In this work, we investigated off-policy policy evaluation (OPE) metrics to evaluate Reinforcement Learning (RL) induced policies and to identify critical decisions in the context of Intelligent Tutoring Systems (ITSs). We explore the use of three common Importance Sampling based OPE metrics in two deployment settings to evaluate four RL-induced policies for a logic ITS. The two deployment settings explore the impact of using original or normalized rewards, and the impact of transforming deterministic to stochastic policies. Our results show that Per Decision Importance Sampling (PDIS), using soft max and original rewards, is the best metric, and the only metric that reached 100% alignment between the theoretical and empirical classroom evaluation results. Furthermore, we used PDIS to identify what we call critical decisions in RL-induced policies, where the policies successfully identify large differences between decisions. We found that the students who received more critical decisions significantly outperformed those who received less; more importantly, this result only holds on the policy that was identified to be effective using PDIS, not on ineffective ones.
Intelligent Tutoring Systems (ITSs) are a type of highly
interactive e-learning environment that facilitates learning by providing
step-by-step support and contextualized feedback to individual
students [
A number of researchers have studied the application of existing
RL algorithms to improve the effectiveness of ITSs [
One challenge is a lack of reliable yet robust evaluation metrics for RL policy evaluation. Generally speaking, there are two major categories of RL: online and offline. In the former category, the agent learns while interacting with the environment; in the latter case, the agent learns the policy from pre-collected data.
Online RL algorithms are generally appropriate for domains where interacting with simulations and actual environments is computationally cheap and feasible. On the other hand, for domains such as e-learning, building accurate simulations or simulated students is especially challenging because human learning is a rather complex, poorly understood process. Moreover, learning policies while interacting with students may not be feasible, and more importantly, may not be ethical. Therefore, to improve student learning, much prior work applied offline RL approaches to induce effective pedagogical strategies. This is done by first collecting a training corpus and the success of offline RL is often heavily dependent on the quality of the training corpus.
One common convention is to collect an exploratory corpus by training a group of students on an ITS that makes random yet reasonable decisions and then apply RL to induce pedagogical policies from that training corpus. Empirical study is then conducted from a new group of human subjects interacting with different versions of the system. The only difference among the system versions is the policy employed by the ITS. The students' performance is then statistically compared. Due to cost limitations, typically, only the best RL-induced policy is deployed and compared against some baseline policies. On the other hand, we often have a large number of RL algorithms (and associated hyperparameter settings), and it is unclear which will work best in our setting. In these high-stake situations, one needs confidence in the RL-induced policy before risking deployment. Therefore, we need to develop reliable yet robust evaluation metrics to evaluate these RL-induced policies without collecting new data before being tested in the real world. This type of evaluation is called off-policy evaluation (OPE) because the policy used to collect the training data, also referred to as the behavior policy, is different from the RL-induced policy, referred to as the target policy to be evaluated. To find reliable yet robust OPE metrics, we explored three Importance Sampling based off-policy evaluation metrics.
The second RL challenge is a lack of interpretability of the RL-induced policies. Compared with the amount of research
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). done on applying RL to induce policies, relatively little work has been done to analyze, interpret, or explain RL-induced policies.
While traditional hypothesis-driven, cause-and-effect approaches offer clear conceptual and causal insights that can be evaluated and interpreted, RL-induced policies are often large, cumbersome, and difficult to understand. The space of possible policies is exponential in the number of domain features. It is therefore difficult to draw general conclusions from them to advance our understanding of the domain. This raises a major open question: How can we identify the critical system interactive decisions that are linked to student learning? In this work, we tried to identify key decisions by taking advantage of the reliable OPE metrics we discovered and the properties of the policies we induced.
Just like the fact that assessment sits at the epicenter of
educational research [
In a series of prior studies on a logic ITS, RL and Markov
Decision Processes (MDPs) were applied to induce four different
pedagogical policies, named MDP1-MDP4 respectively, on one
type of tutorial decision: whether to provide students with a
Worked Example (WE) or to ask them to engage in Problem
Solving (PS). In WEs, the tutor presents an expert solution to a
problem step by step, while in PSs, students are required to
complete the problem with the tutor's support. When inducing each
of four policies, we explored different feature selection methods
and used Expected Cumulative Reward (ECR) to evaluate the
RL-induced policies. ECR of a policy is calculated by average
over the value function of initial states and generally speaking,
the higher the ECR value of a policy, the better the policy is
supposed to perform.
Figure 1 shows the ECRs (blue dashed line) of the four
RLinduced policies, MDP1-MDP4 (x-axis) and the empirical results
of student learning performance (the solid red line) of the
corresponding policies. Here the learning performance is measured by
an in-class post-test after students were trained on the tutor with
corresponding policies (the mean and standard errors of
posttest scores are shown with a red solid line). Figure 1 shows that
our theoretical evaluation (ECR) does not match the empirical
results (post-test) evaluation in that there is no clear relationship
between the ECRs' blue line and the corresponding post-test
in red line across the four policies. This result shows that ECR
is not a reliable OPE metric for evaluating RL-induced policy
in ITSs. Indeed, Mandel et al. [
In summary, we make the following contributions:
We directly compared three IS-based policy evaluation metrics against the empirical results from real classroom studies across four different RL-induced policies. Our results showed that PDIS is the best one and its results can align with empirical results.
As far as we know, this is the first study to compare different deployment settings (original/normalized rewards or deterministic/stochastic policy transformation) on ISbased policy evaluation metrics. Our results showed that settings have a direct impact on the effectiveness of the evaluation metrics. Only PDIS with soft-max transformation and the original reward function agreed 100% with the empirical results.
We investigated using information from the RL-induced policies to identify critical decisions to shed some light on the induced policies. As far as we know, this is the first attempt to differentiate the critical decisions from trivial ones.
In recent years, a number of researchers have applied RL to
induce effective pedagogical policies for ITSs [
Iglesias et al. [
In short, prior work on applying offline RL in ITSs primarily explored ECR as the OPE metric and one common phenomenon is that the theoretical evaluation results do not align with the empirical results from real students.
OPE is used to evaluate the performance of a target policy given
historical data generated by an alternative behavior policy. A
good OPE metric is especially important for real-world
applications where the deployment of a bad or inefficient policy can be
costly [
Importance Sampling (IS) [
Some of the prior work on applying RL to induce pedagogical policies used Markov Decision Processes (MDP) frameworks. An MDP can be seen as a 4-tuple hS ;A;T ;Ri, where S denotes the observable state space, which is defined by a set of features that represent the interactive learning environment and A denotes the space of possible actions for the agent to execute. The reward function R represents the immediate or delayed feedback from the environment with respect to the agent's action(s); r(s;a;s0) denotes the expected reward of transiting from state s to state s0 by taking action a. Once hS ;A;Ri is defined, T represents the transition probability where P (s;a;s0) = P r(s0js;a) is the probability of transitioning from state s to state s0 by taking action a and it can be easily estimated from the training corpus. The optimal policy for an MDP can be generated via dynamic programming approaches, such as Value Iteration.
This algorithm operates by finding the optimal value for each (1) (2) state V (s), which is the expected discounted reward that the agent will gain if it starts in s and follows the optimal policy to the goal. Generally speaking, V (s) can be obtained by the optimal value function for each state-action pair Q (s;a) which is defined as the expected discounted reward the agent will gain if it takes an action a in a state s and follows the optimal policy to the end. The optimal value function Q (s;a) can be obtained by iteratively updating Q(s;a) via equation 1 until convergence: Where is the deterministic policy that maps a given state into an action. In the context of an ITS, this induced policy represents the pedagogical strategy by specifying tutorial actions using the current state.
The following terms will be used throughout this paper.
H = s1 a1;r!1 s2 a2;r!2 s3 a3;r!3 sL; denotes one studentsystem interaction trajectory and HL denotes a trajectory with length L.
G(HL) = PtL=1 t 1rt is the discounted return of the trajectory HL, which generally reflects how good the trajectory is supposed to be.
D = fH1; H2; H3; :::; Hng denotes the historical dataset containing n student-system interaction trajectories. b denotes the behavior policy carried out for collecting the historical data D. e denotes the target policy to be evaluated.
( e) represents the estimated performance of e.
Importance Sampling (IS) is an approximation method that allows the estimation of the expectation of a distribution, p, from samples generated from a different distribution, q. Suppose that we have sample space x and random variable f (x) which is a measurable function from sample space x to another measurable space. We want to estimate the expectation of f (x) over a strictly positive probability density function p(x). Suppose also that we cannot directly sample from distribution p(x), but we can draw Independent and Identically Distributed (IID) samples from probability density function q(x) and evaluate f (x) for these samples. The expectation of f (x) over probability density function p(x) can be calculated as: (7) (8) (9) where p is known as the target distribution, q is the sampling distribution, Ep[f (x)] is the expectation of f (x) under p, p(x)=q(x) is the likelihood ratio weight and Eq[f (x)p(x)=q(x)] is the expectation of f (x)p(x)=q(x) under q. We can then approximate the expectation of f (x) over probability density function p(x) using the samples drawn from probability density function q(x). In the context of OPE, the target distribution, p, is a probability event whose density function is determined by the target policy, and the sampling distribution, p(x), is a probability event whose density function is determined by the behaviour policy. Following the general IS technique, we approximated the expected reward of the target policy using the relative probability of the target and behavior policies. Because of the nature of the underlying MDP, samples in RL are sequential. Therefore, we assumed that each trajectory is a sequence of events whose probability density function is determined by its corresponding policy. Assuming the independence among trajectories and following the multiplication rule of independent events, the probability of occurrence of a trajectory, HL, under policy is
L P (HL) = P1(s1)Y t=1 (atjst)PT (st+1jst;at) (6) where P (HL) is the probability of occurrence of the trajectory HL following a policy , P1(s1) is the probability of occurrence of the state s1 in the beginning of the trajectory and PT is the state transition probability function.
Similar to the original IS technique, the proportional probability of each trajectory occurring under the target policy, e, and behavior policy, b, is used as the likelihood ratio weight for that trajectory. Thus, following the importance sampling technique, the importance sampling discounted return is defined as: Substituting G(HL) with the discounted return and we have:
L L IS( ejHL; b) = (Y e(atjst) )(X t 1rt)
t=1 b(atjst) t=1 After having the individual IS estimator for each trajectory, we can calculate the expected reward of the dataset D by averaging the individual IS estimators for each trajectory as
IS( ejD) =
nD 2 Li 1 X4(Y nD i=1 t=1 b(aitjst)
Li 3 e(aitjsiit) )(X t 1rti)5 t=1 where sit, ait, and rti refer to the ith trajectory at time t and nD is the number of trajectories in D.
IS estimator, a biased but consistent estimator which has lower variance than IS. It normalizes the IS in order to produce a lower variance. At first, it calculates the weight WD for the dataset as the summation of the likelihood ratios for each trajectory as shown in equation 11. Then, it normalizes the IS estimator as shown in equation 12. Finally, the WIS is simply the weighted average of the estimated reward for each sequence in the dataset D, as shown in equation 13.
nD Li WD = X(Y i=1 t=1 b(aitjst)
e(aitjsiit) ) W IS( ejHL; b) = Per-Decision Importance Sampling (PDIS) is also a variant of IS. Like IS, it is also unbiased and consistent. IS has very high variance because for each reward, rt, it uses the likelihood ratio of the entire trajectory. However, the reward at step t should only depend on the previous steps. The variance can be reduced by using the likelihood ratio of the trajectory before step t for the reward rt. It means that the importance weight t for a reward at step t is Q e(ajjsj) . Therefore, the individual j=1 b(ajjsj) PDIS estimator for a trajectory is given in equation 14 and the PDIS for the whole historical dataset D is given in equation 15.
L t P DIS( ejHL; b) = X t 1(Y t=1 j=1 b(ajjsj) e(ajjsj) )rt P DIS( ejD) = 1 XnD nD i=1 t=1
Li t X t 1(Y j=1 b(aijjsij) e(aijjsij) )rti
To evaluate the three IS-based metrics, we explored different deployment settings from two aspects: one is the transformation function to convert the RL-induced deterministic policy to a stochastic policy used in IS-based metrics and the other is reward functions: the original reward function vs. the normalized reward.
As described above, IS-based metrics require the policy to be stochastic but our MDP induced policies are deterministic, i.e. given the current state s, the agent should take the deterministic optimal action a following the optimal policy . To transform the deterministic policy into a stochastic policy, we explore three types of transformation functions: Hard-code, Q-proportion, and Soft-max. The basic idea behind them is: for any given state s, the assigned stochastic probability for an action a should reflect its value of Q(s;a). (16) (17) (18)
(1 " (ajs) = " optimal action otherwise In eqn 16, " is a fixed small probability like " = 0:001 which is assigned to actions with smaller Q-value while 1 " is assigned to the action with the largest Q-value.
(ajs) =
Q(s;a)
Pa0 2AQ(s;a0 ) As shown in eqn 17, the probability to take action a given state s is the proportion of a's Q-value among all possible actions a0 in the state s. Thus, the action which has the highest Q-value is guaranteed to have the highest probability. In practice, because some of the Q-values are smaller than 0, we add a constant to Q-values in the same state so that Q(s;a) 1.
(ajs) =
e Q(s;a)
Pa0 2Ae Q(s;a0 ) Soft-max is a classical function used to calculate the probability distribution of one event over n possible events. In our application, given state s, the soft-max function will calculate the probability of action a over all possible actions using equation 18. The main advantage of soft-max is the output probability range is 0 to 1 and the sum of all the probabilities will be 1 and it can handle negative Q-values. is a weight parameter to the Q-value.
The effectiveness of RL-induced policies is very sensitive to the reward functions. In our application, the range of the reward function is very large [ 200;200], which may cause large variances for IS, especially when a trajectory is long. One effective way to reduce this variance is to use normalized rewards. Therefore, both original rewards and normalized rewards are considered. More specifically, the normalized reward z is defined as: z = maxx(xm)inm(xin)(x) where min and max are the minimum and maximum values of original reward function x and z 2 [0;1].
Figure 2 shows an interface of the logic tutor, which is a
datadriven ITS used in the undergraduate Discrete Mathematics
(DM) course at a large university. It [
Four MDP policies, MDP1 - MDP4 were induced from an
exploratory pre-collected dataset following different feature
selection procedures. The detailed descriptions on our policy
induction process are described in [
In short, despite the fact that ECR showed that our four RLinduced policies should be more effective than the Random, empirical results showed otherwise because of various potential reasons. So we explored other OPE metrics to evaluate MDP1 - MDP4.
Our dataset contains a total of 450 students' interaction logs involved in the strictly controlled studies mentioned above. The goals of our experiment were to: 1) investigate whether any of the three IS-based metrics can be used to align the theoretical and empirical results for our four MDP policies and 2) identify critical decisions that are linked to student learning.
We will describe how we determine whether or not the three ISbased metrics can align the theoretical results with the empirical results for MDP1-MDP4.
For a given RL-induced policy , we first split all students into High vs. Low based on the actual carry-out percentages according to . Since there are only two tutorial choices: WE vs. PS, there is a possibility that each actual decision that the tutor made would agree with the decision according to . For the Random policy, for example, the probability is 50-50. In other words, we can measure each trajectory by the percentage of the tutorial decisions that agree with . If is indeed effective, we would expect that the more the tutorial decisions in a trajectory agree with , the better the corresponding student performance would be. We thus treat all 450 students' interaction log data equally regardless of their original assigned conditions and for each student-ITS interaction log we can calculate a carry-out percentage for using the formula: percentage = NNatgotraele , where Ntotal is the total number of tutorial decisions in the trajectory and Nagree is the number of decisions that agree with . Then, students are divided into High Carry-out (High) vs. Low Carry-out (Low) by a median split on carry-out percentages. Then we empirically evaluate the effectiveness of by checking whether there is a significant difference between the High and Low groups on their post-test scores. Similarly, we compare the High and Low groups' theoretical results. To do so, for each student-ITS interaction trajectory, we estimate its reward by exploring different combinations of the three IS-based OPE metrics with the three policy transformation functions and the two reward functions. More specifically, we treat all the trajectories as generated by Random policy regardless of their original behavior policy. So for each student, we have a total of 18 theoretical evaluations for a given . If is indeed effective and our OPE metric is reliable, we would expect that the more the tutorial decisions in a trajectory agree with , the higher the corresponding theoretical rewards would be and vice versa. Finally, for each of the 18 OPE metric settings, we conduct an alignment test between the theoretical and empirical results on . This is done by comparing the empirically evaluated results and the theoretical rewards using the corresponding OPE metric. More specifically, they are considered to be aligned when: 1. Both empirical and theoretical results were not significant, that is p >= 0:05, or 2. Both results were significant, and the direction of the comparison was the same, that is p < 0:05, and the sign of the t values are both positive or both negative.
All the remaining cases are considered as not aligned. Thus, for each of 18 OPE metrics, we can test whether its theoretical results would align with the empirical results for . Since we have four RL-induced policies, MDP1-MDP4, robust and reliable OPE metrics should align the two types of evaluation results across all four policies.
Next, we will explain how the critical interactive decisions are identified and empirically examined. Note that, there may be critical decisions over which the RL policies have no influence. Hence, we focus only on interactive decisions that are critical. For many RL algorithms, the fundamental approach to induce an optimal policy can be seen as recursively estimating the Qvalues: Q(s;a) for any given state-action pair until the Bellman equation is converged. More specifically, Q(s;a) is defined as the expected discounted reward the agent will gain if it takes an action a in a state s and follows the corresponding optimal policy to the end. Thus, for a given state s, a large difference between the values of Q(s;\P S") and Q(s;\W E") indicates that it is more important for the ITS to follow the optimal decision in the state s. We, therefore, used the absolute difference between the Q-values for each state s to identify critical decisions. Our procedure can be divided into two steps:
for each state, we calculated the absolute Q-value difference between the two actions (PS vs. WE) associated with it. Figure 3 shows the Q-value difference (y-axis) for each state (x-axis) sorted in descending order for MDP1-MDP4 policies respectively. It clearly shows that, across the four MDP policies, the Q-value differences for different states can vary greatly. We used the median Q-value difference to split the x-axis states into critical vs. non-critical states. The states with the larger Q-value differences were critical states and the rest were non-critical ones. For a given RL-induced policy, the critical decisions are defined as those in critical states where the actual carried-out tutorial action agreed with the corresponding policy.
RL-induced policies, we counted the number of critical decisions that each student encountered during his/her training. Then for each policy, students were split into: More vs. Less groups, by a median split on the number of critical decisions experienced in the training process. A t-test was conducted on the post-test scores of More vs. Less groups to investigate whether the students with More critical decisions would indeed perform better than those with Less.
Table 1 shows the empirical evaluation results by comparing the High vs. Low carry-out groups' post-test scores based on the corresponding RL-induced policy. The motivation is that, if a policy is indeed effective, students in the High group should significantly out-perform their Low peers on the post-test. In Table 1, the first column indicates the name of the RL-induced policy; columns 2 and 3 show the mean and standard deviation of classroom post-test scores for High and Low carry-out groups, respectively, and the last column shows the t-test results when comparing post-test between groups. Rows 2-4 show that there is no significant difference between the High vs. Low groups in terms of post-test scores for MDP1, MDP2, and MDP3 policies, but there is a significant difference between the two groups for the MDP4 policy (row 5): t(448) = 2:19;p = :029. This result suggests that, among the four MDP policies, only MDP4 seems to be effective in that the students in MDP4's High carry-out group performed significantly better than those in the Low carry-out group.
Table 2 shows the overall IS-based metrics evaluation results showing the impact of policy transformations and original versus normalized rewards on the outcomes of each IS metric. The first column indicates the type of policy transformation applied and the second column shows whether the rewards are normalized.
The third through fifth columns show the performance of each IS metric, where performance is determined by the percent alignment between the IS policy predictions and empirical post-test results. Among the three policy transformations, Q-proportion is the worst, with none of its six performance results better than the corresponding results of Soft-max or Hard-code. Hard-code performs slightly better than Soft-max in most cases but never reaches a 100% match. For reward normalization, the original reward performs better than the normalized reward for the PDIS metric, but there was no effect on IS or WIS.
Comparing the three IS metrics, PDIS shows the greatest performance with all 12 of the performance results being the best.
For the last two metrics, IS and WIS, the results are exactly the same because WIS is much like multiplying IS by a constant and this kind of re-scale won't change the result of the t-tests. The metric with the best performance is PDIS with Soft-max policy transformation and the original reward, whose performance is 100%. This means that all the t-test results on the PDIS predictions aligned with those on the empirical results Electric-blue cells denote that the theoretical t-test results align with the empirical t-test results (Column 3); Grey cells denote misaligned t-test results. in terms of significance match.
Table 3 shows the detailed results for metric evaluations using the original reward, providing t-test results when comparing post-test results between High and Low carry-out groups. The first column in table 3 shows the type of policy transformation functions applied. The second column shows the four MDP policies considered when splitting the dataset into High and Low carry-out groups. The third column shows the t-test results of the empirical evaluation of High vs. Low carry-out, which served as the ground truth. The fourth and fifth columns show the t-test results for the prediction of the three IS metrics: IS, WIS, and PDIS respectively. In the tables, electric-blue cells denote that the theoretical t-test results align with the empirical t-test results (Column 3) while grey cells denote mismatched t-test results. From this table, we can see that only PDIS with soft-max transformation and the original reward results in all four t-tests aligning with the corresponding empirical results.
IS and WIS are more likely to predict the significant difference between High vs. Low. Meanwhile, Q-proportion tends to cause the metric to predict more significant difference, while Hard-code tends to predict less.
In summary, when comparing groups in the RL-induced policy, our results showed that for the MDP4 policy, the students in the High carry-out group significantly outperformed the students in the Low group, but no significant difference was found in the other three policies. This suggests that the MDP4 policy is an effective policy in that the more it is carried out, the better it performs. However, the partially carry-out situation reduced the power of the MDP4 policy so that it did not significantly outperform the baseline random policy. When comparing the empirical evaluation results with theoretical evaluation results,
PDIS is the best among the three IS-based metrics, reaching MDP1 MDP2 MDP3 MDP4 100% agreement. Our results suggested that proper deployment settings have an impact on the performance of IS-based metrics.
When transforming the deterministic policy to stochastic policy, soft-max is the best one, while Q-proportion is the worst, and Hard-code is stable. The comparison between the original reward and normalized reward indicates that the original reward can better reflect the empirical results despite having larger variance.
Recall that each policy impacts its own critical decisions: those with higher differences between Q-values for possible decisions are considered to be critical, and we split each group of students according to whether the student received More decisions aligned with the critical decisions. Table 4 shows the t-test results comparing the post-test scores between the More vs. Less critical decisions groups. The first column indicates the MDP policies considered when identifying the critical decision. The second and third columns show the average post-test scores of students in the More and Less groups, showing mean(sd). The fourth column shows the t-test results when comparing the post-test scores of the More and Less critical decisions groups. The MDP1 row shows a significant difference between the More vs. Less critical decisions groups for the MDP1 policy, with t(448) = 1:97;p = :049. However, the students in More group perform worse than the Less critical decisions group. The MDP2 and MDP3 rows show that there is no significant difference between the two critical decisions groups in terms of post-test scores for the MDP2 or MDP3 policies. Finally, the MDP4 policy shows a significant difference between the two groups, t(448) = 2:10;p = :036, which means that students with More critical decisions performed significantly better than students with Less. For the MDP4 policy, the identified critical decisions comprised 25% of all decisions. This shows that, although critical decisions are a small proportion of all decisions, they can significantly impact the outcome. Also, the results also show that the Q-values in the MDP4 policy can be used to identify critical decisions aligned with empirical results, but the other three cannot. Based on the results from Table 1, MDP4 was identified as the only effective policy, since its empirical post-test results aligned, with students in the High carry-out group performing significantly better than the Low carry-out group. The critical decisions results suggest that MDP4 is also the only policy where larger differences in Q-values had larger impacts on post-test results. Taken together, these results suggest that only Q-values in effective policies work to influence decisions that impact actual post-test performance. This further inspires us to investigate whether we could verify the effectiveness of a policy in reverse: Given a policy, if the decisions with larger Q-value difference are significantly linked to the student performance, then this policy may be more likely to be effective.
In this work, we explored three IS-based OPE metrics with two deployment settings in a real-world application. Through comparing the effectiveness of four RL-induced policies empirically and theoretically, our results showed that PDIS is the best one for interactive e-learning systems and appropriate deployment settings (i.e., where policy decisions are carried out) are required to achieve reliable, robust evaluations. We also proposed a method to identify critical decisions by the Q-value differences in a policy. In order to verify our method, we investigated the relationship between the number of identified critical decisions and student post-test scores. The results revealed that the identified critical decisions are significantly linked to student learning, and further, that critical decisions can be identified by an effective policy but not by ineffective policies. In the future, we will apply the PDIS metric with soft-max transformation and original rewards to help us induce better RL policies which further improve students' learning in the ITS. Also, when inducing policies, we will consider constraints to avoid the partially carry-out situation that limits the impact a policy can have on outcomes. Furthermore, with the identification of critical decisions, we can reduce the size of resulting policies still further by focusing only on the most monumental decisions rather than meaningless ones.