=Paper=
{{Paper
|id=Vol-1183/bkt20y_paper03
|storemode=property
|title= The Sequence of Action Model: Leveraging the Sequence of Attempts and Hints
|pdfUrl=https://ceur-ws.org/Vol-1183/bkt20y_paper03.pdf
|volume=Vol-1183
|dblpUrl=https://dblp.org/rec/conf/edm/ZhuWH14
}}
== The Sequence of Action Model: Leveraging the Sequence of Attempts and Hints==
<pdf width="1500px">https://ceur-ws.org/Vol-1183/bkt20y_paper03.pdf</pdf>
<pre>
 The Sequence of Action Model: Leveraging the Sequence
                 of Attempts and Hints
              Linglong Zhu                                    Yutao Wang                              Neil T. Heffernan
  Department of Computer Science                  Department of Computer Science              Department of Computer Science
   Worcester Polytechnic Institute                 Worcester Polytechnic Institute             Worcester Polytechnic Institute
  100 Institute Road, Worcester, MA               100 Institute Road, Worcester, MA           100 Institute Road, Worcester, MA
             lzhu@wpi.edu                              yutaowang@wpi.edu                                 nth@wpi.edu



ABSTRACT                                                               multiple model fitting procedures and showed that there are no
Intelligent Tutoring Systems (ITS) have been proven to be              real differences in predictive accuracy between these two models.
efficient providing student assistance and assessing their                  However, little attention is paid to the data generated when
performance when they do their homework. Researchers have              students interact with computer tutors. Shih, Koedinger, and
analyzed how students’ knowledge grows and predict their               Scheines (2010) utilize Hidden Markov Model clustering to
performance from within intelligent tutoring systems. Most of          discover different strategies students used while working on a ITS
them focus on using correctness of the previous question or the        and predict learning outcomes based on these strategies. Their
number of hints and attempts students need to predict their future     work is based on a dataset that consists of a series of transactions
performance, but ignore the sequence of hints and attempts. In         and each transaction is a <Student, Step, Action, Duration> tuple.
this research work, we build a Sequence of Actions (SOA) model         This model takes into account both students’ action, attempt or
taking advantage of the sequence of hints and attempts a student       help request, and action duration. The experimental results of
needed for the previous question to predict students’ performance.     their Stepwise-HMM-Cluster model shows that persistent
A two step modeling methodology is put forward in the work and         attempts lead to better performance than hint-scaffolding strategy.
is a combination of Tabling method and the Logistic Regression.        Some papers have shown the value of using the raw number of
We compared SOA with Knowledge Tracing (KT) and Assistance             attempts and hints. In fact, the National Educational Technology
Model (AM) and combinations of SOA/AM and KT. The                      Plan cited Feng, Heffernan, and Koedinger’s work (2006) and the
experimental results showed that the Sequence of Action model          User Modeling community gave it an award for best paper for
has reliably better predictive accuracy than KT and AM and its         showing that the raw number of hints and attempts is informative
performance of prediction is improved after combining with KT.         in predicting state test scores. Wang and Heffernan (2011) built
                                                                       an Assistance Model (AM) and generated a performance table
Keywords                                                               based on students’ behavior of doing the previous question.
Knowledge Tracing, Educational Data Mining, Student Modeling,          Hawkins et al.(2013) extended AM by looking at students’
Sequence of Action, Assistance Model, Ensemble.                        behavior for the two previous questions.
                                                                             These educational data mining models that utilize the
1. INTRODUCTION                                                        number of assistance students request and the number of attempts
One of the student modeling tasks is to trace the student’s            they make to predict students’ performance have ignored the
knowledge by using student’s performance. Corbett and Anderson         sequencing of students’ interaction with ITS. Consider a thought
(1995) put forward the well-known Knowledge Tracing (KT)               experiment. Suppose you know that Bob Smith asked for one of
based on their observation that the students’ knowledge is not         the three hints and makes one wrong answer before eventually
fixed, but is assumed to be increasing. KT model makes use of          getting the question correct. What if someone told you that Bob
Bayesian network to model students’ learning process and               first made an attempt then had to ask for a hint compared to the
predicate their performance.                                           first requesting a hint and then making a wrong attempt. Would
                                                                       this information (whether he started with an attempt or a hint) add
     A variety of extensions of KT model are put forward in            value to your ability to predict whether Bob will get the next
recent years. Baker, Corbett, and Aleven (2008) build a contextual     question correct? We suspected that a student who first makes an
guess and slip model based on KT that provides more accurate           attempt tends to learn by himself and has higher probability to
and reliable student modeling than KT. Pardos and Heffernan            master the knowledge and answer the next same question correct.
extends KT four parameters model to support individualization
and skill specific parameters and get better prediction of students’        In our previous work, we showed a Sequence of Action
performance. Qiu and Qi et al. find that forgetting is a more likely   (SOA) model that made use of information about the action
cognitive explanation for the over prediction of KT when               sequence of attempts and hints for a student in previous question
considering the time students take to finish their tasks.              better predicted the correctness of a current question.. We
                                                                       reported experimental results of an improvement upon the KT
     Alternative methods to KT model have been developed. For          model. However, we later found a mistake in that experiment. So
example, in order to generate adaptive instructions for students,      this paper serves as a correction of the previous results and as a
Pavlik Jr., Cen, and Koedinger (2009) put forward the                  formal presentation of the SOA model to the community. We
Performance Factor Analysis (PFA) model that can make                  present the SOA model and compare it to the KT model and the
predictions for individual students with individual skills. Gong,      Assistance model, as well as the combined models to see if
Beck, and Heffernan (2010) compared KT with PFA using                  knowing sequence of action information does improve upon a
standard Knowledge Tracing model, or even upon knowing                 problem PRAQZPN, he made one wrong attempt before making
number of hints and number of attempts alone.                          the correct answer and its action sequencing is ‘aa.’
     The raw data and experiment result is available online:
https://sites.google.com/site/assistmentsdata/projects/zhu2014.
1.1 The Tutoring System and Dataset
The data we used originated from the ASSISTments platform, an
online tutoring system for K12 students that gives immediate
feedback to teachers, students, and parents. The ASSISTments
gives tutorial assistance if a student makes a wrong attempt or
asks for help. Figure 1 shows an example of a hint, which is one
type of assistance. Other types of assistance include scaffolding
questions and context-sensitive feedback messages, known as
“buggy messages.”




                                                                             Figure 2. Students’ action records in ASSISTments

                                                                            We used data from one Mastery Learning class. Mastery
                                                                       Learning is a strategy that requires
                                                                                                       . students to continually work
                                                                       on a problem set until they have achieved a preset criterion
                                                                       (typically three consecutive correct answers). Questions in each
                                                                       problem set are generated randomly from several templates and
                                                                       there is no problem-selection algorithm used to choose the next
                                                                       question.
                                                                            Sixty-six 12-14 year-old, 8th grade students participated in
                                                                       these classes and generated 34,973 problem logs. We only used
                                                                       data from a problem set for a given student if they had reached the
     Figure 1. Assistance in ASSISTments. Which is first:              mastery criterion. This data was collected in a suburban middle
            asking for a hint or make an attempt?                      school in central Massachusetts. Students worked on these
                                                                       problems in a special “math lab” period, which was held in
      Figure 1 shows a student who asked for a hint (shown in          addition to their normal math class.
yellow and also indicated by the. button says “Show hint 2 of 4”),           If a problem only has one hint, the hint is the answer of the
but it also shows that the student typed in eight and got feedback     problem and is called the bottom hint. After a student asks for a
that this was wrong. Though Figure 1 shows the number of hints         bottom hint, any other attempt is meaningless because he or she
and attempts, interestingly you cannot tell whether the student        already knows the answer. In the experiment, we only consider
asked a hint first or made an attempt first. This paper’s argument     the problem logs that have at least two hints. And the answer will
is that information is very important.                                 be marked as incorrect if students ask for a hint or the first attempt
     ASSISTments records all the details about how a student           is incorrect. Moreover, we excluded such problem logs where: 1)
does his or her homework and tests from which scientists can get       students quit the system immediately after they saw the question
valuable material to investigate students’ behavior and their          and the action logs were blank ,or 2) after they requested hints,
learning process. These records include the start time and end         but did not make any attempts and no answer was recorded.
time of a problem, the time interval between an attempt, if he or            Here we only consider the question pairs that have the same
she asks for a hint, the number of attempts a student makes, the       skill and skills having only one question were removed because
number of hints a student asks for, as well as the answer and result   they do not help in predicting. Questions of the same skills were
for each attempt a student makes.                                      sorted by start time in ASSISTments. We split equally 66 students
     Figure 2 shows an example of a detailed sequence of action        into six groups, 11 students in each, to run 6-fold cross validation.
recorded by the system. The row in blue means that the answer is       We trained the SOA model and the KT model on the data from
correct, the row in red means that the answer is wrong, and the        five of the groups and then computed the prediction accuracy on
row in orange means the student asked for a hint. We can see that      the sixth group. We did this for all six groups.
this student answered correctly on his first attempt for the first
problem PRAQM5U. The sequence of action is ‘a’ (‘a’ represents         2. INDIVIDUAL MODELS
an attempt). For the second problem PRAQM2W, he asked three
hints continuously before making the correct answer. The
                                                                       2.1 KT
                                                                       Knowledge Tracing (KT) is one of the most common methods
sequence of action is ‘hhha’ (‘h’ represents a hint). For the third
                                                                       that are used to model the process of student’s knowledge gaining
problem PRAQM2F, he alternatively asked for hints and made
                                                                       and to predict students’ performance. The KT models is an
attempts, and the sequence of action is ‘hahaha’. For the last
                                                                       Hidden Markov Model (HMM) with a hidden node (student
knowledge node) and an observed node (student performance             and some students kept trying many times. Some students asked
node). It assumes that a skill has four parameters; two knowledge     for hints and made attempts alternatively and we believe they
parameters and two performance parameters. The two knowledge          were learning by themselves. In the data, there are 217 different
parameters are: prior and learn. The prior knowledge parameter is     sequences of actions. Intuitively, students’ actions reflect their
the probability that a particular skill was known by the student      study attitude and this determines their performance. Based on the
before interacting with the tutor. The learn parameter is the         assumption that students who make more attempts tend to master
probability that a student transits from the unlearned state to the   knowledge better than students who ask for more hints, we
learned state after each learning opportunity, i.e., after see a      divided them into five categories or bins: (1) One Attempt: the
question. The two performance parameters are: guess and slip.         student correctly answered the question after one attempt; (2) All
Guess is the probability that a student will guess the answer         Attempts: the student made many attempts before finally getting
correctly even if the skill associated with the question is in the    the question correct; (3) All Hints: the student only asked for hints
unlearned state. Slip is the probability that a student will answer   without any attempts at all; (4) Alternative, Attempt First: the
incorrectly even if he or she has mastered the skill for that         students asked for hints and made attempts alternatively and made
question.                                                             an attempt at first; and (5) Alternative, Hint First: the students
      The goal of KT is to estimate the student knowledge from his    asked for hint and made attempts alternatively and asked for a hint
or her observed actions. At each successive opportunity to apply a    first. Table 2 shows the division and some examples of the action
skill, KT updates its estimated probability that the student knows    sequences in each category.
the skill, based on the skill-specific learning and performance            Table 2. Sequence of Action Category and Examples
parameters and the observed student performance (evidence). It is
                                                                       Sequence of Action Category/
able to capture the temporal nature of data produced where                                                             Examples
                                                                                Bin Name
student knowledge is changing over time. KT provides both the
ability to predict future student response values, as well as                One Attempt/Bin ‘a’                           a
providing the different states of student knowledge. For this               All Attempts/Bin ‘a+’             aa, aaa, …, aaaaaaaaaaaa
reason, KT provides insight that makes it useful beyond the scope
of simple response prediction.                                                All Hints/Bin ‘h+’                  ha, hha,…, hhhhhhha
                                                                      Alternative, Attempt First/Bin ‘a-
2.2 Assistance Model                                                                                         aha, aahaaha,…, aahhhhaaa
                                                                                    mix’
Motivated by the intuition that students who need more assistance
have lower probability possessing the knowledge, Wang and               Alternative, Hint First/Bin ‘h-
                                                                                                                  haa, haha,…, hhhhaha
Heffernan (2011) built a purely data driven “Assistance” model to                    mix’
discover the relationship between assistance information and               Notice that each sequence ends with an attempt because in
students’ knowledge.                                                  ASSISTments, a student cannot continue to next question unless
     A parameter table was built in which rows represent the          he or she fills in the right answer of the current problem. In Table
number of attempts a student required in the previous question        2, ‘a’ stands for answer and ‘h’ stands for hint. An action
and columns represent the number of hints the student asked for.      sequence “ahha” means that a student makes an attempt and then
Each cell contains the probability that the student will answer the   asks for two hints before he or she types the correct answer and
current question correctly. The attempts are separated into three     moves on to the next question.
bins: one attempt, small number of attempts (2-5 times), and large
numbers of attempts (more than five attempts). Hints are separated    2.3.1 Sequence of Action Tabling
into four bins: no hint, small number of hints (1, 50%], large        After dividing all of sequence of actions into five categories, we
number of hints [50%, 100%), and all hints where students for all     use a Tabling method, which gets the next percent correct directly
hints. Table 1 shows the parameter table gained from our dataset.     from the training data. For each fold, one table is generated by the
As with Wang and Heffernan’s experimental results, the                tabling method by counting the number of total appearance and
parameter table confirms that students requiring more assistance      the number of next correct of each bin. After counting, a next
to solve a problem probably have less corresponding knowledge.        correct percent is calculated by dividing Next Correct Count by
Table 1. Assistance Model parameter table, average across six         Total Count of Bin.
                           folds                                      Table 2. Next correct percent table of training group of fold 1
                       attempt= 1     0<attempt<6      attempt>=6          Bin           Total        Next Correct        Next Correct
 hint_percent = 0        0.8410          0.7963          0.7808           Name           Count             Count               Percent
0<hint_percent<=.5       0.6286          0.6933          0.6741             ‘a’          22964             19157                0.834
.5<hint_percent<1        0.4494          0.6290          0.6522            ‘a+’          3538              2690                 0.760
 hint_percent = 1        0.4293          0.6147          0.6218            ‘h+’           335               172                 0.513
                                                                          ‘a-mix’        2030              1318                 0.649
2.3 The Sequence of Action Model                                         ‘h-mix’           72               37                  0.513
The Sequence of Action (SOA) model we present takes advantage              Table 3 shows the table computed for fold 1. Tables for other
of the order information about how students make attempts and         folds are similar. From Table 3, we can see that the percent of
ask for hints. Different students have different sequences of         next-question-correct is highest among students only using one
actions. Some students answered correctly only after one attempt      attempt since they master the skill the best. They can correctly
answer the next question with the same skill. For students in ‘a+’       3. MODEL COMBINATION
bin, they are more self-learning oriented, they try to learn the skill   Since the SOA model uses completely different information from
by making attempts over and over again. So they get the second           KT, we expected a potential improvement from combing SOA
highest next-question-correct percent. But for students in the ‘h+’      results with the predictions from KT. We combined models using
category, they do the homework only relying on the hints. It is          two different methods.
reasonable that they don’t master the skill well or they don’t even
want to learn, so their next-question-correct percent is very low.            The first method was simply average the SOA and KT
                                                                         predictions. Presumably, if a group of models have high
     The alternative sequence of action reflects students’ learning      accuracies and uncorrelated errors, we can get lower error by
process. Intuitively, these students have positive attitudes for         averaging them. To compare with the combination of AM model
study. They want to get some information from the hint based on          and KT model, we also computed the average of these two
which they try to solve the next problem. But the results for the        models.
two alternative categories are very interesting. Though students in
these two categories alternatively ask for hints and make attempts,            The second method was a linear regression model with
the first action somewhat decides their learning altitude and final      student performance as the dependent variable. This method takes
results. For students who make an attempt first, if they get the         into account the fact that different models’ predictions may have
question wrong, they try to learn it by asking for hints. But for        different weight in the final prediction. If one of the models is
students who ask for a hint first, they seem to have less confidence     more useful than the other, this method will allow us to learn
in their knowledge. Although they also make some attempts, from          which model should be weighted more heavily. SPSS was used to
the statistics of action sequence, they tend to ask for more hints       train linear regression models. The function for KT and AM is:
than making attempts. The shortage of knowledge or the negative                -0.322+0.639*AM_prediction+0.769*KT_prediction;
study attitude makes their performance as bad as the students
                                                                         The function for KT and SOA is:
asking exclusively for hints first.
                                                                             -0. 004+0. 687*SOA_prediction+0. 321*KT_prediction;
2.3.2 Logistic Regression                                                      We did not combine AM and SOA, because both of them use
In this section, we are going to introduce the second part of the
                                                                         information about hints and attempts. From the functions, we can
SOA model that makes use of a logistic regression model and
                                                                         tell that SOA weights heavier than KT, which indicates that SOA
information we get from the first part of SOA, i.e., tabling
                                                                         is more useful than KT in making a prediction.
method.
      Even though the next correct percentage we get from the
tabling method indicates that the action of sequence can reflect
                                                                         4. EXPERIMENTAL RESULTS
the trend of next correct percentage, the table is very rough and is     4.1 Compare AM, SOA and KT
not intelligent enough to be used to predict students’ performance.           To evaluate how well each of the individual models (SOA,
However, we can use it as a feature in our logistic regression           AM, KT) and the combined models fit the data, we used three
prediction model.                                                        metrics to examine the predictive performance on the unseen test
                                                                         set: Mean Absolute Error (MAE), Root Mean Squared Error
     The dependent variable Next Correct of the logistic
                                                                         (RMSE) and Area Under ROC Curve (AUC). Lower values for
regression model has two states: correct and incorrect. The
                                                                         MAE and RMSE and higher values for AUC indicate better
independent variables are Skill_ID and Credit (the next correct
                                                                         model fit.
percentage generated by the tabling method). Skill_ID was treated
as a categorical factor, while Credit was treated as a continuous        Table 5. Prediction accuracy of KT, SOA, AM and Ensemble
factor. There are totally 51 skills of the data. As mentioned in
before, there are six folds and each fold has their own next correct                              MAE          RMSE           AUC
percentage table.                                                                  AM             0.3007       0.3844        0.5795
     We used Binary Logistic Regression in SPSS to train the
                                                                                  SOA             0.2871       0.3767        0.6786
model. Logistic coefficients are fitted through Expectation
Maximization of at most 20 steps. Parts of coefficients of the first                              0.2939       0.3790        0.6735
                                                                                   KT
fold are shown in Table 4.
   Table 4. Coefficients of logistic regression model of fold 1               LR(AM, KT)          0.2874       0.3759        0.6824
                      Parameters           Value                             LR(SOA, KT)          0.2878       0.3762        0.6813
                  β0(Intercept)           -1.679                                AVG(SOA,         0.2876       0.3757         0.6836
                                                                                   KT)
                  β1,0(skill_id 16)       0.322
                                                                              Table 5 shows values of the three metrics from a six-fold
                  β1,1(skill_id 17)       -0.007                         across validation, which are calculated by averaging
                                                                         corresponding numbers obtained from each validation. As with
                  β1,2(skill_id 24)       20.168                         Wang and Heffernan’s results (Wang & Heffernan, 2011), the
                  …….                     ……                             performance of linear regression combination of AM and KT,
                                                                         called as LR(AM, KT) is better than AM itself, which indicates
                  β1,50(skill_id 371)     0.470                          information about the number of hints and attempts improves the
                                                                         prediction of KT model. Overall, the combination of any two
                  β2(Credit)              3.286                          models have higher prediction accuracy and this is especially true
that for the average ensemble of SOA and KT, called as                Table 8. Student level reliability of difference of KT, SOA and
AVG(SOA, KT), which has better accuracy than the other two                                       Ensemble
combinations. Also, the linear regression of AM and KT has
                                                                                                 MAE       RMSE         AUC
better prediction accuracy than linear regression combination of
SOA and KT. However, from the two tailed paired t-test results                  KT vs SOA       0.0000     0.0000     0.0551
shown in Table 6, the statistical difference between any two pairs             KT vs AVG        0.0000     0.0000     0.0000
of model combinations are not significant.
                                                                               SOA vs AVG       0.0000     0.0698     0.0698
     To examine whether there is significant difference between
these models, we performed a 2-tailed paired t-test. The p values          Note that there is no significant difference of AUC between
are shown in Table 6. We observe that most of the differences         KT and SOA. We interpret these results by pointing out that
between two models are reliable, except for when we compare           RMSE and AUC are metrics that are optimized for measuring
some AM and KT combined models with SOA and KT combined               different things, and so this is quite possible.
models. Both SOA and AM use the information about students’                 Table 9 and 10 shows the skill level result across all 51
actions of hints and attempts. There might be a chance that SOA       skills. From Table 9 we observe a very low AUC value for all the
and LR(AM, KT) have some prediction overlap.                          models, which indicates these models do not make a good
    Table 6. Reliability when compare KT, SOA, AM, and                classification at skill level. The t-test p value with remaining
                           Ensemble                                   degrees of freedom 50 is shown in table 10.

                                       MAE       RMSE      AUC            Table 9. Skill level accuracy of KT, SOA and Ensemble
            AM vs SOA                  0.000     0.000     0.000                                   MAE        RMSE        AUC
             AM vs KT                  0.000     0.000     0.000             KT                   0.3064     0.3762      0.4675
       AM vs LG(AM, KT)                0.000     0.000     0.000             SOA                  0.2942     0.3713      0.4769
       AM vs LR(SOA, KT)               0.000     0.000     0.000             AVG(KT, SOA)         0.3003     0.3710      0.492
      AM vs AVG(SOA, KT)               0.000     0.000     0.000       Table 10. Skill Level reliability of difference of KT, SOA and
                                       0.000     0.000     0.037                                  Ensemble
            SOA vs KT
                                       0.298     0.030     0.083                                  MAE        RMSE        AUC
       SOA vs LG(AM, KT)
                                       0.000     0.001     0.006              KT vs SOA          0.0000     0.0136      0.3492
       SOA vs LR(SOA, KT)
                                       0.020     0.000     0.003              KT vs AVG          0.0000     0.0002      0.0003
      SOA vs AVG(SOA, KT)
                                       0.000     0.000     0.000              SOA vs AVG         0.0000     0.3982      0.0059
        KT vs LR(AM, KT)
                                       0.000     0.000     0.000           The student and skill level analysis generate similar
       KT vs LR(SOA, KT)
                                                                      conclusions, that SOA and ensemble outperform KT in all of the
      KT vs AVG(SOA, KT)               0.000     0.000     0.000      three metrics. When we compare the ensemble model with SOA
  LR(AM, KT) vs LR(SOA, KT)            0.265     0.296     0.469      alone, the result is not so clear.
   LR(AM, KT) vs AVG(SOA,              0.271     0.138     0.079
            KT)                                                       5. DISCUSSION AND FUTURE WORK
   LR(SOA, KT)vs AVG(SOA,                                             In this paper, we put forward a Sequence Of Action model that
                                       0.258     0.001     0.010      makes use of sequence of students attempts to answer questions
            KT)
                                                                      and asking for hints. The SOA model consists of two parts. First,
                                                                      the sequence of students’ actions are divided into five categories.
4.2 Further Analysis for SOA and KT                                   A tabling method shows that students who only make attempts
From the last section, we observed the best model is the              tend to answer the next question more correctly than students who
AVG(SOA,KT) model. In order to better investigate this                only ask for hints. This could be caused by students who make
combination, we ran student level and skill level analysis.           more attempts are trying to figure problems out by themselves and
                                                                      it is an efficient way to master knowledge when they are told the
     Tables 7 and 8 shows the student level result across 66          steps to answer these questions by asking for hints. Second, we
students to account for the non-independence of their actions.        built a logistic regression model with next question correct
Take MAE as an example, for each student; a MAE is calculated         percentage as dependent variable and skill_id, credits of sequence
based on all data available for that student. Then an average value   of action bins as independent variables.
for MAE is computed based on MAE of all students. Table 8
shows the t-test p value for each pair of these three models, where        We conducted six-fold cross validation experiments. The
the remaining degrees of freedom on all the tests is 65.              experimental result showed that SOA had reliably higher
                                                                      prediction accuracy than the Knowledge Tracing model and
 Table 7. Student Level accuracy of KT, SOA and Ensemble              Assistance Model. The average combination of the SOA and KT
                              MAE       RMSE       AUC                had the highest prediction. In sum, the sequence of students’
                                                                      actions provided important information in predicting students’
               KT           0.2939      0.3790    0.6738              performance.
              SOA           0.2871      0.3767    0.6786                   This work is the beginning of utilizing the sequence of
        AVG(KT, SOA)        0.2905      0.3765    0.6811              asking for hints and making attempts recorded by intelligent
tutoring systems to better predict student performance. There are      8. REFERENCES
many open spaces for us to explore. For example, the                     Corbett, A. T., & Anderson, J. R. (1995). Knowledge
experimental data we used is from ASSISTments, does SOA                  tracing: modeling the acquisition of procedural knowledge.
model still makes a big difference if use data from other                User Modeling and User-Adapted Interaction, 4, 253–278.
intelligent tutor systems? How much can the performance of SOA
model be improved if combined with other efficient prediction            Baker, R.S.J.d., Corbett, A.T. & Aleven, V. (2008). More
model such as PFA (Pavlik et al., 2009)? What is the SOA                 Accurate Student Modeling Through Contextual Estimation
model’s performance if we use a student action sequence of               of Slip and Guess Probabilities in Bayesian Knowledge
several previous questions when we train the model? How does             Tracing. In: Wolf, B., Aimeur, E., Nkambou, R., Lajoie, S.
SOA perform after individualization? These are some of the               (Eds.) Intelligent Tutoring Systems. LNCS, 5091, Springer
questions that still need to be explored.                                Berlin. pp. 406-415.
                                                                         Feng, M., Heffernan, N. & Koedinger, K.R. (2006a).
6. CONTRIBUTION                                                          Predicting state test scores better with intelligent tutoring
Predicting student performance is an important part of the student       systems: developing metrics to measure assistance required.
modeling task in Intelligent Tutoring Systems. A large portion of        In Ikeda, Ashley & Chan (Eds.). Proceedings of the Eighth
papers at EDM have focused on this. Many models and                      International Conference on Intelligent Tutoring Systems.
techniques have been used to model and investigate students’             Springer-Verlag: Berlin. pp. 31-40.
performance. However, little attention been paid to the temporally       Gong, Y., Beck, J. & Heffernan, N. (2010). Comparing
sequential actions of student when interacting with the tutoring         Knowledge Tracing and Performance Factor Analysis by
system. To our knowledge we are the first to use the temporal            Using Multiple Model Fitting. In Aleven, V., Kay, J &
sequencing of hints and attempts. It turns out that by paying            Mostow, J. (Eds) Proceedings of the 10th International
attention to this we can better predict student performance. In this     Conference on Intelligent Tutoring Systems (ITS2010) Part
paper, we introduce the Sequence of Action model which makes             1. Springer-Verlag, Berlin. pp. 35-44.
use of the click-stream data of the sequence of making attempts
                                                                         Hawkins, W., Heffernan, N., Wang, Y. & Baker, S,J,d..
and asking for hints when students do their homework using an
                                                                         (2013). Extending the Assistance Model: Analyzing the Use
Intelligent Tutoring System. Students’ actions can be very
                                                                         of Assistance over Time. In S. D'Mello, R. Calvo, & A.
different from each other, but we found there are some useful
                                                                         Olney (Eds.) Proceedings of the 6th International
patterns.
                                                                         Conference on Educational Data Mining (EDM2013).
     We can think of several ways to improve upon this. First,           Memphis, TN. pp. 59-66.
our five bins that we put students into were somewhat arbitrary.
                                                                         Pardos, Z. & Heffernan, N. (2010). Modeling
There could be more bins or fewer. If we use more bins, we might
                                                                         Individualization in a Bayesian Networks Implementation of
have very different predictions. The downside is that for some of
                                                                         Knowledge Tracing. In Paul De Bra, Alfred Kobsa, David
these bins we might not have enough data points to reliably fit the
                                                                         Chin, (Eds.) The 18th Proceedings of the International
parameters. One way to make the model better might be to split
                                                                         Conference on User Modeling, Adaptation and
the “All Hints” bin into one that has “Reached Bottom out Hint”
                                                                         Personalization. Springer-Verlag. pp. 255-266.
and one that is “All hints excluding those that reached the bottom
out.” We could also try to pay attention to features like response       Pavlik, P.I., Cen, H., Koedinger, K.R. (2009). Performance
time between hints or the response time after a hint in making an        Factors Analysis - A New Alternative to Knowledge Tracing.
attempt.                                                                 In Proceedings of the 14th International Conference on
                                                                         Artificial Intelligence in Education. Brighton, UK. pp. 531-
     According to our six-fold cross validation experiments and
                                                                         538.
paired two-tailed t-test, both on student level and skill level, our
Sequence of Action model had reliably higher prediction accuracy         Qiu, Y., Qi, Y., Lu, H., Pardos, Z. & Heffernan, N. (2011).
than KT and AM, the later uses the number of hints students ask          Does Time Matter? Modeling the Effect of Time with
for and the number of attempts students make. Furthermore, we            Bayesian Knowledge Tracing In Pechenizkiy, M., Calders,
combined SOA and KT using average and linear regression                  T., Conati, C., Ventura, S., Romero , C., and Stamper, J.
methods, and the ensemble model’s prediction performance was             (Eds.) Proceedings of the 4th International Conference on
much better than either SOA or KT. We also compared                      Educational Data Mining. pp. 139-148.
combination of SOA and KT with combination of AM and KT.                 Shih, B., Kenneth K., & Richard S. (2010). Unsupervised
The experimental result show that SOA contributes more useful            Discovery of Student Strategies. In Baker, R.S.J.d.,
information than AM alone, which indicates that the sequential           Merceron, A., Pavlik, P.I. Jr. (Eds.) Proceedings of the 3rd
information of action does convey more information about                 International Conference on Educational Data Mining. pp.
students’ learning than the statistics information of actions            201-210.
students make.
                                                                         Wang, Y. & Heffernan, N. (2011). The "Assistance" Model:
                                                                         Leveraging How Many Hints and Attempts a Student Needs.
7. ACKNOWLEDGMENTS                                                       The 24th International FLAIRS Conference. Florida.
We acknowledge funding from NSF (#1316736, 1252297,
1109483, 1031398, 0742503), ONR's 'STEM Grand Challenges'
and IES (# R305A120125 & R305C100024).

</pre>