Predicting student performance has been a major task in student modeling. Speci cally, in open-ended domains such as computer science classes, the student submissions contain more information, however they also require more advanced analysis methods to extract this information. Traditional student modeling approaches use knowledge components (KCs) to predict a student's success on speci c practiced skills. These approaches are useful and necessary in helping learning environments like Intelligent Tutoring Systems (ITS) personalize feedback, hints, and identify struggling students. However, when working with programming data, code features provide more information than skill tags representing KCs, and this information is not leveraged by traditional KC models. This work incorporates an implicit representation of KCs into a student model by including features extracted from students' code with data from an undergraduate introductory programming course. This representation is then evaluated by using deep learning predictive models and investigated to see how well they are able to leverage code features to model student knowledge and compare and contrast against other learning models. The study shows a modest, but consistent improvement in models that use time-sequential data with even the simplest code features, implying that these aspects may improve student modelling.
Modeling student learning activities can improve their
learning outcome, and automatically achieving this requires the
system to model student's knowledge through tracking
Knowledge Components (KCs) [
This standard BKT model has been advanced by a lot of
newer models such as Deep Knowledge Tracing (DKT) [
We compare DKT and BKT models in our work, along with
classical deep learning models such as multi-layer perceptron
(MLP) and Long-Short Term Memory (LSTM) to see if code
features improve the ability of KT models, and if it also
has a positive e ect on static or non-time-sequential models.
While skill IDs are hardly available for BKT without manual
labeling, we use the problem or assignment IDs to serve as
the KCs there [
While compared in our work, these standard KT models also
have two important limitations when adding code features
in: 1) Need extra accommodation standard KT models use
correctness of KCs as features, while they do not accept code
features with the original structure. There are some recent
works adding textual information in KT models [
We build preliminary KT models to predict whether novice university CS students struggle with the integration of code features. To verify whether code features can improve the KT models, we experiment with term frequency-inverse document frequency (TF-IDF), simply to show possible e ects of the usage of code features in this task, and compare with other baseline models. To address the aforementioned limitations of classical KT models, we de ne our research question as: Do code features improve the ability to model student struggle pattern in CS in time-sequential and non-timesequential KT models? Our results suggest that simple code features coupled with time sequential information have the potential to improve the KT model performance on student struggle prediction tasks.
The data used in this work is called CodeWorkout1 and is from an undergraduate programming course at a large, public university in the southeast United States, collected in Spring 2019. The dataset includes code submissions from 5 assignments, 50 problems, and 413 students, recording students' attempts in implementing functions in Java. Each assignment contains 10 problems. A student could submit any number of attempts for a problem, each attempt's code is tested against given test cases and given a score from 0 to 1 based on how the code tested. Our statistics show that not all 413 students work on every problem and most students eventually get a full score (score = 1). Assignments are ordered by time stamp, and problems are grouped 1https://pslcdatashop.web.cmu.edu/Project?id=585 in each assignment. Di erent assignments covered di erent concepts, according to their problem descriptions, and the complexity of problems increases over time. The dataset also provides the descriptions of the problems. For example, early problems include simple concepts such as if conditions, while a later problem in the same assignment may ask about nested-if conditions.
Each assignment is a combination of multiple di erent concepts, with at least one new concept or advanced concept being introduced for each assignment. The entire course covers a wide range of introductory concepts including conditional statements, loops, strings, and arrays, which suggests that the knowledge tracing task across all problems is a non-trivial task.
The traditional goal of KT is to model knowledge to predict student performance. Historically, this has been interpreted as whether or not a student gets a problem correct. However, in many programming classes, including the one analyzed here, most students (at least 84% of students in this dataset) got a full score in the problems they attempt. So, it is more important to predict student struggle instead of success. In this programming setting, students can use multiple attempts and nally succeed. This allows us to use their number of attempts as a quantitative measure for their struggle. We de ne a struggling student as someone who uses more attempts than at least T % of the students. In order to nd the threshold for T , we explored the data to nd the division of two relatively separate groups of students corresponding to those who required a lot of e ort to complete the problem, and those who didn't struggle. We found that the 75th percentile of attempts for each problem is a good division of attempts for most problems. A student is identi ed as struggling on the problem if they: 1) didn't pass all the test cases or 2) passed all test cases but took more than the 75th percentile of attempts for that particular problem. So a student who scored 1 on a problem might have also struggled with that problem. The prediction that a student is struggling is the positive class which is also the minority class.
We use simple code features in the form of Term FrequencyInverse Document Frequency (TF-IDF) weights, a text analysis technique that represents documents based on its term frequency. The algorithm uses Term Frequency (TF) to nd the frequency of a word in a document and normalizes these weights with Inverse Document Frequency (IDF), the frequency of the word in the corpus. The nal weights re ect the importance of terms for a particular document. This method is more commonly found in work involving processing text rather than code. However, the novice programming concepts present in this data can be partially represented by keywords since they are focused on familiarizing the students with syntax. More complex concepts, such as stacks and queues are not part of the course curriculum. The vocabulary used in our work consists of keywords in Java such as 'for', 'if', and 'public' that represent some novice concepts a beginner programming course might include. Other keywords such as 'throw' and 'extends' are also included to identify students who might have had a background in programming. These students may not struggle in the problems in this course compared with other students. The models in this experiment predict at the problem level, so each problem done by a student is represented as one input. We de ne a corpus as the attempts across all students who worked on problem p 1 and use the TF-IDF vector of the last attempt a student made on this problem as code feature input. In this way, the frequencies of keywords for each attempt in a problem are calculated and the TF-IDF weights for the keywords in a student's best attempt are included in the code features.
Although other, more complex extraction methods such as code2vec, ASTNN, and pq-grams exist that extract deeper code and structural information, we use TF-IDF for this exploratory study. Our goal is not to use complicated vectorization approaches, but rather to examine whether a simple yet e ective approach can represent information before moving on to more complex approaches. TF-IDF ts this goal because it not only simpli es model architecture, but it puts more weight on relatively rarer keywords and may help in identifying concepts in a problem.
This experiment uses 4 sets of models: 1.) Standard BKT
model [
Attempt-Score Group: The number of attempts the student made (with any score) and the maximum score achieved on unit tests on any attempt for problems 1 to p 1. Code Feature Group: The TF-IDF features of the last attempt at the problems 1 to p 1 (TF-IDF weights). Because of the lack of IDs directly representing KCs in the data, we experiment with three di erent level of IDs (Problem ID, Assignment ID, Uniform ID). The BKT model uses the Uniform ID along with the labels of past struggle, assuming that one ID represents all the KCs in the dataset. The DKT set of models are compared using features across the entire ID-Label group.
The Attempt-Score group is used in the MLP and LSTM based models. Because of these features (Attempts and Score), these models are named Attempt-Score MLP (ASMLP) and Attempt-Score LSTM (AS-LSTM). The Code Feature group is used for MLP, DKT, and LSTM models. We refer all models that don't use TF-IDF features as baseline models. 2.3
2.3.1 BKT
We used the pyBKT implementation of the standard BKT
model [
The general DKT model is designed to process time series data (i.e. multiple problem attempts over time) and uses LSTM networks which use a recurrent structure to serve the target. When predicting the label for student s and problem p, the input sequence includes the features for problems 1 to p 1 done by the student s. We used a xed input sequence length of 50, the maximum number of problems a student can do, and a padding of 0's when needed. The padding makes sure that the input is always a xed length. In addition to ID and/or TF-IDF features, the DKT models require student representation to measure the student performance, in this case, student struggle. We refer to this representation as the struggle feature where the model identi es the student's struggle on past problems. In both DKT and BKT sets of models, this struggle feature is binary because it represents whether or not a student struggled in the past problems.
In the DKT models without code features, models incorporate the struggle feature for problems 1 to p 1 to predict for problem p. There are three of these models with problem ID (D01), uniform ID (D02), or assignment ID (D03). Another two DKT models use code features. Besides the code features, one uses problem ID, and another one uses uniform ID. Both models, D04 and D05, combine TF-IDF vectors (length 63) with the embedded vector of ID and 2https://github.com/seewoo5/KT struggle (length 20) resulting in the LSTM input dimension 63 + 20 = 83.
Attempt and Score MLP (AS-MLP) models with (M02) and without code features (M01) do not take in a sequential input and do not explicitly use the output of one state as input to the next, so they are not time-sequential. However, neural network structure allows MLP to incorporate code features. So, the purpose of these models is to compare the e ect of performance features and code features in a static model on this struggle prediction task. Both AS-MLP models share the same architecture, with three linear layers. M01 uses a batch size of 260, and a learning rate of 0.001 and the AS-MLP model with code features (M02) uses the same batch size with a learning rate of 0.0001.
The inputs for the Attempt-Score-MLP models are less complex than that of any LSTM. Unlike the DKT input structure, the AS-MLP models don't use an empty sequence to predict for the 1st problem a student has attempted. In the AS-MLP model without code features (M01), the input size is two dimensional, including attempts and maximum score of problem p 1, while the other MLP model with code features has an input dimension of 63 + 2 where 63 is the xed length of TF-IDF code features.
We created two Attempt-Score-LSTM (AS-LSTM) models: AS-LSTM with (L02) and without TF-IDF features (L01). These models incorporate performance features (attempt and score) compared to DKT's struggle feature. AS-LSTMs' input is similar to the DKT input, except that AS-LSTM doesn't predict on the 1st problem. In L01 both features represent the student performance on past problems and in L02, these performance features are appended to the TFIDF features. So L01 also inputs a two dimensional array, while L02 inputs a vector sized 63+2. Both AS-LSTM models share the same architecture of one LSTM layer followed by a linear layer. L01 uses a learning rate of 0.001 and a batch size of 260 over 100 epochs. L02 uses a learning rate of 0.000001 and batch size 260 over 250 epochs. We used 8:2 split for data, and performed repeated resampling for AS-MLP, AS-LSTM models, while we use 5 fold cross validation for DKT and BKT models, reporting the average AUC/Recall/Precision and F1 scores in the result section.
TF-IDF features and time-sequential information. Table 1 shows three pairs of models: D01 and D04, D02 and D05, M01 and M02, and L01 and L02 which are all identical except that the latter models (D04, D05, M02, and L02) include TF-IDF features. Before TF-IDF, the DKT models are biased towards label 1, while after adding TF-IDF features, the models are biased towards label 0. So we mainly use AUC, which is more symmetric to bias towards any class, to compare these models' performances in this section. The results in Table 1 show how D04, D05, and L02 can distinguish between classes slightly better than D01, D02, or L01. L02 performs slightly better than D04 and D05 maybe due to its non-binary struggle feature. By looking at the confusion matrices of D04 and D01 in Table 2, it can be calculated that with TF-IDF, the DKT model is 6.2 times more likely to mark an at-risk student as needing help than a non-at-risk student by examining the ratio of FTPP : FT NN . Without TF-IDF, it's only almost twice as likely. Similarly, the confusion matrices of D02 and D05 show that D05 is 5.5 times more likely to mark an at-risk student as needing help while D02 is 3.9 times as likely. Meanwhile, L01 and L02 have a smaller di erence with L01 at about 4.5 times likely to mark an at-risk student as needing help while L02 is about 4.7 times as likely.
While it has been an open question whether the addition
of code features would do so, in our experiments, both L02
and D05 have a range of 3% to 6% improvement in their
AUC. For example, Wang et. al. found that a large number
of code features can lead models to over t to the training
data [
For AS-MLP models, M02 did not improve with TF-IDF features and performed similar to M01 as shown in Table 1. This suggest that time-sequential information is important for KT tasks and that TF-IDF features alone cannot improve model performance.
The current model performances are not ready for implementation in real education settings. This modest performance was predicted because examining the e ect of TFIDF features in modeling student knowledge is a very di cult task due to the open-ended nature of the domain and the wide range of concepts in the dataset. Although the performances of models with TF-IDF features were modest, there is still a 3% to 6% increase in AUC of these models compared to other baselines. This improvement marks potential for more complex code features.
Di culty of Knowledge Tracing for Programming: As hypothesized, this is a di cult KT task because of the wide range of concepts covered throughout the course. As the students progress through the di erent problems and assignments, new concepts are introduced which the models try to infer from either through an ID or student code features. Results across all models in Table 1 show that neither IDs nor code features are particularly successful in representing or inferring KCs. Of all the models in the experiment, BKT is the worst performing model with an AUC of 0.64 which suggests that this particular de nition for student performance (labels) may not be the best t with a standard BKT model. Because there is a lack of tags to directly represent KCs, we use naive ways to represent them in the 3 DKT models. D01 treats each problem as a separate concept, D02 treats each assignment as a separate concept, and D03 treats all 0.33 0.34 0.52 0.74 0.02 0.02 0.07 0.10 0.41 0.44 0.50 0.52 0.01 0.02 0.05 0.00 0.63 0.64 0.69 0.62 0.00 0.00 0.02 0.04 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 concepts the same. As shown in Table 1, they all are biased towards label 1 (struggling student). Further examinations into D01 - D03 from Table 1 show that that they perform similarly, with D03 having the best AUC and D02 with the highest macro F1 score. Because both IDs have similar results, this suggests that neither is better at representing concepts.
Why were DKT and the other KT approaches
unsuccessful?: The original DKT work that the D01 code is based on
states that the algorithm can leverage skill or KC tags but
doesn't need them to perform predictions [
D05
FN = 578 TP = 2251
D02 TP = 888 FP = 1124
FN = 220
TP = 1106
each KC. Where the smallest dataset in [
When code features are added to static models such as ASMLP (M02), there is no improvement in performance. Comparing M02 to the nal time-sequential models, (D04, D05, L02), there is a 7% to 10% increase in AUC suggesting that in the presence of time-series data, the impact of code features is more. However, only one non-time-sequential model is implemented with code features, so it is not clear if these results are robust.
One limitation in this work is that the vocabulary used for TF-IDF may not be generalizable to more advanced curricula. The second limitation is that we only use one type of non-time-sequential and one type of time-sequential models to compare code features. So the results may be speci c to LSTM or MLP based models. In the future, we plan to work with expert-based features like PQ-grams and incorporate other models into our experiment.
In conclusion, in this work, we experimented with models with and without both simple code features and timesequential information. The results show that even these simple code features can a ect model performance and that time-sequential information is important when using these features. These experiments mark the potential that code features have in representing CS KCs over an entire course.