Nation's Report Card Data Mining Competition 2019 (NAEP Competition) aims to understand which students' behaviors are e ective or ine ective during online assessments and to determine how quickly these behaviors can be detected. Speci cally, the ultimate purpose is to be able to classify students as e ective or ine ective based on the logs of their actions on the National Assessment of Educational Progress (NAEP), the largest nationally assessment of what America's students know in various subject areas. To solve this challenge, our proposal is based on modeling the evolution of student behavior throughout the assessment, considering di erent characteristics such as the sequence of activities performed and the order in which they have been carried out. The proposed classi cation model is based on the Long-Short Term Memory (LSTM) recurrent neural network architecture, as it is capable of capturing evolutionary patterns over time. This architecture has been evaluated with the competition dataset and the results obtained are shown, which are very promising.
Due to the great development of online educational platforms, such as Massive
Open Online Courses (MOOCs) or mobile applications, large amounts of
educational data of a very diverse nature are currently being generated: log sequences,
audios, videos, ... [
As a consequence, there is a huge increase in the development of tools based
on Learning Analytics (LA) [
One speci c problem that is attracting the attention of the research
community is the detection of students' behaviour during online assessments through
eye-tracking technology, response time procedures, etc [
Concerning our solution, we conducted an analysis on the competition's
dataset and found important characteristics that potentially classi ed the
students as e ective or ine ective at performing assessments. We observed that the
sequence of activities performed, as well as the order in which they were done,
were good predictors. Accordingly, we suggested to use the Long-Short Term
Memory (LSTM) model, as it is capable of capturing evolutionary patterns over
time. In recent years, the use of such neural networks in the eld of EDM has
increased. For example, in [
The rest of the article is divided as follows. The purpose of the competition and the available dataset are described in Section 2. The Section 3 explains data transformations and feature selection. Then, Section 4 present the architecture of the model used in the competition and its implementation. Finally, in Section 5 the results are shown and some conclusions are outlined. 2
The NAEP Competition [
Accordingly, the nal purpose of the competition was to make a classi er to determine if the students would act e ciently in Block B by having only the sequence of actions performed in Block A.
Furthermore, for this competition, e cient behaviour is de ned as follows: 1. Be able to complete all the problems in block B. 2. Be able to allocate a reasonable amount of time (they said: the minimum possible) to solve each problem, using the 5th percentile as the cut-o . 2.1
The dataset was divided into 6 di erent les, which include the following ones: { data a train.csv: Contains the logs of the actions performed by each student in Block A. It is part of the training dataset, with 1232 students and a total of 438.291 interactions. { data a hidden 10.csv: Contains the actions performed by the students in their rst 10 minutes of activity in Block A. It is part of the test dataset, with 411 students and a total of 47.563 interactions. { data a hidden 20.csv: Contains the actions performed by the students in their rst 20 minutes of activity in Block A. It is part of the test dataset, with 411 students and a total of 110.481 interactions. { data a hidden 30.csv: Contains the actions performed by the students during the rst 30 minutes of activity in Block A. It is part of the test dataset, with 410 students and a total of 143.880 interactions. { data train label.csv: Contains the target variable of students in the training set. { hidden label.csv: Contains the order in which the predictions must be submitted.
The information provided in the rst four les is presented in Table 1. There are 7 di erent attributes, which are described. 3 3.1
As we mentioned previously, the raw dataset contained sequences of actions performed by each student labeled with the timestamp of the moment in which they were performed. In order to extract the desired characteristics to build the classi er, the following transformations were rst performed: { Rows without a timestamp were removed { The problems/items were coded as integers { The 25th, 50th and 75th percentiles1 were calculated, as well as the upper and lower outliers, for the following characteristics:
Time spent by each student for each activity
Time spent for each type of activity 1 Percentiles and outliers were calculated using the training and tests sets
Use of the support functions. According to the study carried out in [
Since we were interested in capturing the evolution of students' behaviour over
time, the model selected was the Long Short-Term Memory (LSTM) [
Speci cally, the architecture we used to carry out the classi er is based on
those proposed by [
Sequences of integers representing the activities performed by each
student in order of completion. The order of completion is determined by the
timestamp of the last action performend in each activity. For example, if the
Input 1 actions were represented by Ax where x determines the number of the activity
and we had the following sequence: A1; A1; A2; A2; A1
the sequence of activities would be [
Input 2 Sequence of integerasctrievpirteyseinnttihneg othrdeepreirncdeinctaitleedfoarbtohvee.time spent on each
Input 3 Sequence of integers that roefptrheseesnutptphoertpefurcnecnttiiolne.for the time spent on each
Input 4 Sequence of integers thabtyretyppreeseonf tatchtieviptye.rcentile of the time used
reduce the dimensionality of the variables and learn a meaningful representation
of the categories in the transformed space. In the case of this competition, we
were looking for the Embedding layer to nd a representation of the di erent
behaviours along the sequences. Therefore, the LSTM layers would be able to
detect patterns over time, taking into account the order of the actions being
performed. Thereafter, the layer that remains to be highlighted is the
GlobalMaxPooling (GMP) layer, which aims to reduce the size of the space in which the
di erent variables are represented. Hence, the number of necessary parameters
is reduced. Furthermore, as detailed in [
Evaluation measures To evaluate the predictions made by the classi ers, the competition organizers used two metrics: the area under the curve (AUC) and Cohen's Kappa, both of which were adjusted. The AUC is a robust metric for evaluating a binary classi er, since it considers the relationship between the false positive and false negative rate according to a discrimination threshold. The aim is to maximize the value of this metric. On the other hand, Cohen's Kappa is a statistical measure that adjusts the e ect of hazard on the classi cation made.
The adjustment made for each of the metrics was the following: AdjustedAU C = (0 2(AU C
if AU C < 0:5 0:5) otherwise AdjustedKappa = (0 kappa if kappa < 0 otherwise
The nal result of the evaluation was calculated using the aggregate score of both metrics:
Concatenate Embedding
Input
Concatenate
Softmax
FC
GMP TimeDistributed
LSTM LSTM
F inal Score = AdjustedAU C + AdjustedKappa Training The classi er was trained following a 5-fold cross-validation process at student-level. Since we worked with a binary classi er, the selected loss function, which we had to minimize during the training, was the binary cross entropy using the Adadelta optimizer and the metrics mentioned above.
In our experiments, we used the BLSTM with forward and backward LSTM
layers with a total of 64 units per layer. The dropout rate was set to 20% and
applied to both the output of the BLSTM and the output of the TimeDistributed
layer. These values were determined by doing a hyperparameter search. Following
the recommendation of [
Finally, as we had to classify the behaviour of the students having limited actions of them ( rst 10 minutes, rst 20 minutes and 30 minutes), we chose to train three di erent models, one for each group. 5
The training results without adjustment are shown in Table 3. As expected, the
results are improving as we have more data available from the students. The
results can be improved, especially for the model of the rst 10 minutes of the
assessment, but we still obtained competitive results, as we were sixth in the
competition. The main problem we faced was under tting. One of the possible
reasons of this under tting may be that we generated few features per student
that summarized their behavior. LSTMs are often used with raw data, allowing
the architecture to discover the features and the relationships between them [
This research is partially funded by the European Regional Development Fund and the National Research Agency of the Spanish Ministry of Science, Innovations and Universities under project grants TIN2017-85179-C3-2-R.