Curriculum-integrated games can provide teachers with data to help them decide when and how to intervene with individual students. Based on our prior work observing teachers using ST Math, teachers may not be able to attend to a dashboard or student screens to determine who might need intervention. We therefore set out to determine how much data we need from the current ST Math gameplay session to predict performance. Based on the available log data that tracks student performance over SETS of puzzles, we performed two experiments to predict performance. The rst uses data from one game level, which is about 3 minutes long, to predict the performance on the next level, and the second uses the rst 6 minutes of gameplay to predict how many levels a student can complete in 20 minutes, a typical class length. Our results show that our data are not negrained enough to allow for paired level prediction, but that 6 minutes of gameplay can be used to rank students in order of performance for a class session. These results can be used as a basis for an alert system that could help teachers prioritize their time in the classroom.
Educational games can be a useful tool for teachers to
provide additional practical learning for students [
In other work, the authors found that when students
possibly need intervention but do not receive it, they might
get frustrated and give up or replay an easier game instead
[
Despite the amount of data gathered with each playthrough, teachers in our system are only provided with a student's current progress in the curriculum and a feature that allows a student to \raise" their hand through the system.
However, this is only visible on the student's screen via a purple hand indicator and often goes unnoticed. Therefore, we sought to determine if there was a way to provide teachers with knowledge regarding students projected progress as fast as possible, so that the teachers can determine who to help from there. With the machine learning techniques that can process such data and help predict outcomes, we wanted to nd the correct technique to answer our question.
Machine learning and educational data mining techniques
have been successfully used in educational game research
for many years [
In this paper, we tried to determine the smallest amount of time needed to predict student outcomes for one gameplay session by investigating multiple feature selection algorithms and prediction models on student gameplay data for an educational game, Spatial Temporal Math (ST Math). We tried two methods of prediction using data analysis and machine learning: 1) Trying to predict student outcomes for playing one level of a game using gameplay data from only the previous level, 2) Using the least amount of time of a student's gameplay data to predict the number of levels they will pass in the next twenty minutes of gameplay. To accomplish this, we tried various machine learning and feature selection methods to nd the most signi cant features needed to predict student outcomes in this educational game. In this study our intention was to give insight to the teachers of ST Math by indicating our best guess for which students
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). could most bene t from teacher intervention on a single day, provided early enough in the gameplay session to allow the teacher to help as many as possible.
ST Math is a curriculum-integrated supplemental
mathematics game for 2nd-4th-grade students that uses spatial
puzzles to teach basic math concepts [
Educational games in classrooms are helpful to teachers
because the students can receive individualized attention and
learning from the game while the teacher gives one-on-one
attention to students who need it.[
Unfortunately, there are many reasons that students who
need help do not receive it. One study found that
middleclass students seek help more directly than working-class
students and end up getting more help as a result[
The task of automatically identifying students who need
help has been explored [
Predictive models are now being integrated into teacher
dashboards or alert systems to enable this aid. In a survey of
K-12 teachers who used intelligent tutoring systems in class,
Holstein et al. found that they were very interested in
having real-time classroom monitoring tools that would help
them decide which students most needed their attention [
The data is collected by MIND Research Institute, who created ST Math. This study was conducted on data from 3rd grade students who played ST Math during the 20162017 school year. The data contains 31 objectives, 154 games, and 669 levels which equals 5,186,269 total level attempts by 8,983 students from 111 schools and 636 teachers. We excluded students who completed objectives not contained in the 3rd grade objectives which removed 21,544 level attempts. These are students who might have been erroneously included from other grades, as the system is used for grades 3-5. For the purposes of our study, we ltered out level attempts where the student only completed one level attempt in our 30 minute gameplay session cuto . This removed 101,849 level attempts, leaving us with a nal dataset of 5,062,876 unique level attempts. The initial data we were provided with had 6 features for each level attempt: STMathID (unique ID for each student), Level, Objective Code, Timestamp, Number of Correct Puzzles, and Total Number of Puzzles. For the purpose of our analysis, we created additional features shown in 1.
We wanted to explore di erent ways of providing teachers with information about student progress, so that they could intervene and monitor progress according to their own classroom goals. Therefore, our intention is to predict the projected progress of a student using the least amount of information and using only same-day gameplay data. Here, we are comparing two methods of data segmentation: predicting the time spent passing the next level based on the time spent passing the current level, and using the least amount of gameplay to predict how many levels would be passed in the next 20 minutes of gameplay. We are attempting to see if a student's performance in the beginning of a gameplay session is a good prediction for their later performance. A gameplay session is de ned as subsequent level attempts that are separated by less than 30 minutes. If two level attempts are separated by 30 minutes or longer, we count the second attempt as a new gameplay session. We decided on a 30 minute cuto because a pause in the game of 30 minutes or more might indicate the student was working on something else in between and we cannot say that the previously played level will have any e ect on the performance of next level. However, we still want to include students who may be truly struggling, receiving help from a teacher or giving help to another student during the playthrough of a level. Only 3.5% of the data for the gap between play sessions was between 30 minutes and 20 hours, while 11.6% of the data was 20 hours or longer between gameplay sessions.
This section details our attempt at using previous level data to predict student outcomes for the next level attempt.
We wanted to see if we can predict how well a student will
do on the next level by using only the data from the
previous level in the prediction. Our aim in this experiment
was to investigate if the features of a single level for a
student can predict whether the student would have needed an
intervention for the next level. This would be bene cial to
teachers because it would give an alert immediately after a
student nishes a level that would tell them that the
student might need help on the next level. We used pairwise
prediction, so the data was grouped into Level A and Level
B, with the constraint that Level A and Level B had to be
in the same objective and the same gameplay session. We
considered Level A to be the rst attempt on a new level, all
subsequent attempts (retries) until the level was passed, and
any replayed levels that happened before or after the level
was passed. Any gameplay that happened between the rst
attempts of two consecutive levels would be counted as Level
A data. Level B is the next attempt on a new level after
Level A, and contains the same information as Level A:
number of attempts, retries, replays before and after passing, up
until the next new level attempt. We expected the
number of attempts to pass a level to provide information about
how many attempts they will need for the next level because
the levels increase in di culty inside objectives and this is
consistent through ST Math. Also, research has shown that
replay that happens before passing a level results in a
negative e ect on performance while replay that happens after
passing a level has been shown to have a positive e ect[
Due to time constraints and the complexity of the feature creation, we used a subset of our total dataset for this analysis. The dataset includes 830 students, and 665 unique objective-level pairs. Objective-level pairs are level pairs within an objective. However, some students did not complete all the objective-levels resulting in a total of 277,975 unique student objective-level pairs. 5.1.1 Pairwise Feature Generation Since the raw data only included gameplay aspects per attempt such as time taken, attempt performance and the kind of attempt (retries, replay, etc.), we engineered 33 additional features for every student objective-level pair. There are 7 attempt categories and 5 metrics per each category. The 7 attempt categories are as follows: overall level attempts, total retry and replay attempts, retry attempts, total replay attempts, total replay attempts before passing the current level, total replay attempts after passing, replay attempts of the same level (current) after passing, and replay attempts of other levels after passing. The 5 metrics for each category are as follows: whether an attempt category occurred (except for overall attempts category), total number of attempts (except for overall attempts category), total time, average time, and average performance.
Next, for each student, we identi ed the consecutive levels
(Level A, Level B ) within each objective that were
completed in the same session. We found a total of 222,258 such
level pairs for 830 students. We explored four di erent ways
to de ne an intervention: if the total time was greater than
the 75th percentile (I-TotalTime), if the average time was
greater than the 75th percentile (I-AvgTime), if the average
performance was less than 25th percentile (I-AvgPerf), and
if the student could not nish a level in the rst attempt
(I-FirstAttempt). Each of these intervention types were
intended to capture a di erent aspect of a student's ability to
complete a level.
5.1.2 Pairwise Feature Selection and Prediction
Models
The analysis was carried out in Python. We normalized the
time-related features and then explored three feature
selection techniques in the scikit-learn[
The feature selection based on LinearSVC with L1 loss and the Random Forest classi er provided the most optimal prediction accuracy. Table 3 shows the results of using a Random Forest classi er for each intervention type. We observed that the average time spent on Level A was selected for each intervention type based target for Level B. The distribution parameters of the features selected are shown in Table 2. We observed that very few features related to the replays were selected. This may be because the consecutive level pair dataset recorded very few rows with retries or replays (18.20%) and even lower just considering replays (0.12%). Such a high degree of sparsity in replay made any replay related features not signi cant enough to contribute towards the predictions. Another interesting observation is that the performance over all the attempts (avgPerfTotal) in Level A was not a signi cant predictor of the intervention for Level B in any of the models primarily because of the small variance recorded for this feature. The small variance is due to the granularity of the data only recording passed level attempts with failed puzzles as 100% performance. On the other hand, the average time in Level A (avgTimePerLevel) was a signi cant predictor of the intervention for Level B for every intervention type based target. We recorded few replays in general and a low variance in the performance related features, so only the time related features were varied enough to capture the relationship between Level A and Level B. The results suggest that the average time spent on an attempt in a level is the most signi cant predictor of whether a student may need assistance in the next level. However, the classi er models for each intervention type did not perform signi cantly better than a baseline classi er that would predict all the observations to be the Majority Class (the class containing more students) as shown in Table 3. This suggests that the relation between the behavior of students in two consecutive levels may be highly varied and that it is di cult to generalize whether an intervention is needed in a level based on only one previous level. It may also suggest that such a prediction may be dependant on how far along students are in their academic year. To investigate the rst scenario, we added a feature for a student's previous performance average, an average of every level attempt until now, in attempt to help distinguish low-performing students from the rest. Previous performance average was selected for each intervention type prediction but had lower feature importance (0.03%) because of the low variance and, therefore, did not a ect the prediction accuracy. To investigate if the time of the academic year had any impact on the prediction, we added a feature to indicate the month in which the sessions occurred. Similar to the previous performance average, this feature was selected but had a low feature importance (0.05%) leading to an insigni cant di erence in the prediction accuracy.
Since only time related features were varied enough to capture the variance in the student behavior in two consecutive level pairs, we explored ways other than feature generation to perform the pairwise prediction. We sliced the data based on aspects, such as replay type or month of the year but, again, obtained similar prediction accuracies; however, the replay related features did get selected and had high importance for the prediction in the data sliced by replay type. To investigate if the variance in the content of the objectives may be a ecting the prediction results, we performed the prediction for each intervention type within a single objective and observed that the prediction accuracy decreased slightly. This suggests that even within one objective, the behavior of a student in one level, as captured in its current granularity, may not accurately predict if they need an intervention in the next level.
The pairwise prediction models may not have generated desirable results because we may need more than just one previous level's data to predict if an intervention is needed. There is not su cient data about each level in this dataset to accurately represent the student's performance and to create good predictions. Therefore, having more ne-grained details about level attempts, including knowing more about how the levels compare to each other within each objective, may improve the prediction accuracy.
We chose to determine if we could predict the number of
levels completed in 20 minutes using only information from
the current session. Using only information from the
current session will allow an easier integration with the current
system with minimal changes needed. Di erent schools and
classrooms have unique ways of using ST Math[
The data we used consisted of 787949 session observations
from 8978 unique students, 111 schools, and 636 teachers. A
session represents a period of time that the student spends
working on ST Math without taking longer than a 30 minute
break (see Section 4 for full de nition). For accurate
predictions, we chose to use the rst 6 minutes of gameplay due to
the average level attempt taking approximately 3 minutes.
We refer to this segment of data used for prediction as a
time \slice". Since our goal was to use the least amount of
information to do the prediction, we wanted this time to be
as short as possible. We initially attempted to use shorter
time slices, but due to a level attempt taking on average
3 minutes, this did not provide a su cient amount of data
to represent the students' gameplay behavior and, in some
cases, eliminated slower students data for that time slice.
Next, we removed sessions under 10 minutes (242750 obs.
- 169863 obs. under 6 min) and sessions over 75 minutes
(10,257 obs.). We chose these cuto s to eliminate short
sessions where predictions would not be useful and long
sessions, in some cases over 4 hours, that were likely
anomalous.
6.1.1 Levels Completed Prediction: Feature
Generation
From the level data, the data was segmented into the rst 6
minute time slice for prediction. Features were aggregated
from this 6 minute time slice to capture what each student
was able to do, such as complete a level, fail a level, retry a
level, or engage in replay. The features generated are based
on performance, time, level attempts/replay features, the
objective the student was in within that slice. The
performance features were: average performance (numeric),
percentage of levels passed out of all attempts (numeric),
and percentage of levels completed out of all attempts
(numeric). The time features from the slice were: the total
time (numeric), the average level time (numeric), number
of passed levels per time (numeric, scaled), number of
completed levels per time (numeric, scaled), and the month of
the session. The level attempt features were: total
number of replays (numeric), total number of levels failed
(numeric), total number of levels passed (numeric), total
number of levels completed (numeric), total puzzles attempted
(numeric), total puzzles completed (numeric), whether they
engaged in replay (binary), whether they re-attempted a
level (binary). Then, there were 31 binary features
representing the objective the student was playing when the
session started.
6.1.2 Levels Completed Prediction: Model Selection
and Feature Selection
For this prediction, we tried a variety of models, tuning of
these models, and alteration of the target variable. Model
and feature selection was accomplished by using scikit-learn[
We tried multiple models, including decision trees, neural networks, and random forests after normalizing the features. Intrepretable machine learning methods are more important because knowing which features are more in uential to predicting performance can give insights to how students learn in games. In the results, we show the 2 best models compared to a baseline. The baseline model is created by always predicting the mean of the completed levels per 20 minutes. The best models were created by testing multiple models and ne-tuning the parameters. The two best models are created from scikit-learn: a 3-layer (50,30,20) neural network (MLPRegressor) using a learning rate of 0.001 and reLU activation function, and a Random Forest (RandomForestRegressor) using mean squared error as the criterion function and setting the minimum samples for a split to be 20.
To evaluate each model, we used the following metrics: Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), adjusted R-squared score, and explained variance (EV). We choose these metrics to evaluate, on average, how accurate each prediction rate was to determine if the error is small enough to still provide a good estimate of the students' projected progress. Both R2 and EV were used to evaluate the variance of these errors and check for biases within the models. All models were evaluated with 10-fold cross validation. We attempted feature selection using scikit-learn lter methods, such as feature importance from tree regression, and a wrapper method (SelectFromModel) with each model. Feature selection did not improve any models, and resulted in signi cantly worse predictions in most cases. This is most likely due to the limited amount of features available. Because our data is not ne-grained, we have a limited amount of information about each student for a level attempt. This indicates that each feature could be providing key information regarding their current progress. Therefore, for the best models the whole feature set was used (see Feature Generation).
This section discusses the results of the Experiment 2 regressions.
Table 5 shows the results of the evaluation metrics for the
two best regression models compared to the baseline model.
The NN and the Random Forest both perform similarly,
both outperforming the baseline model. Although the MAE
does not have a large di erence, the RMSE is much lower.
This indicates that the variance of the errors is signi cantly
smaller for our predictive models. The MAE of 1.2 for our
predictive models means that on average the prediction will
only be around 1 level o for a speci c student, which still
provides a good estimation for the teacher to use. The
adjusted R-squared and explained variance are almost identical
for both models, which happens when the mean of the errors
is approaching zero. Although these scores are not perfect,
in the context of educational data from a system used with
multiple teaching styles, this is a highly meaningful result[
ve most important features from Figure 2 shows a density plot of the predicted values versus the actual values, the yellow/lightest color being the highest density. Both gures show the highest density areas occurs closely to the actual values. The Neural Network appears to have a higher density closer to the line and the points appear to be more compact, although both models show similar predictions. Both gures are zoomed in to focus on the lower level number predictions, although few points have values higher than 10. One note is both models are less tted for the higher values and tends to predict around 10 after the actual value is 10+. However, we are mostly concerned with students who are completing very few levels. If a student falls into the 10+ range of levels completed, the actual value becomes less important due to how much above the average it is. A teacher will still be able to use this information to identify over-performing students and ensure they don't get too far ahead of the class.
With the variability of how this system is used, the models
evaluations are a positive result. For example, during eld
observations of the system, we found many teachers asked
students ahead in the curriculum to help students next to
them during a gameplay session. Therefore, a student may
spend part of the session working as normal, then, after the
teacher has identi ed a struggling student, the teacher may
ask the student next to them to help. This could result in
much higher predicted values than what the student
actually completes. Furthermore, we observed students in some
classrooms initially talking and working at a slower pace in
the rst few minutes of a session as they settled in, then
shortly being asked to focus. This could result in much
lower predictions of the projected number of levels that
student could complete. Since the data does not only include
the sessions where students quietly work by themselves for a
continuous period of time, accurate predictions are di cult.
Furthermore, the data we used focused on only the same-day
gameplay data, not containing any information regarding
how a student has previously performed in other sessions.
This decision was made to limit the changes required to
implement this system in the game. However, including prior
information may improve predictions. One possible way to
control for the e ect of the di erent teaching styles is to
include teacher or school information in the model. However,
this would create very sparse features due to the large
number of teachers and schools that use the system. A future
attempt could identify and categorize the teachers or schools
based on similar styles and add those features to the models.
This prediction can be used in two main ways: identifying
the lowest performing students who may need assistance,
but may not be requesting help, and identifying students
who may be working too fast and getting ahead in the
curriculum. The second usage may not seem like an issue, but
having a large knowledge gap between students may make
a classroom harder to manage and teach. This is a problem
teachers seek to avoid in ST Math that they have
remedied by asking those students to help others or by allowing
them to play games while others catch up[
The results for the levels completed experiment were more promising than the pairwise experiment. For the pairwise prediction, the lack of ne-grained puzzle level data made it di cult to predict whether a student may need intervention based only on their previous level's data. We believe the results for this method of pairwise prediction might improve with more data about how the objectives, games, and levels relate to each other. On the other hand, the prediction model from the levels completed experiment had decent results with the MAE and RMSE indicating that the predictions are generally within 1-2 levels of the actual completed levels for the 20-minute time period. Having additional information, including ner-grained puzzle-level data, should also improve this prediction.
Providing the teachers with a projected completed amount of levels allows us to give the teachers a list of the students ranked by the number of levels they are predicted to complete. This allows the teachers to use their expertise to distinguish the higher- and lower-performing students during that game session, and, importantly, the teachers have the ability to make judgments about interventions according to their discretion. Currently, the teachers only have information on student progress in the overall game curriculum (which objectives each student has nished and how many levels have been completed). Additionally, the only method currently used to support students in seeking help is the raised hand indicator, which has been shown to not always get the teachers' attention due to its location on the students' screens. We believe that incorporating this prediction into the system will be a valuable tool for teachers that will suggest which students are struggling and allow them to decide if they need intervention. Giving teachers these suggestions after only 6 minutes of gameplay time means that the teachers will have more control over the classroom progress because they will have more time to help students get back on track instead of being behind for the entire session and be able to slow down students who are getting too far ahead of the class.
To reduce the amount of time processing the data, we used a representative subset for the pairwise prediction. However, we compared multiple numerical and categorical features between this subset and the entire dataset and determined that it contained almost identical distributions of data points. We created histograms for the distributions of performance, level play time, levels in session, time of session, performance session, and compared the number of schools and teachers represented in the subset to the totals. We were only missing 6 out of 111 schools and we had students from almost half of the teachers (291 out of 636) included in our subset. We do not have ne-grained interaction data, which means we cannot tell exactly how many puzzles a student gets wrong. This lack of information causes our data to be skewed by having many performance scores of 100%, without capturing the full gameplay. However, there are other features that we can use tease out this information, like level time, as students who pass a level while also getting puzzles wrong will most likely take longer because they are doing more problems. We have ner-grained puzzle level data, but it does not match up accurately with our level data. This means that while we can do studies on these datasets separately, we cannot combine them to have the full picture of what a student is doing during the level: which puzzles they see, if any puzzles are repeated during a level, how many puzzles right and wrong, and the time spent on each individual puzzle in a level. These ner granularities could o er valuable information on what a student is doing during a level and their performance compared to the whole student set.
This study aimed to use the least amount of student gameplay data possible to predict which students would bene t from teacher intervention during the remainder of the gameplay session. We tried two granularities of prediction for our analysis. We hypothesized that we could use one level's data (average of 3.5 minutes of gameplay) to predict the next level's outcomes, as this controls for content and difculty, but this hypothesis was not con rmed. The lack of ne-grained level attempt data might not allow us to make a good prediction. Our second hypothesis was that we could use the rst 6 minutes of gameplay (about 2 levels) to predict how many levels the student could complete in the next 20 minutes. This had a reasonable outcome with a MAE of 1.2 and RMSE error of 1.6, meaning that, on average, the prediction is only o by 1-2 levels, which is a good estimation of how many levels a student will complete. We believe this can provide a valuable resource for the teachers who use ST Math in their classrooms, to help them concentrate their time and energy on the students who need it the most. Furthermore, this method allows the teachers to have a certain level of judgment in regards to who needs the assistance, which is imperative in a system that is used in multiple styles. Future work could investigate how this affected the students' performance if we gave this information to teachers.
This research is made possible by support of the National Science Foundation under Grant No. 1544273.