Student Performance Prediction is essential for learning analysis of the students' learning behavior to discovering at-risk students for the early invention to support students. This study transforms the students' reading behavior into a two-dimensional matrix input based on each lecture material's reading behavior. The matrix input will be updated by accumulating the value for each week for performance prediction week by week. The multilayer perceptron neural network is employed to receive the matrix input and give feedback as a student's criteria consist of at-risk or no-risk students. This study considers the accuracy of a model considering between on contents information and weekly information. We also investigate the switching of learning materials' order, the feature importance of the reading operation on an event stream, and the difference in reading behavior between at-risk and no-risk students. These can help the instructors for an early invention to support at-risk students.
Performance prediction is education mining data by collecting students' learning activity logs and
analysis. Then, we receive output to analyze students' learning behavior. The purpose of performance
prediction is to increase students' academic success or discover at-risk students early and invent them
against drop-out or failure [
Recently, Machine Learning (ML) and Deep Learning (DL) have been used for performance
prediction in previous research. This is because ML and DL can learn and recognize the data pattern
with a vast volume and predict the outcome with better accuracy [
From the reason above, this study decided to transform the data to be based on lecture materials for students' performance prediction since the lecture materials are a fundamental object used to hold the lecture. In addition, this paper relies on the neural network (NN) to derive the high-performance classification model for predicting students' reading behavior. This can support instructors to identify at-risk students during the period of courses and invent students early.
The prediction of students' performance becomes the challenge of learning analytics. The previous literature was recently analyzed to present different techniques to predict students' performance, such as statistical methods, machine learning, or deep learning.
Moises et al. [
From these previous works of literature, we found they predicted students' performance by considering the event stream of reading behavior and other factors based on conventional weekly. In contrast to previous work, we present a prediction by considering the reading behavior on each learning material that instructors used in the lecture. This study will transform reading behavior data into the matrix input that can explain each student's reading information.
The clickstream data was received from the International Conference on Learning Analytics &
Knowledge 2022 Data Challenge [
The features are received from the operation on the event stream that concludes the total operations and
reading time computed by using the openLA library [
In this study, the raw data are transformed into two-dimensional matrices for inserted into the model
[
The contents information matrix can input the feature in order of learning materials based on the previous data. Then, this matrix can accumulate values of features depending on students' reading behavior independent of time series. This can protect instructors from switching and not using content for a lecture. So, we can use the previous data to predict students' performance that differs from the model depending on time. Since instructors do not use a learning material to lecture students on that week, the value of the data has a significant change that may cause prediction performance to change.
From the observation data in table 1, we found that the amount of the contents is different. We thus made the matrix for each course is adjusted to have rows and columns equivalent using the matrix size of a course with most lecture materials on the same lecture-type as the main. If the missing value is found, this will be imputed by the average of each feature's value of students on the same course. All value on the matrix is normalized by robust standardization for rescaling features using statistics that are robust to outliers. 3.4.
This study creates MLP prediction models for each type of lecture is trained by the learning behavior in all lecture weeks. The MLP will consist of two hidden layers with the activation function are ReLu. The output layer had two units for classification between no-risk and at-risk students with the activation function is softmax. The optimizer function is Adam for the loss. The matrix will be adjusted to become the vector of size m x n by listing m rows next to each other and fetching them into the same model.
The experiment will predict the students' performance every week during the course period. The matrix will be updated by accumulating data week by week until the course's last week that is shown in figure 2. 3.5.
The evaluation criteria will class the student's performance as a binary classification. The class will
separate students who receive grades A, B, and C into the no-risk group and students who receive grades
D and F into the at-risk group [
This experiment presents the model's prediction result that considers weekly behavior with content
information (Weekly w/ Content) in contrast with weekly behavior without content information
(Weekly w/o Content). Many researchers predicted students' performance using the data based on
conventional weekly in previous. For instance, Okubo et al. [
The weekly prediction data is collected from the reading behavior data from the starting of a lecture of the current week until the starting of next week's lecture every week. Then, this is input into the prediction model as a vector size 1 row with n columns that uses features like a content information model. We evaluate the proposed method where A-2019 and B-2019 courses are used as the training data, and A-2020 and B-2020 courses are used as the test data.
From observing the correspondence of lecture materials on both lecture types from the raw data, an instructor adjusted the teaching styles of the courses by changing the number of contents. We found that A-2019 and B-2019 courses have ten and one content canceled for usage on the lecture in 2020, respectively. Then, A-2020 and B-2020 courses have two and one new content used in the lecture, respectively. Therefore, the matrix input for training and test set is adjusted to have the row and column equivalent by using the matrix size of a course with most lecture materials on the same lecture-type as the main. The missing value thus will be imputed by the mean of each feature's value on each student. However, an A-type lecture on the 2019 course has 21 lecture materials, including 12 main contents and 9 summarized from main contents. Then, an instructor left the main content only and added new content in 2020. In addition, many students chose to read the main contents more than summarized contents significantly on students' reading behavior on A-2019 lectures. We thus transform the training matrix of A-2019 courses to have the same size as the test matrix of the A-2020 course.
The result found that the weekly w/o content model is better than the weekly w/ content model on the lecture A-type. However, these are similar to the lecture B-type. Then, we found that both prediction models had high accuracy on a lecture B-type than a lecture A-type shown in figure 3.
When we observe the result prediction on each model, we found that the weekly w/o content model can classify the final grade A and B students as no-risk students better than the weekly w/ content model on both types of lectures. However, the identity of the final grade C students is very well on both models in a B-type lecture more than an A-type lecture. The detection of the at-risk students of both models on A-type lecture is better than B-type lecture every week. This is because the training data of at-risk students on A-type lecture (16 students) is more than B-type lecture (4 students). Therefore, that affects the model better to analyze the pattern of at-risk students on A-type lecture. The pattern of reading behavior of both no-risk and at-risk students will be shown in section 4.2.
Figure 4a: The graphs show the switching of learning materials' order of a weekly w/ content model on (a) A-2020 and (b) B-2020 courses.
Figure 4b: The graphs show the switching of weekly order of a weekly w/o content model on (a) A-2020 and (b) B-2020 courses.
Furthermore, we simulated the example situation if instructors change the learning materials' order to make information change in an actual class in Figures 4a and 4b. We conducted these experiments by switching the order of learning materials in the weekly w/ content and the weekly information on weekly w/o content on test data every week and inputting the same model on previous experiments.
We found the switch of the contents on a weekly w/ content model made an accuracy drop on some weeks on a course period, especially on an A-type lecture. The reason is that a weekly w/ content ) 18 (%16 ce 14 n tra 12 o p 10 m e 8 i r tau 6 fe 4 e itv 2 a le 0 R model considers each content’s reading behavior as the main. Thus, switching the order materials makes the values of features change. In addition, instructors and students maybe give importance to each lecture equivalent each year. This made the separation of each content on each row on the matrix have new data that a weekly w/ content model has ever been trained to cause an accuracy drop (Figure 4a).
On the other hand, the switch of weekly information on a weekly w/o content model made an accuracy change on some weeks. However, this model can give an accuracy better than the weekly w/ content model. We found that the weekly w/o content model considers the operation count and reading time on each week as central. Let us consider students' behavior, such as reading time. We can find that the no-risk group takes time to read regularly. However, the at-risk group did not spend so much time reading that caused a weekly w/o content model to identify a class of students better than a weekly w/ content model (Figure 4b). 4.2.
In the previous experiment, we conducted the matrix and vector input using features explained in section
3.2. We thus try to reduce the matrix size by exploring the feature importance by using the Random
Forest classifier. This method allows us to know the insight into informative features for classification
[
Features
This experiment was conducted using the vector data from the weekly w/o content to explore the feature importance. We found the 'NEXT,' 'PREV,' 'READING_TIME_MIN,' and 'TOTAL_ACTION' are gave the most feature importance score as a percentage on both types of lecture. These features are the most helpful operation for reading learning materials. Furthermore, this will analyze students' reading behavior on each final grade by using a heatmap in figure 6. The heatmap of reading behavior was computed by the average of operations from the students on each final grade. The rich yellow color means that a feature has the most action and the purple color means that a feature has the poor action. The no-risk students had the most operation on an event stream, which means no-risk students spent much attention and time for reading learning materials on many pages. On the other hand, the at-risk students did not focus on reading lecture materials as no-risk students, which caused these students got to fail.
We thus try to adjust the matrix on weekly w/ content model by reducing the matrix size by using only the high score features composed of 'NEXT,' 'PREV,' 'READING_TIME_MIN,' and 'TOTAL_ACTION' compared with the current features. We found that the reduction of matrix size raised prediction accuracy on an A-type course due to the essential features, making the model more easily recognize the difference between at-risk and no-risk students. However, a B-type course did not have any difference when using the high score features for prediction since the data of at-risk students was few. (Figure 7)
From the result of the experiment, we found that predicting students' performance by considering learning content data can classification students who have a trend to fail or at-risk students if we have much reading behavior pattern of at-risk data sufficient. In addition, we used a heatmap to investigate the reading behavior on each week between no-risk students and at-risk students for a survey on how it affected the model performance. After the 3rd and 4th week of lecture, we found that no-risk students had a few actions for reading learning materials. On the other hand, at-risk students can continue reading and reviewing many learning materials until the end of the course (Figure 8). This causes a model to separate the reading behavior between at-risk and no-risk students.
The reading behavior of at-risk students has a minor operation on the reading of contents materials,
which means these students spent a little time reading. We thus plot graphs of the average reading time
of all students those criteria by final grade on figure 9. Again, we found that the students in the no-risk
group spent much time reading more than at-risk students significantly, according to the result on a
heatmap. Let us consider the reading behavior with the prediction result based on the content
information model. The instructors can invent students who can fail on half of the course period, such
as 3rd or 4th week on the quarter course. Considering the early timing for interventions with past related
literature, such as the author from [
This study employed the multilayer perceptron neural network to classify students' performance as atrisk and no-risk students for support instructors for early interventions. The experimental data were collected from the two different onsite and online lectures in different years. The model for identifying at-risk and no-risk students is based on the clickstream event on each learning material and transformed into a two-dimensional matrix. The features are generated by using the operation of students' reading behavior. We then contrast the model's accuracy with the model considering on weekly information model. From the result, a weekly w/ content model can give accuracy similar to a weekly w/o model. A weekly w/ content model model is a model that considers students' reading behavior depending on the content that is independents of time. However, a weekly w/ content switch made an accuracy drop since students may not focus on each content equivalent. Whereas the weekly switch information on the weekly w/o contents model can give a better accuracy since this model focuses on the total reading behavior each week.
Next, we try to reduce the size of the matrix by using the random forest to investigate the feature importance. We found that both types of lectures' high score operation features were the same as a result. These features are the most necessary action for reading learning materials on the system. Finally, we found the behavior of at-risk students gave little attention to studying and reviewing the learning materials on features on the matrix that can inverstigate on the heatmap after half of period so the teachers should invent to protect these students from failing before half of the course period.
In the future task, we plan to explore how to find an attention weight of each content on the same
type of lecture due to instructors' changing lecture style and students' reading behavior. The purpose is
to improve the weekly w/ contest model because this can still perform well and explain almost all
students' reading behavior well. Then, we want to optimize the model to have higher accuracy by using
other deep learning models, such as the attention model [
This work was supported by JST SPRING Grant Number JPMJSP2136, JST AIP Grant Number JPMJCR19U1, and JSPS KAKENHI Grand Number JP18H04125, Japan.
The following table presents the variables describing the reading operation collected from the students' reading behavior on the e-book system. The description is referred to the dataset received from LAK22 Data Challenge.
F4 F5 F6 F7 F8 F9 F10 F11 F12 F13 F14 F15