Educational data mining is essential to our lives, as it provides the tools for us to analyze education-related problems and achieve better education. Particularly under COVID19, the transition from in-person classes to online learning mode leads to a surge in educational data, and increasing attention has been drawn to educational data mining. However, many long-lasting problems in educational data analysis have not been properly solved due to various challenges. We identify several challenges in the student performance prediction task: (1) Good indicators for the nal grade are hard to nd and the nal grades are only highly correlated to the grades in the previous terms. (2) The lack of one automatic and stable system that automates the data analysis cycle and produces satisfying prediction results. To tackle these two challenges, we implement an ensemble-based machine learning system for the student academic performance prediction task which automates the data analysis cycle in the absence of students' grades from previous terms. Specifically, our model consists of two components, ensemble feature engineering module and ensemble prediction module. Extensive experiment results have shown the superiority of our model over other traditional machine learning models, both in stability, e ciency and accuracy. Speci cally, our model outperforms the best traditional machine learning algorithms by up to 14.8% in prediction accuracy.
∗equal contribution
†corresponding author
Copyright ©2021 for this paper by its authors. Use permitted under
Creative Commons License Attribution 4.0 International (CC BY 4.0)
From traditional lectures to novel E-learning, education
quality is a long-lasting problem that we aim to improve [
An ensemble-based feature engineering algorithm is proposed to enhance the acceptability and accuracy of the feature engineering phase.
A graph-based ensemble prediction model is proposed to make a more stable and accurate student performance prediction.
Extensive quantitative evaluations and qualitative analysis are performed to show the superiority of our proposed model in stability, e ciency, accuracy and explainability.
The road map of this paper is shown as follows. First, a brief summary of related works is explained in section 2. Then, we introduce the details of the proposed system in section 3. Next, the set-up and results of the experiment are detailed and analyzed in section 4. Finally, we conclude this paper with future works in section 5.
Classi cation task is critical in many areas and has
numerous applications, such as spam email detection, face
classi cation, object classi cation, etc [
Ensemble machine learning is based on the idea to combine
and take advantages of multiple models and achieve better
and more stable performance. Two popular ensemble ideas
are bagging and boosting, which bagging aggregates the
results over multiple models, while boosting creates a stronger
model by multiple weak models. Two well-known instances
for bagging and boosting are Random Forest [
Student performance prediction is a long-lasting problem in
the educational data mining eld [
We implement an ensemble-based iterative feature selection algorithm which takes all the features from the beginning and iteratively search for better feature sets. The idea of this algorithm implements the greedy algorithm which follows the problem-solving heuristic to make an optimal choice every single step. Speci cally, the algorithm takes measurement (i.e. importance) of each feature weighted by a list of reliable classi cation models. Every time, each feature I Xi;j = Xi;j + X i=1
PjJ=1 Xi;j
J
I Li = argmax Xi;j j is dropped from the feature set and the classi cation performance (accuracy) change indicates the importance of the feature. To make the feature importance more reliable, we ensemble multiple classi cation models and take the best one as the current best score. Then, it drops the worst negatively-a ecting feature every iteration. Additionally, to leverage the side e ect of the algorithm being too greedy, we set a stop mechanism, parameterized by k, which stops after a continuous lower performance in k iterations and rollback to the previous best feature set. After experiments with di erent values of k, the result showed that if drop positively-a ecting feature more than 3 times, the accuracy will keep decreasing. In practice, we set this parameter to 3 and the list of classi cation models selected are [Random Forest, SVM, XGBoost]. The pseudo-code for the algorithm is shown in code block 1.
We implement a graph-based ensemble classi cation model, which takes several reliable classi cations and clustering models and aggregates the results by the message propagation over a bipartite graph to achieve better performance. The idea is to ensemble over several reliable algorithms to achieve more accurate and stable results. It is very common (e.g. in Kaggle competitions) to ensemble the results of supervised learning models, while it is not that common to study how unsupervised learning models, such as clustering algorithms, can help in the ensemble process. Here, we aim to assist the prediction from the supervised classi cation learning models with certainty from the unsupervised clustering models. We rst initialize all our models (classi cation, clustering) with Grid-searched parameters. We construct the graph G = (V; E), where nodes V represent objects (either a data point X or a group Y ), and edges E represent the connection between a data point and a group (class/cluster) determined by the classi cation/clustering algorithms. Empirically, we utilize three classi cation models and two clustering models. The total groups are 20 for classi cation models. We leave all of the groups on the right side of the bipartite graphs (i.e. 40 groups) because there is no consistent correspondence between di erent clustering models. Speci cally, each edge is built if one classi cation model predicts the class of the sample or clustering algorithms cluster the samples together to one group. Because there is no supervision for the clustering algorithm, we utilize it to enhance the con dence of the classi cation models. We do so by calculating a condence matrix C 2 XI;J , where I is the total number of the samples and J is the total number of the groups. Each entry Xi;j represents the con dence score of a sample being in a group, which is initially measured by the predicting accuracy of the classi cation models. The con dence score is calculated via propagation over the graph: (1) (2) Finally, the prediction class is determined by taking the label with the maximum probability, or certainty.
The groups are shown on the left and the bipartite graph is shown is the right of Fig. 1
Initialize a list of all classi cation models M , model scores P M , all features G, scores for all features P as 0s, a stop counter K, parameterized by k, an empty list of temp features T , a best feature set B, a best accuracy A; for model m in the model list M do
Run the model m;
Record the prediction accuracy to P M ; end Record best accuracy A with highest score in P M ; Record best feature set B with highest score in P M ; Initialize stop counter K = 3; while condition K == 0 is not satis ed do for feature f in the feature list F do
Drop feature f ; for model m in the model list M do
Run the model m; Record max(P M; prediction accuracy) to P M ;
Record max(P M; prediction accuracy) to P ; end
Add feature f ; end if K == 0 then
Resume the best feature set B; Resume the best accuracy A; return the best feature set B, best accuracy A; else if all scores in P > 0 then
Drop the feature with the highest score P drop; if highest score > best accuracy A then
Record best accuracy A with highest score P M ; Record best feature set B with highest score P M ; Drop the feature with the lowest score P drop; K = 1; Continue; end else end
end end
Algorithm 1: Ensemble Feature Engineering Module % !$ ⋯ ⋯ ⋯ ⋯ % !$$ % ⋮ ! ⋮ "# ⋮ !$ ⋮ %$$ ⋮ !$$ % ! ⋮ & %$ ⋮ ⋮
Dataset Description. We take one commonly used
benchmark dataset, Student Performance Data Set [
4.2.1 Ensemble Feature Engineering Evaluation
The nal selected features by our model are school, sex,
age, address, famsize, Pstatus, Fedu, Mjob, Fjob, reason,
guardian, traveltime, studytime, failures, schoolsup,
famsup, paid, activities, nursery, higher, internet, romantic,
famrel, freetime, Dalc, Walc and health. The de nition and
details of each feature are shown in Table 5. We show the
prediction accuracy of all the baseline models on data
provided by di erent feature selection algorithms in table 1.
Clearly, our ensemble feature selection model outperforms
the raw data in all baseline models. To compare with other
feature selection techniques, such as Principle Component
Analysis (PCA) [
In order to see whether popular data re-sampling techniques
are suitable for our problem, we select three well-known data
re-sampling algorithms and take the implementation from
the imblearn library 3. To be speci c, we take one
oversam3https://imbalanced-learn.org/stable/
pling, one undersampling and one combined-sampling
algorithm, respectively. Speci cally, we utilize SMOTE [
In table 4, we show the ablation study of our system, in which we take out one component and see whether the other component improves the performance, compared to the baseline models. Clearly, both the feature engineering module and ensemble prediction module improve the result of our system.
In this paper, we implement an automatic ensemble machine learning system for student academic performance prediction, which consists of ensemble feature engineering and ensemble prediction modules. The extensive experiments have shown that each component of our system outperforms the traditional machine learning methods. Overall, the system further improves the prediction accuracy, up to 14.8%. In the future, we plan to investigate how data augmentation could help in the task. We also hope that our proposed system can inspire others to reinvent traditional machine learning models which can work more e ciently, stably and accurately.
Figure 4: Feature distributions of the dataset [
school sex age address famsize Pstatus Medu Fedu Mjob Fjob reason guardian traveltime studytime failures schoolsup famsup
paid activities nursery higher internet romantic famrel freetime goout Dalc Walc health absences
G1 G2 G3 Meaning School gender
Age
Address
Family Size Parent's Cohabitation Status Mother's Education Father's Education
Mother's Job
Father's Job Reason to Choose this School
Student's Guardian Home-to-School Travel Time
Weekly Study Time # of Past Class Failures Extra Education Support
Family Education Support Paid Classes (Math or Portuguese) Extra-curricular Activities
Attended Nursery School Wants to Take Higher Education
Internet Access at Home with a Romantic Relationship Quality of Family Relationship Freetime after School
Going out with Friends Workday Alcohol Consumption Weekend Alcohol Consumption Current Health Status
# of Absences First Period Grade Second Period Grade
Final Grade
Gabriel Pereira - 'GP' or Mousinho da Silveira - 'MS' Female(F) or Male(M)
from 15 to 22
Urban - 'U' or Rural - 'R' less or equal to 3 - 'LE3' or greater than 3 - 'GT3'
living together - 'T' or apart - 'A' none - 0, primary - 1, 5th-9th grade - 2, secondary - 3, higher - 4 none - 0, primary - 1, 5th-9th grade - 2, secondary - 3, higher - 4 'teacher', 'health' care related, civil 'services', 'at home', 'other 'teacher', 'health' care related, civil 'services', 'at home', 'other close to 'home', school 'reputation', 'course' preference, 'other' 'mother', 'father', 'other' <15min - 1, 15-30min - 2, 30-60min - 3, >60min - 4) <2hrs - 1, 2-5hrs - 2, 5-10hrs - 3, >10hrs - 4 n if n <= 3 else 4 yes or no yes or no yes or no yes or no yes or no yes or no yes or no yes or no from 1 - very bad to 5 - excellent from 1 - very low to 5 - very high from 1 - very low to 5 - very high from 1 - very low to 5 - very high from 1 - very low to 5 - very high from 1 - very bad to 5 - very good from 0 to 93 from 0 to 20 from 0 to 20 from 0 to 20