Low-achievement risk assessment with machine learning

Abstract

In this work, we propose a method for assessing the risk of low-achievement in secondary school with data collected from the Italian ministry of education. Low-achievement is a phenomenon whereby a student, despite completing his or her education, does not reach the level of competence expected by the school system. We train three machine learning models on a large, real dataset through the INVALSI large-scale assessment tests and compare the results in terms of predictive and descriptive performance. We exploit data collected in end-of-primary school mathematics tests to predict the risk of low-achievement at the end of compulsory schooling (5 years later). The promising results of our approach suggest that it is possible to generalise the methodology for other school systems and for diferent teaching subjects.

Keywords

• eol>low-achievement
• performance prediction
• assessment test
• machine learning

1. Introduction

of school efect on their skills acquisition.

An important indicator for school dropout is the ELET Low-achievement at school is a widespread phenomenon rate, which measures the percentage of Early Leavers which has long-term consequences, both for the individ- from Education and Training [6]. It measures a severe ual and for society as a whole. In 2016, above 28% of condition of educational exclusion which refers to young students across Organization for Economic Co-operation people between 18 and 24 with a qualification lower than and Development (OECD) countries underscored the min- upper secondary. Early leavers from education and trainimum level of proficiency in at least one of the three core ing are more likely to be unemployed or employed in subjects according to the Programme for International low-paid jobs with few or no prospects for training and Student Assessment (PISA), which are English reading further career progression; they are more prone to social and comprehension, mathematics, and science [ 1 ]. Low- exclusion and to experience lower levels of health, wellachievement is strongly related to school dropout, i.e., being and life satisfaction; they are also more likely to the discontinuation of education [2], and impact on the experience limited civic participation. In [7] the Eurocultural and professional growth of the individual and pean Commission indicated as one of the main targets to citizen [3, 4]. Indeed, school performance in first grade is be achieved in education the reduction of the ELET rate already a significant indicator of future high dropout risk. from 15% to 10% in the decade 2010–2020. While this In 2019, a study conducted by the National Institute for target was reached in several European countries, Italy Assessment of the Education System (INVALSI) found in 2019 still had an ELET rate of 13.5%[8]. that 20% percent of Italian students had a lower-than- To counteract dropout as soon as possible and to deexpected achievement and, eventually, dropped out of tect low-achievement, we address the following research school [5]. This way, despite the exterior appearance questions: and, in some cases, despite the suficient marks, the students do not reach the adequate level of knowledge which RQ1 Is it possible to quantitatively reprelater on will be needed to successfully continue the stud- sent a student’s knowledge level and build ies or to start a professional career. In this perspective, a model of his or her skill attainment? low achievement can be considered an “implicit” form of school dropout: although some students do not occur into explicit early leaving from school, the result is a lack

RQ2 Is it possible to develop a suitable AI

tool to predict, at an early stage, the risk of low-achievement at secondary school for primary school students?

In the following, we present a case study, focusing on

the Italian context and using data collected from the INVALSI national large-assessment tests in mathematics. In particular, from these tests, we aim to extract the relevant features related to students’ learning in terms of their skill and competence level performance.

In RQ2, we refer to “early stage” meaning to detect risk as soon as possible, i.e., several years in advance, so

3. Methodology

that appropriate countermeasures can be taken, and to design an intervention aimed at reducing risk when it is detected. Concretely, we develop three models able The INVALSI dataset is the result of a large-scale assessto predict the risk of low-achievement at K-10, using ment administered in Italy since the school year 2002/03 student data at K-5. at the levels K-2, K-5, K-8, K-10, and K-13. Students are

In selecting the models, we strive for a balance between tested in Italian, Maths and English. In addition to this, descriptive and predictive performance. Indeed, we want there is a student-survey aimed at gathering informaour solution to be both interpretable and robust. Hence, tion on the social, economic and cultural context. A we consider state-of-the-art machine learning techniques strength point of the dataset is represented by the posthat proved to be efective in preliminary experiments [ 9 ]: sibility to longitudinally link data collected in diferent random forests and neural networks [ 10, 11 ]. On the one school years [22] starting from 2011. In our case study, hand, we exploit random forests to extract rules that we considered data on maths test from two cohorts of facilitate the process of interpreting the outcomes of the students: K-5 of the 2012/13 school year and K-5 of the research and, on the other hand, we test neural networks 2013/14 school year. For the same students, we collected for flexibility, e.g., exploring non-linear correlations, and data from five years later at grade K-10, to be used for performance gain. the definition of the low-achievement target, i.e., the students grade in the test is less than or equal to 2 on a scale from 1 to 5. After merging K-5 datasets with their corre2. Related work spondent K-10 targets, the K-5 2012/13 cohort is made up of 351746 students, while the K-5 2013/14 cohort of Low-achievement is a widely studied phenomenon in the 354987 students. social sciences and education [12, 13]. The problem has There are several features in the dataset and we apalso been addressed in terms of predictive models for low- plied a feature selection process to determine a subset achievement—or dropout risk—for both high school and of relevant features. The datasets also contain a boolean college students. These models exploit diferent machine feature for each test item, where the students’ answers learning techniques, including supervised learning, e.g., correctness are recorded. To enable the use of our prerandom forests, support vector machine and Bayesian dictive models on diferent cohorts of students and to network, unsupervised learning, e.g., k-means and hi- provide a coherent representation of their learning in erarchical clustering, and recommender systems, e.g., terms of areas of knowledge and skills, it is necessary to collaborative filtering [ 14, 15]. Moreover, several kinds release the dataset from the individual items that constiof data have been used to tackle the problem. In [16] tute a certain test. Therefore, we used a knowledge-based the dataset for building the predictive model uses de- approach considering the items classification in terms of mographic data of the students and their grades. Other areas, processes and macro-processes according to the studies are based on students performance, i.e., grades, INVALSI framework for the design of math tests. collected during first semester courses [ 17, 18]. Some in- In Table 1, we give for reference an overview of the clude behavioural data supplemented with other features areas, processes, and macro-processes that have been related to learning results [19], in a mix of cognitive and used in the encoding of the questions. non-cognitive characteristics. In some studies data col- We define one new variable for each area, process, lected through large-scale assessment tests were used to and macro-process. Each of these new features takes design predictive models of student performance through the value corresponding to the percentage of correct several machine learning techniques. In [20], for exam- answers provided by the student for that specific group of ple, the authors refer to data collected through the PISA items, namely, correctness rate. Last, we concatenate the international large-scale assessment tests. computed values to obtain a new flattened representation

In this scenario, we aim to contribute to the research of learning, where each item is a possible indicator and ifeld of AI-based education solutions by presenting a case not its unique representative. Following our strategy, we study for predicting the risk of low-achievement of high represent each student’s learning in the space of ffiteen school students using their performance data collected (15) dimensions, as shown in Table 2. during primary school. As a minor contribution, we We use two techniques to develop our AI-based tool. extract features directly related to students’ learning in The first one is Random forest (RF) [ 10], which is widely terms of knowledge and skills, privileged indicators for used in Educational Data Mining for the high degree of the study of learning [21], thus proposing a knowledge- explainability and efortless interpretation of the results. based method for encoding students’ learning. We believe We trained our models through bootstrap aggregating that this element can improve the interpretability of the (bagging) to reduce the overfitting of dataset and increase results and make this tool useful for students, teachers precision. To tune the model, we performed a grid search. and instructional coordinators. The second technique is based on neural networks, Areas (NU) Numbers (SF) Space and figures (DF) Data and forecasts (RF) Relations and functions Process (P1) Know and master the specific contents of mathematics (P2) Know and use algorithms and procedures (P3) Know diferent forms of representation and move from one to the other (P4) Solve problems using strategies in diferent fields (P5) Recognize the measurable nature of objects and phenomena in diferent contexts and measure quantities (P6) Progressively acquire typical forms of mathematical thought (P7) Use tools, models and representations in quantitative treatment information in the scientific, technological, economic and social fields (P8) Recognize shapes in space and use them for problem solving Macro-process (MP1) Formulating (MP2) Interpreting (MP3) Employing 4. Experimental results which has recently become widespread also in the field of Educational Data Mining and has also been applied in predictive models for student performance [23]. To We carried out all the experiments using the Google deal with our hybrid dataset, we firstly include a prepro- Colaboratory Notebook environment, with the Python cessing step, aimed at encoding the values of categorical programming language and popular machine learning variables into numerical values with a “one-hot” encod- libraries, such as scikit-learn and pandas. ing algorithm. After preprocessing, we implemented two The dataset for all the experiments was pre-processed neural networks based on diferent data transformation cleaning features with many missing values, highly corapproaches. Categorical Embeddings (CE), is a neural relation (computed by 2 measure above 0.5) or specifnetwork that treats the input depending on its type: cat- ically referred to a cohort of students, preventing the egorical inputs are passed through an embedding layer, model to be transferred to new cohorts (e.g., identification numerical ones are fed to a dense layer. Feature Tokenizer code for a class). This features selection process, together Transformer (FTT) [ 11 ] is able to identify the input or the with the engineering of the features related to the items in group of inputs that most influence the output, thanks to the tests, results in a set of 34 features, which refers both attention maps. It is a more complex architecture that ex- to socio-economic and cultural context, demographic ploit a feature tokenizer function to extract tokens from data and learning dimension. For the definition of the the input and then fed these tokens to a Transformer training set we used the data from 2012/13 K-5 cohort. For architecture [24] for classification. the models based on neural networks we split this cohort to generate both training and validation sets (split in 80% and 20% respectively). Finally, we used the K-5 2013/14 cohort to test and measure the model performance. This allowed us to evaluate the validity of the proposed learning encoding that is, the efectiveness of abstraction from specific items to learning in terms of areas, processes and macro-processes. The dataset is unbalanced between underachievement/non-underachievement classes; therefore balancing techniques were applied. In the development of the RF models, a random undersampling technique was used, implemented in the imblearn library.

We trained neural networks using a weighted random sampler, that samples the data to balance classes ratio in the training batches.

Finally, we want to deepen the interpretability of the re

sults of our models, by analysing the feature importance computed on RF model and comparing it with the interpretation of the weights that define the neural networks we have used.

References

[1] OECD, Who and Where are the Low-Performing

Students? , OECD Publishing , 2016 . Predictive models performance [2] D. D. Curtis , J. McMillan , School non-completers: In Table 3, we present the overall results on the test Profiles and initial destinations, 2008. dataset of the above mentioned models: RF, CE, and FTT . [3] K. L. Alexander , D. R. Entwisle , L. S. Olson , Schools, For RF , we considered the best hyper-parameters setting achievement, and inequality: A seasonal perspec-

tive , Educational evaluation and policy analysis 23

( 2001 ) 171 - 191 . Table 3 [4] S. J. Ingels , T. R. Curtin , P. Kaufman, M. N. Alt, Predictive performance of the proposed models on the external X. Chen , et al., Coming of Age in the 1990s: The validation test set (year 2015 /16). Eighth-Grade Class of 1988 12 Years Later ., Eric,

Models Accuracy Precision Recall 2002 .

Random Forest 0.77 0.62 0 . 67 [5] R. Ricci , La dispersione scolastica implicita, 2019 .

CE network 0.76 0.76 0 . 76 [6] S. Flisi , V. Goglio , E. C. Meroni , E. Vera-Toscano,

FTT network 0.78 0.77 0 .78 School-to-work transition of young individuals:

bourg: Publications Ofice of the European Union, determined with the grid search technique: 50 estimators EUR-Scientific and Technical Research Reports in the forest, trained with 30% of random samples , 60 % ( 2015 ). of random features and max depth set to 11 . The FTT [7] E. Commision , Europe 2020 a strategy for smart, outperforms the other predictive models with accuracy, sustainable and inclusive growth, 2010. precision and recall between 77% and 78 %. [8] Istat , Livelli di istruzione e occupazione nazionali,

2020 . 5 . Conclusion [9] A. Zanellati , S. P. Zingaro , M. Gabbrielli , Student

low achievement prediction , in: M. M. Rodrigo, Our results suggest that the challenge of predicting low- N. Matsuda , A. I. Cristea , V. Dimitrova (Eds.), Arachievement risk for primary and secondary school stu- tificial Intelligence in Education, Springer Internadents can be efectively addressed through the use of tional Publishing , Cham, 2022 , pp. 737 - 742 . well-curated datasets and the choice of reliable predictive [10] L. Breiman , Random forests, Machine learning 45 models. Our abstract representation of (INVALSI) tests ( 2001 ) 5 - 32 . and the related encoding for the student achievement [11] Y. Gorishniy , I. Rubachev , V. Khrulkov , A. Babenko, allowed us to transfer the trained models on diferent Revisiting deep learning models for tabular data, cohorts and therefore to obtain a accurate prediction . We 2021 . arXiv: 2106 .11959. believe that the ability to predict low school achievement [12] R. Cassen , G. Kingdon, Tackling low educational with reasonable accuracy five years in advance ofers a achievement , Joseph Rowntree Foundation , 2007 . practical tool for policy makers, managers and educators . [13] D. C. Geary , Consequences, characteristics, and

We are interested in extending our work in several causes of mathematical learning disabilities and directions. First, we want to verify the transferability persistent low achievement in mathematics, Journal of the proposed methodology to other disciplines, us- of developmental and behavioral pediatrics: JDBP ing a representation for students' learning similar to the 32 ( 2011 ) 250. one proposed in this work . Second, we want to increase [14] J. L. Rastrollo-Guerrero , J. A. Gomez-Pulido , the quality of the information provided as input to the A. Durán-Domínguez, Analyzing and predicting predictive models , e.g., by collecting more data and by students' performance by means of machine learnintegrating new data sources. We aim to improve the ing: A review , Applied sciences 10 ( 2020 ) 1042. learning encoding-thus the feature extraction process- [15] B. Albreiki , N. Zaki , H. Alashwal , A systematic in a way that is not knowledge-based to limit the bias. literature review of student performance predic-

Sciences 11 ( 2021 ) 552 . [16] S. Kotsiantis , C. Pierrakeas , P. Pintelas , Predicting

Intelligence 18 ( 2004 ) 411 - 426 . [17] M. A. Al-Barrak , M. Al-Razgan , Predicting students

technology 6 ( 2016 ) 528 . [18] Z. Ibrahim , D. Rusli , Predicting students' academic

decision tree and linear regression , in: 21st Annual

SAS Malaysia Forum, 5th September , 2007 . [19] S. Sultana , S. Khan , M. A. Abbas , Predicting per-

Electrical Engineering Education 54 ( 2017 ) 105 - 118 . [20] A. Pejić , P. S. Molcer , K. Gulači, Math proficiency

ing model , in: 2021 IEEE 19th International Sympo-

IEEE , 2021 , pp. 49 - 54 . [21] L. K. Baartman , E. De Bruijn, Integrating knowl-

tional Research Review 6 ( 2011 ) 125 - 134 . [22] L. Branchetti , F. Ferretti , A. Lemmo , A . Mafia,

in: CERME 9-Ninth Congress of the European Soci-

ety for Research in Mathematics Education , 2015 ,

pp. 1695 - 1701 . [23] A. Hernández-Blanco , B. Herrera-Flores , D. Tomás ,

Complexity ( 2019 ). [24] A. Vaswani , N. Shazeer , N. Parmar , J. Uszkoreit,

2017 , pp. 5998 - 6008 .