Predictors and Early Warning Systems in Higher Education | A Systematic Literature Review

Abstract

The topic of predictive algorithms is often regarded among the most relevant elds of study within the data analytics discipline. Nowadays, these algorithms are widely used by entrepreneurs and researchers alike, having practical applications in a broad variety of contexts, such as in nance, marketing or healthcare. One of such contexts is the educational eld, where the development and implementation of learning technologies led to the birth and popularization of computerbased and blended learning. Consequently, student-related data has become easier to collect. This Research Full Paper presents a literature review on predictive algorithms applied to higher education contexts, with special attention to early warning systems (EWS): tools that are typically used to analyze future risks such as a student failing or dropping a course, and that are able to send alerts to instructors or students themselves before these events can happen. Results of using predictors and EWS in real academic scenarios are also highlighted.

Keywords

• Predictive analytics Early warning systems Learning analytics Learning technologies

-

1.1

Introduction

Context

Over the last couple decades, the meteoric rise of information technologies (IT) has caused deep social and economic transformations worldwide, leading to the growth of new disciplines and activities which are of utmost importance today. Among these disciplines is data analytics, which is currently a huge source of income for many companies | especially, but not exclusively, those in the IT eld |, as well as a very relevant topic for researchers.

Data analytics encompasses the collection of techniques that are used to examine data of a variety of types to reveal hidden patterns, unknown correlations and, in general, obtain new knowledge [ 18 ]. The discipline is often coupled with the term \big data", since analysis tasks are often performed over huge data sets. Other elds of study which are very popular nowadays, such as data mining or machine learning, are close to data analytics and share many relevant techniques.

Depending on the nature of the data that is being analyzed and the objective that the analysis task should ful ll, several sub-disciplines can be de ned under data analytics. Examples of these are text analytics, audio analytics, video analytics and social media analytics. The main focus in this paper, however, will be predictive analytics, which includes the variety of techniques that make predictions of future outcomes relying on historical and present data [ 13 ].

The capability of predicting future events is essential for the proper functioning of some applications. Notable examples among these are early warning systems (EWS), which are capable of anticipating potential risks in the future thanks to present information, accordingly sending alerts to the person or group of people who may be a ected by these risks and/or that are capable of countering them. Their degree of reliability on information technologies greatly varies depending on the context they are applied on.

Early warning systems are mostly known for their use to reduce the impact of natural disasters, such as earthquakes, oods and hurricanes. Upon detection of signs that a catastrophe might happen in the near future, members of the potentially a ected population are alerted and given instructions to prevent or minimize damage [ 24 ]. However, other kinds of EWS have been implemented in a variety of di erent contexts. For instance, they are used in nancial environments to predict economic downturns at an early stage and provide better opportunities to mitigate their negative e ects [ 10 ]. In healthcare, early warning systems are used by hospital care teams to recognize the early signs of clinical deterioration, enabling the initiation of early intervention and management [ 27 ]. 1.2

Research objectives

This document will explore the reported uses of predictive algorithms and early warning systems in the educational context, focusing on higher education environments, most notably university courses. This scenario falls under the umbrella of learning analytics (LA), a particularization of data analytics which is usually de ned as \the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs" [ 26 ].

The study is presented as a systematic literature review, following the general guidelines established by Kitchenham and Charters [ 19 ], attempting to provide an answer to the following research questions: { RQ1: What are the most important purposes of using predictive algorithms in higher education, and how are they implemented? { RQ2: Which are the most notable examples of early warning systems applied in higher education scenarios?

Following this introductory section, this report explains the literature search process and the criteria that was followed to assess the relevance of analyzed documents. Next, the contents of the most relevant papers are summarized, addressing the research questions proposed above. Finally, some insights and discussion are presented at the end of this document. 2 2.1

Document retrieval

Search

The search process consisted in the retrieval of relevant documents available in online libraries and repositories. The selected sources were: { IEEE Xplore Digital Library. { ACM Digital Library. { Elsevier (ScienceDirect). { Wiley Online Library. { Springer (SpringerLink). { Google Scholar.

The following query string was run in each one of these platforms: (" e a r l y w a r n i n g s y s t e m " OR " p r e d i c t i v e a n a l y s i s " OR " p r e d i c t i v e a n a l y t i c s " OR " p r e d i c t i v e a l g o r i t h m ") " e d u c a t i o n " " u n i v e r s i t y " -" d i s a s t e r " -" m e d i c a l " -" h e a l t h "

The purpose of this query was to obtain documents related to the use of EWS and predictive algorithms in university contexts, while disregarding unrelated applications in the elds of natural disaster prediction and healthcare technologies | uses so common that they have entire journals dedicated to them. Additionally, publication dates were restricted to 2012 or newer, and only journal articles, conference proceedings and book extracts were considered.

Table 1 summarizes the results of the search procedure. Notice that due to Google Scholar's nature as an indexer of many di erent sources, some overlapping results with the rest of the libraries are expected. This search engine was included in order to obtain potentially relevant papers which are not available in any of the other digital libraries. 2.2

Rating

In order to rate the retrieved documents according to their relevance, the criteria listed in Table 2 are de ned. For the sake of this study, it is considered important that the paper presents a predictor or EWS which is useful for higher education scenarios, that the inner workings of their algorithms are clearly explained, and that the system has been tested and results exist.

Relevant

The usefulness of the predictive algorithm or EWS in a higher education scenario is limited.

A fair explanation of the data analysis process is provided, although with missing information or technicalities.

The predictive algorithm or EWS is tested in limited scenarios. Results may not be fully convincing.

Not relevant

The paper does not present a predictive algorithm or EWS applied to higher

education.

The document does not give any details

about the data analysis process or the algorithms that were used.

No tests are performed.

The document rating and selection process was carried out in three steps. First, papers were ltered by reading titles and abstracts, discarding those unrelated to the educational eld. Next, introductions and conclusions were analyzed in order to con rm that the documents address the points that were established as rating criteria. The resulting document list was more thoroughly analyzed, disregarding papers that fail to achieve at least a \relevant" rating in any of the three aspects considered for evaluation.

After completing the rating process, a narrower list including the most relevant papers is obtained. Table 1 indicates the amount of documents per source that satisfactorily meet the established criteria.

As previously stated, predictive analytics and EWS are extremely popular disciplines with applications in many di erent knowledge elds, which makes e ciently ltering search results an arduous task. This explains the fact that the amount of selected documents is relatively low compared to the quantity of yielded results. This is particularly true for the Elsevier and Wiley repositories | in the latter case, not even one document was found to be relevant for the educational context. It is also worth mentioning that most of the relevant papers returned by the Google Scholar searcher had already been selected from one of the other online libraries, and only unique articles are re ected as selected in the table. 3 3.1

Content review

Overview

Document Year Input data Prediction goal Key aspects Arnold [ 5 ] 2012 acDapedemrefmoogricmrahapinhscticeos,r,y. (thrLeeev-peloionftrsisckale). aWnCdeolwul-riEedsseWetlaSySbig.ltinesshatleesdd.

Krumm [ 21 ] 2014 peLrMfoSrmeanocret adnadta. (thrLeeev-peloionftrsisckale). SstEeuvWdeerSna.tl UEstsxuepddlioeirnse.r Waddington [ 31 ] 2014 LMS resource use. Final course grade. ISmtupdroevnetmEexnptlourpeorn.

Brown [ 7 ] 2016 StudendtatEax.plorer Risk level changes. SSttuuddeyEntWuEsiSxn.pglothreer Brown [ 8 ] 2017 StudendtatEax.plorer Bheesltpstmustderaeusngutgrsle.isngto SSttuuddeyEntWuEsiSxn.pglothreer Brown [ 9 ] 2018 StudendtatEax.plorer coIn-enureonllcmeeonft. SSttuuddeyEntWuEsiSxn.pglothreer LADA EWS.

Gutierrez [ 14 ] 2018 Gernardoellse,ddcaotaurfsreosm. Risk ocfoufarislein.g the decisSiounp-pmoarktsing of advisors.

SurreyConnect

EWS. Targets laboratory sessions.

Tries to nd the optimal time to apply an EWS in continuous assessment.

Incorporates data on students' life

habits.

Detecting student inactivity in order to predict dropout.

Input data related to e-book interaction.

Using the EWS does not lead to dropout reduction.

Akhtar [ 3 ] Howard [ 16 ] Wang [ 32 ]

2018 Cohen [ 12 ] 2017

LMS log data.

Dropout risk.

Akcap nar [ 2 ]

Plak [ 23 ]

E-book 2019 management system data.

Student 2019 demographics and performance.

Risk of failing the

course.

Risk of failing the

course.

Attendance, 2017 neighwbitohrsin, ltohceation Risk ocfoufarislein.g the

laboratory.

Demographics, 2018 intermediate task results.

Grades, attendance, engagement, library and dorm records.

Final course

grades.

Level of risk for several di erent events. 3.2

Predictive analysis in education

This section addresses RQ1 by summarizing the contents of papers related to the topic of predictive analytics in higher education. As will be shown, student success, performance and grades stand out as the most popular prediction objectives. Within this category, two di erent approaches can be identi ed: success predictors, which try to estimate whether a student will pass or fail a course; and grade predictors, which attempt to anticipate the nal grade of a student. Unique traits in each study include the nature of input data, the data processing algorithms that are used and the scenarios in which they are tested. Success prediction. These applications are mostly based on classi er algorithms.

Ornelas and Ordonez [ 22 ] proposed a Naive Bayesian classi er which was applied in a dozen courses taught at Rio Salado Community College (Arizona, USA). They used data from the institution's LMS as input, divided into two categories: engagement indicators (LMS logins and participation in online activities) and performance (points earned in course tasks). The classi er was able to predict success | that is, the student getting a C grade or better | with an accuracy of over 90% for eleven di erent courses, although not early enough so that it could properly work as an early warning system. This experiment was applied to a fairly big population, with a training sample of 5936 students and a validation sample of 2722.

Thompson et al. [ 28 ] used logistic regression to estimate the chances of student success in an introductory biology course taught during the rst semester of a university major program, with a total of 413 enrolled students. As opposed to the previous case, they exclusively used results from tests with no direct relationship with the course, which were taken right at the beginning of the semester. These were Lawson's Classroom Test of Scienti c Reasoning and the ACT Mathematics Test. Although this was not a perfect model to predict success by any means, it provided a rst estimation of students at risk before the course had even started.

Benablo et al. [ 6 ] introduced procrastination into the picture by surveying students on the time that they spend using social networks and playing online games. A SVM classi er was able to successfully identify underperforming students: 100% precision and 96:7% recall on a 100-instance data set.

Umer et al. [ 30 ] tried to estimate the earliest possible time within a course at which a reliable identi cation of students at risk could be made. The targeted course, an Australian introductory mathematics module with 99 enrolled students, used the continuous assessment system, in which multiple assignments are performed throughout the duration of the course, instead of just a nal exam. The input data were a combination of assignment results and LMS log data. Students were classi ed regarding their nal performance estimation | grades A, B, C, D, as well as failing or dropping the course. After one week, a Random Forest classi er was able to identify students at risk with 70% accuracy. This percentage increased to 87% after ve weeks, a point at which students had already completed 2 out of 7 total assignments.

Kostopoulos et al. [ 20 ] tried to improve the performance of traditional student success classi ers by implementing a co-training method. This technique consists on splitting the available data features into two independent and su cient views. It is particularly useful when the amount of unlabeled data is large compared to the number of labeled examples, since it allows to expand the labeled set by adding initially unlabeled entries that both views can classify with high certainty. This study targeted an introductory informatics module from a Greek open university, involving 1073 students. The feature split that was performed created one view containing students' demographic characteristics and academic achievements provided by their tutors, and a second view including LMS activity data. The co-training method was observed to outperform traditional classi ers such as Naive Bayes, k-NN and Random Forest, providing very accurate identi cations of poor performers towards the middle of the course.

Hirose [ 15 ] attempted to make an estimation of students' abilities using item response theory (IRT). The study was performed in the context of introductory mathematics courses under the continuous assessment system, in which students needed to answer a set of multiple-choice questions each week. Data from around 1100 students was available for this test. Thanks to IRT, question di culty was assessed together with students' abilities, resulting in a more fair judgment. At speci c times during the course, students were classi ed into \successful" or \not successful" using the Nearest Neighbor method. After seven weeks, roughly half the course, a misclassi cation rate as low as 18% was achieved; however, the number of false positives was noticeably high, meaning that many wellperforming students were identi ed as being at risk of failing.

Schuck [ 25 ] presented a unique study which tried to establish a correlation between the level of crime and violence around campus and student success. The experiment was possible thanks to data provided by university representatives of the US Department of Education, as well as the United States' National Center for Education Statistics. Overall, complete data from 1281 higher education institutions was available. The study used multivariate regression models in order to predict graduation rate, that is, the fraction of students who nish their degree within the intended number of years. Input data for this model included the amount of violent incidents and disciplinary measures per number of students, the percent of disciplinary actions that ended up with arrests, as well as student demographic information and school characteristics. As a result of analysis, rates of violence were observed to negatively a ect graduation years, as opposed to the rate of disciplinary measures, which is a positive indicator. Additionally, use of the student conduct system was observed to be better than criminal justice system for minor o enses.

Grade prediction. These applications are mostly built upon regression-based estimators.

There are several examples of grade predictors which take students' previous results as their main source of input data. Tsiakmaki et al. [ 29 ] used nal scores from courses imparted during the rst semester of a Business Administration degree (592 students) in order to predict grades from second semester subjects, obtaining fair results using Random Forest and SVM algorithms. On the other hand, Adekitan and Salau [ 1 ] ran an experiment in a Nigerian engineering school trying to determine how well the grade point average (GPA) over the rst three years of a degree could predict the nal, cumulative GPA over the entire veyear program. Out of the tested analysis algorithms, logistic regression yielded the best result, with a 89:15% accuracy over a 1841 student sample.

Jovanovic et al. [ 17 ] proposed a predictive model to be applied in courses following the ipped classroom teaching method, focusing on student interaction with pre-class learning activities. These activities included videos and documents with multiple choice questions, as well as problem sequences. The model was tested in a rst-year engineering course at an Australian university for three consecutive years, with a number of students ranging between 290 and 486. The study concluded that indicators of regularity and engagement related to pre-class activities had signi cantly superior predictive power than generic indicators such as the frequency of LMS logins.

Chen [ 11 ] assessed the quality and quantity of students' note-taking, both in and after class, to explore the e ects that this could have on academic performance. A population of 38 freshmen students from a Taiwanese university participated in the experiment. Students' notes were retrieved and copied after each lecture by the professor, who rated their quality based on accuracy and completeness regarding the contents of the lecture. The word count, in this case number of Chinese characters, was also recorded. The studio concluded that only the quality of the notes taken during the class was a signi cant predictor of the students' nal grade.

Amirkhan and Kofman [ 4 ] studied the e ects of stress overload | the destructive form of stress | over the GPA obtained by students. The experiment was conducted over two consecutive semesters with a population of 600 freshmen students, who were surveyed mid-semester in order to assess their levels of stress. As a result of predictive analysis, stress was found to be among the strongest performance predictors, having signi cant and negative relationship with nal GPA. However, it did not seem to have a direct relationship with dropout rate. 3.3

Early warning systems in education

This section addresses RQ2 by showing some of the most important EWS that were found in the literature. The applications listed below have the objective of identifying certain risks that students may be exposed to, and do it as soon as possible in order to take proper corrective measures in time. The most common risks to identify are high chances of a student failing or dropping out of a course or degree.

Arnold and Pistilli [ 5 ] present Course Signals, an EWS rst implemented at Purdue University which has become one of the most popular and referenced by the research community. This tool works in conjunction with the LMS Blackboard Vista, using data related to student demographics, performance, e ort and prior academic history. Thanks to an on-demand student success algorithm, instructors can obtain an estimation of the risk level of a student, color coded as green, yellow and red for increasing degrees of risk. The application then allows the instructor to take measures if required, such as sending a message to the student or scheduling a face-to-face meeting. Course Signals has been employed in many courses at Purdue since 2007, registering a signi cant improvement in student grades, as well as a decrease in dropout rate. As opposed to most other EWS, which are not past their experimental stage, Course Signals is a mature and well-established application with proven positive results throughout the last decade.

Another well-known EWS is Student Explorer. As described by Krumm et al. [ 21 ], and similarly to Course Signals, Student Explorer mines e ort and performance data from the institutional LMS in order to assess the likelihood of a student's academic failure. Students are classi ed with the labels \encourage", \explore" and \engage", in increasing order of risk, and student advisors can use this information to take corrective action.

Multiple other papers presented further experiments and improvements over the base Student Explorer application. Waddington and Nam [ 31 ] incorporated LMS resource use as input data, including information such as access to lecture notes or completion of assignments. Analysis using logistic regression determined a direct correlation between resource use and nal grade, with activities related to exam preparation having the strongest positive relationship with performance. Brown et al. performed several studies revolving around Student Explorer. They observed that students had a greater chance of entering the \explore" category if they were in large classes, sophomore level courses and courses belonging to pure scienti c degrees; while underperformance was still the most signi cant reason students entered the \engage" category [ 7 ]. They also used the tool to investigate how to best help struggling students recover. They concluded that students with moderate di culties bene ted the most from assistance planning their study behaviors, while those with severe di culties bene ted from better exam preparation [ 8 ]. Finally, these authors studied the e ect of co-enrollment in multiple courses over performance, establishing a correlation between being enrolled in at least one \di cult course" and a higher chance to experience academic struggles [ 9 ].

Gutierrez et al. [ 14 ] were the developers of LADA, a Learning Analytics Dashboard for Advisors. As its name implies, the main goal of this tool is to support the decision-making process of academic advisors. LADA incorporates a predictive module that estimates the students' odds of success by means of multilevel clustering. Input data includes student grades, courses booked by a student and the number of credits per course. The risk level of a student is calculated by comparing to other students with similar pro les from previous cohorts. LADA was deployed in two di erent universities, and student advisors claimed that the biggest advantage that it provides is being able to analyze a greater amount of scenarios within a given time frame.

Akhtar et al. [ 3 ] created SurreyConnect, a teaching assistant with the objective of supporting computer-aided design (CAD) courses at University of Surrey (England). Most of the utilities of this tool were useful for laboratory sessions, allowing the instructor to share her or his computer screen with students, broadcast the screen of a speci c student to the rest of the class or remotely connect to a student's computer in order to provide help. SurreyConnect also implements an analytics module with the purpose of identifying students at risk of failing the course. In order to do this, the application passively collects data during lab sessions regarding student attendance, location and neighbors within the lab, and time spent in class and doing exercises. A sample of 331 undergraduate students was selected to assess the usefulness of this feature, running an ANOVA test to identify the statistical signi cance of the input data, as well as applying Pearson correlation to identify the independent variables that in uence nal outcomes. Class attendance and time spent on tasks were shown to have a direct connection with learning outcomes, while student positioning in the classroom and sitting with a particular group of students also impacted performance.

Howard et al. [ 16 ] tried to nd out the optimal time to apply an EWS in a course using the continuous assessment system. The study targeted a Practical Statistics course at University College Dublin (UCD) with 136 participant students, in which 40% of the nal grade was awarded for completing certain tasks that were assigned each week throughout the course. The results of these tasks, as well as student demographic information and the number of times they accessed course resources, were collected as input for grade prediction. The data source was the institution's LMS, Blackboard. After testing multiple predictive models, Bayesian Additive Regressive Trees (BART) yielded the best results, being able to predict students' nal grade with a mean absolute error of 6:5% as early as at week 6, exactly halfway through the course. This provides a decently precise prediction for the teacher, early enough so that corrective measures can be taken.

Wang et al. [ 32 ] are the designers of an EWS applied in Hangzhou Normal University (China), with the goal of reducing student dropout and minimizing delays in graduation. This application stands out because it includes types of input data that are not seen anywhere else: besides information regarding students grades, attendance and engagement, it also includes records from the university library and dorm. These extra data enable a closer monitoring of study habits. The EWS assigns students di erent labels depending on the kind of risks they are exposed to, such as the risk of obtaining low grades, graduation delay or dropout. After three semesters, and using a sample of 1712 students, a Naive Bayes algorithm was able to perform risk classi cation with an accuracy of 86%, with grades and library borrowing data being among the key indicators.

Cohen [ 12 ] focused on quantitative analysis of student activity data in order to provide an early identi cation of learner dropout. The study hypothesized that students who drop out of the course will rst become inactive in course websites. The proposed EWS collects student activity data from the Moodle LMS, including number and types of actions performed, as well as their timing and frequency. For the reported test, data from 362 students was collected. The input data were analyzed in a monthly basis in order to detect signi cant activity drops by a student, who would be subsequently agged as at risk. The study concluded that two thirds of agged students would indeed end up dropping their courses or degrees.

Akcap nar et al. [ 2 ] built an EWS intended to be used in courses that use ebooks as learning material, using reading data in order to identify students at risk of academic failure. The data were collected from an e-book management system named BookRoll, used in several Asian universities and which students utilize in order to access course materials. This particularly study obtained information from 90 students registered in an Elementary Informatics course, registering their interactions with BookRoll, for instance, e-book navigation, page highlighting or note taking. Each week, analysis was performed to label students as low or high performing, trying multiple prediction algorithms. It was observed that an accuracy of 79% was achieved just with data from the rst three weeks. Random Forest performed the best with raw data, however, Naive Bayes became the best performing when transforming the input into categorical data.

Finally, Plak et al. [ 23 ] document a case in which the use of an EWS does not provide the expected bene ts. This experiment, conducted at Vrije Unversiteit in Amsterdam, provided student counselors with an analytics monitor that allowed them to identify low-performing students. The EWS used data related to student progress and demographics. However, the introduction of the tool did not lead to a reduction in dropout or an increase in obtained credits. While early identi cation of at-risk students is useful, the underlying problem that causes poor performance is ultimately undetermined. 4

Conclusion

Within the enormous world that is data analytics, the learning analytics eld could be seen as just a small niche, mostly covered by academic research. However, a closer look into the discipline reveals that learning analytics is an extensive subject in its own right. The literature review presented in this document revealed that there exists a considerable variety of studies and applications revolving around predictive algorithms in the educational eld, which is itself an important subject of study within learning analytics.

As an answer to RQ1, it was observed that most of the tools and predictive algorithms that were presented shared similar goals: most commonly, predict student grades, assess their chance of failure or their risk of dropping o a course or degree. Nevertheless, the great variety of educational contexts meant that each study had a unique approach in order to achieve said goals, meaning that speci c implementation aspects greatly di er from case to case. Naturally, the availability of certain types of data, such as those related to student engagement and performance, is the factor that in uences the analysis method the most. However, many other aspects need to be taken into account in order to design a good predictor. Examples of elements that can signi cantly a ect the analytics process are teaching strategy | such as the ipped classroom [ 17 ] | , assessment method | such as continuous assessment [ 16, 30 ] | , geographical context, student demographics or the year within a degree program. Thus, there is not a single predictive algorithm that can be considered better than the rest in all possible scenarios.

As for RQ2, this document covered some of the most important instances of EWS in education. Course Signals [ 5 ] and Student Explorer [ 21 ] are worthy of a special mention. The former has been applied in practical scenarios for over a decade and is one of the most referenced in the literature, while the latter has a pivotal role in several learning analytics experiments.

In general, the introduction of predictors and EWS in educational environments has been helpful in order to optimize the learning process and improve student performance. However, as of the year 2019, they have mostly been used in experimental environments only, with the notable exception being the Course Signals EWS. Additionally, analysis results mean nothing if there is not a person or group of people that are able to interpret them and react accordingly. As of today, these tools are not able to fully take on the gure of a student advisor.

It is worth noticing that data analytics in general has been an extremely active research area for many years, and it still is today. This also applies to learning analytics. As a matter of fact, most of the papers included in this literature review were published in 2017 or later. This means that the subject is most de nitely not fully explored, and that many innovative pieces of work will keep arising in the foreseeable future, in uenced by changes in teaching trends and progress in data analysis techniques.

Overall, this study highlights the many possibilities that predictive analytics provides in order to boost the learning process. At the same time, it is evident that building a single solution that will work well for many di erent types of learning environments is a very di cult task. This remains one of the greatest challenges within the learning analytics discipline.

Acknowledgment. This work is partially nanced by public funds granted by the Galician regional government, with the purpose of supporting research activities carried out by PhD students. (\Programa de axudas a etapa predoutoral da Xunta de Galicia | Conseller a de Educacion, Universidade e Formacion Profesional")

References

1. Adekitan , A.I. , Salau , O. : The impact of engineering students' performance in the rst three years on their graduation result using educational data mining . Heliyon 5 ( 2 ), e01250 (Feb 2019 ). https://doi.org/10.1016/j.heliyon. 2019 .e01250, http://www.sciencedirect.com/science/article/pii/S240584401836924X

2. Akcap nar, G., Hasnine , M.N. , Majumdar , R. , Flanagan , B. , Ogata , H.: Developing an early-warning system for spotting at-risk students by using eBook interaction logs . Smart Learning Environments 6 ( 1 ), 4 (May 2019 ). https://doi.org/10.1186/s40561-019-0083-4, https://doi.org/10.1186/s40561-019- 0083-4

3. Akhtar , S. , Warburton , S. , Xu , W.: The use of an online learning and teaching system for monitoring computer aided design student participation and predicting student success . International Journal of Technology and Design Education 27 ( 2 ), 251 {270 (Jun 2017 ). https://doi.org/10.1007/s10798-015-9346-8, https://doi.org/10.1007/s10798-015-9346-8

4. Amirkhan , J.H. , Kofman , Y.B. : Stress overload as a red ag for freshman failure and attrition . Contemporary Educational Psychology 54 , 297 {308 (Jul 2018 ). https://doi.org/10.1016/j.cedpsych. 2018 . 07 .004, http://www.sciencedirect.com/science/article/pii/S0361476X17301108

5. Arnold , K.E. , Pistilli , M.D.: Course Signals at Purdue: Using Learning Analytics to Increase Student Success . In: Proceedings of the 2Nd International Conference on Learning Analytics and Knowledge . pp. 267 { 270 . LAK '12, ACM , New York, NY, USA ( 2012 ). https://doi.org/10.1145/2330601.2330666, http://doi.acm. org/10 .1145/2330601.2330666

6. Benablo , C.I.P. , Sarte , E.T. , Dormido , J.M.D. , Palaoag , T. : Higher Education Student's Academic Performance Analysis Through Predictive Analytics . In: Proceedings of the 2018 7th International Conference on Software and Computer Applications . pp. 238 { 242 . ICSCA 2018 , ACM, New York, NY, USA ( 2018 ). https://doi.org/10.1145/3185089.3185102, http://doi.acm. org/10 .1145/3185089.3185102, event-place: Kuantan, Malaysia

7. Brown , M.G., DeMonbrun , R.M. , Lonn , S. , Aguilar , S.J. , Teasley , S.D.: What and when: The Role of Course Type and Timing in Students' Academic Performance . In: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge . pp. 459 { 468 . LAK '16, ACM , New York, NY, USA ( 2016 ). https://doi.org/10.1145/2883851.2883907, http://doi.acm. org/10 .1145/2883851.2883907, event-place: Edinburgh, United Kingdom

8. Brown , M.G., DeMonbrun , R.M. , Teasley , S.D.: Don't Call It a Comeback: Academic Recovery and the Timing of Educational Technology Adoption . In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference . pp. 489 { 493 . LAK '17, ACM , New York, NY, USA ( 2017 ). https://doi.org/10.1145/3027385.3027393, http://doi.acm. org/10 .1145/3027385.3027393, event-place: Vancouver, British Columbia, Canada

9. Brown , M.G., DeMonbrun , R.M. , Teasley , S.D.: Conceptualizing Coenrollment: Accounting for Student Experiences Across the Curriculum . In: Proceedings of the 8th International Conference on Learning Analytics and Knowledge . pp. 305 { 309 . LAK '18, ACM , New York, NY, USA ( 2018 ). https://doi.org/10.1145/3170358.3170366, http://doi.acm. org/10 .1145/3170358.3170366, event-place: Sydney, New South Wales, Australia

10. Bussiere , M. , Fratzscher , M. : Towards a new early warning system of nancial crises . Journal of International Money and Finance 25 ( 6 ), 953 {973 (Oct 2006 ). https://doi.org/10.1016/j.jimon n. 2006 . 07 .007, http://www.sciencedirect.com/science/article/pii/S0261560606000532

11. Chen , P.H. : The E ects of College Students' In-Class and After-Class Lecture Note-Taking on Academic Performance. The Asia-Paci c Education Researcher 22 ( 2 ), 173 {180 (May 2013 ). https://doi.org/10.1007/s40299-012-0010-8, https://doi.org/10.1007/s40299-012-0010-8

12. Cohen , A. : Analysis of student activity in web-supported courses as a tool for predicting dropout . Educational Technology Research and Development 65 ( 5 ), 1285 {1304 (Oct 2017 ). https://doi.org/10.1007/s11423-017-9524-3, https://doi.org/10.1007/s11423-017-9524-3

13. Gandomi , A. , Haider , M. : Beyond the hype: Big data concepts, methods, and analytics . International Journal of Information Management 35 ( 2 ), 137 {144 (Apr 2015 ). https://doi.org/10.1016/j.ijinfomgt. 2014 . 10 .007, http://www.sciencedirect.com/science/article/pii/S0268401214001066

14. Gutierrez , F. , Seipp , K. , Ochoa , X. , Chiluiza , K. , De Laet , T. , Verbert , K. : LADA: A learning analytics dashboard for academic advising. Computers in Human Behavior PP ( Dec 2018 ). https://doi.org/10.1016/j.chb. 2018 . 12 .004, http://www.sciencedirect.com/science/article/pii/S0747563218305909

15. Hirose , H.: Success/Failure Prediction for Final Examination Using the Trend of Weekly Online Testing . In: 2018 7th International Congress on Advanced Applied Informatics (IIAI-AAI) . pp. 139 { 145 (Jul 2018 ). https://doi.org/10.1109/IIAIAAI. 2018 .00036

16. Howard , E. , Meehan , M. , Parnell , A. : Contrasting prediction methods for early warning systems at undergraduate level . The Internet and Higher Education 37 , 66 {75 (Apr 2018 ). https://doi.org/10.1016/j.iheduc. 2018 . 02 .001, http://www.sciencedirect.com/science/article/pii/S1096751617303974

17. Jovanovic , J. , Mirriahi , N. , Gasevic , D. , Dawson , S. , Pardo , A. : Predictive power of regularity of pre-class activities in a ipped classroom . Computers & Education 134 , 156 {168 (Jun 2019 ). https://doi.org/10.1016/j.compedu. 2019 . 02 .011, http://www.sciencedirect.com/science/article/pii/S0360131519300405

18. Kempler , S. , Mathews , T. : Earth Science Data Analytics: De nitions, Techniques and Skills. Data Science Journal 16 ( 0 ), 6 (Feb 2017 ). https://doi.org/10.5334/dsj2017-006, http://datascience.codata.org/articles/10.5334/dsj-2017-006/

19. Kitchenham , B. , Charters , S. : Guidelines for performing Systematic Literature Reviews in Software Engineering ( 2007 )

20. Kostopoulos , G. , Karlos , S. , Kotsiantis , S.B. : Multi-view Learning for Early Prognosis of Academic Performance: A Case Study . IEEE Transactions on Learning Technologies PP ( 2019 ). https://doi.org/10.1109/TLT. 2019 .2911581

21. Krumm , A.E. , Waddington , R.J. , Teasley , S.D. , Lonn , S.: A Learning Management System-Based Early Warning System for Academic Advising in Undergraduate Engineering . In: Larusson, J.A. , White , B . (eds.) Learning Analytics: From Research to Practice, pp. 103 { 119 . Springer New York, New York, NY ( 2014 ). https://doi.org/10.1007/978-1- 4614 -3305-7 6, https://doi.org/10.1007/978-1- 4614 -3305-7 6

22. Ornelas , F. , Ordonez , C. : Predicting Student Success: A Nave Bayesian Application to Community College Data . Technology, Knowledge and Learning 22 ( 3 ), 299 {315 (Oct 2017 ). https://doi.org/10.1007/s10758-017-9334-z, https://doi.org/10.1007/s10758-017-9334-z

23. Plak , S. , Cornelisz , I. , Meeter , M., van Klaveren , C. : Early Warning Systems for More E ective Student Counseling in Higher Education { Evidence from a Dutch Field Experiment . In: SREE Spring 2019 Conference. p. 4 . Washington, DC, USA (Mar 2019 )

24. Reid , B. : Global early warning systems for natural hazards: systematic and people-centred . Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 364 ( 1845 ), 2167 {2182 (Aug 2006 ). https://doi.org/10.1098/rsta. 2006 . 1819 , https://royalsocietypublishing.org/doi/full/10.1098/rsta. 2006 .1819

25. Schuck , A.M. : Evaluating the Impact of Crime and Discipline on Student Success in Postsecondary Education . Research in Higher Education 58 ( 1 ), 77 {97 (Feb 2017 ). https://doi.org/10.1007/s11162-016-9419-x, https://doi.org/10.1007/s11162-016- 9419-x

26. Siemens , G.: 1st International Conference on Learning Analytics and Knowledge 2011 j Connecting the technical, pedagogical, and social dimensions of learning analytics ( Jul 2010 ), https://tekri.athabascau.ca/analytics/

27. Smith , M.E.B. , Chiovaro , J.C. , O'Neil , M. , Kansagara , D. , Quinones , A. , Freeman , M. , Motu 'apuaka, M. , Slatore , C.G. : Early Warning System Scores: A Systematic Review . VA Evidence-based Synthesis Program Reports , Department of Veterans A airs, Washington (DC) ( 2014 ), http://www.ncbi.nlm.nih.gov/books/NBK259026/

28. Thompson , E.D. , Bowling , B.V. , Markle , R.E. : Predicting Student Success in a Major's Introductory Biology Course via Logistic Regression Analysis of Scienti c Reasoning Ability and Mathematics Scores. Research in Science Education 48 ( 1 ), 151 {163 (Feb 2018 ). https://doi.org/10.1007/s11165-016-9563-5, https://doi.org/10.1007/s11165-016-9563-5

29. Tsiakmaki , M. , Kostopoulos , G. , Koutsonikos , G. , Pierrakeas , C. , Kotsiantis , S. , Ragos , O. : Predicting University Students' Grades Based on Previous Academic Achievements . In: 2018 9th International Conference on Information, Intelligence, Systems and Applications (IISA) . pp. 1 { 6 (Jul 2018 ). https://doi.org/10.1109/IISA. 2018 .8633618

30. Umer , R. , Susnjak , T. , Mathrani , A. , Suriadi , S.: A learning analytics approach: Using online weekly student engagement data to make predictions on student performance . In: 2018 International Conference on Computing, Electronic and Electrical Engineering (ICE Cube) . pp. 1 { 5 (Nov 2018 ). https://doi.org/10.1109/ICECUBE. 2018 .8610959

31. Waddington , R.J. , Nam , S. : Practice Exams Make Perfect: Incorporating Course Resource Use into an Early Warning System . In: Proceedings of the Fourth International Conference on Learning Analytics And Knowledge . pp. 188 { 192 . LAK '14, ACM , New York, NY, USA ( 2014 ). https://doi.org/10.1145/2567574.2567623, http://doi.acm. org/10 .1145/2567574.2567623, event-place: Indianapolis, Indiana, USA

32. Wang , Z. , Zhu , C. , Ying , Z. , Zhang, Y. , Wang , B. , Jin , X. , Yang , H.: Design and Implementation of Early Warning System Based on Educational Big Data . In: 2018 5th International Conference on Systems and Informatics (ICSAI) . pp. 549 { 553 (Nov 2018 ). https://doi.org/10.1109/ICSAI. 2018 .8599357