In the peculiar realm of higher education, some of the challenges of Public Administration, in terms of quality assurance and data intelligence, can be addressed thanks to the complex ecosystem based on the careers of students and their engagement with the host academia. University governance, ranging from the university Rector and Quality Assurance committee to single heads of degree courses, needs to rely on quantitative and unbiased measures when designing and planning actions. This paper reports on an ongoing project started at Parma University in 2019, that has multiple goals: (1) to collect various sources of students' career-related raw data and to and provide simple access to aggregated analyses through a web portal; (2) to ofer an AI based synthesis, in form of automatically generated reports in natural language; (3) to analyze data to detect and predict potential issues (e.g., students drop-out, classes attendance, graduation time estimations, blockages in the career) that can be promptly highlighted, for immediate intervention. As opposed to the majority of academic analytics implementations, particular care is devoted to minimizing ethics and privacy issues and adhering to explainable AI principles in the generation of synthetic explanations of charts and reports. The results of lines of research (2) and (3) will be integrated in the portal (1) that is currently deployed at Parma University.
In the last decades, learning analytics (LA) received growing attention from educational researchers [1, 2]. Even if there is no general accepted definition of LA, a widely referenced one considers LA as the measurement, analysis and reporting of data about learners, for purposes of understanding and optimizing learning and the environments in which it occurs [3]. A number of benefits arising from LA include the identification of at-risk students, the possibility of developing additional support for coping with academic requirements and expectations [4].
academic experience for students is complex.
In this paper, we present our ongoing work on students’ career analytics at Parma University. We focus here on the core aspect of a student’s career: exams proficiency through her/his academic life. The whole plethora of services (e.g., housing, libraries, counseling, financial support, sport associations) that contribute to a successful experience have an impact that is less directly measurable and its analysis may result in greater privacy concerns. The design, maintenance and improvement of a degree course require the systematic monitoring of its performance, through gathering and analyzing data about students’ careers. The goal is to monitor how the higher education system operates and whether it is reaching, or it will reach with a predictive approach, its objectives and educational targets. To this aim, the central education authority (Ministry) usually defines a set of indicators that are also used to allocate additional resources to universities. Accurate evaluation of these indicators is then essential, and it must include data comparison and efective reporting as well. This scenario involves careful eforts by several actors (quality assurance, degree courses’ council and reviewing committees, joint studentteacher committees, single teachers) at diferent scopes (university, department, degree course, single course).
Planned periodical monitoring promotes action planning and evaluation of feedback about previous actions. While key question is investigated: the implications of eduprojecting students’ careers to simple measures (number cational automation, or, in other words, what kind of of freshmen, drop-out/completion rate, average time to responsibility is given to the automated part while bepass an exam, etc.) captures only a fraction of the complex ing relieved from human activity. Such aspects will be dynamics involved, it certainly helps in detecting clear further discussed in Section 3. symptoms of potential issues to be further investigated.
Our goal is to create a platform that supports academic 1.1. Discussion governance with an efective data-driven pipeline to be integrated into routine activities. The platform enables Before dwelling into the technical details, let us focus on the structuring of the link between students’ outcomes some key goals and novel opportunities ofered by the and policy issues. Its design is rather diferent from typi- framework introduced above. cal architectures and it focuses on three building blocks: From report creation to report explanation Auto1. making information available (Section 2): raw mated help in raw data processing and issues identificadata is processed and aggregated with privacy tion promotes higher quality activities. In particular, it alcompliance. Various and rich metrics are intro- lows one to focus on analyzing possible causes, providing duced in order to capture shades and nuances in context, explaining dynamics and designing correcting career evolutions. A web portal enables browsing actions to mitigate negative trends, rather than spending information with a Role-Based Access Control time on collecting data, handcrafting charts and writing approach for visibility; the report in its descriptive part (which is time-wise pre2. explainable AI-based report creation (Section 3.1): dominant). It can be foreseen a shift towards high-level even aggregated analyses require relevant human and valuable tasks since manual and mechanic operations time to be evaluated. An automated process al- are already performed, which translates into increased lows for identifying outliers (i.e., potential issues) satisfaction and better use of experts’ competencies. This in some metrics and synthesizing a discussion positive impact relies on the usage of an explainable AI about such findings. Moreover, natural language (or xAI, see Section 3.1). This kind of AI can be trusted, generation controls an unbiased description of verified and included in high-stakes risk activities. charts, for better interpretability and comparison; 3. AI-based predictions (Section 4): university online services usage and its correlation to students’ careers can feed predictive models that serve as early detectors of potential issues. Such monitoring can trigger proper actions to be applied while the issues are still developing (and way before the semester is over).
Unbiased analysis The presence of an xAI-assisted pipeline promotes the reduction of manual errors in data transcription and analysis. Commonly, analyses can be operated by hundreds of people with diferent roles (e.g., professors, technicians, managers) and backgrounds. A processing that runs on a common baseline smooths out biases in the collection of data. Moreover, natural language descriptions are processed under uniform metrics (e.g., the same modulation of qualitative adjectives) in order to provide unbiased terminology. These standardized metrics allow fair comparisons among diferent universities, degrees courses and/or the same geographic area.
sified as Learning Analytics and/or Educational Data Analytics. See [7] for a recent review of data mining techniques used for the prediction of students’ drop-out (long-term as well as graduation delays). Interestingly, so far there have been few applications based on Convolutional Neural Networks. There have been specific studies with Italian Universities as case studies: [8], e.g., predicts clusters of students’ early drop-outs based on online polls. Various systems have been implemented in the past decade, mainly in USA academia. Predictive Analytics Reporting (PAR) Framework [9] used data analytics to improve student success and retention. In the original formulation analyzed data were: students’ backgrounds, GPA and general information about their careers. Since then, such systems expanded and covered many other aspects, including social and financial data. Later, academic analytics examples flourished in Europe as well [10].
Concerns about privacy-related issues in treating personal information have been raised [11]. In [12] another Improved accuracy vs privacy In common reports and oficial data of Ministry’s reports, quantitative measures are actually limited to macroscopic trends (e.g., number of students/year, amount of credits earned). Finer-level details can be retrieved from raw data exploration and/or from potentially biased investigations (students’ feedback about classes, teachers’ considerations, etc). Our goal is to include high accuracy (more measures to help uncover small issues that may propagate to large consequences during students’ careers) at a limited cost in terms of personal data to be processed. Clearly, anonymized transcripts data (exams proficiency) are at the basis of our analyses. Moreover, we believe that anonymized information about university digital services usage is enough to serve the purpose. In particular, no social nor financial data are included. Quality and care in monitoring data • the grade (in Italian scale between 18 and 30).
The combination of the above presented measures allows the creation of various analyses (around 40 in the current version of the portal), e.g., the load of non-passed exams over time (the main cause of graduation delays); multi-dimensional combinations of proficiency, based on time/mark/number of attempts; delayed exams; patterns in the order of exams. Dashboards and comparative sections highlight distributions and potential outliers from any metric that suggest further investigation. 3. Explainable AI for reports require significant (human) time. Even the availability of an interactive web portal that ofers a clear presentation of any analysis does not provide a tool for digesting a fast summary, resulting in a data overload for users. An xAI that sieves relevant facts can handle an increase of metrics and data sources (see Section 4).
Frequency of the analysis An xAI processing of analyses allows one to increase the frequency of monitoring and evaluation of actions’ impact. Typically, a monitoring and steering infrastructure meets every 6–12 months (when exams data can be compared). However, it is possible to imagine even real-time monitoring, by moving towards predictive analytics: fresh and available data are needed (by means of a Datalake that collects students’ related information) and predictions can be devised. In this case, potential issues can be predicted and tracked earlier and improve the quality assurance impact.
a web showcase of charts of aggregated data, is only the first step towards a data-driven support system for Feedback to students Our data analysis service can be decision-making. In fact, even at an aggregated level, tuned to provide single students a means to track their the amount of information exposed is very large. For own progress [13], by comparing individual performance example, referred to a large size University as Parma to the one of the associated group (e.g., degree course (8K freshmen/year), having roughly 100 degree courses colleagues, department etc). tracked, with an average of 20 exams for 3 years cycle and 12 for 2 years cycles, browsing the rich set of data aggregations and capturing peculiar aspects of careers 2. A web portal for data analysis becomes time-consuming and dispersive. We envision an additional step devoted to speeding Since 2019, Parma University pioneered the deployment up the task of identifying and reporting issues. Once atof a multi-role internal platform that processes students’ tention points are clearly defined, it is possible to use AI data into aggregated analyses. The source comes from to generate a higher-level report, written in natural lanviews of the Student Management System named Esse3, guage, that describes relevant charts that contain an issue. provided by Cineca (Italian university consortium for Commonly this time-consuming task, namely browsing research support, IT and HPC services). data, charts generation and text writing is at the basis of
The portal is implemented with a combination of any report at any scope level. An automated report can Python procedures (for back-end processing), Angular become the basis for the core interpretation, contextualand Echarts (for front-end service). Users are authenti- ization and decision-making. cated by the university Shibboleth service and identified The issue with browsing data and their graphic repreaccording to their role: the head of degree courses can sentation is not only about time and practicality. Educaaccess the analyses carried out for their courses; the head tional contexts are very sensitive to prejudices, percepof departments can access degree courses belonging to tions and assumptions on how diferent categories of stutheir department as well as comparisons among those dents (slow vs. quick careers, student workers, students courses; administrators can access to all department data with disabilities or learning impairments, non-resident and they can compare all courses in their university. students) can perform and how their pattern of studies
Weekly, a body of roughly 1.5 million rows relative will predictably develop towards the degree or the dropto students and courses spanning 12 years is processed. out. These assumptions not only can heavily influence Information about each anonymized student’s curricu- the analysis of the data, but also generate professors’ lum is processed in order to provide the following main attitudes and behaviors that could possibly condition stumetrics: dent’s learning in order to confirm such assumptions. Using explainable AI for the first level of interpretation can guarantee a bias-free narrative of the most relevant correlation among diferent phenomena and data sets, providing a powerful instrument for all actors involved and, namely, acting as a continuous professional development tool for professors interested in improving their didactics. • a course has been attended, an exam has been
attempted/passed; • the interval between the last day of lessons and the first exam attempt; the interval between the ifrst attempt and the date of the last attempt or in which the exam was passed; • number of attempts;
Since the choice of what data to be retained in the report can introduce biases and, in turn, influence political choices, in our opinion, this process must be transparent and trustworthy. We resort to explainable Artificial Intelligence to control the process of selecting the aspects to be described and how to faithfully describe a particular set of data in natural language. 3.1. Explainable Artificial Intelligence
ture desirable properties of high-risk systems based on AI. Such systems should ensure transparency, exhibit ethical behavior, and support their results in terms of intelligible descriptions, accountability, security, privacy, and fairness [16]. The adoption of AI systems, especially in Public Administration (PA) contexts, depends on the capability of providing a high-level description of their inner activities. This would promote interpretability and Figure 1: Example of xAI chart visual commentary transparency of the inferences that lead to a result.
The urge for explainability in AI applications represents an opportunity for discontinuity with respect to data. Systems able to textually summarize data (e.g, comthe “traditional” approaches adopted in sub-symbolic AI, ing from stock prices, healthcare domain etc.), such as where the AI system acts as a black box. In other words, time-series, can make data more accessible in cases where such systems, usually relying on Machine Learning (ML), the interpretation of visualizations is made dificult or Deep Learning (DL), etc., cannot provide high-level ex- hindered for people with visual impairments or when planations supporting the output of their inferences [17]. readers are not expert or have limited cognitive abilities The design of an architecture that is both explainable and in comprehending and analyzing complex charts. ML/DL free represents a goal of current research in AI. The main challenges involved in D2T are the proper
As strategic choice, we opt for the use of robust and identification of what to describe —i.e., selecting the key of-the-shelf technologies of symbolic AI, to reach xAI descriptive elements in the input data— and how to texcompliance. In this frame of mind, we promote Logic tually describe such elements in generating the output Programming (LP) as the explainable core of an xAI sys- narration. Our approach, presented in [18], designs an tem. LP techniques enable both the representation of xAI-compliant system integrating Python (to perform knowledge at a higher level of abstraction (ranging from raw numerical calculations) and the declarative logicgeneral ontologies to domain-specific knowledge), and based framework of Answer Set Programming (ASP) to the reasoning activity, even by mimicking the human carry out reasoning. We extract the candidate key deway of thinking. Moreover, in LP-based systems, both scriptors of the series, by applying curve fittings: various the inference steps and the outcome of the reasoning of parameterized function prototypes (e.g., lines, polycan be immediately justified by singling out which infer- lines, sinusoids, etc.) are matched against fragments of ence rules have been used by the system and how the input data. This step, performed by a Python program, input knowledge has been processed by these rules. For produces a collection of candidate descriptions of porthese reasons, the resulting framework is not only na- tions of input data, labeled by a measure of accuracy tively explainable, but human users can put themselves (e.g., the Root Mean Square Error involved in the approxin a human-in-the-loop interaction with the xAI system, imation). Then, the ASP engine enters into play: The in order to detect possible flaws in the automated pro- (fragments of) curves are combined to obtain more abcess. This interaction enables a fruitful review of the stract descriptions of larger portions of the series. For knowledge base, e.g., to detect incoherent portions of example, a fragment of data described as a decrease folthe input, inconsistent inference rules, uncertainty or lowed by a fragment where values increase, are “merged” incompleteness in the sources of the input knowledge. in a single description of a valley. An optimization step identifies the descriptions that better represent the data 3.2. Automatic data-to-text series. Fig. 1 shows an example where the analysis detects some prominent details in the data series, such as The data-to-text generation (D2T) task consists in au- drops and peaks. The last step consists in converting the tomatically generating descriptions from non-linguistic qualitative descriptions into simple textual narration. 3.3. Automatic report generation of a general ML approach (in contrast to xAI) can serve the purpose. It is acceptable to cope with a limited error In pursuing transparent processing, an explicit set of in the estimation of student drop-outs if the purpose of attention triggers is defined. For example, a particular the prediction is to activate focused support procedures distribution is considered to be an outlier if it falls below for students. However, when predictions become the the 20th percentile. Such filtering among all analyses basis for other types of interventions, xAI tools should retrieves those alert cases that will compose the detailed be favored for transparent high-stakes decisions. description of the report. Let us present our ongoing research on the AI domain, under the above-mentioned assumptions. In particular, 4. AI predictions we plan to implement two standard approaches to extract insights from the aggregated careers of students: unsuIn this section we discuss our ideas about predictive ana- pervised learning and supervised learning. Unsupervised lytics in the domain of students’ careers. The goal is to learning models, such as clustering and k-nearest neighpredict the evolution of careers so that to anticipate po- bors, discover patterns in untagged data, that is students’ tential disengagement issues ahead of time. As discussed careers and use of services in our case. The rationale in [19], at the beginning of studies, the compensation of behind this first line of research is to detect behavioral diferent students’ backgrounds is fundamental towards a patterns that are a proxy for student engagement. For successful career. Early signs of disengagement dynamics example, the use of services can be clustered to identify during the first months of academic life should be inter- the group of students who are highly involved in the preted as soon as possible by governance, to enact actions learning process, the group of those whose involvement like focused tutoring, mentoring and counseling to mit- wanes as the lessons progress, and finally, the group of igate those phenomena while they are happening. The students who do not seem to have any interest in the typical (manual) revision cycle of courses is performed course. The latter activity is particularly challenging at the end of the teaching period, possibly enriched by because many degree courses have a high percentage the trend of exams performance. However, any action of working students who fall into the third group, and would be efective only on next year’s edition. anomaly detection techniques may be considered. Over
Our plan is to develop a fine time granularity analy- all, this makes it possible to identify critical issues way sis that highlights any change in students’ engagement before the examination session and to intervene with at any time, even before measuring the results of the support activities (second group) or better orientation exams. The accuracy and prediction span strongly de- activities (third group). Unsupervised learning models pend on the type of ingested data. The challenge is to can also be used to identify dificult-to-pass exams, evaluse the least amount of information to produce accurate uate the relative study approach, and determine whether predictions, since privacy issues may arise when inter- study groups, which students tend to form on their own, secting diferent kinds of personal information. Exams can be beneficial and, where appropriate, foster them proficiency is also a consequence of students’ actions with targeted interventions and resources. Complemenand/or attitudes towards studying. After the spread of tary, the supervised learning models solve classification the COVID-19 pandemic, teaching methods have adapted and regression problems where the data consists of lato remote emergency teaching [20] and after the release beled samples, one example is the students’ careers. We of lockdown restrictions the usage of online resources has plan to exploit supervised models, such as generalized linbeen maintained. Digital services can be tracked as fine ear models, logistic regression, support vector machines, descriptors of students’ engagement with the university. decision trees and random forest, to address two main In particular, access to study materials can approximate time-related issues, that is the time taken to pass every the amount of relative commitment the student shows single examination and the time taken to obtain the final in her/his career. Such type of information has difer- title. In the former case, in addition to a basic descriptive ent characteristics compared to simple transcripts of the statistics analysis, supervised models can help in idencareer: it is real-time, accompanies the students through- tifying supporting activities, to be deployed during the out their career and can capture diferent attitudes to course, that could improve performance in terms of final studying. It also correlates with exams proficiency, that, grades and exam pass time. While in the second case, however, has a much slower pace (several months). We constant career monitoring supported by a model that think that the combination of fine and large-scale phe- can predict the progress of examinations passed each nomena can provide the right balance between accuracy year and the date of graduation can be useful for baland privacy. ancing the study load and supplementary resources (e.g.,
The prediction of a student’s career evolution (short teaching tutors, laboratory activities, study groups). and long-term) can be performed with AI tools. It is important to establish whether the amount of opaqueness
dents at risk of academic failure within the educational data mining framework, Social Indicators This paper reviewed an ongoing work on learning analyt- Research 146 (2019) 41–60. ics at Parma University. The multi-role web portal being [9] S. L. Watson, B. Gemin, J. Ryan, Using predictive deployed presents aggregated analyses that help in de- analytics to identify at-risk students in higher edutecting potential blockages in the career of students and cation., Educause rev. 47 (2012) 31–40. in comparing them at diferent levels of aggregation (i.e., [10] A. Bethencourt-Aguilar, et al., The digital transtime-wise, course, degree course, department, university). formation of postgraduate degrees. a study on acaThe portal is the basis for two AI-based challenges: an demic analytics at the University of La Laguna, in: explainable AI-based automatic report generation and Proc. of JICV’21, 2021, pp. 1–4. real-time monitoring and prediction of students’ careers. [11] N. Sclater, A. Peasgood, J. Mullan, Learning analytThe first one can be safely integrated into governance, ics in higher education, Jisc 8 (2016). with advantages in monitoring several metrics with less [12] N. Selwyn, et al., Digital technologies and the auhuman cost in preparing documents. The second one tomation of education—key questions and concerns, ofers a solution for a fast-acting governance in order Postdigital Science and Education (2023) 15–24. to contribute to lowering the drop-out ratio (according [13] R. Yanosky, D. C. Brooks, Integrated planto the EU goals 2030 [21]). Moreover, in a more inte- ning and advising services (IPAS) research, grated didactic perspective, both AI-based instruments 2013. library.educause.edu/resources/2013/8/ can be employed in building typologies of students that integrated-planning-and-advising-services-ipas-research. relate learning styles and approaches with performance [14] A. Barredo Arrieta, et al., Explainable artificial intelin terms of engagement, career length and final outcomes ligence (XAI): Concepts, taxonomies, opportunities (degrees or dropping out). Caveats about privacy and and challenges toward responsible AI, Inf. Fusion ethics for the presented approaches have been discussed. 58 (2020) 82–115.
Even if the academic dynamics are not directly appli- [15] A. Adadi, M. Berrada, Peeking inside the black-box: cable to other Public Administration domains, we believe A survey on explainable artificial intelligence (XAI), that this methodology could have a positive impact on IEEE Access 6 (2018) 52138–52160. many other activities. From our point of view, xAI princi- [16] European Commission, Proposal for ples and report creation, for example, will be a strategic a regulation laying down harmonised asset for lighter and more efective quality assurance. rules on artificial intelligence, 2021. digital-strategy.ec.europa.eu/en/library/ References proposal-regulation-laying-down-harmonisedrules-artificial-intelligence. [1] G. Siemens, Learning analytics: The emergence [17] C. Rudin, Stop explaining black box machine learnof a discipline, American Behavioral Scientist 57 ing models for high stakes decisions and use inter(2013) 1380–1400. pretable models instead, Nature Machine Intelli[2] Y. K. Hui, L. F. Kwok, A review on learning analytics, gence 1 (2019) 206–215.
Int. J. of Innovation and Learning 25 (2019) 197–222. [18] A. Dal Palù, A. Dovier, A. Formisano, An xAI ap[3] G. Siemens, P. Long, Penetrating the fog: Analytics proach for data-to-text processing with ASP, in: in learning and education, Educause rev. 46 (2011). Proc. of ICLP’23 (Tech. Comm.), EPTCS, 2023. [4] D. Ifenthaler, J. Y. Yau, Reflections on diferent [19] C. Aina, E. Baici, G. Casalone, F. Pastore, The delearning analytics indicators for supporting study terminants of university dropout: A review of the success, Int. J. Learn. Anal. AI Educ. 2 (2020) 4–23. socio-economic literature, Socio-Economic Plan[5] C. Romero, S. Ventura, Educational data mining: ning Sciences 79 (2022) 101102.
a review of the state of the art, IEEE Trans. Syst. [20] B. Olasile, S. Emrah, Covid-19 pandemic and online Man Cybern., Part C 40 (2010) 601–618. learning: the challenges and opportunities, Interac[6] L. C. Liñán, Á. A. J. Pérez, Educational data mining tive Learning Environments 8 (2020) 54–72. and learning analytics: diferences, similarities, and [21] European Commission, Council resolution on a time evolution, RUSC. Universities and Knowledge strategic framework for European cooperation in Society Journal 12 (2015) 98–112. education and training towards the European edu[7] F. Agrusti, G. Bonavolontà, M. Mezzini, University cation area and beyond (2021-2030), 2021. dropout prediction through educational data mining techniques: A systematic review, J. of e-learning and knowledge society 15 (2019) 161–182. [8] A. Sarra, L. Fontanella, S. Di Zio, Identifying stu