Evaluating learning experiences and identifying areas for improvement are crucial steps in ensuring effective digital training programs. To this end, our research has developed a comprehensive model for analyzing tracking data from the perspectives of digital learning operators, synthesizing all aspects of training into a unified tool. This approach enables us to create a Macro Index of Performance (MIP) and seven sub-indexes: Results, Study Pace, Course Structure, Computer Adequacy, Community Participation, Educational Tutoring, and Process Tutoring. Using feedback from our first experiments, we have oriented our subsequent research toward analyzing critical situations in tracking data. Our novel framework, LearnalizeR, combines the MIP and sub-indexes to effectively evaluate online courses, identify areas of riskiness, and provide tutors with potential preventive intervention strategies. This study investigates the effectiveness of LearnalizeR through two real-world cases from an Italian public administration, highlighting compelling examples of criticalities and proposed interventions. We aim to demonstrate how LearnalizeR can help address participation and performance criticalities in online courses, ultimately contributing to more effective digital training programs. This paper is structured as follows: after a brief overview of the traditional use of KPIs in evaluating training effectiveness and criticalities, we present the LearnalizeR framework and its components, including MIP and sub-indexes. We then describe the application of LearnalizeR in two real-world cases, highlighting compelling examples of criticalities and proposed interventions. Finally, we summarize our findings and discuss implications for future research.
The big data available on the corporate LMS platforms, from training tracking data to enrollment processes and evaluation systems, represents excellent information potential for continuously improving distance learning projects. The training data, however, are often treated from the point of view of training with a significant focus on hours provided, people involved, test thresholds reached, performance levels, and dropout reduction. This focus on training, i.e., on the training offer, represents the point of view of the training institute or company, which indeed constitutes a component of the problem of training effectiveness and knowledge transfer. Correctly, training institutes and companies must be able to evaluate and monitor the training system offered to the participants. However, extensive debates [1] have arisen on how to effectively evaluate training and its effectiveness, from the issues of the ROI of training to the most recent concept of training impact and the modern KPIs. The latter is helpful for those who organize courses to design new content or methodologies or to redefine support models for training activities. In that case, enhancing these improvements based on the users' actual behavior is equally important. It is essential to adopt a learning perspective, that is, to use data analysis to re-read the experiences and behaviors carried out to redesign the training, starting from the actual impacts and experiences of the participants.
Our perspective on data-based decision-making changes significantly when we consider that choices are derived from natural phenomena rather than just good intuitions or professional expertise. This shift in perspective means that our options for redesign and tutoring intervention become user-oriented and needs-oriented. As such, these interventions aim to validate the effectiveness of designed content by considering real-world experiences. Ultimately, this approach enriches didactic and management options by harmonizing them with typical study behaviors.
To respond to the need of a course evaluation and free learning process from standardized formalisms, practices, and customers, one must equip oneself with data analysis tools and intervention strategies. As part of the LearnalyzeR experience [2], we developed a criticality detecting system that aims to support the intervention of tutors day by day through predictive systems, analysis of participation behaviors, and redesign of courses. As a first step, we defined criticality as the behavior of a non-neglecting part of participants that leads to a course experience that does not satisfy their fruition, participation, times, and performance needs. We oriented the survey tools toward the recognition of:
• Criticalities from already known patterns using only course data. They are calculated during the course, and the course is already critical when observed. • Predictive patterns of criticality with respect to historical behavior. They are calculated during the course, and the course is not critical when observed. • Distorted use of times and verification of units at risk. It can be calculated only at the end of each unit, and then, it can be summarized only at the end of the course.
We will describe our model in details in section 3.
Learning to understand the potential criticalities allows one to adopt a predictive perspective on the problems, reducing the negative impacts of ineffectiveness of the courses, better-managing investments in training and at the same time increase the usefulness and perceived effectiveness and the appreciation of training initiatives. The ROI perspective is managed in progress in terms of containing the risks of ineffectiveness.
Adopting a data analysis perspective also allows to overcome the conformism of the support activities sometimes affected by some typical distortions in the reading of the LMS tracking, as explained in Table 1.
Traditionally, KPIs are used to measure the effectiveness of training. The focus was on the added value that training brought to the company. In the literature, the focus is on training that produces enabling results for business performance. The training system is oriented towards producing specific results. This highlights a training perspective approach, i.e. the enhancement of the training offer and its ability to produce results and economic value. The theme of the evaluation was widely approached already at the end of the 50s with the classic but still significant model of Kirkpatrick [[3]; [4]; [5]; and [6]]. It focused on the theme of the evaluation of the effectiveness of training as well as on the binomial individual-environment, in favor of a logic of sequential 4-step process in which the results of training are progressively brought back by the individual to his context, and therefore require adequate evaluation tools. This dynamic view of the assessment concerns the impacts in terms of:
• Reactions: the degree to which participants find the training favorable, engaging, and relevant to their jobs (typically the evaluation of satisfaction of the training experience); • Learning: the degree to which participants acquire the intended knowledge, skills, attitude, confidence, and commitment based on their participation in the training (alias the evaluation of the added value of training, on the cognitive-affective-attitudinal level); • Behavior: the degree to which participants apply what they learned during training when they are back on the job (typically the manifestation of competent behaviors); • Results: the degree to which targeted outcomes occur as a result of the training and the support and accountability package.
There was an intensification of research since the 80s in particular on the issue of the effectiveness of training and its results. For example, the Baldwin and Ford model [7] highlights how learning to be realized needs the interaction between the person, the appropriate structuring of training and an organizational context that provides opportunities to implement learning. Subsequently, Noe's model [8] highlighted how the transfer of individual learning (linked to the acceptance of feedback, individual expectations and an internal locus of control) in the work passes through the support of the organization, also through social support (support for motivation) to implement adequate performance. Thus, andragogic theories highlight how the transfer of learning is strongly linked to the perception of the usefulness of learning to meet career expectations and satisfaction, within a work context oriented to the management of feedback, to a climate of support between peers and superiors, and the opportunities for application of learning [9]. In all these studies the theme of measurement and objectification of the results has always been the reason for greater criticism of the evaluation models. The main criticisms of Alliger and Janak's (1989) concern the sequentiality and the different meanings and values of the various levels of evaluation, with levels 3 and 4 being more important but often not measured for the difficulties of detection [ [10]; and [11]]. Further criticism is the inextricable connection between the levels, with the consequent difficulty of isolating and measuring the impacts separately. Subsequently, in parallel with the growth of diversified training experiences (also of an experiential type), a reasoning on the issues of measuring actual results was activated in the literature.
An alternative approach to Kirkpatrick [3]; [4]; [5]; and [6] to measure improvements made by the training function is the method of Robert S. Kaplan and David P. Norton [12] typical of evaluating the performance of organizational units using Balance Scorecards. This model considers four different perspectives:
• economic and financial (economic results that effectively describe the relationship between investment in training and the results generated); • customer-market (results in terms of satisfaction of internal customers alias participants); • internal processes (effectiveness of key processes for achieving training objectives); • learning-growth (generated learning outcomes).
The BSC method (also in the specific case of training) is carried out by identifying KPIs (Key Performance Indicators) in the four areas of investigation. This way, a 360°overall evaluation of the analyzed activity is obtained. From the considerations that emerged over time and with the emergence of the knowledge-based economy and a complex and flexible labor market, continuing education has acquired strategic importance for both individuals and organizations, with the need to understand and enhance the value of human capital. Although conceptually, Kirkpatrick's model maintains its value as a complex conceptual framework of intervention in the training evaluation field, it does require corrective measures in the context of meanings and tools.
My son Jim asked me, How much has your model changed since it was introduced in 1959? I replied that the model remains essentially the same. The concepts, principles, and techniques are as applicable today as when I introduced the model. [10]; and [11] If the distinction between learning and behavior is the main merit of Kirkpatrick's model [3]; [4]; [5]; [6], some criticisms arose over time. I.e., that evaluative activities focus on the first level, partly on the second. At the same time, they appear very complex to analyze, and with few and sporadic attempts, the evaluations on the third and fourth (3) levels, moreover, with partly superficial and subjective approaches [13]. From the investigation of Kennedy et al. [14], the problem does not seem to be the lack of interest. Even the training professionals (practitioners) show a broader interest in the higher levels with a lower perception of the usefulness of the lower levels.
The most recent theories look at experimental analysis to identify the measurement of training outcomes and the objective-results comparison through direct evidence between objectives-results [15]; [16]; [17]. Therefore, the models focus on summative evaluation (i.e., tangible, measurable results at the end of the evaluation).
Other interesting models are:
• IPO Systematic approach (input, process, output) of Bushnell [18] contends that Kirkpatrick's model focuses only on what happens after the training, but not the entire training process. • Brinkerhoff's six-stage evaluation model [19]; [20] [21] argued that evaluation systems should be directed toward performance improvement regarding three evaluation domains performance (P), learning (L), and satisfaction (S). • Holton's three-level HRD evaluation and research model. Holton's HRD [22]; [23] All cited research highlights the need to focus on the theme of training transfer as generally defined in the literature. The transfer to the work context depends on the level of involvement and motivation of the participant, the quality of training (understood as attention to learning in concrete reference to the activities to be carried out), and the organizational context (understood as openness to practical experimentation of learning). The latest contributions concerning the training evaluation model highlight the importance of a multi-actor approach in which the different interests and potentially conflicting needs of the stakeholders involved, internal and external to the organization, are represented [16].
A limitation commonly indicated by the literature is the need to equip oneself with adequate tools to identify, study and highlight the results brought by the training. So the topic of data is relevant in any training evaluation process. Our work, therefore, is part of the theoretical framework between level 1 and level 2 of the Kirkpatrick model. We will show a model to reduce the potential effects of ineffectiveness due to the training set, the material, or the object of the training, to facilitate a fluid and practical learning experience to maximize training opportunities and then impact organizational behaviors and final results.
As extensively described in our previous work [2], we developed a model of analysis of the tracking data starting from the experiences of digital learning operators to create a tool capable of synthesizing all aspects of training. We developed a Macro Index of Performance (MIP) and seven sub-indexes: Results (I R ), Study Pace (I SP ), Course Structure (I CS ), Computer Adequacy (I CA ), Community Participation (I CP ), Educational Tutoring (I ET ), and Process Tutoring (I PT ). Each sub-index consists of variables that describe the various aspects of the sub-indexes.
The MIP (and sub-indexes) makes comparable (online, classroom, blended) courses through a unified analysis model. It can discriminate different experiences and indicate the expected outcomes based on similar data, guiding the tutors in the differentiated intervention methods to support and facilitate learning.
Using the feedback received after the first experiments, we oriented the subsequent research to study critical situations in the tracking. Using the values of the MIP (and sub-indexes) and the raw values of the tracking, we could highlight areas of riskiness and provide tutors with potential preventive intervention strategies. The MIP and sub-indexes describe the students' behaviors and indicate any critical issues that characterize the courses. Through the automated analysis of the courses, tutors can
It is essential to define what critical course means concerning a group of participants and for the individual user. Generally, the criticality of a course is considered based on the use of the time allocated and the need to conclude all the teaching activities to reach the minimum requirements. However, a course is not always critical, or at least not for everyone. Generally, there is an increasing participation distribution that peaks on 3/4 of the time available (Figure 1).
We should use different rules when the course (time) progresses. Using this framework, we built a criticality detection system inside LearnalyzeR (our learning analytics). Concerning the various critical issues based on the problems highlighted, we carried out particular interventions that the tutors and the system can implement to prevent the risk of ineffectiveness (Table 2, Table 3, Table 4).
The course critical issues detection algorithms compare the raw data of use and the MIP of all the course users with what the course designers and commitment expect. It is necessary to distinguish two types of expected behaviors: the one during the delivery and the one at the end of the course. The criticality algorithms C PA (Table 2) and C S (Table 3) intercept the expected behaviors during the course, even at the beginning. In contrast, the algorithms for C PE (Table 4) are based on the prediction of performance at the end of the course (final MIP) and depend on the value of the MIP at various steps of the course (available time and completion). For this last criticality, we used a Bayesian beta regression model [2], using the minimum of the MIP estimation interval as a precaution. Then we choosed the threshold value in Table 4 by analyzing the MIP values in the previously closed courses. In the following examples, the chosen threshold value for the minimum MIP forecasted is 70 1 . The model, i.e., its parameters2 , are adaptable, through the retrospective analysis of the closed courses, to the customer's requests, the course structure, and the performance requests.
Our tracking critical issues models allow for timely tutoring actions, which, thanks to the predictive model of the final performance, we can define preventive tutoring.
Tracking critical issues during the course also allows the ex-post evaluation of tutoring actions' design, delivery, and validity.
One defined criticality criteria, we must combine specific intervention criteria based on observed data. Table 5, Table 6, and Table 7 show specific intervention actions for critical issues, as defined in Table 2, Table 3, and Table 4, respectively. Table 5: Interventions proposed for participation's criticalities (C PA ) as defined in Table 2.
Verify that everyone received login information; a2) verify with the client and/or the managers that there are no work contingencies (deadlines, unforeseen events, workloads, etc.) that affect the people's availability; a3) check the sub-index of IT adequacy to verify if there are technological barriers that influence the correct performance of the study activities; a4) check the sub-indices of results and pace of study and verify on the platform the actual times recorded; a5) assess people's level of engagement and motivation for the course.
C PA type B b1) Send reminders to those who have not accessed the platform, remotivating the importance of participating in the course; b2) for those who have started, motivate participation to recover any study gaps compared to the expected; b3) remind everyone of times and deadlines; b4) check the sub-index results (Pace of study and IT adequacy), and verify the raw participation times recorded, signaling users to respect the study times provided to obtain certificates and certifications.
C PA type C c1) Recover participants who have been left behind, for whom it is necessary to ask for an effort of participation; c2) identify and quantify the cases of proper abandonment of the course; c3) check the sub-index results (Pace of study and IT adequacy), and verify the raw participation times recorded.
For people who have lagged in the fruition, i.e. who have not connected or who have short times compared to the expected, it is essential to provide precise indications to complete the course, indicating average daily study time, minimum necessary activities, and operational advice. For people with study times that do not comply with expected, proceed to report that it is essential to respect the study times indicated to obtain certificates and certifications.
C PA type D d1) Communicate to participants that they are at risk of non-completion and non-compliance with the requirements; d2) verify the problems encountered; d3) highlight a possible study plan to recover and complete the course.
For people who have lagged in the fruition, i.e. who have not connected or who have short times compared to the expected, it is essential to provide precise indications to complete the course, indicating average daily study time, minimum necessary activities, and operational advice.
Table 6: Interventions proposed for structure's criticalities (C S ) as defined in Table 3.
C S type Z z1) Verify organizational times and conditions for the schedulation's adequacy with the structure of the course and the expected complexity; z2) verify the incidence of technical factors.
Table 7: Interventions proposed for structure's criticalities (C PE ) defined in Table 4.
Acting on people by remembering tips for effective and productive study. Refer to the tutor for personalized advice.
C PE type B b1) Identify lagging users, and check their actual completion status and the sub-indexes.
C PE type C c1) Identify lagging users, and check their actual completion status and the sub-indexes.
The course is over. Among the course completed participants, more than 10% have an index of less than 70.
In this section, we report examples of some courses in which, through the features of LearnalyzeR, it is possible to view and investigate the reasons that lead to the detection of critical issues with the criteria previously described. Using a medical paraphrase, we want to remember how critical algorithms identify symptoms. At the same time, through LernalyzeR's visualizations of MIP, sub-indices, and variables that compose them, we can understand the causes (diagnosis) and prepare targeted tutoring actions (therapy). As our customers asked, in the following examples, we will show the logic of analysis passing from one course to another without fully showing the critical issues of one or more courses. The first step in LearnalyzeR is the visualization of the median value of the Macro Index of Performance (MIP). While it can assume different states, we synthesized it using a colored dot over the median value using the following chromatic categorization (Figure 2):
• Gray: not calculable; • Red: inadequate performance with many critical elements (< 50); In the rest of the section, we will report some examples from a public administration. The intent is to show how the LearnalizeR framework (MIP, sub-indexes, and visualizations) matches the need to find the causes of course criticalities. We deduced a difference in performances between population 1 (Median MIP around 74, Figure 3) and population 2 (Median MIP around 65, Figure 4)
We observed a more significant variability of study behavior in population 2; there is a long queue on As said, the course structure has an average complexity because, despite having a high methodological complexity (value for variable methodological complexity is 100, the maximum), there is no test, and registration is not mandatory. The running course presents participation criticality of type d, and the sub-index Results have a median 50. In the case under study, this indicator directly gives the value at course completion while there is no test and no badge.
Both participation criticality and similar behavior between the running and concluded courses lead the tutor to undertake engagement actions. Users have good IT adequacy, but there is a problem with open sessions (variable open sessions is 60). The combination of the latter information with the structure criticality highlights to the tutor the non-effectiveness of using the time of use/study. The pace of study, although adequate, presents problems. Users have an adequate time of use, but effective days of study and the number of accesses are deficient.
According to forecasts, 95% of users are hard workers (final MIP predicted between 70.6 and 85.8, 95% C.I.). Still, comparing this running course with the concluded, we expect this forecast to change over time. We expect that there will be a dropout at 50% of course completion. Users in the first half of the course performed well despite problems in the distribution of study time: open sessions and low values of the actual study days variable. Macro-index forecasts are above the threshold, but all users do not go beyond 50% of the completion of both running and concluding courses.
We need to observe the participants' behavior using individual didactical units (ud) to analyze the courses that present critical structure and the problem encountered in example 3 of Section 4.1. In particular, we focused on the number of accesses in the individual ud and the time of use per ud. This analysis, combined with the knowledge of the course structure, allowed us to understand any critical issues in the individual ud:
• high or no number of accesses • high or low time of use • high population behavior variability in the number of accesses and the use time.
The systematic use of a criticality detecting system ensures the application of a regular and rigorous procedure for analyzing behaviors and results. It effectively monitors many courses and participants, even in different training courses for methods and durations. These measures also make it possible to effectively modulate data-driven tutoring systems, effectively analyzing the impacts in terms of effectiveness on the indicator. Adopting a statistical approach also means exploiting the information capital present in historical data to enhance the predictive system. Through the predictability and systematic analysis of the macro-index, sub-indices, and criticalities, tutors can act selectively in the most at-risk situations. They can modulate their intervention between the different categories of problems, individualizing communication and generalizing it or delaying it if valuable and less critical. Therefore, data-based tutoring becomes an intervention system consistent with the training system, independent of the tutors' sensitivities, guaranteeing a systematic and fair approach for all participants. Likewise, adopting a data-driven intervention system allows tutors to evaluate their actions in terms of effectiveness, striving to identify the best solution for individual critical issues. The tutoring action, therefore, starts from the same analysis scheme, guaranteeing tutors intervention methods that specialize in how to communicate with the participant. Above all, on these interventions, it is possible to evaluate the effectiveness with an analysis of the indicators and post-intervention criticalities to determine the actions/tools/messages with more excellent added value and more functional for the different categories of problems. The positive impact of the tutoring action is generally little measured and widely delegated to the tutors' expertise, who need to reread their efforts in terms of effects to evolve the intervention schemes to be an accurate guide of learning.
Our approach allows us to glimpse future steps to take. Integration of new features in the predictive and classification system will be:
• Include the tutoring actions performed following the analysis in the macro-index track, enhancing both the actual tutoring actions and those of a systematic type to evaluate their impacts on subsequent results.
• Study the evaluation thresholds of the sub-indices to decode the critical areas better.
• Integrate into the evaluation of the macro-index with extra measures for the sub-category Results with:
-Focus on the analysis of learning assessment;
-Observed results/work performance;
• Improve the satisfaction measurement in the category didactic tutoring also through NLP analysis.