The ability for learners to self-regulate their learning, is considered as a key factor to achieve academic success. With the increasing popularity of digital learning environments, it has become critical to develop efective ways of supporting self-regulated learning in these contexts to ensure that learners are able to take advantage of the benefits of these platforms, and therefore, measuring self-regulated learning. This paper aims to describe a new approach for analyzing self-regulated learning strategies, while assessing learning skills being acquired by the learner. We propose a two-layer approach that combines an analysis of learners skill levels with the analysis of self-regulation strategies through data traces. This analysis of skills mastery and behaviours leads to qualify the relevance of self-regulated learning strategies. These assessments could serve as a basis to recommend behavioral strategies. This article mainly focus on the presentation of this two-layer system and its first implementation on the Quick-Pi platform dedicated to the learning of the python programming language.
Self-regulated learning (SRL) is the ability for learners to control their own learning, and is considered to be a key factor in achieving academic success. Such ability involves setting goals, monitoring progress, and adapt to changing situations [1]. With the emergence of technologies, online learning is now a popular form of education. In Online Learning Environments (OLEs), the teacher or instructor’s presence is often low. As such, learners require efective SRL skills to be eficient [ 2]. In spite of the significance of SRL, learners encounter obstacles that hinder their ability to regulate efectively, impeding their overall learning progress. Such setbacks can be portrayed as a lack of good strategy use, a lack of metacognitive knowledge or a lack of experience in learning environments. The application of metacognitive strategies necessitates the possession of specific metacognitive skills, which are not universally mastered by all learners. For this reason, it is of major importance to support SRL of each learner in OLEs through tailored guidance according to their skill level [3].
Proceedings of the Doctoral Consortium of the 18th European ConferPortugal question: How to support SRL based on skill level observations ? In this paper, we present and position a system that includes a 2-layer measurement service in charge of collecting and analyzing OLE data, and a recommendadual-layer measurement service consists of two modules: the performance layer, which is primarily responsible for skill assessment, and the behavioral layer, which is responsible for tracing and analyzing SRL strategies. The main objective of the skill tracing module is to identify successful phases where learners show progression of amount of data through their interactions with the plat- tion service aimed at providing support to learners. The their skill mastery level at specific time intervals. We and self-reports [9]. conducted an experiment on the skill tracing module us- Although questionnaires remain a reliable way to meaing the Quick-Pi programming platform to track learners’ sure SRL, scholars in learning analytics pointed out popython programming skills. We hypothesize that during tential limitations [10]. One limitation is their inability specific time intervals when progression of skill mastery to capture the dynamic changes in learners’ adaptation is observed, efective SRL strategies are employed. To and modification of learning tactics and strategies durachieve this, an SRL strategy recognizer module is devel- ing the learning process[11]. As a result, there has been oped to transform raw trace data into identifiable SRL a shift towards exploring more tailored approaches for strategies. Subsequently, these strategies are analyzed us- assessing SRL in OLEs, which will be further discussed ing Process Mining (PM) methods to uncover successful in the upcoming section. transitions between diferent strategy uses and identify behaviors that predict success. These identified behaviors 2.2. SRL in OLEs are then stored and utilized for future recommendations.
The structure of this paper is as follows: first, section The increasing popularity and efectiveness of exercise2.1 provides a background on SRL theory. Then, section based platforms have led to the widespread adoption of 2.2 delves into the manifestation of SRL in OLEs. Finally, large-scale OLEs [12]. The efectiveness of these OLEs section 2.3 introduces the measurement methods utilized hinges significantly on the learner’s capacity to assume to capture SRL. responsibility for their own learning [13]. While previous research on SRL has primarily concentrated on traditional physical settings such as classrooms, there 2. Background and related work is a growing body of scholarship investigating SRL in online contexts (eg. [14]). 2.1. SRL Theory OLEs ofer the benefit of collecting and storing learner SRL encompasses various dimensions, namely cognitive, data for analysis and measurement objectives. According metacognitive, behavioral, motivational, and emotion- to Winne [4], trace data provides observable indicators al/afective aspects, representing a well-established con- that support valid inferences about metacognitive moncept [1]. Zimmerman is credited as one of the pioneering itoring and metacognitive control, which are essential researchers who initially formulated the theory of SRL and fundamental aspects to SRL. In OLEs, trace data are [1]. His work emphasizes the fundamental perspective favored due to their ability to provide precise observathat self-regulation empowers students to be autonomous tions of learners’ interactions with an online platform and assume responsibility for their own learning. This [15]. autonomy is realized through the regulation of the afore- Collecting and analyzing such data enables the provision mentioned aspects of SRL. Ultimately, the primary objec- of feedback on learners’ SRL and promotes their awaretive of adopting SRL is to facilitate the achievement of ness of the learning process. Trace data refers to the data personal goals. produced through learners’ interactions with the online Following Zimmerman’s groundbreaking contributions platform. Log files are regarded as the most feasible data to SRL theory, there has been a notable surge in publi- source due to the level of information they provide and cations within the field and the introduction of various the coding efort and time required for analysis [ 16]. SRL models [5]. Log files encompass a diverse range of information, inSRL models ofer a comprehensive framework that de- cluding details about learning activity sessions, login lineates the processes and sub-processes employed by and logout events, resource views and downloads, uplearners. While some models ofer a broad perspective loaded assignments, attempted quiz items, and forum on SRL [6], others concentrate on specific SRL aspects, posts addressed to both general and specific peers [ 4]. such as emotion/afect [ 7], or metacognition [8]. Approaches that have been proposed to assess SRL in In order to engage in self-regulated learning, learners OLEs, includes mainly the usage of LA and EDM. LA is need to employ strategies that enhance their learning ”the measurement, collection, analysis, and reporting of experience. data and their contexts for the purposes of understanding Self-regulation strategies become evident through the be- and optimising learning and the environments in which haviors exhibited by learners. Consequently, these SRL it occurs”[17], while EDM places its emphasis on explorstrategies can be observed by examining how students ing and analyzing educational data to acquire a deeper employ them in their learning. understanding of students’ learning. In the context of Various assessment instruments have been developed SRL, LA is applied within online settings where users inand proposed for evaluating SRL. Traditional methods of teract with an online platform. Data is captured through SRL assessment involved the utilization of questionnaires various inputs, such as devices (e.g. mouse focus and keyboard typing) and platform events (e.g. opening doc
uments, highlighting text), and is aggregated for analysis purposes, including usage flow analysis, knowledge tracing, and social network analysis. By processing this data, behavioral analysis can be conducted. LA can provide students with information regarding their behaviors and the use of SRL strategies, serving as cues for monitoring and controlling their learning processes.
Viberg et al. [18] demonstrated that the majority (70%) of empirical research in LA focuses on higher education and primarily centers around measuring SRL in OLEs. 2.3. SRL measurement To address our research question, an architecture for a behavioral recommendation system designed to assist learners during their performance is proposed. This section introduces the Skill-based SRL Recommendation System (S-SRL-RS) architecture designed to support learners during their performance in OLEs. The system utilizes learning data collected from the platform, including exercise outcomes and learner interactions. This data is then used to assess the learner’s skill mastery levels and identify successful competence phases. Additionally, trace analysis is conducted to detect SRL strategies employed by the learner. The appropriateness of these strategies is examined considering the learners’ skill levels and individual learning context. The identified strategies are stored for future reference to provide recommendations to learners. The subsequent sections will provide detailed descriptions of each component of the system, and it’s implementation within the Quick-Pi OLE.
In order to support SRL, it is crucial to gain an understanding of learners’ self-regulatory abilities. To observe these abilities, the measurement of SRL becomes essential. This involves constructing indicators that gather data and ofer learners, instructors, and the system valuable information about learners’ interactions and their utilization of strategies during their performance phase.
Empirical research, including the study conducted by Pekrun et al. [19], provides support for the utilization of frequency measures. In their study, they investigated the interplay between achievement goals, achievement emotions, and self-regulation strategies. The findings 3.1. Quick-Pi Learning Environment suggest that the relationship between achievement goals and achievement emotions is partly influenced by the Quick-Pi is an online platform designed to provide highfrequency of self-regulation strategy usage. school students with educational content and interactive An illustration of frequency measures can be observed activities focused on programming connected objects. in the tool NoteMyProgress developed by Pérez-Alvarez These activities are presented as exercises that allow et al. [20]. This tool analyzes data and incorporates a students to work with IoT devices while learning the dashboard equipped with visualizations that enable stu- fundamentals of Python programming. The platform ofdents to monitor their activity and develop an under- fers courses in three diferent programming languages: standing of their self-regulated learning strategies within Blockly, Scratch, and Python. For our experiment, we a selected MOOC course [20]. specifically selected the Python course, which consists Vazquez and Nistal [21] introduced a monitoring system of eight activities. In total, we conducted our experiment that encompassed various learning strategies such as using sixteen exercises from the initial course. To estabtime management, goal setting, and monitoring. They lish a connection between behaviors and existing skills, identified and integrated specific indicators (e.g., resource it is crucial to develop a skill taxonomy of reference that usage time, project engagement time, strategy frequency) outlines the available skills on the platform and their for each strategy, along with potential approaches for dependencies. The construction of this taxonomy will be analyzing and interpreting the obtained results. Subse- elaborated upon in the following paragraph. quently, Manso-Vazquez et al. [22] further explored the significance of relevant data for monitoring SRL. Skill Taxonomy of reference To create our skill taxWhile numerous tools and technologies have been sug- onomy of reference, we manually extracted concepts in gested for measuring and assisting students’ SRL, there the Python programming language by completing each remains a gap in our understanding of how these tools exercise on the platform. These concepts include variactively promote the enhancement of students’ SRL for ables, functions, and others. Then we extract from each domain skill improvement. Hence, addressing our afore- concept the specific doable operations. As shown in figmentioned research question: How to support SRL based ure 2 for instance, a function can be defined or called. on skill level observations ? Then, a description of the expected achievements for each concept and its operations is formulated in table 1, outlining what learners are expected to accomplish through their learning (eg. be able to define a variable).
Twelve skills related to python programming are potentially mobilized in the exercises proposed on this platform. Finally, a skill dependency graph is created to Item Response Theory (IRT), Knowledge Space Theory establish relationships between each skill, where the no- (KST), and Bayesian Knowledge Tracing (BKT) [23]. In tation → indicates that skill is a prerequisite for this thesis, the de-facto standard for student modeling skill . Figure 3 depicts the prerequisite relationship method BKT was opted for. By utilizing a BKT model, among diferent skills. learner performance at a granular skill level can be observed, revealing instances where a progression of their skill mastery level is demonstrated. This allows us to in3.2. Skill Assessment vestigate the SRL strategies employed by learners during Skill modeling serves as the foundation of our system, their learning process. enabling the exploration of behavioral patterns that con- Outside the scope of this work, we propose a BKT model tribute to the advancement of learners’ skill mastery that integrates a structure of skill dependencies, and exlevel. ternal factors such as exercise dificulty [ 24]. We estabIn the field of skill modeling and assessment, the primary lished a specific skill taxonomy of reference, show in proposals can be categorized into three main groups: figure 1 and 3 for the exercises ofered on the Quick-Pi platform, focusing on the python programming domain, sessment can be categorized as either passing ( () = 1 ) and implemented the BKT model accordingly using our or failing ( () = 0 ) the exercise. In our case, we have taxonomy. defined four levels of assessment, which are described in
For the purpose of skill assessment, performance data the assessment result legend shown in figure 5. is collected from the platform. The performance data In a traditional BKT model, it is assumed that a skill is relates to skill assessment and indicates the success utilized at each timestep (exercise). However, in our case, or failure of learners in exercises. Subsequently, the this assumption is not valid as the exercises performed incollected data is inputted into the BKT model to estimate volve distinct subsets of the defined skills. Consequently, the learner’s current level of skill mastery. The model we add a trigger variable to the original BKT model, called updates the probability of skill mastery at each timestep, () to condition the evolution of the skill on whether it where a timestep corresponds to the moment when a was actually mobilized during the exercise ( () = 0 for learner submits their exercise. As a result, the model’s skill ”unused”, and () = 1 for skill ”mobilized”) [24]. output provides the probability that a skill is mastered To incorporate the prerequisites outlined in our skill taxat a specific level of mastery. To demonstrate this, an onomy of reference, we aggregate the prerequisite skills initial experiment was conducted on a learning platform into the variable () as shown in figure 4 on the right to track the skills of learners and identify specific time model, which represents the level of skill attainment for intervals where they successfully achieve mastery in a a given set of prerequisite skills. Therefore, the skill level particular set of skills. () depends not only on the previous skill level ( − 1) , but also on the mastery of the prerequisite skills at the previous timestep, ( − 1) . We incorporate this result Model construction The classical BKT model (left by aggregating () and () into () . model of figure 4) is a random process that uses Boolean Finally, to incorporate external factors () that influvariables to represent the skill level of a learner over time, ences on the skill acquisition, such as exercice dificulties, denoted as () in figure 4. The model simplifies the skill we add a latent variable () which describes the learnlevel to either ”not acquired” ( () = 0) or ”acquired” ing speed of () , and integrate external factors () and ( () = 1 ), while there are more complex scales that can the trigger () in () [24]. be considered [24]. In our case, we define a scale of 4 level of mastery, described in the mastery level scale leg- BKT Application Figure 5 presents the initial findings end of figure 5. of estimating the mastery level of the 12 skills over time At each timestep, the variable () represents the assess- for a specific learner as they progress through their acment of an exercise. In a basic BKT model, a learner’s as- tivity. Throughout the experiment, the learner engages in exercises that primarily involve skills 01 and 02 . 3.3. Behavioral Module During the initial iterations, skill 03 is mobilized and laastseerssiteedr,awtiohnilse. sIkniltlesr0m4s otfo a0s6sescsommeneti,netxocpelpatyfoinr tthhee Touhre sbyeshtaevmioarnaldmhoadsutlheecroonlsetiotuftiedsenthtiefysiencgonadndlayexeramofifrst iteration, direct evaluation of skill 01 is not con- ining SRL strategies initiated by learners during their ducted. However, its continuous utilization over time performance phase. Initially, we will introduce the stratcontributes to an increase in mastery level. Skill 02 egy recognition module, which plays a central role in initially receives a negative evaluation but later shows identifying SRL strategies from raw trace data. Subsesignificant improvement, accompanied by ongoing uti- quently, we will delve into the strategy analysis module, lization, resulting in a faster growth in mastery level. where the detected strategies are thoroughly examined The performance on skill 03 demonstrates variability and analyzed using process mining. initially but gradually improves over time. Other skills that are not directly assessed, such as 05 or 07 , also Strategy Recognition Concurrently with skill assessshow an upward trend in mastery level with practice but ment, behavioral data is collected and analyzed to unmay experience a decline when they are subsequently cover SRL strategies employed by learners during their assessed negatively. learning phase. This process is carried out using the Throughout the activity, various time intervals arise strategy recognition module. This module is designed to where successful skill mastery becomes apparent. No- process raw trace data that captures the various behavtably, skills 01, 02, 03, 04 , and 06 exhibit intriguing ioral actions initiated by learners. Its primary function time intervals characterized by a significant increase in is to encode and interpret this data to identify the SRL their mastery level. These findings lead us to hypothe- strategies employed by learners. size the presence of efective SRL strategies. To explore A trace-based SRL protocol is a methodological approach this further, delving into the analysis of SRL strategies that involves utilizing raw trace data obtained from a digis conducted within the behavioral layer of our system, ital environment, which comprises patterns or sequences which comprises two modules: the strategy recognition of events for measurement purposes [25]. These data are module and the strategy analysis module. The details of then translated into indicators that provide insights into these modules are discussed in section 3.3. learners’ utilization of SRL tactics and strategies [26]. The protocol proposed by [26] establishes the relationships between trace data, learning sessions, learning tacLearning Actions
READ_TASK READ_HELP
NAVIGATION PROGRAMMING
SUBMISSION_FAIL SUBMISSION_SUCCESS
CODE_DEBUG CODE_TEST
Description Learner reads the task set by the exercise Learner seeks help through documentation reading Learner navigates through platform modules Learner is programming, therefore attempting to solve the exercise Learner submits exercise but solution is invalid Learner submits exercise and passes Learner debug their program
Learner experiment their program before submission
constitutes the third component of our system. As mentioned earlier, timesteps correspond to the moments We introduced a system that consists of a two-layer when learners submit their exercises, and these moments measurement service responsible for gathering and trigger the update of probabilities in the BKT model. analyzing data from online learning OLEs. The system At the end of each timestep, we examine specific time includes a skill assessment module that enables instrucintervals where learners demonstrated improvement by tors to understand the underlying factors contributing observing the progression of their skill mastery levels. to learners’ challenges in acquiring specific target skills.
the French National Research Agency (ANR-20-CE380010). The author would like to thank the France-IOI association for providing access to the Quick-Pi.org platform.
Additionally, the author acknowledges Dr. Fahima Djelil, Dr. Jean-Marie Gilliot, and Pr. Gregory Smits for their supervision and input concerning the paper’s language and structure, as well as their support throughout the work’s development. Lastly, the author would like to thank the editors and reviewers for their valuable suggestions and comments aimed at enhancing this work.
This understanding is facilitated by the behavioral module, which provides valuable insights into learners’ behaviors and strategies during their learning process.
This service is complemented by a recommendation service that aims to provide support and guidance to learners.
To provide recommendations, we will suggest behaviors that have been successful for other learners and are beneficial for the specific skill or exercise being addressed. The recommendation system will primarily utilize collaborative filtering, which leverages the experiences of other users with similar profiles who have achieved a high level of skill mastery in a specific skill. Its objective is to support and assist learners facing challenges by suggesting behaviors similar to those of successful learners. Collaborative filtering generates recommendations by considering the relationships between users and items [29]. In our use-case, it can recommend that a learner adopt certain behaviors for a target skill based on the behaviors of similar learners who have successfully mastered that skill. The main question we ask here is: ”What methods can be employed to assess the similarities among learners in order to ofer them suitable behavioral recommendations ?” [15] N. Perry, P. Winne, Learning from learning kits: Learning analytics to reveal links between learngstudy traces of students’ self-regulated engage- ing design and self-regulated learning, in: Interments witgh computerized content, Educational national Journal of Artificial intelligence in EducaPsychology Review 18 (2006) 211–228. doi:1 0 . 1 0 0 7 / tion, volume 31, 2021, pp. 980–1021. doi:1 0 . 1 0 0 7 / s 1 0 6 4 8 - 0 0 6 - 9 0 1 4 - 3 . s 4 0 5 9 3 - 0 2 1 - 0 0 2 4 9 - z . [16] J. You, Identifying significant indicators using lms [27] Y. Fan, J. Saint, S. Singh, J. Jovanovic, D. Gasedata to predict course achievement in online learn- vic, A learning analytic approach to unveiling selfing, Internet and Higher Education 29 (2016) 23–30. regulatory processes in learning tactics, in: Prodoi:1 0 . 1 0 1 6 / j . i h e d u c . 2 0 1 5 . 1 1 . 0 0 3 . ceedings of the 11th International Learning Analyt[17] G. Siemens, Learning analytics: The emergence ics and Knowledge Conference, 2021, pp. 184–195. of a discipline, American Behavioral Scientist 57 doi:1 0 . 1 1 4 5 / 3 4 4 8 1 3 9 . 3 4 4 8 2 1 1 .
(2013) 1380–1400. doi:1 0 . 1 1 7 7 / 0 0 0 2 7 6 4 2 1 3 4 9 8 8 5 1 . [28] E. Villalobos, M. Pérez-Sanagustin, C. Sanza, A. Tri[18] O. Viberg, M. Khalile, M. Baars, Self-Regulated cot, J. Broisin, Supporting self-regulated learning Learning and Learning Analytics in Online Learn- in bl: Exploring learners’ tactics and strategies, in: ing environments: A Review of Empirical Research, I. Hilliger, P. J. Muñoz-Merino, T. De Laet, A. OrtegaAssociation for Computing Machinery, New York, Arranz, T. Farrell (Eds.), Educating for a New NY, USA, 2020, pp. 524–533. doi:1 0 . 1 1 4 5 / 3 3 7 5 4 6 2 . Future;: Making Sense of Technology-Enhanced 3 3 7 5 4 8 3 . Learning Adoption, volume 13450, Springer Inter[19] R. Pekrun, A. Elliot, M. Maier, Achievement goals national Publishing, 2022, pp. 407–420. doi:1 0 . 1 0 0 7 / and achievement emotions: Testing a model of their 9 7 8 - 3 - 0 3 1 - 1 6 2 9 0 - 9 _ 3 0 . joint relations with academic performance, Jour- [29] H. Yago, J. Clemente, D. Rodriguez, Competencenal of Educational Psychology 101 (2009) 115–135. based recommender systems: a systematic literdoi:1 0 . 1 0 3 7 / a 0 0 1 3 3 8 3 . ature review, Behaviour Information Technol[20] R. Pérez-Alvarez, J. Maldonado-Mahauad, M. Pérez- ogy 37 (2018) 958–977. doi:1 0 . 1 0 8 0 / 0 1 4 4 9 2 9 X . 2 0 1 8 .
Sanagustin, Design of a tool to support self- 1 4 9 6 2 7 6 . regulated learning strategies in moocs, Journal of Universal Computer Science 24 (2018) 1090–1109.
doi:1 0 . 3 2 1 7 / j u c s - 0 2 4 - 0 8 - 1 0 9 0 . [21] M. Vazquez, M. Nistal, Design of the monitoring system of a learning organizer, in: 2014 IEEE Global Engineering Education Conference (EDUCON), IEEE, 2014, pp. 235–242. doi:1 0 . 1 1 0 9 /
E D U C O N . 2 0 1 4 . 6 8 2 6 0 9 7 . [22] M. Manso-Vazquez, M. Caeiro-Rodriguez,
M. Llamas-Nistal, Tracking and visualizing time management for self-regulated learners, 2016, pp.
1–5. doi:1 0 . 1 1 0 9 / F I E . 2 0 1 6 . 7 7 5 7 4 1 1 . [23] M. Desmarais, R. Baker, A review of recent advances in learner and skill modeling in intelligent learning environments, User Modeling and UserAdapted Interaction 22 (2012) 9–38. doi:1 0 . 1 0 0 7 / s 1 1 2 5 7 - 0 1 1 - 9 1 0 6 - 8 . [24] Q. Couland, P. Leray, A. Boulahmel, Un modèle générique avec structuration des compétences et facteurs externes pour le bayesian knowledge tracing, in: Proceedings of the 11èmes Journée Francophones sur les Réseaux Bayésiens et les Modèles
Graphiques Probabilistes, 2023. [25] Y. Fan, J. van der Graaf, L. Lim, M. Rakovic,
S. Singh, J. Kilgour, J. Moore, I. Molenaar, M. Bannert, D. Gasevic, Towards investigating the validity of measurement of self-regulated learning based on trace data, Metacognition and Learning 17 (2022) 949–987. doi:1 0 . 1 0 0 7 / s 1 1 4 0 9 - 0 2 2 - 0 9 2 9 1 - 1 . [26] Y. Fan, W. Matcha, N. Uzir, Q. Wang, D. Gasevic,