Among the elements that determine a student's academic success, their ability to regulate their own learning processes is an important, yet typically underrated factor. It is possible for students to improve their self-regulated learning skills, even at university levels. However, they are often unaware of their own behavior. Moreover, instructors are usually not prepared to assess students' self-regulation. This paper presents a learning analytics solution which focuses on rating selfregulation skills, separated in several di erent categories, using activity and performance data from a LMS, as well as self-reported student data via questionnaires. It is implemented as an early warning system, o ering the possibility of detecting students whose poor SRL pro le puts them at risk of academic underperformance. As of the date of this writing, this is still a work in progress, and is being tested in the context of a rst year college engineering course.
Self-regulation skills are key features in order to achieve successful learning results. Many studies have been published showing a good correlation between self-regulation skills and academic performance, also at the higher education level [16]. Good performance is related to a proper acquisition of self-regulation skills while poor performance and drop-out is associated with bad management. Therefore, for the purposes of early-warning systems, it is very interesting to know how students are regulating themselves. This can be a very useful indicator to identify students that are struggling because a poor management of these skills.
Throughout the history of educational research, many authors have invested
e ort in understanding how students regulate their own learning behavior, and
how this a ects their performance and learning outcomes. Initial approaches
were based on the use of questionnaires, usually very large ones, used to inquire
students about their beliefs and strategies regarding the several categories
involved in self-regulation. These instruments have two main issues [
The following are some examples of authors who used learning analytics with
the goal of assessing di erent aspects of students' self-regulation:
{ Several papers by Dragan Gasevic, Jelena Jovanovic and Abelardo Pardo
focus on the analysis of LMS trace data in order to identify students' learning
strategies regarding the use of online resources. In [
As we can see, many authors attempt to make use of data that is available thanks to online tools such as LMS. However, the collection of self-reported data using surveys and questionnaires is still a widely used technique, since even if it is more susceptible to bias, it can provide information that is very di cult or impossible to obtain otherwise.
One common trend of these studies is that they usually focus on a particular aspect of self regulation. In the examples above, we have papers regarding learning strategies, time management and student motivation.
The study presented in this paper aims to cover a wide spectrum of
selfregulated learning components with a simplistic approach, the goal being
obtaining general SRL pro les of students that can be easily interpreted by a
nonexpert user. Moreover, the analysis procedure behind this objective can work as
the basis for an early warning system: pro les are generated as the course is
taking place, allowing teachers to understand the particular SRL aspects in which
students struggle and helping them improve. In the end, the generation and
presentation of these SRL pro les aims to be a learning tool for students, since as
their self-regulation capabilities improve, so will their learning outcomes [
In order to generate these pro les, we use a combination of self-reported data (questionnaires) and observational data (LMS and similar online tools), which can be directly related to some self-regulation aspects. For example, use patterns of an LMS by students can give us an insight on how they manage their time and the use they make of the available learning resources.
This paper is structured as follows: after the present introduction, Section 2 explains the foundations of the study and how the analysis procedure works. Section 3 details how this instrument is being used in a rst year university course, and the results that have been observed so far. Finally, Section 4 o ers a conclusion and possible lines of future work. 2
The following subsections provide some brief reasoning regarding our approach to the division of di erent SRL aspects into categories, as well as detailing the types of both self-reported and observational data that we have at our disposal. 2.1
The classi cation of di erent SRL components into categories is not a particularly novel concept. For example, researchers Zimmerman and Mart nez-Pons proposed a detailed category list in their 1986 study [19], complementing the definition of one of the rst widely known SRL questionnaires, the Self-Regulated Learning Interview Schedule (SRLIS). Particularly, these authors distinguished between 15 di erent categories, including items such as self-evaluation, information seeking, goal-setting, record keeping, or rehearsal and memorization.
This category de nition, however, is not a standard among educational
researchers, as authors who work with SRL categories typically de ne and use a
set that best ts their particular experiment. For example, Perels et al. [14] work
with just six categories: goal setting, motivation, learning strategies, self-e cacy,
self-re ection and problem-solving. On the other hand, Fabriz et al. [
For our purposes, we wanted to de ne a simple, reduced set of SRL categories. This is because of the mid-term goal of reporting SRL information to students and teachers via an early warning system: it is important that the reported data is presented in such a way that is easy to understand and interpret.
As for which categories to choose, we considered the ones that have been observed to be most correlated with academic performance, according to studies such as the ones just cited. Furthermore, we made sure that our available data could be directly associated to these categories.
In the end, we have settled with the following ve categories: 1. Learning strategies. These encompass the variety of ways in which students interact with course resources and undertake tasks. Depending on their learning strategies, students may take super cial or deep approaches to learning (focusing on repetition and memorization, or making an e ort to understand contents), which may be more or less e ective depending on the speci c subject. This category also includes the student's own awareness of the learning strategies that they use and how e ective they are. 2. Time management. The e ectiveness of time management by a student is de ned by the amount of time they spend doing academic tasks, as well as the time frames they choose in order to do so. Ideally, students should be aware of the amount of time that they need to properly prepare their subjects and plan their study sessions around that. Additionally, they should avoid unnecessary delays in task performance, also known as procrastination. The ability of students to allocate time to personal activities or breaks also falls under this category. 3. Resource management. We de ne as resources not only the di erent type of learning materials at the student's disposal, but also elements such as interactions with teachers and other students, or the use of libraries and other study spaces. This category measures the ability of the student to use all of these resources to their advantage in order to improve their learning performance. 4. Self-monitoring and self-assessment. Self-monitoring is the student's capability to realize that they are making progress towards their academic objectives as they study or perform tasks. Meanwhile, self-assessment skills involve re ection on a previous task or study session, making sure that all goals established for said session were accomplished. In both cases, the student must be able to detect de ciencies in their work methods and apply solutions in order to improve them. 5. Motivation and self-con dence. These include several types of emotional factors that directly a ect students' learning, performance and self-regulation capabilities. These factors can be re ected in aspects and actions such as setting and pursuing learning goals, which milestones the student considers as reachable and unreachable, their estimated value of tasks and subjects, or the mental strength to overcome di culties that the course poses. 2.2
Like any other learning analytics-related task, we rely on the availability of student data in order to carry out this study. We will make a distinction between two kinds of data: observational data, which includes online student activity data gathered from LMS and other similar platforms, and self-reported data, which refers to information that is directly provided by students via surveys or questionnaires.
Observational data. This type of data is typically used in learning analytics studies.
In our case, we have two di erent sources of observational data. On the one
hand, course resources were made available to students via Moodle, and as such,
access logs provide useful activity data. While nding an ideal way to process
log data is a very complex problem in and of itself, we have found the methods
detailed by Jovanovic et al. [
On the other hand, the course used the Blended e-Assessment platform
(BeA) [
Self-reported data. Self-regulated learning questionnaires have been widely used by educational researchers for decades, and are still very popular to this day, due to their ability to provide data that is not easily obtainable via observations. For example, information regarding motivational aspects is easy to gather using questionnaires, but very di cult to infer using LMS logs. This is why we consider self-reported data to be a necessary complement to observational data in order to get a complete picture of a student's SRL pro le.
The main problem of self-reported data is the inherent bias of the students when they answer questionnaires or surveys. Information directly provided by students may not be accurate due to di erent factors, such as them having misconceptions about their own reality, or even students willfully lying when answering questionnaires. This is why self-reported data must be contrasted with observational data whenever this is possible.
As a result of their popularity, many di erent questionnaires have been
designed by a variety of authors throughout the years. For our own questionnaires,
we have adapted questions from previously existing ones, modifying them to
better suit our particular context. The questionnaires that served as inspiration
were:
{ Study Process Questionnaire (SPQ) [
The following subsections summarize the context in which this experiment was carried out, the ways data were collected throughout the course, and the provisional results obtained so far. 3.1
The focus of this study is a Computer Architecture course, part of the Degree in
Telecommunications Engineering, taught at University of Vigo, in Spain [
The course has two separate parts that students need to pass: theory and practice, both implementing a continuous assessment system. In the latter, students are presented with weekly assembly programming assignments that they need to solve, and the assessment consists of three exams performed throughout the semester. The theory part, instead of traditional lectures, follows a ipped classroom system: students are provided videos covering the subject contents to watch at home, and classroom sessions are used for questions and problem solving. Additionally, students perform short exams every two weeks, which may allow them to pass the subject without the need to do a nal exam.
A nal assessment system is also provided if the student so prefers, but following the continuous assessment system is encouraged. During the academic year 2020/2021, out of 212 total enrolled students in the subject, 123 followed the continuous assessment system. It is worth noting that this rate of students following continuous assessment is lower than the degree average, and it is explained by the fact that students who are retaking the subject often choose to follow the nal assessment system.
In the practical part of the subject, assignments are made available for students via the institutional LMS, based on Moodle. Grades of this part are also reported using this medium.
As for the theory part, videos, slides, self-assessment tests and any other kind of learning material are also available at the institutional LMS. However, exam handling is performed using BeA. This includes exam signups, grade reporting and reviews. 3.2
As this is the rst year in which this experiment is being performed, the main goal is to gather data from students, both self-reported and observed, and try to de ne basic self-regulated learning pro les. Additionally, identifying correlations between the gathered data and student performance will set the foundations for the implementation of an early warning system.
The use of Moodle in this course allows us to collect data related to student activity. Moodle logs provides information regarding when students log into the platform and which resources they visit. Additionally, the use of BeA provides the possibility of gathering assessment-related data that would be very di cult to obtain and process otherwise.
As explained in Section 2, SRL questionnaires are used in order to collect self-reported data from students. The questions use a 1 to 5 Likert-style scale, through which the student expresses their level of agreement or disagreement with the statement posed in each item. Each of the questions can be directly linked to one of the ve self-regulated categories that were de ned in Section 2.1. Students are never required to answer these questionnaires, but are encouraged to do so.
At the beginning of the course | during its second week |, a 20-item SRL questionnaire was performed during in-person theory sessions. This initial questionnaire includes 4 questions related to each of the ve SRL categories, and has the main purpose of providing basic information for SRL pro ling. Appendix A lists the 20 items that were include in this initial questionnaire.
Additionally, several shorter questionnaires of 7 items each are made available to students through BeA at di erent points of the course. Three of these smaller questionnaires were scheduled throughout the semester, making them available every 4 weeks. The intended purpose of these are tracking the evolution of student views regarding their self-regulation abilities. On top of this, they are designed as brief self-re ection exercises for students.
Regarding the SRL questionnaires, students that followed the continuous assessment system were split into two groups of equal size: an experimental group, which have access and are encouraged to answer the questionnaires as they are made available, and a control group, which are asked to ful ll the rst questionnaire at the beginning of the course, but none of the subsequent ones. 3.3
As of the writing of this paper, we have computed the self-reported data obtained from the initial questionnaire. Having been performed during an in-person session, a total of 113 students completed this questionnaire, a very signi cant fraction of those following the continuous assessment system.
Figure 1 displays the answer distribution for each of the 20 items in this questionnaire, classi ed by their respective self-regulated learning category. The number inside each tile in the graph represents the number of students that provided a particular answer in the corresponding question. On the other hand, Figure 2 represents the averages and standard deviation observed in the answers for each question.
As it can be observed, while there is usually a clearly preferred answer in each question, the variation in answers is not insigni cant. The standard deviation in the answers ranges from 0:583 in the most homogeneous one (question 10), to 1:318 in the most heterogeneous (question 11), and it ranges between 0:8 and 1:0 for most questions. This suggests that it may be possible to identify student clusters depending on their answers to the questionnaire, and possibly link them to strengths or weaknesses regarding speci c self-regulated learning categories.
It is worth noting that there is a negative correlation between averages and standard deviations for each question. Question 10, which as aforementioned is the one for which the lowest standard deviation value was observed, is also the question with the highest average answer value (4:46). Likewise, question 11 was the one with more deviation in its answers, and the one that had the lowest mean answer value (2:68). Generally, this means that there are items for which most students agree with the option that represents the \best practices" in terms of self-regulation, while some other questions are more controversial and varied in terms of their answers.
If we group questions into their respective SRL categories, the average answer values for each one can also be calculated: { Learning strategies: 3:62 { Time management: 3:28 { Resource management: 3:66 { Self-monitoring and self-assessment: 3:66 { Motivation and self-con dence: 3:42
From the results of this questionnaire alone, it is not possible to discern which SRL categories the average student is weaker at. However, this was useful to determine what the focus should be in the following, smaller questionnaires. The categories that resulted in a slightly lower overall score were Time management and Motivation and self-con dence. Additionally, there is the fact that, as explained in Section 2.1, Motivation and self-con dence is arguably the category that is hardest to assess using observational data. Thus, we decided to include mostly questions from this category in future questionnaires.
Figure 3 represents the distribution of answers by students if they are grouped into their respective categories, and con rms the conclusion that is inferred from the average values: the gures for Learning strategies, Resource management and Self-monitoring and self-assessment look almost identical to each other, while Time management and Motivation and self-con dence show slightly lower overall values. So far, we have only fully processed the results from the rst questionnaire. We will be progressively incorporating data obtained from the course, both observational and self-reported, in order to properly assess the SRL pro les of students. We then intend to look for correlations with course performance data, and determine what kind of SRL de ciencies put the student at most risk of failing or abandoning the course. This will allow us to build an early warning system based on self-regulation data.
These are the ways in which we intend to use data at our disposal: { Extra SRL questionnaires performed at di erent points of the course provide further information on the evolution of the students' SRL abilities. As explained in section 3.2, students that follow the continuous assessment are split into control and experimental groups, of roughly 60 students each, and only the ones in the latter are allowed to view and answer the extra questionnaires. So far, we have observed that only about one third of the students in the experimental group actually answered the rst extra questionnaire. Thus, additional measures to foster participation may be required in order to improve the usefulness of the extra SRL questionnaires.
Once the results for all questionnaires during the course have been collected, we intend to validate the questions using Cronbach's alpha coe cient. Particularly, we will check whether questions that address the same SRL category have reliable and consistent answers. The validation outcome will be taken into account to improve the surveys that will be performed during the next academic year. { Moodle activity data can be used to track the use of learning resources by students. On top of being useful to assess the Time management and Resource management by students, these activity logs can also provide hints towards identifying Learning strategies : for example, observing if a student prioritizes some kinds of resources or activities over others, or if there are some topics that a student deliberately avoids. { Finally, BeA data can provide insights towards assessing students' Selfmonitoring and self-assessment. With the help of these data, we could identify the kind of mistakes that students make the most during exams, and even if they repeat similar mistakes across multiple questions or di erent exams. 4
It is unquestionable that self-regulation plays a pivotal role in students' performance and quality of learning at any level of education. However, this aspect is often forgotten due to its relative obscurity, not being taken into account by students and instructors alike. This project aims to raise awareness about self-regulation among the educational community, providing a way to assess the strengths and weaknesses of students in di erent self-regulated learning aspects.
While this work is still at an early stage, we expect that the volumes of data that we handle, both self-reported and observational, can help us build decently reliable SRL pro les at early stages in a course.
The lines for immediate future work were outlined in Section 3.4. We will continue to work with data from the target Computer Architecture course in future academic years. Additionally, we have been contacting other academic institutions of di erent educational levels in order to seek lines of cooperation. It would be ideal to test the ways in which the knowledge acquired from the experiments in Computer Architecture could be applied in other contexts. Acknowledgment. We want to thank Javier Montoto Urrabieta for his support in the development and maintenance of BeA.
This work is partially nanced by public funds granted by the Galician regional government, with the purpose of supporting research activities carried out by PhD students. (\Programa de axudas a etapa predoutoral da Xunta de Galicia | Conseller a de Educacion, Universidade e Formacion Profesional")
This work has received nancial support from the Xunta de Galicia (Centro singular de investigacion de Galicia accreditation 2019-2022) and the European Union (European Regional Development Fund - ERDF), and from the Galician Regional Government under project ED431B 2020/33.
A
The following is a list of the questions that were part of the initial questionnaire, which students lled at the start of the course | originally in Spanish. Beside each question is the SRL category it is associated with: learning strategies (LS), time management (TM), resource management (RM), self-monitoring and selfassessment (MA) or motivation and self-con dence (MC).
Note: questions 11 and 14 are asked in such a way that \negative" answers are those with a higher level of agreement. Therefore, the values of the answers were inverted before analysis (1 becomes 5, 2 becomes 4, and vice versa.)