=Paper=
{{Paper
|id=Vol-3292/paper05
|storemode=property
|title=Measuring and supporting self-regulated learning in blended learning contexts
|pdfUrl=https://ceur-ws.org/Vol-3292/DCECTEL2022_paper05.pdf
|volume=Vol-3292
|authors=Esteban Villalobos,Mar Pérez-Sanagustin,André Tricot,Julien Broisin
|dblpUrl=https://dblp.org/rec/conf/ectel/VillalobosPTB22
}}
==Measuring and supporting self-regulated learning in blended learning contexts==
https://ceur-ws.org/Vol-3292/DCECTEL2022_paper05.pdf
Measuring and supporting self-regulated learning in
blended learning contexts
Esteban Villalobos1,* , Mar Pérez-Sanagustin1 , André Tricot2 and Julien Broisin1
1
IRIT, Université de Toulouse, CNRS, Toulouse INP, UT3, Toulouse, France
2
Université Paul-Valéry Montpellier 3, EPSYLON
Abstract
Despite the positive effects of Blended Learning (BL), several studies have shown that students require high levels of self-
regulation to succeed in these types of practices. Still, there is little understanding of how students organize their learning
in BL authentic contexts. This paper presents the objectives and current status of a project that seeks to understand how
students’ Self-regulated Learning (SRL) strategies manifest themselves in BL contexts holistically and how to foster it through
technological solutions. The contributions of this project will be three-fold. First, we aim to develop novel analytical and
technological solutions to understand better the dynamics of how self-regulated learning unveils in BL contexts. Second
is the development of a dashboard-based support tool for students and teachers. And third, we will provide evaluations of
the analytical framework and support tool in authentic BL contexts. We expect that these contributions will provide the
community with a better understanding of the dynamics of SRL in BL.
Keywords
Self-regulated Learning, Blended Learning, Learning Analytics
1. Introduction gies in a BL course, in a Flipped Classroom (FC), and in
Massive Open Online Courses (MOOCs), showing that
In the last few years, we have seen Blended Learning (BL) students used similar strategies in BL and FC modalities,
approaches becoming more varied and commonly applied but these differed from the tactics used in MOOCs. More-
[1]. This methodology consists in combining online and over, [3] showed that BL students used SRL strategies
traditional in-person activities [2]. Nonetheless, while BL less often than online students. Overall, there seems to
has been shown to have positive effects on learning, many be a strong connection between the course design, the
students often have problems regulating their study [3, 4, learners’ SRL ability profile, and the learning strategies
2]. This has prompted a growing interest in finding out in the course [9, 7].
how to understand and support students’ self-regulation To support students’ SRL, researchers propose differ-
abilities in BL. ent mechanisms. One of these mechanisms is using
Self-regulated Learning (SRL) is defined as a complex dashboard-based tools. These tools provide learners with
process that combines meta-cognitive, motivational, and information about their progress. Although most of these
emotional processes [5]. Recent literature shows that tools have been designed and evaluated in online environ-
students’ SRL ability is a good predictor of their behavior ments with encouraging results [10], only a few works
and success in a course [6]. However, most studies on show how students incorporate them into their learning
SRL have been conducted in online contexts and little strategies and have an impact on their behavior in BL
is known about how these processes manifest in BL [3]. courses [11, 12].
Recent works show that students’ SRL manifests differ- In order to give meaningful SRL support in BL it is im-
ently depending on pedagogical decisions, such as the portant to understand how different external factors (e.g.,
learning context and course modality [3, 7, 8, 9]. For the influence of the teacher or face-to-face classes) and
example, Matcha et al. [9] compared students’ strate- internal factors (e.g., students’ self-regulation abilities)
Proceedings of the Doctoral Consortium of the Seventeenth European affect learners in these contexts. These factors influence
Conference on Technology Enhanced Learning, September 12–16, 2022, how students will interact with the learning material
Toulouse, France. along the course. This represents a particular challenge
*
Corresponding author. in TEL, as it implies that strategies observed will be heav-
$ esteban.villalobos@irit.fr (E. Villalobos);
ily influenced by the dynamics of the system in which
mar.perez-sanagustin@irit.fr (M. Pérez-Sanagustin);
andre.tricot@univ-montp3.fr (A. Tricot); julien.broisin@irit.fr the students operate [13, 14]. This points out the need to
(J. Broisin) develop new holistic approaches to understand the SRL
0000-0002-6026-3756 (E. Villalobos); 0000-0001-9854-9963 behavior of the students better.
(M. Pérez-Sanagustin); 0000-0003-4005-7338 (A. Tricot); This work is part of a 3-year thesis starting in October
0000-0001-8713-6282 (J. Broisin)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License 2021, in which we expect to contribute to the TEL domain
CEUR
Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
by addressing these gaps. Specifically, we propose: (1)
studying new analytics techniques to understand the methods to create an analytical framework for
development of SRL strategies in BL holistically and (2) characterizing the dynamics of students’ strate-
developing technological solutions to support SRL in BL. gies in BL?
2. Objectives and research 2.2. Supporting SRL in BL
questions Researchers have proposed different approaches to sup-
port students’ SRL processes [19]. The most common
The general objective of this project is to investigate the approaches explored are educational prompts and inte-
SRL strategies used by learners in BL scenarios and to grated support systems [20]. These solutions transform
propose and evaluate a Learning Analytics (LA) techno- raw data into ‘actionable insights’ to produce behavioral
logical solution based on user-centered dashboards (for changes in the students [21]. So far, most of this prior
teachers and students) to support those strategies that work has been conducted in online settings, such in Mas-
maximize learners’ performance. Three main objectives sive Open Online Courses (MOOCs), in which students
are derived from this general objective: have low interaction with the teacher [20]. These stud-
ies suggest that dashboards could be an appropriate ap-
• Objective 1: To propose an analytical framework proach for supporting SRL strategies. In particular, the
to study in a holistic manner how students’ SRL strategies of goal setting, strategic planning, time man-
strategies manifest in BL contexts. agement, and monitoring have been shown to be more
• Objective 2: To design a LA dashboard-based effective for promoting students’ motivation and impact
solution for teachers and students to support SRL on course performance.
in BL. There are still very few studies looking at these solu-
• Objective 3: To evaluate the impact of LA solu- tions BL contexts (e.g., [22, 23]). These works in BL have
tion on students’ learning strategies and teachers’ two main limitations. First, the tools focus on support-
decision-making in BL scenarios. ing the students directly, usually overlooking the role
of the teacher. Second, while some tools are based on
2.1. Measuring SRL in BL theoretical models for SRL, there is still much to under-
stand about their impact on students’ SRL strategies. This
Different methods have been proposed for studying how posses the following research questions for the project:
SRL manifests in different learning contexts, especially in
online learning environments. These range from using • RQ2: How useful (interpretable, actionable, and
self reported data [15] to detecting tactics and strate- comprehensive) are the existing indicators pro-
gies by using the trace data collected from the course’s vided in the SRL-support dashboard for students
LMS [16, 7, 17, 18, 9]. The latter has seen many contri- and teachers?
butions from the field of Learning Analytics (LA). Some • RQ3: How do SRL support tools influence stu-
examples of these analytical approaches have used tech- dents’ strategies and teachers’ decision-making
niques derived from temporal analysis and sequence min- in BL scenarios?
ing [17, 16]. Some studies have also made the connection
between these techniques and the SRL theory [16]. Fan
et al. [16] suggests this theoretical backbone may allow 3. Project Methodology
us to overcome the limitations of the context-specific
nature of LA to perform pedagogical interventions that Design Based Research (DBR) will be used as a method-
go beyond course setting. ological approach, which combines experiments in real-
Most of these methods have been applied in online set- world settings with theoretical models [24]. The inter-
tings, and very few have been applied in Blended Learn- ventions will be based on the NoteMyProgress (NMP)
ing settings. The currently applied methods are limited in tool [25], a Moodle plug-in that delivers dashboards with
capturing the impact of factors such as teacher interven- self-regulation indicators in the course to both students
tions and face-to-face classes. In fact, current research and teachers (see Figure 1). Three experimental cycles
applying existing methods in Blended Learning encoun- will be carried out to improve the tool and the analytical
ters difficulties in providing indicators on run-time, as frameworks in an iterative way. After each cycle, the
well as in giving a temporal meaning to the collected data. results will be published as part of the LASER project
From this, we derive the following research question: following an Open Science Framework.
• RQ1: How can pre-existing LA methods and tech-
niques be adapted and combined with qualitative
Figure 1: Examples of visualizations in the NoteMyProgress plug-in
Figure 2: Analytical approach used to evaluate the first design cycle [26].
4. Current Results: First Design and previous achievements. The approach consists of the
following steps:
Cycle
1. Separating the activity of the students into
The first cycle focused on studying students’ behavior in sessions. These correspond to a sequence of
BL. This cycle had three research questions: actions not separated by more than 30 minutes
1. How do students’ learning tactics and strategies of inactivity.
manifest along the BL course? 2. Detecting the underlying tactic of each ses-
2. Does the NMP tool, designed to support students’ sion. A tactic is defined as the underlying pro-
SRL, have an effect on their learning tactics and cess that a student is applying in a given period
strategies? of time [17]. We used a Hidden Markov Model
3. Is there a relationship between students’ learning (HMM) in order to detect students’ tactics.
strategies, course performance, and SRL ability 3. Detecting students’ strategies. Under the ana-
profile? lytical approach proposed by Fincham et al. [17],
strategies are defined as sequences of tactics ap-
This intervention took place between September 2021
plied by the students. In order to include the con-
and January 2022. The study consisted on 241 students
text of the BL course, we included in this model
from two university courses. At the beginning of the
the timing with respect to the face-to-face ses-
course, students completed the informed consent for par-
sions.
ticipation and a questionnaire to assess their level of SRL.
4. Analyzing relationships between strategies
Midway through the course (week 6), they were intro-
and students’ profile. We analyzed how differ-
duced to NMP and invited to refer to it to assess their
ent tactics and strategies applied by the students
study strategies [27]. At the end of the course, they were
related to their SRL ability profile, course perfor-
asked to complete a questionnaire on their sense-making
mance, and previous achievements.
of the tool [25].
The evaluation of one of these courses is detailed in We found that students’ strategies were correlated
[26]. Here, we extended an analytical approach proposed with their previous achievements (GPA) and their self-
in Fincham et al. [17] and analyzed the results with re- reported Self-Regulation ability. We also found that the
spect to students’ SRL ability profile, final performance, tactics used by the students varied across modalities and
Figure 3: Examples of the ‘Student planning and goal setting’ functionalities added to NMP
5. Future work: Second Design
Cycle
The second design cycle focuses on the role of the teacher
in the BL course, as well as on students’ behavior when
they use support for planning their course. This cycle
will take place between September 2022 and January
2023. Based on the insights from the first cycle, new
developments were made to NMP. We developed new
functionalities of student planning and goal setting (see
Figure 3), and gamification (see Figure 4).
We aim to evaluate this intervention based on the tem-
poral dynamics of the students. Our goal is to understand
how external factors (such as feedback and gamification)
Figure 4: Example of the ‘Gamification’ functionalities added and internal factors (such as student planning) affect the
to NMP students’ SRL behavior. Following the recent works by
[14, 28, 29], we will study how context-dependent and
context-independent indicators behaviors throughout the
were based on the pedagogical decisions of the course. course and their potential to give meaningful information
In terms of the usage of NMP we found that even though to students and teachers. In the short term, we will be fol-
some students incorporated the SRL support tool into lowing behavior-based indicators already studied in the
their learning tactics, the use of the tool was relatively literature to provide students feedback week to week. In
sparse. We also found that, even if the use of the tool was the long term, we are looking to develop indicators based
not mandatory, most of the students interacted with the on point processes to capture more complex temporal
indicators relating to Strategic Planning. behavior from the students. This study will be done in
While this gives us some insight into the performance collaboration with the Millennium Nucleus Student Ex-
of the students in the course, this methodology still has perience in Higher Education in Chile (NMEDSUP) to see
some limitations. Mainly, since the methods applied are how this work can be extended to different institutions
"memory-less", we are losing information on the temporal and contexts.
dynamics of the events. Also, this methodology only
allows us to do a retrospective analysis of the course. This
limits our capability to perform meaningful interventions 6. Contribution to TEL domain
on run-time.
This work aims at advancing research in TEL, and in
particular in the study of SRL in BL scenarios, with three
contributions. Firstly, we expect to provide the commu-
nity with an analytical framework for understanding the
dynamics of SRL in BL in a holistic manner and taking
into consideration temporal aspects. These tools will help
in analyzing data but also in proposing indicators that [5] E. Panadero, A Review of Self-regulated Learn-
could serve researchers doing interventions on run-time. ing: Six Models and Four Directions for Re-
Second, we contribute with the NMP tool, a functional search, Frontiers in Psychology 8 (2017) 422.
tool that both teachers and students could use to support URL: http://journal.frontiersin.org/article/10.3389/
SRL, and its evaluation in authentic contexts. The cur- fpsyg.2017.00422/full. doi:10.3389/fpsyg.2017.
rent version of the tool is already openly available1 . And 00422.
third, we expect to contribute with exemplary scenarios [6] J. Maldonado-Mahauad, M. Pérez-Sanagustín, P. M.
on how to apply our analytical framework in BL. Moreno-Marcos, C. Alario-Hoyos, P. J. Muñoz-
These contributions will have implications at the the- Merino, C. Delgado-Kloos, Predicting Learn-
oretical level, the analytical level, and the teaching prac- ers’ Success in a Self-paced MOOC Through
tices level. We expect that our analytical framework and Sequence Patterns of Self-regulated Learning,
proposed tool can give the community greater insights in: V. Pammer-Schindler, M. Pérez-Sanagustín,
into how to understand the different factors that affect H. Drachsler, R. Elferink, M. Scheffel (Eds.),
the dynamics of SRL in BL. We hope that this allows the Lifelong Technology-Enhanced Learning, vol-
community to have a better understanding of how to ume 11082, Springer International Publishing,
support SRL in a holistic manner. Cham, 2018, pp. 355–369. URL: http://link.springer.
com/10.1007/978-3-319-98572-5_27. doi:10.1007/
978-3-319-98572-5\_27, series Title: Lecture
Acknowledgments Notes in Computer Science.
[7] Y. Fan, W. Matcha, N. A. Uzir, Q. Wang, D. Gaše-
This paper has been partially funded by the ANR LASER
vić, Learning Analytics to Reveal Links Between
(156322). The authors acknowledge PROF-XXI, which
Learning Design and Self-Regulated Learning, In-
is an Erasmus+ Capacity Building in the Field of Higher
ternational Journal of Artificial Intelligence in
Education project funded by the European Commission
Education 31 (2021) 980–1021. URL: https://link.
(609767-EPP-1-2019-1- ES-EPPKA2-CBHE-JP). This publi-
springer.com/10.1007/s40593-021-00249-z. doi:10.
cation reflects the views only of the authors and funders
1007/s40593-021-00249-z.
cannot be held responsible for any use which may be
[8] D. Gašević, N. Mirriahi, S. Dawson, S. Joksi-
made of the information contained therein.
mović, Effects of instructional conditions and
experience on the adoption of a learning tool,
References Computers in Human Behavior 67 (2017) 207–220.
URL: https://linkinghub.elsevier.com/retrieve/pii/
[1] K. Pelletier, M. McCormack, J. Reeves, J. Robert, S0747563216307270. doi:10.1016/j.chb.2016.
N. Arbino, 2022 EDUCAUSE Horizon Report, Teach- 10.026.
ing and Learning Edition (2022) 58. [9] W. Matcha, D. Gašević, N. Ahmad Uzir, J. Jovanović,
[2] C. R. Graham, Blended learning systems: Definition, A. Pardo, L. Lim, J. Maldonado-Mahauad, S. Gentili,
current trends, future directions, in: Handbook of M. Pérez-Sanagustín, Y.-S. Tsai, Analytics of Learn-
blended learning: Global Perspectives, local designs, ing Strategies: Role of Course Design and Delivery
San Francisco, CA: Pfeiffer Publishing, Brigham Modality, Journal of Learning Analytics 7 (2020) 45–
Young University, USA, 2004. 71. URL: https://learning-analytics.info/index.php/
[3] J. Broadbent, Comparing online and JLA/article/view/7008. doi:10.18608/jla.2020.
blended learner’s self-regulated learning 72.3.
strategies and academic performance, The [10] R. Pérez-Álvarez, J. Maldonado, M. Pérez-
Internet and Higher Education 33 (2017) Sanagustín, Tools to Support Self-Regulated
24–32. URL: https://www.sciencedirect. Learning in Online Environments: Literature
com/science/article/pii/S1096751617300398. Review: 13th European Conference on Technology
doi:10.1016/j.iheduc.2017.01.004. Enhanced Learning, EC-TEL 2018, Leeds, UK,
[4] J. Broadbent, M. Fuller-Tyszkiewicz, Pro- September 3-5, 2018, Proceedings, 2018, pp. 16–30.
files in self-regulated learning and their corre- doi:10.1007/978-3-319-98572-5_2.
lates for online and blended learning students, [11] M. Pérez-Sanagustín, D. Sapunar-Opazo, R. Pérez-
Educational Technology Research and Devel- Álvarez, I. Hilliger, A. Bey, J. Maldonado-Mahauad,
opment 66 (2018) 1435–1455. URL: http://link. J. Baier, A MOOC-based flipped experience: Scaf-
springer.com/10.1007/s11423-018-9595-9. doi:10. folding SRL strategies improves learners’ time man-
1007/s11423-018-9595-9. agement and engagement, Computer Applica-
tions in Engineering Education 29 (2021) 750–768.
1
https://gitlab.com/laser-anr/notemyprogress-plug-in URL: https://onlinelibrary.wiley.com/doi/10.1002/
cae.22337. doi:10.1002/cae.22337. Scaffolding self-regulated learning with CBLES,
[12] M. Yoon, J. Hill, D. Kim, Designing supports for Journal of Computer Assisted Learning 28 (2012)
promoting self-regulated learning in the flipped 557–573. URL: https://onlinelibrary.wiley.com/doi/
classroom, Journal of Computing in Higher 10.1111/j.1365-2729.2011.00476.x. doi:10.1111/j.
Education 33 (2021) 398–418. URL: https://link. 1365-2729.2011.00476.x.
springer.com/10.1007/s12528-021-09269-z. doi:10. [20] J. Wong, M. Baars, D. Davis, T. Van Der Zee, G.-
1007/s12528-021-09269-z. J. Houben, F. Paas, Supporting Self-Regulated
[13] S. Dawson, S. Joksimovic, O. Poquet, G. Siemens, Learning in Online Learning Environments and
Increasing the Impact of Learning Analytics, MOOCs: A Systematic Review, International
in: Proceedings of the 9th International Confer- Journal of Human–Computer Interaction 35 (2019)
ence on Learning Analytics & Knowledge, ACM, 356–373. URL: https://www.tandfonline.com/doi/
Tempe AZ USA, 2019, pp. 446–455. URL: https: full/10.1080/10447318.2018.1543084. doi:10.1080/
//dl.acm.org/doi/10.1145/3303772.3303784. doi:10. 10447318.2018.1543084.
1145/3303772.3303784. [21] R. L. Jørnø, K. Gynther, What Constitutes an
[14] J. Jovanović, M. Saqr, S. Joksimović, D. Gašević, ‘Actionable Insight’ in Learning Analytics?,
Students matter the most in learning analytics: Journal of Learning Analytics 5 (2018). URL: https:
The effects of internal and instructional conditions //learning-analytics.info/index.php/JLA/article/
in predicting academic success, Computers & Edu- view/5897. doi:10.18608/jla.2018.53.13.
cation 172 (2021) 104251. URL: https://linkinghub. [22] W.-J. Shyr, C.-H. Chen, Designing a technology-
elsevier.com/retrieve/pii/S0360131521001287. enhanced flipped learning system to facilitate stu-
doi:10.1016/j.compedu.2021.104251. dents’ self-regulation and performance, Journal
[15] M. Zhou, P. H. Winne, Modeling academic of Computer Assisted Learning 34 (2018) 53–62.
achievement by self-reported versus traced URL: https://onlinelibrary.wiley.com/doi/10.1111/
goal orientation, Learning and Instruction jcal.12213. doi:10.1111/jcal.12213.
22 (2012) 413–419. URL: https://linkinghub. [23] C. Michel, E. Lavoué, S. George, M. Ji, Sup-
elsevier.com/retrieve/pii/S0959475212000217. porting Awareness and Self-Regulation In Project-
doi:10.1016/j.learninstruc.2012.03.004. Based Learning through Personalized Dashboards,
[16] Y. Fan, J. Saint, S. Singh, J. Jovanovic, D. Gaše- International Journal of Technology Enhanced
vić, A learning analytic approach to unveiling Learning 9 (2017) 204–226. URL: https://hal.
self-regulatory processes in learning tactics, in: archives-ouvertes.fr/hal-01384211. doi:10.1504/
LAK21: 11th International Learning Analytics and IJâĎą.2017.084500.
Knowledge Conference, ACM, Irvine CA USA, [24] P. Reimann, Design-Based Research, in:
2021, pp. 184–195. URL: https://dl.acm.org/doi/ L. Markauskaite, P. Freebody, J. Irwin (Eds.),
10.1145/3448139.3448211. doi:10.1145/3448139. Methodological Choice and Design: Scholarship,
3448211. Policy and Practice in Social and Educational Re-
[17] E. Fincham, D. Gašević, J. Jovanović, A. Pardo, From search, Methodos Series, Springer Netherlands,
Study Tactics to Learning Strategies: An Analytical Dordrecht, 2011, pp. 37–50. URL: https://doi.
Method for Extracting Interpretable Representa- org/10.1007/978-90-481-8933-5_3. doi:10.1007/
tions, IEEE Transactions on Learning Technologies 978-90-481-8933-5_3.
12 (2019) 59–72. URL: https://www.scopus.com/ [25] M. Pérez-Sanagustín, R. Pérez-Álvarez,
inward/record.uri?eid=2-s2.0-85045204080& J. Maldonado-Mahauad, E. Villalobos, C. Sanza,
doi=10.1109%2fTLT.2018.2823317&partnerID= Designing a moodle plugin for promoting learners’
40&md5=84a758598f985bd200d40eab1fb0e45c. self-regulated learning in blended learning, in: Pro-
doi:10.1109/TLT.2018.2823317. ceedins of the Seventeenth European Conference
[18] D. Gasevic, J. Jovanovic, A. Pardo, S. Daw- on Technology-Enhanced Learning - EC-TEL ’22,
son, Detecting Learning Strategies with Analyt- Toulouse, France, In press.
ics: Links with Self-reported Measures and Aca- [26] E. Villalobos, M. Pérez-Sanagustín, C. Sanza, A. Tri-
demic Performance, Journal of Learning Analyt- cot, J. Broisin, Supporting self-regulated learning
ics 4 (2017). URL: https://learning-analytics.info/ in bl: Exploring learners’ tactics and strategies, in:
index.php/JLA/article/view/5085. doi:10.18608/ Proceedins of the Seventeenth European Confer-
jla.2017.42.10. ence on Technology-Enhanced Learning - EC-TEL
[19] A. Devolder, J. van Braak, J. Tondeur, Supporting ’22, Toulouse, France, In press.
self-regulated learning in computer-based learn- [27] P. R. Pintrich, E. V. D. Groot, Motivational and
ing environments: systematic review of effects of Self-Regulated Learning Components of Classroom
scaffolding in the domain of science education: Academic Performance (1990) 8.
[28] J. Jovanovic, N. Mirriahi, D. Gašević, S. Daw-
son, A. Pardo, Predictive power of regular-
ity of pre-class activities in a flipped class-
room, Computers & Education 134 (2019)
156–168. URL: https://www.sciencedirect.
com/science/article/pii/S0360131519300405.
doi:10.1016/j.compedu.2019.02.011.
[29] M. Saqr, J. Jovanovic, O. Viberg, D. Gašević, Is there
order in the mess? A single paper meta-analysis
approach to identification of predictors of success
in learning analytics, Studies in Higher Educa-
tion (2022) 1–22. URL: https://www.tandfonline.
com/doi/full/10.1080/03075079.2022.2061450.
doi:10.1080/03075079.2022.2061450.