Supporting HE Students' Competence in Using SFLA for SRL

Abstract

Higher education (HE) students are increasingly offered autonomy in shaping their education. In these circumstances, self-regulation is important to learn effectively. However, self-regulated learning (SRL) is challenging for students. Learning analytics offer new possibilities to formulate and deliver rich external feedback in support of SRL. But how students actually engage with learning analytics in realworld settings and how this influences their learning is not clear. In a learning environment where feedback is directly presented to students, students seem to lack the necessary competencies to make sense of the information presented. This research further explores how students use student-facing learning analytics (SFLA) for SRL and what competencies would be beneficial. A mixed-method approach will be used to design an intervention to support the use of SFLA for SRL.

Keywords

• Student-facing learning analytics
• self-regulated learning
• student competencies
• Higher Education 1

1. Introduction

In higher education (HE), students can increasingly choose, for instance, mode of delivery, content, time, instructional approach or assessment [ 1 ]. Selfregulation is a critical factor to learn effectively when offered this kind of autonomy [ 2 ]. However, selfregulated learning can be challenging for students [ 3 ], [ 4 ]. For instance in monitoring their learning accurately. Internal and external feedback are fundamental elements for self-regulated learning (SRL) [ 5 ]. Educational data-related technologies, like learning analytics (LA), offer new possibilities to formulate and deliver rich external feedback. For example, feedback can be personalised, delivered in time, on a large scale and (partly) automated [ 6, 7 ]. Measuring or visualising learning actions and on the other hand recommending and guiding improvements with LA, can provide feedback and thereby support SRL [ 3 ].

However, students and staff express concerns about misinterpretation of data and a lack of understanding on how to improve the use of data in learning analytics [ 8 ]. Students use of technology for learning, like for instance LA, is varied and often needs significant training [ 9 ]. How students actually engage with learning analytics in real-world settings and how this influences their learning is still under-researched [ 7, 10, 11 ]. Students are often in need of extra information and differ in the amount and nature of support for SRL they need [ 12 ]. In a learning environment where feedback is directly presented to students, students may not be adequately prepared to

Self-regulated learning (SRL) is an important element when receiving autonomy in learning. SRL has been researched extensively and several models have been developed [ 14 ]. In short, SRL refers to monitoring and controlling one’s learning processes, with a flow of information between object- and meta-level [ 15 ]. Student’s engagement with learning tasks at objectlevel, delivers input for metacognitive processes [ 16 ]. Monitoring in SRL describes a flow of information about learning, providing input for metacognitive thoughts and feelings about specific cognitions [ 15 ]. This results into an action or intention to act, called control or regulation [ 17 ]. With a desired (learning) goal in mind, the student monitors the discrepancy between the current state and goal state. If the student perceives a mismatch it ideally fosters further regulation by setting goals, planning actions and monitoring their progress towards these goals in a continuous loop, until the student’s self-monitored state aligns with the outcome the student desires [ 18 ]. By monitoring their learning, successful students generate internal feedback [ 5 ]. However, students often lack necessary competencies to accurately monitor their learning and for instance tend to overestimate their understanding of learning materials [ 3, 4 ]. Students who lack self-regulatory skills are less capable to generate accurate internal feedback and rely more on external sources of feedback. External feedback is often conveyed by an instructor, but can also be provided via technology in the form of a learning analytics dashboard directly presented to the student (i.e. student-facing learning analytics). Educational data-related technologies like learning analytics (LA) offer promises for delivering personalised feedback at scale [ 7 ]. For students, LA and the provision of relevant study-data offer great opportunities to self-regulate their learning and enhance student-autonomy [ 19–21 ].

1.2. Student-facing Learning

Analytics

Learning Analytics (LA) is commonly defined as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs” [22]. LA provide possibilities for improvement of learning, by offering new means of feedback to support SRL, roughly relating to measuring or visualising learning actions and on the other hand recommending and guiding improvements [ 3 ]. Learning analytics dashboards (LADs) are often used as tools to make data about learners and their contexts available to policy makers, instructors, and students. LADs can positively influence the learning process if LAD design takes into account the regulatory mechanisms underlying selfregulated learning, like planning, monitoring and adapting [23]. If the information is directly presented to students, without conveyance by, for instance, an instructor, a learning analytics intervention (e.g. LAD) is considered student-facing [21]. In HE settings that are increasingly technology-enriched and with greater focus on student autonomy, student-facing learning analytics can be an important source of information for SRL.

1.3. SFLA for SRL

Student-facing learning analytics (SFLA) is a form of external feedback provided by technology and based on study-data, directly aimed at the student. This external feedback can help students to set goals and reflect upon goal achievement [24]. It is a source of information for monitoring and a means to attain an accurate judgement of ‘being on track’. LA can offer valuable insights for SRL, but relevant interactions with LA and how it influences student learning has not been researched extensively [ 7, 10, 11 ]. In recent years, human-centred design of LADs gained more attention and improvements in trace data have improved the possibilities of visualising student behaviour. Nevertheless, the insight in whether, to what extent and how data is used by students is limited and the benefits for student learning and selfregulation are still not fully explored [ 3, 25 ].

At this point learning analytics dashboards (LADs) are rarely grounded in learning theory [20]. Technology-related aspects of LADs, like user acceptance and ease of use are often given more attention than educational aspects, such as addressing learners cognitive and emotional competence [26].

Without downplaying task-related and toolrelated conditions, student competencies are also relevant. Bodily and Verbert [21] show in their literature review on student-facing LADs that actual student use of study-data presented in a system, is generally low. Students tend to not comply to or follow the advice from technology-provided SRL support [ 12 ]. Coping with some degree of imperfect SFLA will probably be a much needed, overarching ability for students in an authentic educational setting. Butler and Winne [5, p. 275] state: “learners' knowledge, beliefs, and thinking jointly mediate the effects of externally provided feedback.”. Shibani [11, p. 326] states that “students possess different levels of skills to meaningfully engage with automated feedback”. Therefore this research will focus on bringing together the most relevant student competencies (i.e. knowledge, skills and attitudes) in regard to using SFLA for SRL.

Also, most research focused on measuring rather than supporting SRL [ 3 ]. Technological support of SRL often does not make clear to learners how their actions relate to progress towards their learning goals [ 12 ]. LADs rarely offer insight in effective learning tactics and strategies [20, 27, 28]. That is, student-facing learning analytics dashboards rarely provide actionable information to regulate learning.

So, besides that students might lack the necessary competencies to accurately self-regulate their learning, the proposed solution of student-facing learning analytics seems to have its own barriers in regard to competencies. Furthermore, insight in how LA can support students to make sense of all these bits and pieces for SRL, is lacking.

This proposed research contributes to these topics by addressing the following questions: • RQ1: How do HE students engage with SFLA for SRL? • RQ2: What competencies and tool characteristics are considered beneficial for SFLA for SRL? • RQ3: How do differences in competencies between students relate to use of SFLA for SRL, in authentic HE settings? • RQ4: Does the proposed intervention support the competencies relevant to the use of SFLA for SRL and the actual use of SFLA for SRL?

2. Methodology of proposed research

2.1. Study 1: Systematic literature review

Study 1 addresses the first and second research question based on available research. A systematic literature review following the PRISMA guidelines [29] will offer insight in how HE students use SFLA for SRL, and specifically what phases of SRL are targeted. We will also retrieve information on which competencies at a student level matter for the uptake of SRL related information. The scope of this review will be studentfacing, technology-provided feedback, instead of feedback provided by an instructor. This means that LA should be directly reported to students.

The review will provide input for the interviews (study 2) and will offer suggestions for collection of relevant trace-data. Meaningful trace data can serve as proxy for use of SFLA for SRL (study 3).

2.2. Study 2: Trace data and interviews

Gathering information on how students use information for SRL in authentic higher education settings will be a challenging task, but is important to enrich and deepen the insight from the literature review (study 1). In a digital (learning) environment we can track the actual behaviour to visualise different steps that students take in the learning process. Tracedata about the use of SFLA for SRL can serve as a proxy for behaviour. But despite the abundance of data and learning technology, some issues in regard to trace data have to be addressed. First, available trace-data is often only slightly indicative for learning and second, LA-research often only observes part of the actual learning process (the part taking place within the digital learning environment) [ 6 ]. This calls for an additional data-collection besides looking at tracedata.

Interviews can extend the scope beyond trace data from the digital learning environment. By using a mixed-method approach, combining quantitative (interpretation of trace-data) and qualitative techniques (interviews), the trace-data can be better interpreted and interviews can be more focussed.

Besides complementing insights from study 1, the interviews will offer suggestions for collection of relevant trace data for use of SFLA for SRL in study 3.

2.3. Study 3: Differences in competencies in authentic educational settings

Study 3 will examine the relation between student competencies and the use of SFLA for SRL by students in authentic educational settings (RQ3). The first and second study yielded insights into what (trace) data is relevant to collect as indicator for use of SFLA for SRL. Student competencies are collected using a questionnaire that builds on the insights on relevant competencies from the first two studies.

To guarantee a minimal level of task and toolconditions of the instruments used in this research we intend to use tooling that has already been tested and improved in regard to user experience and data quality. Furthermore we organize expert reviews to assess the quality of the tools, for instance in regard to ease of use and fit for purpose. To monitor the experienced level of usability of the tool we will include user-experience (for instance the System Usability Scale (SUS) [30]) to the questionnaire in study 3 and 4.

2.4. Study 4: Intervention on

competencies relevant to

SFLA for SRL

We aim to design an intervention that supports the development of students’ competencies relevant for the use of SFLA for SRL and improve their use of SFLA for SRL (RQ4). The exact nature of this intervention has to be decided.

The effect on student competencies will be measured by pre- and post-test use of the questionnaire developed in study 3. The influence on use of SFLA for SRL is shown using relevant trace data as identified in study 2 and observations of students’ use of SFLA for SRL or think-aloud protocols.

3. Current results

At the time of writing, the systematic review is in progress (study 1). From September 2023 onwards, we will conduct the student interviews (study 2) alongside writing up the results of the systematic literature review. Currently we are examining the possible educational practices within our institute in which SFLA is being used. On the shortlist are Brightspace (an LMS recently implemented within our institution) and AvansOne (an application aimed at students, combining information from several sources). The availability of relevant data, maturity and adoption within the organisation are important criteria. We are open to interesting HE contexts outside our own institution.

4. Contribution to TEL domain

The proposed research will build on the interesting work on (student-facing) learning analytics that has been produced in the TEL domain. Specifically, the proposed research will contribute to the domain of student-facing learning analytics (SFLA) and selfregulated learning (SRL), by investigating how students engage with SFLA for SRL, what competences are needed to do so and how these competences and the use of SFLA for SRL can be improved by the developed intervention. The focus on student interaction with, and competencies for use of LA for SLR is underexamined in current research, yet important in regard to life-long learning and educational contexts that ask for student autonomy.

Besides this scientific output, we hope that the more competent students are, the better they can be co-producers in human-centred design of LADs. environments in higher education: A systematic review,” Australas. J. Educ. Technol., vol. 37, no. 6, Art. no. 6, Aug. 2021, doi: 10.14742/ajet.6453. [20] W. Matcha, N. A. Uzir, D. Gašević, and A. Pardo, “A Systematic Review of Empirical Studies on Learning Analytics Dashboards: A Self-Regulated Learning Perspective,” IEEE Trans. Learn. Technol., vol. 13, no. 2, Art. no. 2, Apr. 2020, doi: 10.1109/TLT.2019.2916802. [21] R. Bodily and K. Verbert, “Review of Research on Student-Facing Learning Analytics Dashboards and Educational Recommender Systems,” IEEE Trans. Learn. Technol., vol. 10, no. 4, Art. no. 4, Oct. 2017, doi: 10.1109/TLT.2017.2740172. [22] SoLAR, “What is Learning Analytics?,” Society for Learning Analytics Research (SoLAR). Accessed: Jun. 20, 2022. [Online]. Available: https://www.solaresearch.org/about/what-islearning-analytics/ [23] G. Sedrakyan, J. Malmberg, K. Verbert, S. Järvelä, and P. A. Kirschner, “Linking learning behavior analytics and learning science concepts: Designing a learning analytics dashboard for feedback to support learning regulation,” Comput. Hum. Behav., vol. 107, p. 105512, Jun. 2020, doi: 10.1016/j.chb.2018.05.004. [24] M. Theobald and H. Bellhäuser, “How am I going and where to next? Elaborated online feedback improves university students’ self-regulated learning and performance,” Internet High. Educ., vol. 55, p. 100872, Oct. 2022, doi: 10.1016/j.iheduc.2022.100872. [25] B. T. Wong and K. C. Li, “A review of learning analytics intervention in higher education (2011–2018),” J. Comput. Educ., vol. 7, no. 1, pp. 7–28, Mar. 2020, doi: 10.1007/s40692-01900143-7. [26] Y.-S. Tsai et al., “Learning analytics in European higher education—Trends and barriers,” Comput. Educ., vol. 155, Oct. 2020, doi: 10.1016/j.compedu.2020.103933. [27] D. Gasevic, Y.-S. Tsai, S. Dawson, and A. Pardo, “How do we start? An approach to learning analytics adoption in higher education,” Int. J. Inf. Learn. Technol., vol. 36, Jun. 2019, doi: 10.1108/IJILT-02-2019-0024. [28] A. F. Wise and Y. Jung, “Teaching with Analytics: Towards a Situated Model of Instructional Decision-Making,” J. Learn. Anal., vol. 6, no. 2, Art. no. 2, Jul. 2019, doi: 10.18608/jla.2019.62.4. [29] “PRISMA.” Accessed: Jul. 01, 2023. [Online].

Available: http://prismastatement.org/PRISMAStatement/ [30] J. Brooke, “SUS - A quick and dirty usability scale,” p. 7.

References

[1] K. C. Li and B. Y. Y. Wong , “ Revisiting the Definitions and Implementation of Flexible Learning,” in Innovations in Open and Flexible Education , K. C. Li , K. S. Yuen , and B. T. M. Wong , Eds., in Education Innovation Series. , Singapore: Springer, 2018 , pp. 3 - 13 . doi: 10 .1007/ 978 -981- 10-7995- 5 _ 1 .

[2] R. Boelens , B. De Wever, and M. Voet , “ Four key challenges to the design of blended learning: A systematic literature review , ” Educ. Res. Rev. , vol. 22 , pp. 1 - 18 , Nov. 2017 , doi: 10.1016/j.edurev. 2017 . 06 .001.

[3] O. Viberg , M. Khalil , and M. Baars , “ Self-Regulated Learning and Learning Analytics in Online Learning Environments: A Review of Empirical Research,” in Proceedings of the Tenth International Conference on Learning Analytics & Knowledge , in LAK '20. New York, NY, USA: Association for Computing Machinery, 2020 , pp. 524 - 533 . doi: 10 .1145/3375462.3375483.

[4] B. J. Zimmerman , “ Becoming a Self-Regulated Learner: An Overview,” Theory Pract ., vol. 41 , no. 2 , pp. 64 - 70 , May 2002 , doi: 10.1207/s15430421tip4102_ 2 .

[5] D. L. Butler and P. H. Winne , “ Feedback and SelfRegulated Learning: A Theoretical Synthesis , ” Rev. Educ. Res. , vol. 65 , no. 3 , Art . no. 3 , Sep . 1995 , doi: 10.3102/00346543065003245.

[6] D. Tempelaar , “ Supporting the less-adaptive student: the role of learning analytics, formative assessment and blended learning,” Assess. Eval. High. Educ. , vol. 45 , no. 4 , pp. 579 - 593 , May 2020 , doi: 10.1080/02602938. 2019 . 1677855 .

[7] A. Pardo , O. Poquet , R. Martinez-Maldonado , and S. Dawson , “ Provision of Data-Driven Student Feedback in LA & EDM,” in Handbook of Learning Analytics , SOLAR, 2017 , pp. 163 - 174 . doi: 10 .18608/hla17. 014 .

[8] A. J. Braunack-Mayer , J. M. Street , R. Tooher , X. Feng , and K. Scharling-Gamba , “Student and staff perspectives on the use of Big Data in the tertiary education sector: A scoping review and reflection on the ethical issues , ” Rev. Educ. Res. , vol. 90 , no. 6 , pp. 788 - 823 , Dec. 2020 , doi: 10.3102/0034654320960213.

[9] P. A. Kirschner and P. De Bruyckere , “ The myths of the digital native and the multitasker ,” Teach. Teach. Educ., vol. 67 , pp. 135 - 142 , Oct. 2017 , doi: 10.1016/j.tate. 2017 . 06 .001.

[10] I. Jivet , M. Scheffel , M. Schmitz , S. Robbers , M. Specht , and H. Drachsler , “ From students with love: An empirical study on learner goals, selfregulated learning and sense-making of learning analytics in higher education,” Internet High . Educ., vol. 47 , p. 100758 , Oct . 2020 , doi: 10.1016/j.iheduc. 2020 . 100758 .

[11] A. Shibani , S. Knight , and S. Buckingham Shum , “ Questioning learning analytics? Cultivating critical engagement as student automated feedback literacy,” in LAK22: 12th International Learning Analytics and Knowledge Conference, in LAK22 . New York, NY, USA: Association for Computing Machinery, Mar. 2022 , pp. 326 - 335 . doi: 10 .1145/3506860.3506912.

[12] I. Molenaar , A. Horvers , and R. S. Baker , “ Towards Hybrid Human-System Regulation: Understanding Children' SRL Support Needs in Blended Classrooms,” in Proceedings of the 9th International Conference on Learning Analytics & Knowledge , Tempe AZ USA: ACM, Mar . 2019 , pp. 471 - 480 . doi: 10 .1145/3303772.3303780.

[13] A. Pardo et al., “OnTask: Delivering DataInformed, Personalized Learning Support Actions,” J. Learn. Anal. , vol. 5 , no. 3 , Art . no. 3 , Dec . 2018 , doi: 10.18608/jla. 2018 . 53 .15.

[14] E. Panadero , “ A Review of Self-regulated Learning: Six Models and Four Directions for Research,” Front. Psychol. , vol. 8 , p. 422 , 2017 , doi: 10.3389/fpsyg. 2017 . 00422 .

[15] T. O. Nelson , “ Metamemory: A Theoretical Framework and New Findings,” in Psychology of Learning and Motivation , vol. 26 , Elsevier , 1990 , pp. 125 - 173 . doi: 10 .1016/S0079- 7421 ( 08 ) 60053 - 5 .

[16] A. B. H. de Bruin , J. Roelle , S. K. Carpenter , M. Baars , and EFG-MRE, “Synthesizing Cognitive Load and Self-regulation Theory: a Theoretical Framework and Research Agenda,” Educ. Psychol. Rev., vol. 32 , no. 4 , pp. 903 - 915 , Dec. 2020 , doi: 10.1007/s10648-020-09576-4.

[17] A. B. H. de Bruin and T. van Gog, “ Improving selfmonitoring and self-regulation: From cognitive psychology to the classroom,” Learn . Instr., vol. 22 , no. 4 , pp. 245 - 252 , Aug. 2012 , doi: 10.1016/j.learninstruc. 2012 . 01 .003.

[18] I. Roll and P. H. Winne , “ Understanding, evaluating, and supporting self-regulated learning using learning analytics,” J. Learn . Anal., vol. 2 , no. 1 , Art . no. 1 , Jul . 2015 , doi: 10.18608/jla. 2015 . 21 .2.

[19] J. H. Eggers , R. Oostdam , and J. Voogt , “ Selfregulation strategies in blended learning