Blended learning environments have the potential to provide educators with valuable insights into learner behaviours and strategies. Capturing and analysing learner data, using traditional frequency-based statistical methods, is a challenge if the objective is to understand self-regulated learning (SRL) as a dynamic process. Current research on SRL has recognised the potential of data science methods for analysis of temporal processes. The current studies, however, rely on think-aloud data and other self-report measures, which are costly and intrusive. The use of trace data is a promising alternative, though raw trace data, as captured by contemporary learning environments, is insufficient to represent selfregulation processes (aka micro-level SRL processes) as theorised in well-known models. To address these challenges, this research aims to 1) improve the measurement of SRL by deriving micro-level processes from trace data; 2) analyse these micro-level processes for temporal associations; 3) explore how such temporal associates between micro-level processes are correlated with learning strategies; and 4) assess the impact of formative data-driven feedback on these SRL processes. We undertook a preliminary study and found that certain temporal activity traits relate to performance in the summative assessments attached to the course, mediated by strategy type. In addition, more strategically minded activity, embodying learner self-regulation, generally proves to be more successful than less disciplined reactive behaviours.
Self-regulation is a key skill for strategically mature students as it informs how effectively they process feedback (both internal and external) and act upon it [1]. In addition, administering developmental feedback to students has a significant effect on their learning journey and academic performance [2]. From a metacognitive viewpoint, exponents of self-regulated learning (SRL) are able to succeed by assessing, planning, assimilating, organising, and self-evaluating in an ongoing cycle [3]. Therefore, significant benefits can be realised in identifying, articulating, and optimising patterns of SRL. Much of the research around measuring SRL, however, is based not on authentic process data, but on variants of self-report data capture e.g. [4] [5].
Data generated in blended learning environments, specifically those from learning management systems (LMS), provide opportunities for researchers to unlock insights into learner behaviours and strategies. The use of LMS trace data is a promising alternative to self-report data, as it eliminates potential issues of data objectivity and reliability. However, raw trace data cannot, in and of itself, represent self-regulation processes as theorised in well-known models. Therefore, the derivation of SRL macroprocesses and associated microprocess, as demonstrated by Siadaty et al. [6], provides a strong methodological platform on which the current study can build. 2 2.1
Winne [7] identifies three key aspects of SRL: 1) Cognitive Tactic; 2) Cognitive Strategy; 3) Metacognition. This articulates a learner’s management of their own cognitive tactics, and the development of an overarching knowledge management strategy, encompassing self-awareness. Zimmerman’s model of SRL also provides a strong and conceptually interpretable model: Self-regulation is presented as a cycle of forethought (planning), performance (of learning event), and self-reflection [8].
In the context of Learning Analytics (LA), Winne advises underpinning SRL
research with a proven SRL model. and provides a framework for mapping trace data
events to ‘inferences’ and categorising them to phases of his SRL model [9]. There are
studies that harvest pure trace data to unlock insights into cognitive tactics and learning
strategies e.g. [
Microlevel process analysis is one of the responses to the challenges of capturing and articulating SRL. Cleary consolidates various characteristics of the field into a broad definition: “…a highly specific or fine-grained form of measurement that targets behaviours or processes as they occur in real time across authentic contexts…” [13, p. 330]. Greene and Azevedo [5] present a mapping of micro-level processes e.g., goalsetting or content evaluation, to overarching macro-level processes e.g., planning or monitoring. In this context, the macro-level processes represent elements of the chosen SRL model.
Siadaty et al. [6] build on this substantially by developing a hybrid self-report/tracebased protocol of SRL microanalysis. They posit an SRL model which positions 1) Planning, 2) Engagement, and 3) Evaluation & Reflection as its macro-level processes/SRL phases. Micro-level processes, such as Task Analysis or Working on Task, are categorised to their corresponding macro-level processes. Siadaty et al.’s method was empirically validated in two studies on the self-regulatory patterns of knowledge workers in technology-enhanced environments [14] [15]. These studies provide a critical empirical SRL bedrock. They do not, however, explore inter/intra-strategy temporal differences. Additionally, although the impact of various scaffolding interventions is assessed, it does not represent a true study of feedback as a mediator of SRL. This study seeks to address this empirical gap. 2.3
Process Mining (PM) is an analytical discipline that straddles data mining, machine
learning, and business process modelling. Being data-driven but process-centric, we
can view it as a missing link between data science and process science [9]. In many PM
studies, the processes are of a tactic-level granularity. This study aims to harness PM
for micro and macro-level analysis. PM provides a vital temporal dimension that is not
afforded by traditional statistical methods e.g., [
Bannert et al. [4] use process mining techniques to analyse think-aloud data logged
from a student-group’s navigation through an LMS. Their aim is to provide a
comparison of process models of high and low performing students. Lust et al. [
Providing quality feedback in HE is inherently challenging. These challenges have intensified with the increasing massification of education. Two significant studies, both using the same high-volume LMS trace data, provide valuable insights into the impact of customised automated feedback. Pardo et al. [19] demonstrate a positive association between feedback messaging and both student satisfaction and assessment performance. Fincham et al. [18] detected tactical transition and strategic improvement (linked to assessment performance) as the result of feedback interventions. The current study aims to build on this research to assess the impact of feedback on SRL patterns in learners. 2.4
RQ1. To what extent can micro-level SRL processes be derived from trace data collected by conventional virtual learning environments? RQ2. To what extent can temporal associations between micro-level SRL processes be derived through the analysis of trace data? RQ3. What are the differences in SRL micro-level processes exhibited by students following different learning strategies? RQ4. What is the impact of conditionally administered, analytics-based formative feedback on the micro-level SRL processes of students who follow different learning strategies? 3
The trace data for this study come from two sources: The first were collected from an LMS attached to a computing course at an Australian university. The datasets provide LMS trace data from four cohorts of a course, spanning 2014 to 2017 [20]. The course was based on a flipped classroom pedagogy and the data relate to students’ engagement with the online activities as preparation for the face-to-face learning sessions. Each time a student engaged with an element of the LMS, a learning event record was generated. These events, which are collectively called trace data, provide the source for our analyses. The second will be generated from a course to run at a London university in 2019.
The starting point of PM is a dataset in the form of an event log. The required elements to run a PM algorithm are: Case, a process instance; Activity, a well-defined step in a broader process; Timestamp, providing the temporality that is key to this study. Each LMS event record contains a student ID number (which serves as our PM case), a completed study action (which serves as our PM activity), and a timestamp.
To identify micro-level SRL processes (RQ1), we will extract trace data and utilise the mapping method outlined in the Siadaty (and associated) studies i.e. trace event micro-level process macro-level process/SRL construct. To address RQ2, we will build on the PM techniques explored in our preliminary study (see section 4) and extract temporal relationships from the SRL microprocesses. To address RQ3, we will cluster students in strategy groups, using the methods employed by Bannert et al [4] and Fincham et al [18]. We will perform pair-wise comparative analyses, using appropriate PM algorithms, to articulate the differences in learner strategies from a temporal microlevel perspective. Finally, will consolidate these methods to measure the impact of feedback interventions on learner behaviours, mediated by strategy type (RQ4). 4
We undertook a preliminary study, using the 2014 cohort LMS data; this was submitted to the EC-TEL 2018 conference as a full research paper [21]. We employed the R package pMineR [22] to train process models using first order Markov chains. For this study, we did not undertake micro-level analysis, but employed a coarser, tactic-level granularity in our definition of PM activity. The focus of the study is the analysis of tactical cognitive processes in a temporal/stochastic context, and how it informs learning strategy and performance. We found that certain temporal activity traits relate to performance in the summative assessments attached to the course, mediated by strategy type. In addition, more strategically minded activity, embodying learner self-regulation, generally proves to be more successful than less disciplined reactive behaviours.
It is anticipated that this doctoral submission will be a linked collection of published journal articles. These publications will be interspersed by conference paper submissions to future Learning Analytics and Knowledge (LAK) conferences. Building on the preliminary study, we hope to expand and formalise a process mining/microprocess methodology and replicate it across the remaining LMS cohorts (2015-2017). Finally, it is hoped that data harvested from the London study will provide a means of testing the replicability and scalability of a consolidated LA methodology. 10. 11.