Explaining the Influence of Learning Design Motivational Beliefs Using Learning Analytics on Students´

Abstract

There is an increased interest in pedagogically informed learning design as it is an essential driver for learning. Exploration and understanding of how online learning environments and learning design influence students´ ability to drive their own learning process, i.e., selfregulated learning, is important as it can contribute to the improvement of online programs with a focus on student learning. While a lot of research that examines self-regulated learning behaviour focuses on assessing students´ final learning outcomes and achievement, the proposed research will examine self-regulated learning behaviour, more specifically its subprocesses, related to motivational beliefs which students employ in their online learning. In addition to the survey used to measure trait motivational beliefs, the study will leverage learning analytics to gain a fuller picture of students´ state of motivational beliefs when interacting with learning design online.

Keywords

• 1 learning analytics
• learning design
• self-regulated learning
• motivational belief
• online learning

1. Introduction

With the growth of online education offerings, there is an increased interest in pedagogically informed learning design. Since learning design is considered an essential driver for learning, exploration and understanding of how online learning environments and learning design influence students´ ability to drive their own learning process is important [ 19, 26 ]. This ability is referred to as self-regulated learning (SRL). In extension, what “drives students to drive their own learning process”, i.e., how does motivation feature in this?

Some obvious advantages of online learning are its availability and flexibility in time and space. However, online learning puts a higher demand on students' SRL ability. Poor ability to self-regulate often results in a high rate of dropouts [ 9 ].

To study SRL we can use self-report instruments, such as surveys. There are, however, some known weaknesses associated with self-report data, such as response bias. In other words, what students say they do may not correspond to what they actually do [25].

In order to get more objective measures of SRL behaviour, we can make use of additional data sources, such as capturing the digital footprints (traces) of student behaviour in a virtual learning environment (VLE) [ 8,9,16 ].

Since learning design plays an important role in structuring the pedagogical context where learning occurs (e.g., how courses are designed and delivered, available resources, learning tasks, and assessment) it also plays an important role in providing a framework for analysing and interpreting data about learners’ SRL behaviour [ 8, 9, 12, 16, 21 ].

The present project draws on several study fields, with learning analytics being one of the main data sources in this research area. To understand and to be able to examine how students self-regulate their learning online, how learning design influences this, and how to analyse changes in students’ online SRL behaviour over time, two main issues must be addressed: (1) What is meant by learning design? And (2) What aspects of SRL should we focus on in this context?

2. Background

2.1. Learning design

Current literature shows that researchers and practitioners are approaching learning design from a multitude of perspectives. There is some confusion over terms, concepts and tools within the field, and thus a lack of conceptual clarity, which makes the development of shared understanding difficult. An illustrative example is the many names used for the field itself. Some of the common ones being: “learning design”, “instructional design”, “curriculum design”, “educational design”, “design for learning” and “design-based learning” [ 5, 14 ]. Definitions also vary; for example, Conole [ 3 ] refers to learning design as a “methodology for enabling teachers/designers to make more informed decisions in how they go about designing learning activities and interventions, which is pedagogically informed and makes effective use of appropriate resources and technologies” (p. 121). She emphasises the importance of making the learning design process more explicit and sharable between practitioners/educators. While Matcha et al. [ 16 ] use the terms “instructional design” and “course design” to refer to the structure of learning topics and the corresponding learning activities or tasks. In this context, instructional design is understood as and is driven by the pedagogical approaches and the nature of the discipline. The design of a course is also influenced by the delivery modalities, i.e., when, and how teaching activities are facilitated (e.g., online, face-to-face, flipped classroom, etc.).

2.1.1. Classification

Design of

Learning

Building on this ontological and conceptual diversity, Dobozy [ 5 ] suggests classifying learning design into three types: (1) learning design as a concept, that is a standardised representation of learning sequences and design-based procedures underpinned by learning theories, e.g., cognitive constructivism, social constructivism and social learning, etc.; (2) learning design as a process which illustrates the learning intent, planning and enacting of a particular learning sequence in a context, i.e., subject-specific content; and (3) learning design as a product of the methods, tools and resources, referring to artefacts such as models, templates, and lesson plans with a description of roles and resources needed for a particular learning activity. The second and third approaches to learning design are typically used in learning analytics literature. For example, Mangaroska and Giannakos [ 15 ] refer to learning design as a process of designing effective learning experiences with the use of technological innovations and resources. While Bakharia et al. [ 2 ] chose to see learning design both as a process “of creating and adapting pedagogical ideas” and as a product “of a formalised description of a sequence of learning tasks, resources and support that a teacher constructs for students for an entire, or part of, an academic semester" (p. 330).

2.1.2. Representation of Learning

Design

There is also a niche within the literature that seeks to develop a descriptive framework to capture teaching and learning activities to enable educators to share and reuse ideas and resources [ 4 ]. Most of the learning design models and frameworks developed in the last two decades have focused on tools and representations to support this approach, as well as on mechanisms for sharing its outputs to assist educators in designing learning activities. For example, Conole [ 3 ], and Mor et al. [ 18 ] give a detailed description of learning design representation formats and patterns, which can be effectively adopted by educators/practitioners in planning and facilitating educational activities. Persico and Pozzi [ 22 ] suggest a multidimensional framework for positioning different learning design representations. Maina et al. [ 14 ] review some contemporary trends in the practices and methods of learning design with several tools and resources to support educational practice.

Back in 2012, The Larnaca Declaration on Learning Design made an attempt to provide an overarching theoretical foundation for the field [ 4 ]. It is based on a mix of different approaches, i.e., concepts and shapes, of both research and practice and uses the analogy of music notation metaphor to describe learning design. The idea of such notation is that it should contain enough information to convey “musical” ideas to share between educators. Moreover, the core concepts of learning design are captured and summarised in the Learning Design Conceptual Map (LD-CM) (Figure 1).

It starts with the main objective of “creating learning experiences aligned to particular pedagogical approaches and learning objectives”. How educators make decisions about designing for learning is determined by the Educational Philosophy, Theories and Methodologies, and Characteristics & Values of the Learning Environment. For example, Dobozy´s first type of learning design classification, i.e., concept, might be described in the first and second elements of LD – CM.

How educators plan, engage, reflect and evaluate teaching is determined by Teaching Cycle. In this case, Dobozy’s classification of learning design as a process (i.e., how educators define objectives and plan what teaching methods and strategies to apply to help students to reach the objectives) might be described in Design and Plan stage of the element. Further, her approach of learning design as a product where a set of resources for the students to access might be described in Implementation, while learning activities, i.e., tasks the learners are expected to carry out, with the engagement phase of Teaching cycle, including Level of Granularity.

Many other educational theories and practices could be analysed using the Learning Design Conceptual Map. For example, Pedagogy profile learning design has been developed as part of a Learning Design taxonomy by The Open University Learning Design Initiative (OULDI) [ 3, 26 ]. The systematic research at OU on the relationship between Pedagogy profile, student behaviour and outcomes, among other things, has led to the impact of learning design on decisionmaking and future course design. Such insights could be described and documented in the Reflection and Teaching Development phases of Teaching Cycle and Core Concepts of Learning Design of LD – CM.

Another example is Laurillard’s conversational framework which represents an interaction cycle between teacher and student where each operates at the level of learning outcome and carrying out learning and teaching activities [ 11 ]. The framework has focus on several elements of LD – CM. The framework might be applied to any level of granularity – from the whole course or curriculum to a particular learning activity; interaction with learners in both theory and practical areas of the relevant discipline might be described by engagement in Teaching Cycle, while reactions to teaching and assessment might be described both in Reflection of Teaching cycle and Learner Responses elements. The latter may suffer from at least two problems. First is bias and subjective perception, as learners´ responses are often limited to insights generated from assessments, course evaluations and surveys. Second, as insights are generated over time, it hinders educators/practitioners from making in-time interventions and providing personalised feedback to students. A potential contribution of learning analytics to learning design (captured with Learner Responses) provides an opportunity for deeper tracking of learner activity and more detailed analysis of learners´ self-regulated behaviour at all stages of teaching and learning processes. Moreover, learning analytics could help educators to reflect and compare their practices at all levels of granularity, i.e., from curriculum/ study program down to individual learning activity. However, without a representation of the detailed learning objectives and the expectations in terms of the learner's activities toward them, learning analytics is reduced to monitoring generic behaviours, such as persistence or social interactions [ 19 ].

Regardless of the approach or concept one chooses, learning design is considered to be an essential driver in how courses are designed and delivered, and what resources, learning tasks, assessments, etc., are available. Thus, learning design plays an important role as it provides a framework for analysing and interpreting data about learners´ behaviour and SRL [ 8, 9, 12, 16, 21 ].

SRL is commonly modelled as a cyclical, recursive sequence of processes and subprocesses in which learners proceed through fluctuating phases [ 20 ]. The labels given to each phase vary between researchers. Still, broadly, we are looking at three phases: i) the preparatory or forethought phase, which includes elements such as motivational beliefs, task analysis, planning, and goal setting; ii) the performance phase, which includes task work, strategy use, and monitoring of own learning; and iii) the appraisal phase, which includes performance, feedback and reflection on progress and strategies [24].

The relationship between SRL and student learning has been widely researched over the past decades, and there are substantial amounts of empirical evidence to suggest that an increased ability to self-regulate one´s own learning process is associated with a higher likelihood of academic achievement and performance [ 23, 30, 31 ]. This is particularly evident in the context of online learning, where how and with what content and activities to engage, is acknowledged to be an essential skill for the ability to succeed [ 13, 23, 26 ].

In an ongoing literature systematical review, it appears however that the majority of this research has focused on the behavioural and cognitive aspects of SRL. Considerably less attention has been given to the effective domain of the construct, hereunder motivational beliefs.

As such this is an area, which will be addressed in the current research project. 2.2.

Self-Regulated Learning

2.2.1. Motivation in SRL

Literature suggests that hybrid/ blended and online courses require students to be more selfdisciplined and self-regulated [ 8, 9, 13 ]. SRL as a construct is built up by components belonging to three areas related to the learning process: cognitive, behavioural, and affective/emotional [ 23 ]. A widely agreed upon working definition of self-regulated learning (SRL) is “an active, constructive process whereby learners set goals for their learning and then attempt to monitor, regulate, and control their cognition, motivation, and behaviour, guided and constrained by their goals and the contextual features in the environment” (p. 453) [ 23 ]. In other words, SRL such as learners monitoring and adjusting their own behaviour and actions in relation to their specific learning context.

According to Zimmerman [30] motivation in the field of education is one of the most important pillars through which we can achieve educational goals. Motivational and affective processes are intrinsic parts of this complex system of interdependently connected SRL processes. They trigger and maintain goaloriented behaviours, e.g., engagement, perseverance and ultimately performance on learning tasks by influencing the choice and implementation of learning strategies [ 13 ].

Motivation is an extensive and complex field of research. There are many motivational theories and covering these is beyond the scope of the present paper. However, in the context of SRL, Pintrich [ 23 ] and Littlejohn et al. [ 13 ] have chosen to interpret motivation as a set of “motivational beliefs”. Motivational beliefs are in turn sub-divided into task value, task interest, and self-efficacy, or belief in one´s own ability to succeed with a task.

Students can find motivation from various sources, be they internal such as values, interests, and competence beliefs, or external such as the context of the learning environment, and learner-centered instruction, and levels of motivation can fluctuate over the course of a learning task [ 1 ].

2.2.2. Motivational

Context

Beliefs and

Motivational beliefs do not appear in a vacuum. There are many internal and external conditions that have an impact on “each learner’s choices about how and what to learn … whether learning happens and what is learned” (p.2) [30]. The ability to self-regulate one's learning, including one's motivational beliefs, is mediated by both personal factors such as cognitive and affective and contextual factors such as the learning environment where learning design plays an important role [ 23 ].

Conceived as a set of learning tasks, learning design as a product [ 5 ] can be considered an important contextual factor. Consequently, how students respond to and interact with a learning design will contribute to shaping their SRL, more specifically their motivational beliefs.

2.2.3. Operationalising and

Measuring Motivational Beliefs

How do students´ motivational beliefs manifest within a learning design?

Firstly, it is important to differentiate between trait and state motivational beliefs. “Trait” implies student motivation as an intrinsic disposition, i.e., a general tendency. “State” refers to motivational beliefs as a dynamic construct that changes relative to context and situation. Measures of SRL, which includes motivational beliefs, are typically based on combinations of indicators that may be self-reported responses to items, on-task indicators of responses to individual tasks, or observations by others. However, the most used form of motivation measure continues to be trait-focused, using some form of self-report survey instrument [ 1 ].

With the growth of online learning over the recent years, learning analytics has emerged as a viable measure for SRL. Learning Analytics is defined as “the measurement, collection, analysis, and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” [ 6 ]. Learning analytics can be used to extract and analyse patterns in log data in order to understand the motivated student's behaviour [28].

As mentioned earlier, motivation as a part SRL process has received comparatively little attention, and the same applies within the learning analytics literature [ 13, 28 ].

3. Goal and research questions

The overarching goal of the PhD project is to contribute to understanding how educational context, i.e., learning design, is linked to students' motivational beliefs.

Knowledge about the association between learning design and students' motivational beliefs (as a part of SRL) can enable educators to make more informed choices when deciding on the structure and type of learning tasks in a given learning design. This in turn can help students choose appropriate and effective learning strategies, ultimately increasing their chance to succeed in their online studies.

A lot of research that examines SRL behaviour focuses on assessing students´ final learning outcomes, performance, and satisfaction. This makes sense if SRL is viewed as a trait, i.e., a global and lasting disposition. However, as mentioned earlier, SRL is a series of dynamic processes that fluctuate over time [ 20 ]. Therefore, when exploring students’ SRL behaviour as it “occurs in the learning situation”, the survey approach is less appropriate. By using learning analytics, we can collect data in a more dynamic manner in the learning process. Moreover, extracting trace data can provide us with contextual information relevant to students´ motivational beliefs at a task, or “state”, level.

In order to examine the SRL sub-processes, more specifically processes related to motivational beliefs, students employ in their online learning, the present PhD project seeks to build on the research of Littlejohn and colleagues [ 13 ]. In addition to the survey used to measure trait motivational beliefs, we will leverage learning analytics to gain a more holistic picture of students´ state of motivational beliefs when interacting with learning design online.

The research questions for this project are as follows:

RQ1a: How does observed behaviour as seen in trace data relate to motivational beliefs as measured using a survey?

RQ1b: Relative to survey data, are there systematic variations at a task/state level?

RQ2: Do different types of tasks, e.g., assessments, reflections, discussions, etc., within the learning design give rise to different types of behavioural patterns linked to motivational beliefs?

4. Methods

To answer the research questions of this PhD project, the work is divided into three parts. 1. The first part of the study will be a systematic literature review in order to understand how literature within the field of learning analytics defines and conceptualises learning design and SRL. More specifically, how is learning design defined and conceptualised, and how does this fit with the LD – CP (Figure 1); which aspects of SRL behaviour are observed and operationalised, i.e., what SRL models, domains and phases are used, and how is motivation understood and related to SRL; how does research connect learning design and SRL and what role is motivation given within this context. 2. For the second part, I will collect and analyse Canvas VLE trace data from the students enrolled in the 3 years online bachelor program “Business Economics and Management”. Thus, the second part of this study will use learning analytics to capture observable behaviour and patterns for further investigating motivational beliefs in light of the research questions RQ1a and RQ1b. 3. Following the second part of the project, the third part will use learning analytics to examine whether learning design task types influence motivational belief behaviour in light of the research question RQ2.

At the current time, I am in the process of carrying out the systematic literature review which will be conducted following the five-step methodology [ 8 ]. So far, the research questions to be answered are formulated along with the set of inclusion and exclusion criteria. The search in databases ACM DL, IEEE, ERIC, Elsevier and Web Of Sciences resulted in 25 articles. The completion of the final manuscript for publishing is planned for December 2023.

The doctoral dissertation is a part of the ELEA (Encouraging Self-regulated Learning in Higher Education) project. The project uses multiple data sources, both qualitative and quantitative, to study the dynamic and contextual patterns of students’ self-regulated learning as it develops over time in a study programme. By triangulating behavioural data with data based on self-report and in-depth interviews the aim of the project is to provide a more in-depth understanding of how selfregulated learning occurs and changes over time and relative to context.

6. References

(Eds.), Handbook of Self-Regulation: Theory, Research, and Applications (pp. 451-502). San Diego, CA: Academic Press. [24] Puustinen, M. and Pulkkinen, L. (2001).

Models of self-regulated learning: a review. Scandinavian Journal of Educational Research, 45, 3, 269–86. [25] Quick J., Motz B., Israel J., Kaetzel J. (2020). What College Students Say, and What They Do: Aligning Self-Regulated Learning Theory with Behavioral Logs. LAK '20: Proceedings of the Tenth International Conference on Learning Analytics & Knowledge, March 2020, Pages 534–543. https://doi.org/10.1145/3375462.3375516 [26] Rienties, B., & Toetenel, L. (2016). The impact of learning design on student behaviour, satisfaction and performance: A cross institutional comparison across 151 modules. Computers in Human Behavior, 60, 333–341. [27] Renée S. Jansen, Anouschka van Leeuwen, Jeroen Janssen, Rianne Conijn, Liesbeth Kester. (2020). “Supporting learners' self-regulated learning in Massive Open Online Courses.” Computers & Education,Volume 146, https://doi.org/10.1016/j.compedu.2019.1 03771. [28] Talbi, O., Ouared, A. (2022). Goaloriented student motivation in learning analytics: How can a requirements-driven approach help?. Educ Inf Technol 27, 12083–12121. https://doi.org/10.1007/s10639-02211091-8 [29] Winne, P. H (2020). Construct and consequential validity for learning analytics based on trace data. In Computers in Human Behaviour, vol. 112, nov. 2020. [30] Winne, P. H., & Baker, R. S. (2013). The Potentials of Educational Data Mining for Researching Metacognition, Motivation and Self-Regulated Learning. Journal of Educational Data Mining, 5(1), 1-8. https://doi.org/10.5281/zenodo.3554619 [31] Zimmerman, B. J. (2008). Investigating self-regulation and motivation: Historical background, methodological developments, and future prospects. American Educational Research Journal, 45, 166–183.

References

[1] Ainley , M. , & Ainley , J. ( 2019 ). Motivation and learning: Measures and methods . In K. A. Renninger & S. E. Hidi (Eds.), The Cambridge handbook of motivation and learning (pp. 665 - 688 ). Cambridge University Press.

[2] Bakharia , A. , Corrin , L. , De Barba , P. , Kennedy , G. , Gašević , D. , Mulder , R. , Lockyer , L. ( 2016 ). A conceptual framework linking learning design with learning analytics . In Proceedings of the sixth international conference on learning analytics & knowledge , 329 - 338 .

[3] Conole G. ( 2013 ). Designing for Learning in an Open World . Explorations in the Learning Sciences, Instructional Systems and Performance Technologies 4 .

[4] Dalziel , J. , Conole , G. , Wills , S. , Walker , S. , Bennett , S. , Dobozy , E. , Cameron , L. , Badilescu-Buga , E. , Bower , M. ( 2016 ). The Larnaca Declaration on Learning Design . Journal of Interactive Media in Education, (1): 7 , 1 - 24 .

[5] Dobozy , E. ( 2011 ). Typologies of learning design and the introduction of a “LD-Type 2” case example . eLearning Papers , 27 , 3 - 11 . Retrieved from https://espace.curtin.edu.au/bitstream/han dle/20.500.11937/22874/183178_183178. pdf?sequence= 2

[6] Ferguson , R. ( 2012 ). Learning analytics: drivers, developments and challenges . International Journal of Technology Enhanced Learning (IJTEL) , 4 , 5 /6, 304 - 317

[7] Gabriel , F. , Cloude , E. B. , & Azevedo , R. ( 2022 ). Using learning analytics to measure motivational and affective processes during self-regulated learning with advanced learning technologies. In Using learning analytics to measure motivational and affective processes during self-regulated learning with advanced learning technologies . Springer. https://doi.org/10.1007/978-3- 031 -06333- 6 _ 6

[8] Gašević , D. , Jovanović , J. , Pardo , A. , Dawson , S. ( 2017 ). Detecting Learning Strategies with Analytics: Links with Selfreported Measures and Academic Performance . Journal of Learning analytics , 4 ( 2 ), 113 - 128 .

[9] Gašević , D. , Dawson , S. , Rogers , T. ( 2016 ). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success . The Internet and Higher Education , 28 , 68 - 84 .

[10] Khan , K. S. , Kunz , R. , Kleijnen , J. , & Antes , G. ( 2003 ). Five steps to conducting a systematic review . Journal of the Royal Society of Medicine , 96 ( 3 ), 118 - 121 .

[11] Laurillard , D. ( 2012 ). Teaching as a Design Science: Building Pedagogical Patterns for Learning and Technology . New York: Routledge.

[12] Larmuseau , C. , Elen , J. , & Depaepe , F. ( 2018 ). The influence of students' cognitive and motivational characteristics on students' use of a 4C/ID-based online learning environment and their learning gain Proceedings of the 8th International Conference on Learning Analytics and Knowledge , Sydney, New South Wales, Australia. https://doi.org/10.1145/3170358.3170363

[13] Littlejohn , A. , Hood , N. , Milligan , C. , & Mustain , P. ( 2016 ). Learning in MOOCs: Motivations and self-regulated learning in MOOCs . The internet and higher education , 29 , 40 - 48 .

[14] Maina M. , Craft B. , and Mor Y. ( 2015 ). The Art & Science of Learning Design. Technology Enhanced Learning. SencePublichers.

[15] Mangaroska , K. and M. Giannakos ( 2019 ). Learning Analytics for Learning Design: A Systematic Literature Review of Analytics-Driven Design to Enhance Learning . Ieee Transactions on Learning Technologies Volume 12 ( 4 ), 516 - 534 .

[16] Matcha W. , Gašević D. , Uzir N.A. , Jovanović J. , Pardo A. , Lim L. , Maldonado-Mahauad J. , Gentili S. , PérezSanagustin M. , Tsai Y. ( 2020 ). Analytics of Learning Strategies: Role of Course Design and Delivery Modality . Journal of Learning Analytics Volume 7 ( 2 ), 45 - 71 .

[17] Milikić , N. , Gašević , D. , & Jovanović , J. ( 2020 ). Measuring Effects of TechnologyEnabled Mirroring Scaffolds on SelfRegulated Learning . Ieee Transactions on Learning Technologies , 13 ( 1 ), 150 - 163 .

[18] Mor , Y. , Mellar , H. , Warburton , S. , & Winters , N. (Eds.). ( 2014 ). Practical design patterns for teaching and learning with technology . Springer.

[19] Nguyen , Q. , Rienties , B. , Toetenel , L. , Ferguson , R. , & Whitelock , D. ( 2017 ). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates . Computers in Human Behavior , 76 , 703 - 714 .

[20] Panadero , E. ( 2017 ). A Review of Selfregulated Learning: Six Models and Four Directions for Research . Frontiers in Psychology, 8 ( 422 ). doi: 10 .3389/fpsyg. 2017 .00422

[21] Pardo , A. , Gašević , D. , Jovanovic , J. , Dawson , S. , & Mirriahi , N. ( 2019 ). Exploring Student Interactions With Preparation Activities in a Flipped Classroom Experience . Ieee Transactions on Learning Technologies , 12 ( 3 ), 333 - 346 . https://doi.org/10.1109/TLT. 2018 . 285879 0

[22] Persico , D. and Pozzi , F. ( 2015 ). Informing learning design with learning analytics to improve teacher inquiry . British Journal of Educational Technology , 46 , 2 , 230 - 248 .

[23] Pintrich , P. R. ( 2000 ). The Role of Goal Orientation in Self-Regulated Learning . In M. Boekaerts, P. R. Pintrich , & M. Zeidner