Mapping the Analysis of Students' Digital Footprint to Constructs of Learning

Abstract

This research project explores the importance of learning theories in informing the objective evaluation of learning practice, as evidenced by the analysis of multimodal data collected from the eclectic mix of interactive technologies used in higher education. Frequently, learning analytics research builds models from trace data easily collected by technology, without considering the latent constructs of learning that data measures. Consequently, resulting models may fit the training data well, but tend not generalize to other learning contexts. This study will interrogate educational technology as a data collection instrument for constructs of learning, by considering the influence of learning design on how learning constructs can be curated from these data. Results will inform methodological guidelines for data curation and modelling in educational contexts, leading to more generalizable models of learning that can reliably inform how we act on data to optimize the learning context for students.

Keywords

• learning analytics
• learning design
• mapping 1

1. Background

Learning theories offer explanations of how we learn, and so inform how we interpret models of learning [ 1 ]. Teaching practice informed by both learning theory and real-time information on student learning activities promises pathways to personalized and optimized learning contexts for all students [ 1,2 ]. Arguably, this is the holy grail of higher education. The use of ICT in Higher Education (HE) offers systematic collection of large volumes of data in a learning context. Research in learning analytics over the last 20 years has explored an eclectic mix of data collected by ICT environments including analysis of images, text, audio, data from wearables, and trace data from education technology [ 3 ]. Developments in technology and its use, along with developments in analysis of educational data (learning analytics), solve many of the technical challenges of collecting and analyzing data systematically from learning contexts in the wild. However, the potential of this eclectic mix to serve as data collection instruments for scientific evaluation of latent constructs of learning is still unrealised [ 4 ]. Indeed, the lack of accepted research methodologies based on data collected and curated from education technology cited by Issroff & Scanlon [ 5 ] over twenty years ago persists today [ 6 ]. Ideally, a learning analytics methodology would start with established learning theories to inform a hypothesis and define the latent learning constructs of interest as a first step. This would be followed by designing a valid and reliable data collection instrument to measure these constructs in real learning contexts as a second step. Then, as a third step the data collected by those instruments would be analyzed to provide insights and feedback on the learning that occurs [ 7 ]. In general, the conventional research in learning analytics starts from the last step i.e. to provide insights and feedback on learning [ 3 ]. Data is conveniently collected by educational technology without considering learning theory or learning constructs. B. Motz et al. [ 8 ] have reported that trace data from educational technology can reflect the teaching context that generated it. However, while the validity and reliability of indicators may be established for the specific context that generated them, findings tend not to generalize [ 9 ]. Interestingly, publications that do concur on the generalizability of models use data from, arguably, a naturally ambiguous source, the natural language in student text submissions [ 3 ].

The wide selection of published learning theories evidence that learning is difficult to both define and measure, as it cannot be observed directly [ 10 ]. Theory aims to “systematize and organize what is known about human learning” [ 11,12 ], and so seeks to explain and predict behavior, informing both explanations and potential optimization of models of learning. Therefore, learning theories and learning design choices should be an essential component of any argument informed by learning analytics.

2. Research Goals and Questions

This research project explores the importance of learning design, and the learning theories it actualizes, when informing the objective evaluation of models of learning derived from analysis of multimodal data collected and curated from interactive technologies used in higher education. Results will progress the state of art by informing methodological guidelines in learning analytics to improve the generalizability of future learning analytics models, both supervised and unsupervised.

The research question is: In what ways does including learning design factors affect the generalizability of inferences from learning analytics models trained on ICT data? Based on the research question following are the research objectives: 1. To critically evaluate the state of art on generalizable inferences derived from analyses of educational data, with a focus on inferences about latent constructs of learning process and learning gain. 2. To engage with stakeholders across a variety of learning contexts in HE to understand how they use ICT to enhance student learning and enact their learning design plan. 3. To identify common learning design themes, and their associated learning theories, with respect to how ICT is used. 4. To evaluate if models that account for learning design themes can generalize to other teaching and learning contexts. 5. To propose methodology guidelines for valid inferences from models of learning based on learning design choices.

3. Related Studies

Learning theory explains the psychological and cognitive mechanisms behind how individuals acquire knowledge and skills, focusing on the underlying principles of learning whereas the learning design, on the other hand, applies these theories to create structured educational experiences, using insights from learning theory to inform the development of instructional strategies and materials. They are related in that learning design operationalizes the concepts from learning theory to enhance the effectiveness of teaching and learning processes. Therefore, it is important to discuss the learning theories and design related work first before moving to learning analytics.

The objective of instructional strategies is to enable learning progress. Shuell [ 13 ] discusses meaningful learning progress through various stages, starting with the collection of discrete facts. These facts are then organized into new frameworks, ultimately enhancing one's conceptual strength and/or the ability to perform tasks effortlessly. Similarly, the conceptual framework given by Entwistle & Smith [ 14 ] emphasizes the significance of both teacher and student actions, the role of individual and collective contexts, and the differentiation between 'target' understanding aimed at educational objectives and 'personal' comprehension based on individual perspectives. These elements collectively impact the results of learning in educational settings.

Hassan [ 15 ] argues that to maximize the learning outcomes and to improve the teaching strategies there is need to incorporate cognitive levels, social factors, teamwork, and behavioral elements into integrated learning approaches. Attentiveness to learning theories and feedback on learning strategies through analytics, can play an important role in educational practice, but there is a need for more experimental studies to investigate how theory-based practices are reflective in evidence and learning and digital footprints in online learning settings [ 16 ].

Merrill [17] reported on years of analysis of instructional design theories to uncover common prescriptive principles for designing instructional material. The five key principles identified through this investigation are i) engaging learners in real-world problem-solving, ii) activating existing knowledge as a basis for new knowledge, iii) demonstrating new knowledge, iv) applying new knowledge, and v) integrating new knowledge into the learner's world. Several instructional design theories, including Star Legacy, 4-Mat, instructional episodes, multiple approaches to understanding, collaborative problemsolving, constructivist learning environments, and learning by doing, are examined briefly to showcase how they incorporate these principles. Despite diverse terminologies, these theories share fundamentally similar principles, indicating a commonality in their underlying approaches to learning. A quick comparison between these theories using generative AI is shown in Table 1.

Hernández-Leo et al. [18] presents a framework that outlines three tiers of analytics—learning, design, and community analytics—to facilitate informed decisionmaking in the context of learning design. This method emphasizes the interplay between analytics and design, offering a systematic approach to leveraging data to improve learning experiences. It also suggests interdisciplinary collaboration between educators, designers and data scientists is needed to overcome the challenges of learning analytics implementation.

4. Project Novelty

Currently, systematic collection and curation of data from educational technology has fallen far short of what is needed for generalizable research outputs about learning. The aim of this work is to advance our understanding of how to bridge the gap between the wealth of data collected in HE and reliable inferences about the learning experiences of our students that academic staff can action on. Thus, it will inform guides for academic staff on how to interpret data analytics in the context of their own instructional design.

5. Research Methodology

The research design of this study will be mixed-method exploratory sequential design. In this research design, qualitative data collection and analysis occurs first, followed by quantitative data collection and analysis. We can use this design to first explore initial questions and develop hypotheses. Then we can use the quantitative data to test or confirm our qualitative findings [19] as illustrated in Figure 1.

Data will be collected from three sources: 1. Qualitative data collection from module leaders to capture their learning design plan and their perceived role of educational technology in that design. 2. Activity data from the educational technologies used by modules, in compliance with data usage policies and GDPR. 3. End of term module grades for each student, to be combined with their activity data and then anonymized for cohort level analysis.

Data will be analysed for common patterns of engagement and it’s relationship to learning gain across modules with comparable learning strategies. This will inform if consideration of learning strategies can improve the generalizability of learning analytics models. Module leaders from TU Dublin, Trinity College Dublin, Dublin City University and Allama Iqbal Open University (Pakistan) will be invited to take part.

6. Current Status of the Work

Currently, the project is in its first phase of literature review. A comprehensive literature review will be undertaken to understand the dynamics of work related to learning theories, learning design and learning analytics. Different learning design tools and frameworks being developed by the researchers are under review which will help and guide in developing the interview questionnaire to collect the qualitative data from the module leaders as stated in the research methodology section.

Acknowledgements

This work is conducted with the financial support of the Science Foundation Ireland Centre for Research Training in Digitally-Enhanced Reality (d-real) under Grant No. 18/CRT/6224. [17] M. D. Merrill, “First principles of instruction,” Educational Technology Research and

Development, vol. 50, no. 3, pp. 43–59, Sep. 2002, doi: 10.1007/BF02505024. [18] D. Hernández-Leo, R. Martinez-Maldonado, A. Pardo, J. A. Muñoz-Cristóbal, and M. J.

Rodríguez-Triana, “Analytics for learning design: A layered framework and tools,” British Journal of Educational Technology, vol. 50, no. 1, pp. 139–152, Jan. 2019, doi: 10.1111/bjet.12645. [19] J. W. Creswell, Educational research: Planning, conducting, and evaluating quantitative and qualitative research. pearson, 2015.

References

[1] P. A. Ertmer and T. J. Newby , “ Behaviorism, cognitivism, constructivism: Comparing critical features from an instructional design perspective,” Performance improvement quarterly , vol. 6 , no. 4 , pp. 50 - 72 , 1993 .

[2] E. Warries , “ Theory and the systematic design of instruction ,” Research on instruction: Design and effects , pp. 1 - 19 , 1990 .

[3] G. Gray and Y. Bergner , “ A Practitioner's Guide to Measurement in Learning Analytics: Decisions, Opportunities, and Challenges,” in The Handbook of Learning Analytics , 2nd ed., C. In Lang, G. Siemens, A. Wise , D. Gasevic , and A . Merceron, Eds., Society of Learning Analytics Research , 2022 , pp. 20 - 28 . doi: 10 .18608/hla22. 002 .

[4] K. Kitto , J. Whitmer , A. Silvers , and M. Webb , “ Creating data for learning analytics ecosystems,” Society for Learning Analytics Research , Sydney, Sep. 2020 .

[5] K. Issroff and E. Scanlon, “ Educational technology: The influence of theory,” Journal of Interactive Media in education , vol. 6 , 2002 .

[6] B. A. Motz et al., “A LAK of Direction Misalignment Between the Goals of Learning Analytics and its Research Scholarship,” 2023 .

[7] Y. Bergner , “ Measurement and its Uses in Learning Analytics,” in Handbook of Learning Analytics, C . In Lang, G. Siemens, A. Wise , and D. Gasevic, Eds., Society for Learning Analytics Research (SoLAR) , 2017 , pp. 35 - 48 . doi: 10 .18608/hla17. 003 .

[8] B. Motz , J. Quick , N. Schroeder , J. Zook , and M. Gunkel , “ The validity and utility of activity logs as a measure of student engagement ,” in Proceedings of the 9th international conference on learning analytics & knowledge , 2019 , pp. 300 - 309 .

[9] D. Gašević , S. Dawson , T. Rogers , and D. Gasevic , “ Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success,” Internet High Educ , vol. 28 , pp. 68 - 84 , 2016 .

[10] O. Carlile and A. Jordan , “ It works in practice but will it work in theory? The theoretical underpinnings of pedagogy,” Emerging issues in the practice of university learning and teaching , vol. 1 , pp. 11 - 26 , 2005 .

[11] G. R. Lefrancois , Theories of human learning . Cambridge University Press, 2019 .

[12] H. Thomas , What are learning theories and why are they important for learning design ? Cambridge University Press, 2020 .

[13] T. J. Shuell , “Phases of Meaningful Learning, ” Rev Educ Res , vol. 60 , no. 4 , pp. 531 - 547 , Dec. 1990 , doi: 10.3102/00346543060004531.

[14] N. Entwistle and C. Smith , “ Personal understanding and target understanding: Mapping influences on the outcomes of learning,” British Journal of Educational Psychology , vol. 72 , no. 3 , pp. 321 - 342 , Sep. 2002 , doi: 10.1348/000709902320634528.

[15] O. A. B. Hassan , “ Learning theories and assessment methodologies - an engineering educational perspective ,” European Journal of Engineering Education , vol. 36 , no. 4 , pp. 327 - 339 , Aug. 2011 , doi: 10.1080/03043797. 2011 . 591486 .

[16] J. Wong et al., “Educational Theories and Learning Analytics: From Data to Knowledge,” in Utilizing Learning Analytics to Support Study Success , Cham: Springer International Publishing, 2019 , pp. 3 - 25 . doi: 10 .1007/978-3- 319 -64792- 0 _ 1 .