=Paper=
{{Paper
|id=Vol-3738/paper5
|storemode=property
|title=Mapping the Analysis of Students' Digital Footprint to Constructs of Learning (short paper)
|pdfUrl=https://ceur-ws.org/Vol-3738/paper5.pdf
|volume=Vol-3738
|authors=Kamran Mir,Geraldine Gray,Ana Schalk
|dblpUrl=https://dblp.org/rec/conf/lasispain/MirGS23
}}
==Mapping the Analysis of Students' Digital Footprint to Constructs of Learning (short paper)==
https://ceur-ws.org/Vol-3738/paper5.pdf
Mapping the Analysis of Students' Digital Footprint to
Constructs of Learning
Kamran Mir1,* Geraldine Gray1 and Ana Schalk2
1 School of Informatics and Cybersecurity, Technological University Dublin, Ireland
2 Trinity College Dublin, The University of Dublin, Ireland
Abstract
This research project explores the importance of learning theories in informing the objective
evaluation of learning practice, as evidenced by the analysis of multimodal data collected from
the eclectic mix of interactive technologies used in higher education. Frequently, learning
analytics research builds models from trace data easily collected by technology, without
considering the latent constructs of learning that data measures. Consequently, resulting models
may fit the training data well, but tend not generalize to other learning contexts. This study will
interrogate educational technology as a data collection instrument for constructs of learning, by
considering the influence of learning design on how learning constructs can be curated from
these data. Results will inform methodological guidelines for data curation and modelling in
educational contexts, leading to more generalizable models of learning that can reliably inform
how we act on data to optimize the learning context for students.
Keywords
learning analytics, learning design, mapping 1
1. Background
Learning theories offer explanations of how we learn, and so inform how we interpret
models of learning [1]. Teaching practice informed by both learning theory and real-time
information on student learning activities promises pathways to personalized and
optimized learning contexts for all students [1,2]. Arguably, this is the holy grail of higher
education. The use of ICT in Higher Education (HE) offers systematic collection of large
volumes of data in a learning context. Research in learning analytics over the last 20 years
has explored an eclectic mix of data collected by ICT environments including analysis of
images, text, audio, data from wearables, and trace data from education technology [3].
Developments in technology and its use, along with developments in analysis of educational
data (learning analytics), solve many of the technical challenges of collecting and analyzing
data systematically from learning contexts in the wild. However, the potential of this eclectic
LASI Europe 2024 DC: Doctoral Consortium of the Learning Analytics Summer Institute Europe 2024, May 29-31
2024, Jerez de la Frontera, Spain
∗ Corresponding author.
d23125156@mytudublin.ie (K. Mir); geraldine.gray@tudublin.ie (G. Gray); schalka@tcd.ie (A. Schalk)
0000-0001-7399-7807 (K. Mir); 0000-0002-9694-2558 (G. Gray); 0000-0002-2317-9117 (A. Schalk)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
mix to serve as data collection instruments for scientific evaluation of latent constructs of
learning is still unrealised [4]. Indeed, the lack of accepted research methodologies based
on data collected and curated from education technology cited by Issroff & Scanlon [5] over
twenty years ago persists today [6]. Ideally, a learning analytics methodology would start
with established learning theories to inform a hypothesis and define the latent learning
constructs of interest as a first step. This would be followed by designing a valid and reliable
data collection instrument to measure these constructs in real learning contexts as a second
step. Then, as a third step the data collected by those instruments would be analyzed to
provide insights and feedback on the learning that occurs [7]. In general, the conventional
research in learning analytics starts from the last step i.e. to provide insights and feedback
on learning [3]. Data is conveniently collected by educational technology without
considering learning theory or learning constructs. B. Motz et al. [8] have reported that trace
data from educational technology can reflect the teaching context that generated it.
However, while the validity and reliability of indicators may be established for the specific
context that generated them, findings tend not to generalize [9]. Interestingly, publications
that do concur on the generalizability of models use data from, arguably, a naturally
ambiguous source, the natural language in student text submissions [3].
The wide selection of published learning theories evidence that learning is difficult to
both define and measure, as it cannot be observed directly [10]. Theory aims to “systematize
and organize what is known about human learning” [11,12], and so seeks to explain and
predict behavior, informing both explanations and potential optimization of models of
learning. Therefore, learning theories and learning design choices should be an essential
component of any argument informed by learning analytics.
2. Research Goals and Questions
This research project explores the importance of learning design, and the learning theories
it actualizes, when informing the objective evaluation of models of learning derived from
analysis of multimodal data collected and curated from interactive technologies used in
higher education. Results will progress the state of art by informing methodological
guidelines in learning analytics to improve the generalizability of future learning analytics
models, both supervised and unsupervised.
The research question is:
In what ways does including learning design factors affect the generalizability of inferences
from learning analytics models trained on ICT data?
Based on the research question following are the research objectives:
1. To critically evaluate the state of art on generalizable inferences derived from
analyses of educational data, with a focus on inferences about latent constructs of
learning process and learning gain.
2. To engage with stakeholders across a variety of learning contexts in HE to
understand how they use ICT to enhance student learning and enact their learning
design plan.
3. To identify common learning design themes, and their associated learning theories,
with respect to how ICT is used.
4. To evaluate if models that account for learning design themes can generalize to other
teaching and learning contexts.
5. To propose methodology guidelines for valid inferences from models of learning
based on learning design choices.
3. Related Studies
Learning theory explains the psychological and cognitive mechanisms behind how
individuals acquire knowledge and skills, focusing on the underlying principles of learning
whereas the learning design, on the other hand, applies these theories to create structured
educational experiences, using insights from learning theory to inform the development of
instructional strategies and materials. They are related in that learning design
operationalizes the concepts from learning theory to enhance the effectiveness of teaching
and learning processes. Therefore, it is important to discuss the learning theories and
design related work first before moving to learning analytics.
The objective of instructional strategies is to enable learning progress. Shuell [13]
discusses meaningful learning progress through various stages, starting with the collection
of discrete facts. These facts are then organized into new frameworks, ultimately enhancing
one's conceptual strength and/or the ability to perform tasks effortlessly. Similarly, the
conceptual framework given by Entwistle & Smith [14] emphasizes the significance of both
teacher and student actions, the role of individual and collective contexts, and the
differentiation between 'target' understanding aimed at educational objectives and
'personal' comprehension based on individual perspectives. These elements collectively
impact the results of learning in educational settings.
Hassan [15] argues that to maximize the learning outcomes and to improve the
teaching strategies there is need to incorporate cognitive levels, social factors, teamwork,
and behavioral elements into integrated learning approaches. Attentiveness to learning
theories and feedback on learning strategies through analytics, can play an important role
in educational practice, but there is a need for more experimental studies to investigate how
theory-based practices are reflective in evidence and learning and digital footprints in
online learning settings [16].
Merrill [17] reported on years of analysis of instructional design theories to uncover
common prescriptive principles for designing instructional material. The five key principles
identified through this investigation are i) engaging learners in real-world problem-solving,
ii) activating existing knowledge as a basis for new knowledge, iii) demonstrating new
knowledge, iv) applying new knowledge, and v) integrating new knowledge into the
learner's world. Several instructional design theories, including Star Legacy, 4-Mat,
instructional episodes, multiple approaches to understanding, collaborative problem-
solving, constructivist learning environments, and learning by doing, are examined briefly
to showcase how they incorporate these principles. Despite diverse terminologies, these
theories share fundamentally similar principles, indicating a commonality in their
underlying approaches to learning. A quick comparison between these theories using
generative AI is shown in Table 1.
Table 1
Comparison of Instructional Design Theories [17]
Instructional Design Learner Adaptability Resource
Theory Engagement Intensity
Star Legacy High Moderate High
4-MAT High High Moderate
Instructional Episodes Moderate Moderate Low
Multiple Approaches to High High Moderate
Understanding
Collaborative Problem Solving High Moderate Moderate
Constructivist Learning Very High High Moderate
Environments
Learning by Doing Very High Low High
Flipped Classroom High Moderate High
Inquiry Based Learning Very High Low Moderate
Hernández-Leo et al. [18] presents a framework that outlines three tiers of
analytics—learning, design, and community analytics—to facilitate informed decision-
making in the context of learning design. This method emphasizes the interplay between
analytics and design, offering a systematic approach to leveraging data to improve learning
experiences. It also suggests interdisciplinary collaboration between educators, designers
and data scientists is needed to overcome the challenges of learning analytics
implementation.
4. Project Novelty
Currently, systematic collection and curation of data from educational technology has fallen
far short of what is needed for generalizable research outputs about learning. The aim of
this work is to advance our understanding of how to bridge the gap between the wealth of
data collected in HE and reliable inferences about the learning experiences of our students
that academic staff can action on. Thus, it will inform guides for academic staff on how to
interpret data analytics in the context of their own instructional design.
5. Research Methodology
The research design of this study will be mixed-method exploratory sequential design. In
this research design, qualitative data collection and analysis occurs first, followed by
quantitative data collection and analysis. We can use this design to first explore initial
questions and develop hypotheses. Then we can use the quantitative data to test or confirm
our qualitative findings [19] as illustrated in Figure 1.
Figure 1: Research Design. [19].
Data will be collected from three sources:
1. Qualitative data collection from module leaders to capture their learning design plan
and their perceived role of educational technology in that design.
2. Activity data from the educational technologies used by modules, in compliance with
data usage policies and GDPR.
3. End of term module grades for each student, to be combined with their activity data
and then anonymized for cohort level analysis.
Data will be analysed for common patterns of engagement and it’s relationship to learning
gain across modules with comparable learning strategies. This will inform if consideration
of learning strategies can improve the generalizability of learning analytics models. Module
leaders from TU Dublin, Trinity College Dublin, Dublin City University and Allama Iqbal
Open University (Pakistan) will be invited to take part.
6. Current Status of the Work
Currently, the project is in its first phase of literature review. A comprehensive literature
review will be undertaken to understand the dynamics of work related to learning theories,
learning design and learning analytics. Different learning design tools and frameworks
being developed by the researchers are under review which will help and guide in
developing the interview questionnaire to collect the qualitative data from the module
leaders as stated in the research methodology section.
Acknowledgements
This work is conducted with the financial support of the Science Foundation Ireland Centre
for Research Training in Digitally-Enhanced Reality (d-real) under Grant No. 18/CRT/6224.
References
[1] P. A. Ertmer and T. J. Newby, “Behaviorism, cognitivism, constructivism: Comparing critical
features from an instructional design perspective,” Performance improvement quarterly, vol.
6, no. 4, pp. 50–72, 1993.
[2] E. Warries, “Theory and the systematic design of instruction,” Research on instruction:
Design and effects, pp. 1–19, 1990.
[3] G. Gray and Y. Bergner, “A Practitioner’s Guide to Measurement in Learning Analytics:
Decisions, Opportunities, and Challenges,” in The Handbook of Learning Analytics, 2nd ed.,
C. In Lang, G. Siemens, A. Wise, D. Gasevic, and A. Merceron, Eds., Society of Learning
Analytics Research, 2022, pp. 20–28. doi: 10.18608/hla22.002.
[4] K. Kitto, J. Whitmer, A. Silvers, and M. Webb, “Creating data for learning analytics
ecosystems,” Society for Learning Analytics Research, Sydney, Sep. 2020.
[5] K. Issroff and E. Scanlon, “Educational technology: The influence of theory,” Journal of
Interactive Media in education, vol. 6, 2002.
[6] B. A. Motz et al., “A LAK of Direction Misalignment Between the Goals of Learning Analytics
and its Research Scholarship,” 2023.
[7] Y. Bergner, “Measurement and its Uses in Learning Analytics,” in Handbook of Learning
Analytics, C. In Lang, G. Siemens, A. Wise, and D. Gasevic, Eds., Society for Learning Analytics
Research (SoLAR), 2017, pp. 35–48. doi: 10.18608/hla17.003.
[8] B. Motz, J. Quick, N. Schroeder, J. Zook, and M. Gunkel, “The validity and utility of activity
logs as a measure of student engagement,” in Proceedings of the 9th international conference
on learning analytics & knowledge, 2019, pp. 300–309.
[9] D. Gašević, S. Dawson, T. Rogers, and D. Gasevic, “Learning analytics should not promote one
size fits all: The effects of instructional conditions in predicting academic success,” Internet
High Educ, vol. 28, pp. 68–84, 2016.
[10] O. Carlile and A. Jordan, “It works in practice but will it work in theory? The theoretical
underpinnings of pedagogy,” Emerging issues in the practice of university learning and
teaching, vol. 1, pp. 11–26, 2005.
[11] G. R. Lefrancois, Theories of human learning. Cambridge University Press, 2019.
[12] H. Thomas, What are learning theories and why are they important for learning design?
Cambridge University Press, 2020.
[13] T. J. Shuell, “Phases of Meaningful Learning,” Rev Educ Res, vol. 60, no. 4, pp. 531–547, Dec.
1990, doi: 10.3102/00346543060004531.
[14] N. Entwistle and C. Smith, “Personal understanding and target understanding: Mapping
influences on the outcomes of learning,” British Journal of Educational Psychology, vol. 72,
no. 3, pp. 321–342, Sep. 2002, doi: 10.1348/000709902320634528.
[15] O. A. B. Hassan, “Learning theories and assessment methodologies – an engineering
educational perspective,” European Journal of Engineering Education, vol. 36, no. 4, pp. 327–
339, Aug. 2011, doi: 10.1080/03043797.2011.591486.
[16] J. Wong et al., “Educational Theories and Learning Analytics: From Data to Knowledge,” in
Utilizing Learning Analytics to Support Study Success, Cham: Springer International
Publishing, 2019, pp. 3–25. doi: 10.1007/978-3-319-64792-0_1.
[17] M. D. Merrill, “First principles of instruction,” Educational Technology Research and
Development, vol. 50, no. 3, pp. 43–59, Sep. 2002, doi: 10.1007/BF02505024.
[18] D. Hernández-Leo, R. Martinez-Maldonado, A. Pardo, J. A. Muñoz-Cristóbal, and M. J.
Rodríguez-Triana, “Analytics for learning design: A layered framework and tools,” British
Journal of Educational Technology, vol. 50, no. 1, pp. 139–152, Jan. 2019, doi:
10.1111/bjet.12645.
[19] J. W. Creswell, Educational research: Planning, conducting, and evaluating quantitative and
qualitative research. pearson, 2015.