=Paper=
{{Paper
|id=Vol-3024/paper4
|storemode=property
|title=Adaptable smart learning environments supported by multimodal learning analytics
|pdfUrl=https://ceur-ws.org/Vol-3024/paper4.pdf
|volume=Vol-3024
|authors=Sergio Serrano-Iglesias,Daniel Spikol,Miguel L. Bote-Lorenzo,Hamza Ouhaichi,Eduardo Gómez-Sánchez,Bahtijar Vogel
}}
==Adaptable smart learning environments supported by multimodal learning analytics==
https://ceur-ws.org/Vol-3024/paper4.pdf
Adaptable smart learning environments supported by
multimodal learning analytics
Sergio Serrano-Iglesias1 , Daniel Spikol2 , Miguel L. Bote-Lorenzo1 , Hamza Ouhaichi3 ,
Eduardo Gómez-Sánchez1 and Bahtijar Vogel3
1
GSIC-EMIC Research Group, Universidad de Valladolid, Spain
2
Departments of Computer Science and Science Education, University of Copenhagen, Denmark
3
Department of Computer Science and Media Technology, Malmö University, Sweden
Abstract
Smart Learning Environments and Learning Analytics hold promise of providing personalized support to
learners according to their individual needs and context. This support can be achieved by collecting and
analyzing data from the different learning tools and systems that are involved in the learning experience.
This paper presents a first exploration of requirements and considerations for the integration of two
systems: MBOX, a Multimodal Learning Analytics system for the physical space (human behavior and
learning context), and SCARLETT, an SLE for the support during across-spaces learning situations
combining different learning systems. This integration will enable the SLE to have access to a new and
wide range of information, notably students’ behavior and social interactions in the physical learning
context (e.g. classroom). The integration of multimodal data with the data coming from the digital
learning environments will result in a more holistic system, therefore producing learning analytics that
trigger personalized feedback and learning resources. Such integration and support is illustrated with a
learning scenario that helps to discuss how these analytics can be derived and used for the intervention
by the SLE.
Keywords
smart learning environments, multimodal learning analytics, learning design, across spaces
1. Introduction
Smart Learning Environments (SLEs) hold promise on providing personalized support to learners,
based on their individual needs and context, in order to achieve an effective, efficient and
engaging learning experience [1, 2]. Thanks to the amount of data exposed by learning systems
and tools, such as Learning Management Systems (LMSs), mobile devices and digital tools, SLEs
are able to sense learners’ actions, to analyze and characterize their progression, and to react and
intervene accordingly [3]. In this context, SCARLETT (Smart Context-Aware Recommendation
of Learning Extensions in ubiquiTous seTtings) was proposed as an SLE designed to provide
support to learners during the enactment of across-spaces learning situations, with a special
LA4SLE @ EC-TEL 2021: Learning Analytics for Smart Learning Environments, September 21, 2021, Bolzano, Italy
" sergio@gsic.uva.es (S. Serrano-Iglesias); ds@di.ku.sk (D. Spikol); migbot@tel.uva.es (M. L. Bote-Lorenzo);
hamza.ouhaichi@mau.se (H. Ouhaichi); edugom@tel.uva.es (E. Gómez-Sánchez); bahtijar.vogel@mau.se (B. Vogel)
0000-0002-3110-1096 (S. Serrano-Iglesias); 0000-0001-9454-0793 (D. Spikol); 0000-0002-8825-0412
(M. L. Bote-Lorenzo); 0000-0002-9278-8063 (H. Ouhaichi); 0000-0003-0062-916X (E. Gómez-Sánchez);
0000-0001-6708-5983 (B. Vogel)
© 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
focus on connecting formal and informal learning experiences [4]. SCARLETT is capable of
interacting with third-party tools to collect and combine data about the actions learners perform
in them, in order to keep track of their progression through the different activities. As a result,
with this information SCARLETT can determine appropriate resources and feedback to offer.
However, the data obtained from the interaction with digital devices provides only a partial
picture of the way learners interact during the activities, specially in the physical space. For
example, when a group of students work physically together in a laboratory with one device,
computer logs record only the interactions of the group with the device but not the (verbal
and gestural) interaction between the members of the group. To overcome this limitation, the
efforts from the MultiModal Learning Analytics should be considered.
Multimodal Learning Analytics (MMLA) combines the power of affordable sensor technologies
and advances in machine learning to observe and analyse learning activities from the perspective
of different data sources and modalities [5, 6]. This technology acts as a virtual observer
and analyst of learning activities across multiple contexts between stakeholders, devices, and
resources. Current research explores topics like real-time and automatic video and audio analysis
can support learning by automating the analysis of these activities through the development of
new tools and methods [7, 8]. One the challenges for MMLA has been the use complex systems,
designed for specific controlled settings, in authentic learning settings by non-expert users.
Multimodal Box (MBOX) has taken a different approach away from specialized tools to an
Internet of Things (IoT) concept that allows different sensors to be utilized across spaces [9]. The
aim with MBOX is to provide a smaller footprint for sensors that can be adapted to real-world
learning settings and work together with SLEs like SCARLETT .
In this paper we explore how the adoption of MMLA in SLEs may help in the personalization
of the support offered to learners. This research is performed by means of the integration of
MBOX in SCARLETT. The rest of the paper is structured as follows. Section 2 presents related
work about the adoption of Learning Analytics in SLEs and MMLA. Section 3 describes in more
detail the systems involved. Section 4 presents the learning scenario supported by both systems,
with a technical description of the interactions performed. Section 5 presents the conclusions
of this initial research.
2. Related work
2.1. Learning Analytics in SLEs
Learning Analytics play a significant role towards the personalized support of learners in
SLEs [3]. Based on the different indicators generated by Learning Analytics processes from
the traces of the involved resources, SLEs can make decisions on the appropriate interventions
to perform. Most of the literature about LA in SLEs relate to the traces of learners in the
virtual space. Seanosky et al. [10] developed SCALE to keep track of the progression of learners,
based on the detection of patterns on the learners’ actions. El-Bishouty et al. [11] propose
different mechanisms for the generation of a set of analytics based on learners’ actions in
an LMS that can be assessed with automatic feedback. Still, there are some works that focus
on the physical space, specially in the use of biometric sensors. Dafoulas et al. [12] make
use of a set of biometric sensors, such as heartbeat, emotion detection, sweat levels, voice
fluctuations and voice recognition, to assess learners’ contribution in different collaborative
activities. Nevertheless, learning situations that span multiple activities and spaces, and thus
demand the combination of data from physical and digital traces, still remain a challenge. Under
this premise, SCARLETT builds a learner model from the combination of analytics from the
different activities and resources involved in the learning situation.
2.2. MultiModal Learning Analytics
Over the last several years, Multimodal Learning Analytics (MMLA) has slowly contributed to
supporting learning. RAP [6], a low-cost system for tracking and collecting students’ actions
(voice volume, gaze, and posture) for delivering feedback summary on students’ performance
with multimodal data. Learners’ generated data, including click-streams [13], and sensor
data [14], were used to provide visual representations to improve learning and teaching expe-
riences. However, though these and other research studies have illustrated the benefits from
collecting and analyzing multimodal data in learning situations, their importance is not yet
widely perceived and these systems are absent from almost every real educational settings [15].
3. System description
3.1. SCARLETT: Smart Context-Aware Recommendation of Learning
Extensions in ubiquiTous seTtings
SCARLETT is an SLE designed to facilitate the management and coordination of multiple learn-
ing environments across spaces in order to deploy personalized learning recommendations [4].
Thanks to its adaptor-based architecture, SCARLETT interacts with different learning tools and
systems (Moodle, Canvas, Google Docs, etc.) during the enactment of the learning situations to
facilitate the support for learners. More specifically, SCARLETT covers: (i) the data collection
from the learners’ actions across the involved environments (sense); (ii) the incorporation of
this information into a learner and context model, which represent their current learning state
and conditions (analyze), and (iii) the evaluation of these models to trigger the deployment and
recommendation of learning tasks and resources under the proper conditions (react).
The coordination of these operations is achieved by means of a learning design. In the learning
design, teachers and instructors define the activities learners are expected to participate in, along
with the related resources, objectives and topics. This information helps SCARLETT not only
during the communication with the external tools, but also: (i) to make sense of the analytics
that make up the learner model, by connecting them with the goals and topics discussed; and
(ii), to look for appropriate resources to be presented to learners.
3.2. The Multimodal BOX: MBOX
MBOX is a lightweight toolbox for collecting human interactions for collaborative learning
scenarios [9]. MBOX utilizes a multilayered architecture taking advantage of the edge-fog-
cloud pattern [16]. The multilayered architecture has two main advantages: 1) being scalable
to collect data from supplementary physical and digital data sources, and 2) benefiting from
powerful computational resources when needed. This approach supports system adaptation
for different learning environments and enables a better scaling of computational resources
for diverse learning contexts. MBOX is designed to provide different sensors to detect and
reason about human behavior interactions in learning contexts. The toolbox approach allows
different sensors to be deployed and quickly shifted to support different learning situations.
MBOX focuses on collecting and analyzing social interaction signals from computer vision,
audio processing, biometric sensors, and other sensors.
The edge part is composed of a computational unit single board computers and powerful
microcontrollers). Student groups face a sensing interface (SI) consisting of sensors, such as a
wide-angle camera and microphone array. The SI captures and processes the signals at the Edge
layer, and the data is stored in a time-series database. Additionally, multimodal interaction data
is delivered to the Fog layer to process the data from a broader perspective further and provide
insights on the classroom or at the school level. MBOX has a Feedback and Dashboard Interface
(FDI) which is designed to interface with SCARLLET. The application provides visualization
tools in different layers (cloud and fog), and it coordinates data processing in real-time. Despite
its capabilities for sensing and analizing multimodal data, MBOX is not aimed at tracking
students through multiple learning activities and spaces, nor at providing personalized student
support, by integrating with SCARLETT we can expand to support richer learning experiences.
4. Adoption of MMLA in SLEs
This section describes the concerns to be considered in the integration between MBOX and
SCARLETT for the provision of personalized feedback and recommendations during an across-
spaces learning situation, with major emphasis on the actions performed in the physical space.
4.1. Description of the learning scenario
The currently planned learning scenario is designed for middle school students (grades 8-10), in
which they will explore STEAM subjects at a university lab, as part of a general outreach program.
The program is designed as a set of collected modules that explore real-world science problems
(e.g. climate change & food production and pollution & food waste). The scenarios provide
students with science topics that they can relate to from the different STEAM perspectives:
science, computational thinking, and design and innovation. Each module involves lab work,
teamwork, and group discussions with physical materials, and the production of digital artifacts,
that include online documents, program code, data-logging from sensors and video. The
pedagogical framing is inspired from inquiry-science learning and design processes that provides
a structure for students to investigate.
Prior to the first session of the program, students are granted access to a LMS with resources,
with videos and documents, that describe the activities to perform in the lab. Students are
encouraged to consume these resources as a preparation for the session. As well, within the
LMS they can answer a questionnaire related with basic concepts to be covered during the
following sessions and for reporting topics of interest. In relation with the work performed
in the laboratory, during the first two sessions students work in group and use the available
equipment to collect data, update electronic journals and submit findings from the experiments
Physical Space Virtual Space
Electronic
Laboratory Classroom LMS
Journal
Students Sensing devices Sensing devices
and interfaces and interfaces
API API
MBOX Data exchange SCARLETT
Data processing Sense
Feedback and Cloud
Data traces React
and model
Dashboard Processed
Fog Analyze
analy�cs
Edge
Figure 1: Interaction between MBOX and SCARLETT in the sample scenario
performed. The documents generated from these sessions have to be submitted in the LMS. In
the third session, each group has to discuss the findings obtained and present a final report.
4.2. Integration of the systems
In the present learning situation, both SCARLETT and MBOX provide support in both the
physical and virtual space, collecting data from different sources, analyzing it holistically and
providing feedback. Figure 1 illustrates the expected interaction among both systems. With the
data obtained from each system and environment learners are interacting in, covering data logs
from the tools and recordings of the sessions in classroom and in the laboratory, both SCARLETT
and MBOX can provide partial support on its own. SCARLETT can analyze the answers to
the questionnaire to recommend additional videos to students that need reinforcement based
on learners’ prior knowledge. Similarly, MBOX can detect the level of participation of each
member of the group, the groups, and notify teachers when a specific group is not participating
or discussing during the session. However, the exchange of data and analytics produced in each
system can broaden the opportunities for support and intervention.
This exchange is beneficial in both directions. SCARLETT provides logs from the actions
of learners within the LMS and the student model of each participant. This model contains
different analytics like expertise, interest or grade of participation, derived not only from the
responses to the questionnaire, but from the history of students’ actions with any of the available
resources. The combination of these analytics with discourse analysis from audio recordings
helps MBOX to detect unproductive discussions, based on the main keywords, or distinguish
non-participating students in the discussion due to lack of knowledge. On the other hand, the
discourse analytics reported by MBOX can be used to update the learner model with information
related with struggling students or disengagement, that could trigger further intervertions by
SCARLETT.
At this moment, the interaction among SCARLETT and MBOX is conceived through the
exchange of datasets. Both MBOX and SCARLETT should expose the data obtained from the
supported sensors and platforms and the resulting analytics and make the required changes in
their architectures to integrate such data. Prior to the consumption of the data, data format
and semantics need to be agreed. At least, elements like the type of action (what), the actor
performing it (who) and the environment or space where the action takes place (where) need to
be exchanged. This information will facilitate the integration of this data in the student model
or its use for the provision of feedback and recommendations. From a technical perspective, this
exchange should happen through an API. However, one concern that requires further discussion
is whether data should flow after a push (a server sends the data to subscribed clients) or a pull
(clients asynchronously request data to a server). Although it is reasonable that either MBOX
or SCARLETT periodically requests for any new available data, specially for the provision of
feedback once the activity has been completed, it can delay real-time interventions. This concern
should may be critical for the provision of support during laboratory sessions, where learners
should receive this feedback while the proper activity is taking place. The idea with integration
of MBOX and SCARLETT is to promote a more robust and sustainable MMLA system that
would adapt and continuously support different learning scenarios in a more flexible manner.
5. Conclusions and Future Work
This paper represents a first exploration of the requirements and considerations to be considered
in the adoption of MMLA in SLE from the perspective of MBOX and SCARLETT systems. The
availability of a broader dataset related with the students’ actions, not only from their interaction
with digital resources, but also of their behavior in the physical space raises an opportunity
to offer more meaningful support to learners. However, the exchange of data between the
involved systems raises different concerns, related to architectural changes, the description
of the semantics of the data exposed and the type and timing of support provided to learners,
specially when states a real-time interaction with learners. These aspects raise additional
requirements on how systems receive and consume the data. Future work will continue this
collaboration and further develop the technical integration of the systems and explore the type
of support offered in controlled settings.
Acknoledgements
This research was funded by the Spanish State Research Agency (AEI) and the European Regional
Development Fund (TIN2017-85179-C3-2-R, PID2020-112584RB-C32); and by the European
Regional Development Fund and the Regional Government of Castilla y León (VA257P18)
References
[1] J. M. Spector, Conceptualizing the emerging field of smart learning environments, Smart
learning environments 1 (2014) 1–10.
[2] G.-J. Hwang, Definition, framework and research issues of smart learning environments-a
context-aware ubiquitous learning perspective, Smart Learning Environments 1 (2014) 4.
[3] B. Tabuenca, S. Serrano-Iglesias, A. Carruana-Martin, C. Villa-Torrano, Y. A. Dimitriadis,
J. I. Asensio-Perez, C. Alario-Hoyos, E. Gomez-Sanchez, M. L. Bote-Lorenzo, A. Martinez-
Mones, C. D. Kloos, Affordances and core functions of smart learning environments: A
systematic literature review, IEEE Transactions on Learning Technologies (2021) 1–1.
[4] S. Serrano-Iglesias, E. Gómez-Sánchez, M. L. Bote-Lorenzo, J. I. Asensio-Pérez, A. Ruiz-
Calleja, G. Vega-Gorgojo, Demonstration of SCARLETT: a Smart Learning Environment
to Support Learners Across Formal and Informal Contexts, in: Proceedings of the 16th Eu-
ropean Conference on Technology Enhanced Learning, Springer International Publishing,
2021, pp. 1–5.
[5] P. Blikstein, M. Worsley, Multimodal learning analytics and education data mining: Using
computational technologies to measure complex learning tasks, Journal of Learning
Analytics 3 (2016) 220–238.
[6] X. Ochoa, F. Domínguez, B. Guamán, R. Maya, G. Falcones, J. Castells, The RAP system, in:
Proceedings of the 8th International Conference on Learning Analytics and Knowledge,
ACM, 2018.
[7] M. C. E. Chan, X. Ochoa, D. Clarke, Multimodal learning analytics in a laboratory classroom,
in: Machine learning paradigms, Springer, 2020, pp. 131–156.
[8] P. Chejara, L. P. Prieto, A. Ruiz-Calleja, M. J. Rodríguez-Triana, S. K. Shankar, R. Kasepalu,
Quantifying collaboration quality in face-to-face classroom settings using mmla, in:
International Conference on Collaboration Technologies and Social Computing, Springer,
2020, pp. 159–166.
[9] H. Ouhaichi, D. Spikol, B. Vogel, Mbox: Designing a flexible iot multimodla learning
analytics system, in: 2021 IEEE 21th International Conference on Advanced Learning
Technologies (ICALT), 2021, pp. 122–126.
[10] J. Seanosky, D. Boulanger, V. Kumar, Kinshuk, Unfolding learning analytics for big data,
in: Emerging Issues in Smart Learning, Springer Berlin Heidelberg, 2014, pp. 377–384.
[11] M. M. El-Bishouty, T.-W. Chang, R. Lima, M. B. Thaha, Kinshuk, S. Graf, Analyzing learner
characteristics and courses based on cognitive abilities, learning styles, and context, in:
Smart Learning Environments, Springer Berlin Heidelberg, 2014, pp. 3–25.
[12] G. Dafoulas, C. C. Maia, J. Samuels-Clarke, A. Ali, J. C. Augusto, Investigating the role of
biometrics in education–the use of sensor data in collaborative learning, in: IADIS Int.
Conf. e-Learning 2018, IADIS, Madrid, Spain, 2018.
[13] G. Wang, X. Zhang, S. Tang, H. Zheng, B. Y. Zhao, Unsupervised clickstream clustering
for user behavior analysis, in: Proceedings of the 2016 CHI Conference on Human Factors
in Computing Systems, ACM, 2016.
[14] L. Prieto, K. Sharma, Ł. Kidzinski, M. Rodríguez-Triana, P. Dillenbourg, Multimodal
teaching analytics: Automated extraction of orchestration graphs from wearable sensor
data, Journal of Computer Assisted Learning 34 (2018) 193–203.
[15] R. Wampfler, S. Klingler, B. Solenthaler, V. Schinazi, M. Gross, Affective state prediction in
a mobile setting using wearable biometric sensors and stylus (2019).
[16] K. Portelli, C. Anagnostopoulos, Leveraging edge computing through collaborative ma-
chine learning, in: 2017 5th International Conference on Future Internet of Things and
Cloud Workshops (FiCloudW), IEEE, 2017.