Smart Learning Environments and Learning Analytics hold promise of providing personalized support to learners according to their individual needs and context. This support can be achieved by collecting and analyzing data from the diferent learning tools and systems that are involved in the learning experience. This paper presents a first exploration of requirements and considerations for the integration of two systems: MBOX, a Multimodal Learning Analytics system for the physical space (human behavior and learning context), and SCARLETT, an SLE for the support during across-spaces learning situations combining diferent learning systems. This integration will enable the SLE to have access to a new and wide range of information, notably students' behavior and social interactions in the physical learning context (e.g. classroom). The integration of multimodal data with the data coming from the digital learning environments will result in a more holistic system, therefore producing learning analytics that trigger personalized feedback and learning resources. Such integration and support is illustrated with a learning scenario that helps to discuss how these analytics can be derived and used for the intervention by the SLE.
Smart Learning Environments (SLEs) hold promise on providing personalized support to learners,
based on their individual needs and context, in order to achieve an efective, eficient and
engaging learning experience [
However, the data obtained from the interaction with digital devices provides only a partial picture of the way learners interact during the activities, specially in the physical space. For example, when a group of students work physically together in a laboratory with one device, computer logs record only the interactions of the group with the device but not the (verbal and gestural) interaction between the members of the group. To overcome this limitation, the eforts from the MultiModal Learning Analytics should be considered.
Multimodal Learning Analytics (MMLA) combines the power of afordable sensor technologies and advances in machine learning to observe and analyse learning activities from the perspective of diferent data sources and modalities [ 5, 6]. This technology acts as a virtual observer and analyst of learning activities across multiple contexts between stakeholders, devices, and resources. Current research explores topics like real-time and automatic video and audio analysis can support learning by automating the analysis of these activities through the development of new tools and methods [7, 8]. One the challenges for MMLA has been the use complex systems, designed for specific controlled settings, in authentic learning settings by non-expert users. Multimodal Box (MBOX) has taken a diferent approach away from specialized tools to an Internet of Things (IoT) concept that allows diferent sensors to be utilized across spaces [ 9]. The aim with MBOX is to provide a smaller footprint for sensors that can be adapted to real-world learning settings and work together with SLEs like SCARLETT .
In this paper we explore how the adoption of MMLA in SLEs may help in the personalization of the support ofered to learners. This research is performed by means of the integration of MBOX in SCARLETT. The rest of the paper is structured as follows. Section 2 presents related work about the adoption of Learning Analytics in SLEs and MMLA. Section 3 describes in more detail the systems involved. Section 4 presents the learning scenario supported by both systems, with a technical description of the interactions performed. Section 5 presents the conclusions of this initial research.
Learning Analytics play a significant role towards the personalized support of learners in SLEs [3]. Based on the diferent indicators generated by Learning Analytics processes from the traces of the involved resources, SLEs can make decisions on the appropriate interventions to perform. Most of the literature about LA in SLEs relate to the traces of learners in the virtual space. Seanosky et al. [10] developed SCALE to keep track of the progression of learners, based on the detection of patterns on the learners’ actions. El-Bishouty et al. [11] propose diferent mechanisms for the generation of a set of analytics based on learners’ actions in an LMS that can be assessed with automatic feedback. Still, there are some works that focus on the physical space, specially in the use of biometric sensors. Dafoulas et al. [12] make use of a set of biometric sensors, such as heartbeat, emotion detection, sweat levels, voice lfuctuations and voice recognition, to assess learners’ contribution in diferent collaborative activities. Nevertheless, learning situations that span multiple activities and spaces, and thus demand the combination of data from physical and digital traces, still remain a challenge. Under this premise, SCARLETT builds a learner model from the combination of analytics from the diferent activities and resources involved in the learning situation.
Over the last several years, Multimodal Learning Analytics (MMLA) has slowly contributed to supporting learning. RAP [6], a low-cost system for tracking and collecting students’ actions (voice volume, gaze, and posture) for delivering feedback summary on students’ performance with multimodal data. Learners’ generated data, including click-streams [13], and sensor data [14], were used to provide visual representations to improve learning and teaching experiences. However, though these and other research studies have illustrated the benefits from collecting and analyzing multimodal data in learning situations, their importance is not yet widely perceived and these systems are absent from almost every real educational settings [15].
SCARLETT is an SLE designed to facilitate the management and coordination of multiple learning environments across spaces in order to deploy personalized learning recommendations [4]. Thanks to its adaptor-based architecture, SCARLETT interacts with diferent learning tools and systems (Moodle, Canvas, Google Docs, etc.) during the enactment of the learning situations to facilitate the support for learners. More specifically, SCARLETT covers: (i) the data collection from the learners’ actions across the involved environments (sense); (ii) the incorporation of this information into a learner and context model, which represent their current learning state and conditions (analyze), and (iii) the evaluation of these models to trigger the deployment and recommendation of learning tasks and resources under the proper conditions (react).
The coordination of these operations is achieved by means of a learning design. In the learning design, teachers and instructors define the activities learners are expected to participate in, along with the related resources, objectives and topics. This information helps SCARLETT not only during the communication with the external tools, but also: (i) to make sense of the analytics that make up the learner model, by connecting them with the goals and topics discussed; and (ii), to look for appropriate resources to be presented to learners.
MBOX is a lightweight toolbox for collecting human interactions for collaborative learning scenarios [9]. MBOX utilizes a multilayered architecture taking advantage of the edge-fogcloud pattern [16]. The multilayered architecture has two main advantages: 1) being scalable to collect data from supplementary physical and digital data sources, and 2) benefiting from powerful computational resources when needed. This approach supports system adaptation for diferent learning environments and enables a better scaling of computational resources for diverse learning contexts. MBOX is designed to provide diferent sensors to detect and reason about human behavior interactions in learning contexts. The toolbox approach allows diferent sensors to be deployed and quickly shifted to support diferent learning situations. MBOX focuses on collecting and analyzing social interaction signals from computer vision, audio processing, biometric sensors, and other sensors.
The edge part is composed of a computational unit single board computers and powerful microcontrollers). Student groups face a sensing interface (SI) consisting of sensors, such as a wide-angle camera and microphone array. The SI captures and processes the signals at the Edge layer, and the data is stored in a time-series database. Additionally, multimodal interaction data is delivered to the Fog layer to process the data from a broader perspective further and provide insights on the classroom or at the school level. MBOX has a Feedback and Dashboard Interface (FDI) which is designed to interface with SCARLLET. The application provides visualization tools in diferent layers (cloud and fog), and it coordinates data processing in real-time. Despite its capabilities for sensing and analizing multimodal data, MBOX is not aimed at tracking students through multiple learning activities and spaces, nor at providing personalized student support, by integrating with SCARLETT we can expand to support richer learning experiences.
This section describes the concerns to be considered in the integration between MBOX and SCARLETT for the provision of personalized feedback and recommendations during an acrossspaces learning situation, with major emphasis on the actions performed in the physical space.
The currently planned learning scenario is designed for middle school students (grades 8-10), in which they will explore STEAM subjects at a university lab, as part of a general outreach program. The program is designed as a set of collected modules that explore real-world science problems (e.g. climate change & food production and pollution & food waste). The scenarios provide students with science topics that they can relate to from the diferent STEAM perspectives: science, computational thinking, and design and innovation. Each module involves lab work, teamwork, and group discussions with physical materials, and the production of digital artifacts, that include online documents, program code, data-logging from sensors and video. The pedagogical framing is inspired from inquiry-science learning and design processes that provides a structure for students to investigate.
Prior to the first session of the program, students are granted access to a LMS with resources, with videos and documents, that describe the activities to perform in the lab. Students are encouraged to consume these resources as a preparation for the session. As well, within the LMS they can answer a questionnaire related with basic concepts to be covered during the following sessions and for reporting topics of interest. In relation with the work performed in the laboratory, during the first two sessions students work in group and use the available equipment to collect data, update electronic journals and submit findings from the experiments Students Laboratory Sensing devices and interfaces Feedback and Dashboard
Classroom Sensing devices and interfaces Data processing
Parnoacleysscesd CEFldooguged MBOX
Data exchange
Electronic Journal
API Sense Analyze
SCARLETT
Data traces and model
LMS API React performed. The documents generated from these sessions have to be submitted in the LMS. In the third session, each group has to discuss the findings obtained and present a final report.
In the present learning situation, both SCARLETT and MBOX provide support in both the physical and virtual space, collecting data from diferent sources, analyzing it holistically and providing feedback. Figure 1 illustrates the expected interaction among both systems. With the data obtained from each system and environment learners are interacting in, covering data logs from the tools and recordings of the sessions in classroom and in the laboratory, both SCARLETT and MBOX can provide partial support on its own. SCARLETT can analyze the answers to the questionnaire to recommend additional videos to students that need reinforcement based on learners’ prior knowledge. Similarly, MBOX can detect the level of participation of each member of the group, the groups, and notify teachers when a specific group is not participating or discussing during the session. However, the exchange of data and analytics produced in each system can broaden the opportunities for support and intervention.
This exchange is beneficial in both directions. SCARLETT provides logs from the actions of learners within the LMS and the student model of each participant. This model contains diferent analytics like expertise, interest or grade of participation, derived not only from the responses to the questionnaire, but from the history of students’ actions with any of the available resources. The combination of these analytics with discourse analysis from audio recordings helps MBOX to detect unproductive discussions, based on the main keywords, or distinguish non-participating students in the discussion due to lack of knowledge. On the other hand, the discourse analytics reported by MBOX can be used to update the learner model with information related with struggling students or disengagement, that could trigger further intervertions by SCARLETT.
At this moment, the interaction among SCARLETT and MBOX is conceived through the exchange of datasets. Both MBOX and SCARLETT should expose the data obtained from the supported sensors and platforms and the resulting analytics and make the required changes in their architectures to integrate such data. Prior to the consumption of the data, data format and semantics need to be agreed. At least, elements like the type of action (what), the actor performing it (who) and the environment or space where the action takes place (where) need to be exchanged. This information will facilitate the integration of this data in the student model or its use for the provision of feedback and recommendations. From a technical perspective, this exchange should happen through an API. However, one concern that requires further discussion is whether data should flow after a push (a server sends the data to subscribed clients) or a pull (clients asynchronously request data to a server). Although it is reasonable that either MBOX or SCARLETT periodically requests for any new available data, specially for the provision of feedback once the activity has been completed, it can delay real-time interventions. This concern should may be critical for the provision of support during laboratory sessions, where learners should receive this feedback while the proper activity is taking place. The idea with integration of MBOX and SCARLETT is to promote a more robust and sustainable MMLA system that would adapt and continuously support diferent learning scenarios in a more flexible manner.
This paper represents a first exploration of the requirements and considerations to be considered in the adoption of MMLA in SLE from the perspective of MBOX and SCARLETT systems. The availability of a broader dataset related with the students’ actions, not only from their interaction with digital resources, but also of their behavior in the physical space raises an opportunity to ofer more meaningful support to learners. However, the exchange of data between the involved systems raises diferent concerns, related to architectural changes, the description of the semantics of the data exposed and the type and timing of support provided to learners, specially when states a real-time interaction with learners. These aspects raise additional requirements on how systems receive and consume the data. Future work will continue this collaboration and further develop the technical integration of the systems and explore the type of support ofered in controlled settings.
This research was funded by the Spanish State Research Agency (AEI) and the European Regional Development Fund (TIN2017-85179-C3-2-R, PID2020-112584RB-C32); and by the European Regional Development Fund and the Regional Government of Castilla y León (VA257P18) [3] B. Tabuenca, S. Serrano-Iglesias, A. Carruana-Martin, C. Villa-Torrano, Y. A. Dimitriadis, J. I. Asensio-Perez, C. Alario-Hoyos, E. Gomez-Sanchez, M. L. Bote-Lorenzo, A. MartinezMones, C. D. Kloos, Afordances and core functions of smart learning environments: A systematic literature review, IEEE Transactions on Learning Technologies (2021) 1–1. [4] S. Serrano-Iglesias, E. Gómez-Sánchez, M. L. Bote-Lorenzo, J. I. Asensio-Pérez, A. RuizCalleja, G. Vega-Gorgojo, Demonstration of SCARLETT: a Smart Learning Environment to Support Learners Across Formal and Informal Contexts, in: Proceedings of the 16th European Conference on Technology Enhanced Learning, Springer International Publishing, 2021, pp. 1–5. [5] P. Blikstein, M. Worsley, Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks, Journal of Learning Analytics 3 (2016) 220–238. [6] X. Ochoa, F. Domínguez, B. Guamán, R. Maya, G. Falcones, J. Castells, The RAP system, in: Proceedings of the 8th International Conference on Learning Analytics and Knowledge, ACM, 2018. [7] M. C. E. Chan, X. Ochoa, D. Clarke, Multimodal learning analytics in a laboratory classroom, in: Machine learning paradigms, Springer, 2020, pp. 131–156. [8] P. Chejara, L. P. Prieto, A. Ruiz-Calleja, M. J. Rodríguez-Triana, S. K. Shankar, R. Kasepalu, Quantifying collaboration quality in face-to-face classroom settings using mmla, in: International Conference on Collaboration Technologies and Social Computing, Springer, 2020, pp. 159–166. [9] H. Ouhaichi, D. Spikol, B. Vogel, Mbox: Designing a flexible iot multimodla learning analytics system, in: 2021 IEEE 21th International Conference on Advanced Learning Technologies (ICALT), 2021, pp. 122–126. [10] J. Seanosky, D. Boulanger, V. Kumar, Kinshuk, Unfolding learning analytics for big data, in: Emerging Issues in Smart Learning, Springer Berlin Heidelberg, 2014, pp. 377–384. [11] M. M. El-Bishouty, T.-W. Chang, R. Lima, M. B. Thaha, Kinshuk, S. Graf, Analyzing learner characteristics and courses based on cognitive abilities, learning styles, and context, in: Smart Learning Environments, Springer Berlin Heidelberg, 2014, pp. 3–25. [12] G. Dafoulas, C. C. Maia, J. Samuels-Clarke, A. Ali, J. C. Augusto, Investigating the role of biometrics in education–the use of sensor data in collaborative learning, in: IADIS Int.
Conf. e-Learning 2018, IADIS, Madrid, Spain, 2018. [13] G. Wang, X. Zhang, S. Tang, H. Zheng, B. Y. Zhao, Unsupervised clickstream clustering for user behavior analysis, in: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM, 2016. [14] L. Prieto, K. Sharma, Ł. Kidzinski, M. Rodríguez-Triana, P. Dillenbourg, Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data, Journal of Computer Assisted Learning 34 (2018) 193–203. [15] R. Wampfler, S. Klingler, B. Solenthaler, V. Schinazi, M. Gross, Afective state prediction in a mobile setting using wearable biometric sensors and stylus (2019). [16] K. Portelli, C. Anagnostopoulos, Leveraging edge computing through collaborative machine learning, in: 2017 5th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW), IEEE, 2017.