Adaptable smart learning environments supported by multimodal learning analytics Sergio Serrano-Iglesias1 , Daniel Spikol2 , Miguel L. Bote-Lorenzo1 , Hamza Ouhaichi3 , Eduardo Gómez-Sánchez1 and Bahtijar Vogel3 1 GSIC-EMIC Research Group, Universidad de Valladolid, Spain 2 Departments of Computer Science and Science Education, University of Copenhagen, Denmark 3 Department of Computer Science and Media Technology, Malmö University, Sweden Abstract Smart Learning Environments and Learning Analytics hold promise of providing personalized support to learners according to their individual needs and context. This support can be achieved by collecting and analyzing data from the different learning tools and systems that are involved in the learning experience. This paper presents a first exploration of requirements and considerations for the integration of two systems: MBOX, a Multimodal Learning Analytics system for the physical space (human behavior and learning context), and SCARLETT, an SLE for the support during across-spaces learning situations combining different learning systems. This integration will enable the SLE to have access to a new and wide range of information, notably students’ behavior and social interactions in the physical learning context (e.g. classroom). The integration of multimodal data with the data coming from the digital learning environments will result in a more holistic system, therefore producing learning analytics that trigger personalized feedback and learning resources. Such integration and support is illustrated with a learning scenario that helps to discuss how these analytics can be derived and used for the intervention by the SLE. Keywords smart learning environments, multimodal learning analytics, learning design, across spaces 1. Introduction Smart Learning Environments (SLEs) hold promise on providing personalized support to learners, based on their individual needs and context, in order to achieve an effective, efficient and engaging learning experience [1, 2]. Thanks to the amount of data exposed by learning systems and tools, such as Learning Management Systems (LMSs), mobile devices and digital tools, SLEs are able to sense learners’ actions, to analyze and characterize their progression, and to react and intervene accordingly [3]. In this context, SCARLETT (Smart Context-Aware Recommendation of Learning Extensions in ubiquiTous seTtings) was proposed as an SLE designed to provide support to learners during the enactment of across-spaces learning situations, with a special LA4SLE @ EC-TEL 2021: Learning Analytics for Smart Learning Environments, September 21, 2021, Bolzano, Italy " sergio@gsic.uva.es (S. Serrano-Iglesias); ds@di.ku.sk (D. Spikol); migbot@tel.uva.es (M. L. Bote-Lorenzo); hamza.ouhaichi@mau.se (H. Ouhaichi); edugom@tel.uva.es (E. Gómez-Sánchez); bahtijar.vogel@mau.se (B. Vogel)  0000-0002-3110-1096 (S. Serrano-Iglesias); 0000-0001-9454-0793 (D. Spikol); 0000-0002-8825-0412 (M. L. Bote-Lorenzo); 0000-0002-9278-8063 (H. Ouhaichi); 0000-0003-0062-916X (E. Gómez-Sánchez); 0000-0001-6708-5983 (B. Vogel) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) focus on connecting formal and informal learning experiences [4]. SCARLETT is capable of interacting with third-party tools to collect and combine data about the actions learners perform in them, in order to keep track of their progression through the different activities. As a result, with this information SCARLETT can determine appropriate resources and feedback to offer. However, the data obtained from the interaction with digital devices provides only a partial picture of the way learners interact during the activities, specially in the physical space. For example, when a group of students work physically together in a laboratory with one device, computer logs record only the interactions of the group with the device but not the (verbal and gestural) interaction between the members of the group. To overcome this limitation, the efforts from the MultiModal Learning Analytics should be considered. Multimodal Learning Analytics (MMLA) combines the power of affordable sensor technologies and advances in machine learning to observe and analyse learning activities from the perspective of different data sources and modalities [5, 6]. This technology acts as a virtual observer and analyst of learning activities across multiple contexts between stakeholders, devices, and resources. Current research explores topics like real-time and automatic video and audio analysis can support learning by automating the analysis of these activities through the development of new tools and methods [7, 8]. One the challenges for MMLA has been the use complex systems, designed for specific controlled settings, in authentic learning settings by non-expert users. Multimodal Box (MBOX) has taken a different approach away from specialized tools to an Internet of Things (IoT) concept that allows different sensors to be utilized across spaces [9]. The aim with MBOX is to provide a smaller footprint for sensors that can be adapted to real-world learning settings and work together with SLEs like SCARLETT . In this paper we explore how the adoption of MMLA in SLEs may help in the personalization of the support offered to learners. This research is performed by means of the integration of MBOX in SCARLETT. The rest of the paper is structured as follows. Section 2 presents related work about the adoption of Learning Analytics in SLEs and MMLA. Section 3 describes in more detail the systems involved. Section 4 presents the learning scenario supported by both systems, with a technical description of the interactions performed. Section 5 presents the conclusions of this initial research. 2. Related work 2.1. Learning Analytics in SLEs Learning Analytics play a significant role towards the personalized support of learners in SLEs [3]. Based on the different indicators generated by Learning Analytics processes from the traces of the involved resources, SLEs can make decisions on the appropriate interventions to perform. Most of the literature about LA in SLEs relate to the traces of learners in the virtual space. Seanosky et al. [10] developed SCALE to keep track of the progression of learners, based on the detection of patterns on the learners’ actions. El-Bishouty et al. [11] propose different mechanisms for the generation of a set of analytics based on learners’ actions in an LMS that can be assessed with automatic feedback. Still, there are some works that focus on the physical space, specially in the use of biometric sensors. Dafoulas et al. [12] make use of a set of biometric sensors, such as heartbeat, emotion detection, sweat levels, voice fluctuations and voice recognition, to assess learners’ contribution in different collaborative activities. Nevertheless, learning situations that span multiple activities and spaces, and thus demand the combination of data from physical and digital traces, still remain a challenge. Under this premise, SCARLETT builds a learner model from the combination of analytics from the different activities and resources involved in the learning situation. 2.2. MultiModal Learning Analytics Over the last several years, Multimodal Learning Analytics (MMLA) has slowly contributed to supporting learning. RAP [6], a low-cost system for tracking and collecting students’ actions (voice volume, gaze, and posture) for delivering feedback summary on students’ performance with multimodal data. Learners’ generated data, including click-streams [13], and sensor data [14], were used to provide visual representations to improve learning and teaching expe- riences. However, though these and other research studies have illustrated the benefits from collecting and analyzing multimodal data in learning situations, their importance is not yet widely perceived and these systems are absent from almost every real educational settings [15]. 3. System description 3.1. SCARLETT: Smart Context-Aware Recommendation of Learning Extensions in ubiquiTous seTtings SCARLETT is an SLE designed to facilitate the management and coordination of multiple learn- ing environments across spaces in order to deploy personalized learning recommendations [4]. Thanks to its adaptor-based architecture, SCARLETT interacts with different learning tools and systems (Moodle, Canvas, Google Docs, etc.) during the enactment of the learning situations to facilitate the support for learners. More specifically, SCARLETT covers: (i) the data collection from the learners’ actions across the involved environments (sense); (ii) the incorporation of this information into a learner and context model, which represent their current learning state and conditions (analyze), and (iii) the evaluation of these models to trigger the deployment and recommendation of learning tasks and resources under the proper conditions (react). The coordination of these operations is achieved by means of a learning design. In the learning design, teachers and instructors define the activities learners are expected to participate in, along with the related resources, objectives and topics. This information helps SCARLETT not only during the communication with the external tools, but also: (i) to make sense of the analytics that make up the learner model, by connecting them with the goals and topics discussed; and (ii), to look for appropriate resources to be presented to learners. 3.2. The Multimodal BOX: MBOX MBOX is a lightweight toolbox for collecting human interactions for collaborative learning scenarios [9]. MBOX utilizes a multilayered architecture taking advantage of the edge-fog- cloud pattern [16]. The multilayered architecture has two main advantages: 1) being scalable to collect data from supplementary physical and digital data sources, and 2) benefiting from powerful computational resources when needed. This approach supports system adaptation for different learning environments and enables a better scaling of computational resources for diverse learning contexts. MBOX is designed to provide different sensors to detect and reason about human behavior interactions in learning contexts. The toolbox approach allows different sensors to be deployed and quickly shifted to support different learning situations. MBOX focuses on collecting and analyzing social interaction signals from computer vision, audio processing, biometric sensors, and other sensors. The edge part is composed of a computational unit single board computers and powerful microcontrollers). Student groups face a sensing interface (SI) consisting of sensors, such as a wide-angle camera and microphone array. The SI captures and processes the signals at the Edge layer, and the data is stored in a time-series database. Additionally, multimodal interaction data is delivered to the Fog layer to process the data from a broader perspective further and provide insights on the classroom or at the school level. MBOX has a Feedback and Dashboard Interface (FDI) which is designed to interface with SCARLLET. The application provides visualization tools in different layers (cloud and fog), and it coordinates data processing in real-time. Despite its capabilities for sensing and analizing multimodal data, MBOX is not aimed at tracking students through multiple learning activities and spaces, nor at providing personalized student support, by integrating with SCARLETT we can expand to support richer learning experiences. 4. Adoption of MMLA in SLEs This section describes the concerns to be considered in the integration between MBOX and SCARLETT for the provision of personalized feedback and recommendations during an across- spaces learning situation, with major emphasis on the actions performed in the physical space. 4.1. Description of the learning scenario The currently planned learning scenario is designed for middle school students (grades 8-10), in which they will explore STEAM subjects at a university lab, as part of a general outreach program. The program is designed as a set of collected modules that explore real-world science problems (e.g. climate change & food production and pollution & food waste). The scenarios provide students with science topics that they can relate to from the different STEAM perspectives: science, computational thinking, and design and innovation. Each module involves lab work, teamwork, and group discussions with physical materials, and the production of digital artifacts, that include online documents, program code, data-logging from sensors and video. The pedagogical framing is inspired from inquiry-science learning and design processes that provides a structure for students to investigate. Prior to the first session of the program, students are granted access to a LMS with resources, with videos and documents, that describe the activities to perform in the lab. Students are encouraged to consume these resources as a preparation for the session. As well, within the LMS they can answer a questionnaire related with basic concepts to be covered during the following sessions and for reporting topics of interest. In relation with the work performed in the laboratory, during the first two sessions students work in group and use the available equipment to collect data, update electronic journals and submit findings from the experiments Physical Space Virtual Space Electronic Laboratory Classroom LMS Journal Students Sensing devices Sensing devices and interfaces and interfaces API API MBOX Data exchange SCARLETT Data processing Sense Feedback and Cloud Data traces React and model Dashboard Processed Fog Analyze analy�cs Edge Figure 1: Interaction between MBOX and SCARLETT in the sample scenario performed. The documents generated from these sessions have to be submitted in the LMS. In the third session, each group has to discuss the findings obtained and present a final report. 4.2. Integration of the systems In the present learning situation, both SCARLETT and MBOX provide support in both the physical and virtual space, collecting data from different sources, analyzing it holistically and providing feedback. Figure 1 illustrates the expected interaction among both systems. With the data obtained from each system and environment learners are interacting in, covering data logs from the tools and recordings of the sessions in classroom and in the laboratory, both SCARLETT and MBOX can provide partial support on its own. SCARLETT can analyze the answers to the questionnaire to recommend additional videos to students that need reinforcement based on learners’ prior knowledge. Similarly, MBOX can detect the level of participation of each member of the group, the groups, and notify teachers when a specific group is not participating or discussing during the session. However, the exchange of data and analytics produced in each system can broaden the opportunities for support and intervention. This exchange is beneficial in both directions. SCARLETT provides logs from the actions of learners within the LMS and the student model of each participant. This model contains different analytics like expertise, interest or grade of participation, derived not only from the responses to the questionnaire, but from the history of students’ actions with any of the available resources. The combination of these analytics with discourse analysis from audio recordings helps MBOX to detect unproductive discussions, based on the main keywords, or distinguish non-participating students in the discussion due to lack of knowledge. On the other hand, the discourse analytics reported by MBOX can be used to update the learner model with information related with struggling students or disengagement, that could trigger further intervertions by SCARLETT. At this moment, the interaction among SCARLETT and MBOX is conceived through the exchange of datasets. Both MBOX and SCARLETT should expose the data obtained from the supported sensors and platforms and the resulting analytics and make the required changes in their architectures to integrate such data. Prior to the consumption of the data, data format and semantics need to be agreed. At least, elements like the type of action (what), the actor performing it (who) and the environment or space where the action takes place (where) need to be exchanged. This information will facilitate the integration of this data in the student model or its use for the provision of feedback and recommendations. From a technical perspective, this exchange should happen through an API. However, one concern that requires further discussion is whether data should flow after a push (a server sends the data to subscribed clients) or a pull (clients asynchronously request data to a server). Although it is reasonable that either MBOX or SCARLETT periodically requests for any new available data, specially for the provision of feedback once the activity has been completed, it can delay real-time interventions. This concern should may be critical for the provision of support during laboratory sessions, where learners should receive this feedback while the proper activity is taking place. The idea with integration of MBOX and SCARLETT is to promote a more robust and sustainable MMLA system that would adapt and continuously support different learning scenarios in a more flexible manner. 5. Conclusions and Future Work This paper represents a first exploration of the requirements and considerations to be considered in the adoption of MMLA in SLE from the perspective of MBOX and SCARLETT systems. The availability of a broader dataset related with the students’ actions, not only from their interaction with digital resources, but also of their behavior in the physical space raises an opportunity to offer more meaningful support to learners. However, the exchange of data between the involved systems raises different concerns, related to architectural changes, the description of the semantics of the data exposed and the type and timing of support provided to learners, specially when states a real-time interaction with learners. These aspects raise additional requirements on how systems receive and consume the data. Future work will continue this collaboration and further develop the technical integration of the systems and explore the type of support offered in controlled settings. Acknoledgements This research was funded by the Spanish State Research Agency (AEI) and the European Regional Development Fund (TIN2017-85179-C3-2-R, PID2020-112584RB-C32); and by the European Regional Development Fund and the Regional Government of Castilla y León (VA257P18) References [1] J. M. Spector, Conceptualizing the emerging field of smart learning environments, Smart learning environments 1 (2014) 1–10. [2] G.-J. Hwang, Definition, framework and research issues of smart learning environments-a context-aware ubiquitous learning perspective, Smart Learning Environments 1 (2014) 4. [3] B. Tabuenca, S. Serrano-Iglesias, A. Carruana-Martin, C. Villa-Torrano, Y. A. Dimitriadis, J. I. Asensio-Perez, C. Alario-Hoyos, E. Gomez-Sanchez, M. L. Bote-Lorenzo, A. Martinez- Mones, C. D. Kloos, Affordances and core functions of smart learning environments: A systematic literature review, IEEE Transactions on Learning Technologies (2021) 1–1. [4] S. Serrano-Iglesias, E. Gómez-Sánchez, M. L. Bote-Lorenzo, J. I. Asensio-Pérez, A. Ruiz- Calleja, G. Vega-Gorgojo, Demonstration of SCARLETT: a Smart Learning Environment to Support Learners Across Formal and Informal Contexts, in: Proceedings of the 16th Eu- ropean Conference on Technology Enhanced Learning, Springer International Publishing, 2021, pp. 1–5. [5] P. Blikstein, M. Worsley, Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks, Journal of Learning Analytics 3 (2016) 220–238. [6] X. Ochoa, F. Domínguez, B. Guamán, R. Maya, G. Falcones, J. Castells, The RAP system, in: Proceedings of the 8th International Conference on Learning Analytics and Knowledge, ACM, 2018. [7] M. C. E. Chan, X. Ochoa, D. Clarke, Multimodal learning analytics in a laboratory classroom, in: Machine learning paradigms, Springer, 2020, pp. 131–156. [8] P. Chejara, L. P. Prieto, A. Ruiz-Calleja, M. J. Rodríguez-Triana, S. K. Shankar, R. Kasepalu, Quantifying collaboration quality in face-to-face classroom settings using mmla, in: International Conference on Collaboration Technologies and Social Computing, Springer, 2020, pp. 159–166. [9] H. Ouhaichi, D. Spikol, B. Vogel, Mbox: Designing a flexible iot multimodla learning analytics system, in: 2021 IEEE 21th International Conference on Advanced Learning Technologies (ICALT), 2021, pp. 122–126. [10] J. Seanosky, D. Boulanger, V. Kumar, Kinshuk, Unfolding learning analytics for big data, in: Emerging Issues in Smart Learning, Springer Berlin Heidelberg, 2014, pp. 377–384. [11] M. M. El-Bishouty, T.-W. Chang, R. Lima, M. B. Thaha, Kinshuk, S. Graf, Analyzing learner characteristics and courses based on cognitive abilities, learning styles, and context, in: Smart Learning Environments, Springer Berlin Heidelberg, 2014, pp. 3–25. [12] G. Dafoulas, C. C. Maia, J. Samuels-Clarke, A. Ali, J. C. Augusto, Investigating the role of biometrics in education–the use of sensor data in collaborative learning, in: IADIS Int. Conf. e-Learning 2018, IADIS, Madrid, Spain, 2018. [13] G. Wang, X. Zhang, S. Tang, H. Zheng, B. Y. Zhao, Unsupervised clickstream clustering for user behavior analysis, in: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM, 2016. [14] L. Prieto, K. Sharma, Ł. Kidzinski, M. Rodríguez-Triana, P. Dillenbourg, Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data, Journal of Computer Assisted Learning 34 (2018) 193–203. [15] R. Wampfler, S. Klingler, B. Solenthaler, V. Schinazi, M. Gross, Affective state prediction in a mobile setting using wearable biometric sensors and stylus (2019). [16] K. Portelli, C. Anagnostopoulos, Leveraging edge computing through collaborative ma- chine learning, in: 2017 5th International Conference on Future Internet of Things and Cloud Workshops (FiCloudW), IEEE, 2017.