Smart School Multimodal Dataset and Challenges Luis P. Prieto1 , Marı́a Jesús Rodrı́guez-Triana1,2 , Marge Kusmin1 , and Mart Laanpere1 1 Tallinn University, Tallinn, Estonia 2 École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland lprisan@tlu.ee, mjrt@tlu.ee, marge.kusmin@gmail.com, martl@tlu.ee Abstract. As part of a research project aiming to explore the notion of ‘smart school’ (especially for STEM education) in Estonia, we are devel- oping classrooms and schools that incorporate data gathering not only from digital traces, but also physical ones (through a variety of sensors). This workshop contribution describes briefly the setting and our initial efforts in setting up a classroom that is able to generate such a mul- timodal dataset. The paper also describes some of the most important challenges that we are facing as we setup the project and attempt to build up such dataset, focusing on the specifics of doing it in an every- day, authentic school setting. We believe these challenges provide a nice sample of those that the multimodal learning analytics (MMLA) commu- nity will have to face as it transitions from an emergent to a mainstream community of research and practice. Keywords: Multimodal learning analytics, multimodal teaching ana- lytics, smart school, smart classroom, STEM education, sensors 1 Context: Smart School Project Engaging primary school students in Science, Technology, Engineering and Maths (STEM) learning is difficult, due to the often abstract notions and concepts in- volved. One common alternative proposed to improve engagement and learning about such subjects, is to involve students in scientific inquiries, in which stu- dents are involved in formulating hypotheses and gathering and analyzing real data. Very often, this gathering of data is done outdoors, using increasingly available mobile and sensing technologies [7]. However, the application of these approaches in authentic setting conditions faces simple but quite important con- straints in terms of timing and effort (e.g., logistics of such data gathering trips, matching between weather and curriculum sequence constraints, etc.). At Tallinn University, we are starting a project that takes a different per- spective on this problem of student engagement in STEM and its constraints: instead of (or, in addition to) “taking students to the data”, the Smart School project aims to “bring the data to students”, while still keeping it authentic and relevant to them. The general idea of the project is to support the next generation of scientists and engineers by having them learn in a data-rich school environment. As part of this project, we will instrument a rural primary school with dif- ferent sensing technologies, to enable automatic data collection, including the physical learning environment (i.e., ambient variables like temperature, CO2 , pressure, presence or positioning, etc.), and integrate these data with the digital footprints of learners (e.g., from LMSs and other digital tools). Such multimodal data setup will be used in three different directions: 1) to provide ‘smart building’ capabilities (in terms of energy efficiency, comfort, etc.); 2) to provide analyt- ics about the learning processes (to inform teachers, administrators and even learners); and 3) to be used by learners themselves in STEM education. From the point of view of the multimodal learning analytics (MMLA) com- munity, direction #2 above will imply not only the application of multimodal learning analytics approaches, but will also serve a secondary aim: to bridge the gap between current MMLA research (which is still very experimental, often featuring lab settings and complicated technology setups [8, 4]) and everyday classroom practice, to understand what it takes to make it work within authen- tic, everyday school constraints. In our contribution to the workshop, we address mainly this second direction (the multimodal learning analytics aspects of the project). In the following sec- tion, we outline our initial idea of the technological setup to be used in the initial phases of the project, and the potential multimodal dataset to be generated in the near future. In the next section, we extract several challenges that we face in the implementation of the project’s MMLA, which we believe exemplify quite well common challenges of the MMLA community in the near future. Finally, we include several additional open questions which we hope to discuss with the rest of the workshop participants in the face-to-face sessions. 2 A First Smart School Dataset Given that we want to follow an iterative design-based research methodology for our project, in a first phase we will focus on the identification of a number of relevant learning scenarios and sensing technologies, in close collaboration with primary school teachers. To enable this exploratory investigation, we will instrument a single “smart classroom” with a variety of sensors and learning technologies, which will be made available to innovative teachers from Estonian primary schools to perform pilot classroom experiments (with their actual pri- mary students, in visits to our university). Later on, a whole rural school in central Estonia will be equipped with the technologies that we have found most useful in this exploratory phase. We think that the data recorded during these pilot experiments can provide the basis for a very interesting and varied dataset for the MMLA community, featuring not only multiple data sources, but also multiple kinds of learning tasks, situations and teaching approaches to analyze. Among the research questions that this multimodal dataset will enable us to explore (both in the ‘smart building’, learning analytics and STEM education di- mensions), we can cite: the relationship between physical aspects of the learning and teaching process, and learning (e.g., embodied learning, teacher proxemics, etc.); the relationship between ambient factors (e.g., light, CO2 , etc.) and learn- ing; or the investigation of robust (i.e., generalizable for more than one learning task) multimodal indicators of learning, to help in teacher decision-making. As of this writing, we are in the process of acquisition and installation of the sensors for the first experimental classroom, keeping in mind the different cate- gories of data sources that we want to gather (ambient, physiological, physical, cognitive/learning), as well as the restrictions of a school classroom (in terms of durability, flexibility, etc.). Our initial setup ideas include: – Several static ambient sensors placed in different parts of the classroom, to measure variables like temperature or magnetic field3 . – Several motion capture sensors4 will be placed in strategic areas of the class- room (e.g., the front of classroom, where potentially interesting teacher/student activity is more often happening). These sensors will not only serve to record detailed physical activity data (while remaining relatively privacy- conscious), but also capture audio feeds of different parts of the classroom (with potential for audio direction recognition, as they contain microphone arrays). – In order to track physiological variables like heart rate or galvanic skin re- sponse (commonly used to track affective response, still underexploited in MMLA), a wearable wristband5 will also be part of the setup. Given their (still quite high) cost, only one or a few of these sensors will be part of the initial setup, probably to be worn by the teacher or randomly-selected stu- dents. Additional wearable sensors like mobile eye-trackers (already used, for instance, in “multimodal teaching analytics” [4]) are also being considered. – As an affordable and more flexible alternative to record and stream individ- ual student activity, (cheap) smartphones will be worn by students, logging accelerometer and indoor location/proximity (see next point), as well as au- diovisual feeds, if needed. – Beacons and stickers6 will be placed in strategic places in the classroom, as well as in potentially interesting classroom elements (laptops and tablets, interactive and traditional whiteboards, etc.) in order to track the positions of the different actors in the learning situations. These sensors often can also be used to complement ambient readings of temperature or motion. – Aside from the aforementioned researcher-placed sensors, and given the na- ture of our project (i.e., the use of inquiry-based STEM learning), several teacher- and student-placeable sensors7 will be also made available to people using the classroom, for their own science experiments. 3 For our first prototypes, user-placeable and easily synchronizable sensors will be used, such as PocketLabs sensors (http://www.pocketlabs.com). 4 E.g., Microsoft Kinect. 5 Such as the Empatica E4, https://www.empatica.com/e4-wristband. 6 See, for instance, commercially available models based on Bluetooth LTE like Esti- mote’s (http://www.estimote.com). 7 Again, PocketLab or similar sensors. Aside from these sensors aimed to track the physical space and actions of teachers and learners, we intend to merge these data sources with others coming from the digital space, such as: – Learning analytics-enabled platforms to support IBL processes, such as Graasp/GOLABZ8 can provide classic learning analytics metrics (with relatively low semantic value), such as number of student actions in a certain activity, number of students in a certain IBL phase, etc. – Additional, high semantic value metrics will be extracted from ad-hoc ques- tionnaires, tests and assessments developed by researchers or teachers9 , aimed at measuring different forms of learning gains more directly. 3 Multimodal Learning Analytics Challenges In the ‘Smart School’ project we will face multiple challenges, some of which emerge directly from the messiness and complexity of gathering data in authentic settings (where we will have limited control about the process, and we will face strong contextual restrictions, like time, curriculum or effort): – Pedagogical messiness (or ‘ecumenism’) of schools: even if our project is fo- cused on STEM education and IBL, schools exhibit multiple kinds of teach- ing/learning approaches at different moments (from behavioral teaching of procedural routines to collaborative learning or knowledge building). How- ever, current MMLA is very task- or pedagogy-specific. Thus, we will need to find metrics that are able to address this pedagogical richness and still remain useful for teacher and student decision-making, and mappings be- tween these intermediate metrics and the available data sources and analysis methods and models (which can be context-dependent, or at least “locally trained” to account for the specificities of each classroom). – Physical messiness of schools: given that the setup should be used every day, several times a day, by both adults and children, expensive or delicate equipment should be avoided in favor of heavy-duty sensors and devices. This not only may have an impact on the accuracy of the data sources (making multimodal triangulation even more crucial), but also on how much we can trust the outputs of our automated analyses (making ad-hoc, direct measurements of learning all the more important). – User identification (i.e., mapping the different parts of a data source to the actor generating it) is a very common challenge in MMLA that has not been solved satisfactorily so far, and is expected to be exacerbated by the messiness of an everyday classroom setting (e.g., devices changing hands, students changing sitting positions, etc.). – Schools are inherently a multi-actor setting: Not only we need MMLA met- rics and algorithms that can address our research questions as researchers; 8 http://graasp.eu/ , http://www.golabz.eu/ 9 E.g., powered by Google Forms or similar engines. we also need (potentially different) metrics to aid teachers in run-time de- cision making, as well as (yet different) metrics and visualizations that help students themselves reflect and act upon their own learning progress. – Data gathering, analysis and feedback architecture: We should not disregard other classic challenges of MMLA research, regarding the architecture for gathering data from multiple physical and digital data sources, aligning them (e.g., timestamping issues, etc.), finding suitable and generalizable strategies for data fusion, etc. – Ethics in everyday research: Conducting MMLA ethically, across the whole LA lifecycle, is even more challenging than usual log-based approaches – given the abundance of potentially sensitive data like video or physiological data. Addressing all the subjects’ rights with regard to their data (parental and child consent, anonymization, rights to opt-out and be forgotten, etc.) is already difficult in one-off school experiments – the probability of these issues to arise if our setup is truly used every day will multiply accordingly. We are currently considering several approaches to address these issues from the outset: • To record only feature-level data (e.g., video features instead of video) to preserve privacy – however, this will often break the inspectability and accountability of the analysis results (since raw data will not be available to double-check) • To use only data sources that are ‘anonymous by default’, like move- ment sensors, infrared cameras, audio instead of video, etc. In the same line of thinking, data can be associated to devices and not to people. However, these techniques will make it difficult to provide personalized interventions to help support specific students. • Distinguish between data intended for run-time orchestration-level di- agnostics, versus data for personalized support of specific students (and restrict access to data and modelling accordingly), so that more sensitive data only is used by the actors that actually should act upon it (or by the actor generating the data). • On top of all of the above, we will pay close attention to the emergent field of privacy-conscious analytics (e.g., being able to analyze and cross- reference data without decrypting the data sources, as in [6]). – Finally, and unlike MMLA datasets existing today (which are closed, in the sense that they have a known, defined amount of data), the smart school set- ting has the potential to produce data continuously, in a streaming fashion. Even if in a first phase we will produce closed datasets, the data gathering, alignment, analysis and opening up of such a streaming dataset will provide a host of other interesting future challenges for the MMLA community. We believe the specific challenges above are especially relevant for the MMLA community, since they represent typical challenges that we will have to face as multimodal analyses of learning abandon their current status of ‘emergent/niche technique’ and become more mainstream. Of course, general challenges of learn- ing analytics research still apply to this project, such as finding actionable met- rics, considering algorithmic accountability (i.e., can we trust the results and act on them?) [2], considering and promoting data literacy among teachers and students [3], etc. 4 Open Questions Many of the challenges outlined above represent open questions in the MMLA community, in and of themselves, and we hope to discuss them with the rest of the community during the workshop. Furthermore, we would also like to dis- cuss several other, more specific questions about operationalizing the dataset, including: – What would be a good scale for the dataset (i.e., for how many lessons should we record data), so that the dataset can be useful for the MMLA community? – What should be the scope of the dataset (i.e., record only IBL lessons, vs. record a variety of lessons and approaches) to be most useful for other re- searchers? – Are we missing other potentially useful sources of data (e.g., sensors, digital traces, etc.) that we could add to the setup at a relatively low cost, and with good reliability? – What exact format should the dataset use? xAPI stores are quite popular in the general LA community, but the MMLA community has also exper- imented with other formats like providing a virtual machine with all the (sometimes custom) analysis and visualization tooling incorporated into it. Finally, it is worth noting that this ambitious project is not aiming at the study of a single learning or teaching phenomenon. Rather, the dataset would be aimed for us (and the MMLA community) to explore the potential of these different sensors for studying issues as disparate as: the long-term effect of am- bient factors in learning, the divergences between teachers’ learning designs and their enactment [4], including the influence of diverse enactment and discourse routines [5, 1] in learning performance, and many more. In future contributions, and in the dialogue with the rest of the MMLA community, we hope to identify which sensors and analysis techniques are most adequate to study each of these issues. Acknowledgments This project has received funding from the European Union’s Horizon 2020 re- search and innovation programme under grant agreement No. 669074. References 1. Donnelly, P.J., Blanchard, N., Olney, A.M., Kelly, S., Nystrand, M., D’Mello, S.K.: Words matter: automatic detection of teacher questions in live classroom discourse using linguistics, acoustics, and context. In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference. pp. 218–227. ACM (2017) 2. Drachsler, H., Greller, W.: Privacy and analytics: it’s a delicate issue a checklist for trusted learning analytics. In: Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. pp. 89–98. ACM (2016) 3. Hirsh, S.: Common-core work must include teacher development. Education Week 31(19), 22–24 (2012) 4. Prieto, L.P., Sharma, K., Dillenbourg, P., Rodrı́guez-Triana, M.J.: Teaching analyt- ics: towards automatic extraction of orchestration graphs using wearable sensors. In: Proceedings of the Sixth International Conference on Learning Analytics & Knowl- edge. pp. 148–157. ACM (2016) 5. Prieto, L.P., Villagrá-Sobrino, S., Jorrı́n-Abellán, I.M., Martı́nez-Monés, A., Dimitriadis, Y.: Recurrent routines: Analyzing and supporting orchestration in technology-enhanced primary classrooms. Computers & Education 57(1), 1214–1227 (2011) 6. Schmidlin, K., Clough-Gorr, K.M., Spoerri, A.: Privacy preserving probabilistic record linkage (p3rl): a novel method for linking existing health-related data and maintaining participant confidentiality. BMC medical research methodology 15(1), 1 (2015) 7. Vogel, B., Spikol, D., Kurti, A., Milrad, M.: Integrating mobile, web and sensory technologies to support inquiry-based science learning. In: Wireless, Mobile and Ubiquitous Technologies in Education (WMUTE), 2010 6th IEEE International Conference on. pp. 65–72. IEEE (2010) 8. Worsley, M., Blikstein, P.: Towards the development of multimodal action based as- sessment. In: Proceedings of the third international conference on learning analytics and knowledge. pp. 94–101. ACM (2013)