As part of a research project aiming to explore the notion of `smart school' (especially for STEM education) in Estonia, we are developing classrooms and schools that incorporate data gathering not only from digital traces, but also physical ones (through a variety of sensors). This workshop contribution describes brie y the setting and our initial e orts in setting up a classroom that is able to generate such a multimodal dataset. The paper also describes some of the most important challenges that we are facing as we setup the project and attempt to build up such dataset, focusing on the speci cs of doing it in an everyday, authentic school setting. We believe these challenges provide a nice sample of those that the multimodal learning analytics (MMLA) community will have to face as it transitions from an emergent to a mainstream community of research and practice.
Engaging primary school students in Science, Technology, Engineering and Maths (STEM) learning is di cult, due to the often abstract notions and concepts involved. One common alternative proposed to improve engagement and learning about such subjects, is to involve students in scienti c inquiries, in which students are involved in formulating hypotheses and gathering and analyzing real data. Very often, this gathering of data is done outdoors, using increasingly available mobile and sensing technologies [7]. However, the application of these approaches in authentic setting conditions faces simple but quite important constraints in terms of timing and e ort (e.g., logistics of such data gathering trips, matching between weather and curriculum sequence constraints, etc.).
At Tallinn University, we are starting a project that takes a di erent perspective on this problem of student engagement in STEM and its constraints: instead of (or, in addition to) \taking students to the data", the Smart School project aims to \bring the data to students", while still keeping it authentic and relevant to them. The general idea of the project is to support the next generation of scientists and engineers by having them learn in a data-rich school environment.
As part of this project, we will instrument a rural primary school with different sensing technologies, to enable automatic data collection, including the physical learning environment (i.e., ambient variables like temperature, CO2, pressure, presence or positioning, etc.), and integrate these data with the digital footprints of learners (e.g., from LMSs and other digital tools). Such multimodal data setup will be used in three di erent directions: 1) to provide `smart building' capabilities (in terms of energy e ciency, comfort, etc.); 2) to provide analytics about the learning processes (to inform teachers, administrators and even learners); and 3) to be used by learners themselves in STEM education.
From the point of view of the multimodal learning analytics (MMLA) community, direction #2 above will imply not only the application of multimodal learning analytics approaches, but will also serve a secondary aim: to bridge the gap between current MMLA research (which is still very experimental, often featuring lab settings and complicated technology setups [8, 4]) and everyday classroom practice, to understand what it takes to make it work within authentic, everyday school constraints.
In our contribution to the workshop, we address mainly this second direction (the multimodal learning analytics aspects of the project). In the following section, we outline our initial idea of the technological setup to be used in the initial phases of the project, and the potential multimodal dataset to be generated in the near future. In the next section, we extract several challenges that we face in the implementation of the project's MMLA, which we believe exemplify quite well common challenges of the MMLA community in the near future. Finally, we include several additional open questions which we hope to discuss with the rest of the workshop participants in the face-to-face sessions. 2
Given that we want to follow an iterative design-based research methodology for our project, in a rst phase we will focus on the identi cation of a number of relevant learning scenarios and sensing technologies, in close collaboration with primary school teachers. To enable this exploratory investigation, we will instrument a single \smart classroom" with a variety of sensors and learning technologies, which will be made available to innovative teachers from Estonian primary schools to perform pilot classroom experiments (with their actual primary students, in visits to our university). Later on, a whole rural school in central Estonia will be equipped with the technologies that we have found most useful in this exploratory phase. We think that the data recorded during these pilot experiments can provide the basis for a very interesting and varied dataset for the MMLA community, featuring not only multiple data sources, but also multiple kinds of learning tasks, situations and teaching approaches to analyze.
Among the research questions that this multimodal dataset will enable us to explore (both in the `smart building', learning analytics and STEM education dimensions), we can cite: the relationship between physical aspects of the learning and teaching process, and learning (e.g., embodied learning, teacher proxemics, etc.); the relationship between ambient factors (e.g., light, CO2, etc.) and learning; or the investigation of robust (i.e., generalizable for more than one learning task) multimodal indicators of learning, to help in teacher decision-making.
As of this writing, we are in the process of acquisition and installation of the sensors for the rst experimental classroom, keeping in mind the di erent categories of data sources that we want to gather (ambient, physiological, physical, cognitive/learning), as well as the restrictions of a school classroom (in terms of durability, exibility, etc.). Our initial setup ideas include: { Several static ambient sensors placed in di erent parts of the classroom, to measure variables like temperature or magnetic eld3. { Several motion capture sensors4 will be placed in strategic areas of the classroom (e.g., the front of classroom, where potentially interesting teacher/student activity is more often happening). These sensors will not only serve to record detailed physical activity data (while remaining relatively privacyconscious), but also capture audio feeds of di erent parts of the classroom (with potential for audio direction recognition, as they contain microphone arrays). { In order to track physiological variables like heart rate or galvanic skin response (commonly used to track a ective response, still underexploited in MMLA), a wearable wristband5 will also be part of the setup. Given their (still quite high) cost, only one or a few of these sensors will be part of the initial setup, probably to be worn by the teacher or randomly-selected students. Additional wearable sensors like mobile eye-trackers (already used, for instance, in \multimodal teaching analytics" [4]) are also being considered. { As an a ordable and more exible alternative to record and stream individual student activity, (cheap) smartphones will be worn by students, logging accelerometer and indoor location/proximity (see next point), as well as audiovisual feeds, if needed. { Beacons and stickers6 will be placed in strategic places in the classroom, as well as in potentially interesting classroom elements (laptops and tablets, interactive and traditional whiteboards, etc.) in order to track the positions of the di erent actors in the learning situations. These sensors often can also be used to complement ambient readings of temperature or motion. { Aside from the aforementioned researcher-placed sensors, and given the nature of our project (i.e., the use of inquiry-based STEM learning), several teacher- and student-placeable sensors7 will be also made available to people using the classroom, for their own science experiments. 3 For our rst prototypes, user-placeable and easily synchronizable sensors will be used, such as PocketLabs sensors (http://www.pocketlabs.com). 4 E.g., Microsoft Kinect. 5 Such as the Empatica E4, https://www.empatica.com/e4-wristband. 6 See, for instance, commercially available models based on Bluetooth LTE like Estimote's (http://www.estimote.com). 7 Again, PocketLab or similar sensors.
Aside from these sensors aimed to track the physical space and actions of teachers and learners, we intend to merge these data sources with others coming from the digital space, such as: { Learning analytics-enabled platforms to support IBL processes, such as Graasp/GOLABZ8 can provide classic learning analytics metrics (with relatively low semantic value), such as number of student actions in a certain activity, number of students in a certain IBL phase, etc. { Additional, high semantic value metrics will be extracted from ad-hoc questionnaires, tests and assessments developed by researchers or teachers9, aimed at measuring di erent forms of learning gains more directly. 3
In the `Smart School' project we will face multiple challenges, some of which emerge directly from the messiness and complexity of gathering data in authentic settings (where we will have limited control about the process, and we will face strong contextual restrictions, like time, curriculum or e ort): { Pedagogical messiness (or `ecumenism') of schools : even if our project is focused on STEM education and IBL, schools exhibit multiple kinds of teaching/learning approaches at di erent moments (from behavioral teaching of procedural routines to collaborative learning or knowledge building). However, current MMLA is very task- or pedagogy-speci c. Thus, we will need to nd metrics that are able to address this pedagogical richness and still remain useful for teacher and student decision-making, and mappings between these intermediate metrics and the available data sources and analysis methods and models (which can be context-dependent, or at least \locally trained" to account for the speci cities of each classroom). { Physical messiness of schools : given that the setup should be used every day, several times a day, by both adults and children, expensive or delicate equipment should be avoided in favor of heavy-duty sensors and devices. This not only may have an impact on the accuracy of the data sources (making multimodal triangulation even more crucial), but also on how much we can trust the outputs of our automated analyses (making ad-hoc, direct measurements of learning all the more important). { User identi cation (i.e., mapping the di erent parts of a data source to the actor generating it) is a very common challenge in MMLA that has not been solved satisfactorily so far, and is expected to be exacerbated by the messiness of an everyday classroom setting (e.g., devices changing hands, students changing sitting positions, etc.). { Schools are inherently a multi-actor setting : Not only we need MMLA metrics and algorithms that can address our research questions as researchers; 8 http://graasp.eu/ , http://www.golabz.eu/ 9 E.g., powered by Google Forms or similar engines.
we also need (potentially di erent) metrics to aid teachers in run-time decision making, as well as (yet di erent) metrics and visualizations that help students themselves re ect and act upon their own learning progress. { Data gathering, analysis and feedback architecture : We should not disregard other classic challenges of MMLA research, regarding the architecture for gathering data from multiple physical and digital data sources, aligning them (e.g., timestamping issues, etc.), nding suitable and generalizable strategies for data fusion, etc. { Ethics in everyday research : Conducting MMLA ethically, across the whole LA lifecycle, is even more challenging than usual log-based approaches { given the abundance of potentially sensitive data like video or physiological data. Addressing all the subjects' rights with regard to their data (parental and child consent, anonymization, rights to opt-out and be forgotten, etc.) is already di cult in one-o school experiments { the probability of these issues to arise if our setup is truly used every day will multiply accordingly. We are currently considering several approaches to address these issues from the outset:
To record only feature-level data (e.g., video features instead of video) to preserve privacy { however, this will often break the inspectability and accountability of the analysis results (since raw data will not be available to double-check) To use only data sources that are `anonymous by default', like movement sensors, infrared cameras, audio instead of video, etc. In the same line of thinking, data can be associated to devices and not to people. However, these techniques will make it di cult to provide personalized interventions to help support speci c students.
Distinguish between data intended for run-time orchestration-level diagnostics, versus data for personalized support of speci c students (and restrict access to data and modelling accordingly), so that more sensitive data only is used by the actors that actually should act upon it (or by the actor generating the data).
On top of all of the above, we will pay close attention to the emergent eld of privacy-conscious analytics (e.g., being able to analyze and crossreference data without decrypting the data sources, as in [6]). { Finally, and unlike MMLA datasets existing today (which are closed, in the sense that they have a known, de ned amount of data), the smart school setting has the potential to produce data continuously, in a streaming fashion. Even if in a rst phase we will produce closed datasets, the data gathering, alignment, analysis and opening up of such a streaming dataset will provide a host of other interesting future challenges for the MMLA community.
We believe the speci c challenges above are especially relevant for the MMLA community, since they represent typical challenges that we will have to face as multimodal analyses of learning abandon their current status of `emergent/niche technique' and become more mainstream. Of course, general challenges of learning analytics research still apply to this project, such as nding actionable metrics, considering algorithmic accountability (i.e., can we trust the results and act on them?) [2], considering and promoting data literacy among teachers and students [3], etc. 4
Many of the challenges outlined above represent open questions in the MMLA community, in and of themselves, and we hope to discuss them with the rest of the community during the workshop. Furthermore, we would also like to discuss several other, more speci c questions about operationalizing the dataset, including: { What would be a good scale for the dataset (i.e., for how many lessons should we record data), so that the dataset can be useful for the MMLA community? { What should be the scope of the dataset (i.e., record only IBL lessons, vs. record a variety of lessons and approaches) to be most useful for other researchers? { Are we missing other potentially useful sources of data (e.g., sensors, digital traces, etc.) that we could add to the setup at a relatively low cost, and with good reliability? { What exact format should the dataset use? xAPI stores are quite popular in the general LA community, but the MMLA community has also experimented with other formats like providing a virtual machine with all the (sometimes custom) analysis and visualization tooling incorporated into it.
Finally, it is worth noting that this ambitious project is not aiming at the study of a single learning or teaching phenomenon. Rather, the dataset would be aimed for us (and the MMLA community) to explore the potential of these di erent sensors for studying issues as disparate as: the long-term e ect of ambient factors in learning, the divergences between teachers' learning designs and their enactment [4], including the in uence of diverse enactment and discourse routines [5, 1] in learning performance, and many more. In future contributions, and in the dialogue with the rest of the MMLA community, we hope to identify which sensors and analysis techniques are most adequate to study each of these issues.
This project has received funding from the European Union's Horizon 2020
research and innovation programme under grant agreement No. 669074.
2. Drachsler, H., Greller, W.: Privacy and analytics: it's a delicate issue a checklist for
trusted learning analytics. In: Proceedings of the Sixth International Conference on
Learning Analytics & Knowledge. pp. 89{98. ACM (2016)
3. Hirsh, S.: Common-core work must include teacher development. Education Week
31(19), 22{24 (2012)
4. Prieto, L.P., Sharma, K., Dillenbourg, P., Rodr guez-Triana, M.J.: Teaching
analytics: towards automatic extraction of orchestration graphs using wearable sensors. In:
Proceedings of the Sixth International Conference on Learning Analytics &
Knowledge. pp. 148{157. ACM (2016)
5. Prieto, L.P., Villagra-Sobrino, S., Jorr n-Abellan, I.M., Mart nez-Mones, A.,
Dimitriadis, Y.: Recurrent routines: Analyzing and supporting orchestration in
technology-enhanced primary classrooms. Computers & Education 57(