While Ubiquitous Learning Environments (ULEs) have shown several benefits for learning, they pose challenges for orchestration. Teachers need to be aware of the learning process, which is difficult to achieve when it occurs across a heterogeneous set of spaces, resources and devices. In addition, ULEs can benefit from multimodal analyses due to the heterogeneity of the data sources available (e.g., logs, geolocation, sensor information, learning artifacts). In previous works, we proposed an orchestration system with some analytics features that can gather multimodal datasets during the learning process. Based on this experience, in this paper we describe the technological support provided by the system to collect data from multiple spaces and sources as well as the structure of the generated dataset. We also reflect about the challenges of multimodal learning analytics (MMLA) in ULEs, and we pose some ideas about how the system could better support MMLA in the future to mitigate those challenges.
Learning can occur beyond the walls of the classroom, across different physical and
virtual learning spaces such as museums, streets, the natural environment, a Virtual
Learning Environment (VLE, e.g., Moodle), a website, or even a 3D Virtual World
(3DVW) [
Learning situations conducted in ULEs usually happen across multiple contexts
between people, devices and resources (physical and digital). As a consequence,
ULEs require the gathering of pieces of evidence (i.e., data) from the different spaces
in order to achieve a global view of the learning process [
In order to help teachers orchestrate learning situations conducted in ULEs
involving web, augmented-physical and 3DVW spaces, we proposed a system, which
includes some analytics features that were not conceived with multimodal analytics in
mind [
The structure of the document is the following. In the next section, we describe the mentioned architecture and dataset. In Section 3 we summarize an illustrative scenario in which we used the system. Section 4 presents different challenges for MMLA posed by ULEs. Finally, Section 5 outlines some open questions to be addressed in our future work. 2
In Muñoz-Cristóbal et al. (2016) we proposed an orchestration system of ULEs [
Fig. 1 shows the architecture and dataset of the system [
The orchestration system collects data (by means of different sets of adapters [
They also include information about participants, groups and tools to use. ● Generated artifacts: Initial resources created by teachers and learning products generated by students. The system stores the “raw” artifacts (or links to them). ● User-context interaction: Periodic information about the location and orientation of the users. ● User-tool-context interaction: Basic information about the actions performed by the users in the tools, including space information if available (e.g., location): login, logout, access, creation/update/deletion of artifacts. ● User-artifact-context interaction: Similar information, in this case regarding users’ actions with artifacts (e.g., access, creation, deletion, update of artifacts). The system includes some visualization features: At runtime, the visualization is currently limited to the location of users and artifacts. In addition, the system generates monitoring reports according to the teacher's monitoring designs. These reports provide an overview of the aspects identified by the teacher, showing whether the gathered evidence satisfies teacher’s expectations, and alerting the teacher in case of potential problems (e.g., none of the students of a group accessed a specific resource they had to access). It should be noticed that, despite storing the generated artifacts, current analyses have not been applied yet to their content. 3
This section describes a study that illustrates the use of the architecture and dataset
structure explained in Section 2. The case consisted in the creation and enactment of a
learning situation, called City-Ads, which was carried out in a course on ICT in
Education for pre-service teachers in a Spanish university [
The teacher designed the learning situation using the WebCollage authoring tool
[
City Ads included six activities that were carried out across different physical and virtual spaces. The first activity was a lecture in the classroom about the different types of ads. In the second activity, each student had to select and take a photo of ten advertisements in the streets of the city (the photos were automatically geolocated and integrated with the rest of tools: wiki, AR browsers and VGs). During the third activity, the students used the VGs in the classroom to explore the virtual view of the 1 https://www.google.com/docs/about/ 2 https://www.layar.com/ 3 https://en.wikipedia.org/wiki/Junaio 4 https://www.google.com/earth/ 5 https://www.google.com/streetview/ city, to access the photos of the ads, and to write reports of critical analyses of the ads (using Google Docs that they created inside the VGs in the location of the analyzed ads). In the fourth activity, the students had to create a counter-ad campaign about one of the ads of their mates, and place the resulting document in the location of the corresponding ad in Google Earth. During the fifth activity the students had to access the different artifacts created (photos, reports, counter-ad campaign) at the physical location of the ads in the streets, using AR. Finally, in the sixth activity, the students created a report about the possible use in education of the different technologies utilized.
The dataset resulting from the City Ads study follows the structure described in Section 2. Concretely, the dataset was made up of: the learning design generated by WebCollage; the instantiated learning design generated by GLUEPS-AR; the learning artifacts generated by teachers and students (learning resources, pictures and reports); the path (geolocation and 360º orientation) followed by the students both when using an AR client and when using a VG (in the latter, what is stored is the path followed inside the VG); the access to the different tools and artifacts, and operations performed over artifacts (create, delete, update), including the geolocation of the actions if available.
City Ads is also useful to illustrate some of the advantages and limitations of the orchestration system for MMLA. The main advantage is that it did not require the adhoc embedding of additional instruments or data sources beyond those already employed by the teacher and the students in previous editions of the situation. Other MMLA approaches tend to require artificial settings, with many sensors, videocameras, beacons, etc., in order to capture different types of pieces of evidence about what it is happening. On the contrary, in City Ads, all the information was gathered from the very same tools and devices that the participants were naturally using in the learning situation (VLE, mobile phones, etc.). Moreover, most of the tools were existing tools, many of them already known by the participants (e.g., their usual VLE, Web 2.0 tools, etc.).
However, the main limitation of the orchestration system is that the current data analysis does not fully exploit all the potential advantages of the available dataset. A better MMLA support could have provided the participants with, e.g., runtime and post-hoc indicators regarding performance, location, completion of activities, problems, technical failures, and also prediction of potential issues. This information would have helped detect problems that occurred, such as the work-overload of the students, who were not able to complete some activities; failures in the geolocation of some artifacts; a breakdown of the system during a whole weekend; or the lack of understanding by the students of some of the topics covered in the activity (which was detected in the final report).
During our research on orchestration of ULEs, we identified some issues for MMLA. Table 1 describes different problems that we addressed, and that we consider can be useful to illustrate the challenges that ULEs pose to MMLA.
Data gathering for MMLA is especially complex in ULEs given the need to gather data from multiple different spaces. These include physical spaces outdoors that impose technological constraints such as the need for using battery-operated devices and that sometimes show unpredictable conditions regarding aspects like available bandwidth, GPS coverage that hinder data gathering. The integration of data for later analysis can also be considered more difficult in the case of ULEs, given the need for integrating pieces of evidence coming from different spaces that might have been generated by a same event. Concerning the analysis, we have found some difficulties to gather the amounts of data required to use some learning analytics techniques such as machine learning algorithms. Furthermore, we believe that different spaces in ULEs demand distinct visualizations that should be researched taking into account the specific characteristics of each type of space and the activities that can be carried out in them. 5
This paper reflects on the multimodal features of a monitoring system for ULEs proposed in our previous work. Concretely, we have described the technological support provided by the system to collect data from multiple spaces and sources as well as the structure of the generated dataset. Finally, based on our experience monitoring ULEs and the lessons learnt during the workshop, we have extracted a list of challenges to be addressed by the MMLA community. These challenges affect the different phases of the LA processes, going from the data gathering and integration, to the analyses and visualization.
In our future work, we expect to extend our system to better exploit the MMLA affordances. More concretely, we are considering to: enrich the dataset (e.g., including adaptors for wearable devices); introduce complementary analyses that contribute to a more holistic view of the learning process (e.g., focusing not only on the user activity but also on the user products); increasing the accuracy of the analysis triangulating multiple data source; and, involving final users in the monitoring process so that they can provide evidence or amend the results. resources, all of them with different technological constraints (e.g., internet coverage, bandwidth, battery, etc.). ● Physical spaces have very dynamic and sometimes unpredictable conditions (related to weather, light, location, coverage …). ● A same event or action can generate multiple pieces of evidence in different spaces, resources or devices, requiring identifying duplicates and complementary information. ● Need for integration of evidence from different spaces to have a complete view of the learning scenario. ● Synchronization of different
pieces of evidence. ● Integration of pedagogical intents with learning analytics involving multiple spaces. ● The amount of data collected in many cases is not enough to apply many learning analytics techniques. ● Since the location of the activities can be dynamic and emergent, the analysis may require contextualization. ● Different spaces may demand
different indicators.
Visualization ● Different spaces may demand different visualizations.
granularity and the frequency of the data gathering varied for each data source.
There were places where the GPS signal was not available and the geolocation of the artifacts generated by the students was incorrect.
The operation over a student artifact (e.g., creation, access) was registered by different elements of the architecture, but the geolocation was not registered by some of them.
The same artifact generated in a space (e.g., a picture in a street) was subsequently accessed from a different space (e.g., Google Earth).
In some activities, the students worked simultaneously in different spaces, and their actions were registered by different adaptors, which were not synchronized.
When designing the monitoring process, teachers realized that not every space offered automatically retrievable evidence. Therefore, manual data gathering alternatives (such as observations) had to be included to cover this gap.
In some cases the evidence gathered was simply the accesses to the resources, allowing the inference of very modest indicators.
For certain activities it was crucial to know whether the students had accessed/created the resources in a specific location.
Geolocation was required in actions conducted outdoors and with VGs, but it was irrelevant in actions conducted in web spaces indoor.
Activities in the classroom were overwhelming for the teacher, who tended to lack of available time to consult dashboard. Also, she needed visualization solutions for those cases when she lost visibility of the students (e.g., outdoors activities).