Social media applications have been proposed as a tool to complement students' formal learning experiences, often to increase interactivity and participation. However, evidence regarding the bene ts and challenges of such applications is still con icting. In our latest study to explore this conundrum, we have gathered a multimodal dataset that showcases the teaching and learning processes co-occurring simultaneously on a physical space (face-to-face university lectures) and a digital one (SpeakUp, a social media app). The raw data, provided by di erent sources and informants, were transformed and analyzed using mixed (quantitative and qualitative) techniques. In this contribution, we describe the multiple pieces that composed our dataset, and the steps we took in the multimodal analyses to explore the learning experience occurring in both the physical and digital spaces. This dataset and analysis pipeline illustrates not only challenges and limitations speci c to our study, but also more general ones. Several such challenges and limitations, commonplace in blended learning settings analyzed using mixed (multimodal) methods, are synthesized at the end of our paper.
Technology-enhanced learning (TEL) is, almost invariably, blended in nature:
we create new digital spaces and channels to interact and learn { yet we still
inhabit the physical world and also learn through it. This inherently blended
nature of learning not only has prompted a methodological turn towards mixed
methods [
One example of such research is our ongoing project to study the usage
of social media applications to complement formal, co-located learning
experiences (e.g., face-to-face university courses). Currently there is no consensus as
to whether the bene ts that such applications provide in terms of engagement
and interaction, outweigh their potential cost as a source of distraction [
In this face-to-face course, composed mainly of lectures with more than a hundred students, the usage of a social media tool (SpeakUp3) was proposed in order to foster the (otherwise limited) interaction between students and with the instructors. In a typical usage scenario with SpeakUp, teachers create a chatroom that students can join. Inside the chatroom, any user can anonymously post text messages, comment on existing messages, and up/down-vote them. Moreover, SpeakUp had an added value from the analytics perspective: it allows the chatroom creator to download all the traces collected inside of the room, including both actions and posts.
This paper provides an overview of the multimodal dataset gathered and
the analyses performed in the study. A more detailed account of the setting,
and a partial analysis of the data (including the e ects of SpeakUp usage on
student engagement, distraction, social interaction and teaching style, as well
as their relationship with learning outcomes), are described in [
The dataset for our study was gathered throughout a whole university course on Communication, which involved 6 face-to-face sessions which lasted 90 minutes each. An average of 3 lecturers and 145 students attended per session. We combined quantitative and qualitative data coming from four types of informants (3 teachers, 145 students, 4 assistants, 1 researcher, plus the SpeakUp system itself) using di erent data gathering techniques, namely: questionnaires, observations, video recordings and system logs. Figure 1 o ers an overview of the dataset. In each session, the following data were gathered: { The researcher video recorded the session (focusing mostly on the front of the class) [r vid] { The researcher also wrote down timestamped observations about what was happening during the session (e.g., beginning, start, breaks, activities, topics discussed, interventions, problems with the app, number of students in the room) [r obs] { In parallel, the assistants involved in the course kept track of the students who participated face-to-face in the session (e.g., posing questions or participating in the general discussions) [a obs] 3 http://speakup.info { SpeakUp logs were used to track the activity of the instructors and students joining the chatroom4 (e.g., timestamped posted messages, number of likes and dislikes, etc.) [sp log]
In addition, there were a number of complementary data sources collected at the beginning and at the end of the course: { To understand the student predisposition towards the usage of technology and social media for learning, we conducted a questionnaire at the beginning of the rst session (based on 7-point Likert scale questions) [s que1]. { To gather the student and teacher perceptions on how the tool usage a ected the engagement and attention, we conducted questionnaires at the end of the rst session [s que2] [t que]. While the former was made of 7-point Likert scale questions, the later combined 5-point Likert scale and open questions. 4 Since in SpeakUp, users join anonymously the chatrooms, there was no way to gure out who was behind each user identi er. Thus, we asked the students to freely reveal their identities just for research purposes. It is noteworthy that SpeakUp logs are multimodal since they contain not only activity traces but also the text posted by the users. { At the end of the course, students answered a test composed of multiplechoice questions about the topics discussed in the di erent sessions. The scores [s sco] were used for exploring whether the user's actions on SpeakUp during the topic discussions had an impact on the student answers.
It is noteworthy that, although the study was carried out in an university course on Communication, the data sources used and the data gathering techniques applied are not dependant to the content or the educational context. Thus, the same strategies and techniques could be applied in other settings. 3
As detailed in [
Qualitative analyses. We manually coded all the messages and comments generated during the lesson (thus enriching [sp log]) to determine whether they were relevant to learning, and the direction of the interaction (students to teachers, students to students, students to all, and teachers to students). In a similar way, and in order to understand these topics as they occurred in the face-to-face channel of the classroom, the video recording of the lesson was also coded (enriching [r vid]), according to several categories: which actor was speaking; what topic to appear in the scoring test [s sco] was being discussed, if any; what teacher action was being performed at that moment (e.g., presentation/lecturing, asking questions, providing answers, noting technical or other kinds of problems); who was the target of the interaction, if any (e.g., a teacher, students, or all the class); and nally, what supporting resources were being used, if any (e.g., slides, videos, SpeakUp).
Quantitative analyses. The user activity was measured applying descriptive statistical analyses to the actions tracked in the SpeakUp logs (e.g., number of posted messages, number of likes and dislikes, etc.) [sp log], to the face-toface utterances registered by the researcher and the assistants [r obs] [a obs], and to the results of the manually-coded video [r vid] and SpeakUp comments [sp log]. Furthermore, a clustering analysis (using a k-means algorithm) was performed on each student's activity features [sp log] (number of messages, responses, likes/dislikes, etc.), in order to identify usage pro les. Finally, the data from the users' activity [sp log] were triangulated with the teachers' and students' perceptions from the questionnaires [t que] [s que1] [s que2] and the scores obtained by the students in the nal test [s sco], to understand the impact of such engagement and participation in the learning outcomes.
Despite the richness of the dataset and the usefulness of the analyses described in the previous sections, it presents several limitations, especially apparent in terms of reproducibility and scalability of our approach. While some of these limitations are speci c to the particular implementation of our study, others represent widespread challenges in MMLA that tries to study blended learning settings using a mix of quantitative and qualitative techniques:
Manual data gathering. The fact that several of our data sources originate directly from manual work by human actors (e.g., observations by researchers or assistants). The lack of tooling to easily (and consistently) timestamp, label and export such manual data poses limitations to the scaling and di usion of this kind of e orts.
Data integration. In our study, di erent units of analysis or measure were used. For instance, while the videos allowed us to measure the length of the face-to-face interventions, the logs informed us only about discrete computermediated events, without a duration. This makes merging and comparison di cult, and illustrates a common issue when using multimodal datasets: the heterogeneous nature of the di erent data sources. Despite the e orts put in order to adopt interoperable standards and speci cations (like Caliper or xAPI), this heterogeneity will be hardly avoidable, requiring multiple analysis techniques.
User identi cation and anonymization. The usage of multiple data sources entails the need to identify a certain user across data sources. Computer-mediated user actions are often easy to trace; however, in video or audio data this can be a challenging (if done automatically) or cumbersome task (if done manually). In our particular study, the situation was even more complicated because both the questionnaires and the system logs provided by SpeakUp were anonymous. This brings up the tension between user traceability and privacy, which will be brought to the forefront by the requirements of the recent European General Data Protection Regulation (GDPR EU 2016/679). This kind of regulation may lead, in the near future, to technological tools that only expose anonymous data (hence hindering learning analytics and interventions that target speci c students).
Manual data analysis. As it happens with data gathering, the need for
human involvement during the analysis limits the scalability of our approach. In
our study, both comments and videos were manually analysed by teachers and/or
researchers. Although there are potential solutions that could facilitate the
manual content analyses (e.g., crowdsourcing by letting the users tag themselves the
comments), others like the video analyses remain still a challenge. We
envision that alternative MMLA techniques and approaches (such as speech or text
analyses [
This work was partially funded by the European Union in the context of the Go-Lab Integrated Project (FP7-ICT action, grant no. 317601), SiWay (FP7ICT, IMAILE grant no. 619231-459.76301), the Next-Lab Innovation Action (H2020-ICT, grant no. 731685), and CEITER (H2020-WIDESPREAD-2014-2, CSA action, grant no.669074)