A Data Value Chain to Model the Processing of Multimodal Evidence in Authentic Learning Scenarios

Abstract

Multimodal Learning Analytics (MMLA) uncovers the possibility to get a more holistic picture of a learning situation than traditional Learning Analytics, by triangulating learning evidence collected from multiple modalities. However, current MMLA solutions are complex and typically tailored to speci c learning situations. In order to overcome this problem we are working towards an infrastructure that supports MMLA and can be adapted to di erent learning situations. As a rst step in this direction, this paper analyzes four MMLA scenarios, abstracts their data processing activities and extracts a Data Value Chain to model the processing of multimodal evidence of learning. This helps us to re ect on the requirements needed for an infrastructure to support MMLA.

Keywords

• Multimodal Learning Analytics timodal Learning Scenarios
• Data Value Chain

-

Mul

Introduction

In the last decade the Technology-Enhanced Learning (TEL) community has witnessed the emergence of Learning Analytics (LA) [ 17 ]. A myriad of research works have been published where learning evidence is collected from a learning situation and processed to analyze the situation or to support evidence-based decision making. However, most of these research works only collect and process software logs like Learning Management System (LMS) log. This is an important restriction as they can only provide a partial view of the learning situation. As an example, we can consider the analysis of collaboration in a blended-learning class where a collaborative text editor is used. If students' collaboration is assessed only out of the logs from text editor, we will get a partial view of the collaboration process because interactions may also happen face to face or with the support of other software tools.

In order to overcome these problems Multimodal Learning Analytics (MMLA) was proposed as an LA sub- eld that \triangulates among non-traditional as well as traditional forms of data which represent multimodal learning evidence in order to characterize or model students' learning in complex learning environments" [ 20 ]. To operationalize the de nition of modality in a blended learning context, modality represents the di erent modes of learning progress over different communication channels [ 22 ]. These communication channels represent the interaction between stakeholders and (or) learning resources which can be tracked through di erent techniques like observations, audio, video, observation, and sensors which may end up in one or more than one datasets. Finally, these datasets can be further analyzed based on the low- or high-level features which can help to answer the main objective of MMLA in any speci c case.

MMLA leverages the possibilities of recent digital advancement to provide a holistic view of a learning situation, but it implies a complex technical ecosystem [ 3 ]: multiple modalities of data should be collected, processed and triangulated [ 19 ]; di erent data collection devices are used (e.g., sensors, software tools or questionnaires); and new data visualization tools are required to hide the underlying complexity [ 18 ]. For this reason, building MMLA solutions is a complex and time-demanding task. In a recent review to the topic [ 16 ], we saw that most of the proposals are tailored for a speci c learning situation and it is very di cult to reuse them. Furthermore, there is a lack of guidance about the modelling of di erent data processing activities required in an MMLA solution. This lack of reusability and guidance can potentially hinder the adoption of MMLA proposals, as a signi cant e ort is required to adapt them to each learning situation.

We are working towards a software infrastructure for MMLA that can be adaptable to di erent learning situations. Our approach draws on the Data Value Chain [ 9 ] model to conceptualize the support needed in each phase of the MMLA process by the relevant stakeholders. This paper reports our rst step in this direction. We have analyzed four learning scenarios inspired in real learning situations where MMLA has been used to support multimodal evidence-based teaching and learning practices. In order to process multimodal evidence in these four scenarios, we identify the requirements posed by the involved stakeholders and the external information required to guide the data processing activities of DVC. Hence, to model the data processing activities with the external information and the intended stakeholders, we extract and report a DVC in this paper.

The rest of this paper is structured as follows: First, we outline the state of the art (Section 2). We present four di erent learning scenarios that illustrate di erent cases in which MMLA has been applied (Section 3). Out of them we extract a Data Value Chain [ 9 ] (Section 4), so that we can synthesize the support needed by these scenarios in a single conceptual tool. Finally, we re ect on how an infrastructure could provide a common support to these and other learning situations where MMLA is applied (Section 5).

State of the art

LA community has seen a rapid growth in the last decade due to the possibilities which support evidence-based decision making to educational stakeholders [ 17 ]. However, most of the LA existing projects analyze system logs of the digital platform used in the learning situation [ 7 ]. This mono-modal nature of LA solutions paints a partial picture of the learning process, learning context and the environment where learning progresses. To get a wider and holistic picture, learning evidence collected from one modality needs to be complemented with learning evidence collected from other modalities [ 18 ]. The availability of low-cost sensors and other Internet-of-Things (IoT) devices provides many opportunities to collect learning evidence from multiple modalities [ 18 ].

MMLA is a young research area under the umbrella of LA. MMLA leverages the possibility to collect, process and analyze the multimodal evidence collected from multiple modalities which are available in the digital as well as the physical spaces of a learning situation [ 21 ]. As an approach, MMLA potentially provides pedagogically-meaningful information to support the needs of multiple stakeholders. Unlike other domains (e.g., business, nance, entertainment), multimodal data processing in educational context is complex due to the involvement of multiple stakeholders and cognitive practices [ 10 ]. Hence, to exploit the bene ts which MMLA as an approach can o er in daily teaching and learning practices, recent MMLA studies explore speci c learning scenarios. To the best of our knowledge, most of the scenarios are either controlled or semi-controlled [ 1, 5, 10, 14 ]. In this exploration, the MMLA community has proposed few adhoc and tailored MMLA solutions to t the requirements of speci c learning scenarios [ 1, 5 ]. The heterogeneity and dynamicity of these learning scenarios including the di erent types of learning activities, learning space, learning context, pedagogy, and involved stakeholders adds complexity in developing MMLA solutions which can be adapted to other learning scenarios than to the scenario for what they were developed for [ 4 ]. Moreover, these solutions usually follow di erent data processing activities to process raw multimodal evidence.

MMLA community has started proposing software infrastructure to deal with the reusability issue of most of the existing MMLA solutions. A recent review used the analytical lens of Data Value Chain (DVC) as a model to highlight di erent data processing activities that are being used by existing MMLA architectures to process multimodal evidence of learning [ 16 ]. This study discusses the importance of contextual information of a learning situation which are crucial information to guide the multimodal data processing: the number of students, the number of groups, the attributes of learning situation datasets, etc. However, to the best of our knowledge, the MMLA community has not addressed what type of contextual information is required to guide the processing of multimodal evidence in each data processing activities of DVC. Hence, in order to study the potential of a DVC to model the processing of multimodal evidence, the required extra contextual information of the learning situation, and the stakeholders who can provide such extra information in each data processing activities to make sense out of raw multimodal evidence. In this section, we describe the four MMLA scenarios extracted from our experience in di erent MMLA related research projects. For each scenario, we present its learning space, the type of learning activities, and the stakeholders' involvement and goals. These four scenarios illustrate the heterogeneity of existing learning scenarios and the complexity in implementing MMLA (see Table 1). 3.1

Scenario 1: Open-doors Activity

Iris and Lea are two researchers working on a digital learning lab. They are currently setting up open school approaches where they co-design open-doors events with school teachers to test innovative teaching practices and their impact on learning. In one of these events, a group of 20 students (aged 13 to 16 years old) and one school teacher, Amy, visited the university and participated in learning activities for approximately four hours. The learning activities combined face-to-face and computer-mediated work, and had an emphasis on collaborative and/or inquiry learning, as well as subject integration.

Amy, Iris and Lea plan to study the engagement of each group of students. To this end, they decide to analyze six parameters (totally disengaged, talking to peers, looking to peers, interacting with technology, resources, and other people) from the physical space and four parameters from the digital space (resource access, creation, opening and update). To capture the parameters from the physical space, they use one observation tool through Google Forms 3. Human-observers do the observation based on this form and submit their response every ve minutes during the enactment phase. Moreover, to cover the digital space, they plan 3 Observational form available at http://tiny.cc/adek5y to analyze the four aforementioned parameters from the system log of Graasp4 (a digital platform to manage the Inquiry-Based Learning (IBL)). After the data gathering phase, these two heterogeneous datasets have to be fused and analyzed based on contextual information about the learning situation provided by Amy (the teacher). The analytical results are presented in a timeline chart to show the teacher and the researchers the coherent view of the learning situation based on the multimodal evidence collected across-spaces and enriched with the information about context provided by the teacher.

In this scenario, human actors are responsible of providing the real-time observations that feed the MMLA tool. The teacher is responsible of providing the contextual information needed to enrich the analysis. Moreover, the observation tool needs to be adapted and a signi cant amount of work has to be dedicated to plan, process and exploit the multimodal evidence. 3.2

Scenario 2: Multimodality Supporting Gami cation Research Maria is the teacher of an online-learning university course about Spanish to English translation, usually launched in Moodle with approximately 150 enrolled students. The course activities are con gured to be performed with the Moodle tools (e.g., assessments through quizzes); 3rd party tools (e.g., glossaries through Google Form and Google Spreadsheets), and external social networks (e.g., posting in Twitter), all of them embedded in the LMS. A researcher that works with Maria (Paul) proposed her to include reward-based gami cation strategies to study their e ect on student behavioral engagement. To this end, the course gami cation was co-designed between Maria and Paul. The gami cation design consisted on providing 15 badges associated to di erent activities of the course such as participating in the course glossaries (Google Spreadsheet), watching course videos (H5P), completing course quizzes with a score upper than 90% (Moodle) and posting in Twitter with the course hashtag (Twitter). The gamication is implemented through the GamiTool platform5.

Behavioural engagement in online environments is frequently measured through variables such as the number of page-views, submissions, posts or the activity time [ 6 ]. However, the use of reward-based gami cation provides additional parameters showing the student engagement with the course contents and rewards. One parameter measuring the reward-derived engagement is the time from the moment the student has completed the reward conditions to the moment when the student claims and earns the reward [ 11 ]. Students will potentially show a certain interest on earning rewards when such time is low. On the other hand, students completing the reward-conditions and claiming the rewards at the end 4 Graasp https://graasp.eu/, last access: May 2019. 5 GamiTool (https://gamitool.gsic.uva.es/) is an ecosystem of applications that allow teachers to grain- ne design and deploy reward-based gami cations in multiple LMSs involving activities performed in di erent 3rd party tools. The system allows the implementation of a reward page into the course where students can claim the rewards, being automatically handled by the system [ 12 ]. of the course, potentially show a lower engagement level than the previous students. Therefore, research on reward-derived engagement needs both the timestamp when the reward-conditions are ful lled (from the LMS and the external tools), and the timestamp when the rewards are claimed and issued (from the gami cation platform).

In this situation, the researcher is responsible of retrieving the needed information from the di erent data sources: Moodle, Google Spreadsheets, Twitter, GamiTool, etc. (e.g., through API), isolating the timestamp information, homogenizing the timestamps, and combining them to nally get the expected parameter measuring student engagement. This parameter can be also combined with other behavioural indicators to measure student engagement more precisely. 3.3

Scenario 3: Adaptive treasure hunt

This scenario has been extracted from the one presented in [ 15 ]. Felipe and Eva are the teachers of a Natural Science course in a primary school. In order to help their students understand the features of the local ora, they set a treasure hunt activity in the school playground. During a full session devoted for the activity, 20 students collaborated in four-person groups to spot di erent trees. Each group of students received a tablet device in which, through a custom application, students received hints to nd the next tree. Once the requested tree is reached, they had to scan a QR code that triggered a task related with the current tree.

In previous years, Felipe and Eva experienced that some students found the hints and the tasks too di cult so they were not able to nish the activity. In order to overcome this issue, Felipe and Eva decided to support the treasure hunt with MMLA. They decided to detect struggling students in order to adapt the learning experience of the students and decrease the di culty if necessary (giving more hints or asking for easier tasks). The information required for these analytics comes from both the physical and virtual spaces. On the one hand, a custom application installed in the tablet devices reports the current location of the students every 10 seconds. This information was used to check if the students where in the same place for a long period of time and their distance to the next target tree (registered in Google Maps). On the other hand, when the students accessed each QR code, the server logged the group id that accessed the resource, the initial and nal time-stamp and the score achieved within every associated task. Additionally, teachers can communicate with the di erent groups during the development of the activity by using the custom application (e.g., to solve an unexpected problem).

This case study faced multimodality issues, such as the availability of the data sent by the tablets, due to loss of signal in some areas in the playground; and the aggregation and alignment of data, as the data sources are populating the data in di erent time frames. Moreover, these issues were emphasized as the activity required real time support, so the system had to identify the struggling groups while they were still performing the treasure hunt. 3.4

Scenario 4: Multimodality to Support MOOC Learners Nacho is a lecturer teaching Machine Learning at a well-known university 6. After a successful experience of designing and delivering a MOOC for a rst time, he is planning to deliver a second version of the same MOOC. The course will be launched in the Canvas Network platform, estimating 1000 enrolled participants. The course will be organized in 4 modules (one module per week) involving content pages, forums and compulsory activities -individual and collaborative ones- (e.g., quizzes, peer assignments).

One issue that proved challenging for Nacho during his previous MOOC was the in-time support to the enrolled participants; he devoted a lot of time answering forums' and private messages' questions that in many cases were repeated among learners, when not a signal that participants had not paid enough attention to the content materials. Since his workload is high, he decided to examine the learners' e ort devoted, previous to the communication of a problem to help him control which learner he will assist.

In the current MOOC, Nacho will follow a multimodal approach to tackle the issue of measuring learners' e ort. He will use a system to analyze the learners' self-reported data from the communication threads (posts in discussion forums and private messages) and a dashboard to create a record of the learners' activity traces previous to the communication of the problem. The learners' trace data that the system will consider are the logs available at the Canvas platform (e.g. number of assignments' attempts, time spent in the content material pages, delay of submissions, general course participation, etc.).

Before launching the course, Nacho will have to con gure some conditions, so that the gathered information results meaningful. For example, if several learners state that they face a conceptual problem, but according to their logs they have not watched the course's related video, these learners will receive an alternative way of support and not a direct answer from the instructor (e.g., a noti cation to recheck the video). This way, Nacho can prioritize the learners and see to whom is more urgent to provide support and avoid a possible dropout. Although this approach implies extra work in advance, Nacho is interested to try and see if in that way he can assist rst the learners' who need help and have put their maximum e ort. 4

A Data Value Chain for MMLA

As an step towards the proposal of an MMLA infrastructure that could be reused in di erent scenarios, we analyzed the scenarios described above and we synthesize their data processing activities with the contextual information and the stakeholders which mediate these activities in a DVC. This DVC is represented in Figure 1.

The DVC considers seven multimodal data processing activities which are divided in three groups. The rst group -data discovery- deals with collecting, 6 Reference anonimyzed r o P I n p c u e t s s r E e x q t u r

a i re in d f ? o r m a

d s e A C ttaaenndo itrsaebu )(ttssaaed teh )(ttssaaeed tseo lleup ttrssaaaem ilcvaeeedn llttcceoo iscvaeednd ttaenno lltceo& d • • • a A

n n a R C re ad yS d n irb eom laen free tsae rchn ira taeo teu ve th cn (t o b d s u e e s n u d n d )u ize te a re ta m n s ta

a q a e d l u s e l

e zo r t ire t d en no eh e

P r e p a s r e e t ( s ) • O

• • a C r A S nd lae g d a from trxE d t isaen tsaae irub m tca tsaae frsano rggge ttrcuu repp ,edn O add ()ts tseo lup ftae ()ts rm tae red raed scyn rga tsa f le reu tuo adn taa ad rho izn ()tse ()s ifyn (tsse tsae isen e ) (t d s ) F i g . 1 .

D e r i v e d D V C f r o m t h e s t u d y o f f o u r M M L A s c e n a r i o s

O u t p u t F u s e d d a t a s e t ( s ) S t a k e h o l d e r ( s ) eR IT 78

LASI Spain 2019: Learning Analytics in Higher Education ( h le eP fe i a gh ve fr t - l o ru le (d rm e v - e ta b l a ie sa fu -b th e s a e d io s ) n ed lro ) w o r le an fro an eP a a r m la fr n ly y o ign iss e

r x c m p a s la to lo la d go m ra lg i r a to o e it r re h ch ry it m in h n

m t s e s

O r g a n is In e t d e d g a r t a a t s e e t ( s ) F u s e d d a A t n a s a e l t y ( z s e ) r r a D d A e e lg i a n s p t lu re ro re sa lya V ts se it e e i

h n t c s n m t (s a u tth ts isv ) lre ila e o au su za an ilz lt o a a l n y z o e n d an po Ied d in n h t i s f gh fr y il om teh g h t r c t e r h p u e o c

i m trs la

R e p o r ts D e M c is a io ke n s

F u D s a io ta n D a t a D i s c o v e r y D a t a E x p l o i t a o n annotating, cleaning, synchronizing, transforming and structuring heterogeneous datasets(s); it includes three activities: collect & annotate, prepare, and organize. The second group -data fusion- integrates two datasets based on the features that relate datasets and generate a coherent view of multimodal evidence; it includes only one activity: integrate. The third group -data exploitation- analyzes the fused dataset, visualize the analyzed report and highlight the data points to make decisions; it includes three additional activities: analyze, visualization, and make decisions. Note that this DVC is compliant with other DVCs like Big Data [ 2, 9 ]. Next we provide details about each step: Collect & Annotate: In this activity, we setup the devices and tools to collect multimodal evidence from multiple modalities which may span over digital as well as physical spaces. Once the dataset(s) are generated, the attributes which need to be included in multimodal processing are annotated. This activity currently needs the help of an IT expert to setup devices and platforms to collect multimodal evidence. The list of attributes is the external information which is planned by either teachers, researchers, or both. These actors provide this information during the planning phase, based on the question(s) they expect MMLA solution to answer with the help of an IT expert. For example, in Scenario 3, a custom application is installed in each tablet to receive the location data of the students. Similarly, in Scenario 4, Nacho decides the attributes (like number of visits and time spent) that he is looking to collect from multiple modalities (self-reported data of students and activity log of Canvas). These two di erent datasets and their attributes are annotated.

Prepare: This activity involves tasks to synchronize the di erent dataset(s) under a single reference time zone. Once the dataset(s) are uni ed, they need to be cleaned (like removing missing values) and unwanted attributes need to be removed. The tasks under this activity need external information from researchers like the time zone of every tool or platform that generates a dataset. For example, in Scenario 2, Moodle and Google are hosted in di erent time zones. Hence, to unify these two datasets, they need to be synchronized under a single reference time zone. Similarly, in Scenario 1, Graasp records a total of ten activities of user interaction but teachers and researchers are interested in only four of them. Hence, data related to six unwanted activities need to be removed from the dataset.

Organize: In this activity, dataset(s) need to be structured and aggregated.

Moreover, if needed, selective features should be extracted from the dataset(s). To perform these actions, a list of aggregation and transformation functions are required which are decided by either teachers or researchers. Moreover, a list of values of attributes which de ne the boundary conditions for the aggregation and transformation functions are also needed. For example, in Scenario 1, observational record of students who belong to one group are averaged to generate the group level observation. Also in this scenario, the count of total occurrences of four verbs of Graasp log (which represent the four attributes, planned by teacher and researcher; see Section 3) for every ve minutes time window is calculated.

Integrate: Unlike the usual data integration, this kind of integrations which are based on sets of rules or features, are normally called fusion. This activity merges datasets based on the common features available in them. It requires a list of common features and the relationship network among data sources which are planned by either teachers or researchers. For example, in Scenario 1, the organized dataset of observational responses is merged with the organized dataset of Graasp log based on the three attributes timestamp, and start and end time of the learning activities (start and end time timings help to generate all the ve minute time windows as observation was submitted every ve minutes).

Analyze: This activity includes analytical it, from basic statistical analysis to advanced machine learning algorithms. First, the researcher or the teacher decides the algorithms that have to be used. Then, those algorithms can be used to analyze the fused dataset which represent the coherent view of multimodal evidence of learning. For example, in Scenario 3, once the location data is complemented by the coordinates which were triggered through QR codes, an exploratory analytical algorithm can be implemented to nd out the struggling students. Similarly, in Scenario 2, once the heterogeneous datasets are fused then advanced analytical algorithms can be trained to reveal the behavioral engagement of students.

Visualization: Presenting the results, which include multiple dimensions, to teachers who have limited data literacy, needs careful selection of visualisation tools, techniques and involved algorithms. Once these parameters are planned by teacher and researcher then these can be used to illustrate the analytical results. For example, in Scenario 4, the analyzed dataset which represent the students' e ort devoted to any learning activity (collected from self-reported data and log of digital platform) can be illustrated in a timeline report to illustrate the dataset to the involved teacher.

Make decisions: This activity includes the algorithms which highlight those points of which require attention of the involved stakeholders. To achieve this, teacher and researcher need to provide the rules to nd out such points. For example, the value to lter out the struggling learners should be de ned by the teachers in the case of Scenario 3. Further, this activity would use such boundary conditions to highlight these data points in the report so that it can support multiple stakeholders in decision making. 5

Discussion and Conclusions

The previous section extracts a DVC from four realistic MMLA scenarios derived from di erent research projects. This DVC includes seven data processing activities with the external required information and intended stakeholders in every step to process multimodal evidences of learning. Even if the seven steps can be considered important, there are three of them that are speci cally relevant for MMLA solutions: Prepare, Organize, and Integrate. These three steps have to do with the manipulation of the multimodal data with the contextual information of the learning situation in order to create an aggregated dataset that can be analyzed and reported in the following steps for the sense-making purpose. For this reason, these steps are more relevant in MMLA than in traditional LA solutions. Hence, a data infrastructure that aims to support MMLA should specially focus on these three steps.

Unfortunately, there is not a linear and clearly de ned set of tasks to be done in each data processing activity since data is collected until all the data is aggregated. Instead, some extra information is required for each of these three steps (see Figure 1). Part of this information is related to technical aspects of the data collection process. For example, how to align the data samples and their timestamps is typically an issue. We can expect a technical administrator to provide this information. Some other aspects are related to the activities carried out during the learning situation. For example, the tasks that are carried out by the learners, or how the classroom is con gured, should be known beforehand for the preparation and the organization steps. This information can be expected to be included in the learning design [ 13 ]. Finally, some serendipitous events that may happen in the classroom can also a ect how the data should be prepared and organized. Some examples are machines that do not work properly or atypical student behaviour. These aspects are very di cult to predict and prevent but may have an important impact on the classroom orchestration and the way the data should be organized and interpreted. This kind of information can only be provided by the teacher, or an observer, once the learning situation nishes.

Going back to our initial aim, we see the analysis of these scenarios as an initial step to collect requirements for an infrastructure that supports MMLA in di erent learning situations. If we aim to support specially the data preparation, organization and integration, we need to o er a way to include the technical and pedagogical information mentioned above. This information should be collected from three di erent sources: the technical aspects related to the data collection methods, the learning design, and the classroom observation or orchestration logs. Hence, the infrastructure should include di erent data-input interfaces, as well as a data model able to provide a coherent view of all these con guration parameters.

One limitation of the analysis performed in this paper is that it has focused on data-processing aspects and stakeholders requirements, and has left out data privacy issues. Data privacy is a crucial aspect for the acceptance of learning analytics in general, and has additional implications in MMLA. MMLA approaches consider the use of new data sources, such as IoT devices or data from the learners' contexts, that can constitute new threats to privacy [ 8 ]. We plan to include this factor in future analyses of the problem.

After the analysis of four realistic MMLA scenarios and our previous analysis of the state of the art [ 16 ], the next step is to propose a rst version of an MMLA infrastructure. We are currently working on its architecture and its data model, and we expect to use it in authentic learning situations in the near future. We expect to iterate the design, implementation and evaluation of the architecture for the proposal to mature.

Acknowledgements

This research has been partially funded by the European Union via the European Regional Development Fund and in the context of CEITER (Horizon 2020 Research and Innovation Programme, grant agreement no. 669074). Moreover, this research is partially funded by the European Regional Development Fund and the National Research Agency of the Spanish Ministry of Science, Innovations and Universities under project grants TIN2017-85179-C3-2-R and TIN2014-53199C3-2-R, by the European Regional Development Fund and the Regional Ministry of Education of Castile and Leon under project grant VA257P18, and by the European Commission under project grant 588438-EPP-1-2017-1-EL-EPPKA2KA.

References

1. Chen , L. , Feng , G. , Leong , C.W. , Joe , J. , Kitchen , C. , Lee , C.M. : Designing an automated assessment of public speaking skills using multimodal cues . Journal of Learning Analytics 3 ( 2 ), 261 { 281 ( 2016 )

2. Curry , E. : The big data value chain: de nitions, concepts, and theoretical approaches . In: New horizons for a data-driven economy , pp. 29 { 37 . Springer, Cham ( 2016 )

3. Di Mitri , D. , Schneider , J. , Specht , M. , Drachsler , H.: The big ve: Addressing recurrent multimodal learning data challenges . In: Proceedings of the Second Multimodal Learning Analytics Across (Physical and Digital) Spaces . pp. 6 : 1 { 6 : 5 . CEUR, Sidney, Australia ( 2018 )

4. Di Mitri , D. , Schneider , J. , Specht , M. , Drachsler , H.: From signals to knowledge: A conceptual model for multimodal learning analytics . Journal of Computer Assisted Learning 34 ( 4 ), 338 { 349 ( 2018 )

5. Grafsgaard , J. , Wiggins , J. , Boyer , K.E. , Wiebe , E. , Lester , J.: Predicting learning and a ect from multimodal data streams in task-oriented tutorial dialogue . In: Educational Data Mining 2014 . Citeseer ( 2014 )

6. Henrie , C.R. , Halverson , L.R. , Graham , C.R. : Measuring student engagement in technology-mediated learning: A review . Computers & Education 90 , 36 { 53 ( 2015 )

7. Hernandez-Garc a , A. , Conde , M.A. : Dealing with complexity: educational data and tools for learning analytics . In: Proceedings of the Second International Conference on Technological Ecosystems for Enhancing Multiculturality . pp. 263 { 268 . ACM ( 2014 )

8. Madaan , N. , Ahad , M.A. , Sastry , S.M. : Data integration in iot ecosystem: Information linkage as a privacy threat . Computer Law Security Review 34 ( 1 ), 125 { 133 ( 2018 )

9. Miller , H.G. , Mork , P. : From data to decisions: a value chain for big data . It Professional (1) , 57 { 59 ( 2013 )

10. Ochoa , X. , Worsley , M. : Augmenting learning analytics with multimodal sensory data . Journal of Learning Analytics 3 ( 2 ), 213 { 219 ( 2016 )

11. Ortega-Arranz , A. , Er , E. , Mart nez-Mones, A. , Bote-Lorenzo , M.L. , AsensioPerez , J.I. , Mun~oz- Cristobal , J.A. : Understanding Students Behavior and Perceptions towards Earning Badges in a Gami ed MOOC . Universal Access in the Information Society p. accepted ( 2019 )

12. Ortega-Arranz , A. , Kalz , M. , Mart nez-Mones, A. : Creating Engaging Experiences in MOOC through In-Course Redeemable Rewards . In: Proceedings of the 2018 Global Engineering Education Conference . pp. 1875 { 1882 . IEEE ( 2018 )

13. Rodr guez-Triana, M.J. , Mart nez-Mones, A. , Asensio-Perez , J.I. , Dimitriadis , Y. : Scripting and monitoring meet each other: Aligning learning analytics and learning design to support teachers in orchestrating cscl situations . British Journal of Educational Technology 46 ( 2 ), 330 { 343 ( 2015 )

14. Schneider , J. , Di Mitri , D. , Limbu , B. , Drachsler , H.: Multimodal learning hub: A tool for capturing customizable multimodal learning experiences . In: European Conference on Technology Enhanced Learning . pp. 45 { 58 . Springer ( 2018 )

15. Serrano-Iglesias , S. , Bote-Lorenzo , M.L. , Gomez-Sanchez , E. , Asensio-Perez , J.I. , Vega-Gorgojo , G. : Towards the Enactment of Learning Situations Connecting Formal and Non-Formal Learning in SLEs . In: International Conference on Smart Learning Environments 2019 . pp. 187 { 190 . Springer ( 2019 )

16. Shankar , S. , Prieto , L. , Rodr guez-Triana, M. , Ruiz-Calleja , A. : A review of multimodal learning analytics architectures . In: Proceedings of the 18th International Conference on Advanced Learning Technologies (ICALT) . pp. 212 { 214 . Bombay, India ( 2018 )

17. Siemens , G. , Long , P. : Penetrating the fog: Analytics in learning and education . EDUCAUSE review 46(5) , 30 ( 2011 )

18. Spikol , D. , Avramides , K. , Cukurova , M. , Vogel , B. , Luckin , R. , Ru aldi, E., Mavrikis , M. : Exploring the interplay between human and machine annotated multimodal learning analytics in hands-on stem activities . In: Proceedings of the 6th International Conference on Learning Analytics & Knowledge . pp. 522 { 523 . LAK '16, ACM , Edinburgh, UK ( 2016 )

19. Worsley , M. : Multimodal learning analytics: Past, present, and, potential futures . In: Proceedings of the Second Multimodal Learning Analytics Across (Physical and Digital) Spaces . pp. 1 { 16 . CEUR, Sidney, Australia ( 2018 )

20. Worsley , M. , Abrahamson , D. , Blikstein , P. , Grover , S. , Schneider , B. , Tissenbaum , M. : Situating multimodal learning analytics . In: Proceedings of the 13th International Conference for the Learning Sciences . pp. 1346 |- 1349 . ACM, London, UK ( 2016 )

21. Worsley , M. : Multimodal learning analytics as a tool for bridging learning theory and complex learning behaviors . In: Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge . pp. 1 { 4 . ACM ( 2014 )

22. Worsley , M. , Blikstein , P. : Leveraging multimodal learning analytics to di erentiate student learning strategies . In: Proceedings of the Fifth International Conference on Learning Analytics And Knowledge . pp. 360 { 367 . ACM ( 2015 )