The technological environment that supports the learning process tends to be the main data source for Learning Analytics. However, this trend leaves out those parts of the learning process that are not computer-mediated. To overcome this problem, involving additional data gathering techniques such as ambient sensors, audio and video recordings, or even observations could enrich datasets. This paper focuses on how the data extracted from the observations can be integrated with data coming from activity tracking, resulting in a multimodal dataset. The paper identifies the need for theoretical and pedagogical semantics in multimodal learning analytics, and examines the xAPI potential for the multimodal data gathering and aggregation. Finally, we propose an approach for pedagogy-driven observational data identification. As a proof of concept, we have applied the approach in two research works where observations had been used to enrich or triangulate the results obtained for traditional data sources. Through these examples, we illustrate some of the challenges that multimodal dataset may present when including observational data.
Learning analytics (LA) is an interdisciplinary field mainly based on data coming
from digital traces and digital realms. In order to understand and optimize the learning
process, researchers pay especial attention to what is happening in computer-mediated
contexts. However, the evidence gathered might be incomplete in real-world learning
activities where there face-to-face and digital spaces are frequently combined [
Research has shown that triangulating pedagogically grounded LA with
teachers’ observational data can be effectively used for teacher orchestration and research
purposes [
From the theoretical perspective, there is a need for frameworks that take into consideration the pedagogical semantics in the data collection, integration and analysis. In addition, from the technical point of view, questions remain open about how to model, collect, and integrate the evidence when heterogeneous data sources used. Therefore, we hypothesize that the LA community will benefit from having an integrated solution that aligns pedagogical semantics with xAPI statements.
This paper proposes, first, pedagogy-aware observational data identification approach. To assess its validity, we have chosen existing research that used observations in combination with LA for different purposes. In order to verify whether the approach could be suitable for these cases, we have applied the approach to the observations of such works. Through this proof of concept, we have identified a set of challenges to be overcome when integrating observations with other LA data. 2
Learning Analytics and educational Action Research are two research areas with
similar goals (while the former uses educational data to foster learning, the later aims to
improve the teaching practice), but different methods (LA draws from automatically
collected data, and Action Research from observations) [
The alignment between LA and Action Research entails the integration of
observations as part of data sources used in the analysis. This step could have a clear impact
on the analytics accuracy and representativeness. In most of the cases, part of the
teaching and learning processes are not supported by technology. As demonstrated by
some authors [
In the following section, we propose an approach that tackles the aforementioned problems from a theoretical point of view. Afterwards, the approach is applied to two research studies in order to verify whether it could support the data gathering, collection and integration of the observational data. 3
Three dimensions were taken into consideration in the design of our approach, namely: • • •
The philosophical and research approach that frames the purpose of the LA study; The educational theory and the pedagogical background that sustains the learning scenario; The technological and architectural aspects that condition the data gathering and integration of multiple and heterogeneous data sources;
This section introduces each dimension, reflecting on those areas when the different dimensions overlap. Afterwards, we describe how this approach affects the data gathering, integration and analysis. 3.1
Philosophical approach. Current data gathering and analysis proposals can be
classified in two main coarse-grained sets. Data-driven generate indicators in a
bottom-up fashion, based on available data. Conversely, model-driven approaches need
pre-specified models that guide the data gathering and analysis in a top-down process.
No matter which approach is followed, the selection and definition of the unit of
analysis plays an essential role. Indeed, the unit of analysis is used a critical instrument to
dismiss one approach or another [
Technological context. Research in this field has suggested that it is possible to
organize several heterogeneous data sources in the form of the xAPI statements and
analyze them with a specific framework in mind [
Educational Theory. In our research, from philosophical point of view, we follow
a constructivist approach. Thus, the goal is to enable learners to become actively
engaged constructors of their own experience and knowledge. This motivation triggers
our interest for understanding the learning activity. In order to track constructivist
learning activities, xAPI is ideally suited [
The observation process is supposed to be carried out by an ad-hoc observer or any participant of the scenario, especially the teachers. The process will be supported by a classroom observation application implemented according to the approach presented in the previous section.
To better understand how the approach would be applied, we describe it through the steps of the common protocol that guides the observation process:
Step 1. Be aware of the elements that belong to the learning context. To facilitate
the data gathering (seen as an observer’s task) and to enable the integration, it will be
necessary to register in all the actors and objects in advance. In that way, the observer
will be able to link the events to the corresponding actors and objects. A first
implementation challenge will be to know in advance not only about the actors and objects
but also to extract the corresponding identifiers which are necessary for later
integration and analysis across data sources. To solve this issue, some authors proposed to
use the learning design and its instantiation in the technological environment as
description of the context [
Step 2. Define the areas of focus, the indicators to be obtained in order to illuminated such areas, and the specific events to be observed. We should not forget that we envision the observations as part of a multimodal dataset. Thus, it will be necessary to define, as a whole, how the different areas of interest are informed by the data sources available, and the trackable events. In the case of the observations, the application will be loaded with the vocabulary necessary to describe the events (xAPI verbs).
Step 3. Collect observable events. In this case, the observations will be recorded
following the subject-verb-object structure, using the set of previously loaded
subjects, verbs, and objects. These events will be presented as xAPI statements that will
be timestamped and sent to a learning record storage together with the rest of the
multimodal dataset. It should be noted that a first study was already carried out to ensure
the whether it was feasible to register the observations following the aforementioned
format [
Step 4. Analyze and interpret the results. The observations will be analyzed with the rest of the events tracked by the complementary data sources, extracting the indicators previously chosen for the different areas of focus.
To better support the integration with other data sources we expect to explore the
definition of vocabularies and xAPI Recipes that help us to take also into account the
context as suggested by Bakharia et al. [
To illustrate the potential of our proposal, we have identified 2 research papers that make use of both observations and LA. This section provides a proof of concept of how such research could benefit from an application that implements the approach described in section 3.
Case 1: The first paper describes a study where the teacher reflects on the aspects
to be evaluated in a learning scenario, selects the data sources that are relevant for
each aspect, and finally choses the events to be used in the LA process [
Case 2: The second research paper applied a multiple data gathering techniques for
triangulation in a face-to-face course supported by technology (observations,
questionnaires, logs, and learning outcomes in the form of text) [
In both cases, the processes followed and the unit of analysis is compliant with the proposal presented in this paper. Thus, the envisioned application could have contributed to automatize and simplify the data gathering and integration processes. 5
In this paper, we have discussed the importance of observational data inclusion into MMLA dataset. Based on the literature review, we have proposed an approach and an observational data aggregation solution. The suggested approach is an integrated view that answers to challenges such as standards (xAPI), pedagogy (semantics) and data source (real world data). Based on the proof of concept, we envision that the presented approach could be suitable for pedagogy-aware real-world, observational data identification, and it could serve a basis for development of observational data collection solution in a form of classroom observation app.
In our future work, both the approach and the architecture will establish the basis of the conceptual model/design of an app that will support the structured data gathering during the observation process, and enable xAPI compliant data export for its integration with other data sources. Design-based research methodology will be applied using scenario-based participatory design sessions that are aimed to validate the presented approach and the conceptual model of the app.