Data fusion and parameterization based on qualitative insights are two key challenges in multimodal learning analytics. In this study, we propose Qualitative Parameter Triangulation (QPT) to address these two challenges. In particular, QPT generate optimized parameter values for the type of multimodal learning models that are event-based, process-oriented and connection-structured concerning with recent temporality.
Humans interact and communicate through variant modes. According to [
However, multimodality is not a simple sum of various modes. Instead, multimodality studies
the relationship between diferent modes [
To study interactional processes of learning, [
To represent an event-based, process-oriented and connection-structured multimodal learning process, there are two major challenges.
First, how to fuse multimodal data with varying time scales and frequencies. Depending
on utilities and assumptions involved, there are three categories of data fusion: naive fusion,
low-level fusion and high-level fusion [
Second, how to elicit a quantitative parameter based on qualitative understanding. In the
operationalization step, that is how to transform the qualitative information to quantitative
parameters for further modeling. This challenge is not uncommon in a mixed-method study
and solutions exist in unimodal analysis. For example, quantitative ethnography [
We propose a formulated approach called Qualitative Parameter Triangulation (QPT) to address the two challenges above.
First, modeling involving QPT doesn’t require data fusion; instead, QPT helps with
determining parameter values in a pre-defined function. That is, the dataset can preserve its raw
representation as long as meeting the requirement of evidentiary completeness, ontological
consistency and terminlogical consistency [
Second, QPT provides a formulated structure to help researchers to elicit their hypothesis on the qualtiative data. Instead of asking directly about the relationship between two modes, QPT automatedly constructs networks based on qualitative researcher’s narratives and optimizes parameter values for the next-step modeling.
QPT can be combined with any models that is event-based, process-oriented and
connectionstructured with consideration of temporalities, such as lag sequential pattern mining, process
mining, etc. In this paper, we use Epistemic Network Analysis (ENA) as an example to
demonstrate how QPT can be used to determine parameters: the window size of diferent modes. We
select ENA due to its afordances of modeling interactivity and interdependence in
problemsolving processes [
QPT helps with eliciting assumptions based on qualitative understanding and optimizes the parameters using automated approach for further modeling. For example, if the goal is to determine the length of active-impact window for diferent modes, the inputs include: (1) human’s qualitative interpretation of connections made, given randomly sampled time points, and (2) number of parameters. Then, QPT outputs the optimized window size for each mode that can be used as the window parameters in ENA. We refers to three key concepts in this triangulation: qualitative story, network representation (connections) and parameter determination. By optimizing the parameters, QPT minimize the diferences between qualitative story and quantitative connections.
Start with a multimodal dataset with evidentiary completeness, ontological consistency and terminlogical consistency. Define parameters needed to describe impacts of diferent modes. In this worked example, we defines the two parameters: eye-gazing and log data.
Step 1: Randomly select K lines from the whole dataset.
Step 2: For each line as a referring line, let the qualitative researcher tell a story about the learning event. For example, in the context of a digital learning game, line 10 is an eyegazing event which captures the player looking at a specific object. The qualitative researcher elaborates on their understanding about why the player looked at such an object. Then, based on the content included in researcher’s narrative, we can determine the connections between codes in a network representation. In the example, there is a connection from code A to itself and a connection from code B to code A. Thus, the entry (1,1) and (2,1) in the adjacency matrix are marked as 1; for the rest of entries, mark as 0.
Step 3: Repeat Step 2 for K time. The coded network structures serve as K ground-truth labels.
Step 4: Use an automated optimization algorithm to determine the parameters for eye-gazing and log-data, which result in the least diferences between the ground-truth labels ( ()) and estimated network when plugging in parameters ( (*)) into ENA.For example, let * be the window of impact parameter for eye-gaze mode. If * is 10, the model assumes that one eye-gaze event will have approximately active impact on all events happening in the next 10 seconds. Similarly, we need to optimize * to derive a complex model.
After QPT provides optimized parameters, researchers close the interpretive loop by checking whether the parameters are aligned with their original understanding. With validated parameters and interpretive alignment, use the parameter values to construct an multimodal ENA model.
We randomly select lines from the dataset regardless of modes. Let a randomly selected line be the referring line (), ∈ {1, 2, 3, ..., }. Each () has a cooresponding adjacency matrix to represent what connections were made between any two codes, determined by qualitative understanding by researchers or domain experts. Let () be the matrix, which represents the presence of connections between two codes. Let () be the binary value to indicate the presence of connections between code i and code j, given the referring line (). If () = 1, there is a connection between code i and code j; otherwise, no connection between code i and code j.
A multimodal dataset may include modes. For each mode , we will determine one parameter () as the length of a temporal window, which describes how long one event from such a mode have impacts on other events. Let P be a vector to record all parameters for multimodalities.
QPT optimizes the vector of parameters P* , which represents the active impact windows for diferent modes with least error. With any model M, which describes interdependence between two events using connections, we can plug in P* into function M. That is, (*) = M(*(), P* ). To derive P* , let () ( ∈ {1, 2, 3, ..., }) be the ground truth. Define L((), (*)) as the loss function which describes the sum of diference between () and (*). We optimize P* by: argmin ∑︁ L((), M((), P* ))
P* Start with random value P0 and use gradient descent algorithm to converge to the local minimum of L. Usually, we use gradient descent algorithm to derive the minimum local mean.
In this paper, we propose a method called Qualitative Parameter Triangulation to determine the parameter values in a multimodal learning model. Specifically, this approach address the challenges of data fusion and parameterization based on qualitative insights. Additionally, the key concept of engaging qualitative researchers in the loop ensures the interpretive alignment, which provides potentials for closing feedback loop with other stakeholders in the multimodal study. Future work are as follows: (1) Use empirical data to test the eficacy of QPT. (2) Try diferent models besides ENA. (3) Try diferent methods of optimization (such as using Gibbs sampling to estimate parameters of diferent modes iteratively). (4) Create multimodal interface to facilitate assumption elicitation for research eficiency and closing-loops in human-computer interactions.