This paper presents a theoretical framework for multimodal learner modeling, performance analysis, and feedback generation in personalized experiential learning environments. Effective learning and training programs are crucial in modern workplaces that require complex cognitive and psychomotor skills in individual and team settings. Experiential learning environments have gained popularity for teaching these skills in realistic yet safe contexts. Our framework for assessing learner performance in these environments integrates cognitive task analysis, distributed cognition, and multimodal learning analytics. Drawing on competency-based education theories, the framework introduces a hierarchical learner competency model and Bayesian inferencing techniques to assess individual and team performance. This model provides insights into learner performance at different cognitive levels and tracks performance and behaviors over time. The information derived from multilevel learner modeling is then used to generate explainable and actionable feedback for learners and instructors. This feedback aims to support personalized needs and facilitate improvement.
Success in modern work environments requires the effective interplay of complex cognitive and psychomotor skills in individual and team settings for competent performance outcomes. Because of this increasing complexity and demand, effective learning and training programs have to teach these skills in realistic but safe environments. One of the most common paradigms for this sort of learning and training are personalized experiential learning environments, which train learners in realistic scenarios through direct experiences with task material. The increasing proliferation and demand for experiential learning environments has led to the need for effective modeling techniques that can measure and respond to the personalized learning performance and behavior of individuals and teams of learners. However, an effective learning model in such complex environments comes with a set of unique challenges that include assessment of learner performance and learning behaviors.
In this paper, we present a theoretical framework for multimodal learner modeling, performance analysis, and feedback generation to support debriefing and after-action reviews. Grounded in the theories of competency-based education and experiential learning, our framework combines cognitive task analysis, distributed cognition, and multimodal learning analytics to create a systematic method for analysis of learner behaviors and performance, and generation of effective feedback interventions. Our framework begins with a cognitive task analysis (CTA) of the training environment, generating a detailed hierarchical model of the cognitive, behavioral, and psychomotor tasks and strategies employed by the learners. This CTA model is then combined with a qualitative distributed cognition analysis of the captured learner data to generate needs analysis and feature engineering to support multimodal learning analytics (MMLA) that can effectively structure and interpret the raw learner data. Then, building on the theories of competency-based education, this MMLA is combined with a newly introduced hierarchical learner competency model to generate performance assessments. The hierarchical learner competency model utilizes Bayesian inferencing techniques to generate insights about learner’s cognition at multiple levels of abstraction and map them on to performance metrics that can be evaluated across time. Finally, the insights generated by this multilevel learner modeling can be used to generate explainable and actionable feedback, which is presented back to the learners and their instructors.
In the paper, we will demonstrate each component of our learner modeling frame-work using examples of soldiers training on dismounted battle drills in mixed reality environments. We will then present lessons learned from this implementation and how these techniques might be applicable to a larger class of learning and training environments through the lens of this case study.
In Fall 2021, several infantry fire teams participated in a study at the Fort Camp-bell US Army Installation where they performed training on two dismounted battle drills: Enter and Clear a Room (ECR) and Break Contact (BC) using the Squad Advanced Marksmanship Trainer (SAM-T) system. In the ECR drill, each team is tasked with entering a room that they may not have experienced before, neutralizing all enemy combatants, securing all civilians, and then safely exiting the room. The team must follow proper protocols and best practices while entering and operating in the room to minimize the risks of casualties to the team and civilians who may be present in the room. In the BC drill, each team is tasked with exploring a region where enemy forces may be present, and upon making contact with the enemy forces, breaking the contact and retreating to a safe distance. While breaking contact, the team must follow proper protocols and best practices to avoid casualties from enemy or friendly fire.
All the training took place using the SAM-T system, shown in Figure 1, which is a mixed reality battle simulator that projects a Virtual Battle Space 3 (VBS3) simulation onto a set of screens placed in a simulated physical environment. The team moves around in the physical environment while interfacing with the simulation using modified weapons that have a digital interface. In addition, trained instructors can modify the simulation in real-time to help their trainees develop or reinforce skills and become more proficient in the task domain. The SAM-T system records log information related to events that occur in the simulation, and this can include weapon fire messages, entity positions and health states, and so on. In addition, using the data collector in the Generalized Intelligent Framework for Tutoring (GIFT), we have also collected video and audio of the trainees psychomotor, speech, gaze, and posture as they enact tasks in response to the simulation scenario. Data from these sensors are then synchronized with the VBS3 and SAM-T logs automatically. The data recorded in log files can be played back through a GIFT Gamemaster interface to support debriefing and after-action review.
Competency-based education has gained significant attention in recent years as a powerful approach
to learning and assessment. It is a learner-centered paradigm that focuses on the mastery of specific
competencies or skills rather than the completion of a fixed curriculum
The integration of competency-based education with experiential learning has long been the subject
of research in both theory and implementation and is now often referred to as a completely integrated
research space known as competency-based experiential learning (CBEL)
At a high-level, our theoretical framework can be described using the standard Input-Process-Output (IPO) structure as illustrated in Figure 2. As input to the model, we first take multimodal learner data. This data can come in many forms depending on the specific training environment, but often includes things such as video, audio, eye tracking, and system logs. This data is used qualitatively analyzed and used to form the basis of the task context, which is the other input to the system. Since each task has unique attributes that require associated unique analyses, we add the task context to the model as an input. In our model, this task context takes the form of Cognitive Task Analysis and DiCoT Analysis, which will be described in detail in the next sections. Taking in this raw learner data and task context, the model then processes the data using Multimodal Learning Analytics algorithms, which breakdown into three phases. First, we perform action recognition, which takes the raw unstructured learner data and converts it into a structured and interpretable form. Based on these interpretable actions, we then perform as task segmentation, which breaks down the complete training exercise into meaningful smaller chunks which can be analyzed in detail. Each of these chunks is then passed to the hierarchical competency evaluation phase, which takes the learner data and produces a set of assessments based on the expected behaviors defined in the task context. Finally, this processed learner data and associated assessments are passed to the output phase, which formats them into actionable feedback for the end user that can be used to support debriefing and after-action review processes. This learner feedback takes two forms: (1) an annotated learner activity record which allows instructors and trainees to quickly and easily navigate and review a training session and (2) performance analysis over time which allows instructors and trainees to easily monitor learner progress and identify areas for improvement.
Now having laid the foundation for the theoretical framework for multimodal learner modeling and performance analysis at a high-level, we will examine each component and their associated computational processes in depth. The computational structure of the framework is shown in Figure 3 and breaks down into three primary high-level components: the learner data and feedback (purple); the distributed cognition model (yellow); and the multimodal learning analytics (green), which breaks down further into the cognitive task model (blue), the hierarchical competency model (gray), and the computational processes that connect them (orange). 4.1.
At the center of the analysis framework is the cognitive task model, which represents the primary
learner model of the system, and the first component of the larger MMLA block. A cognitive task
analysis (CTA) of the training environment serves as the starting point for understanding the specific
competencies and skills in which the learners have to become proficient. Cognitive task analysis is a
structured approach to building a model with the help of human experts, which helps us explicate the
cognitive processes and skills required to perform competently in a particular domain
The cognitive task model takes low-level observable actions and links them to higher-level strategies and behaviors. Based on these higher-level strategies and behaviors, we can then perform Event Segmentation, which breaks down the complete training scenario into smaller sections that can be individually analyzed by the competency model, which will be discussed later. Event segmentation is an important step for the assessment process, as each higher-level behavior in the task model may have vastly different requirements, and thus be assessed vastly differently. For example, consider the ECR drill in our case study. The requirements of the team when entering the room, including rapid succession, active firing, moving along the walls, etc., are vastly different from the requirements when exiting the room, including taking the shortest path, disengaging weapons, and vocalizing exits. Evaluating the performance of the team in these highly different phases of the exercise requires the ability to segment the task into these separate phases. In other words, making sense of such com-plex behavior requires a systematic way of breaking it down into smaller units. This segmentation into smaller units would then be passed to the assessment model.
However, for complex and open-ended training environments, the cognitive task model alone does not provide sufficient information to completely disentangle high-er-level behaviors. For example, consider movement patterns in the BC drill from our case study. Without further context, we do not know whether simple movement of the team is part of the initial exploration phase or whether it is part of the break phase after contacting the enemy forces. In order to disambiguate to which phase this movement belongs, we have to introduce additional domain information. 4.2.
In our theoretical framework, this information comes in the form of constraints created from the
Distributed Cognition analysis using the DiCoT framework. Our theoretical framework extends the
insights generated by CTA by combining the CTA model with additional domain information provided
by a distributed cognition analysis. Distributed cognition analysis complements the CTA by examining
how learners interact with the learning environment and distribute cognitive processes across tools,
artifacts, and other individuals. This analysis helps uncover the distributed nature of learning and
cognition, highlighting the ways in which learners leverage external re-sources and collaborate with
others to enhance their performance
Continuing the example of BC, our DiCoT analysis might have revealed that in the training environment, the enemy threat is positioned on the screen. So, movement of the soldiers away from the screen would be interpreted as a retreat, which is part of the break phase. From this simple additional information, we can gather that, in our case study, if movement is toward the SAM-T screen, then it is part of the exploration phase, and if movement is away from the screen, it is part of the break phase. In practice, without this additional domain-specific information from the DiCoT model, the task model would be unable to tell what phase of the drill the soldiers are per-forming. These sorts of probabilistic constraints generated by qualitative DiCoT analysis offer a systematic method of applying a-priori domain knowledge to im-prove the overall learner model. 4.3.
However, in addition to these probabilistic constraints, the DiCoT analysis also pro-vides
information to the computational processes of the MMLA pipeline, specifically providing domain
semantics for feature engineering for action recognition, and metric design for the competency model.
While CTA and DiCoT provide a strong theoretical background for our comprehensive framework,
these two qualitative models alone do not provide the tools to analyze learner data at scale, nor do they
provide a means of assessing learner performance. To fill this gap, our framework utilizes multimodal
learning analytics (MMLA). MMLA is a field that explores the fusion of different data modalities to
gain deeper insights into the learning process
Multimodal learning analytics leverages various data sources, including text, audio, video, physiological signals, and interaction data, among others. By integrating these diverse modalities, researchers can extract rich and contextualized information about students' cognitive, affective, and behavioral aspects during learning activities. This interdisciplinary approach draws from fields like machine learning, computer vision, natural language processing, and data mining to develop sophisticated algorithms and techniques that can analyze, interpret, and visualize multimodal data. The ultimate goal is to enhance educational practices, personalize learning experiences, and provide timely and targeted interventions to improve students' learning out-comes.
In our previous work, we have combined MMLA with cognitive task analysis to understand learner behavior in various learning environments, including traditional computer-based tasks and physical experiential learning tasks. In Biswas et al. (2020), we used a CTA approach to analyze trainee performance in a counterterror-ism simulation called UrbanSim. By recording trainee actions within the simulation environment, we observed their performance at various levels of abstraction and employed quantitative measures to capture their cognitive and metacognitive processes. In Vatral et al. (2021), we employed CTA to develop a hierarchical model and corresponding quantitative metrics for the Enter and Clear a Room (ECR) dismount-ed battle drill. Building upon this research, Vatral et al. (2022a) expanded the CTA methods to investigate teamwork behaviors alongside individual performance and develop the extensible H-ABC teamwork model, which connected overarching teamwork concepts to domain-specific performance metrics. This model was later applied in Vatral et al. (2022b) to analyze the ECR dismounted battle drill from a team perspective. In all of this previous work, we applied MMLA methods with various different modalities, including logs, audio, and video, to quantitatively evaluate performance on the given training tasks. Building upon this series of work, in this paper, we formalize the methods of applying MMLA to CTA and DiCoT into our comprehensive framework. Our framework breaks down the MMLA components into two processes: Action recognition and the hierarchical competency model.
Action recognition, in the case of our theoretical framework, is the process of converting raw sensor
data collected from sensors in the environment into interpretable learner actions. Continuing with the
BC drill example, we previously discussed movement toward or away from the screen as an important
indicator, but we did not specify how we were able to interpret that movement was occurring. Within
the theoretical framework, the action recognition block would be responsible for this step. The exact
specifics of the action recognition process depend very heavily on a combination of what sensors are
equipped in the task environment and what actions need to be determined. In our previous work
Regardless of the sensors and algorithms employed in this step, it is the information from the DiCoT analysis which allows us to define a set of features and potential actions that can occur in the environment. For example, in the BC drill, we already showed the importance of movement patterns, but DiCoT could identify other important components of soldier behavior that require specific action recognition; for example, taking cover behind objects in the space, maintaining a low-to-ground posture when not moving, laying down cover fire, etc. The DiCoT analysis defines the important information, features, and potential actions that need to be calculated used the action recognition step.
The CTA and DiCoT analysis also provides domain-specific information to define the Hierarchical Competency Model and its associated assessment steps. The hierarchical competency model is a structured model for providing assessment of learner competency at multiple levels of abstraction. The competency model is structured very similarly to the cognitive task model, containing many of the same high-level components, but instead of actions at the low levels, the competency model has assessment metrics at the lowest levels. These assessment metrics are crafted based on the DiCoT analysis of the environment, as well as consultation with domain experts and relevant literature, and the metrics are designed to be computable based on the directly observable learner data and associated action recognition. For example, when evaluating the BC drill, one metric might evaluate whether the trainees are staying close to cover when they are not moving.
The parallel structure between the competency model and the cognitive task model allows us to take the event segmentation generated by the task model and generate the relevant assessment metrics for each segment depending on what task is being performed. These assessments for each segment are then propagated up the model to generate assessments of higher-level performance. To perform the propagation, we model the competency model as a dynamic Bayesian network, with directly computable low-level metrics representing the evidence variables, and higher-level competencies representing the unobservable variables conditionally linked to the evidence variables. By observing learner data, computing the relevant metrics, and propagating to high-level competencies over time during multiple training instances, we can then model the progression of learner competency over time. For more information on this Bayesian propagation, see Vatral et al. (2022b). 4.4.
Finally, the last component of the theoretical framework refers to the output given to the end user. The overall goal of this research, as well as research in AIED more generally, is ultimately to improve the learning outcomes of the students and trainees in these environments, so feedback to learners and instructors is a critical component of such learning analytics systems. In this work, we present such feedback using data dashboards which are designed to present conclusions from the analysis in a visual and understandable way to the stakeholders. The design of our feedback dashboard is grounded in theories of explainable AI applied to learning and training environments for debriefing and after-action review. The idea is that users can be presented with data and insights generated from the system in an annotated format that allows the users to understand both how the underlying algorithms work, as well as how the data might be useful for improving performance. Different learning environments often have differing requirements for an effective learning analytics dashboard, but three fundamental principles remain the same in the design of each system: (1) feed-back should be objective and data-driven; (2) users should understand how and why the feedback was generated; (3) all feedback should be designed to supplement and assist, but not replace, traditional instructors.
In our case study, we facilitate the feedback generation using the Gamemaster interface in the
Generalized Intelligent Framework for Tutoring (GIFT), whether our case-study system was
implemented
Now that we have described the general structure of the framework and each of its theory-grounded components, we will give a brief summary of how to implement the framework for a new training exercise or domain based on the lessons learned implementing it for both the ECR and BC drills.
The first step toward implementing this framework is building out the cognitive task model. For most training domains, there exists a set of best practices and procedures that trainees are trying to learn. These practices may be explicitly enumerated in literature related to the training environment, but they also may need to be elicited from experts in the field via interviews and co-design procedures. The important les-son to take away from our implementations is to keep the CTA model general at the high levels and domain-specific at the low levels. This structure allows parts of the model to be re-used across related domains and for multiple drills to evidence the same high-level competencies. An example of this kind of structure is the H-ABC model initially developed in Vatral et al. (2022a) and later utilized in Vatral et al. (2022b; 2023b).
After developing the CTA model, the next step is to collect some sample training data on which we can perform the qualitative DiCoT analysis. As previously de-scribed, the DiCoT analysis will break down the training domain into its five themes and code for their associated 18 principles. After performing the initial analysis, we then compare the identified DiCoT principles to the CTA model to find areas of over-lap that form the basis for the domain-specific constraints. To do this, we take a second pass through the sample training data and attempt to manually code event seg-mentation paths through the CTA model. When we reach an ambiguous point where multiple paths through the CTA model could be plausible, such as those described in the previous section, we find associated principles that have been identified in the DiCoT analysis that can help to clarify these ambiguities and generate a constraint. This process is repeated until all of the sample training data can be definitively ana-lyzed using the CTA model and the generated constraints.
With the completed joint DiCoT and CTA model, we then examine the low-level actions in the CTA model and use DiCoT constraints to develop action recognition features. As previously described, this process will differ depending on what data is available in the training environment, but the general idea is to determine how to calculate each of the defined actions and constraints using the available data. For example, in both of our case study drills, movement is a critically important action because of the physicality of the environment. So, we examine the data that was collected and discover that we can approximate the movement patterns of the soldiers using motion tracking on the video data. Then, we examine the constraints important constraints and define an algorithm to compute a path through the CTA model based on the actions (i.e., motion tracking movement data) and the constraints (e.g., movement toward or away from the screen). After defining these algorithms for each action and associated constraint, the event segmentation procedure can be computed for any new data.
Next, we define the hierarchical competency model (HCM). The HCM is structured in parallel to the CTA model, so much of the work for the high levels is already done. However, for the lowest levels of the model, we have to replace the actions in the CTA model with associated performance metrics that can be calculated for each event segment. This process is again largely based on the predefined best practices and procedures of the training task, which will be elicited from a combination of relevant literature, expert interviews and co-design, and qualitative analysis of the sample training data. In general, the metrics in the HCM will parallel the actions in the CTA model as well, but to a lesser degree than the higher levels which can be ported exactly. For example, compare the HCM for the Break Contact drill shown in Figure 4 to the CTA model for the Break Contact drill shown in Figure 5. There are clearly many parallels between the two, such as the Maintain Cover metric parallel to the Take Cover actions, for example, but the metrics are a more detailed breakdown of actual performance, rather than just the actions taken. Once defined, the low-level metrics can be computed for each event segment identified by the CTA model and the higher-level competencies can be inferred using the Bayesian inference procedures described in Vatral et al. (2022b; 2023b).
Finally, once all of the above components are defined, we can define the feedback component of the framework. This component will again depend heavily on the ex-act specifics of the training environment, but there are some strategies which are readily available from the framework’s previous steps which can be easily leveraged to generate feedback. First, the framework provides a method of generating an annotated learner record using the action recognition and event segmentation. For example, in our case study we generate a video timeline which is annotated with the basic actions of the soldiers (e.g., they entered the room, they took cover, etc.), as well as the higher-level segments of the soldiers’ actions (e.g., entering the room vs securing the room vs leaving the room, or exploration of the area vs retreat from contact). Second, the framework provides a method of generating a progress monitoring visualization. Since the basis for evaluation in the framework is a multi-level competency model, we can easily plot the performance at each level of the model over the course of training to give students and instructors insights into how they are progressing at multiple levels of abstraction. Finally, the framework also provides a method of generating performance comparisons. These comparisons could include trainees to their own past performance, trainees to one another, or trainees to expert models. Regard-less of the type of comparison, since competency evaluations are data-driven met-rics, it is easy to compare performance on similar training drills.
In this paper, we presented a comprehensive theoretical framework for analyzing trainee performance across multiple levels of abstraction using multimodal data from the training environment. Based on the theories of CBEL, CTA, DiCoT, and MMLA, the framework provides a complete analysis path to take raw learner data and develop computational models that generate both learner competency evaluations learner feedback. Throughout the paper, we presented the framework through the lens of an example implementation of soldiers training on dismounted battle drills. We concluded the work by outlining the complete process for implementing the framework with suggestions and lessons learned based on our case-study implementation. Future work should focus on implementing this framework in a variety of settings other than Army battle drill training, which would further validate the frame-work’s efficacy and ensure that it is comprehensive. In addition, future work should focus on developing opensource tools to support the common components of the framework, such as the event segmentation and hierarchical competency modeling steps. These open-source tools would lower the barriers to entry for the framework and allow for consistent implementations that can plug and play between different training domains. With continued development, we hope this framework can represent a comprehensive analysis routine for a wide variety of training domains and play an integral part in standardizing assessment and feedback in competency-based experiential learning environments.