Leveraging multimodal data to augment data-driven coaching policies using machine learning such as reinforcement learning, supervised learning, and unsupervised learning offers the potential to greatly enhance guided experiential learning. In this paper, we discuss an adaptive coaching framework that utilizes both trainee interaction data from a synthetic team training environment as well as data generated from trainee verbal communications during the exercise to support team coaching. We discuss how this multimodal data combined with task performance assessments can guide the selection and empirical evaluation of feedback and coaching strategies. We present how this framework could be applied to crew gunnery simulation-based training events as well as recommendations for the design of simulation activities to best leverage the proposed framework. We conclude with a discussion of a data strategy to ensure data sources are robustly collected and applied across interactive multimodal environments to support multimodal assessment.
Accurate measurement and assessment of taskwork and teamwork skills is an integral for robust adaptive team training. Teamwork skills are observable behaviors and actions that team members engage in to complete a shared goal. A critical step towards the implementation of adaptive instructional systems that support collective and team training is the development of assessment models that can evaluate observable actions and behaviors that are linked to team competencies such as information exchange, team coordination, and team backup behaviors, and provide insights into performance.
Multimodal data analytic approaches that leverage video, audio, sensor-based, and game-trace data can provide insights into learning, engagement, and the social cognitive behaviors that shape learning experiences. Leveraging multimodal data to augment data-driven coaching policies offers the potential to greatly enhance guided experiential learning and model team competencies at the individual and team-levels.
In this paper we discuss an adaptive coaching framework that utilizes both trainee interaction data from a synthetic team training environment as well as data generated from trainee verbal communications during the exercise to support team coaching. We discuss how speech-based assessments can be used to measure coordination, information exchange, and leadership skills, which are difficult to assess with game trace data alone. We present how this framework could be applied to crew gunnery qualification drills, as well as recommendations for the design of simulation activities to best leverage the proposed framework. We conclude with a discussion of a data strategy to ensure data sources are collected in an appropriate manner and applied across interactive multimodal environments to support multimodal assessment of teamwork competencies.
Simulation-based training environments are commonly used to support experiential learning and
skill rehearsal in aviation [
Assessing teamwork processes—the actions and behaviors that team members engage in to support effective team performance outcomes—is critical for diagnosing and prescribing coaching and feedback to adaptively support team development during collective simulation-based training events. In particular, assessing team members' spoken communication can provide relevant insight into the circumstances and actions that shape team performance.
Typically, human observers and coaches are responsible for evaluating team performance during simulation-based training exercises. These evaluators may use standardized scoring cards or rubrics to determine whether trainees successfully met performance and training standards.
Evaluating team competencies is a critical challenge because teamwork skills are multidimensional and require measures that can adequately capture the cognitive, behavioral, and affective processes that affect team performance. Additionally, evaluators may not be able to see what trainees are focusing their attention on and the critical cues that trainees attend to that impact decision making during simulation-based training events. An area that has seen growing use to assist with human observers assessment of collective training is multimodal learning analytics.
Multimodal learning analytics is a rapidly growing field that holds considerable promise for
assessing learning and predicting learner states in simulation-based and synthetic training
environments. By using data from a suite of data streams (e.g., video, audio, eye tracking, facial
expressions, body movement, text responses, simulation data) and ensembling it with machine learning
algorithms, multimodal learning analytics can be used to analyze student learning behaviors and model
learning phenomena such as student engagement (e.g., [
At last year’s AIED team tutoring adaptive instructional system’s workshop, we described an
assessment framework to facilitate automated assessment and team-based coaching capabilities in a
course we are creating using the Generalized Intelligent Framework for Tutoring (GIFT) that focuses
on crew gunnery skills [
The GIFT course utilizes a crew gunnery training mission built in Virtual BattleSpace 3 (VBS3). The crew gunnery exercise involves a progression of training tables in which gunnery crews practice skills and procedures for engaging moving and stationary targets from a mounted-weapons system utilizing the direct fire engagement process. A key component of the direct fire engagement process is the coordination of actions and behaviors between crew members, which we aimed to replicate in our assessment model. During an engagement, crew members each have specific responsibilities (i.e., individual task work) as well as collective responsibilities (i.e., teamwork skills) that require them to coordinate actions and exchange information about potential threats in the environment. When a threat has been identified, crews engage in a fire command sequence. This sequence involves the vehicle commander and the gunner following a prescriptive set of directives, spoken commands, and acknowledgements for engaging identified targets. Crews receive points for each engagement with higher point totals representing better performance. The points are calculated using a rubric that incorporates a combination of the firing vehicle’s position, target type, target movement, target range, and target neutralization time. Crews lose points for committing violations outlined in the fire command protocol.
Our current approach for assessing crew performance involves evaluating performance across with regard to engagement performance, crew duties and crew coordination. Engagement performance reflects a training outcome measure and involves measuring how quickly and how well crews score on each engagement. Crew duties reflect the individual taskwork responsibilities of each crew member to ensure they followed the correct protocol for their positions. Crew coordination reflects the team-level processes and actions that affect a well coordinated response during an engagement. 3.1.
To assess crew engagement performance we utilize the GIFT framework along with custom modifications for gathering game trace data in this particular task. For each engagement, GIFT is able to track performance assessments such as “did the crew eliminate all targets within a defined amount of time?”, as well as tracking other performance metrics such as time to engagement, time to elimination, and time to detection. These metrics are defined through custom condition classes developed for GIFT and VBS3, along with some built-in functionality. After an engagement has completed, either through successful elimination of the target(s), or through the timeout threshold being reached, the scenario is then paused to allow for assessment of teamwork performance. 3.2.
Evaluating coordination behaviors and processes requires triangulating across multiple different channels—simulation states, spoken communication, non-verbal behavior—in order to ascertain whether specific teamwork conditions have been met.
We are currently measuring three dimensions of coordination that we adopted from Oliver’s model
of crew coordination [
Our current protocol for assessing crew coordination behaviors includes utilizing GIFT’s GameMaster interface to bookmark and indicate when a behavioral observation linked to a specific coordination dimension has occurred. Because each engagement follows a linear sequence of events, we are able to anticipate when and what crew members should communicate with one another to promote crew coordination. While this approach provides an excellent source of human labeled data for observed actions and communication, we are also integrating multimodal data approaches that will facilitate automated assessment, which we describe next. In particular we gather video, audio, and simulation-trace data during training exercises that can be used to facilitate crew performance and coordination analytics and inform coaching and feedback strategies. 3.3.
A major source of evidence for evaluating crew coordination is team communication data.
Analyzing team communication is inherently multimodal and multichannel. To support assessment of
crew communication, we plan to use the Team Communication Analysis Toolkit (TCAT). TCAT is a
natural language processing (NLP)-driven framework that analyzes team communication data using
machine learning models to assign descriptive dialogue labels to each utterance and extract salient
patterns in the team’s communication [
The NLP pipeline in TCAT begins with automatic speech recognition (ASR) to transcribe spoken
team communication into text for dialogue analysis. TCAT currently utilizes Microsoft Azure's
Speechto-Text cloud service to generate text transcripts from audio files of team members' communication.
Users can import audio files in mp3 or wav format through TCAT's interface, and ASR processes the
audio data for generating output files with written transcripts saved in an Excel format. The second
component of TCAT's NLP pipeline is the dialogue act recognition and information flow classification,
which categorizes team members' utterances into a set of predefined labels indicating different intents
of speech. Due to significant potential for distributed representation learning provided by deep neural
networks, TCAT currently employs a hybrid framework that combines conditional random fields with
ELMo embeddings [
Accurate analysis of team communication data afforded by TCAT can provide insights into
automatic assessment of team performance, as prior research demonstrated significant correlation
between dialogue label frequencies and squad performance, with high-performing squads exhibiting
significantly more utterances labeled as "Provide information" dialogue act and "Command coming
from the team leader" information flow label, while maintaining consistent rates of information
exchange even during stressful events compared to low-performing squads [
With the goal of integrating automated team communication assessment into our course, we have investigated a natural language processing-based protocol to assess fire command statements for completeness and accuracy and developed a set of fire command labels that can be applied to transcripts using TCAT. 3.4.
A key component of Guided Experiential Learning is providing learners with effective and timely feedback. While automated feedback approaches allow for providing evaluative feedback to learners (i.e. a summary of their performance), it is still an open question as to what is the most effective feedback for a given learner, or team of learners, at a given point in their training experience.
Effective feedback should incorporate more variables than most recent task performance. Feedback decisions should incorporate factors such as prior knowledge or ability level, instructor learning goals, and student learning goals when deciding what feedback to provide. For example, in the crew gunnery scenario, if a team had determined to focus on their team coordination skills in a given session, then feedback focused on that area of the exercise may be more effective than feedback about other competencies such as their shooting accuracy.
One potential way to address these challenges is to operationalize adaptive team feedback as a Reinforcement Learning (RL) problem. Reinforcement Learning refers to a family of machine learning techniques focused on creating agents that perform actions in an environment to optimize a numerical reward. While many types of RL frameworks exist, a common framework for pedagogical applications is a Markov decision process (MDP). MDPs are represented as a set of states S, a set of actions A, a set of transition functions T, and a reward function R. Using one of several different optimization functions, the goal of RL is for an agent to learn a policy which determines which actions to take at what states to maximize the reward signal.
In our current work with crew gunnery, the current state S is defined by combining Incoming
Characteristics and Current Mission Characteristics into one state vector. After each engagement in the
crew gunnery scenario, GIFT sends a JSON message to a feedback server, which converts GIFT
assessment information into a state vector consumable by the learner model. Based on the policy being
used by the learner model, a different feedback message or tutorial adaptation is returned. These
represent the set of actions A in the MDP framework. After a training scenario is completed, the trainees
performance on the final engagement is combined with their incoming ability to generate a reward R.
This is done so the policy will optimize for learning gain/improvement, over performance. For more
details on how this task is represented as an RL problem see [
While the proposed framework is designed to align with common training scenarios (i.e. multiple engagements, multiple competencies involved in each engagement), there are still ways that scenarios and training events can be designed to facilitate the training of the RL models. In the design of the models themselves, it is important to limit the size of the state representations and tutorial actions as much as possible, to lessen the requirements for how much data is needed. Offline RL refers to collecting a dataset and training a policy on it. It is important to collect data that samples as much of the state/action space as possible. One method is to map instructor feedback to the actions available to the tutorial agent. This can allow the model to be trained using data collected from authentic training scenarios and would in theory best align with human instruction. Another variation of this is to have the instructor make their decisions based only on data available to the model. However, this can require significant training for the instructor and is not guaranteed to sample the entire space of states, actions, and rewards. Another approach is to utilize a “random” policy, though for training this is often modified to be a “random yet reasonable” policy to avoid negative outcomes by receiving negative feedback for a correct response. Once a large enough sample has been collected, the model can be trained and deployed in future iterations of the scenario.
Another approach to training the model is to use Online RL. In this scenario, the model is bootstrapped with a base policy and constantly updates the policy as more learners complete the scenario. In addition to many logistical hurdles of deploying this type of model, Online RL models must also be defined with a way of handling tradeoffs between exploration and exploitation. In this case exploration refers to trying new actions as opposed to exploitation which refers to the system only performing whatever action it believes to be optimal at that given time. Common approaches to handling this trade off range from author-defined parameters to systems where the exploration rate varies over time as the system collects more information.
For natural language data, while TCAT’s end-to-end NLP pipeline for analyzing team members' spoken dialogue would enhance team assessment capabilities in a variety of experiential learning domains, there are several challenges, which present opportunities for future research and development (Spain et al., 2022). First, barriers to accurate transcription persist, including interference between multiple simultaneous speakers, the impact of speaker characteristics, and environmental noise, on ASR accuracy. To address these challenges, evaluating various cloud-based ASR services and custom models could help identify an optimal ASR system that can effectively differentiate speakers, recognize domain-specific phrases, handle diverse speaking styles and accents, and deal with background noise. Providing additional training data that represents the target data distribution can help improve ASR’s recognition accuracy in noisy environments. Also, using individual microphones instead of a microphone that collect group-level conversation could dispense with the need for speaker diarization.
Another challenge faced is accurate recognition and classification of dialogue acts. As previously noted, we have developed a set of dialogue act labels to support the automatic recognition of key terms and elements for our target domain (i.e., crew gunnery). However, to support accurate recognition, we need to train our classification models using a sufficiently sized dataset. The limited size of the training dataset for TCAT's dialogue act recognition and information flow classification models highlight the potential of pre-training these models using larger spoken communication datasets collected from diverse training events, with which the NLP models could be fine-tuned to attain improved accuracy. Preliminary results suggest that analyzing linguistic features extracted from team communication transcripts can provide insights into predicting certain performance outcomes, such as information exchange and situation awareness ratings (Spain et al., 2021). Building on these findings, assessment models’ predictive accuracy and generalizability of team performance outcomes could be improved by exploring a hybrid approach of top-down (expert-authored rules) and bottom-up (data-driven) approaches, leveraging human-generated evidence-of-performance rules in conjunction with NLPbased methods that follow an evidence-centered design approach for highly predictive and transferable assessments of team performance.
Finally, it will be important to develop accurate squad communication NLP models that can generalize to multiple training domains and tasks that present a distinct set of challenges. The T5-based NLP approach demonstrated promising near-domain transfer capabilities with respect to dialogue act and information flow classification tasks (Pande et al., 2023). Building on these findings, enhancements made to the deep learning-based communication models such as investigating domain adaptation and utilizing intermediate representations across domains can further improve far transfer of NLP models in analyzing team communication data and assessing team competencies by enabling model transfer to other training domains.
A critical step toward the effective use of multimodal analytics to facilitate team assessment is identifying the features and data sources that can be leveraged to support the team competency assessment. In this paper we discuss how we are utilizing a multimodal assessment framework to guide the development of an RL-based crew gunnery model for an adaptive training course.
In the upcoming months, we will collect training interaction data utilizing the GIFT-based crew gunnery course we are developing and use this data to investigate the effectiveness of different forms of adaptive team-based feedback and coaching using an RL framework. We will also investigate the integration of the TCAT NLP-assessment pipeline to provide automated assessment of crew dialog data. Adding these capabilities will facilitate additional research for multimodal learning analytics to inform experiential learning theory and applications.
The research described herein has been sponsored by the U.S. Army Combat Capabilities Development Command, Soldier Center under cooperative agreement W912CG-19-2-0001. The statements and opinions expressed in this article do not necessarily reflect the position or the policy of the United States Government, and no official endorsement should be inferred. 7. References