For more than a decade, Boeing has had an ongoing program of research focused on user-centered adaptive learning. These efforts have been concentrated on the development of two different flavors of adaptive learning. Our Intelligent Tutoring System (ITS) provides a rich personalized student-centered learning experience through the modeling of system knowledge, problem-solving rules, and real-time assessment of student performance. The learning experience provides dynamic scenario sequencing, tailored student feedback and student performance summary based on the perceived student strengths and weaknesses. In the second implementation, we have extended the adaptive learning capability to simulation-based instruction with the Virtual Instructor (VI). The VI provides adaptive simulation- or game-based instruction by monitoring student actions and simulation events, evaluating student performance in real time for a complex set of behaviors, providing information, hints, learning feedback and recommendations to the student and/or instructor. In this paper, we will discuss two specific prototypes of adaptive learning leveraging those implementations. In the first, we have been working with the U.S. Army Research Laboratory (ARL) to integrate our adaptive learning capability with the ARL's Generalized Intelligent Framework for Tutoring (GIFT). The product is an integrated adaptive prototype that we plan to evaluate as part of an effectiveness study this coming year. In the second implementation, we are developing an intelligent virtual reality-based teammate to enable training of individual technical and teamwork tasks within an intelligent tutoring environment. This synthetic teammate will be integrated with the VI capability to respond to the student in real time to support team training objectives. We will discuss the successes and challenges encountered as we have developed these prototype capabilities.
Each person is unique. People come from different backgrounds, different beliefs,
different experiences and they have different goals. As a result, people learn at different
rates in different ways. Adaptivity is the ability of a system to alter (change) itself to
better fit or function in a given situation. In order to optimize the learning experience
for a unique person, a learning system should adapt to the individual learner or team
for the specific situation. The notion that instruction should adapt to the learner is not
new. Effective instructors and mentors have adapted to a specific learner’s needs since
humans started teaching other humans. The goal of an Intelligent Tutoring System
(ITS) then is to provide automated instruction equivalent to that of a skilled human
tutor. Automating instructional adaptivity is also not new, but has been somewhat
elusive and various techniques have been tried. As early as 1958, the famed psychologist,
B.F. Skinner experimented with Artificial Intelligence and Behaviorism. ITS
development has gained momentum since the 1980’s, with numerous automated tutors being
developed and applied in both university and Department of Defense settings [
Developing expertise is time-consuming and difficult. So how do we optimize a
person’s performance to most efficiently develop expertise? In his book, Flow: The
Psychology of Optimal Experience, Csíkszentmihályi [
Thus, our approach to ITS development has been on building tools and techniques
to place and keep students in the flow of optimal learning. Studies by Boeing [
Boeing’s approach to a learner-centered adaptive training implementation has evolved over the course of the past few years. Initial implementations focused on creation of an architecture and authoring solution in support of intelligent tutoring. The product of this effort was Web-based, SCORM®-conformant computer-based training. More recent efforts focused on development of simulation-based instruction that monitors student performance in real time and adapts the scenario accordingly. More details of both approaches are provided below. 2.1
Figure 1 is an overview of our implementation of the ITS. The design features 3 components: a Student Model, an Instructional Model, and an Expert Model. The student model implements a profile of dynamically maintained variables, each corresponding to one learning objective. These variables are evaluated over a number of observations. As a result, changes due to learning are reflected across exercises, as the score increases due to correct performance, or decreases as errors are made. The amount that scores are changed can be weighted according to the degree to which the action reflects mastery of the learning objective. Amount of change is also adjusted according to the degree of support provided to the student by the ITS in selecting this action.
The instructional model responds to student requests for help or student errors with information on problem-solving strategies. The specificity of the information increases as additional requests are made or additional errors occur. The instructional model is also tasked with providing within-scenario feedback to guide the student, as well as performance summaries across all learning objectives at the end of the lesson scenario.
Our implementation of the expert model is based on a cognitive task analysis
technique known as PARI, for Precursor, Action, Results, and Interpretation [
Extending the ITS adaptive training approach to the more dynamic simulation-based training environment is the goal of the VI implementation. Whether it be a desktop or networked simulation, or even a game-based simulation environment, there are a number of challenges related to the real-time assessment of student performance and scenario adaptation within this fast-paced environment. Similar to our ITS approach, our basic architecture involves a student, expert and instructional model.
Inferring student intent can be more complex in these dynamic environments where even a large variance from the expert over a period of time is a reasonable alternative and not necessarily a “mistake”. The VI implementations use interpretation based on multiple student actions within the scenario context. Contrasted to the ITS, where a single response to a question was the norm, in the VI students may complete any number of actions. Often times, multiple action sequences are equally correct. Our approach to student modeling utilized behavior trees, where an action is interpreted within the context of a given branching structure. A tree can be activated as behaviors are recognized, and multiple trees may be active in parallel. Performance is assessed against detailed learning objectives and feedback is provided based on the interpretation of performance and in a format that is compatible with the particular simulation. Networking capabilities are employed to report performance to any number of data logging or learning records capabilities.
The VI is set up to run as an independent instructional tool to assess performance and provide student feedback in the absence of a human instructor, or may be employed to enhance instructor-based learning through objective metrics tracking, real-time notifications to the instructor and enhanced after-action review. 3 3.1
The Generalized Intelligent Framework for Tutoring (GIFT) program is a U.S. Army
Research Laboratory (ARL) effort to develop a framework for personalized,
on-demand, computer based instruction to improve the speed and quality of Soldier training
[
The prototype uses an aircraft maintenance scenario with aspects of troubleshooting
and part replacement. It uses the knowledge assessment functionality and individual
difference categorization within GIFT to sequence course content to the student and to
adapt course content based on ongoing student parameter characterization. Our ITS
capability provides lesson content for required learning, and adapts within-lesson
content to maximize a student’s ability to successfully pass lesson modules on the initial
attempt. The final evaluated practice module is completed using our virtual
maintenance capability known as Advanced Deployable Accelerated Personalized Training
(ADAPT) [
The first iteration of the integrated prototype has been completed. Current efforts are focused on development of a test plan for the conduct of an adaptive training effectiveness study. Once the design is complete, any required modifications will be made to the adaptive training prototype in support of the effectiveness study and we will collect data to evaluate which adaptive training implementations resulted in reduced time to competence, improved performance outcomes, more effective training transfer and knowledge retention. Findings from the effectiveness study will be used to modify the framework for GIFT, as well as our adaptive training approach.
Training to develop flight crew coordination skill is gaining focus in the commercial aviation community. Currently, flight crew coordination is embedded within flight scenarios performed in large full flight simulators (FFS). Additionally, there are times when an airline is not able to send two pilots to train together, requiring an instructor to play the role of the other pilot. Consequently, there are limited opportunities to hone these competencies. To address this need, we are developing a Virtual Pilot to enable student pilots to conduct flight crew coordination training on their own, without needing a second pilot, instructor or even the use of a FFS.
The Virtual Pilot can be used in an Augmented Reality (AR) or VR environment. The AR use case is to support crew coordination training when the student is in a traditional training device such as a FFS or flat panel trainer, but another pilot is not available to train. In this case the student interacts with the Virtual Pilot wearing AR goggles. In VR mode, the Virtual Pilot is integrated with a VR flight deck environment and the student interacts using a VR headset. In both cases, the student’s speech, inputs to the flight deck, and movements are used by the Virtual Pilot to determine how to respond. To support cases in which an instructor is not available, the Virtual Pilot is integrated with a version of the VI called the VR Instructor that will guide training, monitor progress, provide feedback and interject events or scenarios into training for the purpose of challenging the student or addressing an identified training need.
When integrated with the VI, the Virtual Pilot is capable of performing assigned flight tasks (e.g., role of the Captain) and interact with the student pilot through speech and gestures. The VI will receive the same data inputs from the student as the Virtual Pilot – speech, flight deck interaction and gestures/head movement – and use this data to evaluate the student’s performance against pre-defined performance measures, as described previously. The VI may provide feedback in terms of verbal or textual comments, or by highlighting areas in the cockpit visually, or even providing a vibration or other tactile indicator, such as in the case of directing the student’s attention to a particular instrument. Additionally, the VI may command the Virtual Pilot to perform a task incorrectly depending upon the teaching point to be made. For example, if the student appears to not be monitoring and responding to Virtual Pilot’s actions, the VI may command the Virtual Pilot to perform a task incorrectly for the purpose of prompting the student pilot to speak up and intervene. 4
This paper describes some unique approaches to creating adaptive training solutions. While focused on different applications, both solutions attempt to employ the same basic underlying concepts to develop expertise based on optimizing a learning experience by adapting to the student. One takeaway is that with multiple approaches to adaptivity, each method has challenges. We have learned through experience that there are strengths and weaknesses of different approaches to modeling students, providing feedback, and adapting content. By integrating the Boeing adaptive learning approaches with GIFT, we have identified certain communication and software compatibility challenges. These will be mitigated as we continue to work on a mutual joint solution.
Adaptive learning within simulated training environments can be challenging if access to information for assessing performance have not been built into the simulator or gaming engine. Many times, simulators communicate performance at the mission level, whereas student performance needs to be evaluated at the switch or button push level. Adding the capability to perform automated performance assessment within the simulation proves costly and time-consuming. However, adding access to the events and data at the switch or button level for evaluation by tools like the ITS and VI have proven to be minimal.
Finally, adaptive training has yet to be widely accepted within the educational community. We speculate that this is due in part to the added cost of creating multiple sources of adaptive content (something that is getting better with continuous improvements in authoring capability), as well as a potential increased time to proficiency based on student performance. While ample evidence documents improvements to student training performance, training transfer and long-term knowledge retention based on adaptive training solutions, there is reluctance to adopt these approaches given the potential added complications of each student progressing at his or her own pace.
Near-term future plans include the conduct of studies to evaluate the effectiveness our own adaptive training approaches. Plans are in work to evaluate the effectiveness of the Boeing/GIFT prototype. We will be using cadets at West Point to assess various manipulations of overall curriculum adaptation in an effort to determine which are best utilized to optimize student performance. Based on the results of this study, we plan to modify the prototype to better meet the needs of students. Work continues on the development of the Virtual Pilot to integrate the adaptive lesson content and feedback with its physical avatar. A study is planned to validate the effectiveness of the virtual adaptive learning with this implementation as well.