Time series data has been exploited for use with Case Based Reasoning (CBR) in many applications. We present a novel application of CBR that combines intelligent tutoring using Augmented Reality (AR) and prediction. The MonitAR system, presented in this paper, is intended for use as an intelligent guidance system for astronauts conducting complex procedures during periods of a communication time delay or blackout from Earth. Our approach takes advantage of the relational nature of time-series data to detect a task that the user is completing and diagnose the issue when the user is about to make a mistake.
Astronauts are trained in a myriad of di erent procedures ranging from maintenance to emergency medicine. To alleviate mistakes during stressful situations, guidance during such procedures is advantageous. However with longer spaceight missions comes delays or blackouts in communication with Earth. Such instances would bene t from a training and guidance system that is able to direct the user during the procedure and guide them away from potential mistakes before one is committed. For the duration of this paper we refer to a procedure as a NASA procedure for complex tasks and a plan as the representation of a procedure in a planning language. A task is the smallest step in a plan. Further, a case in MonitAR is comprised of a problem, represented by a series of time step features, and a solution that is the previously mentioned task.
MonitAR is a training and guidance system that predicts the task the user's currently completing. If MonitAR predicts the user will make a mistake, it guides the user back to the correct task using visual cues. The system monitors a user's activity while completing a task taking data at set intervals. The data is collected through the camera of an Augmented Reality (AR) smart glasses device. Features are created using the positions of objects in relation to other objects within the view of the AR device. Over time the relationship between the objects indicate the task the user is completing. MonitAR uses partial data to predict the task the user is completing so as to eliminate potential mistakes. A diagnosis of how the user is completing the task incorrectly, such as if the user is completing tasks in the wrong order, is done to create visual cues. The sequential nature of time series data is manipulated during diagnosis which aids in determining the mistake. Learning is employed when the user indicates to the system that they are completing the task in a previously unknown (to the system) way. The MonitAR system aims to aid astronauts while completing procedures in which the astronaut is not an expert or when the astronaut is performing under stress and would normally be given precise instruction via experts on Earth. This can be generalized to aid in any situation where the user is not an expert in the procedure.
The remaining sections of the paper are broken down as follows. Section 2 discusses related work, section 3 gives an overview of the MonitAR system architecture, section 4 describes how time series data is represented in our system. In section 5 the prediction component of the system is described, section 6 details the diagnosis component of the system, and nally the experiment and conclusion are discussed in sections 7 and 8. 2
Prediction and recognition of users and opponents has been a well researched area in recent years. Less so, however, is the prediction of the task the user is completing. The intelligent tutoring community has shown great strides in modeling the user, and determining how best to help them through a task. The AR community has been doing guided procedures for some time in numerous domains. We believe our system which combines intelligent tutoring and prediction using AR is the rst of its kind. Given the current research and state of technology this area of research seems likely to ourish in the coming years. 2.1
Prediction and recognition of human activity using visual data is an active area
of research. Our approach to prediction leans on this existing body work. Pei
et al. [
Synnaeve et al. [
The Intelligent Tutoring community has published some great work on guiding
users towards better learning. Early systems such as presented by Anderson et
al. [
The visual cues MonitAR presents the user in order to guide them back to
the correct task has similar qualities to a traditional constraint based modeling
ITS, as de ned by Ma et al. [
Grasser et al. [
A survey by [
The full MonitAR system can be seen in g. 1. Procedures are taken from the International Space Station (ISS). While the user executes a task in a plan the AR Interface component collects features from the camera using an object recognition library. At each time step, features are passed to the CBR Task Prediction component. During training this component collects features until the task is complete then writes the case to le. During execution a partial case is compiled and retrieval is executed at each time step. If a case is found during retrieval which is over a prediction similarity threshold the partial case, predicted case, and a case determined to represent the current task (the 'correct case') are given to the Diagnoser component. The Diagnoser merges the partial and predicted case and calculates the di erence between this merged case and the correct case using delta cases which are discussed in later sections. This di erence is used to create visual cues within the AR Interface.
Case Base
Difference Calculator
Visual Explanation Visual Cues
• Object Recognition • Object Tracking • Gathers features Time series data is represented in the MonitAR system in two ways. During prediction of a task the data is represented as distance relationships between recognized objects in view. At each time step the distance of each object in view related to each other object in view averaged over the time step length as distance features. A time step feature is comprised of multiple distance features. A short time step length of 500ms was chosen in order to collect enough data to quickly and correctly predict during relatively short tasks. The position of the hand at the beginning and end of the time step is also annotated and added to the time step feature. The annotation of hand position enables the system to reason on how the hand is moving within the length of the time step. A sample case using partial information that indicates the hand reaching towards obj1 and away from obj2 in two time steps can be seen in Fig.2. HandPosBeg: [0,0,0] HandPosEnd: [25,25,0] DistanceFeature
Hand:Obj1 Dist: 100 DistanceFeature
Hand:Obj2 Dist: 50
HandPosBeg: [25,25,0] HandPosEnd: [50,50,0] DistanceFeature
Hand:Obj1 Dist: 75 DistanceFeature
Hand:Obj2 Dist: 75
During diagnosis of the predicted mistake a delta case is created by merging adjacent time step features. To merge time step features the distance in each distance feature of a time step feature for time t+1 is subtracted from the matching (containing the same objects) distance feature from the time step feature at time t. Using the case in 2 as an example, the merged time step feature for the delta case would have two distance features. The distance feature containing the hand and obj1 would have a distance of -25. The distance feature containing the hand and obj2 would have a distance of 25. Delta cases represent the movement of recognized objects between time steps and provide a way of determining the relationship between objects over time. See section 6 for more detail on delta cases and the diagnosis process.
To handle faulty sensors we employ lters using heuristics based on the way the physical world works. When an object which was previously recognized is not recognized in the current time step, distance features are added to the time step feature at the same position as previously seen. In some instances, for example when a hand grabs an object and occludes it, this heuristic fails. To combat this when a missing object is recognized in a di erent location than it was previously and near an object which can move it, such as a hand, distance features are added to each time step feature where that object was missing using the same distance relations as the object that presumably moved it. Lastly, the user can introduce camera jitter due to slight movements even when standing 'still'. We ran a short experiment and found that a typical user will sway up to 15mm so we accounted for this possible distance change in the similarity function. These input lters solve the most signi cant issues found with the camera, occlusion, and the object recognition library. 5
During execution of a task, time step features are created by the AR system and handed to the CBR Task Prediction component. After a time step feature has gone through the input lters, a set of n cases are retrieved from the case base using the similarity function. The similarity function is comprised of two parts. The rst part is a weighted sum of distances between objects. The second part consists of the distance from the current and projected hand position of the partial case q to the current and next hand position of the case base case c. Sequentiality of the time steps enable a projection of hand positions which give the system more information for the similarity function to use, allowing a quicker prediction. These two parts are weighted and added together to create the similarity score. The full equation is shown in Eqn-1. For the weighted sum of distances we choose to weight time step features using linearly ascending weight, , to model that the later time steps better indicate what the user is trying to accomplish. In the following equation m is the number of time step features in the case base case c, n is number of matching distance features between c and q, Qdf and Cdf indicate the distance feature in case q and c, and nally chf and nhf are the current and next hand positions.
sim(q; c) =
P(
P(1 (Qdf Cdf )) )
n m + ( (chf ) + (nhf )) (1)
The top l cases with a similarity over a threshold tsim are brought back from the case base. If any of the l cases have a greater similarity then a prediction threshold tpred the top case is handed to the Diagnoser to component as a predicted case. 6
The Diagnoser component is responsible for determining the di erence between the predicted task and the task the user should be completing. Visual cues are created during diagnosis that show the user the deviation from the correct task via the AR Interface. The Diagnoser conducts the reuse phase of the CBR work ow to adapt the predicted case to the current situation. To do this we rst merge the predicted case and the partial case to create a complete case by taking the time steps t - tn of the partial case and adding the remaining time steps from the predicted case tn+1 - tm. The cases are merged in order to give the Diagnoser the most grounded information possible, rather than relying solely on the predicted case to be similar to reality. Delta cases are created from this merged predicted case and the correct case for the current task.
From the delta cases we are able determine the target object of each case, by which we mean the object that the hand is moving towards. Two assumptions are made here rst that there is exactly one hand represented in the case and secondly whatever the hand is doing is imperative to the task. In future iterations of this system we will generalize this to a generic type of object. To determine the target object the sum of the distance features between the hand, and each other object is found. The largest sum represents the object that the hand traveled to the fastest and therefore is the target of that delta case. Visual cues are created by nding the target objects of the correct case and the merged predicted case. In the instance that the target objects are di erent, a visual cue of a highlighted green box is drawn over the target object of the correct case, while a red box is drawn over the target object of the merged predicted case ( g. 3).
The di erence routine reuses the time step portion of the similarity function with the delta cases. Since the delta case represents the movement of objects between time steps, the similarity function here indicates the similarity of that movement. This is opposed to when the similarity function is used in prediction where the calculation between the partial and case base cases represent the similarity between the location of static objects. The time steps that have a low similarity (or high di erence) under a di erence threshold are compiled to create the visual cues, the largest di erence is used to create the nal visual cue.
We have created a series of distinct image targets (2D images) to label objects such as the hand or vacuum shown in Fig.3 to make the object recognition task easier. The focus of this project is not intended to include development of improved object recognition algorithms. 7
The results of our prototype MonitAR system are encouraging. For the experiment we tested whether the MonitAR system could correctly predict the task the user was completing. Our initial tests were compared against a naive Bayes task prediction. The CBR system was trained on four plans containing a cumulative thirteen tasks. One hundred cases were created during training which encompassed two users (with di erent handedness) completing the plans. The experiment was run using k-fold cross validation with k = 10.
The naive Bayes prediction is calculated during the retrieval phase replacing Eqn-1 with the following Eqn-2. In Eqn-2 c is a case with task , and q represents the current partial case. The conditional p(qjc ) is the probability a case c encompasses the partial case q and has a solution of task . p(q) is the probability that the case encompasses the partial case q. Finally p(c ) is the probability a case c has a solution of task . The top ten cases are returned during the retrieval step. The case with the highest probability, if it is over the prediction threshold tpred, is the prediction. Both methods used the same prediction threshold tpred. p(c j q) = p(qjc ) p(c ) p(q) (2)
There was a signi cant di erence in the percentage of correct predictions for MonitAR using the similarity Eqn-1 and the Bayesian Eqn-2 (p < 0:0001 using a paired t-test). MonitAR gave on average 148 more predictions than its Bayesian counterpart with an average percent correct prediction of 81% when the tpred = :6 and tsim = :4. Even though the propensity to report false positives is higher using MonitAR due to the sheer amount of predictions made, the gains over Bayesian retrieval, that had an average percent correct prediction of just 43% are signi cant. The average earliest correct prediction was better using Bayesian retrieval: 1.04 seconds vs 1.2 seconds for Bayesian and MonitAR respectively. The experiment was rerun with a tpred = :8 and tsim = :4 the results also showed MonitAR correctly predicting the task at a signi cantly higher percentage. Future experiments will be run to determine the best values for tpred and tsim.
If we look deeper into the results, we can see that certain tasks were easier to predict than others (Fig. 4). In particular T3 did very poorly, this can be explained by the nature of the task which asks the user to remove a battery from a power tool. To do so means the user's hand is reaching toward both the battery and the power tool for most of the case. Instances such as this will be addressed in the future with the addition of more ne grained features. Task T2 also did poorly using either method which we believe is due to the length of the task which was very short. We surmise using the Bayesian probability as a con dence score in conjuncture with Eqn1 will bring the overall correctness and timeliness of the prediction up. This will be explored in future work.
Correct Prediction By Task This paper presented early work done on the MonitAR system for task prediction and mistake diagnosis using visual cues. The system is a novel application of CBR to monitor a user's activity and give visual feedback upon the prediction of deviation to a plan. Our system leans on previous work in plan prediction and recognition and has wide applications within training and procedure guidance domains. The MonitAR system has shown promising results in prediction time and correctness when compared to other approaches. Future work will work encompass a full experimental study to determine the best thresholds to employ and weights as well as the addition to more ne grained features. 9
The material is based upon work supported by the National Aeronautics and Space Administration under Contract Number NNX16CJ22P. Any opinions, ndings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily re ect the views of the National Aeronautics and Space Administration. Copyright, 2016, Adventium Labs - All rights reserved.