motivated to develop conversational systems that can supOur long-term research goal is to develop intelligent sys-port human task learning in physical worlds. Tasks include tems that can support human learning. We are particularlymaintaining and repairing a complex machine such as an interested in developing an approachatpoprenticeship learn- industrial printer or building an artifact such as Ikea furniing which occurs during the physical context of task execu- ture. To be efective trainers, conversational systems have tion and is known to be very efective in learning procedural to be aware of and adaptive to two types of contexts: physitasks such as equipment maintenance or artifact assembly. cal context of task performance and cognitive context of the We describe our initial progress in building such system - human learner. In this paper, we explore how a conversaJarvis - that leverages real-time computer vision, high-level tional system can reason about and adapt to these contexts. inference, and augmented reality technology to monitor and To do this, we bring together deep learning approaches for support human task learning through apprenticeship. computer vision and planning approaches for adaptive instruction reasoning.
actions, the facilitator instructor and the primary learner ACM Reference Format: form a system of joint learning, with the former helping Shiwali Mohan, Kalai Ramea, Bob Price, Mathew Shreve, Hoda the later in achieving critical conditions of learning. The Eldardiry, Les Nelson. 2019. BuildinJagrvis - A Learner-Aware instructor may provide examples through demonstrations, Conversational Trainer.JIonint Proceedings of the ACM IUI 2019 restrict exploration through supervision, provide feedback Workshops, Los Angeles, USA, March 20, 2019. ACM, 9 pages. and encouragement, and scafold learner’s performance. Interactive learning critically distributes the onus of learning 1 INTRODUCTION between the interacting participants. The instructor takes Conversational systems are becoming pervasive in our worldin;itiative in identifying the relevant objects and relationthey support us in accessing information, performing tasks, ships in the shared environment, structuring and decomposand communicating with other people. As strides are made ing tasks, and providing relevant explanations. The learner in AI, ML, and NLP algorithms, it is reasonable to expect takes initiative in actively interpreting the instructions, apthat intelligent conversational systems will become more so-plying them to the current situation, analyzing successes phisticated and will integrate seamlessly with our physical and failures, posing relevant queries that elicit useful inforand online worlds, assisting us in a variety of tasks. We aremation from their instructors, and assimilating information with their world knowledge.
Our research aims at developing intelligent conversational IUI Workshops’19, March 20, 2019, Los Angeles, USA systems that can support human learning vaipaprenticeship. p©e2r0m19iteCdofopyrraicgahdtefomricthpeuirnpdoisveisd. uThisalvoplaupmeresibsyputhbelipsahpeedrasn’dauctohpoyrrsi.gChotepdying Apprenticeship learning is process of acquiring knowledge by its editors. within the actual, physical context of practice. When compared to classrooms where learning is more theoretical and focuses on thewhy, apprenticeship learning focuses on the conversational systems, and human modeling among othhow. Consequently, such training is shown to be very efec- ers. In this paper, we describe the preliminary steps taken tive for learning physical, procedural tasks such as equip-towards building this adaptive conversational system. We ment maintenance or furniture assembly. This paper intro- focus on two problems: duces Jarvis - Just an Augmented Reality/Virtual Instruc- Perceptual state integratioThenc:onversational systion System - that can guide people as they are performing tem should maintain a stable environmental state in new, physical tasks. Through an augmented reality headset, presence of partial vision from ego-centric camera for such as the HoloLens, Jarvis has a shared view of the hu- instruction generation, and man trainee’s world. It can observe the trainee’s workspace Contextual instruction generatio:nThe conversathrough the egocentric camera oHnoloLens and follow hu- tional should adapt its instruction to be most relevant man task execution. If the trainee struggles to make progress given real-time environmental state. towards their task goal, Jarvis provides instructions on how to proceed further through instructional dialog. The dialog To do this, we bring to bear deep learning vision algorithms may incorporate mixed-reality visual elements such as high-[6, 8] and the Soar cognitive architectur7e].[The main conlighting relevant objects, overlaying expected motion, etc. tribution of this paper is integrating deep learning architectures that operate with numeric, continuous representaChallenges for Conversational Trainers tions with rule-based inference architectures that rely on hTere are several challenges that must be addressed in or- relational representations for intelligent adaptive behavior der to build adaptive conversational training systems that in a conversational system. can guide human task performance and learning in physical worlds. There is significant perceptual reasoning com- 2 PRELIMINARIES plexity. First, Jarvis must inspect its video stream in real- We begin by a brief overview of deep learning architectures time to identify relevant objects, their parts, their relation-for computer vision and that of the Soar cognitive architecships with other objects, and their functional state. Then, ture. Later we describe how both of these are brought toit must adapt to partial observability of the domain due to gether in a system that can reliably perform real-time perthe ego-centric camera. As the human trainee looks through ceptual reasoning and provide contextual instructions. HoloLens at the workspace, only a portion of the workspace is visible. The perceptual information changes rapidly as the Deep Learning Architectures for Vision trainee human moves about their workspace. Jarvis must Modern computer vision techniques are based on an old integrate partial perceptual information it receives into aidea called convolutional neural networks (CN8N].s)In[ stable state representation of the environment so that it is this model, paterns are captured in terms of real-valued able to reason about task execution meaningfully. To adapt weight paterns. These paterns are repeated across various instructions to be most relevant, Jarvis must model the task locations with identical weights in order to enable the abilthe trainee is atempting to perform. The task model cru- ity to recognize the same patern in diferent locations in an cially includes relevant objects, relationships, actions, andimage. goals so that Jarvis can dynamically generate plans to help In combination with larger datasets and GPU based comthe trainee achieve task goals based on the current problem putation accelerators, deep versions of convolutional neural state. Human trainees cannot be guaranteed to follow the networks that have many layers of convolutions now domiinstruction script exactly, i.e., human trainees may divert nate the field [ 6]. These deep networks allow paterns to be from a specific script and explore their environment. There- composed hierarchically. Low-level paterns identify edges fore, Jarvis must be adapt to such non-determinism in its or color textures. Higher level paterns identify relevant feareasoning. Further, Jarvis must model the specific trainee, tures such as the presence of eyes. The highest level layers their needs and knowledge, to provide instructions relevant identify semantic entities such as a face. to them. Finally, these instructions should be provided nat- Deep convolutional neural networks typically do well on urally with minimum interference with the trainee’s task large datasets. However, the most popular networks such performance. The conversational training system should be as AlexNet [6], GoolgLeNet [18], VGGNet [17], and ResNet mixed-initiative, providing opportunities to the trainee to [3] have not only shown promising results in large datasets, ask for information relevant to their knowledge and perforb-ut have also revolutionized the field of computer vision mance. through transfer learning, where pretrained weights from hTese challenges are tremendous and require integration these models, specifically those that can identify low-level of algorithms from several diverse AI disciplines includingpaterns can now be used as a starting point for several tasks computer vision, knowledge representation and reasoning,where the dataset size is modest. elaboration rules and suggest an operator to be applied, the selection rules evaluate all the operators that have been proposed, and once a selection is made the application rules make a persistent changes to the working memory graph. This change can only be reversed through another operator-supported process. An operator is the locus of decision making in Soar and this process of proposing, selecting, and applying operator is at the core of deliberative reasoning in Soar.
While specialized architectures have been developed for each task such as image classification and pixel-wise segmentation, we are particularly interested in object detection that localize multiple instance of objects of various classes in an image frame. One example method is the Single shot multibox detector (SSD)9[], which has demonstrated to have high throughput and achieve impressive accuracy. In the SSD architecture, an output array is constructed with distinct dimensions which represent distinct discrete hypotheses about the position, aspect ratio, and scale of an object in each image frame. These dimensions are predicted using a real valued regression function which fine tune the position and dimensions as well as provide a confidence value about Soar has a meta-cognitive process built into it. Where there the class of the object. are no further inferences to be made or there is not a set of rules that can select one specific operator, Soar agent has an impasse. The impasse results in a new sub-state with the preSoar Cognitive Architecture vious state (in which reasoning became stagnant) becoming hTe cognitive reasoning components of Jarvis are imple- a super-state. The sub-state persists until further reasoning mented in the Soar cognitive architectur7e].[Soar has been in the sub-state can determine the next inference step or opused extensively to develop interactive intelligent systems erator for the superstate. For example, if the next operator [4, 10, 12–14] as well as cognitive models of human task cannot be determined for the super-state, the sub-state may learning1[1]. Through its various long-term memories and execute a one-step search to evaluate which available opercorresponding learning algorithms, a Soar agent can repa-tor when applied directly to the superstate results into a resent diferent kinds of knowledge that aid in reasoning goal state. This impasse-driven resolution process is at the about perceptual state as well as instructions. center of all problem solving in Soar.
Working MemoryA: Soar agent’s beliefs about the cur- Long-term Declarative MemorieAsp:art from these rent state are held in its working memory. These beliefs are components, a Soar agent also has an episodic memory that described using relational representations and are derivedis a store of all past experiences of the agent. Episodic memfrom its current and recent sensory data of the world, cur-ory is a graph that eficiently encodes transitions in the workrent goals, and its interpretation of the situation given its ing memory graph over the lifetime of the agent. The Soar goals. The data in working memory is represented as a sym- agent also has a semantic memory that is a store of declarabolic, labeled graph of working memory elements (WMEs). tive facts about the world. The agent can access the contents Working memory input and output bufers provide inter- of these memories by executing a query against them. The faces to the perceptual and interactive interface. WM also information is retrieved in a working memory bufer and is has interfaces to other long-term memories. available for problem solving.
Procedural MemoryS:oar’show-to knowledge is encoded as if-then rules called productions. A rule has two parts: conditions and actions. The conditions part of the rule match against the working memory graph and changes it by either 3 ENVIRONMENT OVERVIEW adding or removing nodes in it as per tahcetions part. The rules fire in parallel. Soar does two kinds of inference:
To study apprenticeship training, we are exploring the domain of equipment operation and maintenance. A printer is a common piece of equipment found in ofice. While the (1) elaboration: Elaboration rules perform non-persistent typical use of printers is fairly straightforward, complicated monotonic inference given a elements in the current tasks such as printing a booklet are not trivial and can benstate. This means that the actions part of the rule is efit from contextual assistance. As printers become more only part of the working memory graph only until the complex in what they can do, maintenance tasks such as conditions part matches the working memory. When replacing various toner cartridges are becoming more chalconditions stop matching, the changes to the working lenging for a typical user. Repairing a complex, electronic memory graph are reversed. machine is challenging for most users of the machine. Fur(2) operator-supported: Operator proposasle,lection, and ther, manufactures have a large variety of oferings that difapplication rules perform persistent inference. Based fer along several dimensions. As a domain, equipment opon the current state,the proposal rules are similar toeration and maintenance provides a wide array of tasks of human trainee canset a task to a goal they are interested in or askwhat’s next? Jarvis responds via text as well. This interaction interface is preliminary in design and we expect Jarvis to interact with the human in a audio-visual conversation in which the next action is described through speech and the relevant components are highlighted in augmented reality. This paper focuses on the knowledge representations and algorithms behind instructional reasoning.
smoothing filter which consists of an appearance threshold and a sliding window across the frames. The appearance threshold is defined as the ratio of the number of times the object is detected in each frame to the total number of frames in the sliding window. For example, if the appearance threshold i0s:8, and sliding window length 2is0 frames, then the objects should have been detected at least 1in6 frames to make to the final detection list. Larger sliding window length leads to more robust predictions, but the final detection list output is slower, so there is a trade-of between latency of detection and stability that must be tuned empirically.
Figure2 shows the perceptual events generated by the vision system. Detected component states such adsocument feeder open, drum cartridge in etc. along with their bounding boxes and detection confidence are included in the event stream. This event stream is fed into the state integrator described below. to reason about the task. These predicates are resolved when there is more information available as the camera moves during task performance. For example, in Figure2 left, the component drum cartridge is not visible. However, if Jarvis was asked to provide assistance forremoving the drum cartridge task, it would assume a the predicateclosed(front_door) and generate an instruction plan. Previous theoretical research [15] suggests using such representation for environments with incomplete information.
Task predicates: Jarvis maintains a set of task predicates that are inferred using the current perceptual information and its long-term model of the world (described in the next section). Examples incluvdiesible(c) when the componentc is in view,accessible(c) when componentc can be acted upon, andrestricted(c) when a componentc is restricted due to some property of the state.
Maintaining Task State World Model. Jarvis has long-term beliefs about the its task As we discussed earlier, the HoloLens camera is ego-centric environment that are stored in its semantic memory indexed and at a time point only has partial perceptual informationby the printer model name. They pertain to the structure of the printer and relationships between diferent components.
In particular, we are interested in modeling three aspects of of the task. For example, Figur2eshows various images collected while the trainee atempts to remove thderum cartridge (visible in the right most image). However, to rea- the task environment: son about and provide assistance for the task, Jarvis must Structure: This part of the model contains a list of all maintain a holistic representation of the current task state. the components (such asfront-door) and their percepTo maintain a stable state representation of the task, we tual states[(open,close]). As we will see later, knowing propose a novel approach of bringing together long-term what components exists in the environment is helpful knowledge of a world model with current perceptions. This in maintaining a holistic task state. approach is implemented in the Soar cognitive architecture Default state predicates: This part of the model enand relies on its memories and inference mechanisms. codes which state predicates can be assumed to be true about which components when no information State Representation. Jarvis’s beliefs about the current state is available from the vision system. For examfprloen,t of the task environment as a set of unary predicates defined door is assumed to beclosed. over the set of relevant components. These predicates are of Occlusion: This part of the model encodes which comfollowing three types: ponents what state occlude which others. For example, front door when closed occludes drum-cartridge andtoner-cartridge.
Restriction: This part of the model encodes which components in what state restrict which others. For example, whenlocked, drum cartridge lock restricts the movement ofdrum cartridge.
Real perceptual predicates: Jarvis uses predicates such asexists(front-door), closed(front-door), orin(drum-cartridge) (describing that the componendtrum cartridge is in the printer) to represent its world. While these predicates are symbolically represented in the cognitive system, they eventually ground out to sub-symbolic information in the deep learning vision system. This connection is unidirectional, i.e. the vision system can State Inference. Jarvis’ state inference system is encoded in inform the cognitive system about existence of certain a set of productions in Soar’s procedural memory and occurs predicates however, the cognitive system has no influ- as follows: ence over the perceptual system’s reasoning & infer- (1) Initialize current task state (CTS): For initializing task ence. state corresponding to a printer, Jarvis queries for a Assumed perceptual predicates: When Jarvis doesn’t relevant world model in its semantic memory. Then, have a complete representation of the task state, it as- it creates a working memory representation of all the sumes a set of perceptual predicates to be true in order components represented in the world model by adding the existence predicate for all of them. It marks the to be removed (by opening thefront door). This is an visibility of each component to be not be true. Let elaboration inference. this working memory representation be the CTS. CTS (5) Infer restrictability: If for a componecn,tthe world represents the beliefs of Jarvis about the state of its model encodes that there exists another component task environment. Initializing the CTS is an operator- c ′ which in a specific state restrictsc and the current supported persistent inference and it changes only when model has a predicate indicatinc′gin the specific state, there is deliberate decision to update it. Persistent CTS c is considered restricted. Jarvis adds a restricted predensures that even when perceptual events flicker in icate forc in its CTS. This is an elaboration inference the vision system, Jarvis’s beliefs about the task are as well.
stable. Every decision cycle, Jarvis’ reads on input from the vision (2) Compare with perceptions: As the perceptual stream system and performs the outlined inference. A typical decibegin coming in, Jarvis compares its CTS with the sion cycle is Soar runs in abo5u0tms. perceptual stream. If a component in the CTS is not visible but the perceptual system is detecting a state Planning Task Actions event for the component, it marks the component visible. Similarly, if a component in visible in the CTS, In the previous section, we described how Jarvis maintains but the perception system is not detecting any state a stable, cohesive state representation of the task (CTS) in a event for the component, that component is marked partially observable environments. Maintaining a cohesive to be not visible. This is an elaboration inference. state is critical for reasoning about the task. Currently, the (3) Update current model: If a component is visible, Jarvis assistance Jarvis provides is limited to what action to do starts tracking any perceptual event detected related next given a task that the trainee is performing and is deterto the component and adds or removes relevant pred- mined using a planning formulation. icates from the current model. For examplef,riofnt
Background planning knowledge. In addition to the state repdoor is visible, Jarvis, upon opening it the CTS will in- resentation, a planning formulation requirgeosaal that is clude open(front-door). This is an operator-supported defined in the terms of the state representation and a set of inference and is persistent. Consequently, if a com- actions defined in terms of their pre-conditions and efects. ponent is not visible, Jarvis maintains the predicates In the current variation, Jarvis knows of two tasrkesm:ovfrom the last time it was visible. These changes are ing a drum cartridge and removing a toner cartridge. These supported by operators and are persistent. goals are represented as a composition of perceptual predi(4) Infer accessibility: If a component is visible, an elabo- cates that perceptual system can detect individually in the ration rule also creates a predicate indicating that the scene. Through an elaboration inference, Jarvis can also decomponent is also accessible to be acted upon. This in- tect when the goal composition is true in the scene. ference encodes the assumption that that if a printer Actions are represented using the operator-supported incomponent is in the human’s field of view (correlated ference in Soar. Operator proposal rules test for the prewith Jarvis’), they can manipulate it. If something conditions of actions against the current working memory is not in the field of view either because the human (containing CTS) and propose the action if there is a match. is looking somewhere else or that they are occluded For example, if the CTS encodes thafrtont door is open, close(frontby another componentfr(ont door occludestoner car- door) will be proposed. Selection of a specific action operatridge in the domain), the occluding component has tor is determined through planning and is described in the following text. Once an operator is selected, the operator application rules will change the CTS in accordance with action part of the rule. Jarvis has two diferent types of operator application rules that are used contextually. These contexts are callepdroblem spaces in Soar. When Jarvis is planning (described in detail in the next section), the application rules encode how the world will change when the particular action operator is applied. I.e, for the acctilosne(frontdoor), the operator application rules encode that the resulting world will contain the predicafrtoent door is closed and not contain the predicatefront door is open. When Jarvis moves into the real-world interactive context, the operator application rules encode a communicative act. For example, for operatocrlose(front-door), the application rule will encode inform(close, front-door). This communicative act is then passed to the human-agent interaction system which may use speech or visual modality for conveying this to the human trainee.
Interactive, Contextual Planning. the goal state. To do this, the selected action operator is applied. The application of the selected operator makes changes in the copied CTS in accordance with the application rules. If the resultant state does not include the goal, an impasse will result again causing sub-state creation and action evaluation to occur recursively. If the resultant CTS contains the goal, evaluation of the action operator will be marked as a success causing the impasse in the super-state to resolve as now the action evaluated a success can be applied. hTe success evaluation will pass up in the recursion chain eventually selecting an action in the initial CTS.
Various kinds of search strategies can be applied adding heuristics about how actions are selected for evaluation as well as constraining the depth of recursion. hTe current implementation uses iterative-deepening search. The output of the planning process is an actionoperator that when applied to the CTS will move the task closer to the selected goal. Let thisrbeesult-operator. (5) Deliver instruction contextually: When the planning operator succeeds, Jarvis moves into the real-world context. In this context, application of threesult-operator generates a communication act of informing the trainee of the action. Currently, the next recommended action is displayed in an interaction window. After completing the action in the world, the trainee can ask what’s next? to trigger planning again. The planning process can be triggered whenever the trainee needs more assistance. A valid plan is generated for the CTS in which assistance was requested. Contextual generation of plans results imnixed-initiative instruction in which the trainee can choose to explore the task by themselves but can ask for support if they aren’t sure how to make progress. (1) Set a planning goal: The human trainee can use the set-task interaction to set a goal for the planner. Upon receiving this instruction, Jarvis instantiates the definition of this goal in its working memory and begins actively tracking if this composition is true in the environment. Upon askingw, hats’ next? after seting a goal, Jarvis can begin planning. (2) Switch problem space (context): Upon being askwedhat’s next?, Jarvis switches to the planning context and creates a copy of the CTS in this context. (3) Complete the world: As we discussed earlier, Jarvis operates under partial-observability. Consequently, it could be the case that when it is asked to plan, it does not know the complete task state. If this is the case, Jarvis applies its world model to include certain predicates in CTS about objects that are not visible to as- 5 SYSTEM DEMONSTRATION sume a certain state about the world. This process re- We can demonstrate real-time contextual instructional supsults in an augmented CTS which is based on some in- port from Jarvis on tasks performed in four scenarios charformation from the perception system and some from acterized by varying trainee behavior.
the world model encoded in Jarvis. (4) Search for the plan: Once the CTS is augmented, Jarvis Regular. In this scenario, the trainee accurately follows Jarvis’ begins planning. The planning process in Soar is build instruction. As shown in Figur3e(left), the task begins in the on itsimpasse structure. In the initial CTS, several ac- initial stateS1 and the trainee selects a taskre-place-tonertions can be taken and consequently, several action cartridge. When prompted for the next action, Jarvis begins operators are proposed. As Jarvis does not have any planning for statSe1 and recommends the actionopen front rules to select between the proposed operators, an im- door to the trainee. The trainee applies this action in the task passe occurs. Due to this impasse, a sub-state is cre- environment and the task transitionsSt2o. The trainee asks ated which will persist until the impasse in the super- for the next action. This interactive task execution continues state is resolved (i.e. a specific action operator can be until the goal state is detected in the environment. selected). In this sub-state, a copy of the CTS is created. hTen, Jarvis evaluates a randomly selected action op- Exploratory. In this scenario, the trainee explores the envierator to see if applying that operator will result inronment by taking actions beyond what was instructed. In
S1 (Figure3), after executing open front-door, the traineeun- Incorrect. In this scenario, the trainee misinterprets the inlocks the toner-cartridge without any prompting from Jarvis, struction. InS1 (Figure 3), upon being instructed toopen consequently landing isn3. Jarvis is able to handle this inde- front-door, the traineeopens the document feeder instead. While pendent action execution by the trainee and when prompted Jarvis is correctly able to reason that the task state still for the next action responds correctly by suggestitnagke is S1 and determine the next actionopen front-door, from toner cartridge out. a training perspective this instruction is not very productive. A good trainer can recognize this as a learning opporInterruptive. In this scenario, the trainee interrupts Jarvis’ tunity and provide more information about the structure of planning operation by executing an unprompted action. In the printer. For example, the trainer can respThaotnids the S1 (Figure 3), as Jarvis is planning, the traineoepens the document feeder. The front-door is in the front, towards the front-door transitioning the task tso2. Jarvis maintains the botom . Currently, Jarvis lacks the representations and reacontext of its planning while it is executing the iterative- soning for such instruction. deepening search. As the context changes due to the trainee’s hTese observations of Jarvis’ behavior directly follow from actions, Jarvis abandons its search and initiates a new search the approach implemented and are arguably trivial. Howwith the current task state. This correctly produces the rele- ever, they are very encouraging because they demonstrate vant actionopen toner cartridge. that these two kinds of architectures can be brought together for reliable behavior in the real-world. The deep learning ar- REFERENCES chitecture can robustly extract state information about vari- [1] John Bransford. 2000H.ow People Learn: Brain, Mind, Experience, and ous components of the machine and Soar can integrate these School. National Academies Press. perceptual events for a stable state representation and in- [2] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zissercorporate structural knowledge of the world for planning. nmaatino.n2a0l1J0o.urThenPaalsocfaClomVipsuutaelr VOibsijoenct8C8,la2s(sJeusn(eV2O0C10))C,3h0a3l–l3e3n8g.eI.nterFurther, these observations illustrate the need for learner [3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep models and relevant pedagogical strategies for reasonable Residual Learning for Image RecognitionC.onference on Computer instructional behavior. Vision and Patern Recognition (2015). [4] J Kirk and J Laird. 2014. Interactive Task Learning for Simple Games.
Advances in Cognitive Systems 3 (2014), 11–28. [5] Kenneth R. Koedinger, Emma Brunskill, Ryan S.J.d. Baker, Elizabeth A.
McLaughlin, and John Stamper. 2013. New Potentials for Data-Driven Intelligent Tutoring System Development and OptimizatiAonIM.ag6 CONCLUSION azine 34, 3 (9 2013), 27. https://doi.org/10.1609/aimag.v34i3.2484 In this paper, we described the initial steps we have taken to [6] Alex Krizhevsky, Ilya Sutskever, and Geofrey E Hinton. 2012. build Jarvis, an intelligent system that can help people learn [7] John E. Laird. 2012.The Soar Cognitive Architecture . MIT Press. [8] Y. LeCun, F.J. Huang, and L. Botou. 2004. Learning methods for physical tasks via apprenticeship training. The use of aug- generic object recognition with invariance to pose and lighting. In mented reality provides an opportunity to provide instruc- Computer Vision and Patern Recognition CVPR , Vol. 2. tional support within the context of task performance. For [9] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, learning procedural tasks in physical domains, this training Scot Reed, Cheng-Yang Fu, and Alexander C. Berg. 2016. SSD: Single can be very efective in comparison to classroom teaching [10] SAhaortoMnuMltiinBionxgeDreatencdtoJro.hInnELCaCiVr.d. 2018. Interactively Learning a Blend in which the student must expend efort to translate theo- of Goal-based and Procedural Tasks.PIrnoceedings of the Thirty Secretical knowledge to the performance context. ond AAAI conference on Artificial Intelligence .
From an AI systems perspective, design of such instruc- [11] Shiwali Mohan, James Kirk, and John Laird. 2013. A Computational tion systems is a significant challenge. The requirement of Model for Situated Task Learning with Interactive InstructionPr.oI-n robust, flexible, real-time instructional behavior in face of [12] cSeheidwinaglsi oMf tohheaTnw,eJlafth mInetserRnaKtiiornka, lACaornofenreMncienoinngCeorg,naitnivdeJMohondeLlianigrd.. 2012. dynamic environment, necessitates computational formula- Acquiring Grounded Representations of Words with Situated Interactions that can reason online. We take an hybrid approach tive Instruction.Advances in Cognitive Systems (2012). towards this system design where we bring together deep [13] Shiwali Mohan and John E Laird. 2014. Learning Goal-Oriented Hilearning architectures for computer vision and high-level erarchical Tasks from Situated Interactive InstructionPr.oInceedings reasoning architecture Soar for online, real-time, adaptive oPfrtehses.Twenty Eighth AAAI conference on Artificial Intelligence . AAAI instructional support in the real-world. We demonstrate the[14] Shiwali Mohan, Aaron Mininger, and John Laird. 2014. Towards an robustness our approach by observing system behavior in Indexical Model of Situated Language Comprehension for Cognitive scenarios with diferent trainee behavior. There are several Agents in Physical WorldsA.dvances in Cognitive Systems (2014). challenges that have not been addressed in this preliminary[15] Raymond Reiter. 1980. A logic for default reasoniAnrgt.ificial intelliwork; from maintaining object identities during replacement [16] gMeantcheew13S,1h-r2e(v1e9,8S0r),ic8h1a–r1a32n. Kumar, Jin Sun, Gaurang Gavai, Robert R tasks to modeling the state of the trainee to provide person- Price, and Hoda Eldardiry. 2017. Augmented Reality For Eficient Colalized training. Our future work will study these challenges lection Of Training Data For Machine Learning. (Aug 2017). and propose hybrid approaches similar to the one described [17] Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convoin this paper. lutional Networks for Large-Scale Image RecognitiIonnte.rnational
Finally, this paper sets the stage for future research in [18] CCohnrfiesrteiancneSoznegLeeadryn, iWngeiRLeipur,eYseanntagtqioinngs (J2ia01,P5)i.erre Sermanet, Scot Reed, design of intelligent technology for apprenticeship training. Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and AnhTere is a long, rich history of using intelligent technology drew Rabinovich. 2015. Going Deeper with Convolutio nIEsE.E Conwhen combined with cognitive modeling to support human ference on Computer Vision and Patern Recognition (2015). learning5[] that have achieved closed to human-level teach- [19] Kurt Vanlehn. 2011. Educational Psychologist The Relative Efectiveing performance 1[9]. However, a significant majority of Tnuetsosroi nfgHSuymsatenmTs.utEodruincagt,ioInnatellPlsiygcehnotloTguisttor4i6n, g4 S(y2s0t1e1m),s1,9a7n–d22O1.ther research threads pursued have looked at augmenting aca- https://doi.org/10.1080/00461520.2011.611369 demic learning in classroom environments, often focusing [20] Lev Vygotsky. 1978. Mind in Society. on learning coursework algebra, sciences, and programming[.21] James V. Wertsch. 1979. From Social Interaction to Higher PsychologApproaches such as ours will greatly enhance the impact of ical ProcessesH.uman Development 22 (1979), 1–22. intelligent tutoring technology by bringing it to the context for workplace training as well as for end-users of complex machines.