Attentional Plan Execution for Human-Robot Cooperation

Abstract

In human-robot interactive scenarios communication and collaboration during task execution are crucial issues. Since the human behavior is unpredictable and ambiguous, an interactive robotic system is to continuously interpret intentions and goals adapting its executive and communicative processes according to the users behaviors. In this work, we propose an integrated system that exploits attentional mechanisms to exibly adapt planning and executive processes to the multimodal human-robot interaction.

Introduction

In social robotics, exible and natural interaction with humans is often needed in the context of structured collaborative tasks. In these scenarios, the robotic system should be capable of adapting the execution of cooperative plans with respect to complex human activities and interventions. Many mechanisms are indeed involved in humans cooperation [ 3 ], such as joint attention, action observation, task-sharing, and action coordination [ 19, 15 ]. Furthermore, communication between humans involve di erent modalities such as speech, gaze orientation, gestures [ 4 ]. Several systems manage the human-robot cooperation by planning (totally or partially) the action sequence for the agents involved in the interaction [ 11, 12, 20 ]; however, planning and replanning processes can be timeexpensive and can therefore impair the naturalness and the e ectiveness of the interaction. In order to exibly combine and orchestrate structured activities and reactive actions we exploit the concept of cognitive control proposed by the cognitive neuroscience literature [ 17, 2 ]. Inspired by supervisory attentional system and contention scheduling models [ 17, 9, 2 ], we propose an integrated framework that combines planning, attentional regulations, and multimodal human-robot interaction in order to exibly adapt plan execution according to the executive context and the multimodal intraction processes.

Multimodal Interaction and Dialogue management. The multimodal HRI framework is appointed to recognize the multiple human commands and actions, such as utterances, gaze directions, gestures or body postures, and to provide an interpretation of users intentions according to the dialogue context. Following the approach proposed by [ 18 ], the multimodal recognition process is composed of three layers: the lower layer contains the classi ers of the single modalities; the middle layer, the fusion engine, exploits a Support Vector Machine (SVM)-based late fusion and provides an integration of the multiple inputs; the upper layer, the dialogue manager [ 14 ], integrates the representation of the dialogue modelled as a Partially Observable Markov Decision Process (POMDP) and accomplishes the semantic interpretation of observations according to the context and the inner knowledge. The main feature of such structure is that the results of each layer are N-best lists of possible interpretations, which are fed to the next layer to solve the ambiguities in cascade.

Human Aware Task Planning. The system is endowed with a Human-Aware Task Planner (HATP) [ 13 ] which is based on a Hierarchical Task Networks (HTN) and a SHOP-like [ 16 ] re nement process. HATP is able to produce hierarchical plans for multi-agent systems (including humans), generating di erent sequences for each agent. Each action has a nite number of precondition links to other actions (which can be part of any sequence), allowing HATP to generate collaborative subtasks where more agents are involved. Furthermore, the actions of the domain can be associated with a duration and a cost function, while speci c social rules could be de ned along with a cost for their violation. By setting a di erent range of parameters the plans can be tuned to adapt the robot behavior to the desired level of cooperation.

Executive System. The executive process is managed by two subsystems: the supervision system [ 10 ] and the the attentional system [ 8, 6, 7 ]. The rst one is to interact with the task planner, monitor the plan execution and formulate replanning requests. The second one exploits bottom-up (stimuli-oriented) and top-down (task-oriented) in uences to regulate the plan execution and the dialogue policy. The attentional system manages a cognitive control cycle that continuously updates the systems working memory (WM) and a set of attentional behaviors (BP) exploiting the task structure de ned in a long term memory (LTM). Each task to be executed is associated with a hierachical structure which is allocated in working memory, the leaves of this structure are concrete attentional behaviors whose activations are top-down and bottom-up regulated by the attentional in uences (see [ 8, 5 ]). In this context, the dialogue policies are assimilated to special interactive behaviors, while the generated human aware plans are exploited as a guidance for the action selection and execution. 3

Case Studies

The proposed architecture has been tested in a case study inspired by a humanrobot co-working domain [ 1 ] proposed in the context of the SAPHARI project (FP7-ICT-287513).

The environment is set as follows: there are three work locations, each containing a slot (slot1, slot2, slot3) and a table that supports a set of objects including a glue bottle (glueBottle) and some brackets (bracket1, bracket2). The user and the robot must cooperatively install the brackets in the slots, by rst cleaning the slot, then applying the glue in the area, and nally xing the bracket on it. In this context, the integrated system can exibly adapt plan execution to unexpected behaviors of the human, avoiding the computation of a new plan while enabling a smoother and more natural interaction. In the following, we provide some examples where the proposed systems allows for overcoming impasses during the interaction.

Handover to Search. In a rst scenario, the robot is waiting for a bracket from the human in order to install it. In the planned sequence the human should bring the object to the robot, however, the human remains idle and does not interact as expected. According to the plan, the robot should keep waiting for the human, however, the attentional system comes into play: in the absence of external stimuli, the activations of the receive behavior decreases with time. Therefore, after some seconds of waiting, since the human does not cooperate in the task, the activations of the search behavior become dominant with respect to receive, hence the robot can abandon the handover behavior and can start searching for the object by itself.

Take to Search. In a second scenario, we consider the case of a robot that should get the bracket1 and give it to the human, which is to x it on a slot. As suggested by the plan, the robot travels to the table in order to get the object, but it is not there. Analogously, to the previous case, since no environmental stimulation is present for the take action, the attentional system switches to a search behavior where the robot inspects other locations looking for bracket1. As soon as the object is found, the take action becomes dominant again allowing the robot to continue the plan.

Handover to Place. In a third scenario, we assume that the human is to obtain the glueBottle in order to glue slot1. Following the plan, the robot tries to perform an handover, but the human moves away from the working space (see Fig. 2). The supervision system cannot replan since the human has not performed its action and the plan is still valid. However, the attentional system can solve the impasse without waiting for the human initiative. Indeed, the bottom-up stimulation of the give decreases as the robot-human distance increases, while the place activations remain stable (depending on the table position). When place wins the contention, the robot places the object on the work location allowing for plan continuation. Once the place is accomplished, the supervision system can manage the action substitution by changing the monitored human action from receive to take.

ACKNOWLEDGMENT

The research leading to these results has been supported by the RoDyMan and SAPHARI projects, which have received funding from the European Research Council under Advanced Grant agreement number 320992 and 287513, respectively. The authors are solely responsible for its content. It does not represent the opinion of the European Community and the Community is not responsible for any use that might be made of the information contained therein.

References

1. Alami , R. , Gharbi , M. , Vadant , B. , Lallement , R. , Suarez , A. : On human-aware task and motion planning abilities for a teammate robot . In: RSS HRCIM ( 2014 )

2. Botvinick , M.M. , Braver , T.S. , Barch , D.M. , Carter , C.S. , Cohen , J.D. : Con ict monitoring and cognitive control . Psychological review 108(3) , 624 ( 2001 )

3. Bratman , Michael, E.: Shared agency. Philosophy of the social sciences: philosophical theory and scienti c practice pp . 41 { 59 ( 2009 )

4. Breazeal , C. : Designing Sociable Robots. MIT Press ( 2002 )

5. Broquere , X. , Finzi , A. , Mainprice , J. , Rossi , S. , Sidobre , D. , Sta a , M.: An attentional approach to human-robot interactive manipulation . I. J. Social Robotics 6 ( 4 ), 533 { 553 ( 2014 )

6. Caccavale , R. , Finzi , A. : Plan execution and attentional regulations for exible human-robot interaction . In: Proc. of SMC 2015 ( 2015 )

7. Caccavale , R. , Finzi , A. , Lucignano , L. , Rossi , S. , Sta a , M.: Attentional top-down regulation and dialogue management in human-robot interaction . In: Proc. of HRI 2014 . pp. 129 { 130 ( 2014 )

8. Caccavale , R. , Leone , E. , Lucignano , L. , Rossi , S. , Sta a , M., Finzi , A. : Attentional regulations in a situated human-robot dialogue . In: RO-MAN . pp. 844 { 849 ( 2014 )

9. Cooper , R.P. , Shallice , T. : Hierarchical schemas and goals in the control of sequential behavior . Psychological Review 113 ( 4 ), 887 { 916 ( 2006 )

10. Fiore , M. , Clodic , A. , Alami , R.: On planning and task achievement modalities for human-robot collaboration . In: International Symposium on Experimental Robotics , Marrakech/Essaouira, June. vol. 1518 , p. 2014 ( 2014 )

11. Fong , T. , Kunz , C. , Hiatt , L. , Bugajska , M.: The human-robot interaction operating system . In: Proc. of HRI 2006 . pp. 41 { 48 ( 2006 )

12. Karpas , E. , Levine , S.J. , Yu , P. , Williams , B.C. : Robust execution of plans for human-robot teams . In: Proc. of ICAPS-15 . pp. 342 { 356 ( 2015 )

13. Lallement , R., de Silva, L. , Alami , R.: HATP: An HTN Planner for Robotics . In: 2nd ICAPS Workshop on Planning and Robotics , PlanRob 2014 ( 2014 )

14. Lucignano , L. , Cutugno , F. , Rossi , S. , Finzi , A. : A dialogue system for multimodal human-robot interaction . In: Proc. of ICMIa . pp. 197 { 204 ( 2013 )

15. Mutlu , B. , Terrell , A. , Huang , C.M.: Coordination mechanisms in human-robot collaboration . In: ACM/IEEE Intl. Conf. on Human-Robot Interaction (HRI)- Workshop on Collaborative Manipulation . pp. 1 { 6 ( 2013 )

16. Nau , D. , Cao , Y. , Lotem , A. , Mun~oz- Avila , H.: Shop: Simple hierarchical ordered planner . In: Proceedings of the 16th international joint conference on Arti cial intelligence-Volume 2 . pp. 968 { 973 ( 1999 )

17. Norman , D.A. , Shallice , T. : Attention to action: Willed and automatic control of behavior . In: Consciousness and self-regulation: Advances in research and theory , vol. 4 , pp. 1 { 18 ( 1986 )

18. Rossi , S. , Leone , E. , Fiore , M. , Finzi , A. , Cutugno , F. : An extensible architecture for robust multimodal human-robot communication . In: In Proc. of IROS-2013 . pp. 2208 { 2213 ( 2013 )

19. Sebanz , N. , Bekkering , H. , Knoblich , G.: Joint action: bodies and minds moving together . Trends in cognitive sciences 10(2) , 70 { 76 ( 2006 )

20. Shah , J. , Wiken , J. , Williams , B. , Breazeal , C. : Improved human-robot team performance using chaski, a human-inspired plan execution system . In: Proc. of HRI-11 . pp. 29 { 36 ( 2011 )