Planning and Execution with Robot Trajectory Generation in Industrial Human-Robot Collaboration

Abstract

The co-presence of a robot and a human sharing some activities in an industrial setting constitutes a challenging scenario for control solutions, requiring highly flexible controllers to preserve productivity and enforce human safety. Standard methods are not suitable given the lack of methodologies able to evaluate robot execution time variability, caused by the necessity to continuously modify/adapt robot motions to grant human safety. This paper presents a novel dynamic planning system for Human-Robot Collaboration (HRC) which leverages an offline motion planning technique and deploys planning and execution features dealing with temporal uncertainty and kinematics both at planning and execution time. The proposed system is deployed in a manufacturing case study for controlling a working cell in which a robot and a human collaborate to achieve a shared production goal. The approach has been shown to be feasible and effective in a real case study.

-

During the last decade, industrial robotic systems have entered assembly cells in order to support human worker in repetitive and physical demanding tasks. However, HumanRobot Collaboration (HRC) scenarios present several issues that must be addressed to realize flexible and effective controllers [ 1 ]. From a low-level control perspective, a generic HRC task can be accomplished through many robot trajectories where each trajectory could be executed concurrently to different human tasks and its execution time depends on the need to modify motion speed in order to grant human safety. Indeed, the robot speed can be modified based on robot direction of motion, human position, velocity and direction and motion (i.e.,speed variation monitoring). From a high-level control perspective, a coordinated task plan should be generated, continuously updated and concurrently performed by the human and the robot aiming at increasing the efficiency (i.e., maximize throughput), supporting the human in a timely manner (i.e., robot tasks synchronized with human tasks) and, again, always keeping the human safe. Literature shows how robot motion and task planning are computationally complex, making difficult their integration in an unified approach, without relying on limiting hypothesis and applicability contexts [ 2, 3 ]. To overcome such issue, some authors (e.g., [ 4, 5, 6, 7 ]) pursued a hierarchical integrated approach that, however, rely on a clear distinction between task and motion planning features. Indeed, typically the task plan is constructed at an abstract, high and discrete level and recursively evaluated just before execution in order to verify the feasibility with respect to spatial/geometric features of the domain. Moreover, these works do not consider temporal information and concurrent execution of human and robot tasks at planning time. Plan-based controllers such as, e.g., T-REX [ 8 ] or IXTET-EXEC [ 9 ], rely on temporal planning mechanisms (exploiting respectively EUROPA [ 10 ] and IXTET [ 11 ]) capable of dealing with coordinated task actions and temporal flexibility. Unfortunately, these systems do not have an explicit representation of uncontrollability features in the domain. Thus, in application scenarios like HRC, the resulting controllers do not endow the flexibility needed to cope with uncontrollable time-varying features and robustly execute plans without strongly relying on replanning mechanisms. Here, we are pursuing an innovative approach for integrating task and motion planning capable of dealing with both temporal and spatial constraints and addressing the uncertainty introduced by the human behavior variability. The approach leverages recent research results, i.e., [ 12 ] and [ 13 ], to provide temporal and geometric models of the human and the robot. This paper presents a planning and execution system fully integrated in such an approach. A key feature of this work consists in modeling the expected behavior of the human at different levels of abstractions dealing with temporal uncertainty and kinematics both at planning and execution time. A system is deployed to realize a flexible plan-based controller capable of dynamically adapting the robot behavior to the human tasks as well as guaranteeing her safety. The system has been applied in a manufacturing case study for controlling a working cell in which a robot and a human collaborate to achieve a shared production goal. A set of experiments have demonstrated both the feasibility and effectiveness of the proposed approach. 2

Human-Aware Control Approach

This paper proposes a human-aware control approach integrating task and motion planning solutions. Moreover, both the solutions, analyzed singularly, represent an advancement with respect to the state of the art. modFuilgeusreco1nstsihtuotwinsg ththee mparoin- 2. DToemmainpdoefrinaitlion tr1a.jeMcototiroiens 0. Collaborative tasks definition posed framework and the se- Task Planner tqhueenincteegoraftesdtecposntirmolplaepmperonaticnhg: 3.trTaajesckto(rrey)spelalencntiniogn & 5. Motion Motion Planner a motion planner, a temporal Plan Executive requests task planner and a plan executive. First, the considered indus- 4. Task plan execution & Monitoring 6. Motion execution & Monitoring trial process is analyzed to iden- Fig. 1: Task and motion planner integration. tify the possible human-robot collaboration scenarios (Step 0). Such step identifies the (collaborative) tasks needed to realize the assembly processes, the resources that can perform the tasks (human, robot or both), and the operational constraints (e.g., precedence or synchronization constraints). Then, for all the possible collaborative tasks of the robot i.e., human and robot simultaneous tasks, a set of robot trajectories is computed (Step 1). Specifically, the motion planner is responsible for generating and executing robot trajectories and guaranteeing the safety of the human operator adapting the robot speed. It relies on an offline and statistical analysis able to identify the volume occupied by the human with a certain probability during the execution of tasks (i.e., the Human Occupancy Volume - HOV) [ 12 ]. Given the HOVs characterized by a different occupancy probability, the motion planner generates, for each couple of simultaneous human-robot task, a set of possible trajectories. The different trajectories that the robot can follow will enter the volume occupied by the human at different levels, thus being characterized by a different execution time and confidence interval. The identified robot and human tasks coupled with the related temporal information (time execution and its variability) are encoded in a temporal planning model (Step 2). This information allows the task planner to characterize the temporal uncertainty concerning the actual duration of human and robot tasks. Considering this model, a task planner generates temporally flexible plans (Step 3) coordinating the operations of the robot and the human and selecting the most suitable trajectories according to the expected collaborative context by taking into account operational constraints and safety settings characterizing the possible collaboration scenarios [ 14 ]. The task planner relies on a temporal planning formalism capable of synthesizing flexible plans by dealing with temporal uncertainty. Then, the plan executive executes the plan (Step 4) by properly dealing with the temporal variability of the robot and the human. Robust plan execution is achieved through temporal flexibility and a replanning mechanism that allow the controller to adapt/modify the plan and robot behavior according to the actual behavior of the human [ 15 ]. The selected robot trajectories (Step 5) are executed by the motion planner which also implements low-level speed separation and variation monitoring to avoid collisions with the human (Step 6). 3

Dynamic Task Planning

The dynamic task planning system is in charge of (i) deciding the tasks the human and the robot must perform; (ii) selecting the most suitable trajectory for robot motions among the set of trajectories generated by the motion planner; (iii) dealing with temporal uncertainty during plan generation and execution; (iv) monitoring the execution and, in case of need, managing possible failures through replanning. Fig. 2 shows a detailed view of the integrated control architecture. It describes the interactions between the deliberative and the executive processes and the role of a ROS-based middleware during the execution of a plan. itahunhnanegvddePeTPArlhbLaceeUnAteiTnnEnTgcIxaNiesemwrkUctpaiutmilthPneivlmtTa(yePneI)iLmnnn[etae1erFnld4iinNg]aiu,.nnes2gdas- 1. bu2f.feprTleaadnsnkePdl3abn.1n.erre3Pbla.nfaniiMlnuFgraaeniluargeer csfeoenmeddmbaacnkd ittrsSygaenm iltraePnnnooM tbooR fcdwrauaintcmhteledywthieoenxrkte[f1onw3rdmh]siacaElhniPsdcmSosLmiignpn[tl1irifie6os]-- 3a. executed lcxEaePn DMispoantictohrer scfeeoenmddmbaacnkd tebpooRO aunHmy<<>>OM by introducing the capability of dealing with temporal Fig. 2: The dynamic task planning control architecture. uncertainty at both planning and execution time. Thus, leveraging the timeline-based approach envisaged in [ 13 ], PLATINUm characterizes the planning domain concerning a HRC scenario by considering the human as a uncontrollable element and the robot as a partially controllable element. Then, following the structured approach described in [ 17 ], the HRC process can be hierarchically characterized in three abstraction levels. The supervision level models the overall processes (e.g., a collaborative assembly process) in terms of tasks needed to realize them. The coordination level models the possible behaviors of the human and the robot in terms of tasks they can perform. The (temporal) behavior of the human is modeled as uncontrollable with lower and upper bounds on task durations according to the information gathered by the offline analysis of the motion planner. Similarly, the (temporal) behavior of the robot is modeled as partially controllable due to the co-presence of the human which may affect robot task execution (e.g., robot motions can be slowed down or suspended according to the position of the human during execution). Finally, the implementation level models the internal constraints that allow a robot to actually execute assigned tasks. Again, the temporal characterization of robot motion tasks leverages information gathered by the motion planner (step 1 in Fig. 1) and encapsulates information about the available trajectories the execution time variability. Synchronization rules model possible assignments of tasks to the human and the robot and the related operational requirements. Such rules specify the possible collaborations between the human and the robot and the temporal constraints that must be satisfied.

The proposed task and motion planning integration approach allows a task planner to reason about the particular collaboration scenario and the related interaction modality by deciding the "most suited" modality of execution of robot tasks in order find a good tradeoff between the safety of the human and the throughput of the production process. 4

Deployment in a Real Scenario

PLATINUM has been deployed in a manufacturing case study integrating the task planning technology described above with a motion planning system for industrial robots [ 12 ]. The reference selected application is a human-robot collaborative environment for the preparation of the load/unload station (LUS) of a flexible manufacturing system (FMS) (Fig.3). At the LUS, machined parts and raw parts have to be unmounted and mounted on ad-hoc fixturing systems, called pallet, by a worker and a robot in order to be machined by the FMS. With the aim to increase productivity and grant human ergonomics and safety, robot trajectories and task allocation have to be respectively adequately designed and planned.

Thus, PLATINUM and its features are leveraged to implement an integrated task and motion planning system capable of selecting different execution modalities for robot tasks according to the expected collaboration of the robot with a human operator. This is the result of a tight integration of PLATINUM with a motion planning system. Indeed, the pursued approach Fig. 3: Experimental environment realizes an offline analysis of the production scenarios in order to synthesize a number of collision-free robot motion trajectories for each collaborative task with different safety levels. Each trajectory is then associated with an expected temporal execution bound and represents a tradeoff between "speed" of the motion and "safety" of the human. The integrated system has been deployed and tested in laboratory on an assembly case study similar to collaborative assembly/disassembly scenario described above. In [ 15 ], an empirical evaluation is provided in order to assess the overall productivity of the HRC cell while increasing the involvement of the robots. The idea is to gradually make free a set of tasks originally preallocated to the human, so to increase the number of degrees of freedom of PLATINUM during the minimization of the assembly time. The results show the effectiveness of PLATINUM in finding well suited distribution of tasks between the human and the robot in different scenarios with an increasing workload for the control system. Indeed, the total assembly time was reduced of 65% (from 259s to 169s) and the percentage of tasks assigned by PLATINUM to the robot moved from 25% to 65%. Thus, PLATINUM instance resulted as capable of increasing the productivity of the production process without affecting the safety of the operator. A wider experimental campaign is now undergoing and will constitute an important pillar for a future longer report. 5

Conclusions

This paper introduces a novel framework in which robot motion planning and task planning are integrated and their synergisms are exploited to cope with the variability of an environment in which an industrial robot is acting together with a human worker. After introducing the integration idea we have described the planning and execution feature that guarantee robustness in coping with temporal uncertainty. The experiment in the real case demonstrates the ability of the system to impact the reduction of the makespan, and to demonstrate time constants able to cope with domain uncertainties. Acknowledgment. The CNR authors are supported by the European Commission within the H2020 research and innovation programme, FourByThree project, grant agreement No. 637095.

References

1. Freitag , M. , Hildebrandt , T. : Automatic design of scheduling rules for complex manufacturing systems by multi-objective simulation-based optimization . {CIRP} Annals - Manufacturing Technology 65 ( 1 ) ( 2016 ) 433 - 436

2. Michalos , G. , Kaltsoukalas , K. , Aivaliotis , P. , Sipsas , P. , Sardelis , A. , Chryssolouris , G. : Design and simulation of assembly systems with mobile robots . {CIRP} Annals - Manufacturing Technology 63 ( 1 ) ( 2014 ) 181 - 184

3. Pellegrinelli , S. , Pedrocchi , N. , Tosatti , L.M. , Fischer , A. , Tolio , T.: Multi-robot spot-welding cells: An integrated approach to cell design and motion planning . {CIRP} Annals - Manufacturing Technology 63 ( 1 ) ( 2014 ) 17 - 20

4. Wolfe , J. , Marthi , B. , Russell , S.J.: Combined task and motion planning for mobile manipulation . In: Proceedings of the 20th International Conference on Automated Planning and Scheduling , ICAPS 2010 , Toronto, Ontario, Canada, May 12 -16, 2010 . ( 2010 ) 254 - 258

5. Kaelbling , L.P. , Lozano-Pérez , T. : Hierarchical task and motion planning in the now . In: 2011 IEEE International Conference on Robotics and Automation . ( 2011 ) 1470 - 1477

6. Srivastava , S. , Fang , E. , Riano , L. , Chitnis , R. , Russell , S. , Abbeel , P. : Combined task and motion planning through an extensible planner-independent interface layer . In: 2014 IEEE International Conference on Robotics and Automation . ( 2014 ) 639 - 646

7. de Silva, L. , Lallement , R. , Alami , R.: The hatp hierarchical planner: Formalisation and an initial study of its usability and practicality . In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . ( 2015 ) 6465 - 6472

8. Py , F. , Rajan , K. , McGann , C. : A systematic agent framework for situated autonomous systems . In: AAMAS. ( 2010 ) 583 - 590

9. Lemai , S. , Ingrand , F. : Interleaving Temporal Planning and Execution in Robotics Domains . In: AAAI-04 . ( 2004 ) 617 - 622

10. Barreiro , J. , Boyce , M. , Do , M. , Frank , J. , Iatauro , M. , Kichkaylo , T. , Morris , P. , Ong , J. , Remolina , E. , Smith , T. , Smith , D. : EUROPA: A Platform for AI Planning, Scheduling, Constraint Programming, and Optimization . In: ICKEPS 2012 : the 4th Int. Competition on Knowledge Engineering for Planning and Scheduling . ( 2012 )

11. Ghallab , M. , Laruelle , H.: Representation and control in ixtet, a temporal planner . In: 2nd Int. Conf. on Artificial Intelligence Planning and Scheduling (AIPS) . ( 1994 ) 61 - 67

12. Pellegrinelli , S. , Moro , F.L. , Pedrocchi , N. , Tosatti , L.M. , Tolio , T. : A probabilistic approach to workspace sharing for human-robot cooperation in assembly tasks . {CIRP} Annals - Manufacturing Technology 65 ( 1 ) ( 2016 ) 57 - 60

13. Cialdea Mayer , M. , Orlandini , A. , Umbrico , A. : Planning and execution with flexible timelines: a formal account . Acta Inf . 53 ( 6-8 ) ( 2016 ) 649 - 680

14. Umbrico , A. , Cesta , A. , Cialdea Mayer , M. , Orlandini , A. : Platinum: A new framework for planning and acting . In: AI*IA 2017, Advances in Artificial Intelligence . ( 2017 ) To appear

15. Pellegrinelli , S. , Orlandini , A. , Pedrocchi , N. , Umbrico , A. , Tolio , T. : Motion planning and scheduling for human and industrial-robot collaboration . {CIRP} Annals - Manufacturing Technology ( 2017 ) -

16. Umbrico , A. , Orlandini , A. , Cialdea Mayer , M. : Enriching a temporal planner with resources and a hierarchy-based heuristic . In: AI*IA 2015, Advances in Artificial Intelligence , Springer International Publishing ( 2015 ) 410 - 423

17. Cesta , A. , Orlandini , A. , Bernardi , G. , Umbrico , A. : Towards a planning-based framework for symbiotic human-robot collaboration . In: 21th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) , IEEE ( 2016 )