We present and discuss a human-robot collaboration system suitable for supervising the execution of structured manipulation tasks in industrial assembly scenarios. As a case study, we consider the application domain proposed in the context of the project (PON R& I 2014-2020) ICOSAF (Integrated collaborative systems for Smart Factory) in which a human operator physically interacts with a collaborative robot (Cobot) to perform multiple item insertion tasks in a shared workspace. The proposed system combines hierarchical task orchestration and human intention recognition during human-robot interaction through hand-guidance. We provide an overview of the system discussing an initial experimental evaluation.
Collaborative robotic systems (CoBots) enable humans and robots to safely work in close
proximity during the execution of shared tasks [
We consider an industrial assembly case study, proposed in the context of the Italian project (PON R& I 2014-2020) ICOSAF (Integrated collaborative systems for Smart Factory) project, whose aim is the design and development of models and methods for collaborative factories. In the assembly operative scenario, a CoBot should support human workers during the insertion of accessories into carbon-fiber monocoque cells for car production. Specifically, we focused on the task of inserting metallic items (called bigheads) on the monocoque, which is usually manually executed. Human-robot collaboration is particularly suitable for this task since both the human dexterity and the robot precision are required. Indeed, the main requirements are: positioning accuracy, repeatability, supervision systems for the application of the correct item, safety, improvement of the operator’s ergonomics. During task execution, both the human and the robot should be able to work on the monocoque, hence concurrent insertion of items should be possible. The operator should always be allowed to hand guide the robot to point the end-efector towards a diferent target or a diferent working area.
The architecture of the human-robot collaborative system is illustrated in Fig. 1. The HighLevel Control System (HL) is responsible for task generation, supervision and execution, while the Low-Level Control System (LL) manages the execution of the trajectories proposed by the high-level system integrating the human physical guidance. Task supervision and orchestration relies on the attention-based executive framework proposed in [3, 4, 5]. During task execution, the human operator can physically interact with the CoBot (force/position feedback) and these interventions are simultaneously interpreted at the diferent layers of the architecture. Depending on the task, the environmental context, and the human interventions, the Executive System (top-down) retrieves hierarchical tasks in the Long Term Memory (LTM) and allocates them in the Working Memory (WM) (see Fig. 2). The primitive operations are associate with behaviors/processes that compete for the execution (Behavior-based System). In
HHiigghh--LLeevveell CCoonnttrrooll SSyysstteemm Attentional Executive System LTM
WM
aalliivvee mm11 tt11 ttNN
mmNN
TTrraajjeeccttoorryy PPllaannnneerr IInntteennttiioonn EEssttiimmaattiioonn TTaarrggeett SSeelleeccttiioonn
Force Position Velocity
Trajectory this case, motion behaviors (e.g., pick, place) are associated with target positions and a trajectories (generated by the Trajectory Planner), while each possible trajectory is continuously monitored in order to estimate (Intention Estimation) the one more aligned with respect to the human guidance. Intention estimation relies on LSTM networks, one for each trajectory, that classify the operator interventions as aligned, deviating, opposed, opposed deviating. The classification results (along with the top-down attentional regulations provided by the WM) are then exploited to influence the CoBot selection of targets and trajectories ( Targret Selection).
Finally, the lowest level of the architecture implements a Shared Controller aimed at mixing the inputs generated by the the human operator (Shared force) and the ones needed to perform robot motion. An (Admittance Controller) integrates the human and the robot guidance.
In order to test the proposed human-robot collaboration system, we implemented the testbed depicted in Fig. 3, which illustrates a mockup representing a surface with positions where operations must be executed. Since our main interest was to test human-robot collaboration, we considered only end-efector movements towards target positions, while the final insertion operations were simulated. We deployed the Kuka LBR IIWA manipulator, controlled via ROS middleware running on a standard version of Ubuntu 18.04 GNU/Linux OS. The ATI Mini 45 Force sensor has been used to detect the human input to command the robot. As for the LSTM network, it has been implemented using TensorFlow1 library through Keras2.
In this setting, we focused on the system ability in following the operator hand-guided operations considering both quantitative (efort and execution time) and qualitative (NASA-TLX [10] inspired questionnaire) assessments. The task assigned consisted in the execution of multiple simulated operations (5 ordered item insertions) on target positions (see Fig. 3). Since the operations’ sequence is provided to the operator, but this is unknown to the robot, the human co-worker is requested to continuously correct the manipulator trajectories via hand-guidance 1https://www.tensorflow.org 2https://keras.io to obtain the requested order of execution. We compared three interaction modalities: proactive, guided, and passive. In the proactive mode, the robot directly heads to the next target without human confirmation, while in the guided mode the robot always waits for an operator physical input to reach the next target. The passive mode is used as a baseline; in this case the robot is fully compliant with respect to the human physical guidance. The experiment was carried out by 40 testers (graduate or post-graduate students). Preliminary results in this experimental setting show the advantage of the proactive and guided modalities with respect to the passive one, with a slight preference for the guided mode. Specifically, as expected, we observed that both guided and proactive modes significantly reduce the operator physical efort (measured as the cumulative impulse applied to the robot) with respect to the passive mode, with less efort measured in the proactive mode (we observed an average efort of 29, 52, 97 for the proactive, guided and passive modes, respectively). On the other hand, qualitative evaluation results show that guided mode was considered as more reliable, readable and satisfactory; users also estimated less pressure and mental demand. In this respect, the guided mode seems preferred to the proactive mode because it provides the users with more control during task execution.
We presented a human-robot collaboration system for CoBots that supports hand-guided human interventions during the execution of structured tasks in an industrial assembly scenario. The proposed system combines hierarchical task orchestration and human intention recognition during physical interaction. We described the application domain and the system design discussing diferent interaction modes (passive, guided, proactive). The collected results show the advantage of the proposed assisted modalities with respect to the passive one. We are currently designing and developing tasks for the real industrial workspace considering issues like positioning accuracy, task reliability, safety, and ergonomics. We are also investigating adaptive multimodal extensions of the framework [11, 12] including additional communication channels [13] (e.g. gestures and speech) along with associated fusion methods [14] and adaptive interfaces [15].
The research leading to these results has been partially supported by the projects REFILLs (H2020-ICT-731590) and ICOSAF (PON R& I 2014-2020). work systems, in: IFIP international conference on advances in production management systems, Springer, 2016, pp. 677–686. [3] R. Caccavale, A. Finzi, Flexible task execution and attentional regulations in human-robot interaction, IEEE Transactions on Cognitive and Developmental Systems 9 (2017) 68–79. [4] R. Caccavale, M. Saveriano, A. Finzi, D. Lee, Kinesthetic teaching and attentional supervision of structured tasks in human–robot interaction, Autonomous Robots 43 (2019) 1291–1307. doi:10.1007/s10514-018-9706-9. [5] R. Caccavale, A. Finzi, Learning attentional regulations for structured tasks execution in robotic cognitive control, Autonomous Robots 43 (2019) 2229–2243. [6] D. A. Norman, T. Shallice, Attention to action, in: Consciousness and self-regulation,
Springer, 1986, pp. 1–18. [7] R. P. Cooper, T. Shallice, Hierarchical schemas and goals in the control of sequential behavior, Psychological Review 113 (2006) 887–916. [8] J. Cacace, R. Caccavale, A. Finzi, V. Lippiello, Interactive plan execution during human-robot cooperative manipulation, IFAC-PapersOnLine 51 (2018) 500 – 505. URL: http://www.sciencedirect.com/science/article/pii/S2405896318332919. doi:https: //doi.org/10.1016/j.ifacol.2018.11.584, 12th IFAC Symposium on Robot Control SYROCO 2018. [9] J. Cacace, A. Finzi, V. Lippiello, Enhancing shared control via contact force classification in human-robot cooperative task execution, in: F. Ficuciello, F. Ruggiero, A. Finzi (Eds.), Human Friendly Robotics, Springer International Publishing, Cham, 2019, pp. 167–179. [10] S. Hart, L. Staveland, Development of nasa-tlx (task load index) : Results of empirical and theoretical research, Human Mental Workload. Advances in Psychology (1988) 139–183. [11] R. Caccavale, E. Leone, L. Lucignano, S. Rossi, M. Stafa, A. Finzi, Attentional regulations in a situated human-robot dialogue, in: The 23rd IEEE International Symposium on Robot and Human Interactive Communication, IEEE RO-MAN 2014, Edinburgh, UK, August 2529, 2014, 2014, pp. 844–849. [12] R. Caccavale, J. Cacace, M. Fiore, R. Alami, A. Finzi, Attentional supervision of humanrobot collaborative plans, in: 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 2016, pp. 867–873. doi:10.1109/ROMAN. 2016.7745221. [13] S. Iengo, S. Rossi, M. Stafa, A. Finzi, Continuous gesture recognition for flexible humanrobot interaction, in: 2014 IEEE International Conference on Robotics and Automation, ICRA 2014, Hong Kong, China, May 31 - June 7, 2014, 2014, pp. 4863–4868. [14] J. Cacace, A. Finzi, V. Lippiello, A robust multimodal fusion framework for command interpretation in human-robot cooperation, in: 26th IEEE International Symposium on Robot and Human Interactive Communication, RO-MAN 2017, Lisbon, Portugal, August 28 - Sept. 1, 2017, 2017, pp. 372–377. [15] J. Cacace, R. Caccavale, A. Finzi, V. Lippiello, Attentional multimodal interface for multidrone search in the alps, in: 2016 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2016, Budapest, Hungary, October 9-12, 2016, 2016, pp. 1178–1183.