Integrating Task and Motion Planning for Manufacturing Processes Alberto Gottardi1,2,∗ , Nicola Castaman3 , Enrico Pagello1 and Emanuele Menegatti2 1 IT+Robotics srl, 36100 Vicenza, Italy 2 Intelligent Autonomous System Lab, Dept. of Information Engineering, University of Padova, 35131 Padua, Italy 3 Professional Industrial consultant, 36061 Bassano del Grappa, Italy. Abstract The synergy between the robot’s abilities and human expertise provides several advantages in the industrial processes. A proper integration of Task and Motion Planning (TAMP), considering the interaction of the user with the environment is essential to maximize the benefits of robot-assisted tasks and to ensure safety in collaborative H-R tasks. However, the state of the art of the TAMP research field should be improved to overcome the limitations of the traditional TAMP approaches to be used in industrial scenarios. We illustrate a list of challenges to be solved in order to improve to manage efficiently the dynamism and uncertainty given by introducing a human operator into a robotized process of draping fiber carbon plies, with reference to the granted European Project DrapeBot. Finally, the proposed TAMP architecture is presented to address the challenges described. Keywords Task and Motion Planning, Human-Robot Collaboration 1. Introduction Human-Robot Collaboration (HRC) has emerged as a significant technological challenge in the industrial landscape in recent years. By combining the precision, efficiency, and repeatability of robots with the intelligence, adaptability and expertise of humans, numerous advantages emerge. Such collaboration reduces operator fatigue, improves ergonomic conditions, and enhances production quality [1, 2]. The robot must continuously interpret human interventions in order to adapt to a cooperative performance while it is executing the task plan [3]. Proper integration of Task and Motion Planning (TAMP) [4], considering both the environment and user needs is essential to maximise the benefits of robot-assisted tasks and to ensure safety in collaborative tasks. A common technique in TAMP is interleaving the symbolic and geometric search processes by calling a motion planner at each step and assigning geometric parameters to the currently symbolic state before proceeding. The interleaving becomes problematic when a planned state is 10th Italian Workshop on Artificial Artificial Intelligence and Robotics (AIRO 2023) ∗ Corresponding author. Envelope-Open alberto.gottardi@it-robotics.it (A. Gottardi); enrico.pagello@unipd.it (E. Pagello); emanuele.menegatti@unipd.it (E. Menegatti) GLOBE https://www.dei.unipd.it/~emg/ (E. Menegatti) Orcid 0000-0002-9229-9411 (A. Gottardi); 0000-0002-3086-5852 (N. Castaman); 0000-0002-8152-0707 (E. Pagello); 0000-0001-5794-9979 (E. Menegatti) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings Figure 1: Example of carbon fiber draping process in an HRC with a securely fixed zone. valid in symbolic space, but geometrically infeasible. To address this, FFRob [5] introduced an FF- like heuristic that integrates geometric information into the FF-search. An alternative approach executes a geometric search on candidate symbolic plans [6]. Similarly, Dantam et al. [7] incrementally generate symbolic plans using an incremental Satisfiable Modulo Theory solver, invoking a motion planner for validation. Most TAMP methods have long processing times and consider a static environment, assuming an ideal, noise-free perception system. Nouman et al. [8] proposed a hybrid condition planner that extends the classical condition planner by integrating feasibility checks into the action conditions. Castaman et al. [9] solve TAMP problems in a changing environment with a receding horizon approach, iteratively solving a reduced planning problem over a receding window. A preliminary study on a conditional TAMP algorithm able to find a plan that minimizes robot efforts while solving assigned tasks has also been illustrated in [10]. In [11] Lagriffoul et al. have presented a platform-independent evaluation method for TAMP by proposing a set of benchmark problems covering the challenging aspects of TAMP. However, these approaches do not completely satisfy the requirements: most TAMP methods have long processing times and consider a static environment, assuming an ideal, noise-free perception system. A second limitation of these works is that they do not consider the human operator, a crucial element in the industrial application under analysis. 2. Case Study Since an industrial environment shared by humans and robots is a highly dynamic scenario, many manufacturing industries have not yet introduced automation in their processes. The unpredictability of human presence must be integrated into TAMP and robot control, leveraging the latest advances in perception and interactions. Thus, this paper offers an enhanced approach that could solve the challenges outlined below for manufacturing processes of an industrial HRC scenario. The following list of challenges was initially identified from the author’s experience in the industrial environment and previous works [2, 9, 10]. The requirements of the industrial environment provided by the application of draping within the Drapebot project emphasised the importance of addressing these challenges with a TAMP approach. CHL1 To integrate the knowledge of humans’ and robots’ capabilities CHL2 To compute feasible action sequences and share them among humans and robots CHL3 To satisfy ergonomic constraints by synchronizing robot and human movements CHL4 To ensure human safety during robot motion CHL5 To continuously monitor the scene by adapting on-line the robot to changes These five challenges can be grouped into three macro categories: collaboration (CHL1-3), safety (CHL4) and monitoring (CHL5). Firstly, CHL1 calls for the seamless integration of human and robot capabilities, emphasizing the need to harness their expertise effectively. CHL2 extends this by highlighting the intricate task of computing a viable action sequence and facilitating its sharing between human and robot counterparts, which is pivotal for synchronized and efficient collaboration. CHL3 complements these by emphasizing the ergonomic alignment of robot movements with human actions, particularly vital in cooperative endeavours involving physical interaction. Moving on to safety, CHL4 addresses the paramount issue of ensuring human safety throughout robot motion, necessitating robust safety mechanisms and real-time risk assessment. Last, the theme of monitoring comes to the fore with CHL5, stressing the robot’s continuous scene observation and adaptive response capabilities to address environmental changes proactively. We have tackled the above challenges within the EU project DrapeBot (https://www.drapebot. eu/), aiming to develop an HRC system that aids operators in carbon fiber draping. In particular, it focuses on the dynamic scheduling of shared human-robot activities within a manufacturing environment where humans and robots collaborate to complete complex tasks like draping [12]. 3. Framework Concept In HRC industrial applications, the TAMP framework is pivotal. It must ensure flexibility in managing work plans, effectively address operator interventions, handle inputs from external sensors (e.g., perception systems and laser scanners), and safeguard operator safety. Furthermore, it should maintain production quality by automating inspection processes for quality control. The use of a TAMP indeed allows the industrial process to take a step forward, bringing an increase in productivity and more efficient use of resources and operator skills by removing the operator from more stressful and repetitive tasks (e.g. transporting patches, in-depth inspection during the process). Therefore, one specific industrial scenario where a human-aware TAMP solution can play a critical role is inside the carbon fiber draping process. Draping involves transporting the carbon ply onto the mould and shaping it to fit. Another vital process is visual inspection to ensure product quality. This process is predominantly manual, carried out by skilled human operators whose expertise is essential for the final product’s high quality. Employing TAMP lets the operator be more free to focus on the draping activity, which is the one that requires the highest skill to obtain the best final quality of the product. Thus, being draping an excellent case for attempting to address the illustrated challenges of a dynamic human-aware industrial scenario, we are developing a TAMP approach within the DrapeBot Figure 2: Dynamic Human-Aware TAMP Framework. project according to the approach outlined by (Fig. 2). Our proposed framework comprises three main modules: Task Planner, Motion Planner and Central Node. Task Planner. The Task Planner [13] orchestrates the operations of humans and robots. It generates a continuously updated plan that will serve as a workflow guideline and consist of the sequence of actions to complete the assigned task. The Task Planner handles human interrupts, dynamically adjusting the computed plan to satisfy collaboration needs or to deal with unexpected events (CHL1, CHL2). Additionally, it employs recovery procedures to revert to a safe state. Since the planner has to deal with different agents, the effort must be divided in such a way that the robot maximizes its contribution and takes care of the heaviest actions (e.g., inspection, small plies transportation) so that the user can minimize his effort and concentrate more on the activity of draping the ply on the mould (CHL2). The details of the approach and the effort graph-based algorithm are described in [13]. A three-tier hierarchical design enhances modularity and adaptability: • Primitive actions: Fundamental operations (e.g., Move, Draping, Inspection, etc.). • Composite actions: Sequences of primitives for complex actions like patch transportation, involving robot movement, carbon fiber detection, etc. • Final Plan: Provides a high-level view of the draping process by laying out the entire sequence of tasks for both human and robot agents. Motion Planner. The Motion Planner computes collision-free trajectory for robots. To ensure operator safety (CHL4), Safe Zones are introduced to restrict robot entry, assuring opera- tor freedom. Motion planning algorithm considers these zones to compute non-collaborative motions (Fig. 1). The collaborative trajectory is the most challenging to compute. The plan- ner must consider human limitations, emphasizing ergonomic constraints based on operator stature (CHL3). Figure 3 displays the window for the cost function of human operations while cooperating with the robot. A skeletal tracking system [14] makes it possible to compute an ergonomic trajectory that all agents can execute. Figure 3: Description of the cost function for the ergonomics motion planner. Central Node. In the classical approaches, Task and Motion planner modules communicate directly to exchange information. However, due to the large amount of data to be handled from the industrial scene and process, our framework provides a Central Node to integrate and optimize task and motion planner modules. The plan is carried out by the Central Node, which also controls and supervises the proper primitives’ activity. Additionally, this module constantly monitors the condition of the workcell using the sensors in the environment (CHL5). The Central Node also handles human gestures that trigger action not foreseen in the plan (CHL1). In that case, it sends the information to the Task Planner module which is in charge of creating a new plan where the requested action is the first action to be performed. 4. Conclusions To improve the management of a dynamic and uncertain scenario by introducing a human cooperating with a robot in an industrial process, we consider the limits of the current state of the art of TAMP approaches. We list a series of challenges that should be addressed, like integrating human skills with robot flexibility, the ergonomic constraints due to the operator’s physical body, the workers’ safety issues, and the processes’ monitoring procedures. These challenges are illustrated with respect to a real carbon fiber draping industrial process. We abstract three different levels: a Task planner computing the process evolution taking into account the robot and human capabilities; a Motion Planner, computing the collision-free trajectories, that uses the Safe Zones for human safety; a Central Node monitoring the actions of the operator in the environment. Acknowledgments This project has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No 101006732 (DrapeBot). References [1] G. Chryssolouris, N. Papakostas, D. Mavrikios, A perspective on manufacturing strategy: Produce more with less, CIRP Journal of Manufacturing Science and Technology 1 (2008). [2] L. Tagliapietra, E. Tosello, E. Menegatti, Curami: human-robot collaboration for intelligent assembly tasks, in: 2nd Italian Conf. on Robotics and Machine Intelligence, I-RIM, Rome, 2020. [3] J. Cacace, R. Caccavale, A. Finzi, R. Grieco, Combining human guidance and structured task execution during physical human–robot collaboration, Intelligent Manufacturing 34 (2022). [4] H. Guo, F. Wu, Y. Qin, R. Li, K. Li, K. Li, Recent trends in task and motion planning for robotics: A survey, ACM Computing Surveys (2023). [5] C. R. Garrett, T. Lozano-Pérez, L. P. Kaelbling, Ffrob: An efficient heuristic for task and motion planning, in: Algorithmic Foundations of Robotics XI: Selected Contrib. of the 11th Intl. Workshop on the Algorithmic Foundations of Robotics, Springer, 2015. [6] S. Srivastava, E. Fang, L. Riano, R. Chitnis, S. Russell, P. Abbeel, Combined task and motion planning through an extensible planner-independent interface layer, in: 2014 IEEE International Conference on Robotics and Automation (ICRA), 2014. [7] N. T. Dantam, Z. K. Kingston, S. Chaudhuri, L. E. Kavraki, Incremental task and motion planning: A constraint-based approach., in: Robotics: Science and systems, volume 12, Ann Arbor, USA, 2016. [8] A. Nouman, V. Patoglu, E. Erdem, Hybrid conditional planning for robotic applications, The Intl. J. of Robotics Research (2020). [9] N. Castaman, E. Pagello, E. Menegatti, A. Pretto, Receding horizon task and motion planning in changing environments, Robotics and Autonomous Systems 145 (2021). [10] N. Castaman, E. Tosello, E. Pagello, Conditional task and motion planning through an effort-based approach, in: 2018 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR), IEEE, 2018, pp. 49–54. [11] F. Lagriffoul, N. Dantam, C. Garrett, A. Akbari, S. Srivastava, L. E. Kavraki, Platform- independent benchmarks for task and motion planning, IEEE Robotics and Automation Letters 3, 4 (2018). [12] M. Terreran, S. Ghidoni, E. Menegatti, E. Villagrossi, N. Castaman, A. Gottardi, C. Eitzinger, et al., A smart workcell for cooperative assembly of carbon fiber parts guided by human actions, in: 4th Italian Conf. on Robotics and Machine Intelligence, I-RIM, Rome, 2022. [13] A. Gottardi, M. Terreran, C. Frommel, M. Schoenheits, N. Castaman, S. Ghidoni, E. Menegatti, Dynamic human-aware task planner for human-robot collaboration in industrial scenario, in: 2023 European Conference on Mobile Robots (ECMR), IEEE, 2023. [14] L. Barcellona, M. Terreran, D. Evangelista, S. Ghidoni, Clustering-based refinement for 3d human body parts segmentation, in: Intl. Conf. on Intelligent Autonomous Systems, Springer, 2022.