Multi-Modal Interaction for Soft Continuum Robots during Post-Earthquake Search Operations Rajashekhar V S1,* , Gowdham Prabhakar2 1 PhD Scholar, Department of Design, Indian Institute of Technology - Kanpur, Uttar Pradesh, India 2 Assistant Professor, Department of Design, Indian Institute of Technology - Kanpur, Uttar Pradesh, India Abstract The post-earthquake search and rescue operations assisted by robots will be efficient if the human- robot interaction (HRI) is easy to perform. It can be done using soft-bodied robots, which are better in performance when compared to their rigid-bodied counterparts. Among them, soft continuum robots (SCR) can be used due to their sleek nature. The SCRs are of two types: soft continuum manipulators (SCM) and soft snake robots (SSR). Although these two robots are functionally different, they are structurally similar. Therefore, these robots can be made modular and reconfigurable. The SCM can assist the medical team with pick-and-place operations, and the SSR can traverse confined spaces that occur after disasters such as earthquakes. It is inferred from our survey that not much of the research work in the literature focuses on the HRI methods for SCR in post-disaster situations. Therefore, in this work, we focus on developing the HRI methods for SCR, which are modular and reconfigurable. Keywords Soft continuum manipulator, Soft snake robot, Multi-modal interaction, Hand gesture, Eye tracker 1. Motivation for research With the number of natural and man-made disasters (such as earthquakes and the collapse of buildings) happening in the world increasing yearly, there is a need for faster search and rescue operations to save lives [1]. This requires trained manpower and assisting devices. Among the assisting devices, robots can play a crucial role in traversing through the debris of collapsed buildings. These robots need to be modular so they can be easily replaced when they run out of power or are damaged during operations in a cluttered environment. Also, the rescuers need to learn how to operate the robots quickly. This can be done only if a simple robotic system with an easy interaction interface is designed. Therefore, my work aims to explore various input modalities that can be used to control the robots in cluttered environments that occur post-disasters. Joint Proceedings of the ACM IUI Workshops 2024, March 18-21, 2024, Greenville, South Carolina, USA * Corresponding author. $ raja23@iitk.ac.in (R. V. S); gowdhampg@iitk.ac.in (G. Prabhakar) € https://vsrajashekhar.weebly.com/ (R. V. S); https://www.gowdhamprabhakar.com/ (G. Prabhakar)  0000-0003-4826-805X (R. V. S); 0000-0002-1579-2843 (G. Prabhakar) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) 1 CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings Rajashekhar V S et al. CEUR Workshop Proceedings 1–6 Figure 1: The conceptual computer aided design of the proposed soft continuum robot (a) The modular mechanism of the soft continuum robot (b) The three modules form the soft continuum manipulator (c) The four modules form the soft snake robot 2. Goal and research questions 2.1. Goal Our work aims to develop an interaction platform modular for a reconfigurable soft continuum robot that can be used as a manipulator and a snake robot. This setup is planned to be used during post-earthquake search operations. A number of modular mechanisms form the SCR, where the conceptual design of one is shown in Figure 1 (a). The three modules can be connected parallelly to form a soft continuum manipulator, as shown in Figure 1 (b). The medical team can use this configuration to handle medical devices sterilely. The four modules can be connected in series to form a soft continuum snake robot, as shown in Figure 1 (c). This configuration can be used for search operations inside the damaged buildings. Since the mechanism is modular, the modules can be easily replaced when damaged. The main focus of the work is to study various feasible means of interaction when the SCR is in the form of a manipulator and snake robot. This is because the functions of the manipulator and snake robot are entirely different. Hence, the input modalities to operate them would also vary. Therefore, we aim to study and map the modalities with the SCR for multi-modal interaction. 2.2. Research questions The following are the research questions I have posed, which I aim to address during my studies. 1. How to design a human-robot interface for a soft continuum robot that is modular and reconfigurable? 2. What kind of interaction modalities are efficient to operate a soft continuum robot in post-earthquake scenarios? 3. Can pneumatic or hydraulic actuators efficiently control the soft continuum robots in post-earthquake scenarios? 4. Will self-reconfiguration of the soft continuum robots be done efficiently so that the search operation is not hindered? 2 Rajashekhar V S et al. CEUR Workshop Proceedings 1–6 I hope that finding answers to these questions will pave the way for creating an interface for soft modular, reconfigurable continuum robots that can be used in multiple scenarios that arise post-earthquakes. 3. Related work that frames the research In the year 2011, after the earthquake and tsunami that occurred in the Tohoku region of eastern Japan, the Fukushima Daiichi Nuclear Power Plant was damaged due to hydrogen explosions, and nuclear reactions were uncontrollable. The robots named Quince, which were belt-driven, had to be retrofitted to carry out the tasks inside the reactor [2]. Sensors such as a dosimeter, camera, laser range scanner, and temperature sensors for motors were mounted on the robot. It can be inferred that the robot has to be tethered to establish communication inside the radiation- prone nuclear reactor. It also involved initial personnel training in the radiation measurement task. The robot was driven using a joystick as an input device, and the feedback was obtained on the display screen. Here, the rescuer had only one mode of interaction with the robotic system. A robot named WALK-MAN, which had the upper body of a humanoid and a wheeled base, was used to perform various tasks inside the damaged building post the 2016 Italy earthquake [3]. The studies involved building 3D maps for the damaged building, measuring the structural damages, recovering objects from the building, and installing sensors inside the house. The communication between the robot and the ground station was wireless. It is reported that scene understanding is a difficult task, and developing autonomous methods is challenging. The literature study on urban search operations showed that snake robots were used for searching operations in real-world applications. This snake robot was built by the Carnegie Mellon University Biorobotics Laboratory and was used for the search operation after the 2017 earthquake in Mexico City. During this operation, the gait parameters for the snake robot had to be adapted manually depending on the feedback displays and camera feed. This highlights the importance of mid-level autonomy [4]. Also, the workers at the earthquake site expected the snake robot to have a microphone and speaker since it had a camera at the head. They also expected the robot to carry food and water to the stranded people. It is also reported that viewing the laptop monitor in the bright sunlight was difficult. In situations like this, a multi-modal interface will play a crucial role in communicating with the robot. It can be found in the literature that robots are being used in real-world applications where disaster has occurred. More of these robots will be used to assist humans during post-disaster situations. Depending on the situation, the mode of communication between the robot and the humans in the ground station can be wired or wireless. 4. Methods/approach to reach the goal The fluid-driven soft devices can be used in human-robot interaction [5]. In recent years, socially assistive soft robots that possess interaction skills have been developed [6]. Understanding these will help build a fluid actuated SCR that can have an efficient interaction. Looking into the studies related to stability and gait control during teleoperation of soft modular robots 3 Rajashekhar V S et al. CEUR Workshop Proceedings 1–6 [7, 8, 9] will help in deciding the type of input modality suitable for searching operations post earthquakes. The soft continuum robot mechanisms in the literature will be studied, and a new mechanism with three degrees of freedom will be designed. It will be modeled using the FreeCAD𝑇 𝑀 software and imported into the SOFA [10] for analysis. The SCR module mechanism for the various actuating mechanisms will be explored. When fluid-based actuators operate the SCRs, they can be extended to give food (liquid) or oxygen (gas) to the humans stranded inside the collapsed building. This factor will be looked into while designing the SCR mechanism. Having decided on the interaction modalities for the SCM and SSR based on the robot design, the robot will be fabricated using an additive manufacturing technique (Fused Filament Fabrication) using thermoplastic polyurethane (TPU) as the material. A mock setup of a collapsed building will be made in an open environment exposed to sunlight. There will be mannequins placed in the middle of the collapsed buildings which are considered to be humans. Then, the experiments with the SCR will be done with the participants. Based on these results, the best way to interact with the SCR will be decided. 5. Next steps to conduct the research Stage 1: Literature survey Conduct a literature survey on various means of input modalities by following the PRISMA 2020 statement [11]. Similarly, do a literature survey for the soft sensors, soft actuators, and soft manufacturing techniques used in soft continuum manipulators and soft snake robots. In this process, identify the gaps in the literature by creating a trend map. Stage 2: Design the soft continuum robot mechanism Based on the gaps identified, design a three-degree-of-freedom soft module mechanism that can bend, twist, expand, and contract. The actuator selection has to be done, followed by the soft sensors to be mounted on the robot. Stage 3: Create a bench-top prototype Using the rapid prototyping techniques, create a bench-top prototype of the SCRs. Then, test their functioning by using the pre-programmed motions. Stage 4: Learn SOFA and perform the analysis Learn the SOFA framework [10] so that the robot can be simulated and the performance can be tested. This can be used in hardware-in-loop simulation. Stage 5: Explore the various input modalities The bench-top prototype has to be inter- faced using the different input modalities such as physiological parameter tracking, eye tracking, hand tracking, body and face tracking, voice recognition, gesture input, in virtual, augmented and mixed reality environment. Stage 6: Create the final setup and perform the experiment The mock setup of the post-earthquake scenario, as mentioned earlier, would be set up. The user study will use the SCR in the manipulator mode with medical assistants as participants. Then, the user study of the soft snake robot will be done using rescue workers who will be assisting in the search operations after the earthquake. The various input modalities will be tested for efficacy with the participants during the experiment, and the data will be collected. Stage 7: Analysis of the data The data collected from the user study will be analyzed 4 Rajashekhar V S et al. CEUR Workshop Proceedings 1–6 using qualitative and quantitative methods. Based on their results, the best modalities for the multi-modal interaction will be chosen by enabling interaction during physical and situational impairments. 6. Conclusions This work describes a conceptual design of a module that forms a modular mechanism for a soft continuum manipulator and soft snake robot. The goals and research questions were presented, focusing on using these robots for post-earthquake search operations. The related works were analyzed, followed by the methods, and the next steps to conduct the research were presented. This work will lead to the findings where the best multi-modal interactions are obtained to control the soft continuum robot during post-earthquake search operations. References [1] D. Alexander, Natural disasters, Routledge, 2018. [2] K. Nagatani, S. Kiribayashi, Y. Okada, K. Otake, K. Yoshida, S. Tadokoro, T. Nishimura, T. Yoshida, E. Koyanagi, M. Fukushima, et al., Emergency response to the nuclear accident at the fukushima daiichi nuclear power plants using mobile rescue robots, Journal of Field Robotics 30 (2013) 44–63. [3] F. Negrello, A. Settimi, D. Caporale, G. Lentini, M. Poggiani, D. Kanoulas, L. Muratore, E. Luberto, G. Santaera, L. Ciarleglio, et al., Humanoids at work: The walk-man robot in a postearthquake scenario, IEEE Robotics & Automation Magazine 25 (2018) 8–22. [4] J. Whitman, N. Zevallos, M. Travers, H. Choset, Snake robot urban search after the 2017 mexico city earthquake, in: 2018 IEEE international symposium on safety, security, and rescue robotics (SSRR), IEEE, 2018, pp. 1–6. [5] P. Polygerinos, N. Correll, S. A. Morin, B. Mosadegh, C. D. Onal, K. Petersen, M. Cianchetti, M. T. Tolley, R. F. Shepherd, Soft robotics: Review of fluid-driven intrinsically soft devices; manufacturing, sensing, control, and applications in human-robot interaction, Advanced Engineering Materials 19 (2017) 1700016. [6] Y.-C. Sun, M. Effati, H. E. Naguib, G. Nejat, Softsar: The new softer side of socially assistive robots—soft robotics with social human–robot interaction skills, Sensors 23 (2022) 432. [7] D. M. Perera, D. D. Arachchige, S. Mallikarachchi, T. Ghafoor, I. Kanj, Y. Chen, I. S. Godage, Teleoperation of soft modular robots: Study on real-time stability and gait control, in: 2023 IEEE International Conference on Soft Robotics (RoboSoft), IEEE, 2023, pp. 01–07. [8] H. Ham, M. Park, T. Park, X. Gao, Y.-L. Park, M. J. Park, Teleoperation of soft robots with real-time fingertip haptic feedback using small batteries, Advanced Materials Technologies (2023) 2300070. [9] F. Stroppa, M. Selvaggio, N. Agharese, M. Luo, L. H. Blumenschein, E. W. Hawkes, A. M. Okamura, Shared-control teleoperation paradigms on a soft-growing robot manipulator, Journal of Intelligent & Robotic Systems 109 (2023) 30. [10] F. Faure, C. Duriez, H. Delingette, J. Allard, B. Gilles, S. Marchesseau, H. Talbot, H. Courte- cuisse, G. Bousquet, I. Peterlik, et al., Sofa: A multi-model framework for interactive 5 Rajashekhar V S et al. CEUR Workshop Proceedings 1–6 physical simulation, Soft tissue biomechanical modeling for computer assisted surgery (2012) 283–321. [11] M. J. Page, J. E. McKenzie, P. M. Bossuyt, I. Boutron, T. C. Hoffmann, C. D. Mulrow, L. Shamseer, J. M. Tetzlaff, E. A. Akl, S. E. Brennan, et al., The prisma 2020 statement: an updated guideline for reporting systematic reviews, International journal of surgery 88 (2021) 105906. 6