With the COVID-19, telepresence robots have re-gained a particular attention as tool to keep in contact with people remotely. Over the years, a lot of studies have demonstrated the eficacy of telepresence robots for communication with respect to other typologies of devices. However, most of the works have focused on the short-term interaction between the robot and the users. Herein, we put the efort to design telepresence robots for continuously caring elderly people in the domestic scenario. With this purpose, in this work, we integrate diferent AI-driven services, developed inside the SI-Robotics project, to enhance the capabilities of a commercial robotic platform and to provide ecological interaction over time.
Telepresence robots are mobile robotic platforms generally endowed with a display, microphone,
speakers, and camera, to enable people to virtually interact from a remote location. Over the
years, telepresence robots have been largely exploited in several applications from the remote
education [
Given these premises, inside the project called SI-Robotics (SocIal ROBOTics for active and healthy ageing), we are currently investigating how to design and develop advanced and customized human-robot interaction for continuously supporting elderly people at home.
Recently, studies have estimated an increment of old-age dependency ratio (number of people age 65 or above compared to those age 15-64) from the current 28% to 50% by 2060. Traditionally, ageing causes a physiological decrease of motor, sensory and cognitive abilities in people that might impede them to be self-suficient and live alone at home (i.e., the place where the most would prefer to stay). In other words, it has emerged the necessity of developing healthcare services, for instance telepresence robots, for continuously assisting elderly people in the daily activities [13, 14] and to monitor them in order to detect the occurrence of impairments and dementia [15]. In the case of telepresence robotics, despite the advance of the robotic platform in recent years [16], their use is still mainly relegated to the remote videoconferencing (e.g., call) [17, 18, 19, 20], having no or very limited autonomy [21] and attention to privacy [20]. Long-term usability and acceptability of these platforms, however, require a certain level of autonomy of the robot, for instance to facilitate the teleoperation for the users that are not very familiar with technology (e.g., family members of the assisted persons, as well as doctors and nurses) as well as to implement proactive and user-centered services designed to keep elderly people company and stimulate them from the cognitive and the physical side (e.g., when the robot is not used for remote communication) [16, 22, 6]. The next sections will provide an overview of the AI based services designed inside the SI-ROBOTICS project, considering the dual role of the robot: (a) being a companion for the primary user (i.e., the elderly); (b) being an intelligent intermediary for the secondary users (i.e., the remote users) to monitor and to interact with the person at home.
To support the continuous interaction (e.g., during all the day), with respect to other works [23, 24, 20], the robot can be used in the two modalities inside the same system: video conferencing to enable the remote communication between the primary and secondary user; intelligent and autonomous companion where the robot operates for assisting the elderly people at home by implementing Communication & Cognitive exercises, Physical exercises and Navigation functionalities. A schematic representation of the system is shown in Fig. 1. The shift of the robot’s role is determined by the Timeline-based planner & executor that copes with the Local Task Manager module and are responsible for the decision-making process. Indeed, if the one side, robots are expected to implement reactive behaviors (e.g., look for the person when he/she does not answer to the robot, avoid obstacles, modify the dialogue, etc.), on the other robots have to operate inside predefined boundaries to be acceptable and to adapt to the necessities of people. Therefore, the first is in charge of planning the personalised activities supporting elderly people (e.g., GoingTo, LookForPerson, Cognitive Exercise, Physical Exercise, VideoConferencing, Dialogue) [25], determining the temporal interval and the corresponding parameters (e.g., the most suitable physical exercises from the specific physiological status). The latter re-adapts the scheduled tasks evaluating the feasibility of performing them in the specific real-time situations and calling the corresponding sub-services according to the relations in the Fig. 1. For instance, before starting a session of cognitive and physical exercises, first, the Local Task Manager requests the activation of camera and then verifies the presence of person close to the robot as well before starting a Dialogue it requires the activation of the microphone (e.g., through the robot interface). In this way, the system can consider constraints related to privacy, by postponing and/or changing the programmed activity when the camera/microphone are not enabled.
Given the importance of creating a bond between the robot and the elderly person to be acceptable, the system personalises the interaction not only by diferentiating the plan of the daily activities, but also in terms of feedback provided (e.g., dialogue and content displayed on the robot’s tablet). Indeed, we aim creating an emphatic interaction through the Emotion recognition module that estimates the person’s emotional status [26](e.g., angry, contempt, disgust, happy, natural, sad, surprise) and triggers diferent dialogues (e.g., when the person is sad, the Dialogue service proposes sentences for the robot to make the person smile, when he/she is angry to understand the reasons) and/or the change of the animation on the tablet.
Finally, the architecture includes the Contextualized navigation module that manages autonomous and semi-autonomous robot’s movements (e.g., when the user also provides direction commands to the robot). This module is designed to facilitate the teleoperation by freeing the human to control any detailed manoeuvres as in the manual modality, and by reaching some navigation goals according to the robot’s perception (e.g., the position of the elderly provided by the skeleton tracker and the surrounding obstacles) [27] and target positions where some activities are expected to be performed (e.g., the predefined place where the session of physical exercises should start).
It is worth highlighting, as represented in Fig. 1, the system exploits the dialogue and the robot’s perception to both provide feedback to the user and to acquire information on the user himself/herself (e.g., cognitive and physical score (e.g., number of correct exercises performed), geometric features of the environment, user’s preference). Thus, the several modules are not independent among them, but, on the contrary, they exchange specific data (e.g., decoded text, decoded emotions, decoded keypoints, obstacle detections) that can activate autonomous behaviors (e.g., not a priori determined at the planner level), such as the implementation of extra dialogues starting from the feedback achieved during the execution of the physical and the cognitive exercises and the output of the navigation.
The AI-based services previously described have been modularly implemented inside the Robotic Operating System (ROS)1, the standard de facto in robotics, where each module corresponds to a ROS node. However, the transfer of the AI-based services to a commercial robot has required the introduction of additional constraints in the design. First of all, a significant limitation is associated with a few resources capabilities in terms of computing power and memory available. To deal with this aspect, we have designed an emotion recognition algorithm that depends on a thin deep architecture with only 3.9M number of parameters with respect to the 50-100M necessary in the standard approaches, reaching the 87.71% of average accuracy [26]. Coherently, we wrapper the MediaPipe Pose2 library inside a ROS package, for its capability of also running in mobile phone without requiring GPU with respect to the traditional algorithms in robotics such as OpenPose3 and SPENCER People Tracking4.
Secondly, to respect the privacy and save resources on the robot, all the services are switched of at the beginning per default except for the Timeline-based planner & executor and Local Task Manager and activate one by one when needed through ROS services.
Given the central importance of the communication in the system, the dialogue module (composed by the Speech-to-text and the Dialogue service) relies on the Google API5 and RASA6 library for learning automated and contextualized robot answers at each iteration. While the navigation system exploits the ROS Navigation stack7 for computing the best trajectories for the robot to reach the computed navigation goal [27].
We are testing the integration of the proposed AI-services on the commercial robot, Ohmni robot, mounting two RGB cameras and a 2D lidar for robot’s perception and telepresence purposes. The platform supports ROS via a Docker virtualization layer. In the ongoing pilot tests, we are evaluating the synchronization of the information exchanged between the multiple 1https://www.ros.org/ 2https://google.github.io/mediapipe/ 3https://github.com/CMU-Perceptual-Computing-Lab/openpose 4https://github.com/spencer-project/spencer_people_tracking 5https://cloud.google.com/speech-to-text 6https://rasa.com/ 7http://wiki.ros.org/navigation modules and the feasibility of the proposed services in the domestic scenario (e.g., not structured environment), representing still a challenge for roboticists (in Fig. 2).
The results of the first tests spotlight some limitations of the current robotic platform not only in terms of limited resources but also regard the battery life when all the services are active simultaneously. Furthermore, the services related to the robot’s perceptions (such as skeleton tracker, emotion recognition) can be influenced from the light conditions as well as the performance of the speech to text is strongly conditioned from the noise in the environment (however it has been testing with the typical domestic sounds such as radio, TV and home appliances active).
Future steps will include the validation of the system with the final potential users, by focusing on diferent levels of robot’s autonomy correlated with the acceptance in terms of privacy.
This work is supported by “SI-Robotics: SocIal ROBOTICS for active and healthy ageing” project (Italian M.I.U.R., PON – Ricerca e Innovazione 2014- 2020 – G.A. ARS01 01120). [2] P. Thompson, S. Chaivisit, Telepresence robots in the classroom, Journal of Educational
Technology Systems (2021) 00472395211034778. [3] G. Claudio, G. Luca, L. M. Luce, Interaction design for cultural heritage. a robotic cultural game for visiting the museum’s inaccessible areas., The Design Journal 20 (2017) S3925– S3934. [4] E. Chang, Museums for everyone: Experiments and probabilities in telepresence robots, in: Exploring Digital Technologies for Art-Based Special Education, Routledge, 2019, pp. 65–76. [5] A. Cesta, G. Cortellessa, F. Fracasso, A. Orlandini, M. Turno, User needs and preferences on AAL systems that support older adults and their carers, J. Ambient Intell. Smart Environ. 10 (2018) 49–70. URL: https://doi.org/10.3233/AIS-170471. doi:10.3233/AIS-170471. [6] S. Laniel, D. Létourneau, F. Grondin, M. Labbé, F. Ferland, F. Michaud, Toward enhancing the autonomy of a telepresence mobile robot for remote home care assistance, Paladyn, Journal of Behavioral Robotics 12 (2021) 214–237. [7] K. Winterstein, L. Keller, K. Hufstadt, N. H. Müller, Acceptance of social and telepresence robot assistance in german households, in: International Conference on Human-Computer Interaction, Springer, 2021, pp. 326–339. [8] B. Isabet, M. Pino, M. Lewis, S. Benveniste, A.-S. Rigaud, Social telepresence robots: A narrative review of experiments involving older adults before and during the covid-19 pandemic, International Journal of Environmental Research and Public Health 18 (2021) 3597. [9] C. Esterwood, L. Robert, Robots and covid-19: Re-imagining human–robot collaborative work in terms of reducing risks to essential workers, Available at SSRN 3767609 (2021). [10] T. Kanda, R. Sato, N. Saiwaki, H. Ishiguro, A two-month field trial in an elementary school for long-term human–robot interaction, IEEE Transactions on robotics 23 (2007) 962–971. [11] T. Kanda, M. Shiomi, Z. Miyashita, H. Ishiguro, N. Hagita, A communication robot in a shopping mall, IEEE Transactions on Robotics 26 (2010) 897–913. [12] M. Niemelä, P. Heikkilä, H. Lammi, V. Oksman, A social robot in a shopping mall: studies on acceptance and stakeholder expectations, in: Social Robots: Technological, Societal and Ethical Aspects of Human-Robot Interaction, Springer, 2019, pp. 119–144. [13] A. W. Group, et al., The 2018 ageing report underlying assumptions and projection methodologies, Economy, finance and the euro publications (2018). [14] G. Mois, J. M. Beer, The role of healthcare robotics in providing support to older adults: a socio-ecological perspective, Current Geriatrics Reports (2020). URL: https://doi.org/10. 1007/s13670-020-00314-w. doi:10.1007/s13670-020-00314-w. [15] M. E. Pollack, Intelligent technology for an aging population: The use of AI to assist elders with cognitive impairment, AI Magazine 26 (2005) 9. [16] B. Isabet, M. Pino, M. Lewis, S. Benveniste, A.-S. Rigaud, Social Telepresence Robots: A Narrative Review of Experiments Involving Older Adults before and during the COVID-19 Pandemic, International Journal of Environmental Research and Public Health 18 (2021).
URL: https://www.mdpi.com/1660-4601/18/7/3597. [17] F. Melendez-Fernandez, C. Galindo, J. Gonzalez-Jimenez, A web-based solution for robotic telepresence, International Journal of Advanced Robotic Systems 14 (2017) 1729881417743738. URL: https://doi.org/10.1177/1729881417743738. doi:10.1177/ 1729881417743738. [18] A. Kristofersson, S. Coradeschi, A. Loutfi, A review of mobile robotic telepresence,
Advances in human-computer interaction 2013 (2013) 1–17. [19] K. M. Tsui, M. Desai, H. A. Yanco, C. Uhlik, Exploring use cases for telepresence robots, in: 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2011, pp. 11–18. doi:10.1145/1957656.1957664. [20] M. Niemelä, L. Van Aerschot, A. Tammela, I. Aaltonen, H. Lammi, Towards ethical guidelines of using telepresence robots in residential care, International Journal of Social Robotics 13 (2021) 431–439. [21] A. Orlandini, A. Kristofersson, L. Almquist, P. Björkman, A. Cesta, G. Cortellessa, C. Galindo, J. Gonzalez-Jimenez, K. Gustafsson, A. Kiselev, A. Loutfi, F. Melendez, M. Nilsson, L. O. Hedman, E. Odontidou, J.-R. Ruiz-Sarmiento, M. Scherlund, L. Tiberio, S. von Rump, S. Coradeschi, ExCITE Project: A Review of Forty-Two Months of Robotic Telepresence Technology Evolution, Presence 25 (2016) 204–221. doi:10.1162/PRES_a_00262. [22] L. Riano, C. Burbridge, M. Mc Ginnity, A study of enhanced robot autonomy in telepresence, in: Proceedings of Artificial Intelligence and Cognitive Systems, AICS ; Conference date: 31-08-2011, AICS, Ireland, 2011, pp. 271–283. [23] T. Bakas, D. Sampsel, J. Israel, A. Chamnikar, B. Bodnarik, J. G. Clark, M. G. Ulrich, D. Vanderelst, Using telehealth to optimize healthy independent living for older adults: A feasibility study, Geriatric Nursing 39 (2018) 566–573. [24] W. Moyle, C. Jones, B. Sung, Telepresence robots: Encouraging interactive communication between family carers and people with dementia, Australasian journal on ageing 39 (2020) e127–e133. [25] R. De Benedictis, G. Beraldo, R. R. Devaram, A. Cesta, G. Cortellessa, Enhancing telepresence robots with AI: Combining services to personalize and react, in: Proceedings of AIxIA 2021 – Advances in Artificial Intelligence, 2021. [26] R. R. Devaram, G. Beraldo, R. De Benedictis, M. Mongiovì, A. Cesta, LEMON: A lightweight facial emotion recognition system for assistive robotics based on dilated residual convolutional neural networks, Sensors 22 (2022) 3366. [27] G. Beraldo, L. Tonin, A. Cesta, E. Menegatti, Brain-driven telepresence robots: A fusion of user’s commands with robot’s intelligence, in: M. Baldoni, S. Bandini (Eds.), AIxIA 2020 – Advances in Artificial Intelligence, Springer International Publishing, Cham, 2021, pp. 235–248.