Engineering a Software System to Support Post-Stroke Patients in their Training Activities with a Humanoid Robot Peter Forbrig, Alexandru Bundea University of Rostock, Chair of Software Engineering, Albert-Einstein-Str. 22, 18059 Rostock, Germany Abstract The paper discusses software architecture for the system E-BRAiN (Evidence-Based Robot- Assistance in Neurorehabilitation) that is intended to support patients after stroke in their rehabilitation. The humanoid robot Pepper gives instructions, provides pictures and videos. Its main task is to motivate patients in their training exercises. A software architecture is suggested that uses a kind of thin robot. The humanoid robot Pepper is used as input and output device only. Most computations are performed externally. Communication between different components is realized by message exchange. The communication between robot and patient is supported by different modalities like spoken, text or buttons. It is supported by a natural language generation system that considers the characteristics of a patient. It can generate e.g. simple and complex language. Additionally, the system adapts speech speed and level of support to those patient characteristics. Keywords 1 Humanoid Robot, Software Architecture, MQTT Application, Rehabilitation Apps 1. Introduction We want to discuss the specific challenges of engineering interactive systems for disabled people in the context of medical training exercises. Within our project E-BRAiN (Evidence-Based Robot- Assistance in Neurorehabilitation) [3] we have been developing a system for post-stroke patients with mild to severe arm paresis and visual impairment. During the last three years, several training tasks supported by a humanoid robot Pepper [19] were implemented. We will first give an overview of the kind of implementations that we developed. Afterwards, we will discuss the challenges for developing such systems and discuss technologies that allow adaptations to different contexts. 2. Rehabilitation training for post-stroke patients The training tasks that we implemented can be grouped into four categories. They are called Arm- Ability Training (AAT), Arm-Basis-Training (ABT), Mirror Therapy (MT) and Neglect Therapy (NT). AAT is intended for patients with mild problems with one arm. They have to execute exercises of different forms that are visualized in the left part of Figure 1. One task is the crossing out of as many as possible Os. An app was developed that checks the correct performance. Additionally, the humanoid robot gives feedback about current and previous performance. The right part of Figure 1 shows a student testing the app. EICS ’22: Engineering Interactive Computing Systems Conference, June 21–24, 2022, Sophia Antipolis, France. EMAIL: peter.forbrig@uni-rostock.de (A. 1); alexandru-nicolae.bundea@uni-rostock.de (A. 2) ORCID: 0000-0003-3427-0909 (A. 1); 0000-0001-8315-3405 (A. 2); ©️ 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) Figure 1: Exercises for the Arm-Ability Training (AAT) according to [10] and digitalized version of a training task (right). The Arm-Basis Training is designated for post-stroke patients with more severe handicaps. They need at the beginning a helping hand of a supporter while performing training tasks. Figure 2 gives an impression of the exercises with arms and fingers. Additionally, one can see one experiment where sensors allow feedback from the robot about the correct performance. Figure 2: Examples of the ABT training movements with a therapist (from [11]). For patients with very severe one-sided handicaps, the Mirror Therapy can be applied. During such therapy, a patient sits at 90 degrees next to a mirror placed on a table with his healthy arm in front of the mirror and the handicapped arm behind the mirror. They have to perform training tasks with their non-affected arm and have to look continuously into the mirror image of their healthy arm while they have to imagine that their disabled arm is moving. According to the medical experts changes in the brain can be induced in this way. Figure 3 gives an impression of how a patient has to sit in relation to the mirror. Figure 3: Situation for exercises with a mirror. Neglect is a problem to orient attention spatially. It can affect all sensory and motor modalities, occurs after hemispheric brain lesions, more frequently after right brain damage, e.g. after stroke [9]. In the visual domain affected patients have difficulties orienting their visual attention to the side opposite to the brain lesion. As a result, patients might visually neglect one side of their body and/or environment, e.g. leave the entire left-hand side of their meals untouched (in case of right brain damage causing neglect). The Neglect Therapy provides exercises to reduce this problem by promoting a shift of visual attention to the neglected side, based on various training techniques including visual exploration training. Figure 4 and Figure 5 provide two examples for visual exploration training tasks. Figure 4: Training task for Neglect where patients have to point at specific objects. Figure 5: Exploring a picture of objects of type cow. Figure 4 gives an impression of the situation where a patient has to point at specific objects. A task could be to point at the object that is different from the other objects or to point at all objects that are different from the majority of objects. Another training task could be to touch on one object and say “now” when this object leaves the screen. Figure 5 demonstrates different stages of one training task where a patient has to point at all objects of type cow. Found objects become red. When a patient stops pointing but did not identify all objects, Pepper gives hints like:” Have a look at the top left corner. There seems to be another cow”. In all cases, the robot provides instructions, jumps in when help is necessary, comments on the results and provides motivation. The spoken language of the robot is additionally displayed as text on a tablet. The robot Pepper is also able to understand spoken language. However, because of possible pronunciation problems of post-stroke patients, we decided to have a specific device to express yes by pushing a button. The question from Pepper: “Are you ready for the next training task?” is an example where a patient has to press this button. 3. The E-BRAiN architecture In our project, we started to implement the first spoken dialogs on the humanoid robot itself. After recognizing the need of integrating further devices like tablets we started with direct communication. However, we run into problems because sometimes the WiFi caused deadlocks. We had a look at different possible technologies and ended up with MQTT (Message Queuing Telemetry Transport) [14]. It is easy to use, does not need many resources and checks whether a message really reached its destination. Having this technology implemented we started to implement the dialogs of the robot externally. In this way, we got a kind of thin robot that only interprets messages and executes them. However, no knowledge is stored on the robot. This software architecture provides a lot of flexibility. New software and new devices can be integrated relatively easily. Figure 6 tries to visualize the concept of the architecture. Management Training Apps Pepper Flickboard MQTT Server Patient Therapist Therapy Administration Tablet1 Tablet2 Natural Language User Models Generation Interaction Server (RosaeNLG) Touch Screen Dialog Scripts Dialog Control Legend Data flow Interaction Programmer Figure 6: Architecture of the E-BRAiN system. All components of the E-BRAiN system communicate via MQTT messages that are managed by a corresponding server, which is the center of our architecture. Clients can register for specific topics and are informed when a message to this topic arrives. On the left-hand side, one can see our administration software, which is a web-application that manages all data of patients, their attributes, medical scores and training results. Additionally, their plans for training exercises are stored in this application. Programmers have to provide Python scripts that represent state-machine specifications. On an interaction, the server runs an interpreter of such state machines. In this way, the dialog scripts can be executed. However, before the scripts are executed they are enhanced by details of included sentences (Natural Language Generation). In this way attributes of the users can influence the generation process. During the interpretation of the final scripts, messages are sent to the MQTT-server. There are four types of messages that will be discussed by example. (1) de.add_message_for_state(’state/ s1’, {’text’ : ’hello’, ’image’ : ’ebrain.jpeg’}) (2) de.add_event_transition(source=’state/s1’, target=’state/s2’, guard=patient_is_ready) (3) de.add_timed_transition(source=’state/s1’, target=’state/s3’, timeout_in_s=5) (4) de.add_interaction_for_state(’state/s3 ’, video=’video.mp4’) The object de represents the dialog engine that will get the messages. The first statement results in the construction of sub-state s1 of state. While entering the state a MQTT message with the corresponding name value pairs (e.g. [text : hello] and [image : ebrain.jpg]) is generated and sent. An event transition evaluates a condition that can be specified as a function. In the case of the example, the function is called patient_is_ready. After a certain interval of time, a timed transition is executed without any side effects. The fourth statement results in a MQTT message for the presentation of a specific video. Additionally, to the video keyword, other options like image, text and say can be used. Similar examples can be found in [7]. Additionally, a visualization of dialog scripts and an interaction pattern are discussed there as well. Some training apps need specific devices like a large touch screen or tablets. Their devices represent a group of training tasks that communicate via MQTT messages as well. 4. Discussion and related work It is not intended to replace a human therapist totally with a robot. The first session of training tasks is performed with a human therapist. The robot comes into the game during the second session only. Nevertheless, a patient can get the instructions of a task again by the robot. However, this is optional and a patient can stop it at any time. 4.1. General design decisions to support disabled patients With the robot Pepper, one can interact touch-based with its tablet. However, interaction is not so easy because the robot is moving and is not always standing very next to a patient. Therefore, a specific tablet was introduced especially for human-robot interaction. It can be located on the table next to a patient. Spoken sentences of the robot are additionally presented as text on this tablet. This text is not as volatile as spoken sentences. In this way, patients can follow the instructions of the robot better. Spoken input to the robot Pepper is possible as well. However, because patients are often not able to speak loud and clear, the input to the robot is reduced to a “confirmation” can also be done by putting the hand on a flickboard. 4.2. Adaptation to specific patients’ needs User modelling in HCI has a long tradition. Fischer [6] provides a nice discussion about this history and about adaptive and adaptable systems. There are general attributes about age and education that are important for interactive systems. However, there are specific aspects that have to be considered for users of the E-BRAiN system. Several attributes describe the medical state of the patient. They are already fixed because of long medical research. They were provided by our experts from medicine. These attributes are evaluated to measure the success of the rehabilitation. However, they also can influence the training task performance. For the ABT such attributes can influence the range of an arm movement and in this way the feedback from the robot. Additionally, the results of the training tasks are stored in our user models. They were relatively simple to identify. There was only the question of granularity. However, to support patients during exercises further attributes might be helpful. For this purpose, the repertory grid method [19] was used to identify knowledge from therapists about patients and helpers. For patients, the attribute “need for attention” and “cognitive performance” were identified. The attributes “authority” and “openness to mistakes made” were collected for helpers. Feedback of the robot can be positive (e.g. “Great results with your 40 points, you’re approaching your goal of playing chess fast!”) neutral (e.g. “You reached 40 points, remember your goal of playing chess to be able to train again.” ) or challenging ( e.g. “With reached 40 points, I am sure you are still holding back your performance.”). Some patients might always need positive feedback. However, others have to be pushed a little bit with challenging sentences. The decision has to be made based on attributes and rules. Bundea et al. [1] discuss these aspects in more details. The adaptation to personal feedback is additionally supported by a natural language generator. RosaeNLG [17] was extended by some members of our project team in such a way that it fits our system [3]. It allows the specification of different language levels from simple to very complex. In this way, the communication can be adapted to the cognitive characteristics of patients. In the literature we did not find a lot about patient models and human-robot interaction. Winkle et al. [21] discuss general design implications for social assistive robots and how the engagement of users can be improved. They used stand-alone applications on Pepper for people that they hired at the university campus. They were not disabled. Quinn et al. [15] describe an ontological model for patients with diabetes. It consists of attributes like age group, physical activity, language and health condition. They used publications provided by medical associations and academic literature. Schrum et al. [16] use a humanoid robot Pepper for patients with dementia. The patients have to copy the body movement of Pepper, while their heart rate was measured by a sensor. Unfortunately, the human-robot interaction is not described in detail. Unfortunately, the same is true for Amato et al. [1]. They use Pepper for children that play games on the tablet of the robot. There seems to be no user model. The Nao robot is used by Feng et al. [4] to interact with children having autism. The authors postulate: "1) If the child makes a wrong action, the robot will point out the mistake and ask the child to do it again. If the child fails three times, this action will be abandoned. 2) If the child is distracted, the robot reminds him to focus attention. 3) When the child completes the task correctly, the robot praises him; when the child completes the task incorrectly, the robot encourages him." About rehabilitation care of elderly people in Japan using Pepper is reported by Tanioka [20]. Different medical signals are measured. As a result, it is reported: "At the end of exercise, sympathetic nerve activity ratio has been increased" and “Elderly persons gain well-being from dialogues and conversations with Pepper robots”. There seems to be an objective and subjective success. Similar to the E-BRAiN project Polak and Levy-Tzedek [12] focus with their system on post- stroke long-term rehabilitation. A patient and a humanoid robot Pepper are located on different sides of a shelve. The patient has to order jars on the shelve. The order is presented on a picture of Peppers’ tablet. Additionally, Pepper provides game instructions and feedback. The feedback is described as follows in the paper: "After each trial, the robot either gives the patient feedback on the timing (e.g., “try to do it faster next time”) or on the success on the task (e.g., “you succeeded!”, “you were not right this time, but I’m sure you will make it next time!”)." Manca et al [9] use Pepper for training tasks for older persons with mild dementia. After serious games Pepper provides feedback like: "Congratulations! Correct answer" or "Very good! Right answer”. The selection is done randomly. Communication acts for dialogs are discussed by Fernandez-Rodicio et al. [5]. They use a robot called Mini and focus on spoken interaction. The target users are elderly people with light dementia. Even though Mini does not look very nice, the results of their experiments were quite positive. The only reference we found where software is running outside a robot is Pulido et al. [13]. They provide an architecture for a mirror game with the Nao robot. The game is called mirror because a patient has to copy the postures of Nao. Because the robot has legs it is a whole-body activity. The correct performance is checked by a Kinect sensor. A decision support system selects the next action. Further devices are not involved. However, this should be possible. The authors focus on an expert friendly system. They want to give health professionals the opportunity to adapt each treatment to every patient. Communication between robot and patient seems to be not in the focus. Summary and Outlooks Software development is challenging. It is especially challenging when users have special needs. Within our E-BRAiN project we have been developing software for supporting the rehabilitation of patients after stroke. We focus on Arm-Ability Training (patients can do already a lot), Arm-Basis- Training (patients need a helper), Mirror Therapy (patients can only move one of their arms well) and Neglect Therapy (patients have problems with special orientation). We discussed some examples of those therapies. To implement a flexible system, we suggested a software architecture that uses the MQTT technology where components communicate via messages. This architecture allows the integration of several external systems that allow the adaptation of dialogs to the specific needs of a patient. This is supported by a natural language generator, a detailed user model and rules for different emotions. The following research questions might be tackled in the future: • Did we analyze the correct number of attributes for patients? • Which further rules for adapting the dialog to patients can be found? • Has the level of natural language communication be adapted during a therapy? First experiments with patients were very positive. However, because of corona restrictions in clinics the number of therapies was very limited yet. Having further hints from patients will help to polish the communication. We also had to learn that patients are not the kind of users we were working beforehand. Ideas of user-centered design cannot be applied as easily. Patients are protected by law. An ethics committee asks for applications of experiments and they want to see medical reasons. Therefore, a certain quality of the software has to be provided and prototyping is difficult. User-centered activities in clinical environments have to be planned detailed and early, an aspect we were not aware of. 5. Acknowledgements This joint research project “E-BRAiN - Evidence-based Robot Assistance in Neurorehabilitation” is supported by the European Social Fund (ESF), reference: ESF/14-BM-A55-0001/19-A02, and the Ministry of Education, Science and Culture of Mecklenburg-Vorpommern, Germany. The sponsors had no role in the decision to publish or any content of the publication. 6. REFERENCES [1] Federica Amato, Marianna Di Gregorio, Clara Monaco, Monica Sebillo, Genoveffa Tortora, and Giuliana Vitiello. 2020. The Therapeutic Use of Humanoid Robots for Behavioral Disorders. Proceedings of the International Conference on Advanced Visual Interfaces. Association for Computing Machinery, New York, NY, USA, Article 98, 1–3. DOI:https://doi.org/10.1145/3399715.3399960 [2] Bundea, A. and Forbrig, P. (2022): Patient and therapist model attributes for social robot stroke therapies based on implicit knowledge from expert interviews, accepted for 15th International KES-HCIS Conference, June 20-22, Rhodes, Greece. [3] E-BRAiN: https://wwwswt.informatik.uni-rostock.de/webebrain/, last visited March 7, 2022 [4] Felske, T., Bader, S., and Kirste, T. (2022): Automatic Generation of Personalised and Context- Dependent Textual, accepted for German Journal of Artificial Intelligence. p. 1-.5 [5] Feng, Y., Jia, Q. and Wie, W. (2018): A Control Architecture of Robot-Assisted Intervention for Children with Autism Spectrum Disorders. 2018 (2018), 1–12. Issue Article ID 3246708. https://doi.org/10.1155/2018/3246708 [6] Enrique Fernández-Rodicio, Álvaro Castro-González, Fernando Alonso-Martín, Marcos Maroto- Gómez, and Miguel Á. Salichs. 2020. Modelling Multimodal Dialogues for Social Robots Using Communicative Acts. Sensors 20, 12 (2020). https://doi.org/10.3390/s20123440 [7] Fischer, G. User Modeling in Human–Computer Interaction. User Modeling and User-Adapted Interaction 11, 65–86 (2001). https://doi.org/10.1023/A:1011145532042 [8] Forbrig, P., Bundea, A., and Bader, S. .(2021):. Engineering the Interaction of a Humanoid Robot Pepper with Post-Stroke Patients During Training Tasks. In Companion of the 2021 ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '21). Association for Computing Machinery, New York, NY, USA, 38–43. DOI:https://doi.org/10.1145/3459926.3464756 [9] Kerkhoff G und Schmidt L. Neglect und assoziierte Störungen. 2018. Hogrefe Verlag, Göttingen (ISBN 9783840928543). [10] Manca, M., Paternò, F.; Santoro, C., Zedda, E., Braschi, C., Franco, R., Sale, A. (2020):, The Impact of Serious Games with Humanoid Robots on Mild Cognitive Impairment Older Adults, International Journal of Human-Computer Studies, 2020, ttps://doi.org/10.1016/j.ijhcs.2020.102509 [11] Platz, T., Denzler, P., Kaden, B., Mauritz, K.-H. (1994). Motor leaning after recovery from hemiparesis, Neuropsychologia, Volume 32, Issue 10, October 1994 pp. 1209-1223, https://doi.org/10.1016/0028-3932(94)90103-1 [12] Platz, T., Pinkowski, C., van Wijck, F., Kim, I.H., di Bella, P., Johnson, G. (2005). Reliability and validity of arm function assessment with standardized guidelines for the Fugl-Meyer Test, Action Research Arm Test and Box and Block Test: a multicentre study. Clin Rehabil.,1 9(4):404-11. doi: 10.1191/0269215505cr832oa. [13] Polak, R.F., and Shelly Levy-Tzedek, S. (2020): Social Robot for Rehabilitation: Expert Clinicians and Post-Stroke Patients’ Evaluation Following a Long-Term Intervention. In Proceedings of the 2020 ACM/IEEE International Conferenc on Human-Robot Interaction (Cambridge, United Kingdom) (HRI ’20). Association for Computing Machinery, New York, NY, USA, 151–160. https://doi.org/10.1145/3319502.3374797 [14] Pulido, J.C., Suarez-Mejias, C., Gonzalez, J.C., Ruiz, A.D., Ferri, P.F., Sahuquillo, P. E. M., De Vargas, C.E.R., Infante-Cossio, P., Calderon, C.L.P., and Fernandez, F. (2019): A Socially Assistive Robotic Platform for Upper-Limb Rehabilitation: A Longitudinal Study With Pediatric Patients. IEEE Robotics Automation Magazine 26, 2 (2019), 24–39. https://doi.org/10.1109/MRA.2019.2905231 [15] Pulver, T. (2019). Hands-On Internet of Things with MQTT: Build connected IoT devices with Arduino and MQ Telemetry Transport (MQTT), Packt Publishing Ltd., Birmingham [16] Quinn S, Bond R, Nugent C (2017) Ontological modelling and rule-based reasoning for the provision of personalized patient education. Expert Systems 34:e12134. https://doi.org/10.1111/exsy.12134 [17] Schrum, M., Park, C.H., and Howard, A. (2019): Humanoid therapy robot for encouraging exercise in dementia patients. In Proceedings of the 14th ACM/IEEE International Conference on Human- Robot Interaction (HRI '19). IEEE Press, 564–565. [18] Stoeckl´e L (2021) Rosaenlg - a natural language generation library for node.js. URL https://rosaenlg.org, version 2.1.7 [19] SoftBank Robotics: https://www.softbankrobotics.com/corp/robots/, last visited November 11th, 2019 [20] Tan, F. B. and M. Hunter, G. (2002):. “The Repertory Grid Technique: A Method for the Study of Cognition in Information Systems.” MIS Quarterly, vol. 26, no. 1, Management Information Systems Research Center, University of Minnesota, 2002, pp. 39–57, https://doi.org/10.2307/4132340. [21] Tanioka, T. (2019): Nursing and Rehabilitative Care of the Elderly Using Humanoid Robots. 66, 1.2. (2019), 19–23. https://doi.org/10.2152/jmi.66.19 [22] Winkle K, Lemaignan S, Caleb-Solly P., Leonards, U.,Turton, A., Bremner, P. (2019): Effective Persuasion Strategies for Socially Assistive Robots. In: 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, pp 277–285