The paper discusses software architecture for the system E-BRAiN (Evidence-Based RobotAssistance in Neurorehabilitation) that is intended to support patients after stroke in their rehabilitation. The humanoid robot Pepper gives instructions, provides pictures and videos. Its main task is to motivate patients in their training exercises. A software architecture is suggested that uses a kind of thin robot. The humanoid robot Pepper is used as input and output device only. Most computations are performed externally. Communication between different components is realized by message exchange. The communication between robot and patient is supported by different modalities like spoken, text or buttons. It is supported by a natural language generation system that considers the characteristics of a patient. It can generate e.g. simple and complex language. Additionally, the system adapts speech speed and level of support to those patient characteristics.
We want to discuss the specific challenges of engineering interactive systems for disabled people in
the context of medical training exercises. Within our project E-BRAiN (Evidence-Based
RobotAssistance in Neurorehabilitation) [
The Arm-Basis Training is designated for post-stroke patients with more severe handicaps. They need at the beginning a helping hand of a supporter while performing training tasks. Figure 2 gives an impression of the exercises with arms and fingers. Additionally, one can see one experiment where sensors allow feedback from the robot about the correct performance.
For patients with very severe one-sided handicaps, the Mirror Therapy can be applied. During such therapy, a patient sits at 90 degrees next to a mirror placed on a table with his healthy arm in front of the mirror and the handicapped arm behind the mirror. They have to perform training tasks with their non-affected arm and have to look continuously into the mirror image of their healthy arm while they have to imagine that their disabled arm is moving. According to the medical experts changes in the brain can be induced in this way. Figure 3 gives an impression of how a patient has to sit in relation to the mirror.
Neglect is a problem to orient attention spatially. It can affect all sensory and motor modalities,
occurs after hemispheric brain lesions, more frequently after right brain damage, e.g. after stroke [
The Neglect Therapy provides exercises to reduce this problem by promoting a shift of visual attention to the neglected side, based on various training techniques including visual exploration training. Figure 4 and Figure 5 provide two examples for visual exploration training tasks.
In our project, we started to implement the first spoken dialogs on the humanoid robot itself. After recognizing the need of integrating further devices like tablets we started with direct communication. However, we run into problems because sometimes the WiFi caused deadlocks. We had a look at different possible technologies and ended up with MQTT (Message Queuing Telemetry Transport) [14]. It is easy to use, does not need many resources and checks whether a message really reached its destination.
Therapist
Therapy Administration
User Models Natural Language
Generation Interaction Server (RosaeNLG)
Pepper MQTT Server
Dialog Scripts Dialog Control
Programmer
Flickboard
Tablet1
Having this technology implemented we started to implement the dialogs of the robot externally. In this way, we got a kind of thin robot that only interprets messages and executes them. However, no knowledge is stored on the robot. This software architecture provides a lot of flexibility. New software and new devices can be integrated relatively easily. Figure 6 tries to visualize the concept of the architecture.
All components of the E-BRAiN system communicate via MQTT messages that are managed by a corresponding server, which is the center of our architecture. Clients can register for specific topics and are informed when a message to this topic arrives.
On the left-hand side, one can see our administration software, which is a web-application that manages all data of patients, their attributes, medical scores and training results. Additionally, their plans for training exercises are stored in this application.
Programmers have to provide Python scripts that represent state-machine specifications. On an interaction, the server runs an interpreter of such state machines. In this way, the dialog scripts can be executed. However, before the scripts are executed they are enhanced by details of included sentences (Natural Language Generation). In this way attributes of the users can influence the generation process. During the interpretation of the final scripts, messages are sent to the MQTT-server. There are four types of messages that will be discussed by example. (1) de.add_message_for_state(’state/ s1’, {’text’ : ’hello’, ’image’ : ’ebrain.jpeg’}) (2) de.add_event_transition(source=’state/s1’, target=’state/s2’, guard=patient_is_ready) (3) de.add_timed_transition(source=’state/s1’, target=’state/s3’, timeout_in_s=5) (4) de.add_interaction_for_state(’state/s3 ’, video=’video.mp4’)
The object de represents the dialog engine that will get the messages. The first statement results in
the construction of sub-state s1 of state. While entering the state a MQTT message with the
corresponding name value pairs (e.g. [text : hello] and [image : ebrain.jpg]) is generated and sent. An
event transition evaluates a condition that can be specified as a function. In the case of the example, the
function is called patient_is_ready. After a certain interval of time, a timed transition is executed
without any side effects. The fourth statement results in a MQTT message for the presentation of a
specific video. Additionally, to the video keyword, other options like image, text and say can be used.
Similar examples can be found in [
Some training apps need specific devices like a large touch screen or tablets. Their devices represent a group of training tasks that communicate via MQTT messages as well.
It is not intended to replace a human therapist totally with a robot. The first session of training tasks is performed with a human therapist. The robot comes into the game during the second session only. Nevertheless, a patient can get the instructions of a task again by the robot. However, this is optional and a patient can stop it at any time. 4.1.
With the robot Pepper, one can interact touch-based with its tablet. However, interaction is not so easy because the robot is moving and is not always standing very next to a patient. Therefore, a specific tablet was introduced especially for human-robot interaction. It can be located on the table next to a patient. Spoken sentences of the robot are additionally presented as text on this tablet. This text is not as volatile as spoken sentences. In this way, patients can follow the instructions of the robot better. Spoken input to the robot Pepper is possible as well. However, because patients are often not able to speak loud and clear, the input to the robot is reduced to a “confirmation” can also be done by putting the hand on a flickboard. 4.2.
Adaptation to specific patients’ needs
User modelling in HCI has a long tradition. Fischer [
Additionally, the results of the training tasks are stored in our user models. They were relatively simple to identify. There was only the question of granularity. However, to support patients during exercises further attributes might be helpful. For this purpose, the repertory grid method [19] was used to identify knowledge from therapists about patients and helpers. For patients, the attribute “need for attention” and “cognitive performance” were identified. The attributes “authority” and “openness to mistakes made” were collected for helpers.
Feedback of the robot can be positive (e.g. “Great results with your 40 points, you’re approaching
your goal of playing chess fast!”) neutral (e.g. “You reached 40 points, remember your goal of playing
chess to be able to train again.” ) or challenging ( e.g. “With reached 40 points, I am sure you are still
holding back your performance.”). Some patients might always need positive feedback. However,
others have to be pushed a little bit with challenging sentences. The decision has to be made based on
attributes and rules. Bundea et al. [
The adaptation to personal feedback is additionally supported by a natural language generator.
RosaeNLG [17] was extended by some members of our project team in such a way that it fits our system
[
In the literature we did not find a lot about patient models and human-robot interaction. Winkle et al. [21] discuss general design implications for social assistive robots and how the engagement of users can be improved. They used stand-alone applications on Pepper for people that they hired at the university campus. They were not disabled.
Quinn et al. [15] describe an ontological model for patients with diabetes. It consists of attributes like age group, physical activity, language and health condition. They used publications provided by medical associations and academic literature.
Schrum et al. [16] use a humanoid robot Pepper for patients with dementia. The patients have to
copy the body movement of Pepper, while their heart rate was measured by a sensor. Unfortunately,
the human-robot interaction is not described in detail. Unfortunately, the same is true for Amato et al.
[
The Nao robot is used by Feng et al. [
About rehabilitation care of elderly people in Japan using Pepper is reported by Tanioka [20]. Different medical signals are measured. As a result, it is reported: "At the end of exercise, sympathetic nerve activity ratio has been increased" and “Elderly persons gain well-being from dialogues and conversations with Pepper robots”. There seems to be an objective and subjective success.
Similar to the E-BRAiN project Polak and Levy-Tzedek [
Manca et al [
Communication acts for dialogs are discussed by Fernandez-Rodicio et al. [
The only reference we found where software is running outside a robot is Pulido et al. [13]. They provide an architecture for a mirror game with the Nao robot. The game is called mirror because a patient has to copy the postures of Nao. Because the robot has legs it is a whole-body activity. The correct performance is checked by a Kinect sensor. A decision support system selects the next action. Further devices are not involved. However, this should be possible. The authors focus on an expert friendly system. They want to give health professionals the opportunity to adapt each treatment to every patient. Communication between robot and patient seems to be not in the focus.
Summary and Outlooks
Software development is challenging. It is especially challenging when users have special needs. Within our E-BRAiN project we have been developing software for supporting the rehabilitation of patients after stroke. We focus on Arm-Ability Training (patients can do already a lot), Arm-BasisTraining (patients need a helper), Mirror Therapy (patients can only move one of their arms well) and Neglect Therapy (patients have problems with special orientation). We discussed some examples of those therapies.
To implement a flexible system, we suggested a software architecture that uses the MQTT technology where components communicate via messages. This architecture allows the integration of several external systems that allow the adaptation of dialogs to the specific needs of a patient. This is supported by a natural language generator, a detailed user model and rules for different emotions.
The following research questions might be tackled in the future: • Did we analyze the correct number of attributes for patients? • Which further rules for adapting the dialog to patients can be found? • Has the level of natural language communication be adapted during a therapy? First experiments with patients were very positive. However, because of corona restrictions in clinics the number of therapies was very limited yet. Having further hints from patients will help to polish the communication. We also had to learn that patients are not the kind of users we were working beforehand. Ideas of user-centered design cannot be applied as easily. Patients are protected by law. An ethics committee asks for applications of experiments and they want to see medical reasons. Therefore, a certain quality of the software has to be provided and prototyping is difficult. User-centered activities in clinical environments have to be planned detailed and early, an aspect we were not aware of.
This joint research project “E-BRAiN - Evidence-based Robot Assistance in Neurorehabilitation” is supported by the European Social Fund (ESF), reference: ESF/14-BM-A55-0001/19-A02, and the Ministry of Education, Science and Culture of Mecklenburg-Vorpommern, Germany. The sponsors had no role in the decision to publish or any content of the publication.
Research Arm Test and Box and Block Test: a multicentre study. Clin Rehabil.,1 9(4):404-11. doi: 10.1191/0269215505cr832oa. [13] Polak, R.F., and Shelly Levy-Tzedek, S. (2020): Social Robot for Rehabilitation: Expert Clinicians and Post-Stroke Patients’ Evaluation Following a Long-Term Intervention. In Proceedings of the 2020 ACM/IEEE International Conferenc on Human-Robot Interaction (Cambridge, United Kingdom) (HRI ’20). Association for Computing Machinery, New York, NY, USA, 151–160. https://doi.org/10.1145/3319502.3374797 [14] Pulido, J.C., Suarez-Mejias, C., Gonzalez, J.C., Ruiz, A.D., Ferri, P.F., Sahuquillo, P. E. M., De Vargas, C.E.R., Infante-Cossio, P., Calderon, C.L.P., and Fernandez, F. (2019): A Socially Assistive Robotic Platform for Upper-Limb Rehabilitation: A Longitudinal Study With Pediatric Patients. IEEE Robotics Automation Magazine 26, 2 (2019), 24–39. https://doi.org/10.1109/MRA.2019.2905231 [15] Pulver, T. (2019). Hands-On Internet of Things with MQTT: Build connected IoT devices with
Arduino and MQ Telemetry Transport (MQTT), Packt Publishing Ltd., Birmingham [16] Quinn S, Bond R, Nugent C (2017) Ontological modelling and rule-based reasoning for the provision of personalized patient education. Expert Systems 34:e12134. https://doi.org/10.1111/exsy.12134 [17] Schrum, M., Park, C.H., and Howard, A. (2019): Humanoid therapy robot for encouraging exercise in dementia patients. In Proceedings of the 14th ACM/IEEE International Conference on HumanRobot Interaction (HRI '19). IEEE Press, 564–565. [18] Stoeckl´e L (2021) Rosaenlg - a natural language generation library for node.js. URL https://rosaenlg.org, version 2.1.7 [19] SoftBank Robotics: https://www.softbankrobotics.com/corp/robots/, last visited November 11th, 2019 [20] Tan, F. B. and M. Hunter, G. (2002):. “The Repertory Grid Technique: A Method for the Study of Cognition in Information Systems.” MIS Quarterly, vol. 26, no. 1, Management Information Systems Research Center, University of Minnesota, 2002, pp. 39–57, https://doi.org/10.2307/4132340. [21] Tanioka, T. (2019): Nursing and Rehabilitative Care of the Elderly Using Humanoid Robots. 66, 1.2. (2019), 19–23. https://doi.org/10.2152/jmi.66.19 [22] Winkle K, Lemaignan S, Caleb-Solly P., Leonards, U.,Turton, A., Bremner, P. (2019): Effective Persuasion Strategies for Socially Assistive Robots. In: 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI). IEEE, pp 277–285