=Paper=
{{Paper
|id=Vol-3404/keynote2
|storemode=property
|title=Engineering a Software System to Support Post-Stroke Patients in their Training Activities with a Humanoid Robot
|pdfUrl=https://ceur-ws.org/Vol-3404/paper9.pdf
|volume=Vol-3404
|authors=Peter Forbrig,Alexandru Bundea
|dblpUrl=https://dblp.org/rec/conf/eics/ForbrigB22
}}
==Engineering a Software System to Support Post-Stroke Patients in their Training Activities with a Humanoid Robot==
https://ceur-ws.org/Vol-3404/paper9.pdf
Engineering a Software System to Support Post-Stroke Patients
in their Training Activities with a Humanoid Robot
Peter Forbrig, Alexandru Bundea
University of Rostock, Chair of Software Engineering, Albert-Einstein-Str. 22, 18059 Rostock, Germany
Abstract
The paper discusses software architecture for the system E-BRAiN (Evidence-Based Robot-
Assistance in Neurorehabilitation) that is intended to support patients after stroke in their
rehabilitation. The humanoid robot Pepper gives instructions, provides pictures and videos. Its
main task is to motivate patients in their training exercises. A software architecture is suggested
that uses a kind of thin robot. The humanoid robot Pepper is used as input and output device
only. Most computations are performed externally. Communication between different
components is realized by message exchange. The communication between robot and patient
is supported by different modalities like spoken, text or buttons. It is supported by a natural
language generation system that considers the characteristics of a patient. It can generate e.g.
simple and complex language. Additionally, the system adapts speech speed and level of
support to those patient characteristics.
Keywords 1
Humanoid Robot, Software Architecture, MQTT Application, Rehabilitation Apps
1. Introduction
We want to discuss the specific challenges of engineering interactive systems for disabled people in
the context of medical training exercises. Within our project E-BRAiN (Evidence-Based Robot-
Assistance in Neurorehabilitation) [3] we have been developing a system for post-stroke patients with
mild to severe arm paresis and visual impairment. During the last three years, several training tasks
supported by a humanoid robot Pepper [19] were implemented. We will first give an overview of the
kind of implementations that we developed. Afterwards, we will discuss the challenges for developing
such systems and discuss technologies that allow adaptations to different contexts.
2. Rehabilitation training for post-stroke patients
The training tasks that we implemented can be grouped into four categories. They are called Arm-
Ability Training (AAT), Arm-Basis-Training (ABT), Mirror Therapy (MT) and Neglect Therapy (NT).
AAT is intended for patients with mild problems with one arm. They have to execute exercises of
different forms that are visualized in the left part of Figure 1. One task is the crossing out of as many
as possible Os. An app was developed that checks the correct performance. Additionally, the humanoid
robot gives feedback about current and previous performance. The right part of Figure 1 shows a student
testing the app.
EICS ’22: Engineering Interactive Computing Systems Conference, June 21–24, 2022, Sophia Antipolis, France.
EMAIL: peter.forbrig@uni-rostock.de (A. 1); alexandru-nicolae.bundea@uni-rostock.de (A. 2)
ORCID: 0000-0003-3427-0909 (A. 1); 0000-0001-8315-3405 (A. 2);
©️ 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
Figure 1: Exercises for the Arm-Ability Training (AAT) according to [10] and digitalized version of
a training task (right).
The Arm-Basis Training is designated for post-stroke patients with more severe handicaps. They
need at the beginning a helping hand of a supporter while performing training tasks. Figure 2 gives an
impression of the exercises with arms and fingers. Additionally, one can see one experiment where
sensors allow feedback from the robot about the correct performance.
Figure 2: Examples of the ABT training movements with a therapist (from [11]).
For patients with very severe one-sided handicaps, the Mirror Therapy can be applied. During such
therapy, a patient sits at 90 degrees next to a mirror placed on a table with his healthy arm in front of
the mirror and the handicapped arm behind the mirror. They have to perform training tasks with their
non-affected arm and have to look continuously into the mirror image of their healthy arm while they
have to imagine that their disabled arm is moving. According to the medical experts changes in the
brain can be induced in this way. Figure 3 gives an impression of how a patient has to sit in relation to
the mirror.
Figure 3: Situation for exercises with a mirror.
Neglect is a problem to orient attention spatially. It can affect all sensory and motor modalities,
occurs after hemispheric brain lesions, more frequently after right brain damage, e.g. after stroke [9].
In the visual domain affected patients have difficulties orienting their visual attention to the side
opposite to the brain lesion. As a result, patients might visually neglect one side of their body and/or
environment, e.g. leave the entire left-hand side of their meals untouched (in case of right brain damage
causing neglect).
The Neglect Therapy provides exercises to reduce this problem by promoting a shift of visual
attention to the neglected side, based on various training techniques including visual exploration
training. Figure 4 and Figure 5 provide two examples for visual exploration training tasks.
Figure 4: Training task for Neglect where patients have to point at specific objects.
Figure 5: Exploring a picture of objects of type cow.
Figure 4 gives an impression of the situation where a patient has to point at specific objects. A
task could be to point at the object that is different from the other objects or to point at all objects that
are different from the majority of objects. Another training task could be to touch on one object and say
“now” when this object leaves the screen.
Figure 5 demonstrates different stages of one training task where a patient has to point at all objects
of type cow. Found objects become red. When a patient stops pointing but did not identify all objects,
Pepper gives hints like:” Have a look at the top left corner. There seems to be another cow”.
In all cases, the robot provides instructions, jumps in when help is necessary, comments on the
results and provides motivation. The spoken language of the robot is additionally displayed as text on
a tablet. The robot Pepper is also able to understand spoken language. However, because of possible
pronunciation problems of post-stroke patients, we decided to have a specific device to express yes by
pushing a button. The question from Pepper: “Are you ready for the next training task?” is an example
where a patient has to press this button.
3. The E-BRAiN architecture
In our project, we started to implement the first spoken dialogs on the humanoid robot itself. After
recognizing the need of integrating further devices like tablets we started with direct communication.
However, we run into problems because sometimes the WiFi caused deadlocks. We had a look at
different possible technologies and ended up with MQTT (Message Queuing Telemetry Transport) [14].
It is easy to use, does not need many resources and checks whether a message really reached its
destination.
Having this technology implemented we started to implement the dialogs of the robot externally.
In this way, we got a kind of thin robot that only interprets messages and executes them. However, no
knowledge is stored on the robot. This software architecture provides a lot of flexibility. New software
and new devices can be integrated relatively easily. Figure 6 tries to visualize the concept of the
architecture.
Management Training Apps
Pepper
Flickboard
MQTT Server Patient
Therapist Therapy Administration
Tablet1
Tablet2
Natural Language
User Models Generation Interaction Server
(RosaeNLG)
Touch Screen
Dialog Scripts
Dialog Control
Legend
Data flow
Interaction
Programmer
Figure 6: Architecture of the E-BRAiN system.
All components of the E-BRAiN system communicate via MQTT messages that are managed by a
corresponding server, which is the center of our architecture. Clients can register for specific topics and
are informed when a message to this topic arrives.
On the left-hand side, one can see our administration software, which is a web-application that
manages all data of patients, their attributes, medical scores and training results. Additionally, their
plans for training exercises are stored in this application.
Programmers have to provide Python scripts that represent state-machine specifications. On an
interaction, the server runs an interpreter of such state machines. In this way, the dialog scripts can be
executed. However, before the scripts are executed they are enhanced by details of included sentences
(Natural Language Generation). In this way attributes of the users can influence the generation process.
During the interpretation of the final scripts, messages are sent to the MQTT-server. There are four
types of messages that will be discussed by example.
(1) de.add_message_for_state(’state/ s1’, {’text’ : ’hello’, ’image’ : ’ebrain.jpeg’})
(2) de.add_event_transition(source=’state/s1’, target=’state/s2’, guard=patient_is_ready)
(3) de.add_timed_transition(source=’state/s1’, target=’state/s3’, timeout_in_s=5)
(4) de.add_interaction_for_state(’state/s3 ’, video=’video.mp4’)
The object de represents the dialog engine that will get the messages. The first statement results in
the construction of sub-state s1 of state. While entering the state a MQTT message with the
corresponding name value pairs (e.g. [text : hello] and [image : ebrain.jpg]) is generated and sent. An
event transition evaluates a condition that can be specified as a function. In the case of the example, the
function is called patient_is_ready. After a certain interval of time, a timed transition is executed
without any side effects. The fourth statement results in a MQTT message for the presentation of a
specific video. Additionally, to the video keyword, other options like image, text and say can be used.
Similar examples can be found in [7]. Additionally, a visualization of dialog scripts and an interaction
pattern are discussed there as well.
Some training apps need specific devices like a large touch screen or tablets. Their devices
represent a group of training tasks that communicate via MQTT messages as well.
4. Discussion and related work
It is not intended to replace a human therapist totally with a robot. The first session of training tasks
is performed with a human therapist. The robot comes into the game during the second session only.
Nevertheless, a patient can get the instructions of a task again by the robot. However, this is optional
and a patient can stop it at any time.
4.1. General design decisions to support disabled patients
With the robot Pepper, one can interact touch-based with its tablet. However, interaction is not so
easy because the robot is moving and is not always standing very next to a patient. Therefore, a specific
tablet was introduced especially for human-robot interaction. It can be located on the table next to a
patient. Spoken sentences of the robot are additionally presented as text on this tablet. This text is not
as volatile as spoken sentences. In this way, patients can follow the instructions of the robot better.
Spoken input to the robot Pepper is possible as well. However, because patients are often not able to
speak loud and clear, the input to the robot is reduced to a “confirmation” can also be done by putting
the hand on a flickboard.
4.2. Adaptation to specific patients’ needs
User modelling in HCI has a long tradition. Fischer [6] provides a nice discussion about this history
and about adaptive and adaptable systems. There are general attributes about age and education that are
important for interactive systems. However, there are specific aspects that have to be considered for
users of the E-BRAiN system. Several attributes describe the medical state of the patient. They are
already fixed because of long medical research. They were provided by our experts from medicine.
These attributes are evaluated to measure the success of the rehabilitation. However, they also can
influence the training task performance. For the ABT such attributes can influence the range of an arm
movement and in this way the feedback from the robot.
Additionally, the results of the training tasks are stored in our user models. They were relatively
simple to identify. There was only the question of granularity. However, to support patients during
exercises further attributes might be helpful. For this purpose, the repertory grid method [19] was used
to identify knowledge from therapists about patients and helpers. For patients, the attribute “need for
attention” and “cognitive performance” were identified. The attributes “authority” and “openness to
mistakes made” were collected for helpers.
Feedback of the robot can be positive (e.g. “Great results with your 40 points, you’re approaching
your goal of playing chess fast!”) neutral (e.g. “You reached 40 points, remember your goal of playing
chess to be able to train again.” ) or challenging ( e.g. “With reached 40 points, I am sure you are still
holding back your performance.”). Some patients might always need positive feedback. However,
others have to be pushed a little bit with challenging sentences. The decision has to be made based on
attributes and rules. Bundea et al. [1] discuss these aspects in more details.
The adaptation to personal feedback is additionally supported by a natural language generator.
RosaeNLG [17] was extended by some members of our project team in such a way that it fits our system
[3]. It allows the specification of different language levels from simple to very complex. In this way,
the communication can be adapted to the cognitive characteristics of patients.
In the literature we did not find a lot about patient models and human-robot interaction. Winkle
et al. [21] discuss general design implications for social assistive robots and how the engagement of
users can be improved. They used stand-alone applications on Pepper for people that they hired at the
university campus. They were not disabled.
Quinn et al. [15] describe an ontological model for patients with diabetes. It consists of attributes
like age group, physical activity, language and health condition. They used publications provided by
medical associations and academic literature.
Schrum et al. [16] use a humanoid robot Pepper for patients with dementia. The patients have to
copy the body movement of Pepper, while their heart rate was measured by a sensor. Unfortunately,
the human-robot interaction is not described in detail. Unfortunately, the same is true for Amato et al.
[1]. They use Pepper for children that play games on the tablet of the robot. There seems to be no user
model.
The Nao robot is used by Feng et al. [4] to interact with children having autism. The authors
postulate: "1) If the child makes a wrong action, the robot will point out the mistake and ask the child
to do it again. If the child fails three times, this action will be abandoned. 2) If the child is distracted,
the robot reminds him to focus attention. 3) When the child completes the task correctly, the robot
praises him; when the child completes the task incorrectly, the robot encourages him."
About rehabilitation care of elderly people in Japan using Pepper is reported by Tanioka [20].
Different medical signals are measured. As a result, it is reported: "At the end of exercise, sympathetic
nerve activity ratio has been increased" and “Elderly persons gain well-being from dialogues and
conversations with Pepper robots”. There seems to be an objective and subjective success.
Similar to the E-BRAiN project Polak and Levy-Tzedek [12] focus with their system on post-
stroke long-term rehabilitation. A patient and a humanoid robot Pepper are located on different sides of
a shelve. The patient has to order jars on the shelve. The order is presented on a picture of Peppers’
tablet. Additionally, Pepper provides game instructions and feedback. The feedback is described as
follows in the paper: "After each trial, the robot either gives the patient feedback on the timing (e.g.,
“try to do it faster next time”) or on the success on the task (e.g., “you succeeded!”, “you were not right
this time, but I’m sure you will make it next time!”)."
Manca et al [9] use Pepper for training tasks for older persons with mild dementia. After serious
games Pepper provides feedback like: "Congratulations! Correct answer" or "Very good! Right
answer”. The selection is done randomly.
Communication acts for dialogs are discussed by Fernandez-Rodicio et al. [5]. They use a robot
called Mini and focus on spoken interaction. The target users are elderly people with light dementia.
Even though Mini does not look very nice, the results of their experiments were quite positive.
The only reference we found where software is running outside a robot is Pulido et al. [13]. They
provide an architecture for a mirror game with the Nao robot. The game is called mirror because a
patient has to copy the postures of Nao. Because the robot has legs it is a whole-body activity. The
correct performance is checked by a Kinect sensor. A decision support system selects the next action.
Further devices are not involved. However, this should be possible. The authors focus on an expert
friendly system. They want to give health professionals the opportunity to adapt each treatment to every
patient. Communication between robot and patient seems to be not in the focus.
Summary and Outlooks
Software development is challenging. It is especially challenging when users have special needs.
Within our E-BRAiN project we have been developing software for supporting the rehabilitation of
patients after stroke. We focus on Arm-Ability Training (patients can do already a lot), Arm-Basis-
Training (patients need a helper), Mirror Therapy (patients can only move one of their arms well) and
Neglect Therapy (patients have problems with special orientation). We discussed some examples of
those therapies.
To implement a flexible system, we suggested a software architecture that uses the MQTT
technology where components communicate via messages. This architecture allows the integration of
several external systems that allow the adaptation of dialogs to the specific needs of a patient. This is
supported by a natural language generator, a detailed user model and rules for different emotions.
The following research questions might be tackled in the future:
• Did we analyze the correct number of attributes for patients?
• Which further rules for adapting the dialog to patients can be found?
• Has the level of natural language communication be adapted during a therapy?
First experiments with patients were very positive. However, because of corona restrictions in clinics
the number of therapies was very limited yet. Having further hints from patients will help to polish the
communication. We also had to learn that patients are not the kind of users we were working
beforehand. Ideas of user-centered design cannot be applied as easily. Patients are protected by law. An
ethics committee asks for applications of experiments and they want to see medical reasons. Therefore,
a certain quality of the software has to be provided and prototyping is difficult. User-centered activities
in clinical environments have to be planned detailed and early, an aspect we were not aware of.
5. Acknowledgements
This joint research project “E-BRAiN - Evidence-based Robot Assistance in Neurorehabilitation” is
supported by the European Social Fund (ESF), reference: ESF/14-BM-A55-0001/19-A02, and the
Ministry of Education, Science and Culture of Mecklenburg-Vorpommern, Germany. The sponsors had
no role in the decision to publish or any content of the publication.
6. REFERENCES
[1] Federica Amato, Marianna Di Gregorio, Clara Monaco, Monica Sebillo, Genoveffa Tortora, and
Giuliana Vitiello. 2020. The Therapeutic Use of Humanoid Robots for Behavioral Disorders.
Proceedings of the International Conference on Advanced Visual Interfaces. Association for
Computing Machinery, New York, NY, USA, Article 98, 1–3.
DOI:https://doi.org/10.1145/3399715.3399960
[2] Bundea, A. and Forbrig, P. (2022): Patient and therapist model attributes for social robot stroke
therapies based on implicit knowledge from expert interviews, accepted for 15th International
KES-HCIS Conference, June 20-22, Rhodes, Greece.
[3] E-BRAiN: https://wwwswt.informatik.uni-rostock.de/webebrain/, last visited March 7, 2022
[4] Felske, T., Bader, S., and Kirste, T. (2022): Automatic Generation of Personalised and Context-
Dependent Textual, accepted for German Journal of Artificial Intelligence. p. 1-.5
[5] Feng, Y., Jia, Q. and Wie, W. (2018): A Control Architecture of Robot-Assisted Intervention for
Children with Autism Spectrum Disorders. 2018 (2018), 1–12. Issue Article ID 3246708.
https://doi.org/10.1155/2018/3246708
[6] Enrique Fernández-Rodicio, Álvaro Castro-González, Fernando Alonso-Martín, Marcos Maroto-
Gómez, and Miguel Á. Salichs. 2020. Modelling Multimodal Dialogues for Social Robots Using
Communicative Acts. Sensors 20, 12 (2020). https://doi.org/10.3390/s20123440
[7] Fischer, G. User Modeling in Human–Computer Interaction. User Modeling and User-Adapted
Interaction 11, 65–86 (2001). https://doi.org/10.1023/A:1011145532042
[8] Forbrig, P., Bundea, A., and Bader, S. .(2021):. Engineering the Interaction of a Humanoid Robot
Pepper with Post-Stroke Patients During Training Tasks. In Companion of the 2021 ACM SIGCHI
Symposium on Engineering Interactive Computing Systems (EICS '21). Association for
Computing Machinery, New York, NY, USA, 38–43.
DOI:https://doi.org/10.1145/3459926.3464756
[9] Kerkhoff G und Schmidt L. Neglect und assoziierte Störungen. 2018. Hogrefe Verlag, Göttingen
(ISBN 9783840928543).
[10] Manca, M., Paternò, F.; Santoro, C., Zedda, E., Braschi, C., Franco, R., Sale, A. (2020):, The
Impact of Serious Games with Humanoid Robots on Mild Cognitive Impairment Older Adults,
International Journal of Human-Computer Studies, 2020,
ttps://doi.org/10.1016/j.ijhcs.2020.102509
[11] Platz, T., Denzler, P., Kaden, B., Mauritz, K.-H. (1994). Motor leaning after recovery from
hemiparesis, Neuropsychologia, Volume 32, Issue 10, October 1994 pp. 1209-1223,
https://doi.org/10.1016/0028-3932(94)90103-1
[12] Platz, T., Pinkowski, C., van Wijck, F., Kim, I.H., di Bella, P., Johnson, G. (2005). Reliability and
validity of arm function assessment with standardized guidelines for the Fugl-Meyer Test, Action
Research Arm Test and Box and Block Test: a multicentre study. Clin Rehabil.,1 9(4):404-11. doi:
10.1191/0269215505cr832oa.
[13] Polak, R.F., and Shelly Levy-Tzedek, S. (2020): Social Robot for Rehabilitation: Expert Clinicians
and Post-Stroke Patients’ Evaluation Following a Long-Term Intervention. In Proceedings of the
2020 ACM/IEEE International Conferenc on Human-Robot Interaction (Cambridge, United
Kingdom) (HRI ’20). Association for Computing Machinery, New York, NY, USA, 151–160.
https://doi.org/10.1145/3319502.3374797
[14] Pulido, J.C., Suarez-Mejias, C., Gonzalez, J.C., Ruiz, A.D., Ferri, P.F., Sahuquillo, P. E. M., De
Vargas, C.E.R., Infante-Cossio, P., Calderon, C.L.P., and Fernandez, F. (2019): A Socially
Assistive Robotic Platform for Upper-Limb Rehabilitation: A Longitudinal Study With Pediatric
Patients. IEEE Robotics Automation Magazine 26, 2 (2019), 24–39.
https://doi.org/10.1109/MRA.2019.2905231
[15] Pulver, T. (2019). Hands-On Internet of Things with MQTT: Build connected IoT devices with
Arduino and MQ Telemetry Transport (MQTT), Packt Publishing Ltd., Birmingham
[16] Quinn S, Bond R, Nugent C (2017) Ontological modelling and rule-based reasoning for the
provision of personalized patient education. Expert Systems 34:e12134.
https://doi.org/10.1111/exsy.12134
[17] Schrum, M., Park, C.H., and Howard, A. (2019): Humanoid therapy robot for encouraging exercise
in dementia patients. In Proceedings of the 14th ACM/IEEE International Conference on Human-
Robot Interaction (HRI '19). IEEE Press, 564–565.
[18] Stoeckl´e L (2021) Rosaenlg - a natural language generation library for node.js. URL
https://rosaenlg.org, version 2.1.7
[19] SoftBank Robotics: https://www.softbankrobotics.com/corp/robots/, last visited November 11th,
2019
[20] Tan, F. B. and M. Hunter, G. (2002):. “The Repertory Grid Technique: A Method for the Study of
Cognition in Information Systems.” MIS Quarterly, vol. 26, no. 1, Management Information
Systems Research Center, University of Minnesota, 2002, pp. 39–57,
https://doi.org/10.2307/4132340.
[21] Tanioka, T. (2019): Nursing and Rehabilitative Care of the Elderly Using Humanoid Robots. 66,
1.2. (2019), 19–23. https://doi.org/10.2152/jmi.66.19
[22] Winkle K, Lemaignan S, Caleb-Solly P., Leonards, U.,Turton, A., Bremner, P. (2019): Effective
Persuasion Strategies for Socially Assistive Robots. In: 2019 14th ACM/IEEE International
Conference on Human-Robot Interaction (HRI). IEEE, pp 277–285