This paper describes the initial findings of a multimodal game which has been implemented on a humanoid robot platform and tested with seniors suffering from dementia. Physical and cognitive activities can improve the overall wellbeing of seniors, but it is often difficult to motivate seniors with dementia to train frequently and long enough using traditional training exercises. The idea of the presented concept is that robot games potentially can supplement occupational- or physiotherapists to promote more frequent and longer periods of training. The implemented multimodal game is based on tactile feedback and includes animated gestures and sounds. The game has been tested in a nursing home with four seniors suffering from moderate to severe dementia.
Due to the demographic challenges of Europe and other regions, it is essential to create
new ways of training seniors (+65) to achieve active and healthy ageing [
The social and recognizable characteristics of autonomous humanoid robots can
potentially be used to make training captivating and as a natural and enjoyable way to stay
active [
The characteristics of an autonomous humanoid robot, makes training engaging as humans tend to perceive robots differently from other machines. The tangible nature of the robot catalyzes interaction, and makes it possible to create many multimodal games which include physical movement. Additionally, a unique feature of using a robot for training, is that the robot can take the initiative pro-actively and start a training session. 2.1
Related research has demonstrated how robots can be a motivating tool for seniors with
a light degree of dementia. Aarhus Municipality has worked with the robot project
"Strengthening the Brain" [
According to Crawford [
Fig. 1. Illustration of the relation between skill and challenge. In state T1 and T4, there is a balance between skill and challenge and the player is in the state of flow. In state T2 and T3 there is no balance, and the player is either bored or frustrated.
It is a feeling characterized by great absorption of energized focus, full involvement,
and success in the process of the activity. As illustrated in Figure 1, Flow can only occur
when there is an appropriate balance between challenge and skill. In the state T1, the
difficulty of the challenge is in an appropriate relation to your (undeveloped) skills. T2
is the situation where you develop your skills to a level where the challenge becomes
too easy and therefore boring. In T3, the difficulty of the challenge is higher than your
skills, leading to discontent and frustration. T4 is also a flow state, but in a more
complex situation than in T1. Flow is not a stable state because your skills will keep
developing [
Although Flow is a well-established construct for examining game experience, an
inherent problem is that it is not immediately obvious how to translate between the flow
construct and an operative description of game-play [
Driven by the exponential growth of processing power, it is now possible to develop robot solutions for the masses at affordable price levels. This development opens for new opportunities in creating robot solutions which can be used in novel areas including physical training of seniors. To ensure robustness, we have chosen to rely on mature hardware technology using a commercially available robot platform. Many commercial platforms are available, but we have chosen to use the Pepper platform which is a humanoid robot from Softbank Robotics (see figure 2). The advantage of this approach is that it reduces development risks and make it is easy and fast to validate new game concepts. 4
The training exercises we implemented using Pepper are light to moderate workouts designed as interactive games in which the robot proactively encourages the seniors to participate. The design of the specific game followed these principles: Simple, recognizable narrative that is easily understood
Simple gameplay and objective, including an uncomplicated competition element Strong visual and aural stimulus using simple and easily recognizable gestures, sounds and tactile interaction 4.1
We have currently implemented several multimodal games using Pepper. However, only one game has been evaluated in a real nursing home setting. This game is designed for 2-5 people, placed in a circle around the robot. The robot acts as a token being passed back and forth between the players. When the robot has approached a player, the goal of the player is to touch the back of the hand of the robot (see figure 3).
If successful, the player will get a game point and the robot will make a short energetic expression using arm gesturing, light and sound (see figure 4).
Based on the player’s number of points, the robot’s arms will slowly change position from being static in front of the robot to gradually more dynamic animated movements. The idea is to make it more and more difficult for players to reach and touch the hand as their number of points increase (see figure 5).
While not touched, the robot will wait passively for 5 seconds. After an encounter or time-out, the robot will move backwards to the initial position and rotate to face the next player. When the players are sitting down, the robot trains the upper torso as well as eye/hand coordination. When the players are standing up, the game also trains balancing and walking sideways. The game flow can be defined in pseudo code: 1 DEFINE n AS NUMBER OF PLAYERS (2-5) 2 DEFINE x AS THE CURRENT PLAYER 3 DEFINE point(x) AS THE NUMBER OF GAME POINTS 4 SET x = 0 5 WHILE x < n 6 Move robot forwards 7 WHILE time < 5 seconds
IF touch == true THEN make energetic expression point(x) = point(x) + 1
GOTO LINE 9 8 END WHILE 9 Move robot back to initial position 10 SET x = x + 1 11 Rotate robot to a new angle 12 Animate arms based on point(x) 13 END WHILE
The flow of the game is illustrated in figure 6 and a video of the game can be found at https://youtu.be/Bb_23Er6JaE After successful interaction (a player touching the robot’s hand), the robot is programmed to make an energetic expression lasting 2-3 seconds using pre-selected animations and sounds. The expressions are randomly picked from a library of pre-selected built-in positive sounds and animations. After each successful interaction, the robot will slowly reposition its arms. Each movement is a function based on the number of game points of each individual player (see table 1). The idea is to gradually make the robot gesture more dynamically for each successful interaction, making it slightly more difficult to touch the robot’s hands as game points increases.
We have evaluated the game on a target group of players consisting in four participants in their 80’es. Three women (A, H and T) and one man (E). All participants of the group suffer from dementia in a moderate to a high degree. The session was planned to last for one hour. Two participants stayed for the entire session, and two participants had to leave after 45 minutes due to fatigue. The experiment was documented using photo and video, but no systematic quantitative measurement was performed. After the training session, the experiment was evaluated with the main occupational therapist and the head of the nursing home.
We observed a positive response by all participants who seemed engaged in the game while playing. The participants tended to move their arms together with the robot when gesturing (see figure 4), and to laugh and smile when the robot was gesturing and playing sounds. The participants would keep commenting the robot by calling it “sweet”, “cute” etc. treating the robot as a living being.
As also observed in earlier experiments [
In a similar experiment [
Although this research is very preliminary, it is our impression that using a multimodal humanoid robot as Pepper, works well with seniors who are suffering from moderate to a high degree of dementia. This becomes clear when comparing the results with a similar game experiment based on a simpler robot, in which participant lost interest faster and feedback were mixed. We did not observe that adapting the game challenge to the skills of the players affected the game experience significantly for this target group, i.e. the physical and social elements of playing with the robot (moving, touching, observing and commenting) overshadowed the competitive element.
Although more research is needed, it is encouraging that humanoid multimodal robot games are not rejected by the target group suffering from dementia – a group who are often difficult to train using traditional tools. However, the games must be very basic.