A virtual reality game for hand rehabilitation after stroke Amal Bouatrous1,4,† , Nadia Zenati2,† , Abdelkrim Meziane3,† and Chafiaa Hamitouche4,† 1 Computer Science Department USTHB University, BP 32, Bab Ezzouar, 16111, Algiers, Algeria 2 Centre for Development of Advanced Technologies (CDTA), Algiers, Algeria 3 Research Center on Scientific and Technical Information (CERIST) 05, Rue des 3 frères aissiou - Ben Aknoun -Algiers,Algeria 4 IMT-Atlantique Bretagne- Pays de la Loire, Brest, France. LaTIM, INSERM, UMR 1101, Brest, France Abstract This paper presents a serious game represented by a virtual environment, that is intended for the functional rehabilitation of the hand of stroke patients. The game was designed in close collaboration with specialized clinicians, and developed using a Leap Motion controller as a hand tracking device and the Unity 3D game development platform. The game simulates fruit harvesting in order to train the movement of the bidigital hand grip. The game has been enjoyed by patients undergoing functional hand rehabilitation. Keywords Serious Games, Virtual Reality, Hand Rehabilitation, Leap Motion Controller, Stroke 1. Introduction Functional rehabilitation in rehabilitation centers requires major improvements in terms of rehabilitation tools, and this is due to the fact that conventional clinical protocols for functional rehabilitation suffer from boredom, which affects patients’ motivation and consequently the recovery of their motor functions. Virtual rehabilitation is one of the new therapies that have been proposed to enhance functional rehabilitation protocols. It is a virtual reality therapy that involves virtual environments, which is a software system that simulates real-world tasks in rehabilitation therapy for stroke patients and helps them to effectively assess and measure patient performance[1]. Serious games are increasingly used for motor rehabilitation[2] after stroke. They involve incorporating gamification into rehabilitation exercises. Gamification is the process of applying codes and mechanisms from the world of video games to areas where they were not intended. Thus, gamification of rehabilitation exercises (exergames) consists in making rehabilitation exercises enjoyable. In order to play, participants must have a physical activity or exercise, which requires them to move the body parts involved in the exercise in RIF 2023: The 12th Seminary of Computer Science Research at Feminine Constantine, Algeria, March 9, 2023 † These authors contributed equally. Envelope-Open amalbouatrous@gmail.com (A. Bouatrous); nzenati@cdta.dz (N. Zenati); ameziane@cerist.dz (A. Meziane); chafiaa.hamitouche@imt-atlantique.fr (C. Hamitouche) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings order to accomplish the required tasks. In this paper, we present a serious game that simulates a traditional clinical exercise for hand motor rehabilitation. The game was designed and developed in close collaboration with specialized clinicians, and it was clinically tested on post-stroke patients with hand motor disabilities. 2. Related Works Upper limb motor deficit is a common symptom of stroke[3], affecting the quality of life of these patients, and improving upper limb function remains a major component of rehabilitation for stroke patients. Therefore, a variety of interventions have been developed for this purpose[4]. In recent years, commercial virtual reality (VR) consoles, also known as exergames, have emerged as a new form of intervention for stroke rehabilitation. These systems allow individuals to participate in tasks ranging from exercises to performing meaningful activities in a virtual environment[5]. S.-H. Yang et al.[6] proposed a virtual reality game for bilateral hand training for home rehabilitation, the movement of the less affected hand was measured by a hand sensor glove using a flexion sensor. The affected hand is then trained by the exoskeleton using the measured hand movements. M. G. Lansberg et al.[7] showed that the Smart Glovea,which is a commercially available noninvasive system consisting of a wearable glove and a tablet, includes 45 games that mimic real-life activities or are based on invented scenarios. Each game is designed to focus on specific sets of hand and wrist movements, may help increase the dose of outpatient rehabilitation. A. de los Reyes-Guzmán et al.[8] presented seven therapeutic virtual reality applications for the rehabilitation of upper limb manipulative abilities in patients with neurological disorders, manipulated using the Leap Motion Controller. Q. U. Ain et al.[9] showed that repetitive use of the hemiparetic upper limb through Xbox Kinect-based upper limb rehabilitation training, in addition to conventional therapy, has promising potential to improve upper limb motor function in stroke patients. K. N. K. Fong et al.[10] presented a TS-VR program based on seven general hand function tasks used in activities of daily living that require upper extremity movement, using a Leap Motion Controller VR device and the Unity3D program for training distal hand function. M. Weiss Cohen and D. Regazzoni.[11] proposed an approach to tracking hand rehabilitation exercises for stroke patients using a leap motion controller as a hand tracking device. Pilatásig et al.[12] presented an interactive system for hand and wrist rehabilitation using the leap motion device and Unity3D software. Our system focuses on the development of a low-cost interactive tool, consisting of a simple game based on a usual activity that takes place in a familiar environment for post-stroke patients, in order to attract them and maintain their motivation. 3. Materials and methods 3.1. The proposed serious game By working closely with clinicians specialized in functional rehabilitation, we have selected a fine hand movement which is the Bidigitale grasp, that involves grasping relatively small objects between the thumb and the index finger of the hand. We chose a fruit harvesting scenario to create a virtual environment that simulates this exercise, which appears to be a familiar activity for all categories of patients. Thus, in the game, the patient is encouraged to harvest fruit from their trees and place them in a basket, grasping them between the thumb and index finger of their hand. 3.2. Hardware and Software The main objective of this work is to propose a non-invasive and low cost tool for motor rehabilitation of the hand, in particular the training of a fine hand movement which is the bidigital grasp. Due to its low cost, small size compared to other devices, and especially its ease of use, lack of markers, and exciting aspects of its technology[13], the Leap Motion Controller LMC optoelectronic system shown in Figure 1 was selected. It captures the movement of both hands and controls a virtual environment. Thus, the LMC captures the fine movements of the hand, so it is potentially more suitable for training fine movements of the upper limb and hand functions[13]. Because of its ease of use and extensive documentation[14], the Unity 3D platform, which enables the development of content for virtual reality and the creation of two- and three-dimensional games, was used to build the virtual environment in which the game takes place. Figure 1: The Leap Motion device. 3.3. Gameplay The game scenario takes place in a virtual grove made up of fruit trees bearing small fruits such as cherries (see Figure 2). Due to the limited field of view of the LMC, we considered a set of nine fruits per tree. In order to play, the player must perform the following tasks: 1. A pointing task: consists of moving the hand towards the fruit and pointing it with the index finger, in order to be able to pick it up. 2. A grasping task: consists of bringing the two fingers of the thumb and index finger as close together as possible in order to pick up the fruit that the index finger is pointing to, by grasping it between the two fingers concerned. 3. A deposit task: consists of moving the thumb and index finger away from each other, in order to deposit the fruit that is between the two fingers in a basket by releasing it from the two fingers involved. The interaction model is avatar-based, so that the user’s hand is represented in the virtual environment by a virtual hand (see Figure 3). Figure 2: The virtual scene representing the game. 3.4. Game Settings To successfully complete the game, the patient must be able to grasp the fruit between the thumb and index finger of their hand after reaching it, and then drop it after grasping it. Thus, as previously stated, in order to grasp the fruit, the patient must bring their two fingers together (see Figure 4), and to release it, they must spread them apart (see Figure 5). This movement was measured in our game by the Euclidean distance between the tips of the two fingers. The distance must be greater than a certain predefined threshold to grasp the fruit, and it must be less to release it. The difficulty of the game is determined by this threshold. The game’s difficulty level can be tailored to the patient’s initial motor status. Before starting to play, the patient calibrates the game by performing the pinch gesture (see Figure 6). Behind the scenes, the system computes the minimum and maximum amplitudes of the patient’s hand movement. These values will then be used to create thresholds against which the patient’s Figure 3: Representation of the user’s hand in the game scene. Figure 4: The task of grasping the fruit. performance during the game will be compared. The entire calibration procedure lasts 30 seconds. Figure 5: The task of dropping the fruit. Figure 6: The calibration scene of the game. 3.5. Exercises History We considered some parameters to measure during the game in order to quantify the patient’s performance. The Table 1 depicts these parameters. Indeed, each patient has their own file in a database, at the end of the game, these parameters are recorded in their file, and the therapist can consult these data to determine the patient’s level of motor skills. Table 1 Game Settings Parametrey Meaning Time The time that elapses between the beginning and the end of the game. For each success, a reward is given in the form of a number of coins that appear Score_S in the scene(see Figure 7),the score is the sum of all the awards given. Score_RF Represents the number of fruits harvested A list of nine items, each linked to a fruit and representing the patient’s motion skills performance in completing the tasks required to harvest each associated fruit. Figure 7: The representation of rewards in the game. 4. Preliminary Tests and Discussion The usability and acceptability of the proposed game were tested on a group of post-stroke patients with hand motor impairment. The LMC hand motion sensor was connected to a PC computer (Intel(R) Core™i7-3520 M CPU @ 2.90 GHz 290 GHz, NVIDIA GeForce 840M graphics Figure 8: The Hardware configuration of the game. card, 12GB RAM). The complete hardware architecture of the system is shown in Figure 8. The patients who played the game found it enjoyable and beneficial to their hand rehabilitation. Patients in particular, reported that the proposed game’s features were very well integrated, and that it was an innovative and motivating tool for them, as it can help them better engage in their rehabilitation sessions. However, because the number of patients who participated in the clinical evaluation was so small, more patients must be recruited to properly assess the acceptability and the usability of the proposed game. Figure 9 shows a patient playing the game. Indeed, the selection of patients who can use the game was conducted by the therapists. Figure 9: A post-stroke patient playing the game. 5. Conclusion and Future Works In this article, we presented a serious game for the functional rehabilitation of the hand. It is a simulation of a clinical exercise for a fine hand movement, namely the Bidigitale grip. The game was designed and developed in close collaboration with clinicians specialized in functional rehabilitation. A first evaluation of the acceptability of the tool on patients has been performed. The game was very well received by the patients. They indicated that it was a new tool that could help them recover their functionality. Nonetheless, the clinical evaluation sample was too small to objectively assess the tool’s acceptability and usability by patients. The addition of new patients is required, which we intend to do in the near future. Acknowledgment The authors would like to thank everyone who contributed to the completion of this work. Beginning with the clinicians of the Douera University Hospital in Algiers, Pr Houria Kaced, Dr Amine Brahimi, and Dr Sara Ait Ziane, with whom they collaborated throughout the game’s setup process. They would also like to thank Dr. Olivier Rémy-Néris, Dr. Myriam Thiebaut- Clerin, and Ms. Sandrine Chedet of the CHRU of Brest in France. They would like to thank the patients who took part in the device’s clinical evaluation. Finally, they thank Dr. Chayma Zatout (CERIST) for her participation in the system’s evaluation on patients, as well as the great efforts she made to accompany them during this phase, assisting them. References [1] N. F. Rakib, N. H. Mahmood, N. Ramli, N. A. Zakaria, and M. A. A. Razak, “Preliminary Results of Hand Rehabilitation for Post Stroke Patient using Leap Motion-based Virtual Reality,” in 2020 IEEE Student Conference on Research and Development (SCOReD), Batu Pahat, Malaysia, Sep. 2020, pp. 259–262. doi: 10.1109/SCOReD50371.2020.9250985. [2] F. Pereira, S. Bermúdez i Badia, R. Ornelas, and M. S. Cameirão, “Impact of game mode in multi-user serious games for upper limb rehabilitation: a within-person randomized trial on engagement and social involvement,” J NeuroEngineering Rehabil, vol. 16, no. 1, p. 109, Dec. 2019, doi: 10.1186/s12984-019-0578-9. [3] J.-S. Kwon, M.-J. Park, I.-J. Yoon, and S.-H. Park, “Effects of virtual reality on upper extremity function and activities of daily living performance in acute stroke: A double-blind random- ized clinical trial,” NRE, vol. 31, no. 4, pp. 379–385, Nov. 2012, doi: 10.3233/NRE-2012-00807. [4] J.-H. Shin, S. Bog Park, and S. Ho Jang, “Effects of game-based virtual reality on health-related quality of life in chronic stroke patients: A randomized, controlled study,” Computers in Biology and Medicine, vol. 63, pp. 92–98, Aug. 2015, doi: 10.1016/j.compbiomed.2015.03.011. [5] A.-V. Nguyen et al., “Virtual reality exergaming as adjunctive therapy in a sub-acute stroke rehabilitation setting: facilitators and barriers,” Disability and Rehabilitation: Assistive Technology, vol. 14, no. 4, pp. 317–324, May 2019, doi: 10.1080/17483107.2018.1447608. [6] S.-H. Yang et al., “An Instrumented Glove-Controlled Portable Hand-Exoskeleton for Bilateral Hand Rehabilitation,” Biosensors, vol. 11, no. 12, p. 495, Dec. 2021, doi: 10.3390/bios11120495. [7] M. G. Lansberg et al., “Home‐based virtual reality therapy for hand recovery after stroke,” PM&R, p. pmrj.12598, May 2021, doi: 10.1002/pmrj.12598. [8] A. de los Reyes-Guzmán et al., “RehabHand: Oriented-tasks serious games for upper limb rehabilitation by using Leap Motion Controller and target population in spinal cord injury,” NRE, vol. 48, no. 3, pp. 365–373, Apr. 2021, doi: 10.3233/NRE-201598. [9] Q. U. Ain et al., “Additional Effects of Xbox Kinect Training on Upper Limb Function in Chronic Stroke Patients: A Randomized Control Trial,” Healthcare, vol. 9, no. 3, p. 242, Feb. 2021, doi: 10.3390/healthcare9030242. [10] K. N. K. Fong, Y. M. Tang, K. Sie, A. K. H. Yu, C. C. W. Lo, and Y. W. T. Ma, “Effects of task-specific virtual reality training using a Leap Motion Controller on hemiparetic upper extremity in patients with stroke,” In Review, preprint, Oct. 2020. doi: 10.21203/rs.3.rs- 55924/v2. [11] M. Weiss Cohen and D. Regazzoni, “Hand rehabilitation assessment system using leap motion controller,” AI & Soc, vol. 35, no. 3, pp. 581–594, Sep. 2020, doi: 10.1007/s00146-019- 00925-8. [12] M. Pilatásig et al., “Interactive System for Hands and Wrist Rehabilitation,” in Proceedings of the International Conference on Information Technology & Systems (ICITS 2018), vol. 721, Á. Rocha and T. Guarda, Eds. Cham: Springer International Publishing, 2018, pp. 593–601. doi: 10.1007/978-3-319-73450-7_56. [13] M. Iosa et al., “Leap motion controlled videogame-based therapy for rehabilitation of elderly patients with subacute stroke: a feasibility pilot study,” Topics in Stroke Rehabilitation, vol. 22, no. 4, pp. 306–316, Aug. 2015, doi: 10.1179/1074935714Z.0000000036. [14] M. Alimanova et al., “Gamification of Hand Rehabilitation Process Using Virtual Real- ity Tools: Using Leap Motion for Hand Rehabilitation,” in 2017 First IEEE International Conference on Robotic Computing (IRC), Taichung, Taiwan, Apr. 2017, pp. 336–339. doi: 10.1109/IRC.2017.76.