In 2019, stroke was the second most common cause of death and the third most common cause of disability-adjusted life years globally [1,2] and in Norway [2]. As healthcare is constantly evolving and new studies have been conducted, death rates have decreased [3]. This leads to an increasing number of patients who need extensive rehabilitation. The goal of rehabilitation is to give the affected as much independence as possible and be able to go back to work and enjoy the hobbies they love.
Keeping the motivation of the patient high is one of today's problems during traditional rehabilitation, as the patients can often perceive the exercises as monotone and repetitive [4]. Another challenge with stroke is the significant difference in impairments for each patient [5,6]. The project aims to tackle these problems by developing two fully immersive VR games with adjustable difficulty to supplement traditional rehabilitation. However, the work presented is limited to evaluations with physical therapists at this stage. This limitation is to first see the acceptability and usability of the games as a supplement. The therapists are a good starting point in assessing the usefulness. They are also the entry point of taking the games into
The International Health Data Workshop (HEDA-2022), June [19][20][21][22][23][24]2022, Bergen, Norway Envelope severin.eliassen@gmail.com (S. A. Eliassen); Harald.Soleim@hvl.no (H. Soleim); Atle.Birger.Geitung@hvl.no (A. B. Geitung); Lars.Peder.Vatshelle.Bovim@hvl.no (L. P. V. Bovim) the rehabilitation process and recruiting patients to a clinical trial. Then, after achieving a satisfying result, patients can be involved to check the motivational and engagement gains and the effectiveness of the proposed games.
In a related project by Førde hospital trust [7], Beat Saber1 is being used to supplement the rehabilitation of sitting and standing balance among stroke survivors. They observe with Beat Saber that patients have increased motivation and engagement in the treatment, in accordance with [9]. Technology-assisted rehabilitation is not a new concept but rather an emerging field.
Tobler-Ammann et al. [10] designed nine exergames for rehabilitation of visuospatial neglect after stroke. The games were Virtual Reality (VR) experiences developed for use on a computer monitor and with the Novint Falcon haptic controller. This is a 3D touch controller with force feedback [11]. They used the technology acceptance model, group interviews with the therapists, and individual interviews with the patients to evaluate the games. The study found that the therapist was more skeptical about using these games than the patients. Patients perceived the games as a motivating and nice change from regular rehabilitation. However, they could become repetitive, and some had trouble seeing their relevance to real life at first or at all. Both groups of users were not convinced to use exergames after the trial period, as the patients preferred to use their time on other activities after being discharged.
Faria et al. [12] compare traditional paper-pencil cognitive training methods with adaptive training in VR. The article uses Reh@City v2.0 [13], a VR game that simulates eight daily life activities. The game is experienced through a computer monitor, where interactions happen through a motion-tracking camera. These activities are spread across eight different locations in a city, including a bank, a store, and a post office. For the evaluation, they recruited 35 outpatients who had an ischemic stroke2 at least six months prior to the study (18 in the pen-paper group and 17 in VR). Their findings do not show a significant performance difference between the paper-pencil intervention and VR, but VR users experienced more intensive training, leading to more repetitions and challenging tasks. The authors concluded that it could lead to more cognitive improvements, but further research with a larger sample size and other rehabilitation tasks is necessary.
Gamito et al. [15] designed a Virtual Reality cognitive training application. The VR application included a scenario that consisted of several daily life activities for training cognitive functions such as working memory, visuospatial orientation, selective attention, recognition memory, and digit retention. They recruited 20 stroke patients, where 10 used the VR application and 10 were in the control group. Overall their results indicate benefits in memory and attention but no significant results in visual memory.
The presented articles are not an exhaustive list of related works. There exists a wide range of projects and studies on rehabilitation using VR, both fully immersive and not, such as [9,16,17,18,19,20,21].
As is evident from the presented related work, gamification of the rehabilitation process is not a new trade, and quite a bit of novel work is done, both in terms of cognitive and physical rehabilitation of stroke survivors. However, the use of fully immersive VR for cognitive stroke rehabilitation is rather understudied, as a lot of the related work is done through screen-based VR [10], [12], [17], and some require specialized tools and controllers [10], [12]. In addition, there is work that uses keyboard and mouse interaction instead of fully utilizing the potential of head-mounted displays (HMD) [20].
The work presented in this paper distinguishes itself by the games fully utilizing commercially available HMDs for presenting and interacting with the virtual environment, giving a fully immersive experience. Although an HMD could be argued as a specialized tool, it is a much more common item than others (such as the Falcon) and is expected to see significant growth in the coming years [22]. In addition, non of the presented work looks at the possibility of using a tablet application for viewing and setting the game settings of multiple game sessions.
The remainder of the paper is organized as follows. The following section briefly presents the methods and tools used. Then in section three, we take a look at the developed solution. Section four contains the gathered results, and the last section is the discussion and conclusion.
The research method uses a Design Science [23] inspired approach.The problem stems from stroke rehabilitation, with an increasing need for rehabilitation tools to increase the patients' motivation and give therapists more ways to individualize for each patient. Before starting the development, we held two idea workshops with domain experts. The development iterations then went two weeks at a time and included internal evaluations at the end of each. When the applications reached a satisfactory point, domain experts were recruited for evaluation (Three physiotherapists and a doctor). The evaluation used the System Usability Scale (SUS) [23] and semi-structured interviews. Chapter 4 presents the SUS statements. For simplicity, the SUS questionnaire was translated into Norwegian.
The evaluation process let the therapist try the applications as thoroughly as they wanted. One of the authors was present during the evaluation, taking a passive observing role while taking notes of the impressions and what seemed unintuitive. The setup used a Meta Quest HMD connected to a desktop computer for testing the application without the streaming possibility. When the therapists felt satisfied with the amount of testing both as a "patient" and therapist, the HMD was unplugged. Then they were shown the tablet application while the author took the role of a patient. After using the applications, they filled out the SUS questionnaire. After the SUS questionnaire, the evaluation transitioned into a semi-structured interview. The interview process contained questions about possible improvements, the tasks' relevancy, and their thoughts about controlling from a tablet. This evaluation process was over two rounds, where the second round included most of the suggested improvements from round one.
The games and tablet applications were made using the Unity game engine [24]. This game engine supports deployment to multiple platforms such as Android, Windows, and IOS. The VR application uses Unity's XR-interaction toolkit [25] and Auto Hand [26] (from the unity asset store) for VR interaction and the OpenXR standard [27]. Other toolkits and SDKs are available, such as Oculus Integration, but this is limited to running on the Oculus head-mounted displays (HMD)s [28]. Therefore OpenXR is used for simple support for multiple HMDs. The primary HMD used during the development was the Oculus Quest 2. Alongside the Quest 2, we tested the VR application compatibility with these: Pico Neo 3, HTC Vive 2 pro, and HTC Cosmos. Since the games have an intention to be used with and without a therapist present, it felt natural to develop them for wired and standalone HMDs. The games use low-polygon graphics as the standalone HMDs have limited computational resources. We used various packages from the Unity asset store for many of the 3D models, the color picker, and networking.
The following sub-sections present the two games with justification, implementation details, and the tablet application.
The painting game is a simple game where the user gets a set of boxes, paint, and some paintbrushes, as shown in Figure 1. The goal of this game is for the user to stack the boxes in a designated area and paint them in the correct order. This process is possible in any order, e.g., painting then stacking or stacking then painting. A "pre-painted" stack of boxes determines the color order of the boxes, which resides on another table.
The game intends to work with eye-hand coordination. This skill is essential for daily life functioning, as it is central in activities such as eating and using tools [29] and is susceptible to impairment after a stroke [30]. According to Crawford et al. [29], eye-hand coordination involves visuals, eye, head, and arm motor systems, memory, and attention. This skill involves processing visuals, translating them into a planned hand action, and executing said plan. Some studies show a correlation between cognitive functions and the sensory-motor system [31], while several parts of the latter are involved in eye-hand coordination [29]. Training this skill is done by the user stacking and painting boxes, where they need to plan the motor action of stacking and painting. The executive functions are also involved, as the user must strategize in what order to stack and paint. The game also has additional use-cases. One of these is the inclusion of neglect training. Neglect is a deficit in one's attention to the opposite side of the body than where the lesion resides [32]. Training of neglect is doable by the table with the prepainted stack is movable. Thus the therapist can move the table into the field where the patient's neglect is affected, forcing the patient to shift their focus there.
One of the ways to individualize for each user is that the paintable boxes can be made smaller and larger. The smaller ones require more fine motor skills and higher accuracy of eye-hand coordination. Other ways to individualize are fewer and more boxes, and the number of colors and the colors themselves can be changed.
The box painting game includes two types of feedback, visual and haptic. The controller vibrates when a user places a box at the correct spot or on top of another box. Whether the boxes are in the correct order does not matter, as the user can paint them before or after stacking. The visual feedback occurs when the game finishes. When the user successfully replicates the pre-painted stack, the user's stack shoots a small blast of confetti to signal victory, and the score panel appears. The confetti effect was with Unity's particle system.
For the user to not get stuck, the game includes an indicator that appears when the user has not placed a box within the first 15 seconds or the user holds a box for two seconds. These visuals are a light border at the place point and a "bouncing" arrow over it. The arrow has a billboard shader that always renders the arrow to face the camera. Both of these effects were made using Unity's shader graph.
Both games include nine different environments (maps), all in the low-poly graphics style. The low-poly style is used as it is important to have less sensory stimulus during the early stages of stroke rehabilitation. Students that worked on a related project at Western Norway University of Applied Sciences made these maps. They feature locations such as a desert, lake, Stonehenge, and island, to name a few. Another feature in both games is the possibility to adjust the player's height, position, and rotation. This feature gives the possibility for use in standing and sitting positions and for patients with less mobility in their arms to use the games. The movement also enables them to adjust the games for mobility exercise. With this easy access, the rehabilitation process could become more straightforward.
The second game is based on the Stroop test, which according to Scarpina and Tagini [33] "is used to assess the ability to inhibit cognitive interference". The traditional test is based on a set of words written in colored ink, and then the participants are instructed to say the color of the ink and not read the word. The plane game takes inspiration from this task, but it uses colored figures instead of words. The figures reside on a banner behind several planes that fly across the environment, while the user has the same set of colored figures as buttons, as shown in Figure 2. This task works on the inhibition part of the executive functions. Logan and Verbruggen [34] state that response inhibition is "... the suppression of actions that are no longer required or that are inappropriate, which supports flexible and goal-directed behavior in ever-changing environments. " A large part of this game is that the user should not only click on the correct color or figure but that it might switch as a new plane spawns. This behavior forces users to suppress the action of clicking what is no longer correct and constantly process
The plane game scoring system The panel resides at a fixed location, as shown in Figure 2.
As with the first game, the plane game contains a part of eye-hand coordination as the game uses physical buttons in the virtual environment for the clicks. The user then has to plan the action of clicking the correct button and carry out this movement plan. This game is on a higher "level" than the painting game as the main concept is a more cognitively demanding task. A small aspect of neglect is included, as planes can spawn either from the user's left or right side. This makes the user shift their focus towards the side the planes are coming from while shifting focus back to the center panel to know what to click.
After a correct button click, the corresponding colored figure on the plane's banner will disappear. A custom scoring system has been implemented in this game to measure performance and increase motivation.
The scoring system is inspired by those seen in games such as Beat Saber 1 . This artificial scoring system gives the user increased motivation by rewarding the use of fewer attempts to click the correct button and increasing the replay value, as stated by Boller and Kapp [35]. They also state the importance of having variance in the scoring system.
A multiplier is included in the scoring system to introduce variance, starting at 1x, with the base score of 100 pr correct. The multiplier increases when the player clicks on the correct buttons five consecutive times. This can be increased to 2, 4, and 8, as seen in Table 1. If the user presses the wrong button, the multiplier decreases. Say the player has 12 consecutive correct, they are at 4x, with three to reach 8x. If the player clicks the wrong button, they are set down to 2x and only need three more to reach 4x again (i.e., one could say that the correct amount in a row is reduced by five.).
This game has some similar adjustabilities as the first game. The number of different colors/figures can be set between 2-4, the plane speed is adjustable, and if enabled, will increase and decrease based on if the user clicks the correct button. They can set the time a game should last or the number of planes that will spawn. The therapist can set the game to last, for example, 30 planes, or they could set the game to last about 2 min and 30 sec (planes will spawn for the specified amount of time). Just like the box painting game, this game has a setting that allows for work on neglect. By default, the planes spawn on the right side of the environment and fly in the left direction. However, one can set the planes to spawn from the left in the game settings. This is not mixable, so planes can only spawn from one direction at a time. With this setting, the therapist can make the patient shift what side to focus on for each round and make the patient shift focus to one side as they notice the planes coming from that side. When the last plane despawns or all figures are correctly clicked, the game finishes, and the game saves the score and settings to the high score table.
The patients' progress gets tracked with a high score table, which was one of the features requested from the first evaluation round. The table contains information about the used game settings for that record, streamlining progress tracking. When users start the application with the games, they can create a nickname, select an existing one, or select "guest". All of which use no authentication. The nickname list and the score list are saved locally as JSON strings with Unity's built-in PlayerPrefs class. When a user finishes a game, the score list string is retrieved first, then the score for the new game is stringified, added to the retrieved score string, then lastly saved back into PlayerPrefs.
The user in the virtual environment has an in-game menu that includes map selection, game settings, high score, a help section, and language selection. The VR menu is an adapted 2D menu laid out on a physical 3D object (cube) so that it resembles a tablet. An advantage of this layout is that 2D menus use well-established principles, and most are familiar with how they function and the interaction they require. The VR menu uses a ray-cast technique for interaction. According to LaViola et al. [36], this is one of the most straightforward and most efficient techniques of selection when used at not too long distances on medium to large objects.
The VR menu has a large mode (Fig. 3A) and a small tablet-like mode (Fig. 3B). Opening the menu in large mode moves it to a fixed location. The large menu mode gives higher precision when selecting menu items but has drawbacks. The menu is in a fixed location determined by the application, which might occlude the virtual environment, especially in the plane game. Because of how the tablet moves and enlarges, loss of immersion might occur, as stated in the article by Wang et al. [37]. The therapist's menu is a separate menu that the VR user cannot see and is only available on the monitor of the connected computer, as seen in Figure 3C. One advantage of this design and implementation is that it allows for remote control for a therapist with the help of external software, such as TeamViewer. The user can then start a session at home and let the therapist connect to their computer, so they do not need to travel to the rehabilitation center for their session.
The menu layout for both games is equal when it comes to design but differs in the difficulty/settings available. By having menus equal in design, the users are less prone to be confused and not find the correct buttons. After learning the menu system for one game, it should be easy to use it for the other.
As with the games, this application was made in Unity to support multiple platforms easily. Figure 4 shows the simplistic tablet application, with a large stream in the top left corner, and below the "main" stream is space for three more, which are smaller. The therapist can hold the stream to select it to send game settings to that user and tap the small streams to make them
The goal of this app is to make the setup more mobile, as, in one of the idea workshops, there were mentions of problems with physical space at rehabilitation centers when it comes to using Virtual Reality tools. Alongside being more mobile, the application opens for home visits where the therapist can bring the tablet and connect to the patient's HMD. Since it allows up to four streams simultaneously, it facilitates group sessions where the patients can get social interactions and competition. Although competition can increase motivation, they must be on the same level so that the score discrepancy does not decrease motivation. The package FMETP STREAM V2 [38] enables networking, encoding, and decoding of the in-game camera. This pack takes the game view on the HMD, converts it to byte arrays, and sends them with UDP to the tablet app. These are then converted back into textures for display. The pack is used due to time constraints on the project and its ease of use, as it features automatic network discovery.
The games have two different builds, one with control from a tablet and one without. This situation is due to introduced limitations due to performance issues, as the game view encoding is a rather taxing process. All maps except the neutral one have to be disabled. Another feature not included is the ability to move the table in the box painting game. Due to the limited time available, this feature was deemed less necessary than other features and thus left out of this version. Even with the introduced limitation, the video stream does not have a satisfying quality level and is therefore used more like a "proof of concept. "
As mentioned earlier in Section 2, the evaluation used SUS and semi-structured interviews. The original SUS statements can be seen in Table 2. Before filling out the SUS questionnaire, all evaluators ranked their previous experience with VR on a scale from one to five to indicate if the perceived user-friendliness could stem from a lack of experience. The proposed solution has undergone two rounds of evaluation. The first round featured three therapists. During the I think that I would like to use this system frequently. 2. I found the system unnecessarily complex. 3. I thought the system was easy to use. 4.
I think that I would need the support of a technical person to be able to use this system. 5.
I found the various functions in this system were well integrated. 6.
I thought there was too much inconsistency in this system. 7.
I would imagine that most people would learn to use this system very quickly. 8. I found the system very cumbersome to use. 9.
I felt very confident using the system. 10. I needed to learn a lot of things before I could get going with this system. evaluation with two of the therapist (E1 and E2 in the same session), some technical difficulties occurred, giving the evaluators a worse impression and resulting in a lower SUS score than what would have been without it. The mean score ended at 69.2, which is, according to Bangor et al. [23], an OK score and falls just above the average stated by Sauro [39], which is 68 after looking at 500 studies. This shows that the system has room for improvement. Although the mean score is important, individual responses can give a clearer view of the positives and negatives. Table 3 shows that therapist one (E1) scored 65 on the SUS scale. E1 indicated medium prior experience with VR by giving the question a score of 3. What is interesting to note about the results of E1 is to compare the answer Q4: "I think that I would need the support of a technical person to be able to use this system, " with the answer to Q7: "I would imagine the most people would learn to use this system very quickly, " and Q8: "I found the system cumbersome to use. " By giving Q8 a score of 1 and Q7 a score of 4, he indicates that the system is easy to use and learn, but at the same time states that he would need the help of a technical person by giving Q4 a score of 4. This result could indicate that the technical problems at the beginning of the evaluation highly influenced the results.
Therapist two (E2) had no previous experience with VR and ended with a SUS score of 60, as can be seen in Table 3. E2 did not note the same discrepancy between questions that could have somewhat of a connected meaning. As E1 did, E2 gave a four to Q4. Since this evaluator did not have prior experience with VR, it is hard to assess if it would be due to setting up VR or the technical problems encountered. The response of E2 to Q2: "I found the system unnecessarily complex, " and Q3: "I thought the system was easy to use, " speaks well for the application, as a therapist without prior VR experience found it easy to use.
As for the semi-structured interview, E1 and E2 agreed that the games are relevant for cognitive rehabilitation for a select group of users. They state that the tasks in the plane game are cognitively demanding and that both games contain a fair amount of motoric abilities. The evaluators also stated that the games could fit as a distractor or pastime activity during balance training. Both evaluators think that the games fit well as a supplement to traditional rehabilitation, as they found them fun, engaging, and innovative. As for the tablet application, they find it useful as the setup becomes more mobile.
E3 is the therapist with the most VR experience in the first round, as they noted a score of four for previous experience. This evaluator did not have any technical difficulty with the games themselves but could not test the tablet application. They did not think the application was difficult to use or think they would need help from a technical person.
In accordance with E1 and E2, the last evaluator of the first set considers the games relevant for cognitive rehabilitation following a stroke. E3 stated that the painting game could see a wide range of uses, and while the game demands less focus and concentration, it still demands attention and speed due to measuring time. As for the plane game, it is more demanding in terms of focus switching and spatial orientation. E3 believes that the game would fit, as patients are more focused and engaged while it is "game-based," where they would be responsible for solving the tasks and orient themselves in the virtual environment.
The second round of evaluations went more smoothly than the first, and no evaluators experienced technical problems. This round also featured improvements and suggestions from the first. All evaluators scored above what is considered good by Bangor [23], with the average at 83.3. Both E1 and E2 (same session again), which were present in round one, show a significant increase in the SUS score, as seen in Table 4. The increased score stems from no technical hiccups and added improvements. Interestingly E2 (who has the least amount of experience with VR) answered more positively on Q4 than E1, but both think they are likely to need help from a technical person. In the semi-structured interview, both evaluators thought that the changes made the games more relevant, especially the scoreboard, as comparing scores between sessions became easier. Other minor changes made the games more intuitive as well.
The second round also included E4 (a specialized doctor) who has prior experience with VR (score 4 out of 5) and games in general. This evaluator gave a SUS score of 97.5 and found parts of the user interface more intuitive than those with less experience. While the games do include haptic feedback in the form of controller vibration, E4 suggested adding more along with sound effects. As for the semi-structured interview, E4 stated that the games are relevant for some and that it is a good starting point. As for the relevancy, the games would fit as a supplement, as there is the concept of visuospatial skills, and the plane game is cognitively demanding. However, they stated that the plane game could be a bit much, and there should be a lower difficulty setting. While discussing the tablet application, they stated that it would be a good addition, especially if the clinic lacks a computer setup.
This paper presents two games and a tablet application3 intended to supplement traditional cognitive rehabilitation after a stroke. The rehabilitation process gets more effective by increasing patient motivation and engagement and giving the therapists more tools to individualize tasks. Based on the few who had the time to evaluate the games, the results show that their usability is at an acceptable level. The evaluations also suggest a potential to increase patient motivation and engagement. The evaluators were quite optimistic about the streaming solution as a concept; therefore, it should be revisited and optimized. In conclusion, the proposed solution shows potential use. However, as the sample size is small, more evaluation needs to be done, and preferably include patients at a later point to see the actual effect.
As mentioned in section 3.5, the game is built in two different versions, and these should be merged so that the therapist only needs to relate to one version. The merge is achievable in several ways, whereas one is to introduce a centralized state by, for example, using Unity's scriptable objects. Alongside the state, create a "manager component" responsible for enabling and disabling components for performance based on if streaming is enabled. Secondly, the games and streaming solution needs optimization to increase the video quality and later be able to re-introduce all maps when streaming is enabled.
During the second round of evaluations, E4 mentioned a lack of haptic and auditory feedback. There is no auditory feedback at the time of writing and only haptics when a box gets placed in the box painting game. Including these might result in more immersion and better inform the user when something happened (box painted, button clicked, game victory).
As mentioned earlier, the number of evaluators who could test the solution is relatively low. Therefore, more evaluations with therapists are needed to assess the usability and have their perspective on the potential motivational increase of the games. Alongside having more evaluations with domain experts, some features need to be enhanced before potentially testing the usability and efficiency with post-stroke patients.