This paper presents the FP7 European Project GAME-ABLING[1] developed from December 2012 to January 2015. This project aimed at the development of a platform for the creation of games for patients with Cerebral Palsy (CP). A key point of the platform is that the framework can be used by personal with no specific skill in game creation, permitting caregivers and parents its utilization. The system is composed of (i) a framework that encompasses the several tools developed to run and control the games, (ii) the authoring tool to easily allows the creation of new games, and (iii) the analyzing tool that generate statistics on the impact of the games in CP patients. Due to motor and cognitive constraints of CP patients, specific sets of games were developed. Also an extensive group of peripherals can be employed beyond the usual game controllers, including color and depth cameras, Nintendo Wiimote and balance boards. This paper describes the system elements and the results obtained during the evaluation of the games with real patients.
Cerebral Palsy (CP) is one of the most frequently conditions in childhood, with an incidence of 2 per 1,000 live births. In the EU there is 1.3 out of 15 million persons with CP in the world. This neurological disorder affects body movement, balance and posture and almost always is accompanied by other cognitive or sensory impairments like mental retardation, deafness and vision problems. The severity of these problems varies widely, from very mild and subtle to very profound. These disabilities lead to an inactive lifestyle which reduces the patient’s physical health, social participation, and quality of life. Therapy costs (up to €45,000 by year) cannot be afforded by most of the families. Playing Video games is a useful treatment that promotes and maintains more active and healthful lifestyle in these persons. However accessibility to videogames is hardly applied for them. The FP7 Project GAME-ABLING developed a software tool for creating interactive video games in an intuitive manner so non-expert personnel (caregivers and parents) can develop customized games. GAME-ABLING are accessible games with the aim of improve physical activity of disabled people. This is a different approach that benefits both caregivers and therapists, who are able to design and easily customize games for their patients, and also for disabled people, who are able to play with the games developed by caregivers of therapist improving their fitness while playing. Games created by GAME-ABLING can be controlled in a variety of ways, allowing the use of body movements and voice. The images recorded by the webcam are processed using computer vision techniques in order to track head, eyes, and hands as well as to detect some basic gestures like hand shapes (open hand, closed fist or hand pointer shape) and face gestures (aperture of mouth and opening/closing the eyes). Voice tones and all these movements and gestures are also translated into user actions that could be used to control the videogames, similar to a joystick or a mouse.
GAME-ABLING also supports other type of standard input game controllers such as mouse, keyboard, joystick, Wii controller, and balance board. According with the type and severity of disability of patient, caregivers can decide which type of game design and which type of game controller input can be used. Game performance and player feedback is registered in a database and analyzed by caregivers or therapists to determine the physical fitness improvement.
The GAME-ABLING system is composed of the following elements: (i) image analysis module, (ii) user input modules, (iii) game development framework, (iv) game authoring tool, (v) analysis tool and activity database, and (vi) games. The structure of the complete system can be seen in Fig. 1. The idea of the whole system is to provide the tools to create, use, evaluate, and share the games by a community of users with specific requirement (CP patients) and non-expert programming skills (caregivers and parents). In the following subsections each part is described. 2.1.
This module is able to capture movements of the different parts of the body, especially focusing in the obtaining of head and hands movement employing two different types of cameras, generalist color cameras and depth cameras, such as Kinect. Also a database with images and videos were collected to test the module.
Data Collection. During the GAME-ABLING project, we have created a database of videos recorded at gaming sessions in the facilities of one of our partners, the Associació Provincial de Paràlisi Cerebral (APPC) in Tarragona, Spain. The goal of this database is to test the computer vision algorithms in similar conditions as those encountered during the use of the games. Furthermore, we annotated the some images of this dataset manually to train our head and body-part detectors. We also collected another dataset in APPC and developed an annotation software and annotated them in URV. These databases were created scrupulously following the ethical issues involved in the obtaining and processing of data from Cerebral Palsy patients.
Image Analysis. A series of algorithms for analyzing and tracking the head movements based on skin color segmentation and a state-of-art face detection method were developed. These algorithms work with standard color webcams and the depth cameras. Although the algorithms are accurate, there were some practical challenges when tested in real scenarios on the CP patients depending on their severity level. We observed that the algorithms worked accurately with levels 1 and 2 patients, which are those with the least level of impairment. However, level 4 and 5 patients (the highest levels) were not able to control the games due to the fact that their facial and body-part appearance and movements are much more different than the patients with levels 1 and 2. In order to solve such difficulties we also utilized the depth information for segmentation purposes, which also increases the computational efficiency and makes it more robust against illumination. This approach can deal with one of the important challenges that we faced in the project: patients with higher severity level are barely able to control their heads and hold it in a frontal or near-frontal position. As the results, the state-of-are methods are not able to detect the face and, consequently, users are not able to control the games. 2.2.
A series of libraries were developed in the project to access different input devices to control the game through a wide range of possibilities, i.e., direct motion controllers, image-based controllers, and audio-based controller. These libraries allow an application to read inputs from the following list of devices: Nintendo Balance board, usual game controllers (joysticks, gamepads, and switch buttons), Nintendo Wiimote controller, keyboard and mouse, color and depth cameras, and a microphone. The inputs from these devices can now be read, adapted when necessary and used by games to perform different game actions. As regards the audio devices, a simple method to capture and measure the intensity of the voice or blows into the microphone allowing the device to control some game actions. The inclusion of cameras as game controllers was accomplished as two modules for integrating Microsoft Kinect and color cameras. 2.3.
The main core of the system consists of the common game framework. A configurable launcher was created for two different families of games. The launcher is capable of loading the game description and assets, loading different input modules, displaying a control panel and finally allowing the user to play the game. The work focused on the development of a software framework that assisted in the development of configurable games and the integration of general purpose and specialized input devices in order to support a wide range of patients/disabilities. The approach followed was to abstract user input and to decouple devices from games, which facilitates the adaptation of games to future use cases. Moreover, a modular architecture was designed to allow for the independent development and deployment of base games and input modules. This is important as it helps the development throughout the project lifetime but also facilitates future developments and allows the introduction of a licensed development scheme; allows the commercialization of modules; helps the deployment process when different platforms are involved and when upgrading an existing installation. Finally, a central application is provided that coordinates all existing modules and presents the user with a single environment through which to configure and control game sessions 2.4.
The main objective of this tool is to provide an intuitive, easy to use with no special expertise required, authoring environment that combines configurable games and the various game elements into complete – ready to played - games. The approach followed was the development of a web application, capable also to be executed offline and supporting all major browsers and a step by step approach in developing the games. A common look and feel was adapted across all base games, although the game play of each base game is considerable different. The authoring tool allows to load user-created assets, like avatars, sounds, backgrounds, and their position in the window.
A number of considerations were taken into account in the design of the authoring tool: (i) usability: the solution is very easy to use and self-explanatory, (ii) deployment: the solution is hassle-free of installation and OS considerations, and (iii) effectiveness: The solution is effective in generating a wide range of games. 2.5.
This module generates for each patient databases with data of the performance of the patients during the gamming sessions and an activity report that visualizes it creating graphs related with the patient performance. The relational database is capable of storing two set of information: configuration of individual games/users and the actual data of the played games. The purpose of this database is to use data to analyze the development of the patient. The analysis tool was developed in order to study the information stored in the database. This tool allows to access, combine and visualize information stored in the database according to criteria of specialized personnel (therapists, caregivers, and psychologists), so they get the best possible profit of the activity database. It is a graphical web interface tool with diagrams and statistics, which inform the interested users of the evolution of each patient, and based on this information, they will be able for further customization the games for each patient. The data generated during the games is anonymized and follows the strict requirements of the Spanish Data Protection Act (LOPD). 2.6.
Following the suggestions of the CP experts involved in the project, three main families of game typologies were implemented in the project, namely, XY games, ‘spot-thedifference’ games, and memory games:
XY Games: these are games where an avatar is moving in one or two axes: vertical and/or horizontal and the main goal of the player is to collect items (Friends) and avoid enemies (foes). With each Friend collect it, he/she gains points and with each foe hit he/she loses lives.
Spot-the-Difference Games: in these games the player needs to find small differences in two images that look identical.
Memory Game: Here the player is presented with a grid of images, where every image exists twice. After few seconds all the images are hidden and the player needs to select the same images sequentially.
Evaluation of the Game Authoring Tool (GAT) and Rehabilitation Games developed for CP patients of different levels of motor impairment and age was done by specialists from International Clinic of Rehabilitation (ICR) in Ukraine and the Associació Provincial de Paràlisi Cerebral de Tarragona (APPC) in Spain, partners of the project employing the following procedure. Therapists were taught how to use the GAT and supplied with assets to develop games. They also learned how to use games with different gaming hardware – balance board, Kinect sensor, camera, special goniometer joystick, and others. Thus, they could develop their own games under supervision. Totally 12 games were selected for testing on 32 patients with CP of different level of motor disability. Therapists filled out the questionnaire that was analyzed and conclusions about the GAT usability was obtained. Every patient participated in 6 to 8 gaming sessions of 15-20 min duration under the supervision of the therapist. During training sessions different gaming hardware was used. After the sessions, parents were interviewed and filled out the questionnaire that was later analyzed.
Analysis of the questionnaires indicates that the majority of the patients highly rated rehabilitation games - 82% ranked the rehabilitation games as excellent and 18% as good, and nobody stated that the games are of poor or average quality. Also the majority of the parents liked graphical and sound effects of the games and stated that the game difficulty was adjusted properly to the motor abilities of the child. Parents suggested in their comments that there should be a larger collection of the games of different difficulty levels with nice animated characters, focusing not only on motor development but also at training cognitive functions. There were also suggestions to integrate the games into social networks for disabled children.