This paper describes a camera-based approach towards creating a tangible interface for serious games. We introduce our game for dental health targeted at school children which implements the Nintendo WiiMote as infrared camera. Paired with a gesture-recognition system, this combination allows us to apply real-world items as input devices. Thereby, the game tries to address different aspects of dental hygiene along with the improvement of children's motor skills. In our focus group test, we found that tangible interfaces offer great potential for educational purposes and can be used to engage kids in a playful learning process by addressing their childlike curiosity and fostering implicit learning.
INTRODUCTION
Research has shown that insufficient dental hygiene of
children and teenagers has a negative impact on the
development of their second dentition: Children whose
primary teeth have been affected by cavity are at a
significantly higher risk of developing caries during
adolescence and adult life [
In this paper, we describe our tangible approach towards interface design for a game for dental health which features a user interface based on real-world items and thereby tries to convey psychomotor skills along with factual knowledge on dental hygiene.
RELATED WORK The issue of dental hygiene of children and teenagers has rarely been addressed by serious games.
A ubiquitous approach towards teaching tooth brushing to school children has been suggested by Chang et al. [3]. Their system tracks children’s brushing behavior while cleaning their teeth and presents brushing results on an LCD display. Additionally, they conducted an accompanying study which showed an improvement of brushing strategies and a significantly longer overall brushing time. Nevertheless, the application only includes few game elements and focuses on sensorimotor skills instead of game play, which might affect player’s motivation in the long run.
Dental Attack [4] is a serious game featuring 3D graphics in which the player has to protect a large tooth from cavity by administering correct cleaning behaviour. It aims at improving the player’s knowledge and motivation regarding dental hygiene, but possible positive effects of the game have not been validated in the context of an evaluation.
Additionally, a wide range of flash games featuring the topic of dental hygiene is available on the internet. Furthermore, tangible interfaces for children have frequently been discussed and a variety of playful applications has been designed.
TICLE was designed by Scarlatos et al. and is an early
approach to game design based on tangible interfaces for
children. It is a tabletop tangram game which supervises
player’s movements and offers help when necessary [
Ryokai et al. [
Furthermore, Ho et al. [8] approached serious game design from the perspective of tangible user interfaces and created a health game aimed at children attending primary school which implements the Nintendo WiiMote.
GAME CONCEPT The game concept presented within this paper is based on the idea of fusing game mechanics and learning units in order to create an immersive user experience. It features a combination of two complementary game modes which incorporate different didactic goals. Furthermore, the concept aims at the implementation of intuitional interaction paradigms designed to engage the player in game play and to facilitate learning processes. A detailed description of the game concept and its didactic foundation was published in [7].
Game Design Our game design suggests the combination of two game modes, nutrition-mode and cleaning-mode, to implement in-game challenges which address different learning objectives.
In nutrition-mode, the player has to take care of a group of teeth advancing through the game in a style similar to 2D side-scrolling games (cf. Figure 1). Each level consists of a set of comestibles which the teeth have to chew. Additionally, a time limit which represents an average playing value is implemented to support the calculation of a comprehensible high score at the end of the level. While the group encounters challenges represented by a variety of comestibles causing negative effects ranging from contamination and plaque to dental decay, the player may unlock bonus items such as fluoride gel or chewing gum by entering quick mini game challenges or decide to switch to cleaning-mode in order to administer cleaning treatment. After switching to cleaning-mode, the player is asked to choose from a set of three cleaning instruments displayed at the top of the screen (cf. Figure 2). He may decide to use a tooth brush, mouthwash or dental floss to remove traces of comestibles and plaque. Depending on the player’s choice, an adequate input device allowing the conduction of different cleaning gestures has to be selected (cf. Enhanced Tangible Interaction).
During game play, the user is given the possibility of returning to each of the game modes as necessary. At the end of each level, scores are calculated based on the average status of the group of teeth and the overall quality of cleaning behavior administered by the player. Thereby, the player receives additional feedback regarding his ingame performance. The game tries to raise children’s motivation by putting them in charge of a group of teeth while equipping them with different tools. During game play, children are continuously asked to make choices and consider consequences of their actions while being provided with immediate feedback. Thereby, the game aims at the improvement of their self-efficacy while challenging their factual knowledge regarding dental hygiene. Furthermore, our game concept addresses motivational aspects by the presentation of teeth as vulnerable creatures which depend on the player’s care and protection.
Additionally, the transfer of factual knowledge is fostered by the visualisation of dental decay caused by the consumption of comestibles during nutrition-mode. Depending on the type of comestible which was encountered by the group of teeth supervised by the player, plaque or contamination is caused. Furthermore, the speed of decay is affected by the core ingredients of each comestible, such as sugar (e.g. chocolate) or acid (e.g. cola). Besides, the game accentuates dependencies between eating habits and their impact on specific requirements of adequate means of cleaning through different game mechanics.
The development of sensorimotor skills is addressed by the implementation of cleaning-mode, which requires gesturebased interaction and includes the application of real-world items as input devices (cf. Enhanced Tangible Interaction). INTERACTION DESIGN Interaction design for children and teenagers is a challenging task due to the particularly wide range of cognitive and sensorimotor skills of the target audience. This requires an inclusive design approach which takes different stages of children’s development into account [2]. Our game tries to address this issue by offering two interaction concepts which require different levels of sensorimotor precision and both try to reduce children’s cognitive load by implementing facile input paradigms. The basic approach towards interaction design includes mouse and keyboard input, whereas the enhanced interaction concept is based on the implementation of tangible controllers, i.e. the Nintendo WiiMote and real-world items.
Additionally, we believe that our tangible approach to interface design may address a broader audience and activate children’s curiosity, which would be ideal regarding our target audience of both younger children and teenagers.
Basic Interaction Design The game features a conventional user interface based on the combination of mouse and keyboard input. The player is offered the possibility of interacting through point-andclick operations as well as keyboard input: During nutrition-mode, the group of teeth is moved using arrow keys. After switching to cleaning-mode by using the mouse to click on a particular tooth, further mouse input is required to perform different cleaning actions. In order to evaluate the quality of the player’s attempts to use different cleaning instruments, we implemented a gesture recognition system which is also used within the enhanced tangible interaction and is described in the following section.
The advantage of this basic approach is the wide availability of mouse and keyboard as input devices. Additionally, many children and teenagers have previous experience with similar interaction paradigms, which is likely to provide an easy entry to game play. However, input in cleaning-mode requires accurate mouse movements which might not be suitable for younger children. Therefore, an enhanced tangible interaction applying real-world items as input devices was implemented.
Enhanced Tangible Interaction
In contrast to our suggestion for a basic interaction scheme,
the enhanced design embeds real world items into the
game. It introduces infrared input recognition which is
based on the idea of implementing a gesture-based interface
which allows young children to actively participate in game
play and engages older kids due to its innovative nature.
In this context, we decided to use Nintendo’s WiiMote as
primary input device because it is one of the most popular
controllers for digital games. Furthermore, it provides us
with a variety of information on user input through its
button interface, translatory sensors and the additional
infrared interface. Additionally, the availability of a wide
range of third party software such as the WiimoteLib [
The game requires a total of two WiiMotes: One Mote is used as regular pointing device, another mote is located in front of the player and serves as infrared camera. In the following section, both input paradigms will be described.
WiiMote Input The primary WiiMote controller is implemented as pointing device. This setup requires a wireless infrared sensor bar similar to the bar delivered with the Nintendo Wii, which is positioned in front of the player in order to track input via the infrared interface. Furthermore, button input is registered to trigger in-game events.
During nutrition-mode, this remote is used to control the group of teeth representing the player, to participate in mini-games in order to unlock bonus items and to switch to cleaning-mode when necessary. In cleaning-mode, the pointing mote may be used to select an appropriate cleaning device. Once a device has been selected, the player needs to exchange the pointing remote with the corresponding reflector-based input device.
Camera-based Input Instead of using the WiiMote as pointing device in combination with a stationary sensor bar, it is also possible to implement the remote as fixed infrared camera and move a source of infrared light instead. Thereby, it is possible to track different input gestures conducted by the user if he or she is equipped with adequate input devices. This allows us to implement a tangible user interface which is based on the idea of utilizing real world items as input devices.
WiiMote
Input Input Items Brush Cup Floss
Input Detection
Gesture Recognition
System Display / Audio
Game
Input Evaluation
User Feedback
Technically, there are two different approaches based on the idea of casting infrared light on a stationary infrared camera which we considered for our setup. Both require the implementation of additional LEDs similar to those used for the original sensor bar. The LEDs utilized for the array should ideally have a peak wavelength of 950nm, otherwise tracking results are not stable enough and are easily influenced by other sources of infrared light, e.g. sunlight. At first, we tried to invert the regular pointing system by attaching LEDs directly to the objects which would then be used as input devices. This requires a power supply which either needs to be attached to the object or has to be connected by wire. Unfortunately, this may be a disadvantage when designing for children since the prototypical input devices we developed were very fragile and could easily be damaged when dropped or carelessly handed over to another person. Because we were intending to use the interface for teenagers and younger children, it was also problematic that most LEDs which are currently available have an operating temperature up to 75°C and therefore cannot be used in interface design for kids without the risk of injury. However, this method returns good transmission results and offers the possibility of tracking single input devices as LEDs may be used as markers which are recognized by the system during game play. Thereby, it is possible to determine whether the player is using the correct input device.
Alternatively, it is possible to implement a system which
does not require the attachment of LEDs to the hardware
user interface: The installation of an LED array casting
infrared light towards the player in combination with the
camera WiiMote allows us to track user input without
attaching any electronics to the input devices of our choice.
Because we wanted to be able to track more than one input
event at a time at a later point of development, we decided
to adapt a multi-touch finger tracking system originally
designed by Chung Lee [
To improve tracking results, Chung Lee suggests that players attach reflective tape to their finger tips which increases the amount of infrared light being thrown back at the camera remote. Instead of adding reflective tape to the player’s fingers, we decided to attach it to a tooth brush, a cup and an imitation of dental floss. In order to achieve an acceptable transmission result, we used Scotchlite Solas GradeTM, which is a type of reflective tape that is commonly used in shipping. Besides a high flexibility, the tape concentrates light rather than having a dispersive effect which increases the amount of light being returned to the camera remote.
Interface and Recognition System
LED Array Camera WiiMote
Gesture recognition
IR light reflection
User
Tools with reflective tape (Tooth brush, cup or dental floss)
Game In order to receive a strong infrared signal, we used an LED array consisting of 40 LEDs with a wavelength of 940nm (+/- 50nm). We installed the LED array on a wooden rack with a little hinge which allows for the adjustment of the LED array in a very flexible manner. This is particularly important because the angle of the array needs to be adjusted according to each player’s body size to grant an optimal tracking result. On top of the rack, a clamp was installed which allows us to mount the camera WiiMote accordingly (cf. Figure 3).
The advantage of this method is the fact that the input devices are lightweight and fully functional without any additional electronics or other attachments which might disturb the user (e.g. wires connecting the device to the system). If necessary, all input devices can be replaced at low cost, because average retail items can be used which only need to be enhanced with reflective tape.
Figure 5 shows how input information is processed by the system described in the previous section. First, the LED array casts infrared light towards the objects utilized by the player. Second, light is reflected by the input device and thus recorded by the camera WiiMote. Then, 2D coordinates representing the relative position of the input device to the camera are obtained and translated into mouse coordinates. Therefore, it is possible to apply regular mouse gesture recognition algorithms during the next step of information processing. Afterwards, information regarding the quality of player input is passed on to the game and corresponding feedback is displayed to the player. For each instrument in cleaning mode, a corresponding object was chosen and laminated with reflective tape. Thus, the game features a tooth brush, a cup as well as an imitation of dental floss made of cord and wooden pearls (cf. Figure 6).
The following section describes the set of characteristic input gestures which is required to play the game. To use the cup filled with mouth wash, the player has to lift it up to his face and pretend to drink out of it. Thereby, the camera records a vertical movement which can easily be identified by the gesture recognition system. Dental floss is used by picking up the cord with both hands and slowly lifting both ends in an alternating rhythm. Since the tracking system supports multiple input sources, two vertically moving objects are detected and submitted to the game as player input.
If the player uses the cup or dental floss, the game only checks whether the corresponding gesture has been completed. However, if the tooth brush is used, the quality of the gesture has to be evaluated. Therefore, a more complex interpretation process is required which is able to determine the precision of each player’s cleaning behavior. Thus, we implemented a simple algorithm which allows us to judge whether player input was good, average or bad. If the tooth brush is used correctly, a circular movement is registered by the system. As this is a continuous process, the cleaning period is split into cycles. Once one cycle within the brushing process is finished, the recognition system evaluates the accuracy of the player’s cleaning movement by comparing player input to an ideal circular movement. Figure 6 shows the recognition process: If user input is registered which may be similar to circle A (good) or circle C (bad), it is compared to the ideal circle B. Based on four distinctive points of the player’s input gesture, the area of the circle is approximated and compared to the ideal circle which is based on a set of four predefined values. It was necessary to reduce the amount of values drastically to reach an acceptable level of system performance and to be able to deliver immediate visual feedback to the user. In order to determine whether the input cycle has been finished, each circle is split into four areas (cf. Figure 7). The transition between two areas can be detected by a corresponding change of the X and Y values of mouse coordinates. Depending on the alteration of these coordinates, the system defines an initial starting point at which the cycle started and is later expected to end. Thereby, the system is able to distinguish between circular strokes and mere vertical or horizontal scrubbing. Finally, the user is provided with visual feedback regarding the individual cleaning performance.
In contrast to regular mouse and keyboard input, the enhanced interaction design offers a more complex, yet intuitional tangible interface which we expect to have a positive impact on player’s motivation and engagement. In the following chapter, we describe the results of a first focus group test during which both interaction paradigms were compared.
FOCUS GROUP TESTING In order to evaluate the two interaction concepts presented within the previous chapter, a stable and fully playable prototype which offers both mouse and keyboard input as well as WiiMote and item-based interaction was created. The focus-group test was conducted with nine school children with a mean age of 10 (range 9 to 11). The group consisted of three girls and six boys, all of the children were from socially disadvantaged families and therefore at a generally higher risk of suffering from the consequences of insufficient dental hygiene. Therefore, the focus group represented one of the core audiences of our design concept. Besides, all participants had previous experience using the computer and playing games.
Setting and Method
At the beginning of the test, the children were divided into
two groups. Group A consisted of five children and was
later on presented with the enhanced tangible interaction
using both WiiMote and real-world items. Group B
consisted of four children who were asked to play the game
using keyboard and mouse. Each group was granted one
hour of playing time followed by a fifteen-minute
structured group interview. Before the start of the playing
session, the interaction methods as well as keyboard and
button mappings were explained. During game play,
children were observed and asked questions about their
ingame actions according to the Active Intervention method
suggested for the evaluation of interactive products with
children subjects [
Results and Interpretation Children’s comments during the playing session showed that group B generally had no difficulties approaching the game, because all children had previous experience using the combination of mouse and keyboard as input devices for digital games. On the contrary, participants of group A needed assistance when switching from WiiMote to itembased interaction. Difficulties were primarily caused by the fact that the specific version of the LED array required careful player alignment and did not take children’s restlessness into account. In that context, using a regular tooth brush and cup was advantageous because these items turned out to be very robust and are barely affected by careless treatment. Additionally, comments showed that children within group A were enthusiastic about the use of real-world items as input devices and were highly motivated to enter cleaning-mode to take proper care of their virtual teeth. In this regard, we observed that children within group A generally showed a higher willingness to cooperate with their peers than participants within group B. We believe that the increase in participants’ communication can be accredited to the availability of several input devices which encourages collaborative game play rather than indicating a 1:1 relationship between one user and the system through a regular mouse and keyboard setup.
The observations which were made during both playing sessions were generally supported by children’s statements and comments in the context of the follow-up interview. All of the subjects within group A claimed that they preferred cleaning-mode over nutrition-mode because it allowed for the use of real-world items as input devices, which they commented on as “fun”, “magic” and “exciting”. Participants of group B reported that they enjoyed playing the game but did not make any distinction between both modes. When questioned about the user interface in particular, they explained that they found the interface to be usable, but their comments did not show the same curiosity and enthusiasm as those of participants within group A.
In general, the focus-group test showed that new technologies offer great opportunities for educational games because they have the potential to engage children in a playful learning process. Teaching staff positively highlighted the fact that their students actively discussed ingame actions as well as aspects of dental hygiene, which in their opinion offers a good opportunity of picking up the topic of dental hygiene in a more formal context. Thus, the game could probably be used in a classroom context to introduce the topic of dental hygiene. Furthermore, the implementation of tangible interfaces in combination with gaming applications may foster implicit learning processes, because the player’s desire to master the game requires mastery of the input devices, too.
DISCUSSION The focus group test showed that the tangible approach towards interaction design for serious games has the potential of engaging children in game play and learning processes. Nevertheless, this first approach only included few subjects and should be understood as an indication of future research focuses rather than an extensive evaluation allowing definite conclusions regarding usability, interface design and learning success.
One of the biggest advantages of the reflector-based approach to interface design presented within this paper is the fact that all tangible input devices required during cleaning-mode are lightweight and robust as they do not utilize fragile technology. Additionally, they can easily be replaced at low cost since the current implementation is based on conventional cleaning instruments enhanced with reflective tape, which is ideal for the design of tangible interfaces for children.
However, the current implementation does not allow the game to determine whether the player utilized the correct cleaning instrument. This may be problematic if the game is played without supervision. Besides, we learned about a number of usability issues regarding the setup process of the game during the focus group test, for instance establishing a Bluetooth connection between the WiiMote and a PC is rather difficult due to connectivity issues and requires a level of technical expertise which should not be assumed. Furthermore, the current setup of the WiiMote and LED array is very prototypical (cf. Figure 3) and is not suitable for unsupervised use by children and teenagers yet. FUTURE WORK Future work will include the replacement of the camera WiiMote by a regular infrared camera to address connectivity issues and to facilitate the setup process which is required to play the game. This is especially important as a comprehensive clinical study requires an increased level of usability regarding the technical installation as the system needs to be set up by both teaching and medical staff. Additionally, we are planning on introducing a new rack to carry the camera WiiMote and the LED array which is suitable for repeated use in the context of a clinical study. Besides, it is planned to add a basic logging system to the game which tracks the most important in-game actions performed by the player, such as the use of cleaning instruments or the frequency of cleaning sequences during game play. Additionally, metadata such as the overall playing time and high scores could be included in the logfiles. Implementing a logging system offers the opportunity of supporting possible general findings of the evaluation by individual player data, which may support the interpretation process of evaluation results.
Additionally, our hypothesis regarding the activation of implicit learning processes through tangible user interfaces for serious games needs to be tested within an evaluation of learning success. This includes the revision of additional hypotheses concerning the transfer of factual knowledge as well as psychomotor aspects. We hope to conduct these tests in the context of a clinical study which is currently being prepared.
Furthermore, we would like to conduct further research
regarding the aspect of the development of game mechanics
in combination with didactic goals. The fusion of game
elements and learning units to positively influence
children’s self-perception and thereby affect behavioral
outcomes has previously been proven to be effective in the
context of a serious game designed for children diagnosed
with cancer [
ACKNOWLEDGMENTS We would like to thank our project partners Prof. Dr. Elmar Hellwig and Johan Wölber of the School of Dentistry of the University Medical Center Freiburg for the creation of a serious game for dental health and their excellent support regarding all dental issues. Furthermore, we would like to thank Manuel Grundmann and Michael Mayer for their work on the game design and prototypical implementation of the game within their graduate research project.
&
Children (IDC ’07):
[last
access: (2009). [last access: