We report on challenges we encountered when using automatic measurements for a longer term exploratory study (8-10 sessions, 9 participants) with people with Profound Intellectual and Multiple Disabilities (PIMD). In the overall study, which we will publish elsewhere, one element that we investigated was if we were able to persuade users of this target group to move more by providing interaction with an interactive ball. This paper focuses on the challenges we had regarding the use of our method for automatic measurement of movement based on camera recordings during this study. With this paper we like to remind researchers to not rely blindly on the outcome of automatic measurements and instead to analyze measures in depth, which can become difficult when using extensive sets of data. 1
Not all people have the combination of cognitive and/or physical capabilities to be able
to enjoy modern sources of leisure such as computer games, watching TV, or interactive
theme-park rides. Especially for people with Profound Intellectual and Multiple
Disabilities (PIMD) there is a limited amount of suitable entertainment [
People with PIMD are dependent on their caregivers, have low intellectual capabilities
(immeasurable), and have disabilities that reinforce each other and can not be
compensated for [
In our project, which was instigated by the care organization Dichterbij, we emphasized
the opportunities offered by new interactive technologies. For example, the Kinect depth
camera or the Arduino prototyping platform can help develop suitable activating leisure
activities that persuade users from this target group in a pleasurable way to exhibit
certain wanted behavior. Larsen and Hedvall showed that interactive technology allows to
create new pleasurable experiences tailored to people with PIMD [
One approach which seemed promising was to make use of automatic measurements,
both for measuring study outcomes as well as for triggering the interaction patterns,
for instance applying automatic measurements to indicate preferences by measuring
movements that are almost invisible to the eye [
With this paper, we want to achieve two things: 1) we want to inspire our research community to also make people with special needs benefit from persuasive technology and share some of our process and choices to this end, and, more importantly, 2) remind the reader that unforeseen side-effects and incorrect values in automatic measurements challenge us in using extensive data for this kind of system. This holds for evaluations, especially for the analysis as shown in this paper, but in truly automated interventions also for the performance of the system during the intervention itself. The remainder of this paper is structured as follows. We will continue with related work on the user group and (related) interactive entertainment for people with special needs. In the third section we will briefly describe our prototype and underlying design principles. In the fourth section we will then explain our automated measurement for movement and our experiences with this. We will conclude by discussing some of our vision and the challenges that become apparent when working with this target group attempting to use automatic measurements over longer periods time. 2
To our knowledge there are only a few examples available of research that develops
truly interactive technology for people with PIMD [
The related work that can be found seems to point towards personalized and tailored
interactions, which also fits the Persuasive System Design model [
In our design efforts we build on our earlier work and that of Caltenco et al., also
using interactive body-controlled physically present objects [
We developed an interactive ball that can be remotely controlled based on gross body movements and vocalizations, see Figure 1. During our development there were two different versions of this interactive system. Version 1 was controlled fully automatically on basis of recognized body movement using a Kinect sensor. We only used this version with students and at the start of a pilot with 5 participants from the target group. In version 2 the ball was remotely controlled by a facilitator, this was the ‘final’ version that we used for the longer term (8-10 sessions) study. 2 The ball moves based on changing the center of gravity by rotating two weighted arms connected to a servo inside the ball. This allows for a gentle movement, unlike the more direct and quick movements of the commercially available interactive ball Sphero. With a WeFly wifi-hotspot, simple string based commands were sent to the ball. We programmed a simple Graphical User Interface (GUI) in C++ (using the Qt framework) that allowed the facilitator to move the ball with the keyboard cursors and to use some fields to set the speed. Key inputs could be used to play 17 different sounds. Most of these sounds were made using free virtual (synthesizer) instruments, 6 were recordings from an online audio database (mainly animal sounds and bells). The sounds were played in front of the user over standard PC speakers. We painted the ball in highly 2 Currently we also have a simple tablet application to control the ball’s movement, change the
LEDs, and play sounds from within the ball. contrasting blue and yellow colors. After some sessions we also added a rattle, so the ball made more noise. This could make it easier to follow the ball, especially important for people with more limited visual capabilities.
As stated above our first concept was based on an automated detection of the user’s movements with the Kinect depth camera and software. At the same time we used a webcam and a simple background subtraction method to track the position of the ball. We created an interaction in which the ball’s position (left/right on a predefined path) was dependent on the position of the head. To test the technical feasibility and to improve this interaction we tested this with 40 students. During these tests we made some small technical improvements (e.g. in the implementation of sending commands) but the system worked reliable enough at the end of these tests.
However, during the first tests with people from the actual target group we realized that one fully automated interactive system was not flexible enough to deal with the heterogeneity of the user group. For instance, we planned to use the head positions as the main method of interaction, but already the first PIMD-user did not move her head left/right often enough, so this data could not be used as main input. Besides the lacking adaptability to the user’s abilities and interaction methods, the first concept was also not tailored enough to preferences of the user. Solving this technically could become quite hard. Therefore, we switched to a Wizard of Oz approach as it seemed to be a good first step to see whether the interactive ball in whatever way could be beneficial for people with PIMD.
Before designing the interaction we have spoken with several caregivers and a therapist about the project and did a literature research. We also watched play sessions, visited daily living environments, and were informed about a variety of activities for this user group. We believe this is essential in coming to a suitable interaction. However, no overgeneralization should take place: things seen for one user can not simply be transferred to another. Therefore, we tailored the interaction patterns of the ball in an iterative and individualized way. We continued to work closely together with caregivers, in order to analyze the user’s behavior and experience and to help improve the interactive system, during a first set of habituation sessions for each participant.
We chose for a ball as this is a general shape used often in traditional playful interactions, furthermore we saw one person from the target group—with relatively high capabilities— trying to push a big ball during an observational session. We think a big physical object has benefits over showing objects on a screen, as we expect people with visual disabilities to be more likely to be able to identify and follow this. As a start of the interaction we preferably use the movements of the user that were likely to occur, such as the rocking behavior of the upper body (see Figure 1), using their ‘vocalizations’ or the fiddling of their hands. After this (unconscious) initiation of interaction, the user has to learn the link over time between action and response, something we knew beforehand not all users will be capable of. Furthermore, we were not even sure whether some users would recognize the ball and its behavior. To increase chances for these kinds of recognition, sessions were held several times. To further improve chances of success we also tailored to the possible actions of the user, as well as the responses that needed to be stimulating enough for the user to make a cause-effect link more plausible. We added sound feedback as we knew that some people from the user group are more sensitive to this than merely visual feedback.
The exact interaction protocol depended on the user. More info on this protocol will be published elsewhere; here we only briefly describe the general way of interaction. In this protocol actions where described like, user action: leans to our left, ball: move ball to our left, sound: play sound type 1. For the implemented protocol for the longer term study we started from the premises of what could be expected from a user as well as what we designed that the interactive ball could do: move, make sounds, and change appearance using bright LEDs. This resulted in some links that might not be as intuitive as they could have been. We made the ball respond to upper body movement and/or focus of attention. The ball was not placed within reach of the user, to trigger them to show alertness farther away than their close encounters, as well as to make the ball less likely to bump against a person or to harm the participant in any other way. When the user moved his upper body, the ball was remote controlled by a facilitator to start to roll. The ball also made sounds when an attempt for interaction was made or if the ball was kept in the user’s visual focus for some time. Furthermore, for some users the ball moved from side to side when it was not been interacted with for some time. This was done in order to (re)gain the attention of the user. Sounds were also played to further grab the attention of the user, and some categories of sounds were removed or played more often depending on (dis)liking it. 4
For one of our outcome measures (the only one we discuss in this paper) we used computer vision techniques to automatically detect movement responses. This allowed us to make use of the benefits of automatic measurements without burdening the participants too much. For this target group it is important to consider downsides of for instance placing body worn sensors, as this might have detrimental effects for overly sensitive people. In our study design we also took into account an extended time for getting accustomed to a new situation.
We carried out a pilot with five people from the target group, followed by the final study with 9 participants. One of the researchers was present to control the ball, one other researcher was taking notes of noticeable and relevant behavior and (for a set number of sessions) take a structured interview, and one caregiver was present to help the participant if needed (e.g. one participant had phlegm that resulted in coughing severely and several times needed to be helped and calmed). We used three sessions to tailor the interaction protocol to each participant, and then had up to two more additional sessions, where these 3-5 sessions were used to get the user acquainted with the people present, the ball, and the room. We then used the last 5 sessions to do our measurements. We compared the movement during interaction with periods where the ball was not present; exact details are discussed in the other papers. We compared on a person-to-person basis and used the data only for descriptive statistics including whether they moved more, less, or showed about the same amount of movement. Beside these automatic measures there were also other measures for the final study which are outside the scope of this paper. 4.1
To have a measure for movement we used a fairly simple method. It is a slightly simpler
version of Motion Energy Analysis as used by Ramseyer & Tsacher [
We investigated the resulting measurements thoroughly. Figure 3 shows post-filtered results. We used Matlab R2012a to generate the graphs and filter out noise-polluted results, which we will discuss next. The meaning of the results regarding the outcome measure will be published elsewhere.
Using similar graphs we saw numerous peaks of measured movements for participants3 where we did not remember seeing this during the session. For instance, one participant slept through the entire session but still we saw several peaks in the measured 3 This measured movement is represented in amount of pixels that differed beyond a threshold, where we subtracted the average movement of the entire session of the day for representation purposes. movement4. This made abundantly clear that there was noise interfering with the measurements. Therefore, we tried to sync the measured frames with the video recordings and we saw several reasons for this occurring, we did not notice these aspects in the pilots. 4.3
One, the camera apparently made use of auto-focus. At certain points the camera would
focus differently and this resulted in measured peaks, when changing focus and often
a second or so later changing back to original focus, we hypothesize this could have
been related to facial recognition of the camera. Two, in a similar fashion we noticed
something that seemed like a color filter change, combined with our own experience
this seemed to occur when sunlight intensity changed (even though we equipped the
room with curtains), see Figure 4. Three, the camera sometimes shook resulting in
short peaks, this occurred when the ball was moved in a certain rhythm back and forth.
Fourth, at certain moments feet of people present entered the footage, we cropped the
images to prevent this as much as possible, see Figure 4. Fifth, although very limited
the reflections of the moving ball could be seen in one of the wheelchairs. Sixth, we
knew from previous work that the clothing of the day can influence the amount of
pixels changed (compare a checkered blouse with a black sweater). This mainly made
the results harder to compare between days, which was less of an issue with our study
4 In a normal population this sleeping participant might be considered an outlier and could be
withdrawn from analysis. For our main study, we kept this participant because such “outliers”
can in fact occur very regularly with this target group.
setup. Seventh, the wheelchair also moved due to the heavy movements of some of the
participants which might influence the results when looking only at the averages.
Although we did not anticipate any of this, to a certain amount our context thus violated
the static camera and light conditions prerequisites of MEA [
For our project we were motivated by being confronted with the issues that the target group had such limited possible leisure activities. The observations of the target group and conversations with caregivers helped us to clarify that for some people of the target group even a bit of movement increase can be of added value. Early on in the project, we realized that expecting statistical results over the generalized population, a population which although small in size is very heterogeneous, might be too much to expect. Even targeting a specific sub-group selected beforehand can be hard as it was not always predictable for which users it might or might not be beneficial. We tried to fit this heterogeneity also in our research approach, placing a lot of emphasis on analysis of individual cases. In short, we knew it would be a time consuming research where there would not be ‘a cure for all’ of a large population but that we could inspire others to also create more interactive leisure activities for this target group.
In our study design we took into account the variability within and between participants for this target group. This also had advantages to deal with some of the shortcomings of the measurement tool (e.g. impact of clothing changes on SMEA). Furthermore, we did a pilot study but refrained from in-depth analysis of the entire sessions (they were not an effect study after all), it is however good to mention that if we had done this analysis it might have prevented some of the issues (e.g. auto-focus) from occurring. We do believe an analysis of any ‘new’ automatic measurement tool should be done thoroughly as unexpected events can influence results. Although there were severe shortcomings of our approach it did give an interesting and useful result, so we still argue that automatic measures can be an effective tool.
However, in our case the measurement ‘errors’ could be related quite easily to outliers, but what happens if results are not recognizable so easily as outliers, would they be able to impact results in an unnoticeable way? Furthermore, when data is gathered over a even longer period of time, would it not be impossible to investigate these outliers one by one? In our case we knew that habituation for this target group was important. However, in many cases we (can) only measure persuasive systems over a short term. Furthermore, the ethical considerations for this target group and several health related issues, especially when measured over a longer time period, can also take up a long time5. Therefore, is it conceivable that systems can be beneficial for end-users and from a technological point of view be developed in a research setting, but are less likely to be researched as evaluations might become too complicated and time consuming? For this target group some are able to crawl others had a hard time even moving their head. It is important that the automatic measurement tool can deal with both. However, even if the tool can measure both, the same kind of data might have a different meaning from person to person in our specific context. How can we take into account such large difference in possibilities, and interpreting their data, between participants? In short we like to conclude: 1) that developing persuasive entertainment for people with PIMD is challenging but seems worthwhile to investigate, 2) that automatic measurements can be a useful tool for longer term measurements but that shortcomings should not be overlooked and might require a thorough analysis and manual filtering step.
Acknowledgements This research was supported by the Dutch national program COMMIT, and received additional funding of Dichterbij, the health care organization involved. We like to thank Wietske van Oorsouw and Petri Embregts of University of Tilburg, as well as Sophie Wintels, for the joint setting up and carrying out the study reported elsewhere and for the hours and hours of manual annotation. We also like to thank our professors Dirk Heylen and Vanessa Evers for their time and valuable input in the project, Kitt Engineering for developing the hardware for the ball for us. And most importantly all the participants and the involved family members, as well as all the caregivers and other employees at Dichterbij making time in their already very busy schedules, these are the people actually making this project possible.