DESIGN IMPLICATIONS FOR A UBIQUITOUS AMBIENT SOUND DISPLAY FOR THE DEAF

Abstract

Previous work already investigated the value of ambient sound visualizations for deaf and hearing-impaired people. Our work builds upon these results and specifically explores the applicability of the ceiling for such visualizations. Thus, we gathered design requirements based on a participatory design process including expert interviews, an online questionnaire as well as a design workshop at a local organization for deaf people. Results from the workshop showed that people highly approved the idea to use the ceiling for ambient visualizations. However, they also expressed a strong need for a supplementary traditional display to get more detailed information about occurring sounds. The implications that we derived in the present study build the basis for a prototype that we currently develop at our research group.

Keywords

• Peripheral displays
• ambient sounds
• deaf
• hearing impaired

1. Introduction

Matthews et al. (2005) identified different classes of ambient sounds that constantly provide us with important clues about our environment: serendipitous events (e.g. children playing in the next room), problematic things (e.g. fire alarm), and critical information (e.g. knocking on the door). Another study revealed that deaf people have difficulties to keep track of this kind of information. Assistive technologies exist that help them to stay aware of specific events, such as a telephone call or the doorbell. However there is no tool available that consistently provides them with visualisations of all sounds in an environment (Matthews et al., 2005) . For example, Matthews et al. (2005) found out in one of their interviews that a deaf couple once triggered the fire alarm while cooking and did not notice this until a hearing friend informed them. Deaf parents have difficulties to maintain awareness about their children’s activities (Clarke, 2001) . Such situations cause additional stress for deaf people (Clarke, 2001) .

As a new approach in this research field, we propose to use ambient large-scale projections for visualizing ambient sounds. Specifically, we evaluate the ceiling as applicable ambient large-scale display due to the following facts: • In a previous study deaf people stated to prefer large-sized displays for such applications (Matthews et al., 2005) . • Recent advances in technology promote the integration of ambient technologies into the home, although it might take another decade until the technology for such an application will actually be available on the market. • The ceiling has the advantage that it sits at the periphery of attention. Thus we exploit the fact that deaf people have enhanced visual attention to the periphery (Bavelier et al., 2000) . The ceiling also allows spatial arrangement of acoustic icons. • The ceiling is further always present and always available. This is an advantage over an ambient sound display application that runs on a desktop computer, since people move around and are not always near their desks (Matthews et al., 2006) .

2. Related Work

Ho-Ching et al. (2003) built two prototypes that supported both monitoring and notification of sounds. An evaluation showed that people preferred the prototype, which presented sounds as positional ripples. Matthews et al. (2005) conducted design interviews to guide the development of ambient sound displays. Based on their work, they formulated a set of visual design preferences and functional requirements. In a follow-up project Matthews et al. (2006) developed several functional prototypes, which they evaluated in different studies. The Single Icon application displayed recognized sounds as icons and unrecognized sounds as rings. Pitch and volume of unrecognized sounds were encoded through colour and number of rings. The Spectrograph with Icon application additionally displayed a black and white spectrograph. This spectrograph served as a footprint and aimed to help identifying sounds through a more detailed representation of volume and pitch. Both prototypes used the sound recognition system from Malkin et al. (2005) for sound identification.

Chatter1 is a table that reacts to sound frequencies with changing light patterns. Volume is visualized through brightness of the pattern. Deaf or hearing-impaired people can learn the connection between patterns and acoustic events, such as the ringing of a doorbell.

Commercial products that support notification of acoustic events include sensors that are attached to the doorbell, placed in front of the entrance, or located anywhere inside a room (e.g. for monitoring young children). Furthermore, specific electronic devices are available, such as alarm clocks or fire alarm systems (Hersh and Johnson, 2003) . Those devices typically either use flashlights or vibration or both to notify deaf people of an event.

3. Collecting Design Requirements

The design process of our prototype was guided by expert interviews, an online questionnaire, and finally by new insights resulting from a design workshop.

3.1 Expert interviews

The expert interviews were held at a local association for deaf and hearing-impaired people, called WITAF2, which is the German abbreviation for “Knowledge, information, tradition, news, and demands from and for deaf people”. The interview partners were an almoner, who has been working at WITAF for several years, and a technical assistant, who is responsible for installing assistive technologies at people’s homes. The purpose of the interviews was to develop a common understanding for the situation of deaf people and to gather initial design ideas. We therefore decided to use unstructured interviews.

An important issue that was raised by the almoner was that deaf people suffer additional stress due to the fact that they cannot notice acoustic events. We discussed the possibility of using the ceiling as an ambient display with both of them. Surprisingly, we received very different opinions. The technical assistant was concerned about the additional information overload. The almoner was very interested in introducing such an application to her clients. In an informal discussion after the interviews we discovered that this ambivalence might come from the age difference between the two experts. A conclusion of this insight was that especially deaf adolescents, who are already familiar with information technologies, would be interested in such an application.

The interviews also had the purpose to initiate a personal contact to the people from WITAF, which was a crucial factor for the success of the successive steps in our design process. 1 http://myweb.tiscali.co.uk/anna.hiltunen/chatter.htm 2 http://www.witaf.at

3.2 Questionnaire

The goal of the questionnaire was to collect information about assistive devices people currently use. We also hoped that we could verify the assumption that especially people familiar with new technologies, such as the Internet, would be interested in ambient sound displays. However, the methodological problem that we had to face was that we were only able to distribute the questionnaires over WITAF mailing lists. This means that we only reached people, who were already familiar with PCs and the Internet. Nevertheless, we were able to strengthen our assumption later on during the design workshop (see below).

Since we knew from other work (Chan 2003) as well as from the initial interviews that deaf people have difficulties to fill out long questionnaires, our questionnaire consisted of short closed questions. To further ensure its applicability for the designated context, it was developed in cooperation with the almoner from WITAF. In the first part we asked about devices deaf people actually use and whether they prefer vibration or flashlight for notification. In the second part we presented two sketches of ambient sound systems and asked whether they would be interested in such an application. The questionnaire was posted on WITAF mailing lists. From 40 people that were subscribed to the mailing lists, eight (4 male, 4 female) filled out a valid questionnaire.

The average age of the participants was 32.8 years (SD=10.5). All of the respondents used a mobile phone and a PC. All but one used a fax machine, three of them had a mobile phone that supported video transmitting, and five of them used a web cam with their PC. All participants stated that they used a doorbell sensor and a special alarm clock (four with flash lights, two with vibration, and two with a combination of both). Four participants had children and three of them used a baby-monitoring device. The generally preferred notification method was flashlights (4), followed by light signals (2) and vibration (1).

The average acceptance rate of the first sketch (a screen showing a map with icons for occurring sounds) was 4 (SD=1) on a scale from 1 to 5 with 5 being the best. The second sketch (a projection of icons onto the ceiling) was rated 3.3 (SD=1.8). One respondent stated that he would like to see a presence awareness monitor for people in his surrounding incorporated into the device from the first sketch. Opinions against the concept drafts were concerns about high electricity consumption (both sketches) and installation costs as well as aesthetical considerations (second sketch only).

3.3 Design workshop

The goal of the design workshop was to collect information about the target audience and to develop design ideas in a participatory design process. It was held during one of WITAF’s club evenings (Figure 1, left). Ten participants (6 male, 4 female) aged between 22 and 30 years took part in the workshop. According to the analysis of the background questionnaires, all of them had experience with PCs and were interested in new technologies. Furthermore, other deaf people that attended the club evening continuously passed by the sessions and joined in the discussions. We did not collect background data from those people.

The workshop was structured into three parts. The first part was a demonstration of a simple application that projected a visualisation onto the ceiling, which reacted on acoustic input. In the second part we presented drafts for possible applications, which where also projected onto the ceiling. Participants discussed the sketches in the group and afterwards rated each application individually by attaching Post-it notes to the corresponding design sketch printed on a paper. In the third part participants had to answer design questions and were encouraged to sketch design ideas for ceiling applications. (Figure 1, right) The sound reactive demo application was appreciated with great interest and curiosity. Participants spontaneously started interacting with the prototype. They clapped their hands, screamed, and did all other sorts of things to produce noise and watched the corresponding patterns that were displayed on the ceiling.

Results and comments for the design sketches gathered in the second part of the workshop can be found in Table 1 (Post-it notes that we could not interpret were omitted). During the discussions it was revealed that icons were slightly preferred over sound ripples, although we could not observe this trend in the results from the ratings. Location of sounds was rated to be very important. Of the presented design concepts, participants liked the iconic representation of sounds within an overview map of the flat or house best. During the presentations they also discussed other possible implementations of novel systems to provide sound awareness. The ideas ranged from floor projections (in shopping malls), over displays hanging at the wall, to augmented reality glasses that superimpose the real environment with virtual traces of sounds. In the final discussion of this part participants agreed that they would like to have a combination of an ambient ceiling projection and a display that hangs on the wall like a picture. The display should provide an overview of acoustic events in the entire flat or house. One of the participants said that he would appreciate a multifunction ceiling application to combine the functionality of all different devices he currently uses in one central system.

Design sketch

Sound ripples

Patterns

Icons Positional sound ripples

Positional icons Sound ripples in map Icons in map 3

1

Comments from the participants 4 Difficult to recognize, does not attract

attention 6 Good visual appearance, difficult to

memorize 6 Too small 7 Supports orientation and localization of

sounds 5 Very practical 9 10 Very clear due to icons In the third and final part of the workshop, we asked participants to form three groups. Each group was handed out a set of cards with design questions and tasks. The questions aimed to reveal their requirements for ceiling applications in general and for ambient sound visualisation in detail. The design tasks encouraged them to sketch icons for sounds and possible arrangements of a ceiling display in their home. We incorporated the results from this session into our final concept. An interesting result was that people were not interested in ambient sounds emitted from the street, their neighbours’ flats or some home appliances, such as the dishwasher. This contradicts some of the results of Matthews et al. (2005) . Other findings were confirmed, like the fact that deaf people sometimes forget to turn off their appliances, since they lack the acoustic information. Examples for this included cookers and water taps, another group mentioned the washing machine. Two groups mentioned the following issues as important information, which they would like to get informed about by a ceiling display: mobile phone, fax machine, and baby monitoring. Other (less important) issues were weather conditions, traffic conditions, and calendar data. One group said that they would also like to be able to call people, who are located in other rooms, via the ceiling. Results from this session were probably biased due to the fact that all participants used their desktop computers on a regular basis. They therefore adhered to the metaphors and interaction paradigms known from desktop computer environments. This phenomenon was strongly supported by some of the results. For example two groups mentioned that they would like to have a screen saver view on their ceiling in case there is no other activity going on. The icons sketched by one group (Figure 2, left) also feature a strong desktop-like character. We also asked the participants to point out where they would like to have ceiling displays inside their own flats. However, we did not receive meaningful answers to this question. Instead they produced sketches to show how they thought a ceiling display should be arranged inside a room. They further told us that they would like to see similar displays in public places, like hospitals, train stations, airports, and subways.

During the workshop we were able to confirm the assumption that older people would not be interested in an ambient sound display. Some older WITAF members, who spontaneously joined in the discussions for a short time, explicitly stated that they did not need such an application. “My dog never barks and I don’t care about the noise inside my room,” one participant stated. In contrast the workshop participants were very enthusiastic about the idea and presented concepts.

4. Design Implications

Matthews et al. (2005) already identified a number of general functional requirements for ambient sound applications: identify what sound occurred, view a history of displayed sounds, customize the information that is shown, and determine the accuracy of displayed information.

All requirements, except for the second one, were confirmed in our study. Further research will be necessary to investigate this variance probably caused by individual preferences of different users. Thus, the third requirement is very crucial.

Additionally to these requirements we derived the following implications that are specifically relevant for an application that is designed for an ambient technologies environment: • Use either the entire ceiling or multiple areas as projection surface • Provide a low-level awareness of sounds through the ceiling display • Determine the location of occurring sounds • Use aesthetic visualizations for the ceiling display • Provide a second traditional display for higher-level sound awareness that features an overview of the entire living space (e.g. a desktop screen fixed on a wall) • Use icons for sound representation (especially on the additional display) • Show location of other people in the environment on the map The ceiling projection therefore has to be developed according to the guidelines of ambient display design. It should represent information about location and source of sounds in a peripheral, glanceable and aesthetic way. The additional display acts as a primary display that provides the user with the same information in a higher resolution once he/she was attracted to the information by the ceiling display.

5. Conclusion and Future Work

In this paper we presented a survey and evaluation of design requirements for applications to visualize ambient sounds. As a methodological framework we used interviews, an online questionnaire, and organized a design workshop. The main results from the workshop were: (1) participants were highly interested in visualisations of ambient sounds; (2) icons were preferred to sound ripples; (3) location of sounds was rated to be very important; (4) overall they voted for a combination of an ambient ceiling projection and a display that hangs on the wall like a picture and provides an overview of the entire flat or house.

Our experiences from the design workshop showed that it is a big challenge to develop applications that are based on ambient technologies in a participatory design process. The reason for this is that people tend to adhere to traditional interaction concepts known from desktop computers. Confronting them with simple prototypes that demonstrate the possibilities of new technologies helps, but it is sometimes difficult or impossible to prototype such applications with out-of-the-box hardware. For example the application that we suggest should run on a display that spans the entire ceiling, however, the projector that we used for the design workshop only illuminated a fraction of the room’s ceiling.

We are currently working on a functional prototype that implements the requirements collected in the present study. The prototype uses multiple microphones for sound location, which is based on Scott and Dragovic’s (2005) work. For sound identification we will use automatic sound classification (Temko et al., 2006) . We plan to evaluate this application during one of WITAF’s club evenings. Eventually the goal is to evaluate the application in a real context over a longer period. Acknowledgements: Thanks to Valerie Clarke, our contact at WITAF, who helped us organizing the workshop and did the sign language interpretation during the workshop. We also thank all the participants from the questionnaires and the design workshop at WITAF.

References

Bavelier , D. , A. Tomann , C. Hutton , T. Mitchell, D. Corina , G. Liu and H. and Neville ( 2000 ). Visual Attention to the periphery is enhanced in congenitally deaf individuals , The Journal of Neuroscience , vol. 20 : RC93 , pp. 1 - 6 .

Chan , P. ( 2002 ). An analysis and measurement of the needs of the deaf regarding assistive communication devices , Thesis Research Paper, Parsons MFA Design & Technology.

Clarke , V. ( 2001 ). Unerhört (unheard) , In German, Ziel-Verlag.

Hersh , M.A. and M. A . Johnson (Eds.) ( 2003 ), Assistive Technology for the Hearing Impaired, Deafblind and Deaf, Springer Verlag.

Ho-Ching , F.W.-L. , J. Mankoff , and J.A. Landay ( 2003 ). Can you see what I hear? the design and evaluation of a peripheral sound display for the deaf , Proc. of the SIGCHI conference on Human factors in computing systems , pp. 161 - 168 .

Malkin , R. , D. Macho and A . Temko ( 2005 ). First evaluation of acoustic event classification systems in the CHIL project , Proc. of 2005 Workshop on Hands-Free Speech Communication and Microphone Arrays.

Matthews , T. , J. Fong and J. Mankoff ( 2005 ). Visualizing non-speech sounds for the deaf , Proc. of ACM SIGACCESS Conference on Computers and Accessibility , pp. 52 - 59 .

Matthews , T. , J. Fong , F.W.-L. Ho-Ching and J . Mankoff ( 2006 ). Evaluating non-speech sound visualizations for the deaf , Behaviour & Information Technology , vol. 25 ( 4 ), pp. 333 - 351 .

Scott , J. and B. Dragovic ( 2005 ). Audio location: accurate low-cost location sensing , Proc. of the 3rd International Conference on Pervasive Computing , pp. 1 - 18 .

Temko , A. , R. Malkin , C. Zieger , D. Macho , C. Nadeu and M. Omologo ( 2006 ). CLEAR evaluation of acoustic event detection and classification systems , Proc. of CLEAR Evaluation Workshop.