The Challenge of Evaluating Situated Display based Technology Interventions Designed to Foster ‘Sense of Community’ Keith Cheverst, Nick Taylor, Mark Rouncefield Areti Galani and Christian Kray Computing Department ICCHS and School of Computing Science Lancaster University, Lancaster, UK Newcastle University, Newcastle upon Tyne, UK {k.cheverst, n.taylor3, m.rouncefield@ {areti.galani,c.kray}@ncl.ac.uk lancaster.ac.uk ABSTRACT work is provided in [14] with one of the first systems of this In this paper we discuss the obdurate problems associated genre being GroupCast [10]. Typically such systems focus with evaluating the extent to which technological on highlighting the technical difficulty associated with interventions – in particular those based on mobile and implementing the system or the methods used to ensure ubiquitous technologies – can be judged to have „improved appropriate and well-informed design. However, in addition a sense of community ‟ in their given deployment settings. to the difficulty of successfully designing and implementing We report on experiences gained from several deployments systems, the evaluation of their „success‟, i.e. their ability to of ubiquitous systems that share this design goal, and foster and support a sense of community, is also a analyze common issues we observed during real life use of challenging problem, due to a number of issues. For these systems. Based on these we discuss some of the key example, when deploying technologies to support challenges for evaluating ubiquitous systems of this genre. community it is likely that social practices will shift in order to accommodate the new technology. Furthermore, it Author Keywords is likely that the technology will be tailored by its users, Technological intervention, mobile and ubiquitous sometimes in unanticipated ways (i.e. through computing, CSCW, long-term deployment, evaluation, appropriation) to accommodate the social practices it is methods, community, situated displays. intended to support. For example, technology can reshape notions of space and proximity and thus the boundaries of ACM Classification Keywords 'community‟, re-conceptualizing what it means to be local, H5.m. Information interfaces and presentation (e.g., HCI): connected etc. Hence, community is an achieved social Miscellaneous. construct, a „persuasion‟, of mutual ties, orientations and obligations, pointing to the ability of technology to reshape INTRODUCTION and redefine how people see themselves [13]. Our research is interested in the design of technical systems that may prove useful in promoting or 'affording' some One of the difficulties of evaluating how well a given sense of community. McMillan and Chavis [11] identify technological intervention may support notions of four inter-related elements associated with sense of community is that the effect of the intervention is community: dependent on the interaction between a combination of technologies and their affordances (including those brought i. membership, about through the placement of the technologies) and ii. influence, particular communities and their dynamics. Furthermore, the evaluation techniques themselves must adapt to these iii. integration and fulfillment of needs, and, dynamics, evolving alongside the system. iv. shared emotional connection. In [15] we discuss the need to consider the following A number of technical systems based around ubicomp factors when designing technologies to support notions of technologies, most notably situated displays, have recently community: been developed and deployed with the intention of 1. membership - recognisable members and membership supporting sense of community. A good overview of this categories, allied with recognisable boundaries Copyright is held by the authors. 2. identity and representation - how people can represent UbiComp '08 Workshop W2 -- Ubiquitous Systems Evaluation (USE '08) themselves and manage their 'identities' September 21st, 2008 This position paper is not an official publication of UbiComp '08. 3. managing spatial relations - need to manage spatial relations to integrate the real and the virtual 1 4. rhythms - the highly predictable rhythm of everyday SYSTEM DEPLOYMENTS activity sets the grounds for shared expectations and We have experienced „community‟ use with several of our comprehension of behaviour - successful communities deployed systems based around „situated‟ displays. For carry intelligible rhythms of interaction and awareness example, with the Hermes office door display system (that - which vary according to the community and is linked enabled office owners to post awareness related messages to issues of awareness and 'sense of place'. on digital displays situated outside their office) we describe in [2] how usage of the system was considered by many 5. community development - the community should be users as directly relating to notions of community, e.g. one able to reflect and learn from experience, to develop door display owner made the following comment when „robust sociality' asked why he used the system: 6. history and change - the ability to develop a history “there is a community associated with my doorplate, you through recording and archiving various interactions know people have to be able to get to my doorplate, and The remainder of this paper is structured as follows. In that probably makes them one of the staff or colleagues, section two we describe our general approach towards the and that affects what information I could put on there and I design, deployment and evaluation of technology don‟t want burglar Bill with his web browser to go – oh interventions (where sense of community is at least one of look [name]‟s in such-and-such I‟ll go and burgle his house the aims) in a range of settings. Next, in section three, we now.” summarise two of our current deployments for which we In the following sub-sections we describe two of our wish to evaluate their effectiveness in supporting sense of current technology deployments that are undergoing community. These two deployments comprise the Wray evaluation and which were designed to support notions of Photo Display, a touch screen based interactive system, community. which is situated in the Post Office of a rural village in the North of England, and the Campus Coffee Display, a wall- The Wray Photo Display mounted broadcasting screen, which is situated in a café at The Wray Photo Display [16] is deployed in the Post Office the intersection of Newcastle University‟s campus and the of a rural village situated in the North of England. The city‟s main shopping area. In this respect both systems are system enables members of the village to post photos (or located within the activity zone of established local short video clips) to be shown on the display and to create communities and visitors to the area. Finally, we discuss the and moderate their own photo categories. The photo display pertinent issues that we have experienced when considering was conceived as a technology probe and has run the evaluation of these systems. continuously (capturing log data) in its current location (see figure 1 below) since October 2006. APPROACH It is apparent from related literature and our own research that it is essential to understand the social and physical richness of a given setting in order to avoid inappropriate design. Consequently, our approach draws from a range of approaches including ethnographic studies, use of cultural and technology probes [6], focus groups and design workshops. We have investigated several settings in the course of our studies including Lancaster University campus, a public café, and domestic settings such as family homes and residential care facilities. By using a range of settings we aim to increase our confidence in the generality of our findings. Our methodology is iterative: observe, design and deploy, Figure 1. The Wray Photo Display situated in the Village Post observe etc., where these stages are closely coupled and all Office. The Comments Book can be seen just to the right of the hold key (technical and practical) challenges. display. Our general approach is one of „co-realisation‟ [5] whereby In order to evaluate the usability and usefulness of the technical modification is rooted in ongoing ethnographic system we have held a number of participatory design study. The evaluation approach, therefore, both informs and workshops and focus groups. However, perhaps the most is being informed by the evolving character of the system to useful single method for obtaining qualitative feedback reflect the dynamic relationship between the system and its regarding the system has been via a comments book which socio-spatial context. has been placed next to the display since its first deployment. This book has enabled both members of the village and visitors to the village to express their opinions regarding the display and its content. To date over 60 individual comments have been left in the comments book while waiting at the counter. Nevertheless, the display was but suggestions for additional functionality have also been perceived as being beneficial as a reminder about upcoming left via e-mail. A page from the comments book containing local events and complementary to other similar community a comment relating to issues of community is shown in resources, such as the weekly newsletter. The slow pace of figure 2. the presentation was also positively received as being in line with the general „feel‟ and use of the café. In the course of the focus study we discussed with the participants three alternative designs of a more interactive system that would enable customers to interact with the display through their mobile phone. The proposed interactive features would provide a means for visitors to the café to comment either on the cultural events currently being shown on the screen, or on objects exhibited at nearby museums, or on user-defined topics. Feedback from the focus study indicated that, although the public nature of the display might serve well the promotion of community activities, the ownership of the content, its management, and the protocols of content contribution (including the interaction mechanisms) would be difficult to negotiate in such a socially and politically diverse environment. Figure 2. A Sample Page from the Wray Photo Display Comments Book. The last comment on this page reads: “What a superb idea, especially for those that are new to the village, and a delight for those who were born here and to go down memory lane...” It is interesting to note that this comment speaks of notions of membership, identity and history that were introduced Figure 3. The Campus Coffee display ‘in the wild’. earlier. Indeed the most popular category of photos viewed Regarding this latter finding, we return to the comment on the display is that of historical photos. made in the introduction, and the fact that here what is Campus Coffee being evaluated is the product of both setting and The Campus Coffee system at a local café (see figure 3) has technology. been running continuously for about two years now [7,8]. It provides information about upcoming cultural events in the ISSUES quarter of the city where the café is located. The initial In applying our approach to these deployed systems, we version of the system delivered content updated by the have come across a series of recurring issues, which we researchers and was designed to be non-interactive and discuss in this section. slow-paced. As a new addition to other modes of local How long does a deployment need to be in place? information in the café, it functions as a low-key technology probe. Both the Wray Photo Display and Campus Coffee systems have been deployed for relatively long periods of time – In order to assess the customers‟ perception of the system especially in the context of typical ubicomp systems. as a source of local information and to look into options for However, the question remains: how long does a further community engagement through the incorporation deployment need to be in place before it can sensibly be of interactive features, we conducted brief in-situ evaluated against success criteria based on improved questionnaires, observations and focus group sessions. In community and coordination in the setting? A key element line with previous findings, users most frequently classified of our research methodology is the use of substantial their use of the system as opportunistic, i.e. glancing at it deployed installations. The long term use of novel 3 technologies, especially their collaborative and community display technologies and have been deployed for relatively effects, cannot be deeply understood through short-term long periods of time and received daily use. The Wray experiments or „toy‟ installations. This development and Photo Display system has certainly received positive deployment enables longitudinal studies as well as being a comments from members of the community; however, technology demonstrator for dissemination and inspiration. questions over its „inclusivity‟ still remain. With the Campus Coffee system it has been interesting to observe What are appropriate techniques for evaluating technology the cost/benefit analysis that has led participants of a focus probes with respect to community? study group to favor calm/controlled content presentation We have utilized both qualitative and quantitative measures over potential haphazard community generated content. but to-date it is the use of qualitative methods that have Clearly part of the cost/benefit analysis taking place in this yielded most insight. One problem with the use of case is informed by the participants‟ use of the café in the quantitative measures based on log analysis, for example, is first place. Therefore, it highlights strongly the fact that that it is difficult to produce figures on how many different with the technology interventions discussed in this paper, members of a community view the content (not least how what is being evaluated is the product of both setting and they feel about the content). With the Wray system, we did technology – and this reveals the emphasis in situated not wish for the interaction design to require viewers of the displays. Furthermore, it indicates that an evaluative content to log themselves in and out of the system, as is approach that would investigate the correlation between often the case with similar systems [10,4]. There is the community dynamics and system usage patters and possibility of exploring the use of monitoring devices such perceptions might be particularly helpful in the design of as web cams but these, of course, introduce numerous and sustainable community-centered technology. difficult privacy and control issues, see [12] for an initial As part of our future work, we hope to extend our use of discussion on this topic. We have also highlighted qualitative evaluation methods but also explore further the additional complexity added to this issue by the need to potential of more quantitative methods, such as the use of adapt our evaluation approach to individual communities „Sense of Community Index‟ developed from the field of and technologies. psychology [3, 9]. We also hope to explore how to design How to introduce the system to the community? and evaluate technology interventions to support a sense of community in further different and (again difficult to study) The Wray Photo Display was introduced as a working sensitive settings, including rural townships in South interactive system, and has evolved over time in response to Africa. user feedback. With the Campus Coffee system we took a slightly more conservative approach by repurposing an ACKNOWLEDGMENTS existing non-interactive and very ambient system with the This work is supported by the EPSRC funded CASIDE scope to introduce interactive aspects in response to user project (grant ref: EP/C005589) and a Microsoft Research consultation. While both systems are relatively similar in PhD Scholarship. We would like to thank the villagers of the function they provide, specifically the delivery of Wray and the staff and visitors of Campus Coffee as well as community/locale related content, the reaction to them has the student union of Newcastle University. been quite different. In particular, the interactive features associated with the Wray Photo Display have been received REFERENCES enthusiastically, but with the Campus Coffee system the 1. Anderson, R.J., and Sharrock W.W. (1993) 'Can suggestion of altering the design concept of the existing Organisations Afford Knowledge', in Journal of technology deployment to one in which a great degree of Computer Supported Cooperative Work (JCSCW) Vol community-generated content could be entered and 1, No. 3, pp. 143-161. 1993. displayed received negative reaction. It is interesting to speculate on how the Campus Coffee deployment would 2. Cheverst, K., A. Dix, D. Fitton C. Graham, and M. have been received if the initial deployment had been based Rouncefield, (2008) Situatedness of Awareness on this suggested design concept. The implication for Information: impact on the design and usage of evaluation being that the way a technology intervention is awareness systems, Book chapter to appear in: introduced can have a significant impact on the adoption Awareness Systems: Advances in theory, methodology and appropriation of the technology (to support sense of and design. Springer HCI Series – Eds: Panos community). Markopoulos and Boris de Ruyter - Springer. 2008. 3. Chipuer, H. M., & Pretty, G. M. H. (1999). A review of CONCLUDING REMARKS AND FUTURE WORK the Sense of Community Index: Current uses, factor In this paper we have discussed the difficult issue of how to structure, reliability, and further development. Journal evaluate the success of technology interventions that have of Community Psychology, 27(6), 643-658. 1999. „supporting notions of community‟ as their design goal. The 4. Grasso, A. Roulland F. and Snowdon, D. (2006) two systems presented in this paper, which both share the Informing the community: The roles of interactive aforementioned design goal, are based around situated public displays in comparable settings. In Purcell, P. (ed.) Networked neighborhoods. Springer, 373-395. supporting local and remote mobile phone interaction 2006. in situated public display Deployments‟, submitted to 5. Hartswood, M., Procter, R., Slack, R., Voß, A., the International Journal of Mobile Human Computer Buscher, M., Rouncefield, M., and Rouchy, P. (2002). Interaction (IJMHCI): special issue on Advances in Co-realisation: Towards a Principled Synthesis of Evaluating Mobile and Ubiquitous Systems. 2008. Ethnomethodology and Participatory Design. 13. Mynatt, E.D., V.L. O„Day, A. Adler, and M. Ito, Scandinavian Journal of Information Systems, 14(2). 9- (1998) „Network communities: Something old, 30. 2002. something new, something borrowed...‟, Computer Supported Cooperative Work, 7(1-2), 123–156, 1998. 6. Hutchinson, H., Mackay, W., Westerlund, B., Bederson, B. B., Druin, A., Plaisant, C., Beaudouin- 14. O‟Hara, K., M. Perry, et al (2003) Public and Situated Lafon, M., Conversy, S., Evans, H., Hansen, H., Displays: Social and Interactional aspects of shared Roussel, N. and Eiderbäck, B. (2003) Technology display technologies, Kluwer. ISBN 1-4020-1677-8. probes: inspiring design for and with families. In Proc. 2003. of the SIGCHI Conference on Human Factors in 15. Rouncefield, M., K. Cheverst, A. Dix, M. Gibbs and C. Computing Systems (CHI 03), (Ft. Lauderdale, Florida, Graham, (2005) “Workshop Position Paper: Apr. 05-10, 2003), ACM Press, 2003, 17-24. 2003. Understanding space, place and 'community'”, in Proc 7. Kray, C., Galani, A., and Cheverst, K. (2007) Engaging of Interact ‟05 workshop on „Space, Place and with Cultural Content on Ambient Displays. Urban Experience in HCI‟, see: Screens 2007, Manchester, UK. 2007. http://www.infosci.cornell.edu/place/, Sept 2005. 8. Kray, C., Galani, A. and Rohs, M. (2008). Facilitating 16. Taylor, N., Cheverst, K., Dix, A., Race, N. Fitton D., Opportunistic Interaction with Ambient Displays, in Rouncefield, M. and Graham, C. (2007). Probing Workshop on Designing and Evaluating Mobile Phone- Communities: Study of a Village Photo Display. In Based Interaction with Public Displays at CHI 2008. Proc OZCHI 2007. 9. Long, D.A., & Perkins, D.D. (2003). Confirmatory 17. Taylor, N., Cheverst, K., Rouncefield, M. and Factor Analysis of the Sense of Community Index and Shahram, S. (2008) Encouraging Community Spirit Development of a Brief SCI. Journal of Community with Situated Displays, in Proc. of AISB International Psychology, 31, 279-296. 2003. Symposium on Persuasive Technology, University Aberdeen, April 2008. 10. McCarthy, J. F., Costa, T. J., and Liongosari, E. S. (2001). UniCast, OutCast & GroupCast: Three Steps Toward Ubiquitous, Peripheral Displays. Proc 3rd international Conference on Ubiquitous Computing, pp. 332-345. 2001. 11. McMillan, D.W., and Chavis, D.M. (1986). Sense of community: A definition and theory, p. 16. 1986. 12. Müller, J., K. Cheverst, D. Fitton, N. Taylor, O. Paczkowski, A. Krüger, (2008) „Experiences of 5