Ambient Façades Bernhard Wally Alois Ferscha Studio Pervasive Computing Applications Department of Pervasive Computing Research Studios Austria Johannes Kepler University Linz Thurngasse 8/16, A-1090 Vienna Altenberger Strasse 69, A-4040 Linz bernhard.wally@researchstudio.at ferscha@pervasive.jku.at ABSTRACT styles but concentrate on the visualization on data on fa- Public spaces get increasingly equipped with displays in çades. terms of shopping window plasma screens, electronic ad- RELATED WORK vertisements at the point of sale, kiosk systems at points of Ambient displays have been thoroughly discussed within interest, etc. While this trend enables numerous applica- the last decade, starting with early instantiations as physical tions in the pervasive display systems domain, it also has displays in the late 90s, characterizing ambient displays as effects on how people perceive urban environments. In this entities that “present information within a space through work we describe the concept, implementation and first subtle changes in light, sound, and movement, which can experiences from a real life setup of an ambient façades be processed in the background of awareness” [1]. Even framework expanding the idea of public displays to façades though some of the concepts have been proposed earlier, of arbitrary buildings without modifications on the build- the ambientROOM and two ambient fixtures have been ings themselves. With such a framework it is possible to presented in [1] and [2] describing indoor mounted displays integrate information into buildings in a very unobtrusive comprising light, sound, airflow and physical motion as the way and without interference with the building fabric. ambient actuators. In [3] the concept of ambient media has Keywords been broadened to “the use of our surroundings for infor- Ambient Displays, Content Adaption, Simulation, Public mation display”, which represents a key concept of what Display Systems we think of ambient displays: integration into our lives by either imitating commonplace objects or by extending ex- INTRODUCTION isting objects with somewhat smart behavior. Public displays are being increasingly used for displaying diverse information, including corporate propaganda at in- By specifying different zones of interaction (ambient, noti- store installations, advertisements at the point of sale and fication, interactive) a hybrid approach is prosecuted: de- location-aware information at points of interest. In Vienna, pending on the distance of a prospective user to the ambient the headquarter of the UNIQA insurance company, the display (the Hello.Wall) the type of interaction is deter- UNIQA Tower, has been covered with more than 180.000 mined [4, 5]: in the ambient zone the display shows general LEDs that are controlled based on video signals with 25 information about the overall level of activity, number of frames per second [18, 19]. people in the building, etc. In the notification zone the am- bient display reacts on the physical presence of a specific We believe that public displays can be perfectly used as person and provides means for explicit interaction with the visual interfaces for ambient information systems by lever- ambient display over a handheld device. In the interaction aging the ever increasing availability of such displays and zone the user can interact with the display at a very low one of the most interesting features of ambient displays: level and allows for playful and narrative interactions. information hiding. Depending on the level of abstraction, the information depicted in ambient displays can be under- An extension of this concept is presented in [8] where the stood by almost anyone passing by or it can be revealed to three zones are interpreted as four phases of interaction informed people only – uninitiated people just see images, (ordered from far way to close): ambient display, implicit icons, figures, etc. interaction, subtle interaction and personal interaction. The basic idea is that the ambient display resembles a common Implicit and explicit interaction metaphors and techniques context that should not be destabilized by the other phases. have already been discussed in the literature and even Transitioning from one phase to another should be very though we don’t believe that this topic is solved (on the smooth and happen only if certain “interruptibility” is de- contrary – feasible solutions still need to be invented), we tected. do not attempt to give an answer on specific interaction Regarding the type of visualization within ambient displays an interesting concept has been presented in [12], using particle systems, as they are able to “accurately portray Copyright © 2008 for the individual papers by the papers' complex data with breadth, depth, and elegance”. Particle authors. Copying permitted for private and academic systems seem chaotic and incomprehensible in the first purposes. Re-publication of material from this volume place but can be rich in information, if used with caution. requires permission by the copyright owners. We, too, believe there is a certain power within particle systems as they can deliver information extracted from the only informed people know about the informative value of single particles and a particle system has an overall appear- such displays. ance (shape, volume, etc.) that can unveil even more infor- Users’ experiences with an at-home ambient display have mation. The ambient display framework described in this been presented in [7] with the CareNet display which sup- work also makes use of the low-level and high-level state- ports an ambient and an interactive mode. Situated in the ments of a high number of objects on an ambient display. field of elder care it was shown that people with different [16] shows a possible solution for displaying text in ambi- roles used the display in different ways: basically, the less ent displays in an aesthetically pleasing way by using ki- the people were integrated into the care-process, the more netic typography (animated text) for displaying e-mail mes- often they actively used the display (interactively), while sages in the AmbientMailer system. This work is interest- seriously dedicated people used the display as ambient in- ing, as (especially high throughput) textual displays often formation system. lack aesthetic emphasis [9]. In [13] the success of ambient displays is identified as the In [10] a general purpose software framework for informa- combination of effectiveness in promoting awareness and tive art display systems is presented and some general as- the level of enjoyment in the users. This statement is de- pects of typical ambient displays are depicted, including rived after observing users and installations of four differ- themes, symbols and connotations. On the basis of real ent ambient information systems of both tangible and (ab- paintings, methods for integrating information therein are stract) 2D display type. proposed and implemented in the peripheral display frame- In [14] a taxonomy for ambient displays is proposed com- work. Subsequent research led to the proposal of more prising a set of design dimensions that can be applied to the user-oriented, participatory design process for ambient dis- various systems and allow a detailed classification. With plays [11], by letting the user decide on the specific theme the 19 projects already included in their taxonomy, a ten- a peripheral display is operated at. Different elements of dency to private, visual and highly abstract displays has various artworks are manipulated to resemble sensor data been determined. However, we believe the number of pub- or abstract context information thus leaving the decision for lic ambient displays is going to increase with the rise of the concrete piece of painting used for displaying ambient public displays in general. information to the user. A very critical look at public displays is taken in [15] One of the rather seldom seen examples of large public where large ambient displays in public settings have been displays is presented in [17], explaining a detailed observa- observed regarding their use practices. It is stated, that tion of the multi-touch display called City Wall. While the large public displays are not necessarily eye-catching and emphasis of this project lies on the multi-user interaction appealing, but that glancing and attention is a rather com- possibilities, it also shows some interesting aspects of how plex process. One of the key statements is that “people people approach public displays. Depending on the current make extremely rapid decisions about the value and rele- usage of the display, people need to wait for a free slot if vance of large display content”, devaluating content that too many people are interacting already, or they can start takes more then a few brief seconds to absorb. Also the interacting immediately if nobody is using the display. The displayed format is very important for the perception: video empiric data shows however, that there are usually at least is more attractive than text, animated text or still images. two steps involved: (1) noticing that there is a display, (2) interacting with the display. One conclusion of [17] is that Regarding these findings of previous work, we propose the “City Wall’s large physical size appeared to support mak- virtual façade framework for using suitable façades of ing interactions visible”. During eight days of operation buildings as solid basis for ambient information display. 1199 people interacted with the system. VIRTUAL FAÇADE FRAMEWORK Evaluation of Ambient Information Systems Examining façades as hosts for ambient displays is a very Regarding the evaluation of ambient displays, several ap- exciting thing, as the discrepancy between private data and proaches have been presented, such as a method to evaluate public accessibility is very high. Nevertheless, the aesthet- the comprehension of such displays [6]: it is argued that ics of fascinating buildings can offer a great set of struc- there are three levels of comprehension, each being a pre- tures “to lean on” (cf. Figure 1). requisite of the next: 1. That information is visualized 2. What kind of visualization is visualized 3. How the information is visualized The author emphasizes that it is important to consider the first two steps in the system design process and not start (blindly) at level 3 [6]. We believe however, that some set- tings, especially when involving public displays, single or even all three steps are not explained on purpose, so that • Content management: A content management system supports the integration of different resources (images, videos, streams) at runtime and provides a way to de- fine the position and shape of structures and ornaments of the façade to project on. For better results, the defini- tion should take place on-site, when projection distance and angle are known. Additionally, the support for on- site structure definition paves the way for automated Figure 1: Interesting features of a façade include borders of mechanisms, e.g. via a camera based system driving windows, various areas (separated by different colors, edge detection or other image processing algorithms. shapes, etc.) and ornamentation. Aside from these requirements we also had a picture in mind of what we would like to achieve. A relatively coarse illustration thereof is depicted in Figure 2 and Figure 3. Purpose In order to be able to support future development of façades as displays in combination with the ambient display meta- phor we decided to implement a robust framework as a basis for further ideas and implementations. We enunciated the requirements for the framework very roughly, as we wanted to narrow the choice of technical (a) (b) (c) solutions as less as possible: Figure 2: The façade of the MuseumsQuartier in Vi- • Text: There might be a need to display text of any size, enna/Austria to project on (a), an automated structure de- font type and color. However, with regards to ambient tection algorithm, such as Difference of Gaussians (b), and displays, text is usually avoided in favor of graphical the final fragmentation into separate regions, using e.g. solutions – thus it is a minor requirement. Flood Fill algorithm (c). • Still images: Support for embedding images into the visualization including scaling functions (each axis in- dependently) and, of course, free positioning. • Moving images: The framework should be able to ren- der videos and support both live camera streams and produced videos: • Live camera streams: Since our first façade was to (a) (b) (c) be the one of the Theatre Linz, we opted to integrate Figure 3: The façade of the Theatre Linz (a), after edge the possibility to render live camera streams to the detection (b), with detected regions (c). façade. This thought was driven by the idea to pre- sent the current action on stage simultaneously out- System Architecture side. Based on our visions and derived requirements we decided on a simple system architecture comprising a software • Produced videos: In addition to live video, our sys- framework running on a PC which renders the visuals to a tem should support readily available videos in order projector system and receives data from several resources to visualize perfectly pre-arranged content and se- as well as user input for the content management system. A lected scenes. Also, in case of a live camera failure, rough system architecture is illustrated in Figure 4. locally available videos could be applied to the visu- alization. • Fragmented objects: The visual content is required to be displayed fragmented, as one of our main claims is to adapt visuals to the structure of façades and they often comprise compact areas discontinued by some orna- mentation, windows or the like. It should be possible to load a single resource and split it into several parts for wide spread display. • Dynamics: The framework was supposed to support animated content by means of moving, rotating frag- mented objects, either by specifying the animation over a separate tool (even at runtime) or by introducing some Figure 4: System architecture of the ambient façades kind of automated animation mechanism. framework: different content types are handled by a content management system and forwarded to a rendering engine which outputs the visuals to a projector system facing a Figure 5 depicts the implemented software architecture for suitable façade. the demonstrator. A user input module allows interacting with the scene during runtime by adding/removing obsta- cles, throwing requisites and defining/undefining black A separate control channel gives the chance to modify pa- areas in the projected image (such as to exclude windows rameters at runtime – a basic feature of ambient display from being projected on). frameworks, as this control channel is used to send e.g. sensor data to the visualization system which in turn can modify size, position, speed, color or similar features of visualized objects for sensor data representation. The con- trol channel is also used to configure the visualization sys- tem regarding a specific façade setup (distance, angles, structures, etc.). Technical Implementation Hardware Our setup was executed on an IBM laptop with a 1.7 GHz Pentium M CPU and an ATI Mobility Radeon 7500 inte- grated graphics card running Windows XP SP2. The pro- Figure 5: The software architecture used for the Ambient jected image was required to fill an area of at least 4.5x6 Façade Framework wraps the underlying C-libraries into square meters on somewhat light façades. To provide a convenient C++ classes; instance management is handled bright and high-contrast picture we decided to use a Barco from a central entity “Scene Manager”. SLM R12+ Performer large venue projector with 12000 ANSI Lumen, to be positioned about 18 meters from the The central management entity is responsible for rendering building. The resolution chosen for the projection was the components by providing a simple scene graph, which 1024x768 pixels. For receiving live video streams we is altered by user input or a parallel process generating ran- added a Logitech QuickCam Pro 9000 webcam connected dom pieces to be integrated as falling objects into the via USB 2.0. scene. It calls the appropriate functions of the underlying Software C-libraries and is supported by a separate thread responsi- Before we started implementing a structured framework, ble for continuously buffering webcam content in a byte we did some technology research and created simple labo- array to be used as texture. ratory demos in order to be able to estimate implementation The user input is performed using a pointing device such as effort and feature richness of the tested components. One of a mouse for positioning obstacles, black areas and for the key findings was that our framework is only required to throwing requisites around. The basic workflow is to define support two dimensional positioning, moving, etc. as we façade structures and unprojectable areas once the applica- intended to project on flat surfaces only and wanted to in- tion is running and projected onto the façade. The demon- teract with structures of these surfaces. It occurred to us strator is then ready to go and starts dropping requisites that a 2D physics engine would help our efforts a lot, espe- from somewhere above the screen into the scene. With a cially by solving the question how to animate components keystroke the direction of gravity can be adjusted to any of as to provide constant motion. A quick research in the top-down, bottom-up, left-right, right-left. The requisites physics engine “market” disclosed the Chipmunk 2D phys- are generated using random numbers and can differ in type ics engine which is licensed under the unrestrictive MIT (shape, texture), initial coordinates, initial velocity and di- license and is written in pure C99, which led us to the deci- rection and angular rate. The interval between the creations sion to use OpenGL as the rendering engine. Even though of two consecutive requisites is between 100 and 600 ms. we did not want to support full 3D applications, the use of a The coordinates of each requisite are tracked and compared three dimensional graphics engine allowed easily integrat- to the borders of the viewport; in combination with the cur- ing different layers, usually referred to as z-order of visual rent direction of gravity the requisites are deleted and re- components. spective memory freed if a certain distance threshold has The visuals would be implemented as textured meshes of been exceeded and the objects are not to return to the view- arbitrary shapes and sizes. Texturing meshes with still im- port anymore. ages was offered by the DevIL library, uniformly colored Of course, also elements that are not managed by the phys- meshes were pigmented using OpenGL’s glColor* func- ics engine can be included, to realize static elements, e.g. tions. AVI video files were read using the Video for Win- used for fragmented video visualization, as depicted in dows API and the grabbed frames were converted into tex- Figure 6. ture compatible byte arrays. Live video streams were real- ized with the OpenCV library through the HighGUI API. To ensure the correct color order of the webcam content, the respective pixel buffer is displayed in GL_BGR_EXT format. REAL LIFE SETUP We tested our Ambient Façade Framework during a per- formance of La Traviata at the Theatre Linz to mainly find out two things: 1. Is the technical realization good enough regarding brightness and contrast of the projected image and the size of the fragments? Figure 6: The frameworks shows fragmented video content 2. What is the (subjective) overall visual impression like? only in areas not masked. The mask is adjusted according The first question can be answered quickly: the chosen to the underlying façade structures. Barco projector illuminated the façade of the Theatre Linz at an amazing level of brightness and contrast. Of course, One important aspect of the projection based system was to the façade was a very complaisant screen as it was unen- avoid bright light flooding the rooms behind the façade and lightened and had a very pale yellowish color resulting in probably blinding or disturbing people working or lingering almost no color variation. The displayed visuals were good therein. To overcome this issue, we added a mask layer on to perceive, however some of the objects used for the dy- top of the rendered scene where black (not to be projected) namics simulation turned out to be too small. areas could be defined. Ultimately, even if a collision de- The overall visual impression of our live demonstration tection would fail, a requisite falling into a window would was outstanding. Invited representatives of the Theatre Linz not be visualized but filtered out by the black masking and our colleagues were impressed by the quality of the layer. It is therefore possible to use this layer to display displayed content and the ease of use concerning the setup fragmented video slices by simply erasing parts of the con- process which took roughly one minute to mark structures tent from the overall video (cf. Figure 6). and ornamentation using a simple pointing device. The We implemented the concept of textures as abstract as pos- dynamics engine emerged to be very attractive and created sible, ending up with a system that allows comfortable ex- a very harmonic relation between the façade and the dis- change of textures and sharing of textures between multiple played objects. Changes in gravitation were easy to follow objects regardless of the texture type (image, color, video, and the bouncing elements made sure that there is motion at none). any time. Animated elements were not necessary for dis- playing video streams, as the moving images are attractive The performance of the system was satisfying and ran flu- enough when displayed on their own, as static elements ently on the specified (aging) system. The most influential filling certain areas of the façade. bottleneck was the physics engine as it considerably slowed down if more than two hundred objects were to be consid- ered. A built-in simulation mode helps understanding the basic behavior of implemented features by rendering the com- plete scene to a separate texture and blending it on top of a façade. The section of the façade to be projected on can be adjusted to any extend required. It is possible to view the whole façade or just the part where the projection will take place (cf. Figure 7). (a) (b) Figure 8: The framework at runtime, projecting on the fa- çade of the Theatre Linz: requisites fall down the façade and interact with structural elements of the façade (a). Fragmented video elements are projected on the two pillars on the façade. The live demonstration did not incorporate any sensor data, (a) (b) but was controlled manually, because we mainly wanted to Figure 7: The simulation mode of the Ambient Façade test the visual appearance rather than the correct transfor- Framework allows viewing the section to project on (a) or mation of sensor data into ambient information objects. the whole building with the visuals blended on top (b). In the simulation depicted here a fruits theme was used in- stead of the theatre theme illustrated in Figure 8. CONCLUSION 3. Gellersen, H.W., Schmidt, A., Beigl, M., Ambient Me- We have presented the design and implementation of an dia for Peripheral Information Display, Personal and ambient façades framework that uses façades of buildings Ubiquitous Computing, Vol. 3, No. 4, pp. 199-208, De- and their underlying structures and ornamentation together cember, 1999. with large venue projection technology to form a new type 4. Streitz, N., Röcker, C., Prante, T., Stenzel, R., van Al- of ambient display in urban spaces. The presented frame- phen, D., Situated Interaction with Ambient Informa- works is able to display dynamic particles resembling tion: Facilitating Awareness and Communication in pieces of information regardless of their type (video, im- Ubiquitous Work Environments, in Proceedings of the ages, text) by considering physical barriers on a façade, 10th International Conference on Human-Computer In- which can be edited at runtime and customized to various teraction (HCI International 2003), pp. 133-137, June façades. 22-27, 2003. The current status of the demonstrators has shown some 5. Prante, T., Röcker, C., Streitz, N., Stenzel, R., potential for further improvements. In order to adhere to a Magerkurth, C., Hello.Wall – Beyond Ambient Dis- fully automated configuration of masks and obstacles, im- plays, in Adjunct Proceedings of the 5th International age processing methods could be of a great help. By detect- Conference on Ubiquitous Computing (UBICOMP ‘03), ing edges in an image taken from the façade, it would be pp. 277-278, October 12-15, 2008. possible to automatically define obstacles like window bor- ders and ornamentation. Edge detection combined with 6. Holmquist, L.E., Evaluating the Comprehension of recognition of connected areas would enable the automated Ambient Displays, in Extended Abstracts on Human finding of areas for video display. Of course, camera and Factors in Computing Systems, part of CHI 2004, pp. projector need to be calibrated in a way that allows the 1545-1545, April 24-29, 2004. mapping of camera-based coordinates to coordinates within 7. Consolvo, S., Roessler, P., Shelton, B.E., The CareNet the projected renderings. Currently such a feature is not Display: Lessons Learned from an In Home Evaluation implemented in the framework, but the structures need to of an Ambient Display, in Proceedings of the 6th Inter- be defined by hand. national Conference on Ubiquitous Computing (UBI- ACKNOWLEDGMENTS COMP ‘04), pp. 1-17, September 7-10, 2004. The authors would like to thank Heinrich Schmitzberger 8. Vogel, D., Balakrishnan, R., Interactive Public Ambient who implemented parts of the code related to fragmented Displays: Transitioning from Implicit to Explicit, Public objects and supported the preparation and realization of the to Personal, Interaction with Multiple Users, in Pro- live demonstration at the Theatre Linz with his valuable ceedings of the 17th Annual ACM Symposium on User experience in large scale, long range projections. Berna- Interface Software and Technology (UIST ’04), pp. 137- dette Emsenhuber prepared the displayed video files for 146, October 24-27, 2004. development and the demonstration and additionally 9. Pousman, Z., Stasko, J., A Taxonomy of Ambient In- documented the live demonstration with still and moving formation Systems: Four Patterns of Design, in Pro- images. Dominik Hochreiter was another member of the ceedings of the Working Conference on Advanced Vis- live demonstration team and helped with the technical setup ual Interfaces (AVI ’06), pp. 67-74, May 23-26, 2006. and lens and distance calculation. 10. Ferscha, A., Informative Art Display Metaphors, in Finally, we would like to thank Thomas Königstorfer, Proceedings of the 4th International Conference on Uni- commercial chairman of the board of the Theatre Linz, for versal Access in Human-Computer Interaction (UAHCI his encouragement to try our framework on the façade of 2007), pp. 82-92, July, 2007. the Theatre Linz during a performance of La Traviata. We also appreciate his precious comments and feedback re- 11. Ferscha, A., A Matter of Taste, in Proceedings of the 2nd garding further development. European Conference on Ambient Intelligence (AmI 2007), November, 2007. REFERENCES 12. Walker, K., Smoke Signals: Particle Systems for Ambi- 1. Wisneski, C., Ishii, H., Dahley, A., Gorbet, M., Brave, ent Information Display, Internet: S., Ullmer, B., Yarin, P., Ambient Displays: Turning http://www.exhibitresearch.com/kevin/ioe/smoke.pdf, Architectural Space into an Interface between People last viewed: July, 2008. and Digital Information, in Proceedings of the 1st Inter- national Workshop on Cooperative Building (CoBuild 13. Stasko, J., Doo, M., Dorn, B., Plaue, C., Explorations ’98), pp. 22-32, February, 1998. and Experiences with Ambient Information Systems, in Proceedings of the Workshop for Ambient Information 2. Dahley, A., Wisneski, C., Ishii, H., Water Lamp and Systems at the 5th International Conference on Perva- Pinwheels: Ambient Projection of Digital Information sive Computing (PERVASIVE 2007), pp. 36-41, May into Architectural Space, in Conference Summary on 13, 2007. Human Factors in Computing Systems (CHI ’98), pp. 269-270, April 18-23, 1998. 14. Tomitsch, M., Kappel, K., Lehner, A., Grechenig, T., Towards a Taxonomy for Ambient Information Sys- tems, in Proceedings of the Workshop for Ambient In- formation Systems at the 5th International Conference 17. Peltonen, P., Kurvinen, E., Salovaara, A., Jacucci, G., on Pervasive Computing (PERVASIVE 2007), pp. 42- Ilmonen, T., Evans, J., Oulasvirta, A., Saarikko, P., “It’s 47, May 13, 2007. Mine, Don’t Touch!”: Interactions at a Large Multi- 15. Huang, E., Koster, A., Borchers, J., Overcoming As- Touch Display in a City Centre, in Proceedings of the sumptions and Uncovering Practices: When Does the 26th Annual SIGCHI Conferences on Human Factors in Public Really Look at Public Displays?, in Proceedings Computing Systems (CHI ’08), pp. 1285-1294, April 5- of PERVASIVE 2008, pp. 228-243, May 19-22, 2008. 10, 2008. 16. Minakuchi, M., Kidawara, Y., Kinetic Typography for 18. UNIQA Versicherungen AG, Der UNIQA Tower – Ambient Displays, in Proceedings of the 2nd Interna- LED Design, Internet: tional Conference on Ubiquitous Information Manage- http://tower.uniqa.at/content/led_koerperdesign.php?nav ment and Communication (ICUIMC ’08), pp. 54-57, =2|0, last viewed: July, 2008. 2008. 19. LICHT KUNST LICHT AG, UNIQA-Tower, Internet: http://www.lichtkunstlicht.de/, last viewed: July, 2008.