Public spaces get increasingly equipped with displays in terms of shopping window plasma screens, electronic advertisements at the point of sale, kiosk systems at points of interest, etc. While this trend enables numerous applications in the pervasive display systems domain, it also has effects on how people perceive urban environments. In this work we describe the concept, implementation and first experiences from a real life setup of an ambient façades framework expanding the idea of public displays to façades of arbitrary buildings without modifications on the buildings themselves. With such a framework it is possible to integrate information into buildings in a very unobtrusive way and without interference with the building fabric.
systems as they can deliver information extracted from the
single particles and a particle system has an overall
appearance (shape, volume, etc.) that can unveil even more
information. The ambient display framework described in this
work also makes use of the low-level and high-level
statements of a high number of objects on an ambient display.
[
In [
One of the rather seldom seen examples of large public
displays is presented in [
Evaluation of Ambient Information Systems
Regarding the evaluation of ambient displays, several
approaches have been presented, such as a method to evaluate
the comprehension of such displays [
The author emphasizes that it is important to consider the
first two steps in the system design process and not start
(blindly) at level 3 [
Users’ experiences with an at-home ambient display have
been presented in [
In [
In [
A very critical look at public displays is taken in [
Purpose In order to be able to support future development of façades as displays in combination with the ambient display metaphor we decided to implement a robust framework as a basis for further ideas and implementations.
We enunciated the requirements for the framework very roughly, as we wanted to narrow the choice of technical solutions as less as possible: •
Text: There might be a need to display text of any size, font type and color. However, with regards to ambient displays, text is usually avoided in favor of graphical solutions – thus it is a minor requirement. • Still images: Support for embedding images into the visualization including scaling functions (each axis independently) and, of course, free positioning. •
Moving images: The framework should be able to render videos and support both live camera streams and produced videos: • Live camera streams: Since our first façade was to be the one of the Theatre Linz, we opted to integrate the possibility to render live camera streams to the façade. This thought was driven by the idea to present the current action on stage simultaneously outside. • Produced videos: In addition to live video, our system should support readily available videos in order to visualize perfectly pre-arranged content and selected scenes. Also, in case of a live camera failure, locally available videos could be applied to the visualization. • Fragmented objects: The visual content is required to be displayed fragmented, as one of our main claims is to adapt visuals to the structure of façades and they often comprise compact areas discontinued by some ornamentation, windows or the like. It should be possible to load a single resource and split it into several parts for wide spread display. •
Dynamics: The framework was supposed to support animated content by means of moving, rotating fragmented objects, either by specifying the animation over a separate tool (even at runtime) or by introducing some kind of automated animation mechanism. •
Content management: A content management system supports the integration of different resources (images, videos, streams) at runtime and provides a way to define the position and shape of structures and ornaments of the façade to project on. For better results, the definition should take place on-site, when projection distance and angle are known. Additionally, the support for onsite structure definition paves the way for automated mechanisms, e.g. via a camera based system driving edge detection or other image processing algorithms. Aside from these requirements we also had a picture in mind of what we would like to achieve. A relatively coarse illustration thereof is depicted in Figure 2 and Figure 3.
System Architecture Based on our visions and derived requirements we decided on a simple system architecture comprising a software framework running on a PC which renders the visuals to a projector system and receives data from several resources as well as user input for the content management system. A rough system architecture is illustrated in Figure 4. which outputs the visuals to a projector system facing a suitable façade.
A separate control channel gives the chance to modify parameters at runtime – a basic feature of ambient display frameworks, as this control channel is used to send e.g. sensor data to the visualization system which in turn can modify size, position, speed, color or similar features of visualized objects for sensor data representation. The control channel is also used to configure the visualization system regarding a specific façade setup (distance, angles, structures, etc.).
Technical Implementation Hardware Our setup was executed on an IBM laptop with a 1.7 GHz Pentium M CPU and an ATI Mobility Radeon 7500 integrated graphics card running Windows XP SP2. The projected image was required to fill an area of at least 4.5x6 square meters on somewhat light façades. To provide a bright and high-contrast picture we decided to use a Barco SLM R12+ Performer large venue projector with 12000 ANSI Lumen, to be positioned about 18 meters from the building. The resolution chosen for the projection was 1024x768 pixels. For receiving live video streams we added a Logitech QuickCam Pro 9000 webcam connected via USB 2.0.
Software Before we started implementing a structured framework, we did some technology research and created simple laboratory demos in order to be able to estimate implementation effort and feature richness of the tested components. One of the key findings was that our framework is only required to support two dimensional positioning, moving, etc. as we intended to project on flat surfaces only and wanted to interact with structures of these surfaces. It occurred to us that a 2D physics engine would help our efforts a lot, especially by solving the question how to animate components as to provide constant motion. A quick research in the physics engine “market” disclosed the Chipmunk 2D physics engine which is licensed under the unrestrictive MIT license and is written in pure C99, which led us to the decision to use OpenGL as the rendering engine. Even though we did not want to support full 3D applications, the use of a three dimensional graphics engine allowed easily integrating different layers, usually referred to as z-order of visual components.
The visuals would be implemented as textured meshes of arbitrary shapes and sizes. Texturing meshes with still images was offered by the DevIL library, uniformly colored meshes were pigmented using OpenGL’s glColor* functions. AVI video files were read using the Video for Windows API and the grabbed frames were converted into texture compatible byte arrays. Live video streams were realized with the OpenCV library through the HighGUI API. To ensure the correct color order of the webcam content, the respective pixel buffer is displayed in GL_BGR_EXT format.
Figure 5 depicts the implemented software architecture for the demonstrator. A user input module allows interacting with the scene during runtime by adding/removing obstacles, throwing requisites and defining/undefining black areas in the projected image (such as to exclude windows from being projected on).
The central management entity is responsible for rendering the components by providing a simple scene graph, which is altered by user input or a parallel process generating random pieces to be integrated as falling objects into the scene. It calls the appropriate functions of the underlying C-libraries and is supported by a separate thread responsible for continuously buffering webcam content in a byte array to be used as texture.
The user input is performed using a pointing device such as a mouse for positioning obstacles, black areas and for throwing requisites around. The basic workflow is to define façade structures and unprojectable areas once the application is running and projected onto the façade. The demonstrator is then ready to go and starts dropping requisites from somewhere above the screen into the scene. With a keystroke the direction of gravity can be adjusted to any of top-down, bottom-up, left-right, right-left. The requisites are generated using random numbers and can differ in type (shape, texture), initial coordinates, initial velocity and direction and angular rate. The interval between the creations of two consecutive requisites is between 100 and 600 ms. The coordinates of each requisite are tracked and compared to the borders of the viewport; in combination with the current direction of gravity the requisites are deleted and respective memory freed if a certain distance threshold has been exceeded and the objects are not to return to the viewport anymore.
Of course, also elements that are not managed by the physics engine can be included, to realize static elements, e.g. used for fragmented video visualization, as depicted in Figure 6.
One important aspect of the projection based system was to avoid bright light flooding the rooms behind the façade and probably blinding or disturbing people working or lingering therein. To overcome this issue, we added a mask layer on top of the rendered scene where black (not to be projected) areas could be defined. Ultimately, even if a collision detection would fail, a requisite falling into a window would not be visualized but filtered out by the black masking layer. It is therefore possible to use this layer to display fragmented video slices by simply erasing parts of the content from the overall video (cf. Figure 6).
We implemented the concept of textures as abstract as possible, ending up with a system that allows comfortable exchange of textures and sharing of textures between multiple objects regardless of the texture type (image, color, video, none).
The performance of the system was satisfying and ran fluently on the specified (aging) system. The most influential bottleneck was the physics engine as it considerably slowed down if more than two hundred objects were to be considered.
A built-in simulation mode helps understanding the basic behavior of implemented features by rendering the complete scene to a separate texture and blending it on top of a façade. The section of the façade to be projected on can be adjusted to any extend required. It is possible to view the whole façade or just the part where the projection will take place (cf. Figure 7).
(a) (b) Figure 7: The simulation mode of the Ambient Façade Framework allows viewing the section to project on (a) or the whole building with the visuals blended on top (b). In the simulation depicted here a fruits theme was used instead of the theatre theme illustrated in Figure 8. REAL LIFE SETUP We tested our Ambient Façade Framework during a performance of La Traviata at the Theatre Linz to mainly find out two things: 1. Is the technical realization good enough regarding brightness and contrast of the projected image and the size of the fragments? 2.
What is the (subjective) overall visual impression like? The first question can be answered quickly: the chosen Barco projector illuminated the façade of the Theatre Linz at an amazing level of brightness and contrast. Of course, the façade was a very complaisant screen as it was unenlightened and had a very pale yellowish color resulting in almost no color variation. The displayed visuals were good to perceive, however some of the objects used for the dynamics simulation turned out to be too small.
The overall visual impression of our live demonstration was outstanding. Invited representatives of the Theatre Linz and our colleagues were impressed by the quality of the displayed content and the ease of use concerning the setup process which took roughly one minute to mark structures and ornamentation using a simple pointing device. The dynamics engine emerged to be very attractive and created a very harmonic relation between the façade and the displayed objects. Changes in gravitation were easy to follow and the bouncing elements made sure that there is motion at any time. Animated elements were not necessary for displaying video streams, as the moving images are attractive enough when displayed on their own, as static elements filling certain areas of the façade.
(a) (b)
The live demonstration did not incorporate any sensor data, but was controlled manually, because we mainly wanted to test the visual appearance rather than the correct transformation of sensor data into ambient information objects. CONCLUSION We have presented the design and implementation of an ambient façades framework that uses façades of buildings and their underlying structures and ornamentation together with large venue projection technology to form a new type of ambient display in urban spaces. The presented frameworks is able to display dynamic particles resembling pieces of information regardless of their type (video, images, text) by considering physical barriers on a façade, which can be edited at runtime and customized to various façades.
The current status of the demonstrators has shown some potential for further improvements. In order to adhere to a fully automated configuration of masks and obstacles, image processing methods could be of a great help. By detecting edges in an image taken from the façade, it would be possible to automatically define obstacles like window borders and ornamentation. Edge detection combined with recognition of connected areas would enable the automated finding of areas for video display. Of course, camera and projector need to be calibrated in a way that allows the mapping of camera-based coordinates to coordinates within the projected renderings. Currently such a feature is not implemented in the framework, but the structures need to be defined by hand.
ACKNOWLEDGMENTS The authors would like to thank Heinrich Schmitzberger who implemented parts of the code related to fragmented objects and supported the preparation and realization of the live demonstration at the Theatre Linz with his valuable experience in large scale, long range projections. Bernadette Emsenhuber prepared the displayed video files for development and the demonstration and additionally documented the live demonstration with still and moving images. Dominik Hochreiter was another member of the live demonstration team and helped with the technical setup and lens and distance calculation.
Finally, we would like to thank Thomas Königstorfer, commercial chairman of the board of the Theatre Linz, for his encouragement to try our framework on the façade of the Theatre Linz during a performance of La Traviata. We also appreciate his precious comments and feedback regarding further development.