The Micro Smart Grid Tabletop — A real world case study for tangible multitouch technologies for communicating complex concepts Sebastian Meier Potsdam University of Applied Sciences & IXDS, Berlin contact@sebastianmeier.eu ABSTRACT In 2012 the InnoZ explored possibilities to extend their In this paper we present a case study for an interactive surface, an educational material to explain the topic of MSGs to other explanatory tangible multitouch system in an exhibition context. scientists, the public, as well as decision and policy makers. The The system is used as an expressive representation of the complex InnoZ campus itself is a MSG, which is made visible by live data concept of Micro Smart Grids (MSGs) and their influence on our shown on a control terminal. The envisioned tool should go a step future energy infrastructure. We built the experience upon further and explain the structure of MSGs as well as their impact research on integrating tangibles into playful and exploratory on our future energy infrastructure. learning environments. Beyond the well-researched learning experience with tangibles, we developed the system to stimulate 2. DESIGN OF THE SYSTEM and foster discussions on the subject of MSGs. Besides the technology behind the system we will discuss limitations and 2.1 Design Process advantages of combining tangible and multitouch technologies. The design process followed the principles of human To conclude we present reflections on the design and interaction centered design (Abras, Maloney-Krichmar, & Preece, 2004), by concepts that were embedded into the system. including not only stakeholders but future users in the design process from the beginning. The human centered design was executed in an iterative and adaptive manner (Smith et al., 2012), General Terms meaning that e.g. insights gathered from paper-prototyping (see Design, Reliability, Experimentation, Human Factors Figure 1) were quickly implemented in feasibility prototypes to test the interactions on the actual hardware. Especially when Keywords working with large multitouch systems and tangible interactions, Tangible, Touchscreen, Surface, Interaction, Case Study the prototyping becomes an important part of the design process, as it is the only way to experience the interactions e.g. in regards to responsiveness. This allowed us to identify and resolve 1. INTRODUCTION problems in the interface and interaction design throughout the Global warming, sustainability and lately resilience are process and by that limiting the revisions at the end of the topics that matter to an audience beyond the scientific community. development process. Due to the growing interest and the urgency presented by these topics, new tools for scientific communication are required. Tools that allow scientists as well as decision and policy makers to communicate complex issues to non experts, to foster an inclusive discussion on the basis of facts and scientific results. The Berlin based research institute InnoZ is focusing on those topics, especially how transport, energy and information and communications technologies (ICTs) are involved in shaping the urban future and how new products and technologies in these areas of research are adopted by society (Wolter, 2014). The InnoZ runs a showroom in their Berlin headquarters, allowing visitors to explore new technologies and learn more about future changes in our energy and transport infrastructure. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that Figure 1. Paper-prototyping of the tangible interface copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other 2.2 System Overview uses, contact the Owner/Author. The initial aim of our designs was to create a tool that would allow the research team to use the proposed system as an Copyright is held by the owner/author(s). explanatory system to educate visitors on the matter of MSGs. As TacTT '14, Nov 16 2014, Dresden, Germany a basis for the explanatory system we created a virtual model of a 10 years ahead and in 20 years ahead. The three scenarios MSG as well as a model of the surrounding energy infrastructure. influence the datasets for the components behind the model. All The model was built upon statistics and estimations on energy parameter “states” are not only visually supported by highlighting consumption and production in Germany. On top of the model the current state in the control bar, but also the visual appeal is visualization was implement to explain the workings of the changing. For example modifying wind speed changes the speed underlying model, respectively a MSG. The components of the of the clouds or changing daytime changes the brightness (see model, consumers, producers, storage components and MSG- Figure 3). This allows the spectators to create a connection controller were represented by tangible tokens that could freely be between the changes of parameters and the changes within the combined by the users. In this paper we will refer to the tangible, system, e.g. less brightness equals less light and thereby less solar physical objects that are placed on the interactive surface as energy. tokens. Those tokens were an attempt to overcome the initial barrier of interacting with a complex system. Thus we made use of the attractiveness and playfulness of tangible interactions (Nagel, Pschetz, Stefaner, Halkia, & Müller, 2009; Wakkary & Hatala, 2006). We employed an interactive surface with tangible interactive tokens, thereby creating a setting for multi-user interaction and observation. 2.3 Interaction Concept 2.3.1 The components As explained above, the basis for the system is a complex model based on statistics and estimations of energy production, consumption and storage. By analogy with the collected data the Figure 3. Parameter daytime: left-night, right-day model is made up of three categories of components: consumers, producers and energy storage. The consumers are private 2.3.3 Relationships households and industrial consumers, their consumption is The system displays two types of relationships between the dependent on the external parameters daytime and time of year, components added to the current state through the tokens. On the e.g. during daytime and especially during the winter the one hand the information exchanged between consumers, consumption is higher. The energy producers are divided by fossil producers and storage components are being visualized through fuels including nuclear energy and renewable energy. The latter fine grey lines and on the other hand the more important part, the are, similar to the consumers, dependent on external parameters. flow of energy between the components, is visualized through a This includes e.g. solar energy’s dependency on cloudiness or flow of particles. While of course the speed of transporting the wind energy’s dependency on wind speed. The category of energy energy is always the same, the visualization makes use of particle storages holds electric cars, water pumps, natural gas energy density within the stream to indicate higher and lower production converters as well as large batteries. In addition to the three main levels and changes the color of the particles to allow users to components a forth component, the MSG-controller is available to differentiate between fossil (orange) and renewable (green) create a MSG. energy (see Figure 4). Each component is represented by several tokens that can be combined freely by the users. For a more intuitive interaction the tokens belonging to the same category share the same form factor: triangle = producer, circle = consumer, square = storage, hexagon = MSG-Controller (see Figure 2). The specific identity of the component is printed on top of the token in the form of an icon. Figure 2. The set of tokens Figure 4. Flow of fossil and renewable energy 2.3.2 Parameters As mentioned in 2.3.1 the model has several parameters that 2.3.4 Levels of Information influence the individual components. The whole model is running The whole system consists of several layers of information. on a time loop of day/night as well as months, which influences The first layer consists of the tokens representing the components the energy consumption as well as the production of energy e.g. and their connections through the visualization of the model. The through solar cells. In addition to the time data, the weather second layer is visualizing the “state” of the parameters that parameters wind and clouds can influence the energy production influence the system. The third layer is a layer of information that of the renewable energy producers. Beyond daytime and time of can be activated per component on demand, it shows more the year, the system offers three time scenarios: present-time, in detailed information on every component (see Figure 5). The users can use the last layer of information to read short texts about allows the users to change quickly between different states e.g. every component and learn more about their impact on the energy night and day and thereby observe and learn what this means e.g. infrastructure through visualized statistics. Those layers represent for solar energy production. levels of information complexity, starting from the lowest level of learning how MSGs work in general. Down to the level of facts If the user is interested in learning more about a specific component, she can click on info buttons that float along the 2D e.g. how much energy is produced by a certain power plant and the estimated production throughout the next 20 years. Through representation of the components. A small window will pop up this we allow the user to dive deeper into the system and thereby allowing the user to discover more information on that allowing a deeper engagement (Hornecker, 2008). component. In order to create a non-fragmented view of the visualization and the interactions happening, we had to create interfaces and visualizations that were accessible from every side of the table. While the main interaction through tangibles is possible from every side and the observations as well, the manipulation of properties is only possible from the two long sides of the table due to the limited space on the screen. 2.4 Theoretical Framing of the Interaction In recent years, several descriptive frameworks for tangible interaction and tangible interfaces have been developed (Hornecker, 2006; Hornecker & Buur, 2006; Mazalek & van den Figure 5. Detailed information layer with graphs Hoven, 2009; Shaer & Hornecker, 2009). In the following section 2.3.5 Interaction we build upon the framework by Hornecker et al, because in our context a specific focus lies upon the learning experience by The user can start her journey of exploring the system with a means of the interaction. This has also been covered by blank plate or with a template that works like a puzzle game, in Hornecker’s work, which extends beyond the framework itself which the user sees 2D tokens on the screen and needs to place the 3D markers on top of it and by that assembling a basic setup to into the area of learning through tangible interaction or rather further explore the system. Through those two modes the user can tangible interfaces. Horneckers framework builds upon four main receive different levels of guidance. pillars: tangible manipulation, spatial interaction, embodied facilitation and expressive representation. Those categories are By placing a token on the table, a visual 2D representation of not exclusive, which means that applications that embed those the token shows up below the token, the shape is corresponding to properties span across multiple pillars. the shape of the token to create a visual connection between token Through the visual correlation of tokens and visualization on and visualization. Furthermore, the two-dimensional the screen and the different visualization schemes we created a representation behaves according to the category it represents: producers are generating energy particles and distributing those to strong coupling between the physical and the digital. We were available consumers; consumers are aggregating energy particles able to establish a consistent visual storytelling and an intuitive and consuming them; storage components are aggregating unused and seamless interaction, following the pillars of tangible energy to store them. The storage components differ from the manipulation and expressive representations. In terms of other components as their 2D representation have a visual expressiveness, our approach sees the tokens as essential parts of the construction of the system through the users, even though they indicator of the amount of energy that is stored in the individual don’t represent a change within the system as it would be required component. During the design process the responsiveness and according to the definition of expressiveness by Ullmer and Ishii visual correspondence between tokens and visualization played an important role, as we tried to achieve a high “perceived coupling” (Ullmer & Ishii, 2000). The theme of expressiveness is further effect to create an intuitive interaction with a feeling of “direct more picked up by Marshall et al in the area of learning through haptic manipulation” (Hornecker, 2006; Hornecker & Buur, tangibles, which is essential for our approach (Marshall, Price, & 2006). Rogers, 2003). From their point of view, our approach is not expressive; instead it fulfills the requirements for what they call The MSG-Controller plays a significant role, as it is essential an exploratory system. As we try to use the tangibles to help users to understanding the workings of MSGs. If no MSG-Controller is understand the underlying model and make use of the tangibles as present on the table, the components almost behave as if “present-at-hand , the user will be more likely to focus on the way everything is connected to everything. As soon as the MSG- the system works, rather than reflecting on the history of their Controller is added to the mix, the new token adds a sense of own interaction with it” (Marshall et al., 2003 p3). smartness and structure to the relationships and connections between the components. Through the interaction of adding and 3. REVIEW OF TECHNOLOGY removing the MSG-Controller token, the user can instantly Papers that focus on the interaction with tangibles often only observe how the visualization and the structure in the underlying briefly explain the actual technology used for the experiments. model beneath changes. This makes it hard for other researchers and practitioners to build In order to create a more realistic setting, the user can modify upon the experience gained on the technology side. This is why the parameters described in 2.3.2, allowing for the audience to we will spend the next section on elaborating the technological observe changes in the energy system according to changes in the side of our project as well as the pros and cons. environment. Again, similar to the effect of the MSG-controller, For the multitouch table we employed a large LCD system as the instant modification of the model through the parameters a display. Like most common multitouch systems that don't use capacitive touch technology, the system uses multiple HD cast shadows accurate for arbitrary times of day; to throw infrared-enabled cameras to track touch events as well as fiducials reflections off glass facade surfaces; to affect a real-time and (printed 2D codes). Using infrared tracking systems has the visually coincident simulation of pedestrian-level wind- advantage of being able to use touch events as well as tangible flow"(Underkoffler & Ishii, 1999 p1). In this example, the real- objects both in combination with printed fiducials. Then again, the world model of wind and light can be explored through tangible infrared technology has several downsides which become present tokens that represent buildings and interact with the projection, in an exhibition context. Even though the technology uses the allowing the user to directly manipulate the virtual model and infrared light spectrum, it is still sensitive to other spectrums of simulation through the tokens. the light. Changing light intensity in the surrounding sometimes calls for readjustment of the system, the same is true for light Illuminating Light, published one year earlier by the same sources that heat up throughout the day or, even worse, direct research group at MIT (Underkoffler & Ishii, 1998) is a "rapid sunlight. Sources of light and/or heat can create noise on the prototyping [system] of laser-based optical and holographic infrared image. In order to achieve very precise tracking results, layouts" (Underkoffler & Ishii, 1998 p1). The tool uses tangible tokens to enable users to set up a system of light rays and objects which are required for tracking fiducials, the trackers need to be adjusted carefully. This sometimes conflicts with exhibition that redirect, concentrate or burst the beams, which is visualized spaces being lively spaces. An interactive piece build for group in a projected overlay, similar to Urp. interaction will be under a lot of physical friction. Sometimes exhibits additionally need to be relocated e.g. for rearranging or 4.2 Relationships special events. All those factors have an impact on the system and The second category consists of projects, that similar to the often create need for readjustments. first category, build upon real-world models, but rely on a more abstract visualization, as the real-world representation is too The design of the tangible objects is quite limited by the complex to visualize. Instead of showing the real connections required integration of the printed fiducials. A big problem of between the components of the model, the visualization focuses tangible tokens for interfaces is visual or tactile feedback, as Ishii on a representation that displays the relationship between and Ullmer illustrate “unlike malleable ‘bits’, ‘atoms’ are components and their influence on each other, allowing the user to relatively inflexible” (Ishii & Ullmer, 2012 p469). Our first manipulate those influences and relationships and thereby foster attempt was to insert a solid light-reinforcing material into the deeper understanding. black tokens to allow us to use the emitted light from the screen and make it become visible at the sides of the token. This In the area of sound- or music-interfaces the most cited case confronted us with two problems: for one, light reinforcing study is the Reactable by Reactivision (Jordà, Geiger, Alonso, & plastics only achieve really good results with low ambient light, Kaltenbrunner, 2007), which allows the creation of electronic additionally we were required to use special infrared see-through music through tangible tokens, that are visually connected to each material for the 2D codes. The latter was the reason we were other by the underlying display. The user can manipulate the unable to pursue the plan of creating responsive tokens as the see- music by using different tokens, changing orientation of the through 2D codes didn't meet our requirements for precise tokens, or modifying the distance between objects. tracking results. In the end, black and white 2D codes on solid The IP network simulation is using a similar approach for black tokens were used. visualizing IP networks (Kobayashi, Hirano, Narita, & Ishii, 2003). The set of tokens allow the user to create new nodes and 4. RELATED WORKS modify the parameters, creating a virtual representation of an IP Our interactive system stands in a line of projects dealing with network and thereby explore the inner workings of such. tangible interfaces that visualize flow and relationship. Most The last project from the subcategory of projects that focus projects from the category of what Ishii and Ullmer call on visualizing relationships between components is the Maeve "interactive surfaces" or "workbench surfaces" (Ishii & Ullmer, project (Nagel et al., 2009). Maeve is an interactive multitouch 2012) that are similar to our project fall into two subcategories: installation that is enriched with tangible cards, which can be one category focuses on visualizing the flow of real world entities placed on the multitouch installation. Maeve uses architectural like wind, light or in our case energy. The other category contains metadata collected by the research project Mace (Stefaner et al., systems that visualize relationships of objects through a flow- and 2007). Each card represents an item from the database. Placing a network-like visualization. card onto the table allows the user to explore connections and relationships between the item and other items in the database. 4.1 Real world entities The first category consists of projects that create tangible interfaces, or rather tangible interactive visualizations, which 5. REFLECTIONS build upon real-world models. These projects use attributes that Our reflections derive from the observation of users exist in the real world and map them onto the interface or surface. interacting with the system. The evaluation was collected through The projects try to keep the abstraction between the real-world interviews with three designers who were involved in the model and the artificial model as small as possible in order to development and the deployment, as well as two instructors who create an exploratory tangible system which, according to Marshal work with the table in the showroom. et al, allows for the learner to explore the system through practical exploration and thereby learn more about the underlying 5.1 Still a WOW-Effect structure (Marshall et al., 2003). Even though tabletop multitouch systems are frequently used e.g. in exhibitions or trade fairs, those systems are still able to The most prominent example is Urp — the urban planning create a WOW-effect among visitors. At the same time, the workbench (Underkoffler & Ishii, 1999) which "allows physical widespread use of multitouch technology in smartphones, tablets architectural models [to be] placed on an ordinary table surface to and even household appliances creates similar expectations towards big multitouch systems in regards to interactions. A good study to highlight some of the remarkable features of tangible example to illustrate this is the design of the information popups. interfaces. As other research underlines, we were able to observe In one of the first prototypes, those windows were that large tangible interactive surfaces provide fertile ground for programmatically floating elements, but the tests showed that social interaction and the exploration of real-world models. users were trying to apply common multitouch interaction Furthermore we were able to observe that large tangible interfaces patterns, like “pinch” to zoom and “drag” to move the window can foster discussions on the topic visualized in the interface. On (see Figure 6). In the next iteration we implemented those the other hand we also observed that more focused single-user interactions, which helped creating a more fluent interaction. interactions were not well received by the users. 6.1 Further Research The area of learning through tangible interfaces in regards to children is widely covered in the research community (O'Malley & Fraser, 2004), while this topic lacks the same level of detail in regards to adults. In some regards, similar effects with adults were observed by the related works mentioned above, while other researchers point out that some tactile thinking disappears while we grow up (e.g. finger counting) (Goswami, 2004). This might lead to other results in terms of learning through tangible interfaces. Our reflection is only looking at the overall experience of the tool, anyhow we discovered a lack of information on the Figure 6. User trying to pinch-zoom an information window difference between the interaction of children and adults with tangible interfaces throughout our literary research. Consequently, 5.2 Social Interaction we believe that more research on this topic could help to apply the An observation reported by all interviewed persons was the knowledge from the vast amount of research on learning through tabletop’s function as a catalyst for social interaction, especially tangible interfaces with children to the interaction of adults. fostering discussions among the visitors. The interaction was not only between visitors and instructor but also among the groups themselves. Beyond verbal discussions that arose, common 7. ACKNOWLEDGMENTS The project was put to life by the IXDS team (Jonathan observations were the physical collaborations while using the Cohen, Andre Knörig, Felix Lange, Sebastian Meier, Dimitar table, originating from the “non-fragmented” visibility of the Ruszev, Julia Werner and Reto Wettach) in cooperation with the visualization, giving it an overall performative quality (Hornecker prototype - Moers Peter Sailer GbR who helped with the & Buur, 2006), a phenomenon often described in similar settings theoretical framework as well as an in-depth data aggregation. (Huang, Mynatt, Russell, & Sue, 2006; Nagel et al., 2009). The technology, especially the display unit by multitouch.fi, and furniture was assembled by werk5 GmbH. 5.3 Differences between adults and children As intended by the design, the physical tokens were not only In a recent project we were able to use the latest multitouch an eye-catcher, but also attracted attention and interest leading to and fiducial tracking technology and we were pleased to see that an immediate interaction with the tokens and thereby the system. the dependency from ambient light was a lot better than with the The tokens had this effect on children and young people as well as technology we used for the system described in this paper. on adults. Nonetheless, it was reported that the intuitive usage and Nothing the less, the described problems remain, if only less playful exploration was more powerful on the younger severe. generations. They seemed to understand the workings of the model a lot faster than adults through their more playful 8. REFERENCES interaction with the table. [1] Abras, C., Maloney-Krichmar, D., & Preece, J. (2004). User- centered design. In Encyclopedia of Human-Computer 5.4 Learning Interaction. It is hard to elaborate on the long-time learning effect triggered by the table as the groups visiting the space are only [2] Goswami, U. (2004). Neuroscience and education. British using the table for a limited time. Accordingly, the envisioned Journal of Educational Psychology, 74(Pt 1), 1–14. deeper layers of information were not accessed in many cases. doi:10.1348/000709904322848798 Most users concentrated on the basic structure of the MSG. [3] Hornecker, E. (2006). An Encompassing View on Tangible Beyond the factor of available time, we believe that the user’s Interaction: A Framework. CHI 2006 (pp. 1–4). limited use of access to the detailed information also depends on [4] Hornecker, E. (2008). “I don’t understand it either, but it is the fact that the process of accessing the detailed information is a cool” – Visitor Interactions with a Multi-Touch Table in a very focused interaction that in turn is hard to share with a bigger Museum (pp. 1–8). Presented at the IEEE Tabletop. group. In contrast to the focused interaction the token-based interaction can be executed in a collaborative manner and the [5] Hornecker, E., & Buur, J. (2006). Getting a Grip on Tangible results can easily be visually shared with a larger group. Interaction: A Framework on Physical Space and Social Interaction (pp. 1–10). Presented at the CHI 2006. 6. CONCLUSION [6] Huang, E. M., Mynatt, E. D., Russell, D. M., & Sue, A. E. The MSG tabletop allows users to explore a complex real- (2006). Secrets to success and fatal flaws: the design of world model through a playful and explorative interaction. The large-display groupware. IEEE Computer Graphics and results from the observation show that the system is a good case Applications, 26(1), 37–45. [7] Ishii, H., & Ullmer, B. (2012). Tangible User Interfaces. In J. [14] Shaer, O., & Hornecker, E. (2009). Tangible User Interfaces: Jacko, The Human-Computer Interaction Handbook. CRC Past, Present, and Future Directions. Foundations and Press. Trends® in Human–Computer Interaction, 3(1-2), 1–137. [8] Jordà, S., Geiger, G., Alonso, M., & Kaltenbrunner, M. doi:10.1561/1100000026 (2007). The reacTable: Exploring the Synergy between Live [15] Smith, P. J., Beatty, R., Hayes, C. C., Larson, A., Geddes, N. MusicPerformance and Tabletop Tangible Interfaces. D., & Dorneich, M. C. (2012). Human-Centered Design of Presented at the TEI '07: Proceedings of the 1st international Decision Support Systems. In J. Jacko, The Human- conference on Tangible and embedded interaction. Computer Interaction Handbook. CRC Press. [9] Kazue Kobayashi, Mitsunori Hirano, Atsunobu Narita, and [16] Stefaner, M., Vecchia, E. D., Condotta, M., Wolpers, M., Hiroshi Ishii (2003) A tangible interface for IP network Specht, M., Apelt, S., & Duval, E. (2007). MACE - enriching simulation. In CHI '03 Extended Abstracts on Human Factors architectural learning objects for experience multiplication in Computing Systems (CHI EA '03). ACM, New York, NY, (Vol. 4753, pp. 322–336). Presented at the EC-TEL'07: USA, 800-801. DOI=10.1145/765891.766000 Proceedings of the Second European conference on [10] Marshall, P., Price, S., & Rogers, Y. (2003). Conceptualising Technology Enhanced Learning: creating new learning tangibles to support learning (pp. 101–109). Presented at the experiences on a global scale, Berlin, Heidelberg: Springer- IDC '03: Proceedings of the 2003 conference on Interaction Verlag. doi:10.1007/978-3-540-75195-3_23 design and children, New York, New York, USA: ACM. [17] Ullmer, B., & Ishii, H. (2000). Emerging frameworks for doi:10.1145/953536.953551 tangible user interfaces. IBM Systems Journal, 39(3), 915– [11] Mazalek, A., & van den Hoven, E. (2009). Framing tangible 931. doi:10.1147/sj.393.0915 interaction frameworks. Artificial Intelligence for [18] Underkoffler, J., & Ishii, H. (1998). Illuminating Light: An Engineering Design, Analysis and Manufacturing, 23(03), Optical Design Tool with a Luminous-Tangible Interface 225–235. doi:10.1017/S0890060409000201 (pp. 1–8). Presented at the CHI 2006. [12] Nagel, T., Pschetz, L., Stefaner, M., Halkia, M., & Müller, B. [19] Underkoffler, J., & Ishii, H. (1999). Urp: A Luminous- (2009). mæve – An Interactive Tabletop Installation for Tangible Workbench for Urban Planning and Design (pp. 1– Exploring Background Information in Exhibitions. In 8). Presented at the CHI 2006. Human-Computer Interaction. Ambient, Ubiquitous and [20] Wakkary, R., & Hatala, M. (2006). Situated play in a Intelligent Interaction (Vol. 5612, pp. 483–491). Berlin, tangible interface and adaptive audio museum guide. Heidelberg: Springer Berlin Heidelberg. doi:10.1007/978-3- Personal and Ubiquitous Computing, 11(3), 171–191. 642-02580-8_53 doi:10.1007/s00779-006-0101-8 [13] O'Malley, C., & Fraser, D. S. (2004). Literature Review in [21] Wolter, D. F. (2014). InnoZ: Schwerpunkte. Innoz.De. Learning with Tangible Technologies. Retrieved September 9, 2014, from http://innoz.de/schwerpunkte.html