In this paper we present a case study for an interactive surface, an explanatory tangible multitouch system in an exhibition context. The system is used as an expressive representation of the complex concept of Micro Smart Grids (MSGs) and their influence on our future energy infrastructure. We built the experience upon research on integrating tangibles into playful and exploratory learning environments. Beyond the well-researched learning experience with tangibles, we developed the system to stimulate and foster discussions on the subject of MSGs. Besides the technology behind the system we will discuss limitations and advantages of combining tangible and multitouch technologies. To conclude we present reflections on the design and interaction concepts that were embedded into the system.
Global warming, sustainability and lately resilience are topics that matter to an audience beyond the scientific community. Due to the growing interest and the urgency presented by these topics, new tools for scientific communication are required. Tools that allow scientists as well as decision and policy makers to communicate complex issues to non experts, to foster an inclusive discussion on the basis of facts and scientific results.
The Berlin based research institute InnoZ is focusing on
those topics, especially how transport, energy and information and
communications technologies (ICTs) are involved in shaping the
urban future and how new products and technologies in these
areas of research are adopted by society
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Copyright is held by the owner/author(s). In 2012 the InnoZ explored possibilities to extend their educational material to explain the topic of MSGs to other scientists, the public, as well as decision and policy makers. The InnoZ campus itself is a MSG, which is made visible by live data shown on a control terminal. The envisioned tool should go a step further and explain the structure of MSGs as well as their impact on our future energy infrastructure.
The design process followed the principles of human
centered design
The initial aim of our designs was to create a tool that would
allow the research team to use the proposed system as an
explanatory system to educate visitors on the matter of MSGs. As
a basis for the explanatory system we created a virtual model of a
MSG as well as a model of the surrounding energy infrastructure.
The model was built upon statistics and estimations on energy
consumption and production in Germany. On top of the model
visualization was implement to explain the workings of the
underlying model, respectively a MSG. The components of the
model, consumers, producers, storage components and
MSGcontroller were represented by tangible tokens that could freely be
combined by the users. In this paper we will refer to the tangible,
physical objects that are placed on the interactive surface as
tokens. Those tokens were an attempt to overcome the initial
barrier of interacting with a complex system. Thus we made use
of the attractiveness and playfulness of tangible interactions
As explained above, the basis for the system is a complex model based on statistics and estimations of energy production, consumption and storage. By analogy with the collected data the model is made up of three categories of components: consumers, producers and energy storage. The consumers are private households and industrial consumers, their consumption is dependent on the external parameters daytime and time of year, e.g. during daytime and especially during the winter the consumption is higher. The energy producers are divided by fossil fuels including nuclear energy and renewable energy. The latter are, similar to the consumers, dependent on external parameters. This includes e.g. solar energy’s dependency on cloudiness or wind energy’s dependency on wind speed. The category of energy storages holds electric cars, water pumps, natural gas energy converters as well as large batteries. In addition to the three main components a forth component, the MSG-controller is available to create a MSG.
Each component is represented by several tokens that can be combined freely by the users. For a more intuitive interaction the tokens belonging to the same category share the same form factor: triangle = producer, circle = consumer, square = storage, hexagon = MSG-Controller (see Figure 2). The specific identity of the component is printed on top of the token in the form of an icon. 10 years ahead and in 20 years ahead. The three scenarios influence the datasets for the components behind the model. All parameter “states” are not only visually supported by highlighting the current state in the control bar, but also the visual appeal is changing. For example modifying wind speed changes the speed of the clouds or changing daytime changes the brightness (see Figure 3). This allows the spectators to create a connection between the changes of parameters and the changes within the system, e.g. less brightness equals less light and thereby less solar energy.
The system displays two types of relationships between the components added to the current state through the tokens. On the one hand the information exchanged between consumers, producers and storage components are being visualized through fine grey lines and on the other hand the more important part, the flow of energy between the components, is visualized through a flow of particles. While of course the speed of transporting the energy is always the same, the visualization makes use of particle density within the stream to indicate higher and lower production levels and changes the color of the particles to allow users to differentiate between fossil (orange) and renewable (green) energy (see Figure 4).
As mentioned in 2.3.1 the model has several parameters that influence the individual components. The whole model is running on a time loop of day/night as well as months, which influences the energy consumption as well as the production of energy e.g. through solar cells. In addition to the time data, the weather parameters wind and clouds can influence the energy production of the renewable energy producers. Beyond daytime and time of the year, the system offers three time scenarios: present-time, in
The whole system consists of several layers of information.
The first layer consists of the tokens representing the components
and their connections through the visualization of the model. The
second layer is visualizing the “state” of the parameters that
influence the system. The third layer is a layer of information that
can be activated per component on demand, it shows more
detailed information on every component (see Figure 5). The
users can use the last layer of information to read short texts about
every component and learn more about their impact on the energy
infrastructure through visualized statistics. Those layers represent
levels of information complexity, starting from the lowest level of
learning how MSGs work in general. Down to the level of facts
e.g. how much energy is produced by a certain power plant and
the estimated production throughout the next 20 years. Through
this we allow the user to dive deeper into the system and thereby
allowing a deeper engagement
The user can start her journey of exploring the system with a blank plate or with a template that works like a puzzle game, in which the user sees 2D tokens on the screen and needs to place the 3D markers on top of it and by that assembling a basic setup to further explore the system. Through those two modes the user can receive different levels of guidance.
By placing a token on the table, a visual 2D representation of
the token shows up below the token, the shape is corresponding to
the shape of the token to create a visual connection between token
and visualization. Furthermore, the two-dimensional
representation behaves according to the category it represents:
producers are generating energy particles and distributing those to
available consumers; consumers are aggregating energy particles
and consuming them; storage components are aggregating unused
energy to store them. The storage components differ from the
other components as their 2D representation have a visual
indicator of the amount of energy that is stored in the individual
component. During the design process the responsiveness and
visual correspondence between tokens and visualization played an
important role, as we tried to achieve a high “perceived coupling”
effect to create an intuitive interaction with a feeling of “direct
haptic manipulation”
The MSG-Controller plays a significant role, as it is essential to understanding the workings of MSGs. If no MSG-Controller is present on the table, the components almost behave as if everything is connected to everything. As soon as the MSGController is added to the mix, the new token adds a sense of smartness and structure to the relationships and connections between the components. Through the interaction of adding and removing the MSG-Controller token, the user can instantly observe how the visualization and the structure in the underlying model beneath changes.
In order to create a more realistic setting, the user can modify the parameters described in 2.3.2, allowing for the audience to observe changes in the energy system according to changes in the environment. Again, similar to the effect of the MSG-controller, the instant modification of the model through the parameters allows the users to change quickly between different states e.g. night and day and thereby observe and learn what this means e.g. for solar energy production.
If the user is interested in learning more about a specific component, she can click on info buttons that float along the 2D representation of the components. A small window will pop up allowing the user to discover more information on that component.
In order to create a non-fragmented view of the visualization and the interactions happening, we had to create interfaces and visualizations that were accessible from every side of the table. While the main interaction through tangibles is possible from every side and the observations as well, the manipulation of properties is only possible from the two long sides of the table due to the limited space on the screen.
In recent years, several descriptive frameworks for tangible
interaction and tangible interfaces have been developed
Through the visual correlation of tokens and visualization on
the screen and the different visualization schemes we created a
strong coupling between the physical and the digital. We were
able to establish a consistent visual storytelling and an intuitive
and seamless interaction, following the pillars of tangible
manipulation and expressive representations. In terms of
expressiveness, our approach sees the tokens as essential parts of
the construction of the system through the users, even though they
don’t represent a change within the system as it would be required
according to the definition of expressiveness by Ullmer and Ishii
Papers that focus on the interaction with tangibles often only briefly explain the actual technology used for the experiments. This makes it hard for other researchers and practitioners to build upon the experience gained on the technology side. This is why we will spend the next section on elaborating the technological side of our project as well as the pros and cons.
For the multitouch table we employed a large LCD system as a display. Like most common multitouch systems that don't use capacitive touch technology, the system uses multiple HD infrared-enabled cameras to track touch events as well as fiducials (printed 2D codes). Using infrared tracking systems has the advantage of being able to use touch events as well as tangible objects both in combination with printed fiducials. Then again, the infrared technology has several downsides which become present in an exhibition context. Even though the technology uses the infrared light spectrum, it is still sensitive to other spectrums of the light. Changing light intensity in the surrounding sometimes calls for readjustment of the system, the same is true for light sources that heat up throughout the day or, even worse, direct sunlight. Sources of light and/or heat can create noise on the infrared image. In order to achieve very precise tracking results, which are required for tracking fiducials, the trackers need to be adjusted carefully. This sometimes conflicts with exhibition spaces being lively spaces. An interactive piece build for group interaction will be under a lot of physical friction. Sometimes exhibits additionally need to be relocated e.g. for rearranging or special events. All those factors have an impact on the system and often create need for readjustments.
The design of the tangible objects is quite limited by the
required integration of the printed fiducials. A big problem of
tangible tokens for interfaces is visual or tactile feedback, as Ishii
and Ullmer illustrate “unlike malleable ‘bits’, ‘atoms’ are
relatively inflexible”
Our interactive system stands in a line of projects dealing with
tangible interfaces that visualize flow and relationship. Most
projects from the category of what Ishii and Ullmer call
"interactive surfaces" or "workbench surfaces"
The first category consists of projects that create tangible
interfaces, or rather tangible interactive visualizations, which
build upon real-world models. These projects use attributes that
exist in the real world and map them onto the interface or surface.
The projects try to keep the abstraction between the real-world
model and the artificial model as small as possible in order to
create an exploratory tangible system which, according to Marshal
et al, allows for the learner to explore the system through
practical exploration and thereby learn more about the underlying
structure
The most prominent example is Urp — the urban planning
workbench
Illuminating Light, published one year earlier by the same
research group at MIT
The second category consists of projects, that similar to the first category, build upon real-world models, but rely on a more abstract visualization, as the real-world representation is too complex to visualize. Instead of showing the real connections between the components of the model, the visualization focuses on a representation that displays the relationship between components and their influence on each other, allowing the user to manipulate those influences and relationships and thereby foster deeper understanding.
In the area of sound- or music-interfaces the most cited case
study is the Reactable by Reactivision
The IP network simulation is using a similar approach for
visualizing IP networks
The last project from the subcategory of projects that focus
on visualizing relationships between components is the Maeve
project
Our reflections derive from the observation of users interacting with the system. The evaluation was collected through interviews with three designers who were involved in the development and the deployment, as well as two instructors who work with the table in the showroom.
Even though tabletop multitouch systems are frequently used e.g. in exhibitions or trade fairs, those systems are still able to create a WOW-effect among visitors. At the same time, the widespread use of multitouch technology in smartphones, tablets and even household appliances creates similar expectations towards big multitouch systems in regards to interactions. A good example to illustrate this is the design of the information popups. In one of the first prototypes, those windows were programmatically floating elements, but the tests showed that users were trying to apply common multitouch interaction patterns, like “pinch” to zoom and “drag” to move the window (see Figure 6). In the next iteration we implemented those interactions, which helped creating a more fluent interaction.
An observation reported by all interviewed persons was the
tabletop’s function as a catalyst for social interaction, especially
fostering discussions among the visitors. The interaction was not
only between visitors and instructor but also among the groups
themselves. Beyond verbal discussions that arose, common
observations were the physical collaborations while using the
table, originating from the “non-fragmented” visibility of the
visualization, giving it an overall performative quality
As intended by the design, the physical tokens were not only an eye-catcher, but also attracted attention and interest leading to an immediate interaction with the tokens and thereby the system. The tokens had this effect on children and young people as well as on adults. Nonetheless, it was reported that the intuitive usage and playful exploration was more powerful on the younger generations. They seemed to understand the workings of the model a lot faster than adults through their more playful interaction with the table.
It is hard to elaborate on the long-time learning effect triggered by the table as the groups visiting the space are only using the table for a limited time. Accordingly, the envisioned deeper layers of information were not accessed in many cases. Most users concentrated on the basic structure of the MSG. Beyond the factor of available time, we believe that the user’s limited use of access to the detailed information also depends on the fact that the process of accessing the detailed information is a very focused interaction that in turn is hard to share with a bigger group. In contrast to the focused interaction the token-based interaction can be executed in a collaborative manner and the results can easily be visually shared with a larger group.
The MSG tabletop allows users to explore a complex realworld model through a playful and explorative interaction. The results from the observation show that the system is a good case study to highlight some of the remarkable features of tangible interfaces. As other research underlines, we were able to observe that large tangible interactive surfaces provide fertile ground for social interaction and the exploration of real-world models. Furthermore we were able to observe that large tangible interfaces can foster discussions on the topic visualized in the interface. On the other hand we also observed that more focused single-user interactions were not well received by the users.
The area of learning through tangible interfaces in regards to
children is widely covered in the research community
The project was put to life by the IXDS team (Jonathan Cohen, Andre Knörig, Felix Lange, Sebastian Meier, Dimitar Ruszev, Julia Werner and Reto Wettach) in cooperation with the prototype - Moers Peter Sailer GbR who helped with the theoretical framework as well as an in-depth data aggregation. The technology, especially the display unit by multitouch.fi, and furniture was assembled by werk5 GmbH.
In a recent project we were able to use the latest multitouch and fiducial tracking technology and we were pleased to see that the dependency from ambient light was a lot better than with the technology we used for the system described in this paper. Nothing the less, the described problems remain, if only less severe.