Developing Multi-touch Software through Creative Destruction

Abstract

Tangible and Gesture-Based Interfaces are made possible by hardware like the Microsoft® Surface Device. Despite the commitment of big business, the killer application is yet to be found. To address this challenge, we investigate the innovation process. Against this background, a workshop was set up and conducted. The clarification of focuses during brainstorming is achieved by introducing a triad of innovation, which revealed potentials for entertaining multi-touch applications. We respond to these potentials with the custom-made solutions realized during our workshop.

-

MOTIVATION Natural user interface, Gesture-based Interfaces, Software design, Creativity techniques Multi-touch technology is becoming available to customers today. The direct interaction paradigm supported by multitouch devices has been the focus of researchers for a long period of time [ 6 ][ 8 ][ 16 ], and is being transported into the mainstream with mature products like iPhone or iPad from Apple Inc. [ 2 ] or Laptops equipped with N-Trig® technology like the Touchsmart series from Hewlett Packard [ 9 ] or the Dell Latitude™ XT2 [ 4 ]. The Microsoft® Surface device demonstrates how the technology scales to larger setups, which will be available to consumers in the future. Multi-touch interfaces excel in user experience, which is more entertaining due to the direct, unencumbered interaction with digital content. BACKGROUND In contrast to the mature and robust hardware, software products for large-scale multi-touch devices are lagging behind. To overcome this discrepancy, we consider commoditisation as described by Simon Wardley [ 15 ]. We seek to adapt his conclusions to multi-touch technology. Figure 1 shows how novel ideas, generated from discoveries or inventions, are transformed into common and ubiquitous appliances that are central to the development of a service infrastructure. This can be illustrated by the example of electricity [ 15 ]. From its inception, there have been innovative inventions and prototypes. This was followed by custom-made solutions to power specific appliances. Products like electric generators or the light bulb were followed by a network of different services offered by power companies, providing the appropriate infrastructure to make electricity a ubiquitous phenomenon. With multi-touch technology, we are currently seeing a lot of custom-made solutions. In this contribution, we present approaches to exploit custom-made solutions to generate knowledge and reusable components in order to evolve into the stage of products. Especially the entertaining qualities and ease of use of these products is the key to their success. In order to generate knowledge concerning the development of multi-touch software, it was decided to build on the key instruments of commoditisation. The goal In order to generate knowledge concerning the development of multi-touch software, it was decided to build on the key instruments of commoditisation. The goal was to exploit creative destruction and adaptation of existing solutions (cp. Figure 1). As commoditisation shows, there is an increasing pressure on the developers to gain advantages in the innovation process. WORKSHOP SETTING As a consequence of the previous considerations, a workshop was set up by a group of researchers and business experts in product development with the following terms and conditions. Ten students were picked from applicants with knowledge in software developing and fine arts. None of them had any experience with multi-touch technology so far. The goal was to support creative destruction, in contrast to the more technical approach of professionals in the field of software development. The initial kick-off meeting included a traditional brainstorming [ 12 ]. Thoughts were organized with regard to different focuses in a triad of innovation (cp. Figure 2). Multi-touch technology presents certain challenges, e. g. the orientation of text in a bargaining situation. On the other hand, concrete solutions are made possible by the properties of multi-touch tables, e. g. collaborative work with simultaneous interaction on one device. Other ideas are of a more conceptual nature. This is the case for interfaces based on tangible objects or 3D modelling software. Creative destruction was enforced by presenting the innovative technology to students without going into technical details and limitations the technology currently exhibits. As a result, the capabilities were questioned and sometimes even overestimated, revealing requirements for future versions of the hardware. For instance, identification of unique users and mapping of touch interaction to user’s hands was discussed. Some of these modalities already exist in research setups [ 5 ]. After the initial brainstorming, the students were acquainted with the concrete features and capabilities of the multi-touch table. Adaptation of these properties was crucial to implement the ideas from the brainstorming (cp. Figure 1).

Students were asked to choose their focuses and refine the scenarios to an application and a working prototype. This was realized by providing a Microsoft® Surface Device and workstations running Windows Vista and Visual Studio with the Surface SDK [ 11 ]. As a result, the following technical capabilities were addressed: • • • •

Multi-user setup of table allows collaborative work Physical objects can be used for interaction (tagging)

Recognition of 52 simultaneous touch contacts

Bluetooth, Infrared and Wireless LAN available

Over the course of 14 days, five teams of two students worked on one project each. Each day commenced with a status report in order to continuously evaluate the process of creative destruction by the workshop advisers and to allow peer review among the students. The first week was focused on developing conclusive concepts and getting to know the technical properties of the Microsoft® Surface Device. The second week was dedicated to the implementation of the prototypes. Our initial non-technical approach enhanced the creative quality of the resulting applications. In addition, advisers offered consulting to support the process of creative destruction.

During the workshop the standard Surface SDK was adapted to the requirements of each individual approach. At this point, knowledge about the workflow in software development for multi-touch technology was generated. New controls that adapt and extend the functionality provided by the Surface SDK are introduced.

In the next section, we present the five applications and their underlying scenarios that resulted from our workshop of creative destruction.

CASE STUDIES This section describes the custom-made solutions designed and implemented during the workshop (cp. Figure 1). Different modes of interaction emerged from each application. A reference to all interaction modes is provided in Table 1, along with the specific switching patterns used in each scenario. This is further proof that universal concepts like WIMP (Windows, Icons, Menus, Pointing devices) have to be abandoned in order to design multi-touch interfaces [ 3 ]. Individual solutions adapted to their specific requirements are advisable.

Navigatable

Tangible

Design Helper

SurfaceReader

MySpace

TagIt

Modes of interaction

Navigation vs. modification

Presentation vs. edit

Alignment vs. annotations

Collaboration vs. private work

Tagging vs. filter

Switching pattern

Gesture based in 3D content (using a five fingers gesture)

Tangible based (put down interaction PUK)

Menu based (tap on icon)

Gesture based in

2D content (pull forward gesture to bring up private workspace )

Tangible based (interaction with content or application area)

Navigatable

The Navigatable application was focused on the challenge of adapting natural human gestures to the multi-touch scenario. Human gestures are three-dimensional, similar to 3D computer graphics. In this context, multi-touch interfaces constitute a dimensional discrepancy. Data from a two-dimensional interface has to be interpreted and mapped to a three-dimensional space. In our approach, users can switch between the navigation and modification modes by using a five fingers gesture performed with one hand (Figure 3, top picture). As visual feedback, the background changes to a darker shade of grey. Auxiliary objects are used to translate models on each of the three axes in space. This is done by dragging one of the coloured manipulation spheres (Figure 3, bottom picture). Multi-touch is further exploited by dragging two manipulation spheres simultaneously. Consequently, the translation axis in space is defined between the two spheres. Tapping on a manipulation sphere rotates the selected object or part of the object to the corresponding section of the model.

By putting the whole hand on the surface, every modification is reversed and the model is presented without auxiliary objects. Models are loaded into Navigatable by putting actual file cards on the table. They show a preview picture of the model on one side and a tag is applied on the flipside for recognition purposes.

In navigation mode, the familiar pinch gesture with two fingers is used to zoom in and out. Rotating is performed with one finger and a swiping motion and panning of the camera is realized by a three fingers swipe. It is possible to select an object in the scene by tapping on it. This will change the camera focus to the selected object and the pivot point for rotating is set to the object’s centre.

Navigatable answers to the challenge of using the twodimensional interaction surface of a multi-touch table as a proxy for three-dimensional objects. In this innovative undertaking, a set of tools and gestures were defined, which can be applied to 3D modelling software in general. The seemingly complex gesture interaction necessary to navigate in three-dimensional space is easily learned due to the entertaining qualities of multi-touch interaction. They consist of direct feedback and unencumbered contact with the multi-touch surface, which displays digital content. Tangible Design Helper Designing a laptop cover in a collaborative, tangible based manner is possible with our Tangible Design Helper application. Existing design templates are available and custom pictures and content can be added.

Apart from gestures based on fingers, the program employs a tangible input device. The Personal User Key (PUK) is used to interact with the system and constitutes an evident element to avoid conflicts between users. Only one user is able to use the PUK at any time, in order to allow undisturbed interaction. Removing the PUK from the interaction surface disables all editing features and switches into presentation mode. Standard gestures to rotate and to zoom in and out of the laptop cover are available to the users surrounding the multi-touch table.

In interaction mode, elements on the cover can be selected and resized by all users with finger gestures. The PUK is managed by one user at a time and offers different tools to change fonts and colors (Figure 4, bottom picture) and allows inserting of new elements (Figure 4, in the middle). Tangible Design Helper incorporates simultaneous interaction of multiple users on the laptop cover and the PUK as a singular element to avoid conflicts. Natural understanding of the users at the table is exploited to reach agreements regarding the management of the PUK and its functionality.

For instance, this tool can be used in a store to personalize a product in a comfortable and spontaneous way. Multitouch tables are often used in business to consumer settings. Tangible Design Helper demonstrates sales conversations with actual participation of the customer. This custom-made solution is also a showcase for point of sale situations in which the setup of physical objects on the interaction surface can influence the system state (cp. Figure 1). The SurfaceReader application enables users to read and annotate electronic documents in a collaborative manner. Documents are duplicated for each user and aligned to be read properly. Annotations are made by switching into annotation mode (Figure 5, bottom picture). To this end, a menu is located at the edge of each document. A master key (red circular button in Figure 5, top picture) can be dragged onto one of the duplicates. The navigation and annotations are reflected on each of the other documents. The SurfaceReader realizes a bargaining table with electronic documents and can be extended to a distributed setting. Documents can be shared on several tables via network connection. Cooperative work as well as teaching and learning are realized and assisted by the concepts introduced in this solution.

TagIt The TagIt application supports users in tagging their photo collections. Tangible objects shaped like stamps are used. Tagging is facilitated, and users are motivated to actually organize and classify their photo collection. Collaborative tagging is possible by employing more than one stamp. Each stamp is individually associated with a set of tags. Users can activate their stamp-pad from the side of the table (on the bottom and right of Figure 6, top picture). New tags can be defined (Figure 6, bottom picture) and combined to be associated with a stamp. Besides the viewing and tagging mode, it is also possible to put a stamp on the background of the photo application. The filter mode is activated, and only photos matching the tags of the particular stamp are displayed. The TagIt application demonstrates how skills, rules and knowledge of users can be exploited in natural user interfaces [ 13 ]. The notion of indexing content by stamping is a cultural tradition that has been known even before printing was invented by Gutenberg in the 15th century. The concept has proven to be stable even in applications like Adobe’s Photoshop® toolbox. Multi-touch rediscovers the actual manual process. MySpace The MySpace application targets collaborative work on multi-touch tables. Simple whiteboard or brainstorming activities can be assisted by introducing personalized, private workspaces and a shared workspace for collaborative interaction.

Personalized tags identify users and serve to provide data and content. Private areas are activated at the edges of the table by putting down a user tag (Figure 7, top picture). Private workspaces allow interaction only from the corresponding location of the user. This is achieved by matching the finger orientation provided by the Surface SDK to the correct workspace [ 14 ]. Pictures, text, and other objects can be arranged in private workspaces. To make any object publicly available for collaborative manipulation, it is dragged into the shared workspace with three fingers. The ring menu (Figure 7, bottom picture) is used to store objects and to access manipulation tools. For instance, text labels can be associated with each other by means of the connector tool. Moreover, this ring menu is fully customizable and refactored to be reused in other applications. If one of the private workspaces is not needed, it can be minimized by using a four finger pull gesture. This increases the size of the shared workspace. It also frees up space for the private workspaces of other users. Similar to the SurfaceReader application, a distributed version of the MySpace application is conceivable. Users in various places can work with several multi-touch tables on joint activities. To this end, personalized tags could be used to carry private content and the individual workspace layout from one table to the other. This supports the knowledge management of each user over time and different locations.

CONCLUSIONS AND FUTURE WORK In this contribution, we presented a working approach to leveraging the potentials of multi-touch technology. The workshop setting we described can help to put forth innovative and sustainable concepts. As a rationale for this setting, the framework of commoditisation with its key instruments of creative destruction and adaptation was exploited (cp. Figure 1). In Figure 2, thoughts and ideas are arranged according to their focuses on conceptual work, challenges and solutions. Prior to the definition of custommade solutions, this triad revealed potentials of multi-touch technology. This is one approach to support creative destruction. Other techniques, such as TRIZ [ 1 ] are available. However, due to the complexity, it is not suitable for the presented workshop setting. Similarly, methods like Synectics [ 7 ] require more time and training to be effective. In the future, we seek to further deconstruct existing technical solutions. In a following workshop, fundamental capabilities of multi-touch interaction will be investigated. Further custom-based solutions can be developed and transformed into stable and mature products.

ACKNOWLEDGMENTS Thanks are due to the staff of Professur Mediengestaltung at Technische Universität Dresden for supporting the practical work of the students. We are indebted to the inspiring and motivated work of the students who participated in the workshop: Claudia Zimmer, Kristin Dietze, Marie Schacht, Tino Winkler, Christian Klauss, Julian Eberius, Martin Herrmann, Frank Harnisch, Mathias Müller and Florian Schneider.

Finally, thanks are due to T-Systems Multimedia Solutions GmbH for supplying the Microsoft Surface Device and providing technical assistance during the workshop.

References

1. Altshuller , G. 1973 . The Innovation Algorithm: TRIZ, Systematic Innovation and Technical Creativity , Original publication in Russian. English translation by L. Shulyak , S. Rodman , Technical Innovation Center, Inc., USA, 1999 .

2. Apple Inc . Apple - iPhone - Mobile phone, iPod, and Internet device . Website. Available at: http://www.apple.com/iphone/. Retrieved 2/24/ 2010 .

3. van Dam, A. 1997 . Post-WIMP user interfaces . Commun. ACM 40 , 2 (Feb. 1997 ), 63 - 67 .

4. Dell Latitude Laptops . Website. Available Online: http://www.dell. com/latitude. Retrieved 2 /24/ 2010 .

5. Echtler , F. , Huber , M. , and Klinker , G. 2008 . Shadow tracking on multi-touch tables . In Proceedings of the Working Conference on Advanced Visual interfaces (Napoli , Italy, May 28 - 30, 2008 ). AVI '08. ACM , New York, NY, 388 - 391 .

6. Fitzmaurice , G. W. , Ishii , H. , and Buxton , W. A. 1995 . Bricks: laying the foundations for graspable user interfaces . In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Denver, Colorado, United States, May 07 - 11, 1995 ). I. R. Katz , R. Mack , L. Marks , M. B. Rosson , and J. Nielsen, Eds. Conference on Human Factors in Computing Systems . ACM Press/Addison-Wesley Publishing Co., New York, NY, 442 - 449

7. Gordon , W. J. J. 1961 . Synectics: The development of creative capacity . Harpercollins College Div.

8. Hauptmann , A. G. 1989 . Speech and gestures for graphic image manipulation . SIGCHI Bull. 20 , SI (Mar. 1989 ), 241 - 245 .

9. Hewlett-Packard Development Company, L.P. HP Touchsmart PCs . Website. Available at: http://www.hp.com/unitedstates/campaigns/touchsmart/. Retrieved 2/24/ 2010 .

10. Microsoft®. Microsoft® Surface. Website. Available at: http://www.microsoft.com/surface/. Last Checked: 2 /24/ 2010 .

11. Microsoft® Developer Center . Microsoft Surface SDK . Website. Available at: http://msdn.microsoft.com/enus/library/ee804845.aspx. Retrieved 2 /24/ 2010 .

12. Osborn , A. F. Applied Imagination . Scribner, June 1979

13. Rasmussen , J. 1987 . Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models. In System Design For Human interaction, A . P. Sage, Ed. IEEE Press, Piscataway, NJ, 291 - 300 .

14. Wang , F. and Ren , X. 2009 . Empirical evaluation for finger input properties in multi-touch interaction . In Proceedings of the 27th international Conference on Human Factors in Computing Systems (Boston, MA, USA, April 04 - 09 , 2009 ). CHI '09. ACM , New York, NY, 1063 - 1072 .

15. Wardley , S. Innovation, the Future and Why Nothing is Ever Simple . Available at http://www.capgemini.com/technologyblog/2008/12/simon_wardley_and_innovation_l.php. Retrieved 2010 - 02 -19.

16. Wolf , C. G. and Morrel-Samuels , P. 1987 . The use of hand-drawn gestures for text editing . Int. J. Man-Mach. Stud . 27 , 1 (Jul. 1987 ), 91 - 102