Towards a Periodic Table of Gestural Interaction Dietrich Kammer, Mandy Keck, Rainer Groh Technische Universität Dresden 01062 Dresden dietrich.kammer@tu-dresden.de ABSTRACT of natural sciences, cannot be established in applied The periodic table is the first means of chemists to structure sciences such as human-computer interaction. However, it their field of research. It comprises all chemical elements is still an interesting and beneficial endeavor in order to with their most prominent features. The system is so advance and consolidate the field of research. The proposed important that students of chemistry usually learn it by heart periodic table does not claim to have an immediate during their university education. Scientists in the field of application for engineering multimodal gestures. Its value is human-computer interaction lack such a concise system that on a meta-level, in order to identify differences, covers all aspects of gestural interaction. Although commonalities, and requirements for engineering software considerable research exists, scientists rarely agree on that processes gestural input. common aspects and systems to classify, collect, and share their research. This position paper attempts to pacify The periodic table of gestural interaction requires rivaling scientific views towards gestural interaction and its researchers to think in terms of Semiotics: what are the properties, benefits, and applications. By collecting meta- fundamental syntactic elements that constitute a gestural properties of multimodal gestures, the proposed periodic system and how are they combined, interpreted, and table tries to provide a common ground for classification received by users? We first provide a brief background on and debate among researchers and practitioners working the periodic table of chemical elements and then address with gestural interfaces. existing research on gestural interaction, followed by an attempt at constituting a first draft of the periodic table for Author Keywords gestural interaction. Gestures, Multimodal, Classification, Formalization. BACKGROUND ACM Classification Keywords The periodic table of the chemical elements is a table that D.2.2 Software Engineering: Design Tools and Techniques; registers elements and their atomic numbers, electron H.5.2 Information Interfaces & Presentation: User configurations, and other chemical properties in a tabular Interfaces manner [17]. It is not only a collection of known elements; due to its layout according to physical and chemical rules, it INTRODUCTION also serves to predict further elements that have not yet Scientists in the field of human-computer interaction rarely been synthesized or discovered. agree on common aspects of gestural interaction. While this Eighteen columns and seven rows constitute the periodic attitude serves to investigate various routes and illuminates table. There is a double row of elements below those different key aspects, the consolidation of knowledge is columns and rows (see Figure 1). Colors signify if the necessary to advance and consolidate a field of research. In elements belong to a metal category or non-metal category. this position paper, we propose to collect and agree on certain meta-properties of different gestural interaction Rows and columns are called periods and groups, styles. respectively. Some of the groups have special names like halogens or noble gases. Groups show trends with respect A periodic table of gestural interaction is the main to the contained elements, i.e. common properties such as metaphor, which serves as motivation to classify atomic the same electron configuration in the outermost shell of the gesture building blocks and debate their properties. The atom. Periods represent less important trends: atomic periodic table is the main tool for chemists to structure their radius, ionization energy, electron affinity, and field of research. It is so important that undergraduate electronegativity. students are required to learn it by heart. It is clear that such a profound system, which is grounded in fundamental facts There are different layouts of the periodic table of chemical elements, for instance, the lanthanides and actinides can be EGMI 2014, 1st International Workshop on Engineering Gestures for integrated, which makes the table considerably broader. Multimodal Interfaces, June 17 2014, Rome, Italy. Due to different requirements and views, several layouts Copyright © 2014 for the individual papers by the papers' authors. Copying permitted only for private and academic purposes. This volume is and categorizations are possible with regard to the chemical published and copyrighted by its editors. elements of the periodic table. http://ceur-ws.org/Vol-1190/. 30 RESEARCH ON GESTURAL INTERACTION It is out of the scope of this paper to collect the complete research that exists on gestural interaction. However, in order to setup a first draft of the intended periodic table, it is necessary to address a number of approaches that seek to consolidate knowledge on gestural interaction. Declarative approaches to specify multi-touch gestures can be found in the literature such as GDL [12,13]. Other formalization attempts for multi-touch gestures are GeForMT [9,10] and Proton [14,15]. Wobbrock et al. consider further aspects towards a taxonomy of multi-touch gestures [23]. GISpL [5] and Mudra [8] address multimodal gestures. The Behaviour Markup Language is Figure 1: Structure of the periodic table of the chemical an XML dialect to describe multimodal and spatial gestures elements. [22]. The Conversational Gesture Transcription system also The chemical elements themselves are described by their uses a formal notation to describe spatial human gestures atomic number, which directly refers to the electron [21]. For sketching gestures, a sketch language has been configuration. The designated name is often in Latin and is developed by Bimber et al. [2]. Another domain-specific the foundation for the official symbol, which is an language for sketching has been proposed by Hammond internationally agreed code that consists of one, two, or and Davis [7]. Spindler et al. propose an interaction three letters. vocabulary for spatial interaction using magic lenses [18,19]. Epps et al. investigate different hand shapes, which can be used in spatial gestures [6]. All of the declarative approaches to describe gestures aim at facilitating the implementation of gestures used in specific interaction techniques. In the next section, we propose to collect these interaction techniques in a periodic table of gestural interaction. Figure 2: Chemical element with its configuration and designation. PERIODIC TABLE OF GESTURAL INTERACTION Most of the research in gestural interaction addresses the The periodic table has also been called “nature’s rosetta atomic building blocks of gestures and how they can be stone” [1]. There are other instances, where the periodic captured and processed. Interaction techniques developed table of chemical elements serves as a means to structure a by researchers use these building blocks for simple or field of research, such as the periodic table of visualization compound tasks in an interface. We propose to view these methods for management [16]. This periodic table uses interaction techniques as the elements of our periodic table. concrete information visualization methods as basic The periodic table should reflect some basic distinctions of elements. Coloring is used for categories, while rows show the gestural interaction techniques such as: a basic trend towards more complex visualization techniques. The categories are data visualization, • Degrees of freedom information visualization, concept visualization, strategy • Complexity visualization, metaphor visualization, and compound visualization. The columns have no explicit meaning, but • Continuous or discrete evaluation show another trend towards more complex and compound visualizations. Atomic numbers are omitted and the main • Hardware, i.e. enabling technologies symbol or abbreviation is put in the center of each element. The complexity used in our periodic table refers to the Within each element, the coloring of the symbol shows subjective intricacy involved in an interaction technique. whether the visualization method is a process or structure For instance, a technique involving two hands or multiple visualization and icons express if the visualization method fingers can be rated more complex than a simple tap with is used for overview, detail, divergent thinking, or one finger. It is important to note that the intended periodic convergent thinking. These symbols can be combined and table makes most of the distinction from the point of view inspired the icons used in the periodic table of gestural of the sensing technology. Hence, we will not consider the interaction. movement necessary to reach a button on a keyboard. The same goes for reaching out to a multi-touch screen. 31 Figure 3: First draft of a periodic table for gestural interaction, the elements correspond with different interaction techniques. line with our work, which focuses on lexicalic and Ta pragmatic aspects of input structures [3,4]. INTERACTION TECHNIQUE TAP The first draft of our periodic table in Figure 3 groups the interaction techniques according to their enabling Information Position and technologies and complexity, as well as with regard to the mode (number of degrees of freedom. The degrees of freedom (DOF) cannot fingers, duration, be expressed with specific numbers and represent a general repetition) tendency in the table: from 2D space to 3D space. It is also Formalization 1F(POINT) or conceivable to relate input DOF to output DOF in order to achieve a more sophisticated layout. 2F(POINT) In addition, the complexity of the different interaction Use case Activation, techniques is open to debate. Those techniques of similar selection, context complexity should be found on the same row in the table. menus, The axis for complexity is chosen similarly to the periodic information table of visualization methods for management [16]. Hence, Visualization Short highlighting simple techniques are found in the top rows of the table and Feedforward of interactive more involved interaction techniques are situated at the interface bottom. elements A name, a short symbol, and a number of properties, which are represented by icons, describe each interaction Interaction Manipulation technique. The interaction techniques such as tap, scrolling, goals and panning are already established quasi-standards. Other Restrictions Size and techniques such “lense tilt” or “lense move” are currently precision of being researched [20]. The properties expressed by icons in fingers each element refer to the use of the technique in a continuous (online) or discrete (offline) manner. Table 1: Crib sheet for interaction technique “tap” Furthermore, a distinction in interaction techniques for navigation and manipulation tasks is made. There are other frameworks for gesture-based interactions, which also integrate application domains and concrete design guidelines [11]. Early work of Buxton is more in 32 Scr Pi INTERACTION TECHNIQUE SCROLLING INTERACTION TECHNIQUE PINCH Information Direction of Information Centre of movement gesture and (horizontal or relative vertical) adjustment of distance Formalization 1F(LINE) between both Use case Navigation in fingers lists, interfaces Formalization JOIN[1F(LINE) with dynamic * 1F(LINE)] or dimensions SPLIT[1F(LINE) Visualization Truncated list * 1F(LINE)] Feedforward items, Use case Zoom in and consecutive zoom out, numbering Scaling of Interaction Navigation objects goals (orientation) Visualization Truncated Restrictions Long Feedforward interface, short interfaces or fade in of scroll texts need bars optimization Interaction Manipulation goals (Scaling) or Table 2: Crib sheet for interaction technique “scrolling” Navigation Each interaction technique should be accompanied by a (level of detail) more detailed crib sheet, which gives a short overview of Restrictions Used in the technique, its use cases, and application. Table 1 gives combination an example of such a crib sheet, which also uses GeForMT with panning to provide a formal expression of the multi-touch gesture involved in the interaction technique [9]. Table 3: Crib sheet for interaction technique “pinch” GeForMT uses a simple math formula syntax involving contact functions such as 1F(…) to express that 1 finger FUTURE WORK touches the multi-touch surface. Atomic gestures describe The periodic table can be used to codify knowledge in the certain movements such as lines (LINE) or circles field of gestural interaction. Especially with the advent of (CIRCLE) or static contacts such as POINT for a simple multimodal interfaces, this becomes increasingly important. touch of the surface or HOLD for a longer contact. Other An interactive periodic table could be established on the formalizations or notations are conceivable as well and internet, which would allow cooperative work on building a should be provided in order to exchange gestures across knowledge base on gestural interaction. Furthermore, an different frameworks. interactive table allows drilling down on the elements and Table 2 and 3 show additional examples of crib sheets for show detailed information, such as the implementation with multi-touch interaction techniques. The examples show use various declarative approaches to describe multimodal cases when these interaction techniques are suitable and gestures. how they can be visualized in an interface. Thus, a Researchers should try to classify their developed feedforward is suggested for the developer of a gestural interaction techniques according to an established set of interface. The restrictions listed in the crib sheets can alert concerns as used in the periodic table. In the following, the the developer if certain problems exist when using an table can be adapted and optimized. Especially if the interaction technique, or if there are interplays with classification breaks down or becomes ambiguous, additional techniques. additional rules need to be devised in order to achieve a sound assessment of complexity and DOFs used in an interaction technique. 33 ACKNOWLEDGMENTS 14. Kin, K., Hartmann, B., DeRose, T., and Agrawala, M. The European Union and the Free State Saxony through the Proton: multitouch gestures as regular expressions. European Regional Development Fund (ERDF) supported Proceedings of the 2012 ACM annual conference on this work. Thanks are due to Axel Pötzsch for his support of Human Factors in Computing Systems, ACM (2012), this research. 2885–2894. 15. Kin, K., Hartmann, B., DeRose, T., and Agrawala, M. REFERENCES Proton++: a customizable declarative multitouch 1. Baum, R.M. Celebrating the periodic table. Chemical framework. Proceedings of the 25th annual ACM & engineering news 81, 36, 28–29. symposium on User interface software and technology, 2. Bimber, O., Encarnacao, L.M., and Stork, A. A multi- ACM (2012), 477–486. layered architecture for sketch-based interaction within 16. Lengler, R. and Eppler, M.J. Towards a periodic table virtual environments. Computers & Graphics 24, 6 of visualization methods of management. Proceedings (2000), 851–867. of the IASTED International Conference on Graphics 3. Buxton, W. Lexical and pragmatic considerations of and Visualization in Engineering, ACTA Press (2007), input structures. ACM SIGGRAPH Computer Graphics 83–88. 17, 1 (1983), 31–37. 17. Scerri, E.R. The periodic table: a very short 4. Buxton, W. Chunking and phrasing and the design of introduction. Oxford University Press, Oxford; New human-computer dialogues. Proceedings of the IFIP York, 2011. World Computer Congress, (1986), 475–480. 18. Scholliers, C., Hoste, L., Signer, B., and De Meuter, 5. Echtler, F. and Butz, A. GISpL: gestures made easy. W. Midas: a declarative multi-touch interaction Proceedings of the Sixth International Conference on framework. Proceedings of the fifth international Tangible, Embedded and Embodied Interaction, ACM conference on Tangible, embedded, and embodied (2012), 233–240. interaction, ACM (2011), 49–56. 6. Epps, J., Lichman, S., and Wu, M. A study of hand 19. Spindler, M. and Dachselt, R. Exploring information shape use in tabletop gesture interaction. CHI ’06 spaces by using tangible magic lenses in a tabletop Extended Abstracts on Human Factors in Computing environment. CHI ’10 Extended Abstracts on Human Systems, ACM (2006), 748–753. Factors in Computing Systems, ACM (2010), 4771– 7. Hammond, T. and Davis, R. LADDER, a sketching 4776. language for user interface developers. Computers & 20. Spindler, M., Tominski, C., Schumann, H., and Graphics 29, 4 (2005), 518–532. Dachselt, R. Tangible views for information 8. Hoste, L., Dumas, B., and Signer, B. Mudra: a unified visualization. ACM International Conference on multimodal interaction framework. Proceedings of the Interactive Tabletops and Surfaces, ACM (2010), 157– 13th international conference on multimodal 166. interfaces, ACM (2011), 97–104. 21. Spindler, M. Spatially Aware Tangible Display 9. Kammer, D., Henkens, D., Henzen, C., and Groh, R. Interaction in a Tabletop Environment. Proceedings of Gesture Formalization for Multi-touch. Software: the 2012 ACM International Conference on Interactive Practice and Experience (in press), (2014). Tabletops and Surfaces, ACM (2012), 277–282. 10. Kammer, D., Wojdziak, J., Keck, M., Groh, R., and 22. Trippel, T., Gibbon, D., Thies, A., et al. CoGesT: a Taranko, S. Towards a formalization of multi-touch formal transcription system for conversational gesture. gestures. ACM International Conference on Interactive Proceedings of LREC 2004, (2004), 2215–2218. Tabletops and Surfaces, ACM (2010), 49–58. 23. Vilhjálmsson, H., Cantelmo, N., Cassell, J., et al. The 11. Karam, M. and Schraefel, M.C. A taxonomy of Behavior Markup Language: Recent Developments Gestures in Human Computer Interaction. ACM and Challenges. In C. Pelachaud, J.-C. Martin, E. Transactions on Computer-Human Interactions, André, G. Chollet, K. Karpouzis and D. Pelé, eds., (2005). Intelligent Virtual Agents. Springer Berlin / 12. Khandkar, S. and Maurer, F. A Domain Specific Heidelberg, 2007, 99–111. Language to Define Gestures for Multi-Touch 24. Wobbrock, J.O., Morris, M.R., and Wilson, A.D. User- Applications. Proceedings of the 10th Workshop on defined gestures for surface computing. Proceedings of Domain-Specific Modeling (DSM’10), (2010). the SIGCHI Conference on Human Factors in 13. Khandkar, S.H., Sohan, S.M., Sillito, J., and Maurer, F. Computing Systems, ACM (2009), 1083–1092. Tool support for testing complex multi-touch gestures. ACM International Conference on Interactive Tabletops and Surfaces, ACM (2010), 59–68. 34