Towards a Periodic Table of Gestural Interaction

Abstract

The periodic table is the first means of chemists to structure their field of research. It comprises all chemical elements with their most prominent features. The system is so important that students of chemistry usually learn it by heart during their university education. Scientists in the field of human-computer interaction lack such a concise system that covers all aspects of gestural interaction. Although considerable research exists, scientists rarely agree on common aspects and systems to classify, collect, and share their research. This position paper attempts to pacify rivaling scientific views towards gestural interaction and its properties, benefits, and applications. By collecting metaproperties of multimodal gestures, the proposed periodic table tries to provide a common ground for classification and debate among researchers and practitioners working with gestural interfaces.

-

INTRODUCTION Scientists in the field of human-computer interaction rarely agree on common aspects of gestural interaction. While this attitude serves to investigate various routes and illuminates different key aspects, the consolidation of knowledge is necessary to advance and consolidate a field of research. In this position paper, we propose to collect and agree on certain meta-properties of different gestural interaction styles.

A periodic table of gestural interaction is the main metaphor, which serves as motivation to classify atomic gesture building blocks and debate their properties. The periodic table is the main tool for chemists to structure their field of research. It is so important that undergraduate students are required to learn it by heart. It is clear that such a profound system, which is grounded in fundamental facts of natural sciences, cannot be established in applied sciences such as human-computer interaction. However, it is still an interesting and beneficial endeavor in order to advance and consolidate the field of research. The proposed periodic table does not claim to have an immediate application for engineering multimodal gestures. Its value is on a meta-level, in order to identify differences, commonalities, and requirements for engineering software that processes gestural input.

The periodic table of gestural interaction requires researchers to think in terms of Semiotics: what are the fundamental syntactic elements that constitute a gestural system and how are they combined, interpreted, and received by users? We first provide a brief background on the periodic table of chemical elements and then address existing research on gestural interaction, followed by an attempt at constituting a first draft of the periodic table for gestural interaction.

BACKGROUND The periodic table of the chemical elements is a table that registers elements and their atomic numbers, electron configurations, and other chemical properties in a tabular manner [ 17 ]. It is not only a collection of known elements; due to its layout according to physical and chemical rules, it also serves to predict further elements that have not yet been synthesized or discovered.

Eighteen columns and seven rows constitute the periodic table. There is a double row of elements below those columns and rows (see Figure 1). Colors signify if the elements belong to a metal category or non-metal category.

Rows and columns are called periods and groups, respectively. Some of the groups have special names like halogens or noble gases. Groups show trends with respect to the contained elements, i.e. common properties such as the same electron configuration in the outermost shell of the atom. Periods represent less important trends: atomic radius, ionization energy, electron affinity, and electronegativity.

There are different layouts of the periodic table of chemical

elements, for instance, the lanthanides and actinides can be integrated, which makes the table considerably broader.

Due to different requirements and views, several layouts and categorizations are possible with regard to the chemical elements of the periodic table.

The chemical elements themselves are described by their

atomic number, which directly refers to the electron configuration. The designated name is often in Latin and is the foundation for the official symbol, which is an internationally agreed code that consists of one, two, or three letters.

The periodic table has also been called “nature’s rosetta stone” [ 1 ]. There are other instances, where the periodic table of chemical elements serves as a means to structure a field of research, such as the periodic table of visualization methods for management [ 16 ]. This periodic table uses concrete information visualization methods as basic elements. Coloring is used for categories, while rows show a basic trend towards more complex visualization techniques. The categories are data visualization, information visualization, concept visualization, strategy visualization, metaphor visualization, and compound visualization. The columns have no explicit meaning, but show another trend towards more complex and compound visualizations. Atomic numbers are omitted and the main symbol or abbreviation is put in the center of each element. Within each element, the coloring of the symbol shows whether the visualization method is a process or structure visualization and icons express if the visualization method is used for overview, detail, divergent thinking, or convergent thinking. These symbols can be combined and inspired the icons used in the periodic table of gestural interaction.

RESEARCH ON GESTURAL INTERACTION

It is out of the scope of this paper to collect the complete

research that exists on gestural interaction. However, in order to setup a first draft of the intended periodic table, it is necessary to address a number of approaches that seek to consolidate knowledge on gestural interaction.

Declarative approaches to specify multi-touch gestures can

be found in the literature such as GDL [ 12,13 ]. Other formalization attempts for multi-touch gestures are GeForMT [ 9,10 ] and Proton [ 14,15 ]. Wobbrock et al. consider further aspects towards a taxonomy of multi-touch gestures [ 23 ]. GISpL [ 5 ] and Mudra [ 8 ] address multimodal gestures. The Behaviour Markup Language is an XML dialect to describe multimodal and spatial gestures [ 22 ]. The Conversational Gesture Transcription system also uses a formal notation to describe spatial human gestures [ 21 ]. For sketching gestures, a sketch language has been developed by Bimber et al. [ 2 ]. Another domain-specific language for sketching has been proposed by Hammond and Davis [ 7 ]. Spindler et al. propose an interaction vocabulary for spatial interaction using magic lenses [ 18,19 ]. Epps et al. investigate different hand shapes, which can be used in spatial gestures [ 6 ].

All of the declarative approaches to describe gestures aim at

facilitating the implementation of gestures used in specific interaction techniques. In the next section, we propose to collect these interaction techniques in a periodic table of gestural interaction.

PERIODIC TABLE OF GESTURAL INTERACTION

Most of the research in gestural interaction addresses the

atomic building blocks of gestures and how they can be captured and processed. Interaction techniques developed by researchers use these building blocks for simple or compound tasks in an interface. We propose to view these interaction techniques as the elements of our periodic table.

The periodic table should reflect some basic distinctions of the gestural interaction techniques such as:

• • • •

Degrees of freedom

Complexity

Continuous or discrete evaluation

Hardware, i.e. enabling technologies

The complexity used in our periodic table refers to the

subjective intricacy involved in an interaction technique. For instance, a technique involving two hands or multiple fingers can be rated more complex than a simple tap with one finger. It is important to note that the intended periodic table makes most of the distinction from the point of view of the sensing technology. Hence, we will not consider the movement necessary to reach a button on a keyboard. The same goes for reaching out to a multi-touch screen.

Ta

INTERACTION TECHNIQUE TAP

Position and

mode (number of fingers, duration, repetition)

2F(POINT)

Activation, selection, context menus, information

Short highlighting of interactive interface elements

Manipulation

Size and precision of fingers

Formalization

1F(POINT) or

There are other frameworks for gesture-based interactions, which also integrate application domains and concrete design guidelines [11]. Early work of Buxton is more in

line with our work, which focuses on lexicalic and pragmatic aspects of input structures [ 3,4 ].

The first draft of our periodic table in Figure 3 groups the interaction techniques according to their enabling technologies and complexity, as well as with regard to the degrees of freedom. The degrees of freedom (DOF) cannot be expressed with specific numbers and represent a general tendency in the table: from 2D space to 3D space. It is also conceivable to relate input DOF to output DOF in order to achieve a more sophisticated layout.

In addition, the complexity of the different interaction

techniques is open to debate. Those techniques of similar complexity should be found on the same row in the table.

The axis for complexity is chosen similarly to the periodic

table of visualization methods for management [ 16 ]. Hence, simple techniques are found in the top rows of the table and more involved interaction techniques are situated at the bottom.

A name, a short symbol, and a number of properties, which are represented by icons, describe each interaction technique. The interaction techniques such as tap, scrolling, and panning are already established quasi-standards. Other techniques such “lense tilt” or “lense move” are currently being researched [ 20 ]. The properties expressed by icons in each element refer to the use of the technique in a continuous (online) or discrete (offline) manner.

Furthermore, a distinction in interaction techniques for navigation and manipulation tasks is made.

INTERACTION TECHNIQUE SCROLLING

INTERACTION TECHNIQUE PINCH Information Formalization Use case

Centre of gesture and relative adjustment of distance between both fingers

JOIN[1F(LINE)

* 1F(LINE)] or

SPLIT[1F(LINE) * 1F(LINE)]

Zoom in and

zoom out,

Scaling of objects

Truncated interface, short fade in of scroll bars

Manipulation

(Scaling) or

Navigation (level of detail)

Used in combination with panning

Direction of movement (horizontal or vertical)

Navigation in lists, interfaces with dynamic dimensions

Truncated list items, consecutive numbering

Navigation (orientation)

Long interfaces or texts need optimization

Each interaction technique should be accompanied by a

more detailed crib sheet, which gives a short overview of the technique, its use cases, and application. Table 1 gives an example of such a crib sheet, which also uses GeForMT to provide a formal expression of the multi-touch gesture involved in the interaction technique [ 9 ].

GeForMT uses a simple math formula syntax involving contact functions such as 1F(…) to express that 1 finger touches the multi-touch surface. Atomic gestures describe certain movements such as lines (LINE) or circles (CIRCLE) or static contacts such as POINT for a simple touch of the surface or HOLD for a longer contact. Other formalizations or notations are conceivable as well and should be provided in order to exchange gestures across different frameworks. FUTURE WORK

The periodic table can be used to codify knowledge in the

field of gestural interaction. Especially with the advent of multimodal interfaces, this becomes increasingly important. An interactive periodic table could be established on the internet, which would allow cooperative work on building a knowledge base on gestural interaction. Furthermore, an interactive table allows drilling down on the elements and show detailed information, such as the implementation with various declarative approaches to describe multimodal gestures.

Researchers should try to classify their developed interaction techniques according to an established set of concerns as used in the periodic table. In the following, the table can be adapted and optimized. Especially if the classification breaks down or becomes ambiguous, additional rules need to be devised in order to achieve a sound assessment of complexity and DOFs used in an interaction technique.

ACKNOWLEDGMENTS

The European Union and the Free State Saxony through the

European Regional Development Fund (ERDF) supported this work. Thanks are due to Axel Pötzsch for his support of this research.

References

1. Baum , R.M. Celebrating the periodic table . Chemical & engineering news 81 , 36 , 28 - 29 .

2. Bimber , O. , Encarnacao , L.M. , and Stork , A. A multilayered architecture for sketch-based interaction within virtual environments . Computers & Graphics 24 , 6 ( 2000 ), 851 - 867 .

3. Buxton , W. Lexical and pragmatic considerations of input structures . ACM SIGGRAPH Computer Graphics 17 , 1 ( 1983 ), 31 - 37 .

4. Buxton , W. Chunking and phrasing and the design of human-computer dialogues . Proceedings of the IFIP World Computer Congress , ( 1986 ), 475 - 480 .

5. Echtler , F. and Butz , A. GISpL: gestures made easy . Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction , ACM ( 2012 ), 233 - 240 .

6. Epps , J. , Lichman , S. , and Wu , M. A study of hand shape use in tabletop gesture interaction . CHI '06 Extended Abstracts on Human Factors in Computing Systems , ACM ( 2006 ), 748 - 753 .

7. Hammond , T. and Davis , R. LADDER , a sketching language for user interface developers . Computers & Graphics 29 , 4 ( 2005 ), 518 - 532 .

8. Hoste , L. , Dumas , B. , and Signer , B. Mudra: a unified multimodal interaction framework . Proceedings of the 13th international conference on multimodal interfaces , ACM ( 2011 ), 97 - 104 .

9. Kammer , D. , Henkens , D. , Henzen , C. , and Groh , R. Gesture Formalization for Multi-touch . Software: Practice and Experience (in press), ( 2014 ).

10. Kammer , D. , Wojdziak , J. , Keck , M. , Groh , R. , and Taranko , S. Towards a formalization of multi-touch gestures . ACM International Conference on Interactive Tabletops and Surfaces , ACM ( 2010 ), 49 - 58 .

11. Karam , M. and Schraefel , M.C. A taxonomy of Gestures in Human Computer Interaction . ACM Transactions on Computer-Human Interactions , ( 2005 ).

12. Khandkar , S. and Maurer , F. A Domain Specific Language to Define Gestures for Multi-Touch Applications . Proceedings of the 10th Workshop on Domain-Specific Modeling (DSM'10) , ( 2010 ).

13. Khandkar , S.H. , Sohan , S.M. , Sillito , J. , and Maurer , F. Tool support for testing complex multi-touch gestures . ACM International Conference on Interactive Tabletops and Surfaces , ACM ( 2010 ), 59 - 68 .

14. Kin , K. , Hartmann , B. , DeRose , T., and Agrawala , M. Proton: multitouch gestures as regular expressions . Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems , ACM ( 2012 ), 2885 - 2894 .

15. Kin , K. , Hartmann , B. , DeRose , T., and Agrawala , M. Proton++: a customizable declarative multitouch framework . Proceedings of the 25th annual ACM symposium on User interface software and technology , ACM ( 2012 ), 477 - 486 .

16. Lengler , R. and Eppler , M.J. Towards a periodic table of visualization methods of management . Proceedings of the IASTED International Conference on Graphics and Visualization in Engineering, ACTA Press ( 2007 ), 83 - 88 .

17. Scerri , E.R. The periodic table: a very short introduction . Oxford University Press, Oxford; New York, 2011 .

18. Scholliers , C. , Hoste , L. , Signer , B. , and De Meuter , W. Midas: a declarative multi-touch interaction framework . Proceedings of the fifth international conference on Tangible, embedded, and embodied interaction , ACM ( 2011 ), 49 - 56 .

19. Spindler , M. and Dachselt , R. Exploring information spaces by using tangible magic lenses in a tabletop environment . CHI '10 Extended Abstracts on Human Factors in Computing Systems , ACM ( 2010 ), 4771 - 4776 .

20. Spindler , M. , Tominski , C. , Schumann , H. , and Dachselt , R. Tangible views for information visualization . ACM International Conference on Interactive Tabletops and Surfaces , ACM ( 2010 ), 157 - 166 .

21. Spindler , M. Spatially Aware Tangible Display Interaction in a Tabletop Environment . Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces , ACM ( 2012 ), 277 - 282 .

22. Trippel , T. , Gibbon , D. , Thies , A. , et al. CoGesT: a formal transcription system for conversational gesture . Proceedings of LREC 2004 , ( 2004 ), 2215 - 2218 .

23. Vilhjálmsson , H. , Cantelmo , N. , Cassell , J. , et al. The Behavior Markup Language: Recent Developments and Challenges . In C. Pelachaud, J.-C. Martin , E. André , G. Chollet, K. Karpouzis and D. Pelé, eds., Intelligent Virtual Agents. Springer Berlin / Heidelberg, 2007 , 99 - 111 .

24. Wobbrock , J.O. , Morris , M.R. , and Wilson, A.D. Userdefined gestures for surface computing . Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , ACM ( 2009 ), 1083 - 1092 .