=Paper=
{{Paper
|id=Vol-1190/paper5
|storemode=property
|title=Towards a Periodic Table of Gestural Interaction
|pdfUrl=https://ceur-ws.org/Vol-1190/paper5.pdf
|volume=Vol-1190
|dblpUrl=https://dblp.org/rec/conf/eics/KammerKG14
}}
==Towards a Periodic Table of Gestural Interaction==
https://ceur-ws.org/Vol-1190/paper5.pdf
Towards a Periodic Table of Gestural Interaction
Dietrich Kammer, Mandy Keck, Rainer Groh
Technische Universität Dresden
01062 Dresden
dietrich.kammer@tu-dresden.de
ABSTRACT of natural sciences, cannot be established in applied
The periodic table is the first means of chemists to structure sciences such as human-computer interaction. However, it
their field of research. It comprises all chemical elements is still an interesting and beneficial endeavor in order to
with their most prominent features. The system is so advance and consolidate the field of research. The proposed
important that students of chemistry usually learn it by heart periodic table does not claim to have an immediate
during their university education. Scientists in the field of application for engineering multimodal gestures. Its value is
human-computer interaction lack such a concise system that on a meta-level, in order to identify differences,
covers all aspects of gestural interaction. Although commonalities, and requirements for engineering software
considerable research exists, scientists rarely agree on that processes gestural input.
common aspects and systems to classify, collect, and share
their research. This position paper attempts to pacify The periodic table of gestural interaction requires
rivaling scientific views towards gestural interaction and its researchers to think in terms of Semiotics: what are the
properties, benefits, and applications. By collecting meta- fundamental syntactic elements that constitute a gestural
properties of multimodal gestures, the proposed periodic system and how are they combined, interpreted, and
table tries to provide a common ground for classification received by users? We first provide a brief background on
and debate among researchers and practitioners working the periodic table of chemical elements and then address
with gestural interfaces. existing research on gestural interaction, followed by an
attempt at constituting a first draft of the periodic table for
Author Keywords gestural interaction.
Gestures, Multimodal, Classification, Formalization. BACKGROUND
ACM Classification Keywords The periodic table of the chemical elements is a table that
D.2.2 Software Engineering: Design Tools and Techniques; registers elements and their atomic numbers, electron
H.5.2 Information Interfaces & Presentation: User configurations, and other chemical properties in a tabular
Interfaces manner [17]. It is not only a collection of known elements;
due to its layout according to physical and chemical rules, it
INTRODUCTION also serves to predict further elements that have not yet
Scientists in the field of human-computer interaction rarely been synthesized or discovered.
agree on common aspects of gestural interaction. While this Eighteen columns and seven rows constitute the periodic
attitude serves to investigate various routes and illuminates table. There is a double row of elements below those
different key aspects, the consolidation of knowledge is columns and rows (see Figure 1). Colors signify if the
necessary to advance and consolidate a field of research. In elements belong to a metal category or non-metal category.
this position paper, we propose to collect and agree on
certain meta-properties of different gestural interaction Rows and columns are called periods and groups,
styles. respectively. Some of the groups have special names like
halogens or noble gases. Groups show trends with respect
A periodic table of gestural interaction is the main to the contained elements, i.e. common properties such as
metaphor, which serves as motivation to classify atomic the same electron configuration in the outermost shell of the
gesture building blocks and debate their properties. The atom. Periods represent less important trends: atomic
periodic table is the main tool for chemists to structure their radius, ionization energy, electron affinity, and
field of research. It is so important that undergraduate electronegativity.
students are required to learn it by heart. It is clear that such
a profound system, which is grounded in fundamental facts There are different layouts of the periodic table of chemical
elements, for instance, the lanthanides and actinides can be
EGMI 2014, 1st International Workshop on Engineering Gestures for integrated, which makes the table considerably broader.
Multimodal Interfaces, June 17 2014, Rome, Italy.
Due to different requirements and views, several layouts
Copyright © 2014 for the individual papers by the papers' authors.
Copying permitted only for private and academic purposes. This volume is and categorizations are possible with regard to the chemical
published and copyrighted by its editors. elements of the periodic table.
http://ceur-ws.org/Vol-1190/.
30
RESEARCH ON GESTURAL INTERACTION
It is out of the scope of this paper to collect the complete
research that exists on gestural interaction. However, in
order to setup a first draft of the intended periodic table, it is
necessary to address a number of approaches that seek to
consolidate knowledge on gestural interaction.
Declarative approaches to specify multi-touch gestures can
be found in the literature such as GDL [12,13]. Other
formalization attempts for multi-touch gestures are
GeForMT [9,10] and Proton [14,15]. Wobbrock et al.
consider further aspects towards a taxonomy of multi-touch
gestures [23]. GISpL [5] and Mudra [8] address
multimodal gestures. The Behaviour Markup Language is
Figure 1: Structure of the periodic table of the chemical an XML dialect to describe multimodal and spatial gestures
elements. [22]. The Conversational Gesture Transcription system also
The chemical elements themselves are described by their uses a formal notation to describe spatial human gestures
atomic number, which directly refers to the electron [21]. For sketching gestures, a sketch language has been
configuration. The designated name is often in Latin and is developed by Bimber et al. [2]. Another domain-specific
the foundation for the official symbol, which is an language for sketching has been proposed by Hammond
internationally agreed code that consists of one, two, or and Davis [7]. Spindler et al. propose an interaction
three letters. vocabulary for spatial interaction using magic lenses
[18,19]. Epps et al. investigate different hand shapes,
which can be used in spatial gestures [6].
All of the declarative approaches to describe gestures aim at
facilitating the implementation of gestures used in specific
interaction techniques. In the next section, we propose to
collect these interaction techniques in a periodic table of
gestural interaction.
Figure 2: Chemical element with its configuration and
designation. PERIODIC TABLE OF GESTURAL INTERACTION
Most of the research in gestural interaction addresses the
The periodic table has also been called “nature’s rosetta
atomic building blocks of gestures and how they can be
stone” [1]. There are other instances, where the periodic
captured and processed. Interaction techniques developed
table of chemical elements serves as a means to structure a
by researchers use these building blocks for simple or
field of research, such as the periodic table of visualization
compound tasks in an interface. We propose to view these
methods for management [16]. This periodic table uses interaction techniques as the elements of our periodic table.
concrete information visualization methods as basic The periodic table should reflect some basic distinctions of
elements. Coloring is used for categories, while rows show the gestural interaction techniques such as:
a basic trend towards more complex visualization
techniques. The categories are data visualization, • Degrees of freedom
information visualization, concept visualization, strategy
• Complexity
visualization, metaphor visualization, and compound
visualization. The columns have no explicit meaning, but • Continuous or discrete evaluation
show another trend towards more complex and compound
visualizations. Atomic numbers are omitted and the main • Hardware, i.e. enabling technologies
symbol or abbreviation is put in the center of each element. The complexity used in our periodic table refers to the
Within each element, the coloring of the symbol shows subjective intricacy involved in an interaction technique.
whether the visualization method is a process or structure For instance, a technique involving two hands or multiple
visualization and icons express if the visualization method fingers can be rated more complex than a simple tap with
is used for overview, detail, divergent thinking, or one finger. It is important to note that the intended periodic
convergent thinking. These symbols can be combined and table makes most of the distinction from the point of view
inspired the icons used in the periodic table of gestural of the sensing technology. Hence, we will not consider the
interaction. movement necessary to reach a button on a keyboard. The
same goes for reaching out to a multi-touch screen.
31
Figure 3: First draft of a periodic table for gestural interaction, the elements correspond with different interaction techniques.
line with our work, which focuses on lexicalic and
Ta pragmatic aspects of input structures [3,4].
INTERACTION TECHNIQUE TAP The first draft of our periodic table in Figure 3 groups the
interaction techniques according to their enabling
Information Position and technologies and complexity, as well as with regard to the
mode (number of degrees of freedom. The degrees of freedom (DOF) cannot
fingers, duration, be expressed with specific numbers and represent a general
repetition) tendency in the table: from 2D space to 3D space. It is also
Formalization 1F(POINT) or conceivable to relate input DOF to output DOF in order to
achieve a more sophisticated layout.
2F(POINT)
In addition, the complexity of the different interaction
Use case Activation, techniques is open to debate. Those techniques of similar
selection, context complexity should be found on the same row in the table.
menus, The axis for complexity is chosen similarly to the periodic
information table of visualization methods for management [16]. Hence,
Visualization Short highlighting simple techniques are found in the top rows of the table and
Feedforward of interactive more involved interaction techniques are situated at the
interface bottom.
elements A name, a short symbol, and a number of properties, which
are represented by icons, describe each interaction
Interaction Manipulation
technique. The interaction techniques such as tap, scrolling,
goals
and panning are already established quasi-standards. Other
Restrictions Size and techniques such “lense tilt” or “lense move” are currently
precision of being researched [20]. The properties expressed by icons in
fingers each element refer to the use of the technique in a
continuous (online) or discrete (offline) manner.
Table 1: Crib sheet for interaction technique “tap” Furthermore, a distinction in interaction techniques for
navigation and manipulation tasks is made.
There are other frameworks for gesture-based interactions,
which also integrate application domains and concrete
design guidelines [11]. Early work of Buxton is more in
32
Scr Pi
INTERACTION TECHNIQUE SCROLLING INTERACTION TECHNIQUE PINCH
Information Direction of Information Centre of
movement gesture and
(horizontal or relative
vertical) adjustment of
distance
Formalization 1F(LINE)
between both
Use case Navigation in fingers
lists, interfaces Formalization JOIN[1F(LINE)
with dynamic * 1F(LINE)] or
dimensions SPLIT[1F(LINE)
Visualization Truncated list * 1F(LINE)]
Feedforward items, Use case Zoom in and
consecutive
zoom out,
numbering
Scaling of
Interaction Navigation objects
goals (orientation) Visualization Truncated
Restrictions Long Feedforward interface, short
interfaces or fade in of scroll
texts need bars
optimization Interaction Manipulation
goals (Scaling) or
Table 2: Crib sheet for interaction technique “scrolling”
Navigation
Each interaction technique should be accompanied by a (level of detail)
more detailed crib sheet, which gives a short overview of
Restrictions Used in
the technique, its use cases, and application. Table 1 gives
combination
an example of such a crib sheet, which also uses GeForMT
with panning
to provide a formal expression of the multi-touch gesture
involved in the interaction technique [9].
Table 3: Crib sheet for interaction technique “pinch”
GeForMT uses a simple math formula syntax involving
contact functions such as 1F(…) to express that 1 finger FUTURE WORK
touches the multi-touch surface. Atomic gestures describe The periodic table can be used to codify knowledge in the
certain movements such as lines (LINE) or circles field of gestural interaction. Especially with the advent of
(CIRCLE) or static contacts such as POINT for a simple multimodal interfaces, this becomes increasingly important.
touch of the surface or HOLD for a longer contact. Other An interactive periodic table could be established on the
formalizations or notations are conceivable as well and internet, which would allow cooperative work on building a
should be provided in order to exchange gestures across knowledge base on gestural interaction. Furthermore, an
different frameworks. interactive table allows drilling down on the elements and
Table 2 and 3 show additional examples of crib sheets for show detailed information, such as the implementation with
multi-touch interaction techniques. The examples show use various declarative approaches to describe multimodal
cases when these interaction techniques are suitable and gestures.
how they can be visualized in an interface. Thus, a Researchers should try to classify their developed
feedforward is suggested for the developer of a gestural interaction techniques according to an established set of
interface. The restrictions listed in the crib sheets can alert concerns as used in the periodic table. In the following, the
the developer if certain problems exist when using an table can be adapted and optimized. Especially if the
interaction technique, or if there are interplays with classification breaks down or becomes ambiguous,
additional techniques. additional rules need to be devised in order to achieve a
sound assessment of complexity and DOFs used in an
interaction technique.
33
ACKNOWLEDGMENTS 14. Kin, K., Hartmann, B., DeRose, T., and Agrawala, M.
The European Union and the Free State Saxony through the Proton: multitouch gestures as regular expressions.
European Regional Development Fund (ERDF) supported Proceedings of the 2012 ACM annual conference on
this work. Thanks are due to Axel Pötzsch for his support of Human Factors in Computing Systems, ACM (2012),
this research. 2885–2894.
15. Kin, K., Hartmann, B., DeRose, T., and Agrawala, M.
REFERENCES Proton++: a customizable declarative multitouch
1. Baum, R.M. Celebrating the periodic table. Chemical framework. Proceedings of the 25th annual ACM
& engineering news 81, 36, 28–29. symposium on User interface software and technology,
2. Bimber, O., Encarnacao, L.M., and Stork, A. A multi- ACM (2012), 477–486.
layered architecture for sketch-based interaction within 16. Lengler, R. and Eppler, M.J. Towards a periodic table
virtual environments. Computers & Graphics 24, 6 of visualization methods of management. Proceedings
(2000), 851–867. of the IASTED International Conference on Graphics
3. Buxton, W. Lexical and pragmatic considerations of and Visualization in Engineering, ACTA Press (2007),
input structures. ACM SIGGRAPH Computer Graphics 83–88.
17, 1 (1983), 31–37. 17. Scerri, E.R. The periodic table: a very short
4. Buxton, W. Chunking and phrasing and the design of introduction. Oxford University Press, Oxford; New
human-computer dialogues. Proceedings of the IFIP York, 2011.
World Computer Congress, (1986), 475–480. 18. Scholliers, C., Hoste, L., Signer, B., and De Meuter,
5. Echtler, F. and Butz, A. GISpL: gestures made easy. W. Midas: a declarative multi-touch interaction
Proceedings of the Sixth International Conference on framework. Proceedings of the fifth international
Tangible, Embedded and Embodied Interaction, ACM conference on Tangible, embedded, and embodied
(2012), 233–240. interaction, ACM (2011), 49–56.
6. Epps, J., Lichman, S., and Wu, M. A study of hand 19. Spindler, M. and Dachselt, R. Exploring information
shape use in tabletop gesture interaction. CHI ’06 spaces by using tangible magic lenses in a tabletop
Extended Abstracts on Human Factors in Computing environment. CHI ’10 Extended Abstracts on Human
Systems, ACM (2006), 748–753. Factors in Computing Systems, ACM (2010), 4771–
7. Hammond, T. and Davis, R. LADDER, a sketching 4776.
language for user interface developers. Computers & 20. Spindler, M., Tominski, C., Schumann, H., and
Graphics 29, 4 (2005), 518–532. Dachselt, R. Tangible views for information
8. Hoste, L., Dumas, B., and Signer, B. Mudra: a unified visualization. ACM International Conference on
multimodal interaction framework. Proceedings of the Interactive Tabletops and Surfaces, ACM (2010), 157–
13th international conference on multimodal 166.
interfaces, ACM (2011), 97–104. 21. Spindler, M. Spatially Aware Tangible Display
9. Kammer, D., Henkens, D., Henzen, C., and Groh, R. Interaction in a Tabletop Environment. Proceedings of
Gesture Formalization for Multi-touch. Software: the 2012 ACM International Conference on Interactive
Practice and Experience (in press), (2014). Tabletops and Surfaces, ACM (2012), 277–282.
10. Kammer, D., Wojdziak, J., Keck, M., Groh, R., and 22. Trippel, T., Gibbon, D., Thies, A., et al. CoGesT: a
Taranko, S. Towards a formalization of multi-touch formal transcription system for conversational gesture.
gestures. ACM International Conference on Interactive Proceedings of LREC 2004, (2004), 2215–2218.
Tabletops and Surfaces, ACM (2010), 49–58. 23. Vilhjálmsson, H., Cantelmo, N., Cassell, J., et al. The
11. Karam, M. and Schraefel, M.C. A taxonomy of Behavior Markup Language: Recent Developments
Gestures in Human Computer Interaction. ACM and Challenges. In C. Pelachaud, J.-C. Martin, E.
Transactions on Computer-Human Interactions, André, G. Chollet, K. Karpouzis and D. Pelé, eds.,
(2005). Intelligent Virtual Agents. Springer Berlin /
12. Khandkar, S. and Maurer, F. A Domain Specific Heidelberg, 2007, 99–111.
Language to Define Gestures for Multi-Touch 24. Wobbrock, J.O., Morris, M.R., and Wilson, A.D. User-
Applications. Proceedings of the 10th Workshop on defined gestures for surface computing. Proceedings of
Domain-Specific Modeling (DSM’10), (2010). the SIGCHI Conference on Human Factors in
13. Khandkar, S.H., Sohan, S.M., Sillito, J., and Maurer, F. Computing Systems, ACM (2009), 1083–1092.
Tool support for testing complex multi-touch gestures.
ACM International Conference on Interactive
Tabletops and Surfaces, ACM (2010), 59–68.
34