An increasing number of today's consumer devices such as mobile phones or tablet computers are equipped with various sensors. The extraction of useful information such as gestures from sensor-generated data based on mainstream imperative languages is a notoriously difficult task. Over the last few years, a number of domain-specific programming languages have been proposed to ease the development of gesture detection. Most of these languages have adopted a declarative approach allowing programmers to describe their gestures rather than having to manually maintain a history of event data and intermediate gesture results. While these declarative languages represent a clear advancement in gesture detection, a number of issues are still unresolved. In this paper we present relevant criteria for gesture detection and provide an initial classification of existing solutions based on these criteria in order to foster a discussion and identify opportunities for future gesture programming languages.
With the increasing interest in multi-touch surfaces (e.g. Sony Tablet, Microsoft Surface or Apple iPad), controller-free sensors (e.g. Leap Motion, Microsoft Kinect or Intel’s Perceptual SDK) and numerous sensing appliances (e.g. Seeeduino Films and Nike+ Fuel), developers are facing major challenges in integrating these modalities into common applications. Existing mainstream imperative programming languages cannot cope with user interaction requirements due to the inversion of control where the execution flow is defined by input events rather than by the program, the high programming effort for maintaining an event history and the difficulty of expressing complex patterns.
EGMI 2014, 1st International Workshop on Engineering Gestures for Multimodal Interfaces, June 17 2014, Rome, Italy.
Copyright © 2014 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors. http://ceur-ws.org/Vol-1190/.
For example, commercial multi-touch hardware has evolved
from simple two-finger support to multi-user tracking with up
to 60 fingers1. Similarly, commercial depth sensors such as
the Microsoft Kinect were introduced in 2010 and supported
the tracking of 20 skeletal joints (i.e. tracking arms and limbs
in 3D space). Nowadays, numerous depth sensors such as the
Leap sensors or the DepthSense cameras by SoftKinetic also
provide short-range finger tracking. Recently, the Kinect for
Windows supports facial expressions and the Kinect 2
supports heart beat and energy level tracking. This rapid
evolution of novel input modalities continues with announcements
such as the Myo electromyography gesture armband [
In this paper, we consider a gesture as a movement of the hands, face or other parts of the body in time. Due to the high implementation complexity, most gesture recognition solutions rely on machine learning algorithms to extract gestural information from sensors. However, the costs of applying machine learning algorithms are not to be underestimated. The capture and annotation of training and test data requires substantial resources. Further, the tweaking of the correct learning parameters and analysis for overfitting require some expert knowledge. Last but not least, one cannot decisively observe and control what has actually been learned. Therefore, it is desired to have the possibility to program gestures and to ease the programming of gestural interaction. We argue that research in software engineering abstractions is of utmost importance for gesture computing.
In software engineering, a problem can be divided into its
accidental and essential complexity [
Gesture programming languages are designed to support
developers in specifying their gestural interaction requirements
more easily than with general purpose programming
languages. A domain-specific language might help to reduce
the repetitive boilerplate that cannot be removed in existing
languages as described by Van Cutsem2. General purpose
programming languages such as Java sometimes require an
excessive amount of constructs to express a developer’s
intention which makes them hard to read and maintain. Van
Cutsem argues that languages can shape our thought, for
instance when a gesture can be declaratively described by its
requirements rather than through an imperative
implementation with manual state management. A gesture programming
language can also be seen as a simplifier where, for example,
multiple inheritance might not be helpful to describe gestural
interaction. Finally, domain-specific languages can be used as
a law enforcer. Some gesture languages, such as Proton [
With these existing solutions, gestures are described in a domain-specific language as a sequence or simultaneous occurrences of events from one or multiple fingers. The modularisation and outsourcing of the event matching process paved the way for the rapid development of more complex multi-touch gestures. With the advent of novel hardware such as the Microsoft Kinect sensor with a similar or even higher level of complexity, domain-specific solutions quickly became a critical component for supporting advanced gestural interaction. The definition of complex gestures involves a number of concerns that have to be addressed. The goal of this paper is to enumerate these criteria for gesture programming languages and to provide an overview how existing approaches focus on different of these criteria.
We define a number of criteria which can shape (1) the choice of a particular framework, (2) the implementation of the gesture and (3) novel approaches to solve open issues in gesture programming languages. These criteria were compiled based on various approaches that we encountered over the last few years, also including the domains of machine learning or template matching which are out of the scope of this paper. We aligned the terminology, such as gesture spotting 2http://soft.vub.ac.be/ tvcutsem/invokedynamic/node/11 and segmentation, and performed an indicative evaluation of nine existing gesture languages as shown in Figure 1. For each individual criterion of these nice approaches we provide a score ranging from 0 to 5 together with a short explanation which can be found in the online data set. A score of 1 means that the approach could theoretically support the feature but it was not discussed in the literature. A score of 2 indicates that there is some initial support but with a lack of additional constructs that would make it useful. Finally, a score in the range of 3-5 provides an indication about the completeness and extensiveness regarding a particular criterion. Our data set, an up-to-date discussion as well as some arguments for the indicative scoring for each criterion of the different approaches is available at http://soft.vub.ac. be/ lhoste/research/criteria/images/img-data.js. We tried to cluster the approaches based on the most up-to-date information. However, some of the criteria could only be evaluated subjectively and might therefore be adjusted later based on discussions during the workshop. An interactive visualisation of the criteria for each of the approaches can be accessed via http://soft.vub.ac.be/ lhoste/research/criteria. Note that the goal of this assessment was to identify general trends rather than to draw a conclusive categorisation for each approach.
The following criteria have an effect on the software engineering properties of the gesture implementation. Furthermore, these criteria might require the corresponding features to be implemented by the processing engine.
By modularising gesture definitions we can reduce the effort to add an extra gesture. In many existing approaches the entanglement of gesture definitions requires developers to have a deep knowledge about already implemented gestures. This is a clear violation of the separation of concerns principle, one of the main principles in software engineering which dictates that different modules of code should have as little overlapping functionality as possible. Therefore, in modular approaches, each gesture specification is written in its own separate context (i.e. separate function, rule or definition).
Composition allows programmers to abstract low-level complexity by building complex gestures from simpler building blocks. For instance, the definition of a double tap gesture can be based on the composition of two tap gestures with a defined maximum time and space interval between them. A tap gesture can then be defined by a touch down event shortly followed by a touch up event and with minimal spatial movement in between. Composition is supported by approaches when a developer can reuse multiple modular specifications to define more complex gestures without much further effort.
Customisation is concerned with the effort a developer faces
to modify a gesture definition in order that it can be used in
a different context. How easy is it for example to adapt the
existing definition of a gesture when an extra condition is
required or the order of events should be changed? For
graphical programming toolkits, the customisation aspect is
broadened to how easy it is to modify the automatically generated
code and whether this is possible at all. Note that in many
machine learning approaches customisation is limited due to
the lack of a decent external representation [
Kammer et al. [
Negation is a feature that allows developers to express a context that should not be true in a particular gesture definition. Many approaches partially support this feature by requiring a strict sequence of events, implying that no other events should happen in between. However, it is still crucial to be able to describe explicit negation for some scenarios such as that there should be no other finger in the spatial neighbourhood or a finger should not have moved up before the start of a gesture. Other approaches try to use a garbage state in their gesture model to reduce false positives. This garbage state is similar to silence models in speech processing and captures nongestures by “stealing” partial gesture state and resetting the recognition process.
Some gestures such as a pinch gesture for zooming require feedback while the gesture is being performed. These socalled online gestures can be supported in a framework by allowing small online gesture definitions or by providing advanced constructs that offer a callback mechanism with a percentage of the progress of the larger gesture. Note that small online gesture definitions are linked with the segmentation criterion which defines that gestures can form part of a continuous event stream without an explicit begin and end condition.
Offline gestures are executed when the gesture is completely finished and typically represent a single command. These gestures are easier to support in gesture programming languages as they need to pass the result to the application once. Offline gestures also increase the robustness due to the ability to validate the entire gesture. The number of future events that can change the correctness of the gesture is limited when compared to online gestures.
Several conditions of a gesture definition can be partially or fully contained in another gesture definition. This might be intentional (e.g. if composition is not supported nor preferred) or unintentional (e.g. if two different gestures start with the same movement). Keeping track of multiple partial matches is a complex mechanism that is supported by some approaches, intentionally blocked by others (e.g. Proton) or ignored by some approaches.
A stream of sensor input events might not contain explicit
hints about the start and end of a gesture. The
segmentation concern (also called gesture spotting) gains importance
given the trend towards the continuous capturing and
freeair interaction such as the Kinect sensor and Z-touch [
The expiration of input events is required to keep the memory and processing complexity within certain limits. The manual maintenance of events is a complex task and most frameworks offer at least a simple heuristic to automatically expire old events. In multi-touch frameworks, a frequently used approach is to keep track of events from the first touch down event to the last touch up of any finger. This might introduce some issues when dealing with multiple users if there is always at least one active finger touching the table. Another approach is to use a timeout parameter, effectively creating a sliding window solution. An advantage of this approach is that the maximum memory usage is predefined, however a slack value is required. A static analysis of the gesture definitions could help to avoid the need for such a static value.
In order to allow concurrent interaction, one has to keep track
of multiple partial instances of a gesture recognition process.
For instance, multiple fingers, hands, limbs or users can
perform the same gesture at the same time. To separate these
instances, the framework can offer constructs or native
support for concurrent gesture processing. In some scenarios, it
is hard to decide which touch events belong to which hand or
user. For example, in Proton the screen can be split in half
to support some two player games. A better method is to set
a maximum bounding box of the gesture [
Concerns such as portability, serialisation and embeddability form the platform independence of an approach. Portability is defined by how easy it is to run the framework on different platforms. Some approaches are tightly interwoven with the host language which limits portability and the transfer of a gesture definition over the network. This transportation can be used to exchange gesture sets between users or even to offload the gesture recognition process to a dedicated server with more processing power. The exchange requires a form of serialisation of the gesture definitions which is usually already present in domain-specific languages. The embeddability has to do with the way how the approach can be used. Is it necessary to have a daemon process or can the abstractions be delivered as a library? Another question is whether it is possible to use the abstractions in a different language or whether this requires a reimplementation.
The dynamic nature of user input streams implies the
possibility of an abundance of information in a short period of
time. A framework might offer a maximum computational
boundary for a given setting [
The integration of graphical user interface (GUI) components
removes the need for a single entry point of gesture callbacks
on the application level. With contextual information and
GUI-specific gesture conditions the complexity is reduced.
For instance, a scroll gesture can only happen when both
fingers are inside the GUI region that supports scrolling [
Whenever a gesture is recognised, an action can be executed.
In some cases the developer wants to provide a more detailed
activation policy such as trigger only once or trigger when
entering and leaving a pose. Another example is the sticky
bit [
Dynamic binding is a feature that allows developers to define a variable without a concrete value. For instance, the x location of an event A should be between 10 and 50 but should be equal to the x location of an event B. At runtime, a value of 20 for the x location of event A will therefore require an event B with the same value of 20. This is particularly useful to correlate different events if the specification of concrete values is not feasible.
Refining gesture parameters or outsourcing gesture definitions to gesture services requires a form of runtime modification support by the framework. The refinement can be instantiated by an automated algorithm (e.g. an optimisation heuristic) by the developer (during a debugging session) or by the user to provide their preferences.
The primary goal of gesture languages is to provide an
abstraction level which helps developers to express complex
relations between input events. However, with multimodal
setups, input continuously enters the system and the user
expects the system to immediately react to their gestures.
Therefore, performance and the scalability when having many
gesture definitions is important. Some approaches, such as
Midas, exploit the language constructs to form an optimised
direct acyclic graph based on the Rete algorithm [
The modularity of gestures allows for a much better scalability in terms of complexity. When adding an extra gesture, no or minimal knowledge about existing definitions is required. However, when multiple gestures are recognised in the same pool of events, the developer needs to check whether they can co-exist (e.g. performed by different users) or are conflicting (i.e. deciding between rotation, scaling or both). A lot of work remains to be done to disambiguate gestures. For example, how do we cope with the setting of priorities or disambiguation rules between gestures when they are detected at a different timestamp? How can we cope with these disambiguation issues when there are many gesture definitions? Furthermore, it is unclear how we need to deal with many variants of a similar gesture in order that the correct one is used during the composition of a complex gesture.
When multiple gesture candidates are detected from the same event source, the developer needs a way to discriminate between them. However, this is not a simple task due to the lack of detail in sensor information, unknown future events and the uncertainty of the composition of the gesture.
The identification problem is related to the fact that sensor
input is not always providing enough details to disambiguate
a scenario. Echtler et al. [
The annotation of gestures with different priority levels is a
first form of prioritisation which can have an impact on the
gesture recognition accuracy. However, it requires
knowledge about existing gestures and if there are many gestures it
might not be possible to maintain a one-dimensional priority
schema. Nacenta et al. [
One of the major issues with gesture disambiguation is that information in the near future can lead to a completely different interpretation of a gesture. A gesture definition can, for instance, fully overlap with a larger, higher prioritised gesture. At a given point in time, it is difficult to decide whether the application should be informed that a particular gesture has been detected or whether we wait for a small time period. If future events show that the larger gesture does not match, users might perceive the execution of the smaller gesture as unresponsive. Late contextual information might also influence the fusion process of primitive events that are still in the running to form part of more complex gestures. The question is whether these fused events should be updated to reflect the new information and how a framework can support this.
Noise is an important parameter when dealing with gestural interaction. The jittering of multi-touch locations or limb positions might invalidate intended gestures or unintentionally activate other gestures. To analyse gestures based on imprecise primitive events, a form of uncertainty is required. This might also percolate to higher level gestures (e.g. when two uncertain subgestures are being composed). The downside is that this introduces more complexity for the developer.
Whenever a gesture candidate is found, it might be verified using an extra gesture classifier. An efficient segmentation approach may, for example, be combined with a more elaborate classification process to verify whether a detected recognition is adequate. Verification can also be used to further separate critical gestures (e.g. file deletion) from simple gestures (e.g. scaling). Note that couples of classifiers (i.e. ensembles) are frequently used in the machine learning domain. In order to further increase the gesture recognition accuracy, a developer can offer a form of user profiling for gestures that are known to cause confusion. Either a gesture is specified too precisely for a broader audience or it is specified too loosely for a particular user. This influences the recognition results and accidental activations. Therefore, the profiling of users by tracking undo operations or multiple similar invocations could lead to an adaptation of the gesture definition for that particular user. User profiling is valuable to improve recognition rates but it might also be interesting to exploit it for context-sensitive cases. Future work is needed to offer profiling as a language feature.
The description of a gesture requires a number of primitive statements such as spatial and temporal relations between multiple events as described in the following.
We define the spatial specification of a gesture as the
primitive travelled path to which it has to adhere. The path can
be formed by sequential or parallel conditions (expressed by
using temporal constructs) where events are constrained in a
spatial dimension such as 10 < event1:x < 50. The use of
relative spatial operators (e.g. event1:x + 30 > event2:x as
used in [
Gestures can be described in multiple conditions. However, these conditions cannot always be listed in a sequential order. Therefore, most gesture languages allow the developer to express explicit temporal relations between the conditions. An example of such a temporal relation is that two events should (or should not) happen within a certain time period.
Many frameworks offer additional features to describe a
gesture. For instance, Kammer et al. [
Scale invariance deals with the recognition of a single gesture trajectory regardless of its scale. Similarly, rotation invariance is concerned with the rotation. Most approaches offer this feature by rescaling and rotating the trajectory to a standard predefined size and centroid. However, a major limitation is that scale and rotation invariance requires segmentation and therefore does not work well for online gestures.
In order to debug gestures, developers usually apply
prerecorded positive and negative gesture sets to see whether the
given definition is compatible with the recorded data. Gesture
debugging tools have received little attention in research and
are typically limited to the testing of accuracy percentages.
It might be interesting to explore more advanced debugging
support such as notifying the developer of closely related
gesture trajectories (e.g. gesture a is missed by x units [
Gesture development can be carried out in a code-compatible
graphical manner such as done with tablatures [
We can identify some general trends with regard to open issues and underrepresented criteria in existing work. Figure 2 reveals that some criteria such as (1) modularisation and (7) offline gestures are well supported by most approaches and are important to provide a minimal ability to program gestures in a structured manner. Commonly used multi-touch gestures such as pinch and rotate are (6) online gestures supported by most frameworks. However, additional work can be done to streamline the implementation of online gestures by providing (2) composition support, a deeper (14) GUI integration and more advanced (16) activation policies. We believe that these challenges can be resolved with additional engineering effort in existing systems.
However, we see that more challenging issues such as (9) segmentation, (19) scalability in terms of complexity or dealing with (22) future events are poorly or not at all supported in existing approaches. (9) Segmentation is crucial to deal with continuous streams of information where no hints are given by the sensor with regard to potential start and end conditions. The recent trend towards near-touch sensors and skeletal tracking algorithms makes the segmentation concern of crucial importance.
With the adoption of the discussed frameworks there is an increasing demand to deal with (19) scalability in terms of complexity. It is currently rather difficult to get an overview on how many gestures work together. For instance, when a two-finger swipe needs to be implemented by composing two Midas GDL-K GeForMT GDL-E Proton GestureAgents EventHurdle GestIT ICO 30 20 10 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 0 Figure 2. A stacked bar chart of the summed values for each criterion primitive swipes. However, more than one primitive swipe variation might exist (long/short, fast/slow) and choosing the correct one without introducing conflicts is challenging. A way to deal with the issue is to provide test data but this is work intensive. There might be software engineering-based abstractions that could improve this situation.
Information coming from (22) events in the near future can lead to a completely different interpretation of a gesture. Especially in a multimodal context where context or clarifying information comes from additional sensors. An active gesture might fully overlap with a larger and higher prioritised gesture or be part of a composition that might or might not succeed. In other cases the gesture should not be triggered at all due (late) context data. There are currently no adequate abstractions helping developers to deal with these concerns. Finally, we would like to highlight that (23) uncertainty and (24) user profiling abstractions are also lacking. Dealing with uncertainty is currently completely hidden from the programmer. Nevertheless, a gesture might be better neglected when composing from two or more uncertain subgestures. Additionally, when a user consistently undoes an operation executed by a particular gesture, the gesture might require some adjustments. This can be checked by profiling users and by offering several resolutions strategies.
Gesture programming languages allow developers to more easily express their interaction patterns. When designing such a language, a number of concerns need to be addressed. Our goal was to categorise and explicitly expose these design decisions to provide a better understanding and foster a discussion about challenges, opportunities and future directions for gesture programming languages. We observed a number of underrepresented concerns in existing work and highlighted challenges for future gesture programming languages.