Considering the technological advances in touch-based devices, gesture-based interaction has become a prevalent feature in many application domains. Information systems are starting to explore this type of interaction. Currently, gesture specifications are hard-coded by developers at the source code level, hindering its reusability and portability. Similarly, defining new gestures in line with users' requirements is further complicated. This paper describes a model-driven approach to include gesture-based interaction in desktop information systems and a tool prototype to: capture user-sketched multi-stroke gestures and transform them into a model, automatically generating the gesture catalogue for gesture-based interaction technologies and gesture-based interface source code. We demonstrate our approach in several applications, ranging from case tools to form-based information systems.
New devices come together with new types on interfaces (e.g. based on gaze,
gesture, voice, haptic, brain-computer interfaces). Their aim is to increase the
naturalness of interaction [
Several issues may hinder the wide adoption of gesture-based interaction
in complex information systems engineering. Gesture-based interfaces have
been reported to be more difficult to implement and test than traditional
mouse and pointer interfaces [
This paper introduces an MDD approach and a tool for gesture-based IS interface development, which is intended to allow software engineers focusing on the key aspects of gesture based information system interfaces; namely, defining gestures and specifying gesture-based interaction. Coding and portability efforts are alleviated by means of model-to-text (M2T) transformations. 2 2.1
The representation of gestures according to related literature can be classified
into some categories: (a) based on regular expressions [
In this research work we propose a model-driven approach in order to
represent gestures with a high-level of abstraction, enabling
platformindependence and reusability. By providing the proper transformations, it is
possible to target several gesture recognition technologies. We focus on
userdefined, multi-stroke, semaphoric gestures [
Gesture-based interfaces can play two major roles in IS engineering,
depending on whether we intend to incorporate this natural interaction into (i) CASE
tools or (ii) into the IS themselves. In the former case, the interest is to
increase the IS developers’ efficiency, whereas in the latter the aim is to
increase IS usability, especially in operations in the field, where the lack of a
comfortable office space reduces the ergonomics of mouse and keyboard.
In both cases, gesture-based interface development methods and tools are
needed. Some examples of methods and tools are described in [
In this work, we propose a similar flow to that of [
gestUI is a user-driven and iterative method that follows the MDD paradigm. The main artefacts are models which are conform to the Model-Driven Architecture, a generic framework of modelling layers that ranges from abstract specifications to the software code (indicated at the top of Fig. 1).
The computation-independent layer is omitted because gestUI already
assumes that the IS is going to be computerised. Note that gestUI is expected
to be integrated into a full interface development method (represented with
generic activities and artefacts in grey). Such a method can either be
modeldriven or code-centric. gestUI is user-driven because users participate in all
non-automated activities; and it is iterative because it intends to discover the
necessary gestures incrementally and provides several loopbacks. In the
platform-independent layer, the gestures are defined (activity A1 in Fig. 1) by
the developer but, preferably, in collaboration with representative users of the
IS. Gestures are defined by sketching and are stored in the ‘Gesture catalogue
model’, and is part of a larger ‘Interaction requirements’ specification. In the
platform specific layer, a concrete gesture-recognition platform is selected
(we currently support three platforms: quill [
The gestUI tool is developed using Java and Eclipse Modelling Framework. As shown in Fig. 2, the tool is structured in three modules. The numbers in brackets indicate the method activity each component supports. The method’s internal products are not shown. The relationship with the external gesture recogniser is represented.
It supports the definition of new multi-stroke gestures by means of an interface implemented in Java in which the user sketches the gestures. The set of gestures sketched by the user constitutes the ‘Gesture catalogue model’, conforms to the metamodel defined in this work (Fig. 3). 4.2
It requires as data: the ‘Gesture catalogue model’, the target technology specified by the developer, and the target folder to save the output. Depending on the target technology, a different M2T transformation is executed which creates the PSGS, in the corresponding file format (i.e. XML for $N and iGesture and GDT 2.0 for quill). The transformation rules are written in Acceleo. The PSGS can be imported in a third-party gesture recogniser to test the gestures.
It allows the developer and the user to specify what action to execute whenever the gesture-based IS interface recognises a gesture. In a model-based IS interface development, the actions are specified in the interface model. In a code-centric interface development, they are implemented in the interface itself. We currently provide automated support to code-centric developments made in Java; that is, the gestUI module parses the source code of the user interface to obtain a list of actions. This module therefore requires two inputs: the previously created ‘Gesture catalogue model’ and the user interface (e.g. a Java code). The output of this module is the ‘Gesture-based interaction model’ and the same source code but now supporting the gesture-based interaction.
When generating the user interface Java source code, many references are included (e.g., to libraries to manage gestures, to libraries of the gesturerecognition technology (e.g. $N)), and some methods are added (e.g., to execute the gesture-action correspondence, and to capture gestures). Additionally, the class definition is changed to include some listeners, then the source code should obviously be compiled. 5
We demonstrate the integration of gestUI within a code-centric interface development method. For illustration purposes, we use a fictional, simple university management case and we narrate the project as if it actually happened. Fig. 4 shows the domain class diagram of a university with several departments, to which teachers are assigned and which manage classrooms. For the sake of brevity, we will just consider the two screens; namely, the initial and department management screens.
In the first method iteration, the university representatives tell the developer that they would like the gestures to resemble parts of the university logo. Thus, they use the Gesture catalogue definition module to create a first version of the ‘Gesture catalogue model’ containing these three gestures: for departments, || for teachers and for classrooms. However, when the first interface design is available (see sketch in Fig. 5), they soon realise that other gestures are needed. This way, by defining new gestures, and after testing them, they determine that navigation will be done by means of the abovementioned gestures, but that similar actions that appear across different screens shall have the same gestures (e.g. the gesture shall be used to create both new departments and teachers).
The Model transformation module allows generating the PSGS for any of the available gesture-based recognition technologies (i.e. $N, quill and iGesture). The developer only needs to choose a single technology but we chose to demonstrate the multiplatform features of the gestUI method by generating the three gesture files. Using the appropriate tool, the users can test the gestures. Fig. 6 shows the gestures being recognised by the SN, quill and iGesture tools so the gestures have been properly converted by the Model transformation module.
The developer assigns the gesture-action correspondence in collaboration with the user, supported by the Gesture-action correspondence definition module. The correspondences are informally shown in Fig. 5, next to each action button. Once the Java source code of the traditional interface is available, then the components that support the gesture-based interaction are generated. In this case, the chosen underlying gesture-recognition technology is $N; the users felt more comfortable with multi-stroke gestures (especially with regards to tracing some letters and symbols) so quill was discarded. The final IS interface consists of several screens that allow managing university information. Users can still interact with the IS in the traditional way (i.e. using the mouse), but now, they can also draw the gestures with a finger on the touchbased screen in order to execute the actions. Fig. 7 represents a specific interaction with the IS interface in which a department is being created. We describe gestUI, a model-driven method, and the tool that supports it to specify multi-stroke gestures and automatically generating the information system components that support the gesture-based interaction. We validated the method and tool by applying them to a case and generated the Platformspecific gesture specification for three gesture-recognition technologies, to illustrate the multiplatform capability of the tool. The gestures were successfully recognised by the corresponding tools. We then automatically generated the final gesture-based interface components and integrated them into the IS interface. The advantages of the proposal are: platform independence enabled by the MDD paradigm, the convenience of including user-defined symbols and its iterative and user driven approach. Its main current limitations are related to the target interface technologies (currently, only Java) and the fact that multi-finger gestures are not supported. These limitations will be addressed in future work. We also plan further validation by applying the approach to the development of a real IS and to extending a CASE tool with gesture-based interaction (the Capability Development Tool being developed in the FP7 CaaS project). We also plan to integrate gestUI into a full-fledged model-driven framework capable of automatically generating the presentation layer, in order to extend it with gesture-based interaction modelling and code generation. The author is grateful to his supervisors Sergio España and Óscar Pastor for their invaluable support and advice. This work has been supported by SENESCYT and Univ. de Cuenca - Ecuador, and received financial support from Generalitat Valenciana under Project IDEO (PROMETEOII/2014/039).