Currently, the process of definition of custom gestures and the inclusion of gesture-based interaction in user interfaces of information systems is complicated because to the diversity of devices, software platforms and development tools available. In order to help to resolve this situation we had proposed gestUI, a model-driven method, whose objectives are custom gesture definition and gesture-based user interface source code generation. This paper describes gestUI using the “Template for the Documentation of Method Components in CaaS” and it demonstrates the application of gestUI in order to include gesture-based interaction in the tool support called Capability Design Tool (CDT) of CaaS project.
Currently, there are some types of devices that the users employ (e.g., mobile phone,
tablet, notebook), in conjunction with this devices, there are some types of interaction
(e.g., gesture-based, gaze-based, voice-based interaction) [
Interfaces supporting gesture-based interaction have been reported to be more
challenging to implement and test than traditional mouse and pointer interfaces [
We had proposed gestUI [
In order to support CDD, the CaaS project has planned the following components [
CDD methodology: an agile methodology for identification, design and delivery of context aware business models. The notion of capability is formalised by means of a metamodel containing the following elements: goal, key performance indicator (KPI), context, capacity and, ability.
Capability delivery patterns: they are generic organisational design and business process which can be easily adapted, reused, and executed.
CDD environment: tool support for design and runtime of CDD solutions. Its first release is called Capability Design Tool (CDT), it is designed as an integrated development environment built on the Eclipse Framework, using the Eclipse Modelling Framework (EMF) technologies. This tool supports capability modelling according to the capability metamodel including context modelling and goal, process and concepts models The contributions of this paper are: (i) the description of the method gestUI by means of the “Template for the Documentation of Method Components in CaaS” and, (ii) the demonstration of use of gestUI with the aim to include gesture-based interaction in a CASE tool, in this case, Capability Design Tool (CDT).
The remainder of this paper is organized as follows. Section 2 contains the state of the art. Section 3 describes gestUI. Section 4 includes a description of Capability Design Tool (CDT). Section 5 contains details of the application of the method gestUI in a CASE tool, in this case CDT of CaaS project. The paper ends with some conclusions and future work in Section 6.
This section presents a related literature review of gesture-based human-computer interaction.
Vanderdonckt [
The authors of [
Carvalho et al. describe in their work [
gestUI [
Applying model-driven development method the results obtained with gestUI are: ─ Model-driven gesture catalogue which contains the definition of gestures using XML. This definition is platform-independent and it can be imported in frameworks that employ gestures as input in their process. ─ Source code to include in the specification of user interface in order to include gesture-based interaction in the user interface of information systems. The method produces source code in the selected technology containing the definition of gesture-based interaction. Considering the “Template for the Documentation of Method Components in CaaS” we describe gestUI by means of following method components: b.
Method component to define customised gestures: it permits the definition of customised gestures in order to obtain a platform-independent gesture catalogue to include in the specification of the user interaction of an information system.
Method component to generate platform-specific gesture specification: it permits stakeholders to obtain gestures specification according to the platform specified by them. The process includes a model-to-text transformation with the aim of produce a platform-specific gesture catalogue.
Method component to design gesture-based interaction: it consists of the specification of the relation between gestures and actions contained in an information system. A parsing process is executed on the user interface source code to determine the actions in such user interface that will be associated with previously defined gestures.
Method component to generate gesture-based interface. It is added source code in the user interface source code containing the gesture-based interaction.
Method component to test gestures. This component permits to test the defined previously gestures. If the user is not according with the definition of a gesture, it is possible redefine the gesture.
The method components have an execution sequence that permits obtain some products: gesture-catalogue model, gesture-based interaction model, platform-specific gesture specification, and gesture-based interface. Each product that is obtained in a method component is the input to other method component, as shown in Fig. 2. The method component “Definition of gestures” contains a process to define gestures in order to conform a gesture catalogue model. The information of the procedure of this method component is: (1) Input: set of strokes defined by coordinates (X, Y) and a timestamp; (2) Output: a platform-independent gestures catalogue; (3) Tool support: a user interface where the user draws a gesture.
The procedure of this method component comprises of different work steps to be performed:
The user sketches one or more strokes using his/her finger on the screen. These strokes are stored in a data structure, conforming a gesture catalogue model.
If the gesture is according to the requirements of the user/system, then the gesture is saved.
The set of gestures that have been sketched by the user defines the gesture catalogue model, which is conforms to gesture metamodel (see Fig. 3). The next method component is called “Generation of platform-specific gesture specification”. With this method component the platform-specific gesture specification is obtained by means of a model transformation. The information of the procedure of this method component is: (1) Input: the platform-independent gesture catalogue, the target platform and target folder; (2) Output: the platform-specific gesture specification; (3) Tool support: a model transformation.
The user specifies a set of gestures that will conform the platform-specific gesture catalogue. The gestures are selected from a repository containing gestures previously defined.
Then, the user specifies the target platform in order to apply transformation rules to obtain the specific solution.
Finally, the user specifies the target folder where the gesture catalogue will be stored.
Other method component is called “Design of gesture-based interaction”. In order to obtain this design, gestUI requires the specification of a user interface where actions (commands) will be executed using gestures. This method component applies a parsing process on the user interface source code with the aim of analyse it to obtain actions included in this code (e g., actions defined using structure “action perform” in Java). Then, the user defines the correspondence between gestures and actions selecting the gestures contained in the catalogue and specifying the action to execute by means of this gesture.
The information of the procedure of this method component is: (1) Input: user interface source code; (2) Output: the definition of the correspondence gestureaction; (3) Tool support: a user interface where the user specify the input to this procedure.
The different work steps to be performed in this method component are:
The user specifies a user interface source code, for instance, source code in Java containing “action listener” which specify actions to execute.
Parsing the source code in order to find actions included, e.g., actions defined in the “action listener” contained in the source code.
The user define a correspondence between actions and gestures previously selected from the gesture catalogue model.
The method component called “Generation of gesture-based interface” permits the generation of source code of user interface considering gesture-based interaction. The information of the procedure of this method component is: (1) Input: user interface source code, correspondence gesture-action; (2) Output: user interface source code containing gesture-based interaction; (3) Tool support: a user interface where the user specify the input to this procedure.
The different work steps to be performed in this method component are: The user interface source code is specified in order to include gesture-based interaction specification.
The correspondence gesture-action is specified.
A process is executed in order to analyse the source code and to insert the source code related with the actions and the gestures defined previously. The output of this process is the source code of the user interface containing the gesture-based interaction.
The last method component, called “Testing gestures”, permits that the users test the gestures defined with this method by means of an existent framework (e. g., iGesture, $N, quill). In this case, as first step, the gesture catalogue, previously defined using gestUI, is imported in the framework, then in the next step, the user employs the functionalities available in the framework in order to test the gestures. The information of the procedure of this method component is: (1) Input: gesture catalogue obtained by means of a model transformation; (2) Output: information about the gesture testing process; (3) Tool support: an existent framework to test the gesture catalogue.
The different work steps to be performed in this method component are: A gesture catalogue defined by means of gestUI is the input to this procedure.
The gesture catalogue is imported in the framework selected to test the gestures.
The user employs the gestures defined to do some tasks in the framework. If each gesture satisfies the requirements of the user, then the gesture is confirmed in the catalogue.
Else, the user requires the definition of a new gesture in order to replace the gesture that doesn’t satisfy the requirements of the user. 4
The tool support for design and runtime of solutions based on Capability Driven Development (CDD) is called Capability Design Tool (CDT) (see Fig. 4, left), it is designed as an integrated development environment built on the Eclipse Framework, using the Eclipse Modelling Framework (EMF) technologies. This tool supports capability modelling according to the capability metamodel including context modelling and goal, process and concepts models. CDT permits draw goal model, context model, CDD model, concept model. In this paper, we consider goal model in order to define the gesture catalogue to apply gestUI to define gestures. The elements of the palette corresponding to the goal model are shown in Fig. 4 (right). 5
gestUI can be applied in any information system with the aim of include gesturebased interaction in the user interface. In this paper, we consider CDT in order to apply this method to include gesture-based interaction in its user interface to draw goal diagrams.
The first step is the definition of gesture catalogue. In Table 1 we show an excerpt of the gesture catalogue definition in order to apply gestUI to draw goals models in CDT.
create Goal create Constraint Therefore, employing the procedure specified in the first method component the user defines a set of gestures in order to execute actions in CDT. Then, using a user interface, the user employs his/her finger to sketch strokes which define a gesture, see Fig. 5. Then, according to the description of this method component the output of the procedure is the set of gestures conforms to the gesture catalogue model (see Fig. 6), each gesture is defined as a set of strokes with points defined by coordinates (X, Y) and the corresponding timestamp.
Then, considering as input this gesture catalogue model and applying a model transformation we obtain a platform-specific gesture catalogue. In Fig. 7 is shown an excerpt of set of gestures according to the initial definition of gestures. The next method component is used to define the gesture-based interaction by means of the specification of the correspondence gesture-action, which is dependent on the function of the information system. In Table 1, it is specified the actions to execute using gestures defined.
The last method component permits the automatic generation of the source code including the gesture-based interaction with the customised gestures. In this case, we require as input the source code of the user interface, the specific-platform gesture catalogue. Then, the user select a gesture contained in the catalogue and an action defined in the user interface source code in order to define the correspondence gesture-action. Finally, the process has finished and the CASE tool called Capability Design Tool (CDT) has included the gesture-based interaction which can be used to draw goal models (Fig. 8).
gestUI, a model-driven method for developing multi-stroke gesture-based user interfaces is described in this paper using the “Template for the Documentation of Method Components in CaaS”. We demonstrate the application of the method and supporting tools in the CASE tool called Capability Design Tool (CDT) which is built using Graphiti (an Eclipse framework with the structure of a plug-in) that is used to draw diagrams. We produced the ‘Platform-specific gesture specification’ for this CASE tool in order to illustrate the multiplatform capability of the approach. The gestures were successfully recognised using $N as a gesture recognizer. Then, we automatically generated the final gesture-based interface components and integrated them into the user interface.
Some advantages of gestUI are: (i) its platform independence enabled by the modeldriven development paradigm, (ii) the convenience of including user-defined symbols and (iii) its iterative and user driven approach.
Future work will be developed along the following lines: (i) to include a feature to that the user can define gestures according to his/her preferences during the execution of the information system, (ii) developing a Technical Action Research in order to validate this method in the “Capability as a Service for Digital Enterprises” Project (CaaS project).
Otto Parra is grateful to his supervisors Sergio España and Óscar Pastor for their invaluable support and advice. This work has been supported by Secretaría Nacional de Educación, Ciencia y Tecnología (SENESCYT) and Universidad de Cuenca of Ecuador, and received financial support from Generalitat Valenciana under Project IDEO (PROMETEOII/2014/039).