In this paper we show how it is possible to represent the CARE properties (complementarity, assignment, redundancy, equivalence) modelling the temporal relationships among inputs provided through di erent modalities. For this purpose we extended GestIT, which provides a declarative and compositional model for gestures, in order to support other modalities. The generic models for the CARE properties can be used for the input model design, but also for an analysis of the relationships between the di erent modalities included into an existing input model.
INTRODUCTION
The characteristics of gestural input, such as its long
temporal extension, or the possibility to support parallel
interaction even for single-users, set di erent challenges
that are shared with multimodal interaction. In
particular, di erent modelling techniques and formalisms are
shared between gestural and multimodal interaction
research. As we discuss better in the related work section,
the considered modalities in literature are usually
graphical and vocal. However, the wide availability of input
devices allowing gestural interaction and the progresses in
voice recognition technologies require a general approach
for combining modalities, focusing more on User Interface
(UI) design rather than technology integration.
In this paper, we enlarge the spectrum of GestIT [
RELATED WORK
In this paper, we exploit a formal notation for de ning
the input sequences that can be recognized by the UI. The
idea of describing di erent types of input through a formal
notation has been widely investigated in literature, using
di erent formalisms. For instance, Finite State Machines
(FSM) have been integrated into widely adopted window
toolkits, such as Java Swing, by Appert et al. [
The combination of di erent interaction modalities
needed a formalism able to integrate di erent
concurrent information sources. In [
This paper starts from a compositional and declarative
model for gestural interaction and extends it in order
to support input from di erent modalities. We consider
the GestIT model in [
Ground terms GestIT models the input through an expression de ning the input temporal sequence. The ground term expressions represent the basic building blocks for such description. Each of them describes an atomic event, that is a noti cation that cannot be further decomposed. In general, they are associated to a value change of a feature, which is a single data tracked by an input device. For instance, we can de ne a ground term representing the current mouse pointer position, or a key press on a keyboard. In the gestural modality, we associate a ground term to each point tracked by the recognition device, such as the skeleton joints for MS Kinect, or the touch points for a multitouch screen. In the vocal modality, we can associate a ground term to the recognition of a word (or a phrase) pronounced by the user.
The noti cation can be optionally associated to a condition, which can be exploited receiving only a subset of all possible state changes associated to a ground term. This is useful in the gestural modality for receiving movements that follow a speci c trajectory (e.g. linear), or in order to calculate di erential characteristics of the gesture performance (e.g. the speed). In the vocal modality, we can associate a word or a grammar to an utterance recognition.
In this paper, we associate a symbol to each considered ground term. For instance, we indicate with Hr the position of the right hand, with Sl the position of the left shoulder etc.. We represent a generic ground term with the symbol Gt. For the vocal modality, we use the symbol V for indicating the recognition of a word.
A boolean predicate can be associated to a ground term for ltering the noti cation. We represent it through its name in square brackets, immediately after the ground term name: Gt[p] represents a generic ground term Gt associated to a predicate p. For instance V [quit] expresses the recognition of the word \quit", where V is the vocal ground term and quit is the predicates that restrict the recognition only to the considered word. A predicate p can be associated to more than one ground term in both the formal notation and in the library supporting the GestIT model.
In GestIT [
Composition operators Starting from ground terms, it is possible to de ne the input temporal sequence composing the di erent expression through the following set of temporal operators: Iterative Operator, represented by the symbol, repeats an input expression an inde nite number of times.
Sequence Operator, represented by the symbol, connects two (or more) expressions to be executed in sequence, from left to right.
Parallel Operator, represented by the symbol k, connects two (or more) expressions that can be recognized at the same time.
Choice Operator, represented by the symbol [ ], allows to select one among the connected components in order to complete the entire expression.
Disabling Operator, represented by the [> symbol, de nes that an expression stops the recognition of another one, typically used for stopping iteration loops. Order Independence, represented by the j j sym= bol, de nes that the connected sub-expression can be performed in any order.
The choice, parallel and order independence operators are commutative. The sequence, parallel, choice and disabling operator are binary and associative. The order independence operator is a n-ary operator, it can be de ned through a choice of sequences, as we detail better while describing equation 7.
In GestIT, a composed expression is represented by the class ComplexExpr, where it is possible to include the operands (either simple or composite) and to specify the temporal operator. This class is independent from the modality and it can be used for connecting ground terms from di erent input devices.
Auxiliary functions We de ne two auxiliary functions for de ning the CARE properties on input expressions. The rst function answers the following question: which modalities are needed for completing the considered input expression? Obviously, since it may be possible to provide the same information through di erent modalities, it is possible to exploit di erent sets of modalities for completing the same expression.
We formally model this association through the M od function, which maps an expression to a family of sets. The family contains all the possible sets of modalities that the user can exploit for completing the considered input expression. We provide an inductive de nition for this function in equation 1, starting from a ground term and showing how to build the family of sets for two generic expressions connected through a composition operator. M is the set of all possible input modalities.
M od : Exp 7! fS1:::Sng where Si
M; i 2 [1; n] M od(Gt) = fmg; m 2 M M od(Exp ) = M od(Exp)
Si 2 M od(Exp2)g M od(Exp1[ ]Exp2) = fSijSi 2 M od(Exp1) _ M od(Exp1 op Exp2) = fSi;jjSi;j = Si [ Sj;
Si 2 M od(Exp1); Sj 2 M od(Exp2)g op 2 f
; [>; j=j ; kg A ground term Gt is assigned to a single modality, therefore the function M od maps it to a single set with a single element. The iterative operator does not add any element to the family of sets.
The choice operator allows selecting between one of the sub-expressions. From the modality point of view, it is possible to choose only one of the sets provided either by the left or the right operand. Therefore, the resulting family of sets for the composed expression is obtained considering all sets from both operands.
For the other composition operators, the user needs to complete both operands, which means that she can select a set of modalities for completing the left operand and another set for completing the right operand. Therefore, in order to obtain the resulting family of sets, we rst calculate all possible pairs of sets, selecting one set from left and one set from the right operand families. After that, we calculate for each pair the union of the two sets and we obtain the resulting family for the composed expression.
The second function associates an input expression with a semantic label identifying which input data provides to the application. Considering two input expressions Exp1 and Exp2, the value returned by Input function is the same if they provide an equivalent input for the application. The possible values for this function are related to the speci c application. The function is de ned in equation 2: Datain is a set of semantic labels associated to all the possible user's input needed by the considered application, and d is a particular label. (1)
T ap [ ] N um T ap = Start1 This association between an expression and its semantic label represents the abstraction of the \message" that the system needs from the user for completing an operation. The same input can be provided through di erent modalities. For instance, we can consider an application allowing the user to enter a phone number either dialling it through a virtual keyboard or to pronouncing it vocally, as modelled in equation 3. In this case, we consider two di erent input expressions: one de ning the screen taps (T ap) and one for the vocal interaction (N um). The rst one is simply a sequence of a touch start (Start1) and a touch end (End1). The second expression allows the user to dictate the number digit by digit. Such expressions provide the same data to the application logic (represented by the phoneN umber label).
N um = V [zero] [ ] V [one] [ ] : : : [ ] V [nine] (3)
Input(T ap) = Input(N um) = phoneN umber
In this paper, we assume that the Input function has
been de ned for each expression in the input model. The
straightforward way for obtaining this mapping is to
require its speci cation from designers, but this may
limit the acceptance of the modelling technique.
We think that it is possible to provide an automated
labelling procedure based on the reverse engineering of
code. If we consider the T ap and N um expressions, the
behaviour associated to their completion should contain
a call to the same application logic, e.g. through a
command pattern [
MODELLING CARE PROPERTIES
The CARE properties [
In all de nitions, we suppose that the UI is currently in a state st and that the user wants to change this state to st+1. Exp is the input expression de ning all the possible UI options for reaching st+1 from st. In addition, we consider two modalities in the de nitions, but they can be easily extended to the general case. Assignment De nition. The modality m is assigned from state st to reach st+1 if no other modality can be used for moving from st to st+1.
Model. We can de ne this property on the expression Exp simply checking that all ground terms belong to the modality m, as de ned in equation 4.
Assignment(Exp; m) , M od(Exp) = fmg (4) Equivalence De nition. Two (or more) modalities m1 and m2 are equivalent for changing the state of an application from st to st+1 if it is necessary and su cient to use any one of the modalities for changing the state. This means that the user is free to select exactly one modality among the ones supported by the interface for completing one action.
Model. We can de ne the equivalence between two different modalities connecting the expressions for m1 and m2 with a choice operator as shown in equation 5 (Exp1 is assigned to m1 and Exp2 is assigned to m2). However, this is not su cient for being compliant with the de nition. For ensuring that st+1 is the same independently from which expression the user selects, we must ensure also the same type of input is provided by the user through both expressions. This is modelled by the condition Input(Exp1) = Input(Exp2).
, Exp = Exp1 [ ] Exp2 ^ M od(Exp1) = fm1g ^ M od(Exp2) = fm2g ^ Input(Exp1) = Input(Exp2) (5) Redundancy De nition. Two (or more) modalities are used redundantly to reach state st+1 from state st, if they have the same expressive power (e.g. the user provides the same information to the UI through di erent channels or vice versa).
Model. As discussed in [
P arRedundancy(Exp; fm1; m2g) , Exp = Exp1 j=j Exp2 ^ M od(Exp1) = fm1g ^ M od(Exp2) = fm2g ^ Input(Exp1) = Input(Exp2) , Exp = Exp1 jj Exp2 ^ M od(Exp1) = fm1g ^ M od(Exp2) = fm2g ^ Input(Exp1) = Input(Exp2) (6)
The two versions model the property in a similar way. The only di erence is the temporal operator connecting the two expressions Exp1 and Exp2, which are assigned to only one modality (respectively m1 and m2). The rst de nition, SeqRedundancy, connects the two expressions through the order independence operator, which guarantees that the two input sub-expression must be both completed in order to reach the state st+1. The user is free to select which modality she wants to use rst. Indeed the order independence is by de nition a choice between all possible sequences for executing the input actions, as shown in equation 7.
Exp1j=jExp2 = (Exp1
Exp2)[ ](Exp2
Exp1) (7)
In the second de nition, Exp1 and Exp2 are connected
through the parallel operator, which allows the execution
of the input actions assigned to the di erent modalities
at the same time. It is worth pointing out that the
sequences of actions recognized by expressions satisfying
SeqRedundancy can be also recognized by those that
satisfy P arRedundancy. However, as also remarked in [
In both cases, the command for changing the UI state is associated to the completion of Exp, which ensures that the UI changes its state only when the input from both modalities has been provided.
Complementarity De nition. Two (or more) modalities are used for reaching the state st+1 from st, all of them must be used for changing the state, but no one is able to complete the change individually.
Model. In this case, there is no need to enforce a particular temporal sequence for the input actions in Exp. The only requirement is that M od(Exp) contains a single set, whose elements are all the considered modalities. Considering the de nition we provided for the function M od, this guarantees that the user completes the input expression using all modalities at least once. The formal de nition is shown in 8.
, M od(Exp) = fm1; m2g
(8)
(9)
SAMPLE APPLICATION
In this section we redesign the interaction applying the
CARE properties to the touchless recipe browser
presented in [
All presentations respond to the input only if the user stands in front of the screen, in order to ignore movements or speech when they are not intended for interacting with the application (e.g. movements for chopping a carrot or talking with other persons in the kitchen). We can model this input ltering by recognizing the user pose in front of the screen. It is su cient that the user's shoulder joints are contained in a plane roughly parallel to the screen. The interaction is modelled in equation 9.
AppIn = F ront
F ront = (Sl[p] k Sr[p])
N otF ront = (Sl[p] k Sr[p]) The de nition of the expressions F ront and N otF ront is similar, since they track the parallel movement of the left and right shoulder joints (represented by the Sl and Sr ground terms). They are complementary, since all movements recognized by F ront are not recognized by N otF ront and vice versa. This is modelled by the p predicate, which tests whether the two joints are parallel with respect to the screen plane. N otF ront uses the logical negation of the same predicate.
Interact contains the expression de ning the interactions supported by the application, which we re ne later on. The sequence operator between F ront and Interact guarantees that no input is processed while the user is not in front of the screen. The disabling operator between Interact and N otF ront ensures that the input tracking nishes as soon as the user moves from the screen front position. Such design of the interaction, assigns the expression of the user's will to interact to the gestural modality, as demonstrated in equation 10.
M od(F ront) = M od(N otF ront) = = M od(Sl) [ M od(Sr) = g (10) While interacting with all presentations, it is possible to quit the application or to go back to the previous presentation. In order to avoid unintended terminations, we ask a con rmation for the quit command, as usual in many interfaces. In the vocal modality, the user says the word \quit" for quitting the application, while in the gesture modality we associate this command to a wave gesture.
The wave gesture is depicted in gure 1. For convenience, we set the origin of our coordinate system on the elbow of the considered hand, obtained at each frame simply de ning a translation of the original coordinate system, using the vector de ned by the elbow position. The gesture starts when the hand point reaches the second quarter in our coordinate system, with a positive Y and a negative X value, depicted in gure 1. Then, the user has to move the hand in the rst quarter of the coordinate system, with both values of X and Y positive, as shown in gure 1, part 2. At this point, there are two alternatives: either the user repeats the wave, returning to the situation in gure 1, part 1, or she can conclude the gesture moving the hand in the third quarter, as depicted in gure 1, part 3.
We model the gesture with the expression in equation 11. The position of the right hand is represented by Hr; x is true if the hand point has a positive value for the X coordinate, x is true otherwise; y is true if the hand point has a positive value for the Y coordinate, y is true otherwise.
(Hrf1; g[x ^ y] [> Hrf1; g[x ^ y])f1; g [> Hrf1; g[x ^ y] (11) In order to support the exit con rmation using di erent modalities, it is possible to follow di erent strategies. We can ask the user to provide the input sequentially or in parallel, since voice and gestures do not cause con icts. The two situations are modelled in equation 12, respectively by the Seqquit and by the P arquit expressions. Since the equivalence between the inputs entered through the expressions using the vocal and the gesture modality holds (considering the interaction semantics we associated them in this speci c case) we can conclude that both SeqRedundancy(Seqquit; fv; gg) and
Another strategy is to allow only one modality for the rst request (e.g. vocal) and then a con rmation through either the vocal or the gesture modality (the vice versa is also possible, with a symmetric modelling technique). In this case, the command received through the vocal command \enables" a selection between a gesture and the same vocal command for con rmation. The strategy is modelled in equation 13 by the AssEquivquit expression. We have an assignment property for the rst request and an equivalence property for the con rmation.
AssEquivquit = V [quit] (W ave[ ]V [quit]) (13) We assign the back command to the vocal modality (V [back]). Therefore, the interaction with a generic presentation (Interact, see equation 9) consists of a presentation-speci c (P resIn) expression in choice with the back and the quit command, as de ned in equation 14, where Expquit is a place holder for either Seqquit, P arquit or AssEquivquit.
Interact = P resIni [ ] V [back] [ ] Expquit (14) In the rst presentation, which is shown in gure 2, the user points a target on the screen and selects it closing the hand. The interaction is modelled in equation 15: the user moves the dominant hand an inde nite number of times (Hr ), until she selects an item closing the hand (we indicate the feature for the hand opening with oHr, and we restrict the recognition only to the closed state). In this case, we assigned the interaction to the gestural modality. The interaction for the dish selection presentation exploits the same gesture set.
P resIn1 = Hr [> oHr[closed] M od(P resIn) = g (15) The presentation for preparing the selected dish is shown in gure 3. It allows reading the description of the steps that are needed for cooking a dish, through a text and a video. The video can be played continuously, in order to have an overall idea on the whole preparation process. Otherwise, the video can be played step-by-step, pausing the playback at the end of each step. In this case, the user requests to watch the next (or previous) step explicitly. In addition, she can jump randomly from one step to another, moving the video timeline knob. Finally it is possible to change the volume setting for the video description. The interface supports the interaction with the expression in equation 16. The change between the continuous and the step-by-step playback is supported through the \continuous" and the \step" vocal commands (respectively V [cont] and V [step].
In order to navigate the recipe steps, the user can select among the gestural and the vocal modality. The next step (N ext) can be visualized through the \next" vocal command (V [next]) or (choice operator) through a swipe from left to right. The latter gesture can be modelled through an iterative movement of the dominant hand with a speed higher than a speci c threshold (Hr [spr]), disabled by a movement slower than this threshold (Hr[spr]). We use a symmetric modelling approach for visualizing the previous step.
The steps can be randomly navigated performing a grab gesture and dragging the timeline knob (Random): it consists of closing the dominant hand (oHr[closed]), followed by its iterative movement (Hr ), disabled by opening the hand (oHr[open]). The same gesture can be used for changing the volume setting, after having pronounced the \volume" vocal command.
The V olume expression in equation 16 satis es the complementary property, since M od(V olume) includes only one set containing both the vocal and the gesture modality. This means that the input expression cannot be completed without using all the considered modalities. Instead, in N ext we have an equivalence between the two sub-expressions since they provide the same input through di erent modalities (the same holds for P revious). The \continuous" and the \step" commands are assigned to the vocal modality.
P resIn3 = V [cont] [ ] V [step] [ ] V olume [ ]
N ext = V [next] [ ] (Hr [spr] [> Hr[spr])
P rev = V [prev] [ ] (Hr [spl] [> Hr[spl]) Random = Drag
Drag = oHr[op]
Hr [> oHr[op] M od(V olume) = fg; vg
M od(V [step]) = M od(V [cont]) = fvg M od(Random) = fgg Input(V [next]) = Input(Hr [spr] [> Hr[spr]) (16) CONCLUSIONS AND FUTURE WORK In this paper we extended an existing declarative model for gesture de nition in order to support other modalities. With this extension, we provided the de nition of the CARE properties for managing multimodality, representing them through the temporal input modelling. This is useful for providing automatic model checking procedures, able to identify the properties inside existing models. This can support designers in the UI creation process or during the reverse engineering of existing applications. In future work, we aim to include the multimodal support in the next release of the GestIT library, together with tool support for the model creation and checking. ACKNOWLEDGEMENTS We gratefully acknowledge Sardinia Regional Government for the nancial support (P.O.R. Sardegna F.S.E. Operational Programme of the Autonomous Region of Sardinia, European Social Fund 20072013 - Axis IV Human Resources, Objective l.3, Line of Activity l.3.1 \Avviso di chiamata per il nanziamento di Assegni di Ricerca"