We deal with a yet untreated issue in debates about linguistic interaction, namely a particular multi-modal dimension of meaning-dependence. We argue that the shape interpretation of speechaccompanying iconic gestures is dependent on its co-occurrent speech. Since there is no prototypical solution for modeling such a dependence, we offer an approach to compute a gesture's meaning as a function of its speech context.
Speakers often convey multi-modal content by
pointing at things or shaping their contours while
talking. The semantics of the verbal part is
intertwined not only with the communicative
situation and the agent’s informational situation, but
also with the semantics of the non-verbal part.
So, one information providing system (gesture)
depends on another one (language) for its
interpretation. In gesture research, there are at least
three claims about how a gesture’s interpretation
depends on its accompanying speech context: (i)
The classification of gestures is speech-dependent
(see, e.g.,
The iconic gestures we are concerned with are
spontaneous movements of hands or fingers that
do not have a lexical meaning. Here, we employ
McNeill’s conception of a stroke and its
semantic synchrony with the accompanying speech
Humans do not gesticulate geometrical shapes.
If one takes a closer look at roundish-looking
gestures, one quickly notices that such gestures are
mostly if not always spiral. If a speaker iterates
such a sloppy gesture, it looks helix-like.
Moreoever, gestures that are intended to be angular
are often roundish. This sloppiness is
presumably due to the physiological features of humans,
time limits, etc. Despite this fact it is common
to interpret gestures as conveying meanings like
round0 or square0. It seems natural to interpret,
say, a roundish gesture as an imperfect sign for the
meaning round0. Roundish gestures can be
interpreted as approximating geometrical shapes like
circles. If so, the gesture’s speech-independent
morphological features alone, such as its hand
shape, movements, could provide the core of the
gesture’s meaning. This view has been (implicitly
or explicitly) suggested by authors of formal
theories of gesture meaning (which range from
employing HPSG (e.g.,
First, gestures that share all relevant morphological features (i.e., that are of the same type) can be interpreted differently given different speech contexts. If a helix-gesture accompanies an utterance like ‘The window is round’ it is likely to be taken as meaning circular0 or round0. If it accompanies ‘The townhall features a staircase’ it is likely to be interpreted as meaning spiral0. Depending on the standard of precision at stake, a roundish gesture might be interpreted as conveying round0 when accompanied by ‘ball’, but as conveying angular0 when accompanied by ‘box’. Such an ambiguity is also found when the sloppiness of the gesture is extreme. Take a look at the examples given in Fig. 1. In Fig. 1a the speaker is uttering ‘But not round spiral staircases, but so eh. If the house is rectangular, can the stairs outside be [truncation].’ (English translation, gesture stroke underlined) The emphasis on ‘rectangular’ and the overlapping stroke together with other parts of the dialogue suggests that the
ometrical shapes, such as circles and cylinders, for their
underspecified gesture meanings (e.g.,
Second, gestures with a significantly different gesture morphology can represent the same meaning. For instance, different gestures can convey the meaning rectangular0 if they relate to the same utterance segment, etc. Take as examples the ones shown in Fig. 2. In Fig. 2a the speaker utters ‘It is just a rectangular building.’ Compare this displaying of rectangular0 with Fig. 2b which is identical to Fig. 1a. Although the gestures display some similarity, they are clearly different. Nonetheless, they both seem to mean rectangular0 or angular0. Here, the general observation is that different types of gesture accompanying the same or semantically similar utterance segments can select the same gesture meaning as value: (II) Different types of gestures accompanying the same utterance segment have one and the same meaning. (I) and (II) support the idea that the meaning of an iconic gesture is determined to a significant extent by the meaning of its accompanying speech.2
(a) ‘rectangular building’ (b) ‘rectangular house’ This dependence of gesture meaning on speech meaning has not been modeled. The gesture theories mentioned above could only cope with it by substantially underspecifying the gesture’s meaning. This would allow the meaning of, say, a spiral gesture to be compatible with utterance segments with conflicting meanings, such as ‘round’ and ‘rectangular’. But this would render the gesture’s meaning too weak. It would not allow for recognizing the gesture’s contribution to the communicated content and it would not fit the intuition that iconic gestures have a rich meaning on their own. There is also no prototypical solution to be found in other formal semantics: Formal semantics travels the inverse route, so to speak, modeling the context dependence of speech, whereas we model a dependence on speech as context.
A new model of the meaning of iconic gestures should meet at least the following desiderata: (a) The meaning of a gesture is determined to a significant extent by the meaning of the accompanying speech. A similar gesture morphology is not sufficient for a similar/identical meaning and a different gesture morphology is not sufficient for a different meaning. (b) Nonetheless, its morphology is not irrelevant for determining a gesture’s meaning. Not just any gesture can have the meaning round0, for instance, a clearly articulated angular gesture cannot. So, a gesture’s meaning is not completely determined by speech. Moreover, gesture content can contradict speech meaning. Our corpus has one remarkable instance in which a ‘cup-upwards-word’ is accompanied by a ‘cupdown-wards’ gesture.
From a formal point of view, (II) does not present new obstacles over and above those encountered in the context of observation (I). A roundish gesture accompanying, say, ‘clock’ or ‘window’ could either be drawn with one indexnot, in principle, restricted to gesture-word relations. finger or shaped or modeled with both hands. According to our annotation practices, these would be different gestures, in part due to the different handshapes used. In addition, more subtle differences in terms of gesture morphology could arise. According to the account presented here, the different gestures might all yield JroundK if combined with JclockK or JwindowK.3 Arguments supporting that would have to be given for (I), too.
For (I) our account has to specify the speechdependent meaning of the gesture. Here is an outline of our approach: The gesture meaning is a function of the gesture’s initial (topological) meaning based on its morphology and the speech context. The gesture’s morphology is described by attribute-value pairs (AVMs) concerning hand shape, movements, etc. One computes the initial meaning of the gesture mapping the AVMs onto a logical formula. The final gesture meaning is a function of the initial meaning and the speech context. Then, speech meaning and final gesture meaning can be combined to gain a multi-modal proposition (see Fig. 3).
Multi-modal Proposition Speech and Gesture Interface via Compositional Semantics
Final Gesture Meaning Initial Gesture Meaning Formal Meaning/ Partial model of speech context
Syntax Analysis
AVM-Representation Speech Transcription
Gesture Annotation
namic semantics, because we need a device that
is able to model the evolution of the interpretation
of gesture processes and speech processes as well
as their interaction. The interaction handles
compositionality of non-speech and speech meanings.
No known static semantics can fulfill such
desiderata. We use the -calculus, a recent extension of
Milner’s -calculus (see,
count of gesture meaning. We suggest a change of the initial meaning gained from the described morphology. It is triggered by the meaning of the accompanying speech, given that restrictions like the satisfaction of an approximation function hold.
Implementing our approach roughly works as follows: The initial semantics of the gesture formulated in -terms is passed onto a channel containing the gesture’s speech context. The speech context may modify the gesture meaning in various ways (see, e.g., (I)). Assume that the (a) Basic intuition: contact point of spiral gesture and word gesture’s initial meaning is spiral0, its speech ‘ball’. The spiral gesture + ‘ball’ yields round0, according to context ball0. Roughly, ball0 is sent to spiral0 (b). which changes it to round0 and finally uses it as ch1 bae ch2 ro < λz∃f ∃c∃r∃thrc (spiral0 ∧ a modifying information. So, transported and approximates(f (spiral0), c, x) = r ∧ modified meanings are treated in the end as fixed r ≥ thrc ∧ circle0(x) ∧ context(c) ∧ z ∈ iploluinsttrsa.tedI.n TFhieg.exa4mypoleu ucttaenrasnecee itshe‘Nbeabseicn dideema [J{rcJoucuipnr-cdbl0eo/Ktrt,ooJm]c[leKo,lcs.ke.-].f[}a⊥c)/e→rK,oJ]m>ir(rboarKe,)JsignK, JballK, Ball ist eine Kiste.’ (Engl.: ‘Next to the ball there is a box.’) As shown in Fig. 4a, the idea is (obf,) sIaf-ye,lJsbearlulKleafsorroiunntder0preting a spiral gesture in the context that a spiral gesture in the context of objects like JballK and other roundish things designates round0 Figure 4: Modeling with the - -calculus. (observe the use of meta-language and object language expressions here which is vital) and ? (undefined) else. So, the multi-modal meaning 4 Conclusion and further research of ‘ball’ + spiral gesture is ball0(x) ^ round0(x). We argued that gestures have a speech-dependent More specifically, if the partial model input ‘J: : :K’ meaning and proposed to model their meanings as to (2), instantiating bae, yields z 2 fJcircleK; a function of the gesture’s initial meaning and the Jclock-faceK; JmirrorK; JsignK; JballK; Jcup-bottomK; speech context employing the -calculus. On ac: : : g and the projection of spiral0, f (spiral0), ap- count of this, gestures with the same morphology proximates circle0 in context c to degree r can have even conflicting meanings if they appear the threshold in c then round0 is substituted for in different speech contexts, e.g., we can assign ro, [round0=ro], and output on ch2; else ? is sub- meanings like rectangular0 vs. circular0 to similar stituted for ro, [?=ro], and is output on ch2. The gestures. For future research we aim at integratz 2 clause and the threshold shall guarantee that ing the speech context’s influence on the gesture not just any gesture can mean round0. Gestures classification and individuation as well as the role accompanying a phrase whose extension is not of rhetorical relations, and at expanding our model an element of the set (say, ‘square’), as well as for analyzing more complex gestures. gestures that do not approximate a circle to the context-sensitive threshold cannot mean round0. 5 Acknowledgements The threshold can be determined algorithmically through a simulation device as shown in Pfeiffer et al. (2013) for two-dimensional cases. For three-dimensional cases we still rely on intuition.
We are grateful to three reviewers for their critical
comments. We tried to accommodate most, but
some suggestions, such as the generalization of
the model to discourse data (see, however,