The spontaneous gestures that accompany spoken language are particularly suited to conveying spatial information, yet their briefness, individuality, and lack of conventional linguistic structure impede their integration into NLU systems. The current work characterizes spontaneous size gestures in a manual task corpus, clarifying their form, discourse role and representation as a first step toward incorporating them into NLU systems.
When gesture carries the primary load of communication, as in the major sign
languages, it develops linguistic properties such as verb subcategorization [
Yet the spontaneous hand gestures that accompany speech are increasingly
recognized both as a cognitive aid to the gesturer, and an encoding of meaning [
The current work pursues the incorporation of spontaneous gesture into NLU systems: much groundwork must be laid. Amid the fluidity and abstractness of spontaneous gesture, we focus on concrete gestures with (relatively) straightforward spatial interpretations. We seek to answer the questions:
What is the discourse purpose of the gestures? Do the gestures constitute intended communication? To what extent are they lexicalized? What are their semantics?
How can they be related to the semantics of the co-ocurring speech? We collected a reference corpus for dialogue with intonation and gesture in a physical task context. The subjects were twelve pairs of University of Chicago undergraduate and graduate students, who were familiar with each other and had some cooking experience. They were recorded while performing a 30-45 minute cooking task (making chocolate truffles), using a single camera and lapel microphones. Some elements of the task include locating ingredients and equipment, dividing the labor, choosing flavorings, and activities such as measuring and washing up.
The resulting eight hours of videotape were examined for spatial gestures. These
included pointing, displaying, miming of physical actions and manner[
All of the size gestures in our corpus stemmed from the recipe step: “Take a hunk of set ganache and roll into a walnut-sized ball between your palms.” An example can be seen in Illustration 1, where subject Chris reads the recipe step aloud, envisions the ball he will roll, and enlists Jason to confirm the ball size. In total he performs the gesture for about three seconds; Jason eventually turns his head to view it for about 800ms. We will refer to this example and similar gestures as 'the ball size gesture'.
Of twelve pairs of subjects, two did not communicate about truffle size beyond reading the recipe. Ten discussed truffle size verbally; of these, three did not use gestures, and three used displays of ganache (dough). Four used size gestures: three ball size gestures and one caliper size gesture1. Gestures were used in two main ways: to inform the partner of a desired size, or to request confirmation that a size was correct. In one case, multiple ball gestures were used to explain how an incorrect ball size leads to difficulties in baking. All gestures were used with co-occurring speech. 1 A 'caliper gesture' shows the size of a small object using parallel thumb and forefinger .
We classify five of the seven gestures as intended communication, on the basis that: in three cases the gesturer used motion or location to attract visual attention; in two cases the gesturer made a verbal reference to the gesture (e.g.“like this?”), and in one case both were used. For the seventh gesture (the incorrect ball size explanation) we have no evidence that the gesture per se was intended communicatively. A further analysis of gaze and uptake in these cases is in progress. Although this is a very small sample, most of these gestures showed evidence of communicative intent.
We initially suspected that the ball size gesture was strongly lexicalized in comparison with spontaneous gesture generally. In all cases the thumb and forefinger circle to touch each other and embrace a notional ball, and are displayed as the focal side of the gesture. However, there is notable variation in other parameters. Either hand could be used, as in ASL. The position of the other three fingers is not conventionalized (where it might or might not be constrained in a sign language.) The location of the gesture relative to the gesturer is not as conventionalized as it would be in ASL. In the table, we refer to the gesturer as G and the observer as O.
The third column, the explanation of how two balls may melt into each other while baking, is more typical of spontaneous gesture in showing dynamic configurational elements with extended duration. The ball size gesture is not as conventionalized as an ASL gesture – nor can we say what lexicon it would belong to. More work is needed on this point. The ball size gesture contrasts with the caliper gesture in form.
Orientation Location Path
Duration (ASL=250ms) left 'OK' splayed
At G's eye level static >3000ms (G) > 700ms (O) right 'OK' curled
both 'OK', 'OK' splayed, splayed
Finally we consider semantic representation. A size is a property of a physical object,
generally represented as a value on a scale, where a scale is a partial ordering on a set
of elements. The majority of verbal size descriptions followed the recipe text: 'the
size of a” small object, or simply mentioned a small object: walnut, half a walnut,
meatball. The comparative “...smaller”, and (negated) intensifier “don't make it too
big!” also occurred. The scale in this case seems to be based on the generics (types)
of ball shaped food items, and the asserted relation is purely qualitative. Qualitative
representations [