=Paper=
{{Paper
|id=Vol-1419/paper0077
|storemode=property
|title=Pointing to an Invisible Object behind a Wall: Comprehension of Pointing with a Bent Index Finger
|pdfUrl=https://ceur-ws.org/Vol-1419/paper0077.pdf
|volume=Vol-1419
|dblpUrl=https://dblp.org/rec/conf/eapcogsci/KobayashiY15
}}
==Pointing to an Invisible Object behind a Wall: Comprehension of Pointing with a Bent Index Finger==
https://ceur-ws.org/Vol-1419/paper0077.pdf
Pointing to an invisible object behind a wall:
Comprehension of pointing with a bent index finger
Harumi KOBAYASHI (h-koba@mail.dendai.ac.jp)
Division of Information System Design, Tokyo Denki University,
Ishizaka Hatoyama, Saitama, 350-0394 Japan
Tetsuya YASUDA (t-yasuda@jumonji-u.ac.jp)
Department of Human Developmental Psychology, Jumonji University
Sugasawa Niiza, Saitama, 352-8510 Japan
Abstract understanding of pointing at referents and placing referents
Pointing is a gesture that people use to specify and convey
in different situations. Clark, Schreuder, & Buttrick (1983)
information about objects in the environment. Previous and Clark (1996) proposed that people use “common ground”
research has mostly explored peoples’ comprehension and as implicit mutual knowledge in human communication.
production of others’ “straight” pointing gestures, that is, Common ground can include a variety of information: how
pointing at an object with arm and index finger kept straight people convey information using words and sentences,
while aiming at a visible object. However, we seem to use knowledge about the history and culture of the speakers and
various types of pointing in addition to typical straight addressees, mutually shared knowledge about specific
pointing to denote both visible and invisible objects. This
study examined comprehension of pointing with a “bent” people and events, and knowledge of what is going on in the
index finger at an invisible object behind a wall. The current communication. Thus, common ground includes
experimenter pointed either at an object in front of a wall or both general knowledge about the world and specific
one behind a wall with a straight index finger or a bent index knowledge of the specific task that the conversation partners
finger, and the participants guessed which object was being are conducting. Previous research has focused mostly on
denoted. The results were that when the participants looked at language and verbally describable information included in
straight pointing, they thought objects in front of the wall
common ground. Non-verbal information such as gestures
were being denoted. However, when they looked at bent
pointing, they thought objects behind the wall could be must also be comprehended using common ground as to
denoted. The study suggested that people have “common how people use gestures in different situations; however,
ground” in terms of interpretation of different types of usage of gestures as common ground has not yet been
pointing. thoroughly explored.
Keywords: gesture; declarative pointing; common ground; Some research has explored peoples’ comprehension and
non-linguistic information production of pointing gestures when they use some
language such as demonstratives such as “this” or “that”
Introduction while indicating objects in the environment (Bangerter,
2004; Coventry, Valdés, Castillo, & Guijarro-Fuentes,
When we communicate with others, we often draw their
2008; Coventry, Griffiths, & Hamilton, 2014). Most of
attention to objects about which we wish to communicate
pointing studies have examined typical pointing gestures,
(Tomasello, 2008; 2014). Pointing is a gesture that people
we named it “straight” pointing because pointing at an
use to specify and convey information about referents. For
object is done with the addresser’s arm and index finger
example, when one asks a friend about a landmark in the
kept relatively straight while aiming at a visible object in the
city, she will point at the landmark using her index finger or
environment (Coventry et al., 2008; Doherty, Anderson, &
her hand.
Howieson, 2004; Jaswal & Hansen, 2006; Kobayashi, 2007).
Clark (2003) discussed use of attention-getting gestures
In this situation, the addresser can easily share information
in various cases. He noted that pointing at a referent and
about visible objects, using visual joint attention and
placing a referent are both useful ways to convey
common ground.
information about referents, but that people index objects
How, then, can we point at invisible objects such as
differently. In pointing, a person directs the addressee’s
objects behind obstacles? In the authors’ lab, we observed a
attention to the referent object; for example, a customer may
person pointing at a magnet pin that was attached on the
point at a package of a medicine that is difficult for her to
other side of a steel board. The addresser bent his index
reach but is easy for the clerk. In placing, a person puts a
finger when he pointed at the invisible, but known, magnet
referent object in the area of an addressee’s attention; for
pin. We observers immediately understood the meaning of
example, a customer may place a package of medicine on
his pointing gesture, although pointing with a bent index
the checkout counter where a clerk waits. These
finger seems relatively unusual. We might have common
communications are possible without saying any words. In
ground with regard to non-verbal gestures, or at least
order to communicate smoothly, people must share mutual
477
knowledge about how we should interpret others’ various, of the table between object #2 and object #3. Participants
occasionally unusual, gestures. In the case of referring to were randomly assigned to all conditions.
invisible objects, sharing information about the referent may
be more difficult for both the addresser and the addressee
because visual joint attention is difficult to establish. The
role of common ground in human non-verbal gesturing
seems to be more important when people refer to invisible
objects.
We examined people’s common ground regarding
pointing gestures. This study focused on comprehension of
pointing with a bent index finger at an invisible object
behind a panel. Because this is the first study to examine the
Fig. 1: Examples of the two types of pointing gestures: “a”
role of a bent index finger, we did not examine the
possibility that the general posture of the arm and the index denotes the shape of the “straight” pointing; “b” denotes the
finger as a whole may have a role in this study. The shape of the “bent” pointing.
experimenter pointed at an object in front of a panel or
behind a panel with a straight index finger or a bent index
finger. There were objects either in front of or behind the
panel, and the participants guessed which object was being
indicated. We expected that if the experimenter used a bent
index finger in pointing, the participant would interpret this
as referring to the object behind the panel, but if he used a
straight index finger in pointing, the participant would
interpret this as referring to the object in front of the panel.
The reason is that the bent index finger seems to suggest
that the “pointing trajectory” (imaginary trajectory of
pointing gesture) can go over the panel.
Method
Fig. 2: Experimental setup in the with-obstacle condition.
Participants
Each object was placed 10 cm apart from the adjacent bottle.
Twenty Japanese undergraduate university students (M
age = 21.2 years; 3 females) participated. The experiment First, the experimenter and the participant looked at all
was conducted in accordance with Tokyo Denki the bottles placed on the table. Each bottle was placed 10
University’s code of ethics. cm apart from the adjacent bottle. Bottle #1 was placed 30
cm away from the edge of the side of the table where the
Procedure experimenter sat. Then, the participant sat on the
The experimental conditions consisted of two types of experimenter’s chair and looked at the table. Then, the
pointing (straight vs. bent) and obstacle placement (with vs. experimenter put the panel in between bottles #2 and #3,
without). and the participant again looked at the table. Thus, the
With regard to pointing condition, “straight pointing” was participant experienced the experimenter’s view in both
when the experimenter pointed at the referent with her arm with the obstacle and without the obstacle conditions (Fig.
extended horizontally and her index finger kept straight (Fig. 3). Then, in the straight pointing with the obstacle condition,
1a), “Bent pointing” was when the experimenter pointed at the experimenter put the panel in between bottles #2 and #3
the referent with her arm extended slantwise and her index and said to the participant, “I cannot see bottles #3 and #4.
finger kept somewhat bent (Fig. 1b). The “with-obstacle” Now, I will point at one of the four bottles.” Then, the
condition was when there was a small black opaque panel experimenter pointed at bottles using either with the straight
(W: 25 cm x H: 40 cm) on the table. “Without-obstacle” index finger or the bent index finger. With each pointing
was when there was no panel on the table. gesture, the experimenter said, “Now I am pointing at
Fig.2 shows the experimental setup. On the table, there something. Which bottle would you guess I am pointing at?
were 4 small bottles (W: 2.3 cm × H: 8 cm) designated 1, 2, Please answer with the number of the bottle.” The
3, and 4, respectively, on a label of each bottle. The participant responded orally using the bottle number. The
experimenter sat on one side of the table, wearing black bottle number corresponded to the distance from the edge of
sunglasses during the experiment so that participants could the table: Bottle #1’s distance was 30 cm; #2, 40 cm; #3, 50
not see the experimenter’s gaze direction. The participant cm; and #4, 60 cm, respectively. In addition, the bottle
sat at the table at a right angle to the experimenter. In the numbers corresponded to visible or invisible status within
with-obstacle condition, the panel was placed in the middle the with-obstacle condition: bottles #1 and #2 were visible,
478
Fig. 3: Experimenter’s view during the experiment in the without-obstacle condition (left) and in the with-obstacle
condition (right). Before the experiment, each participant first sat on the experimenter’s chair and looked at the table with
and without the obstacle to know the experimenter’s views of the both conditions.
and bottles #3 and #4 were invisible. Consequently, the = 160.457, p < .001, ηp2= 0.894. There were also significant
participants answered using a scale of 1,
2, 3, and 4. In the interactions of Pointing × Obstacle, F(1,19) = 6.935, p
without-obstacle condition, all four bottles were visible. < .005, ηp2 = 0.300, and Pointing × Distance, F(3,57) =
Because the experimenter wore sunglasses, the participant 6.935, p < .005, ηp2 = 0.148.
could not see the experimenter’s eye gaze. To explore the significant Pointing × Obstacle interaction,
In the bent-pointing with-obstacle condition, the the simple main effects of Pointing within each Obstacle
procedure was the same as with the straight pointing with- condition and the simple main effects of Obstacle within
obstacle condition except that bent pointing was used. In the each Pointing condition were analyzed. Pointing differences
without-obstacle conditions, after both the experimenter and in pointing with the obstacle (F(1,38) = 34.139, p < .001, ηp2
the participant looked at the four bottles, the experimenter = 0.473.) and without the obstacle (F(1,38) = 80.265, p
pointed at a bottle in random order, and the participant < .001, ηp2 = 0.679.) were significant. Obstacle differences
guessed which bottle was being pointed at. in the straight pointing were significant, F(1,39) = 13.023, p
There were two pointing conditions (straight and bent), < .001, ηp2 = .255. To explore the significant Obstacle ×
and in each pointing condition, there were two obstacle Distance interaction, the simple main effects of Pointing
conditions (with obstacle, without obstacle); in each within each Obstacle condition and the simple main effects
pointing and obstacle combination, there were four bottle of Obstacle within each Pointing condition were analyzed.
(distance) trials. There were 4 blocks in the order of Obstacle difference was significant for object #4, F(1,76) =
pointing, straight-pointing and with-obstacle, straight- 12.552, p < .001, ηp2 = 0.142. Distance differences in the
pointing and without-obstacle, bent-pointing and with- straight and the bent pointing conditions were significant,
obstacle, bent-pointing and without-obstacle. In each block, except when the experimenter pointed at objects #3 and #4
the order of the bottle was random, and there were totally 16 in the without-obstacle condition (p < .05).
trials. Overall, the order of these blocks was
counterbalanced between the participants. Discussion
The experimenter was trained to show the same pointing This study examined how people interpret the
gesture in either the bent or the straight conditions in the experimenter’s pointing with a bent index finger at an
aspects of speed of the movement, the height of the wrist, invisible object behind a panel. The experimenter pointed at
and the distance from the participant’s body. bottles that were placed either in front of the panel or behind
the panel using a straight index finger or a bent index finger,
Results and the participants guessed which object was being
Fig. 4 shows the participant’s responses when the indicated. The results were that in the with-obstacle
experimenter pointed at each object in each condition. A 2 condition, straight pointing tended to be interpreted as
(Pointing: straight, bent) × 2 (Obstacle: with, without) × 4 referring to objects #1 and #2, whereas in the without-
(Referent: #1, #2, #3, #4) ANOVA was performed with the obstacle condition, straight pointing tended to be interpreted
number of the bottle that the participant responded as the as referring to all objects, including objects #3 and #4.
dependent measure. There were significant main effects of However, interestingly, the situation was different when
Pointing, F(1,19) = 78.042, p < .001, ηp2= 0.804; Obstacle, bent pointing was used. In the with-obstacle condition, bent
F(1,19) = 6.163, p < .01, ηp2= 0.245; and Distance, F(3,57) pointing tended to be interpreted as referring to all the
479
Fig. 4: Participant’s responses when the experimenter pointed at each object in each condition. Here, “a” denotes the
straight pointing condition, and “b” denotes the bent pointing condition. X-axis shows the bottle number that corresponds
to the distance from the edge of the table: Bottle 1’s distance was 30 cm; Bottle 2, 40 cm; Bottle 3, 50 cm; and Bottle 4,
60 cm, respectively. The bottle numbers also correspond to visible or invisible status in the with-obstacle condition:
bottles #1 and #2 were visible, and bottles #3 and #4 were invisible in this condition. In the without-obstacle condition, all
four bottles were visible. Y-axis shows the number of the bottle the participant responded.
objects, including #3 and #4, and there was no difference
between the with-obstacle condition and the without- Acknowledgments
obstacle condition. Thus, the results indicate that
participants interpret the straight pointing as referring to all We would like to thank all participants who participated
objects when the panel was not present and the objects in in our experiment. We thank Kota Kuwayama for data
front of the panel when the panel was present. In contrast, collection. This study was partially supported by the Japan
they tend to think the bent pointing as referring to all Society for the Promotion of Science (JSPS): Grant-in-Aid
objects irrespective of the presence or absence of the panel.
for Scientific Research (C) #24530793 (H.K.), and Grant-in-
The current experiment did not disentangle if the effect
Aid for Young Scientists (B) # 26870549 (T.Y.).
could be due to the bent index finger or to the general
posture of the arm and the index finger as a whole. In future
References
research, the roles of the bent index finger per se and it and
Bangerter, A. (2004). Using pointing and describing to
the arm as a whole must be examined. In addition, the achieve joint focus of attention in dialogue.
current experiment did not perfectly control the speed of the Psychological Science, 15, 415-419.
pointing gesture. The speed of pointing may have an effect Clark, H. H. (1996). Using language. Cambridge:
in estimating the distance of the “imaginary trajectory.” Cambridge University Press.
Clark, H. H. (2003). Pointing and placing. In S. Kita (Ed.).
Future research must examine this issue.
Pointing: Where language, culture, and cognition
In conclusion, the study showed that people could meet (pp. 243-268). Mahwah, NJ: Lawrence Erlbaum.
interpret pointing at an invisible object when bent index Clark, H. H., Schreuder, R., & Buttrick, S. (1983). Common
finger was used in pointing. It suggests that people know ground at the understanding of demonstrative
the meaning of the “bent” index finger based on “common reference. Journal of Verbal Learning and Verbal
Behavior, 22, 245-258.
ground” in their interpretation of different types of pointing.
480
Coventry, K. R., Griffiths, D., & Hamilton, C. (2014). Jaswal, V. K. & Hansen, M. B. (2006). Learning words:
Spatial demonstratives and perceptual space: children disregard some pragmatic information that
Describing and remembering object location. conflicts with mutual exclusivity. Developmental
Cognitive Psychology, 69, 46-70 Science , 9, 158–165.
Coventry, K. R., Valdés, B., Castillo, A., & Guijarro- Kobayashi, H. (2007). The effect of touching object parts on
Fuentes, P. (2008). Language within your reach. Near- learning novel object part names among young
far perceptual space and spatial demonstratives. children and adults. Studies in Language Sciences 6,
Cognition, 108, 889-895. 61-76.
Doherty, M. J., Anderson, J. R., & Howieson, L. (2009). Tomasello, M. (2008). Origins of human communication.
The rapid development of explicit gaze judgment Cambridge: MIT press.
ability at 3 years. Journal of Experimental Child Tomasello, M. (2014). A Natural History of Human
Psychology, 104, 296-312. Thinking. Harvard University Press.
481