Pointing is a gesture that people use to specify and convey information about objects in the environment. Previous research has mostly explored peoples' comprehension and production of others' “straight” pointing gestures, that is, pointing at an object with arm and index finger kept straight while aiming at a visible object. However, we seem to use various types of pointing in addition to typical straight pointing to denote both visible and invisible objects. This study examined comprehension of pointing with a “bent” index finger at an invisible object behind a wall. The experimenter pointed either at an object in front of a wall or one behind a wall with a straight index finger or a bent index finger, and the participants guessed which object was being denoted. The results were that when the participants looked at straight pointing, they thought objects in front of the wall were being denoted. However, when they looked at bent pointing, they thought objects behind the wall could be denoted. The study suggested that people have “common ground” in terms of interpretation of different types of pointing.
When we communicate with others, we often draw their
attention to objects about which we wish to communicate
Some research has explored peoples’ comprehension and
production of pointing gestures when they use some
language such as demonstratives such as “this” or “that”
while indicating objects in the environment
How, then, can we point at invisible objects such as objects behind obstacles? In the authors’ lab, we observed a person pointing at a magnet pin that was attached on the other side of a steel board. The addresser bent his index finger when he pointed at the invisible, but known, magnet pin. We observers immediately understood the meaning of his pointing gesture, although pointing with a bent index finger seems relatively unusual. We might have common ground with regard to non-verbal gestures, or at least knowledge about how we should interpret others’ various, occasionally unusual, gestures. In the case of referring to invisible objects, sharing information about the referent may be more difficult for both the addresser and the addressee because visual joint attention is difficult to establish. The role of common ground in human non-verbal gesturing seems to be more important when people refer to invisible objects.
We examined people’s common ground regarding pointing gestures. This study focused on comprehension of pointing with a bent index finger at an invisible object behind a panel. Because this is the first study to examine the role of a bent index finger, we did not examine the possibility that the general posture of the arm and the index finger as a whole may have a role in this study. The experimenter pointed at an object in front of a panel or behind a panel with a straight index finger or a bent index finger. There were objects either in front of or behind the panel, and the participants guessed which object was being indicated. We expected that if the experimenter used a bent index finger in pointing, the participant would interpret this as referring to the object behind the panel, but if he used a straight index finger in pointing, the participant would interpret this as referring to the object in front of the panel. The reason is that the bent index finger seems to suggest that the “pointing trajectory” (imaginary trajectory of pointing gesture) can go over the panel.
Twenty Japanese undergraduate university students (M age = 21.2 years; 3 females) participated. The experiment was conducted in accordance with Tokyo Denki University’s code of ethics.
The experimental conditions consisted of two types of pointing (straight vs. bent) and obstacle placement (with vs. without).
With regard to pointing condition, “straight pointing” was when the experimenter pointed at the referent with her arm extended horizontally and her index finger kept straight (Fig. 1a), “Bent pointing” was when the experimenter pointed at the referent with her arm extended slantwise and her index finger kept somewhat bent (Fig. 1b). The “with-obstacle” condition was when there was a small black opaque panel (W: 25 cm x H: 40 cm) on the table. “Without-obstacle” was when there was no panel on the table.
Fig.2 shows the experimental setup. On the table, there were 4 small bottles (W: 2.3 cm × H: 8 cm) designated 1, 2, 3, and 4, respectively, on a label of each bottle. The experimenter sat on one side of the table, wearing black sunglasses during the experiment so that participants could not see the experimenter’s gaze direction. The participant sat at the table at a right angle to the experimenter. In the with-obstacle condition, the panel was placed in the middle of the table between object #2 and object #3. Participants were randomly assigned to all conditions.
Fig. 2: Experimental setup in the with-obstacle condition.
Each object was placed 10 cm apart from the adjacent bottle.
First, the experimenter and the participant looked at all the bottles placed on the table. Each bottle was placed 10 cm apart from the adjacent bottle. Bottle #1 was placed 30 cm away from the edge of the side of the table where the experimenter sat. Then, the participant sat on the experimenter’s chair and looked at the table. Then, the experimenter put the panel in between bottles #2 and #3, and the participant again looked at the table. Thus, the participant experienced the experimenter’s view in both with the obstacle and without the obstacle conditions (Fig. 3). Then, in the straight pointing with the obstacle condition, the experimenter put the panel in between bottles #2 and #3 and said to the participant, “I cannot see bottles #3 and #4.
Now, I will point at one of the four bottles.” Then, the experimenter pointed at bottles using either with the straight index finger or the bent index finger. With each pointing gesture, the experimenter said, “Now I am pointing at something. Which bottle would you guess I am pointing at? Please answer with the number of the bottle.” The participant responded orally using the bottle number. The bottle number corresponded to the distance from the edge of the table: Bottle #1’s distance was 30 cm; #2, 40 cm; #3, 50 cm; and #4, 60 cm, respectively. In addition, the bottle numbers corresponded to visible or invisible status within the with-obstacle condition: bottles #1 and #2 were visible,
Fig. 3: Experimenter’s view during the experiment in the without-obstacle condition (left) and in the with-obstacle condition (right). Before the experiment, each participant first sat on the experimenter’s chair and looked at the table with and without the obstacle to know the experimenter’s views of the both conditions. and bottles #3 and #4 were invisible. Consequently, the participants answered using a scale of 1, 2, 3, and 4. In the without-obstacle condition, all four bottles were visible.
Because the experimenter wore sunglasses, the participant could not see the experimenter’s eye gaze.
In the bent-pointing with-obstacle condition, the procedure was the same as with the straight pointing withobstacle condition except that bent pointing was used. In the without-obstacle conditions, after both the experimenter and the participant looked at the four bottles, the experimenter pointed at a bottle in random order, and the participant guessed which bottle was being pointed at.
There were two pointing conditions (straight and bent), and in each pointing condition, there were two obstacle conditions (with obstacle, without obstacle); in each pointing and obstacle combination, there were four bottle (distance) trials. There were 4 blocks in the order of pointing, straight-pointing and with-obstacle, straightpointing and without-obstacle, bent-pointing and withobstacle, bent-pointing and without-obstacle. In each block, the order of the bottle was random, and there were totally 16 trials. Overall, the order of these blocks was counterbalanced between the participants.
The experimenter was trained to show the same pointing gesture in either the bent or the straight conditions in the aspects of speed of the movement, the height of the wrist, and the distance from the participant’s body.
Fig. 4 shows the participant’s responses when the experimenter pointed at each object in each condition. A 2 (Pointing: straight, bent) × 2 (Obstacle: with, without) × 4 (Referent: #1, #2, #3, #4) ANOVA was performed with the number of the bottle that the participant responded as the dependent measure. There were significant main effects of Pointing, F(1,19) = 78.042, p < .001, ηp2= 0.804; Obstacle, F(1,19) = 6.163, p < .01, ηp2= 0.245; and Distance, F(3,57) = 160.457, p < .001, ηp2= 0.894. There were also significant interactions of Pointing × Obstacle, F(1,19) = 6.935, p < .005, ηp2 = 0.300, and Pointing × Distance, F(3,57) = 6.935, p < .005, ηp2 = 0.148.
To explore the significant Pointing × Obstacle interaction, the simple main effects of Pointing within each Obstacle condition and the simple main effects of Obstacle within each Pointing condition were analyzed. Pointing differences in pointing with the obstacle (F(1,38) = 34.139, p < .001, ηp2 = 0.473.) and without the obstacle (F(1,38) = 80.265, p < .001, ηp2 = 0.679.) were significant. Obstacle differences in the straight pointing were significant, F(1,39) = 13.023, p < .001, ηp2 = .255. To explore the significant Obstacle × Distance interaction, the simple main effects of Pointing within each Obstacle condition and the simple main effects of Obstacle within each Pointing condition were analyzed. Obstacle difference was significant for object #4, F(1,76) = 12.552, p < .001, ηp2 = 0.142. Distance differences in the straight and the bent pointing conditions were significant, except when the experimenter pointed at objects #3 and #4 in the without-obstacle condition (p < .05).
This study examined how people interpret the experimenter’s pointing with a bent index finger at an invisible object behind a panel. The experimenter pointed at bottles that were placed either in front of the panel or behind the panel using a straight index finger or a bent index finger, and the participants guessed which object was being indicated. The results were that in the with-obstacle condition, straight pointing tended to be interpreted as referring to objects #1 and #2, whereas in the withoutobstacle condition, straight pointing tended to be interpreted as referring to all objects, including objects #3 and #4.
However, interestingly, the situation was different when bent pointing was used. In the with-obstacle condition, bent pointing tended to be interpreted as referring to all the 60 cm, respectively. The bottle numbers also correspond to visible or invisible status in the with-obstacle condition: bottles #1 and #2 were visible, and bottles #3 and #4 were invisible in this condition. In the without-obstacle condition, all four bottles were visible. Y-axis shows the number of the bottle the participant responded. objects, including #3 and #4, and there was no difference between the with-obstacle condition and the withoutobstacle condition. Thus, the results indicate that participants interpret the straight pointing as referring to all objects when the panel was not present and the objects in front of the panel when the panel was present. In contrast, they tend to think the bent pointing as referring to all objects irrespective of the presence or absence of the panel.
The current experiment did not disentangle if the effect could be due to the bent index finger or to the general posture of the arm and the index finger as a whole. In future research, the roles of the bent index finger per se and it and the arm as a whole must be examined. In addition, the current experiment did not perfectly control the speed of the pointing gesture. The speed of pointing may have an effect in estimating the distance of the “imaginary trajectory.” Future research must examine this issue.
In conclusion, the study showed that people could interpret pointing at an invisible object when bent index finger was used in pointing. It suggests that people know the meaning of the “bent” index finger based on “common ground” in their interpretation of different types of pointing.
We would like to thank all participants who participated in our experiment. We thank Kota Kuwayama for data collection. This study was partially supported by the Japan Society for the Promotion of Science (JSPS): Grant-in-Aid for Scientific Research (C) #24530793 (H.K.), and Grant-inAid for Young Scientists (B) # 26870549 (T.Y.).