Pointing to an invisible object behind a wall: Comprehension of pointing with a bent index finger Harumi KOBAYASHI (h-koba@mail.dendai.ac.jp) Division of Information System Design, Tokyo Denki University, Ishizaka Hatoyama, Saitama, 350-0394 Japan Tetsuya YASUDA (t-yasuda@jumonji-u.ac.jp) Department of Human Developmental Psychology, Jumonji University Sugasawa Niiza, Saitama, 352-8510 Japan Abstract understanding of pointing at referents and placing referents Pointing is a gesture that people use to specify and convey in different situations. Clark, Schreuder, & Buttrick (1983) information about objects in the environment. Previous and Clark (1996) proposed that people use “common ground” research has mostly explored peoples’ comprehension and as implicit mutual knowledge in human communication. production of others’ “straight” pointing gestures, that is, Common ground can include a variety of information: how pointing at an object with arm and index finger kept straight people convey information using words and sentences, while aiming at a visible object. However, we seem to use knowledge about the history and culture of the speakers and various types of pointing in addition to typical straight addressees, mutually shared knowledge about specific pointing to denote both visible and invisible objects. This study examined comprehension of pointing with a “bent” people and events, and knowledge of what is going on in the index finger at an invisible object behind a wall. The current communication. Thus, common ground includes experimenter pointed either at an object in front of a wall or both general knowledge about the world and specific one behind a wall with a straight index finger or a bent index knowledge of the specific task that the conversation partners finger, and the participants guessed which object was being are conducting. Previous research has focused mostly on denoted. The results were that when the participants looked at language and verbally describable information included in straight pointing, they thought objects in front of the wall common ground. Non-verbal information such as gestures were being denoted. However, when they looked at bent pointing, they thought objects behind the wall could be must also be comprehended using common ground as to denoted. The study suggested that people have “common how people use gestures in different situations; however, ground” in terms of interpretation of different types of usage of gestures as common ground has not yet been pointing. thoroughly explored. Keywords: gesture; declarative pointing; common ground; Some research has explored peoples’ comprehension and non-linguistic information production of pointing gestures when they use some language such as demonstratives such as “this” or “that” Introduction while indicating objects in the environment (Bangerter, 2004; Coventry, Valdés, Castillo, & Guijarro-Fuentes, When we communicate with others, we often draw their 2008; Coventry, Griffiths, & Hamilton, 2014). Most of attention to objects about which we wish to communicate pointing studies have examined typical pointing gestures, (Tomasello, 2008; 2014). Pointing is a gesture that people we named it “straight” pointing because pointing at an use to specify and convey information about referents. For object is done with the addresser’s arm and index finger example, when one asks a friend about a landmark in the kept relatively straight while aiming at a visible object in the city, she will point at the landmark using her index finger or environment (Coventry et al., 2008; Doherty, Anderson, & her hand. Howieson, 2004; Jaswal & Hansen, 2006; Kobayashi, 2007). Clark (2003) discussed use of attention-getting gestures In this situation, the addresser can easily share information in various cases. He noted that pointing at a referent and about visible objects, using visual joint attention and placing a referent are both useful ways to convey common ground. information about referents, but that people index objects How, then, can we point at invisible objects such as differently. In pointing, a person directs the addressee’s objects behind obstacles? In the authors’ lab, we observed a attention to the referent object; for example, a customer may person pointing at a magnet pin that was attached on the point at a package of a medicine that is difficult for her to other side of a steel board. The addresser bent his index reach but is easy for the clerk. In placing, a person puts a finger when he pointed at the invisible, but known, magnet referent object in the area of an addressee’s attention; for pin. We observers immediately understood the meaning of example, a customer may place a package of medicine on his pointing gesture, although pointing with a bent index the checkout counter where a clerk waits. These finger seems relatively unusual. We might have common communications are possible without saying any words. In ground with regard to non-verbal gestures, or at least order to communicate smoothly, people must share mutual 477 knowledge about how we should interpret others’ various, of the table between object #2 and object #3. Participants occasionally unusual, gestures. In the case of referring to were randomly assigned to all conditions. invisible objects, sharing information about the referent may be more difficult for both the addresser and the addressee because visual joint attention is difficult to establish. The role of common ground in human non-verbal gesturing seems to be more important when people refer to invisible objects. We examined people’s common ground regarding pointing gestures. This study focused on comprehension of pointing with a bent index finger at an invisible object behind a panel. Because this is the first study to examine the Fig. 1: Examples of the two types of pointing gestures: “a” role of a bent index finger, we did not examine the possibility that the general posture of the arm and the index denotes the shape of the “straight” pointing; “b” denotes the finger as a whole may have a role in this study. The shape of the “bent” pointing. experimenter pointed at an object in front of a panel or behind a panel with a straight index finger or a bent index finger. There were objects either in front of or behind the panel, and the participants guessed which object was being indicated. We expected that if the experimenter used a bent index finger in pointing, the participant would interpret this as referring to the object behind the panel, but if he used a straight index finger in pointing, the participant would interpret this as referring to the object in front of the panel. The reason is that the bent index finger seems to suggest that the “pointing trajectory” (imaginary trajectory of pointing gesture) can go over the panel. Method Fig. 2: Experimental setup in the with-obstacle condition. Participants Each object was placed 10 cm apart from the adjacent bottle. Twenty Japanese undergraduate university students (M age = 21.2 years; 3 females) participated. The experiment First, the experimenter and the participant looked at all was conducted in accordance with Tokyo Denki the bottles placed on the table. Each bottle was placed 10 University’s code of ethics. cm apart from the adjacent bottle. Bottle #1 was placed 30 cm away from the edge of the side of the table where the Procedure experimenter sat. Then, the participant sat on the The experimental conditions consisted of two types of experimenter’s chair and looked at the table. Then, the pointing (straight vs. bent) and obstacle placement (with vs. experimenter put the panel in between bottles #2 and #3, without). and the participant again looked at the table. Thus, the With regard to pointing condition, “straight pointing” was participant experienced the experimenter’s view in both when the experimenter pointed at the referent with her arm with the obstacle and without the obstacle conditions (Fig. extended horizontally and her index finger kept straight (Fig. 3). Then, in the straight pointing with the obstacle condition, 1a), “Bent pointing” was when the experimenter pointed at the experimenter put the panel in between bottles #2 and #3 the referent with her arm extended slantwise and her index and said to the participant, “I cannot see bottles #3 and #4. finger kept somewhat bent (Fig. 1b). The “with-obstacle” Now, I will point at one of the four bottles.” Then, the condition was when there was a small black opaque panel experimenter pointed at bottles using either with the straight (W: 25 cm x H: 40 cm) on the table. “Without-obstacle” index finger or the bent index finger. With each pointing was when there was no panel on the table. gesture, the experimenter said, “Now I am pointing at Fig.2 shows the experimental setup. On the table, there something. Which bottle would you guess I am pointing at? were 4 small bottles (W: 2.3 cm × H: 8 cm) designated 1, 2, Please answer with the number of the bottle.” The 3, and 4, respectively, on a label of each bottle. The participant responded orally using the bottle number. The experimenter sat on one side of the table, wearing black bottle number corresponded to the distance from the edge of sunglasses during the experiment so that participants could the table: Bottle #1’s distance was 30 cm; #2, 40 cm; #3, 50 not see the experimenter’s gaze direction. The participant cm; and #4, 60 cm, respectively. In addition, the bottle sat at the table at a right angle to the experimenter. In the numbers corresponded to visible or invisible status within with-obstacle condition, the panel was placed in the middle the with-obstacle condition: bottles #1 and #2 were visible, 478 Fig. 3: Experimenter’s view during the experiment in the without-obstacle condition (left) and in the with-obstacle condition (right). Before the experiment, each participant first sat on the experimenter’s chair and looked at the table with and without the obstacle to know the experimenter’s views of the both conditions. and bottles #3 and #4 were invisible. Consequently, the = 160.457, p < .001, ηp2= 0.894. There were also significant participants answered using a scale of 1, 2, 3, and 4. In the interactions of Pointing × Obstacle, F(1,19) = 6.935, p without-obstacle condition, all four bottles were visible. < .005, ηp2 = 0.300, and Pointing × Distance, F(3,57) = Because the experimenter wore sunglasses, the participant 6.935, p < .005, ηp2 = 0.148. could not see the experimenter’s eye gaze. To explore the significant Pointing × Obstacle interaction, In the bent-pointing with-obstacle condition, the the simple main effects of Pointing within each Obstacle procedure was the same as with the straight pointing with- condition and the simple main effects of Obstacle within obstacle condition except that bent pointing was used. In the each Pointing condition were analyzed. Pointing differences without-obstacle conditions, after both the experimenter and in pointing with the obstacle (F(1,38) = 34.139, p < .001, ηp2 the participant looked at the four bottles, the experimenter = 0.473.) and without the obstacle (F(1,38) = 80.265, p pointed at a bottle in random order, and the participant < .001, ηp2 = 0.679.) were significant. Obstacle differences guessed which bottle was being pointed at. in the straight pointing were significant, F(1,39) = 13.023, p There were two pointing conditions (straight and bent), < .001, ηp2 = .255. To explore the significant Obstacle × and in each pointing condition, there were two obstacle Distance interaction, the simple main effects of Pointing conditions (with obstacle, without obstacle); in each within each Obstacle condition and the simple main effects pointing and obstacle combination, there were four bottle of Obstacle within each Pointing condition were analyzed. (distance) trials. There were 4 blocks in the order of Obstacle difference was significant for object #4, F(1,76) = pointing, straight-pointing and with-obstacle, straight- 12.552, p < .001, ηp2 = 0.142. Distance differences in the pointing and without-obstacle, bent-pointing and with- straight and the bent pointing conditions were significant, obstacle, bent-pointing and without-obstacle. In each block, except when the experimenter pointed at objects #3 and #4 the order of the bottle was random, and there were totally 16 in the without-obstacle condition (p < .05). trials. Overall, the order of these blocks was counterbalanced between the participants. Discussion The experimenter was trained to show the same pointing This study examined how people interpret the gesture in either the bent or the straight conditions in the experimenter’s pointing with a bent index finger at an aspects of speed of the movement, the height of the wrist, invisible object behind a panel. The experimenter pointed at and the distance from the participant’s body. bottles that were placed either in front of the panel or behind the panel using a straight index finger or a bent index finger, Results and the participants guessed which object was being Fig. 4 shows the participant’s responses when the indicated. The results were that in the with-obstacle experimenter pointed at each object in each condition. A 2 condition, straight pointing tended to be interpreted as (Pointing: straight, bent) × 2 (Obstacle: with, without) × 4 referring to objects #1 and #2, whereas in the without- (Referent: #1, #2, #3, #4) ANOVA was performed with the obstacle condition, straight pointing tended to be interpreted number of the bottle that the participant responded as the as referring to all objects, including objects #3 and #4. dependent measure. There were significant main effects of However, interestingly, the situation was different when Pointing, F(1,19) = 78.042, p < .001, ηp2= 0.804; Obstacle, bent pointing was used. In the with-obstacle condition, bent F(1,19) = 6.163, p < .01, ηp2= 0.245; and Distance, F(3,57) pointing tended to be interpreted as referring to all the 479 Fig. 4: Participant’s responses when the experimenter pointed at each object in each condition. Here, “a” denotes the straight pointing condition, and “b” denotes the bent pointing condition. X-axis shows the bottle number that corresponds to the distance from the edge of the table: Bottle 1’s distance was 30 cm; Bottle 2, 40 cm; Bottle 3, 50 cm; and Bottle 4, 60 cm, respectively. The bottle numbers also correspond to visible or invisible status in the with-obstacle condition: bottles #1 and #2 were visible, and bottles #3 and #4 were invisible in this condition. In the without-obstacle condition, all four bottles were visible. Y-axis shows the number of the bottle the participant responded. objects, including #3 and #4, and there was no difference between the with-obstacle condition and the without- Acknowledgments obstacle condition. Thus, the results indicate that participants interpret the straight pointing as referring to all We would like to thank all participants who participated objects when the panel was not present and the objects in in our experiment. We thank Kota Kuwayama for data front of the panel when the panel was present. In contrast, collection. This study was partially supported by the Japan they tend to think the bent pointing as referring to all Society for the Promotion of Science (JSPS): Grant-in-Aid objects irrespective of the presence or absence of the panel. for Scientific Research (C) #24530793 (H.K.), and Grant-in- The current experiment did not disentangle if the effect Aid for Young Scientists (B) # 26870549 (T.Y.). could be due to the bent index finger or to the general posture of the arm and the index finger as a whole. In future References research, the roles of the bent index finger per se and it and Bangerter, A. (2004). Using pointing and describing to the arm as a whole must be examined. In addition, the achieve joint focus of attention in dialogue. current experiment did not perfectly control the speed of the Psychological Science, 15, 415-419. pointing gesture. The speed of pointing may have an effect Clark, H. H. (1996). Using language. Cambridge: in estimating the distance of the “imaginary trajectory.” Cambridge University Press. Clark, H. H. (2003). Pointing and placing. In S. Kita (Ed.). Future research must examine this issue. Pointing: Where language, culture, and cognition In conclusion, the study showed that people could meet (pp. 243-268). Mahwah, NJ: Lawrence Erlbaum. interpret pointing at an invisible object when bent index Clark, H. H., Schreuder, R., & Buttrick, S. (1983). Common finger was used in pointing. It suggests that people know ground at the understanding of demonstrative the meaning of the “bent” index finger based on “common reference. Journal of Verbal Learning and Verbal Behavior, 22, 245-258. ground” in their interpretation of different types of pointing. 480 Coventry, K. R., Griffiths, D., & Hamilton, C. (2014). Jaswal, V. K. & Hansen, M. B. (2006). Learning words: Spatial demonstratives and perceptual space: children disregard some pragmatic information that Describing and remembering object location. conflicts with mutual exclusivity. Developmental Cognitive Psychology, 69, 46-70 Science , 9, 158–165. Coventry, K. R., Valdés, B., Castillo, A., & Guijarro- Kobayashi, H. (2007). The effect of touching object parts on Fuentes, P. (2008). Language within your reach. Near- learning novel object part names among young far perceptual space and spatial demonstratives. children and adults. Studies in Language Sciences 6, Cognition, 108, 889-895. 61-76. Doherty, M. J., Anderson, J. R., & Howieson, L. (2009). Tomasello, M. (2008). Origins of human communication. The rapid development of explicit gaze judgment Cambridge: MIT press. ability at 3 years. Journal of Experimental Child Tomasello, M. (2014). A Natural History of Human Psychology, 104, 296-312. Thinking. Harvard University Press. 481