Behavioral Insights on Influence of Manual Action on Object Size Perception Annalisa Bosco, Patrizia Fattori Department of Pharmacy and Biotechnology University of Bologna Bologna, Italy patrizia.fattori@unibo.it Abstract— Visual perception is one of the most advanced function hand movements [1–4]. For example, the orientation of human brain. The study of different aspects of human perception is enhanced during preparation of grasping action perception currently contributes to machine vision applications. compared with a pointing for which object orientation is not Humans estimate the size of objects to grasp them by perceptual important [5,6]. This “enhanced perception” is triggered by the mechanisms. However, the motor system is also able to influence intention to grasp and is important to examine objects with the the perception system. Here, we found modifications of object size perception after a reaching and a grasping action in different maximum possible accuracy. If we consider the effects of contextual information. This mechanism can be described by the action execution on visual perception of object features, there Bayesian model where action provides the likelihood and this is ample evidence for visual perception changes in the latter is integrated with the expected size (prior) derived from the oculomotor system, but little is known about the perceptual stored object experience (Forward Dynamic Model). Beyond the changes induced by different types of hand movements. In action-modulation effect, the knowledge of subsequent action order to evaluate the influence of different hand movement on type modulates the perceptual responses shaping them according visual perception, we tested a feature-specific modulation on to relevant information required to recognize and interact with object size perception after a reaching and a grasping action in objects. Cognitive architectures can be improved on the basis of different contexts. these processings in order to amplify relevant features of objects and allow to robot/agent an easy interaction with them. II. MATERIALS AND METHODS A total of 16 right-handed subjects (11 females and 5 males, Keywords—visual perception, object recognition, motor output, ages 21–40 years; with normal or corrected-to-normal vision) human functions, context information. took part in the experiment. The experiment was performed by two groups of participants. One group of 8 subjects performed I. INTRODUCTION the Prior knowledge of action type experiment (PK condition) The majority of machine vision and object recognition and the other group (8 participants) performed the No prior systems today apply mechanistic or deterministic template knowledge of action type (NPK condition). All subjects were matching, edge detection or color scanning approach for naive to the experimental purpose of the study and gave identifying different objects in the space and also to guide informed consent to participate in the experiment. Procedures embodied artificial intelligent systems to interaction with were approved by the Bioethical Committee of the University them. However, fine disturbances in the workspace of a robot of Bologna and were in accordance with the Declaration of can lead to failures, and thus slow down their performance in Helsinki. identification, recognition, learning and adapting to noisy A. Apparatus and Setup environment, compared to human brain. To go beyond these Participants were seated in an environment with dim limitations robots with intelligent behavior must be provided background lighting and viewed a touchscreen monitor (ELO with a processing architecture that allows them to learn and IntelliTouch, 1939L), which displayed target stimuli within a reason about responses to complex goals in a complex world. visible display of 37.5 X 30.0 cm. To stabilize head position, The starting point for the development of such intelligent the participants placed their heads on a chin rest located 43 cm systems is the study of human behavior. Humans frequently from the screen, which resulted in a visual field of 50 x 40 estimate the size of objects to grasp them. In fact, when deg. The display had a resolution of 1152 X 864 pixels and a performing an action, our perception is focused towards object frame rate of 60 Hz (15,500 touch points/cm2). For stimulus visual properties that enable us to execute the action presentation, we used MATLAB (The MathWorks) with the successfully. However, the motor system is also able to Psychophysics toolbox extension [7]. The stimuli were white, influence perception, but only few studies reported evidence red and green dots with a radius of 1.5 mm and 10 differently for action-induced visual perception modifications related to Proceedings of EUCognition 2016 - "Cognitive Robot Architectures" - CEUR-WS 21 sized white, red and green bars all 9 mm large and whose manual estimation phases. Grip aperture was calculated length was: 30, 33.6, 37.2, 40.8, 44.4, 48, 51.6, 55.2, 58.8, considering trial intervals in which the velocities of the index 62.4 mm. Hand position was measured by a motion capture and thumb markers remained <5 mm/s [8]. Grip aperture was system (VICON, 460; frequency of acquisition 100 Hz), defined as maximum distance within this interval. To evaluate which follows the trajectory of the hand in three dimensions the effect of different hand movement on size perception, we by recording infrared light reflection on passive markers. compared the manual perceptual responses before the Participants performed 10 blocks of 10 trials each. Each trial movements with those after the movements by using two- consisted of three successive phases: Pre-size perception, tailed t-test with independent samples. Reaching or Grasping movement, Post-size perception (Fig. To evaluate the magnitude of the effect of NPK and PK 1). In Pre-size perception and Post-size perception phases conditions on perceptual responses before the movement we (phases 1 and 3), a white or green central fixation target stayed calculated the average difference between the two responses on the screen for 1 s; then, a white or green bar was presented, and we compared the responses between the two conditions by for 1 s, 12 deg on the left or on the right side of the central a t-test analysis. We extracted relevant features from the fixation target and, after an acoustic signal, it disappeared. The perceptual responses before the movement and we used them participants were required to manually indicate the perceived to predict the NPK and PK conditions. For this purpose, we horizontal size of the bar. All participants indicated the bar performed a linear-discriminant analysis (LDA-based sizes by keeping the hand within the starting hand position classifier), as implemented in Statistics and Machine Learning square and the distance between subject eyes. In the Reaching toolbox (Matlab). Pre movement manual responses of NPK or Grasping movement phase (phase 2), after 1 s, the white or and PK conditions were vertically concatenated to build the green central fixation point was followed by a bar identical for feature space composed by 958 trials. Fivefold cross- position and size to that of phases 1 and 3. Participants were validation was performed by using the 80% of trials for required to perform a reaching (closed fist) or grasping action training and the 20% for testing the data, so to ensure that the (extension of thumb and index fingers to “grasp” the classifier was trained and tested on different data. Specifically, extremities of the bar) towards the bar after the acoustic the classifier was trained on the training subset and the signal, respectively. The type of actions was instructed by the obtained optimal decision criteria was implemented on the colors of the stimuli (fixation point and bar). In fact, if the testing subset. The prediction results were obtained for this color of the stimuli was white, participants were required to testing subset. This procedure was repeated 5 times, so that all perform a reaching movement whereas, if the color was green, trials were tested and classified basing on models learned from they were required to perform a grasping movement. In PK the other trials. The prediction results for all the trials were condition, the color of fixation points and bars was white or taken together to give an averaged prediction result with green in all three phases of trial and in this way the standard deviation. We considered statistically significant the participants knew in advance (from phase 1) which action type accuracies which standard deviations did not cross the was required in the movement phase (phase 2). In the NPK theoretical chance level of 50%. We used a LDA classifier as condition, the sequence of the three phases was identically decoder of the two conditions. LDA finds linear combination structured as in the PK condition, but we changed colors of of features that characterizes or separates two or more classes fixation points and bars from white/green to red in phases 1 of objects or event [9,10]. In fact, LDA explicitly attempts to and 3. The color of stimuli during phase 2 remained white or model the difference between the classes of data. For all green according to the movement type, reaching or grasping statistical analyses the significant criterion was set to P < 0.05. respectively. By this color manipulation, participants could not know in advance the successive action type. III. RESULTS We assessed the effects of action execution on perceptual responses comparing the single subject responses before the movement with those after the movement and calculating the difference between these. Fig. 2 shows these differences in grey color for reaching movement on the horizontal axis compared with those of grasping movement on vertical axis. Filled and empty circles are referred to PK and NPK condition, respectively. The majority of subjects fell below the diagonal suggesting that they corrected the perceptual estimation after the grasping movements with respect to the reaching movement. In particular, they perceived significantly smaller the bars after a grasping movement with respect to a reaching movement (P < 0.05). The averaged differences in Fig. 1. Task sequence. Circle = fixation point, Rectangle = stimulus, Hand = PK and NPK conditions are reported in Fig. 2 as black and size indication by manual report, Speaker = acoustic signal to respond. white dots, respectively. Both dots are below the diagonal suggesting that, globally, subjects perceived smaller after a grasping action compared with a reaching action. B. Data analysis To analyze the effect of the NPK and PK conditions on size perception, we focused the analyses on manual size reports After data collection, finger position data were interpolated at before the movement execution (Pre size perception phase). 1000 Hz, then data were run though a fifth-order Butterworth We computed the difference between the Pre size perception low-pass filter [8]. For data processing and analysis, we wrote reports in PK condition and the Pre size perception reports in custom software in MATLAB to compute the distance NPK condition. This difference allowed to highlight the between index and thumb markers during the pre- and post- Proceedings of EUCognition 2016 - "Cognitive Robot Architectures" - CEUR-WS 22 amount of change in size perception in the two conditions Bekkering and Neggers [2] analysed the performance of tested. As it is shown in Fig. 3A, we found that the amount of subjects that were required to grasp or point to an object of a change in reaching was -11.89 mm ±0.98 mm and in grasping certain orientation and color among other objects. They -11.36 mm ±1.08 mm, and in both cases, they were demonstrated that fewer saccadic eye movements were made significantly deviated from baseline (t-test, P < 0.05). to wrong orientations when subjects had to grasp the object Generally, the subjects tended to perceive smaller the sizes than point to it. Recently, Bayesian theory has been applied to presented in the condition where they were aware about the formalize processes of cue and sensorimotor integration subsequent action (PK condition) compared with the condition [13,14]. According to this view, the nervous system combines where they were uncertain about the successive movement prior knowledge about object properties gained through (NPK condition). To evaluate whether the strength of this former experience (prior) with current sensory cues effect was due to a perceptual bias or to different neural (likelihood), to generate appropriate object properties processings, we used a LDA decoder to classify the manual estimations for action and perception. Hirsinger and responses according to the NPK and PK condition (see coworkers [15], by application of a size-weight illusion Material and Methods). In other words, we checked whether paradigm, found that the combination of prior and likelihood we were able to predict the PK and NPK conditions from for size perception were integrated in a Bayesian way. Their perceptual responses before the movement execution, as this model consisted in a Forward Dynamic Model (FDM) that technique represents a powerful method to reconstruct represented the stored object experience. The FDM output was experimental conditions and functional movements from the experience-based expected size and was referred as the neural responses using different types of classifiers [11,12]. prior. The prior then was integrated with the likelihood, which Fig. 3A shows decoding results as confusion matrix and the represented the afferent sensory information about object size. corresponding mean accuracy expressed in percentage. We A feedback loop with a specified gain provides the FDM with found a good correlation between the real conditions and the the final estimate of size, which serves as learning signal for decoded conditions, as it is illustrated in Fig. 3B. The adapting object experience. In the present study, we can apply accuracies of decoding were significantly higher of 50% a similar model for size perception after an action execution. (66,8% for PK and 60.54% for NPK) as shown in Fig. 3C. In our case, the objects were visual, not real objects and no haptic feedback was given after the execution of movement. So, the likelihood was represented by the matching of the fingers with the outer border of objects with/or the proprioceptive signals coming from the hand posture that are integrated with the prior. We found that the knowledge of action type was a factor modulating size perception. In fact, subjects perceived smaller the bars during the condition where they knew the subsequent action (PK) compared with the other condition where they did not know the subsequent action (NPK) for both reaching and grasping. A further demonstration of that was related to the possibility to predict with significant accuracy (>50%) the two conditions from perceptual responses before movement (see Fig. 3B-C). This approach is typical for neural responses and represents a novelty for this type of behavioral variables. The Fig. 2. Differences between perceptual responses before and after the significance of these results is in line with evidence from movement. Filled grey dots are differences in PK condition and empty grey behavioral research suggesting that motor planning processes dots are differences in NPK condition. Black and white dots are the mean increase the weight of visual inputs. differences in PK and NPK conditions, respectively. IV. DISCUSSION In the present study, we found direct evidence for a perceptual modification of a relevant feature as object size before and after the execution of two types of hand movement. These changes depended on two factors: the knowledge of the subsequent action type and the type of action executed. Changes in perception were sharpened after a grasping action compared with a reaching. Specifically, subjects perceived objects smaller after a grasping movement than after a reaching movement. The study of action effects exerted by the skeletomotor system on perception has been focused on the evidence that relevant features of objects, such as size or orientation, prime the perceptual system in order to execute a Fig.3. A, Mean differences of perceptual responses between PK and NPK more accurate subsequent grasping movement. conditions in reaching and grasping. B, Confusion matrix of decoding results. Indeed, Gutteling et al. [5] demonstrated an increased C, Mean decoding accuracy for classification of NPK and PK conditions. perceptual sensitivity to object orientation during a grasping Error bars are standard deviation. *P<0.05, significant level. preparation phase. The effect of action-modulated perception has also been shown to facilitate visual search for orientation. Proceedings of EUCognition 2016 - "Cognitive Robot Architectures" - CEUR-WS 23 Hand visual feedback has been found to have a greater impact REFERENCES on movement accuracy when subjects prepare their [1] Craighero, L., Fadiga, L., Rizzolatti, G., Umiltà C. Action for movements with the prior knowledge that vision will be Perception A Motor - Visual Attentional Effect. J Exp Psychol Hum available during their reaches [16,17]. More interestingly, Percept Perform. 1999;25: 1673–. [2] Bekkering H, Neggers SFW. Visual search is modulated by action motor preparation facilitates the processings of visual intentions. Psychol Sci a J Am Psychol Soc / APS. 2002;13: 370– information related to the target of movement. Similarly to 374. doi:10.1111/j.0956-7976.2002.00466.x Gutteling et al. [5] for object orientation, Wykowska et al. [18] [3] Hannus A, Cornelissen FW, Lindemann O, Bekkering H. Selection- reported that the detection of target size was facilitated during for-action in visual search. Acta Psychol (Amst). 2005;118: 171– 191. doi:10.1016/j.actpsy.2004.10.010 the planning of grasping but not during the planning of [4] Fagioli S, Hommel B, Schubotz RI. Intentional control of attention: pointing. All these studies show the capacity of the brain to Action planning primes action-related stimulus dimensions. Psychol modulate the weight of visual inputs and provide an Res. 2007;71: 22–29. doi:10.1007/s00426-005-0033-3 illustration of the importance of the context in visual [5] Gutteling TP, Kenemans JL, Neggers SFW. Grasping preparation enhances orientation change detection. PLoS One. 2011;6. information processing. In line with all these studies, our doi:10.1371/journal.pone.0017675 findings suggest that the knowledge or not of subsequent [6] Gutteling TP, Park SY, Kenemans JL, Neggers SFW. TMS of the movement type defines a context that modulates the anterior intraparietal area selectively modulates orientation change perceptual system. When subjects knew the subsequent detection during action preparation. J Neurophysiol. 2013;110: 33– 41. doi:10.1152/jn.00622.2012 movement, the perceptual system was within a definite context [7] Brainard DH. The Psychophysics Toolbox. Spat Vis. 1997;10: 433– and perceived object smaller, scaling the measures according 436. doi:10.1163/156856897X00357 to hand motor abilities. In the other case, subjects were in an [8] Bosco A, Lappe M, Fattori P. Adaptation of Saccades and Perceived uncertain context about the successive action, and the Size after Trans-Saccadic Changes of Object Size. J Neurosci. 2015;35: 14448–14456. doi:10.1523/JNEUROSCI.0129-15.2015 perceptual system used different rules to scale the size reports. [9] Fisher RA. The use of multiple measurements in taxonomic In both cases, the defined and undefined context can be problems. Ann Eugen. 1936;7: 179–188. predicted. All the mechanisms described in the present study [10] McLachlan GJ. Discriminant analysis and statistical pattern could implement models of cognitive architecture of vision- recognition. Wiley Intersci. 2004;. ISBN 0-4. [11] Schaffelhofer S, Agudelo-Toro A, Scherberger H. Decoding a wide based reaching and grasping of objects located in the range of hand configurations from macaque motor, premotor, and peripersonal space of a robot/agent. Additionally, the evidence parietal cortices. J Neurosci. 2015;35: 1068–81. that the perceptual system is dynamically modulated by doi:10.1523/JNEUROSCI.3594-14.2015 contextual information about subsequent movement type can [12] Townsend BR, Subasi E, Scherberger H. Grasp movement decoding from premotor and parietal cortex. J Neurosci. 2011;31: 14386–98. be used to improve cognitive architectures. For example one doi:10.1523/JNEUROSCI.2451-11.2011 or multiple focus of attention signals can be sent to the object [13] Körding KP, Wolpert DM. Bayesian decision theory in representation of robot/agent in order to amplify relevant sensorimotor control. Trends in Cognitive Sciences. 2006. pp. 319– features and at the same time inhibits distractors. 326. doi:10.1016/j.tics.2006.05.003 [14] Van Beers RJ, Wolpert DM, Haggard P. When feeling is more ACKNOWLEDGMENT important than seeing in sensorimotor adaptation. Curr Biol. 2002;12: 834–837. doi:10.1016/S0960-9822(02)00836-9 We thank F. Daniele for helping in the data collection and [15] Hirsiger S, Pickett K, Konczak J. The integration of size and weight in data analysis. This work was supported by Firb 2013 N. cues for perception and action: Evidence for a weight-size illusion. RBFR132BKP (MIUR) and by the Fondazione del Monte di Exp Brain Res. 2012;223: 137–147. doi:10.1007/s00221-012-3247- 9 Bologna e Ravenna. [16] Zelaznik HZ, Hawkins B, Kisselburgh L. Rapid visual feedback processing in single-aiming movements. J Mot Behav. 1983;15: 217–236. doi:10.1080/00222895.1983.10735298 [17] Elliott D, Allard F. The utilization of visual feedback information during rapid pointing movements. Q J Exp Psychol A Hum Exp Psychol. 1985;37: 407–425. doi:10.1080/14640748508400942 [18] Wykowska A, Schubö A, Hommel B. How you move is what you see: action planning biases selection in visual search. J Exp Psychol Hum Percept Perform. 2009;35: 1755–1769. doi:10.1037/a0016798 Proceedings of EUCognition 2016 - "Cognitive Robot Architectures" - CEUR-WS 24