Eye-hand coordination during visual search on geographic displays

Abstract

Eye movement analysis can provide insights into the cognitive processes in human mind and have been successfully utilized to study visual tasks such as reading and exploration of digital displays. However recording eye movements requires costly equipment and the face-to-face individual data collection forces us to work with only a limited number of participants. Interactive displays are alternatively evaluated through mouse movements which can be collected considerably easier than eye movements. It is however an open question if and how these two types of movement are linked. In this project, we study the link between eye and mouse movements to understand the eye-hand coordination speci cally with visual search tasks in geographic displays.

Keywords

• eye tracking
• mouse tracking
• visual search
• trajectory analysis

-

Our eyes and hands move in coordination to execute many everyday tasks, e.g., when we play, fold laundry or pour co ee into a cup. This coordination has been widely studied in psychology and cognitive science [ 10, 4 ]. Eye-hand coordination is expressed also in the pointing behavior; arguably a very fundamental geographic task. We point at the target destination during way nding or when we describe directions (even if the target is out of sight). We might also point at a target on a map, or trace the path that we might take. Additionally, when using interactive maps, eye-hand coordination also plays a strong role as we zoom, pan, tilt, rotate etc. However, the use of hands seems to di er between people as well as tasks, and these di erences are not yet well documented. We believe that studying eye-hand coordination patterns can be helpful in understanding the map reading process. Therefore, in this study. we take mouse movements as a proxy for hand movements (e.g. similarly to [ 2 ]), record both mouse and eye movements in a user experiment with various visual search tasks and analyze some of the similarities and di erences to explore the gaze patterns in relation to mouse use.

While eye movement recordings can be useful in evaluating visual displays [ 3 ], obtaining them is (still) expensive and eye tracking process has various practical drawbacks to administer, especially for large populations. Such concerns lead to considerations to use mouse movements instead. In fact, the term mouse eye tracking has been used, assuming that if we move the mouse or click somewhere, we look at the cursor position [ 11 ]. Mouse movements are easier and cheaper to record than eye movements and can be done automatically and on a large scale. Various studies investigated the patterns of eye and mouse movements in non-geographic contexts [ 8, 12, 1, 9 ]. However, it is still an open question if and how these two types of movements are linked, especially for map reading tasks such as localization (visual search), identi cation, comparison or pattern recognition. In this paper we study the link between eye and mouse movements for a fundamental map reading task: visual search.

It is important to note that from physics perspective two movement types are distinctly di erent: While the eyes typically move in discrete jumps (saccades) between xations producing irregular movement and jagged trajectories; the hand (and therefore the mouse) moves in a continuous motion producing smooth trajectories. However, both trajectory types are generated by the same process (visual search on the screen) and are co-located in space and time. Therefore, we believe using methods from trajectory analysis and visualisation [ 5, 6 ] to investigate eye and mouse movements is appropriate.

More speci cally, we investigate two questions related to visual search in geographic displays: 1) Do people use the mouse during visual search even when it is not necessary? We are interested in identifying if and how frequently people use their mouse based on user characteristics, map type and task di culty. 2) When the mouse is used, how does the temporal mouse movement compare to the temporal gaze movement? Does the eye follow the mouse, vice versa, or are they independent from each other? 2

Data collection

Eye and mouse movement data were simultaneously collected in a controlled lab experiment [ 7 ]. 37 voluntary participants (11 male, 26 female) executed various visual search tasks on two map types (satellite and cartographic, g. 1) using a total of 52 stimuli. In this paper we present a preliminary analysis of all 37 participants while they execute one task on one stimulus ( g. 1a). The task was to match the clipped area underneath the map to its location on the map, and click on it. 3

Distance to target

A common metric used for analysing gaze data for visual search tasks is to calculate the distance from gaze position to the target position. We used this principle on both gaze and mouse data and calculated the time series of distances between gaze and mouse positions to the target (speci ed as the centroid of the square target area on the map). We expected two possible outcomes: 1) For participants who use the mouse for visual search, the time series for gazetarget and mouse-target distances would mirror each other. 2) For participants who do not use the mouse, the gaze-target distance would vary over time, while the mouse-target distance would be constant up to the moment when the user identi ed the location of the target and grabbed the mouse to click on it (whereupon the time series of mouse to target distance should suddenly drop to zero). Our preliminary analysis shows that both cases indeed occur ( g. 2).

Visualising spatio-temporal similarity of trajectories

To further investigate the similarity of eye and mouse movements, we visualised the relevant trajectories in a space-time cube (STC). Figure 3 shows the cubes (toppled, so that the bottom is on the left and time starts from zero on the left) for the two cases from gure 2, i.e. 3a corresponds to 2a and 3b to 2b. The temporally-di erent tracks of the mouse being stationary ( g 3a) vs. the mouse and the eye following one another ( g 3b) can clearly be seen. We plan to use 3D trajectory analysis in the STC space to further quantify similarity of movement. 5

Conclusions and outlook

This paper presents preliminary results from a spatio-temporal analysis of eye and mouse trajectories in visual search. Data analysis is currently in progress investigating similarities in scan paths (gaze trajectories) and mouse trajectories. In the process, we develop methods to quantify these similarities based on trajectory analysis. However, in this paper, we speci cally focus on the possible impact of the studied map types in the unnecessary mouse use and whether there are group di erences between participants.

Acknowledgements : This work is supported by the Royal Society grant IE120643. We would like to thank Floris Heim from University of Zurich for data collection. JV and UD are supported by the EU FP7 Marie Curie ITN GEOCROWD (FP7-PEOPLE-2010-ITN-264994).

References

1. Arsenault R and Ware C , 2000 . Eye-hand co-ordination with force feedback . Proc of CHI 2000 , 408 - 414 .

2. Bieg H-J , Chuang LL , Fleming RW , Reiterer H and Bultho HH , 2010 . Eye and pointer coordination in search and selection tasks . Proc of the 2010 Symposium on Eye-Tracking Research & Applications - ETRA10 , 89 .

3. Coltekin A , Fabrikant SI and Lacayo M, 2010 . Exploring the e ciency of users visual analytics strategies based on sequence analysis of eye movement recordings . Int J GISience 24 ( 10 ): 1559 - 1575 .

4. Crawford JD , Medendorp WP and Marotta JJ , 2004 . Spatial transformations for eye-hand coordination . Journal of Neurophysiology 92 ( 1 ): 10 - 19 .

5. Demsar U and Virrantaus K , 2010 . Space-time density of trajectories: exploring spatio-temporal patterns in movement data . Int J GISience 24 ( 10 ): 1527 - 1542 .

6. Dodge S , Weibel R and Laube P , 2011 . Trajectory Similarity Analysis in Movement Parameter Space . Proc of GISRUK 2011 .

7. Heim F , 2014 . A Visual Search E ciency Study : An Evaluation of Labels, Road Junctions and Landmarks in 2D Orthogonal Maps . MSc Thesis , University of Zurich.

8. Huang J , White RW and Buscher G , 2012 . User See, User Point: Gaze and Cursor Alignment in Web Search . Proc of CHI 2012 .

9. Jiang X , Atkins MS , Tien G , Bednarik R and Zheng B , 2014 . Pupil responses during discrete goal-directed movements . Proc of CHI 2014 , 2075 - 2084 .

10. Johansson RS , Westling G , Backstrom A and Flanagan JR , 2001 . Eye-hand Coordination in Object Manipulation . Journal of Neuroscience 21 : 6917 - 6932 .

11. Mac Aoidh E , Bertolotto M and Wilson DC , 2012 . Towards dynamic behaviourbased pro ling for reducing spatial information overload in map browsing activity . Geoinformatica 16 : 409 - 434 .

12. Rodden K , Fu X , Aula A and Spiro I , 2008 . Eye-mouse coordination patterns on web search results pages . Proc of CHI 2008 .