Triangulating Eye Movement Data of Animated Displays

Abstract

We present a cross-validation approach to assess animations of movement data. Specifically, we investigate if and how display design, data complexity and user background and training might influence participants' decision-making with animated designs. Our triangulation approach is based on eye-tracking records, galvanic skin conductance responses, and electroencephalography data. We raise data analysis issues and data synchronization challenges for discussion at the workshop relating to data integration at various resolutions. With this empirical triangulation approach, we hope to better understand user decision-making with animated displays, and aim to develop sound animation design guidelines.

Keywords

• Animation
• movement data
• decision-making
• human-subject experiment
• eye tracking
• electroencephalography
• galvanic skin response

Introduction

The need to more deeply understand how participants make decisions with animated displays representing moving objects is the main motivation of our study. The current lack of empirically founded design choices, and our still limited knowledge about cognitive and emotional processes involved in task-related decision-making with animations hinder the development of effective and efficient animated displays [ 7,1 ]. Despite these constrains, spatio-temporal data are increasingly depicted and explored with dynamic visual analytics displays. This might be because intuitively moving objects seem adequately represented with congruent movement changes over time in animated displays, and thus pattern recognition might be facilitated compared with static displays [ 8 ].

In this work in progress paper, we introduce a human-subject experiment aimed to examine how map-related (i.e., animation design type), data-related (i.e., data complexity and data context) and user-related factors (i.e., individual and group differences) might affect information exploration and decision making with animated displays in high-risk decision-making contexts. This user-study aims to systematically evaluate current semi-static and novel continuous Air Traffic Control (ATC) radar displays across ATC expert and ATC novice decision makers. We collected data from four different sources, i.e., eye movement data, galvanic skin responses (GSR), electroencephalography (EEG), and responses from standard questionnaires. We present our proposed research framework based on data triangulation and detail our cross-validation analysis process and related methodological challenges as a discussion basis for the workshop, to get additional feedback for further analysis refinement. At the meeting, we also intend to present a subset of preliminary results emerging from the proposed methodological triangulation analysis. 2 2.1

Methods

Experimental design and procedure

We designed a human-subject experiment according to a mixed factorial design considering how users’ cognitive processes and emotional states might inter-relate with display design choices. Our experiment stimuli are based on simplified French ATC radar screens, in which aircraft positions are updated every four seconds. We aim to compare these currently employed semi-static displays with novel continuously animated displays, and elicit if and how the above-mentioned (i.e., map-related, data-related, and user-related) factors might influence participants’ visuo-spatial decisionmaking.

The experiment was conducted with eighteen ATC experts at the Ecole Nationale de l'Aviation Civile (ENAC) in Toulouse and nineteen ATC novices at Temple University in Philadelphia according to a between-subject design. The experiment task consisted of watching aircrafts (four or eight) moving in the same direction, but at different speeds, in a series of randomized semi-static (N=16) and continuous animations (N=16). Figure 1 shows the employed data collection equipment set up at ENAC in Toulouse. Participants were asked to detect the accelerating aircraft as soon as possible per mouse click. Stimuli were shown on a color monitor screen at 1920x1200 spatial resolution. The animated portion of the experiment took on average 16 minutes. We recoded participants’ eye movements using a Tobii TX300 eye tracker, coupled with a mobile galvanic skin conductance response (GSR) recorder1. Participants’ brain activity was monitored by means of a mobile electroencephalograph2. We also collected written responses with a Short Stress State Questionnaire (SSSQ) [ 4 ] and additional background information. 2.2

Psycho-physiological data collection

We are currently analyzing participants’ cognitive states by examining the frequencies of the recorded EEG signals, and the GSR across animation designs, data complexity, and other user factors. For example, EEG signals with high frequencies and low amplitudes can be triggered by aroused participants, suggesting both alerted cognitive and motivated emotional states. Conversely, less motivated and engaged participants should exhibit EEG signals at a lower frequency, but with higher amplitudes [ 5 ]. These psycho-physiological data can be cross-validated with participants’ self-reports collected with the SSSQ questionnaire (i.e., engagement, distress, and worry). Figure 2 shows a participant’s GSR, processed with the BIOPAC3 software. 1 Smartband by http://www.bodymonitor.de 2 EPOC by http://emotiv.com/ 3 BIOPAC by http://www.biopac.com/ depicts the time line of GSR recordings, along which specific events have been marked up. Event-related potentials (ERP) are represented as blue drop symbols, that is, where significant changes in SCL occurred. The red columns indicate the time and duration of specific viewing events (i.e., the 16 animated stimuli). The red lines within each stimulus viewing event indicates the moment the participant clicked into the display to indicate target aircraft detection. 2.3

Methodological triangulation of gaze data with GSR and EEG

To further investigate how psycho-physiological responses might interact with display design during high-risk decision making with animated displays, we intend to combine collected gaze data with GSR and EEG records. The purpose of this analysis is to examine how external stimulus features or events (e.g., perceived visual cues) might interact with internal cognitive and emotional states. The event-related potentials (ERP) analysis seems useful here [ 6 ]. Specifically, we intend to examine participants’ saccadic eye-movement-related potentials (SERP) and eye-fixation-related potentials (EFRP) to triangulate viewers’ arousal states, brain activity, and eye movement behaviors when making visuo-spatial decisions with animated displays. 2.4

Methodological Challenges

We identified the following methodological challenges related to our proposed data triangulation approach, which we would like to discuss at the workshop:  How do we effectively synchronize triangulated data detectable at different signal latency durations (i.e., time between stimulus and event-related response)? For example, event-related potentials of GSR occur between 1–5s after an event [ 3 ], recorded eye-fixations have a duration of about 200-300ms (for visual stimuli) and 3080ms for saccades, respectively. EEG ERP (P300 = related to decision-making) are elicited at about 250-500ms after a stimulus [ 5 ].  How do we effectively synchronize triangulated data recorded at different temporal resolutions? For example, eye movement data can be recorded with a sampling rate of 300Hz, compared to 2048Hz for EEG and 10Hz for mobile GSR, respectively.  How do we effectively and efficiently compare participants’ scan paths with randomly moving AOIs (i.e., sequence analysis)? For example, tested aircrafts started to move at randomly selected starting points in the display, to avoid potential learning effects, and consequently aircraft (AOI) inter-distances change randomly across stimuli. 3

Summary and Outlook

We present our long-term empirical research framework to assess animated visual displays including eye movement analysis, triangulated with psycho-physiological (e.g., GSR and EEG) response data. We intend to investigate emotional and cognitive state variations across display design conditions, data complexity and context, and across users with different training and spatial abilities. The proposed methodological triangulation might enable us to better understand how users’ individual differences, cognitive, perceptual, and emotional states might influence their visuo-spatial decision-making with animated displays. Further analysis might include the relationship between participants’ arousal intensity and their affective states (i.e., positive and negative valence, such as engagement, joy, stress, boredom, etc.) [ 2 ].

Once we more deeply understand how cognitive and emotional states might interact with display design choices during visuo-spatial decision-making, we are able to develop display design guidelines to support effective and efficient visuo-spatial decision making with perceptually salient, cognitively supportive, and emotionally engaging graphic displays.

Acknowledgment

We cordially thank all test participants from ENAC (France) and Temple University (USA) for their participation in the experiment. We also thank Christophe Hurter, JeanPaul Imbert and Maxime Cordeil at ENAC, as well as Tim Shipley and Kelly Bower at Temple University for their precious collaboration and continued support for the development and conduction of our user study.

References

1. Fabrikant SI et al., 2008 , Novel Method to Measure Inference Affordance in Static Small Multiple Displays Representing Dynamic Processes . The Cartographic Journal , 45 ( 3 ): 201 - 215 .

2. Barrett LF and Russell JA , 1999 , Structure of Current Affect. Current Directions in Psychological Science , 8 : 10 - 14 .

3. Figner B and Murphy RO , 2011 , Using Skin Conductance in Judgment and Decision Making Research . In: Schulte-Mecklenbeck M , Kuehberger A , Ranyard R , editors. A Handbook of Process Tracing Methods for Decision Research . New York: Psychology Press.

4. Helton WS , 2004 , Validation of a Short Stress State Questionnaire . Proceedings of the 48th Meeting of the Human Factors and Ergonomics Society , 48 ( 11 ): 1238 - 1242 .

5. Holmqvist K et al., 2011 , Eye Tracking : A Comprehensive Guide to Methods and Measures . Oxford: Oxford University Press.

6. Luck SJ , 2005 , An Introduction to the Event-Related Potential Technique . Cambridge: MIT Press.

7. Tversky B et al., 2002 , Animation: Can It Facilitate?. International Journal of Human Computer Studies , 57 ( 4 ): 247 - 262 .

8. Ware C , 2013 , Information Visualization: Perception for Design . San Francisco: Morgan Kaufmann.