Effects of Latency Jitter and Dropouts in Pointing Tasks Andriy Pavlovych* Wolfgang Stuerzlinger† York University, Toronto, Canada and many computer games have significant delays, with 80– ABSTRACT 150 ms being most common [1]. Interactive computing systems frequently use pointing as an input Spatial jitter is caused either by hand tremor or noise in the modality, while also supporting other forms of input. We focus on device signal or both. Some devices also exhibit additional noise pointing and investigate the effects of variations, i.e. jitter, in the during movements. Hand jitter only exacerbates this problem, input device latency, as well as dropouts, on 2D pointing speed especially in devices used in free-space. Temporal jitter, or and accuracy. First, we characterize the latency, latency jitter, and latency jitter, refers to changes in lag with respect to time. dropouts in several common input technologies. Then we present 2.1 Characterizing Latency, Latency Jitter, Spatial an experiment, where we systematically explore combinations of Jitter, and Dropouts dropouts, latency, and latency jitter on a desktop mouse. The To measure the latency, a video camera simultaneously filmed the results indicate that latency and dropouts have a strong effect on motion of both the mouse and the cursor. The average delay of the human performance; moderate amounts of jitter in latency do not mouse cursor motion relative to motion of the mouse was change performance in a significant way in most cases. determined to be 8 ± 2.8 ms at the centre of the screen. More than 99.5% of the updates happened within 8–11 ms of the previous KEYWORDS: Latency, jitter, Fitts’ law, pointing, dropouts. sample. Practically all of the remaining samples followed within 5–8 ms. We never observed a dropout in a mouse. 1 INTRODUCTION Optical sensing method employed by the mouse appears to filter Latency, or lag, is the delay in device position updates [2]. the spatial jitter in hardware. Likewise, hand jitter, or hand tremor, Latency and spatial jitter have been previously demonstrated to does not appear to be an issue in our experiments, as resting the significantly impact human performance in both 2D and 3D tasks mouse on a physical surface largely eliminates tremor. Based on [3], [5], [6], [8]. Recent interest in remote application use our measurements and the fact that our participants were young, (application as a service, [7]), as well as a renewed interest in we assume the input had no significant jitter of either kind. interactive network gaming [4] highlights the need for systematic study of this phenomenon. Also, the pointing devices are affected 3 EXPERIMENT 1 to varying degrees in the reliability of position tracking. Any The first experiment compares effective throughputs under failure of the sensing gives rise to dropouts in the sequence of various magnitudes of latency, time jitter, and dropouts. Twelve position reports. students participated in the experiment. The study lasted 40–50 We present two empirical studies that systematically investigate minutes. The software, implemented a standard Fitts’ 2D task of the effects of dropouts and latency jitter on human performance. 13 targets in a circle. The experiment was within subjects, and the The studies employ Fitts’ law, a well-established model of order in which the various combinations of the factors were pointing device performance. In our experiments, we used a presented was randomized (without replacement), to compensate mouse as an exemplary low-latency, low-jitter device, and for asymmetric learning transfer effects. Each participant artificially added latency and latency jitter to it, to match the range completed 100 “rounds” with different latencies, latency jitters, of latencies and jitter present in other commonly used devices, as dropout durations, and dropout intervals, as described below. well as in computer networks. We also varied the number of samples the system was omitting (“dropping”) and the periodicity 5.0 of such omissions (Experiment 1), or the number and the 0 ms dropouts percentage of the omitted samples (Experiment 2). The main goal 4.0 125 ms dropouts was to determine, all else being equal, the effects of dropouts and 250 ms dropouts latency jitter on device performance at varying mean latencies. Throughput, bps 500 ms dropouts 3.0 As one can often trade some latency for a decrease in latency jitter, typically through time-domain filtering, and extrapolate the missing and delayed samples, knowing the interrelationships 2.0 between the factors allows a designer to make an informed decision in choosing an appropriate filter and its parameters. 1.0 2 BACKGROUND Latency is the time from when the device is physically moved to 0.0 0 50 100 150 the time the corresponding update appears on the screen. For technical reasons, it is hard to avoid latency. And it is known that Latency, ms latency adversely affects human performance in both 2D pointing [3] and 3D pointing [9]. Common LCD displays have update rates Figure 1. Throughput vs. latency and dropout duration. Here and of only 60 Hz and may exhibit lags of 40 ms [5]. If a DLP further, error bars represent standard error. projector is used, latencies as high as 100 ms may be encountered, The experiment had four independent variables in a (1×1 + 1×2 * e-mail: andriyp@cs.yorku.ca + 1×3 + 1×4) × (3×3 + 1) = 10×10 arrangement: † e-mail: wolfgang@cs.yorku.ca • Latency (constant part): 10*, 40, 100, and 160 ms; 30 • Dropout duration: 0*, 125, 250, 500 ms; durations this transition happens at higher percentages, e.g., after • Intervals between dropouts: 0*, 500, 1000, 2000 ms. 10%, for 80 ms-long dropouts, as can be observed in Figure 2. • Latency jitter (normally distributed, in addition to the One of the surprising findings (Exp. 1) was that latency jitter, constant value above): σ = 0* ms for 10 ms latency, 0, ±20 that is, variations of latency with time, had little effect on ms for 40 ms latency, 0, ±20, ±40 ms for 100 ms performance, resulting in the worst case in an 8.5% drop in performance at 100 ms base latency and jitter with σ = 40 ms. latency, σ = 0, ±20, ±40, ±60 ms for 160 ms latency; Compared to the dramatic drops with increasing latency or In the above list, * denotes the baseline condition, i.e., minimum dropouts, such a small drop is likely to be of little practical latency, no latency jitter, and no dropouts. We chose a Poisson significance. Moreover, we can hypothesize that a higher, yet distribution for dropouts, as it is often used to model independent constant, latency could result in worse performance compared to events, i.e., the time an event occurs does not depend on the just keeping the latency variations at their original level. previous occurrence. The indices of difficulty (ID), ranged evenly from 2.44 to 5.76 bits. 5.0 The dependent variable was effective device throughput. 3.1 Results 4.0 The effect of latency on throughput was significant, Throughput, bps 0% F3,33 = 200.43, p < .0001. The interaction between the latency and 3.0 1% dropout duration was also significant, F9,99 = 11.59, p < .0001. Figure 1 illustrates the results. 2% 2.0 5% 4 EXPERIMENT 2 10% In this experiment investigate the effect of lower dropout 1.0 20% percentages more thoroughly, to determine whether infrequent dropouts still have a measurable effect on throughput. Also, we 0.0 aim to determine if there is a threshold for dropout duration, after 0 50 100 150 which the throughput starts to drop progressively. Latency, ms This experiment had three independent variables in a 4 × (5×5 + 1) = 4×26 arrangement, for a total of 104 combinations. In the following list, * denotes the baseline condition, i.e., minimum Figure 2. Throughput vs. levels of latency and dropout %. latency, no latency jitter, and no dropouts. The dependent variable was effective device throughput (in bits per second). All other For small dropout durations (up to 40 ms), dropout percentages aspects were similar to the preceding experiment can be relatively large (up to 20%), without noticeable effects on • Latency (constant): 10*, 40, 100, and 160 ms; performance. On the other hand, longer dropouts (e.g. 160 ms) • Dropout duration: 0*, 10, 20, 40, 80, 160 ms; have significant effects even at low percentages (5% and more). • Dropout percentage: 0*, 1, 2, 5, 10, 20%. While long dropouts have a dramatic impact on performance, they are encountered in fewer situations, and, overall, their impact 5 RESULTS AND OVERALL DISCUSSION on performance is either similar to, or lighter than the impact of The effect of latency on throughput was significant, frequently encountered latency levels. Initial indications exist that F3,33 = 359.40, p < .0001. No other significant interactions were interpolating dropouts by filtering may be of little or no use: for observed. Figure 2 illustrates the results. short intervals – because short dropouts have little effect on The effect of dropout duration on the throughput was performance, and for large dropouts – due to this not being significant, F5,55 = 3.08, p < .05. According to a Tukey-Kramer feasible. To summarize, while both latency and dropouts have test, only the 160 ms condition was different from the others. The detrimental effect on pointing performance, normally distributed effect of dropout percentage on the throughput was significant, latency jitter seems to have no noticeable effects. Filtering in F5,55 = 16.55, p < .0001. According to a Tukey-Kramer test, no order to combat latency jitter may actually be harmful, as the statistically significant difference exists between the 0, 1, 2, and filter-added latency may outweigh any potential advantages. 5% conditions. The interaction between the dropout percentage Finally, we estimate that both latency and dropout duration are and duration was significant, F16,176 = 2.18, p < .01. multiplicative factors for predicting the throughput. This suggests For low latencies, below approximately 40ms, we observed no incorporating them into a homogeneous model for estimating the significant differences in throughput, consistent with the first human pointing performance in the presence of latency and experiment and a previous study [5]. The significant interaction dropouts. This is a subject of future research. between latency and dropout percentages seems to be due to the REFERENCES 20% dropout condition, which has a significant drop of [1] Console Gaming: The Lag Factor. http://www.eurogamer. performance, F1,11 = 8.17, p < .05, even at low latencies, whereas net/articles/digitalfoundry-lag-factor-article. the lower dropout conditions don’t have such behaviour, [2] Foxlin, E. 2002 Motion tracking requirements and F1,11 = 0.09, ns; see Figure 2. technologies. Handbook of virtual environments: Design, im- For dropout durations of up to 80ms, there seems to be no plementation and applications, Lawrence Erlbaum, 163- 210. significant effect on throughput, F4,44 = 0.48, ns. For dropout [3] MacKenzie, I. S., and Ware, C. 1993 Lag as a determinant of percentages up to 5% we observe no significant drop in human performance in interactive systems. ACM CHI ’93, performance, relative to the no-dropout condition. Looking at 488. dropout durations of 160 ms we see a significant drop in [4] Online Gaming Sees Significant US Growth. Accessed Dec. performance above 5%, F1,11 = 24.54, p < .0001, and no drop 2009. http://www.edge-online.com/news/online-gaming- sees-significant-us-growth before that, F2,22 = 1.93, p = 0.16. However, for lower dropout 31 [5] Pavlovych, A. and Stuerzlinger, W. 2009. The tradeoff between spatial jitter and latency in pointing tasks. In Proc. EICS '09. ACM Press , 187-196. [6] So, R. H. Y., and Chung, G. K. M. 2005. Sensory Motor Responses in Virtual Environments: Studying the Effects of Image Latencies for Target-directed Hand Movement. IEEE Engineering in Medicine and Biology Society, 5006-5008. [7] Software as a service. http://en.wikipedia.org/wiki/Software_as_a_Service [8] Teather, R., Pavlovych, A., Stuerzlinger, W. and MacKenzie, S. 2009. Effects of tracking technology, latency, and spatial jitter on object movement, IEEE 3DUI 2009, 43-50. [9] Ware C., and Balakrishnan, R. 1994. Reaching for objects in VR displays: lag and frame rate. ACM TOCHI 1, 4, 331-356. 32