-Many areas of software engineering require good program code reading skills. We analyse the process of program reading using gaze tracking technology. We performed a study with six subjects, who performed four code reading tasks. The errors the embedded into program sources code and the lines of code with the areas were analysed as Areas of Interest (AoI). We formulated a research hypothesis and tested it using a one-way analysis of variance (ANOVA) test. The results of the study confirmed our research hypothesis that the number of fixations on AoI is larger than the number of fixations on other areas.
Program code reading skills are important in many areas
of software engineering, especially, in adopting good code
writing practices and techniques, understanding how
programs work, identifying cases of poor programming style
and bad design, and delivering effective software
maintenance. Examples include program tracing and
searching for bugs, code smells and design anti-patterns [
Eye movements are directly related to cognitive and
information processing processes, and through these
processes, visual information is used to stimulate the brain and
to understand the given task. There are two assumptions
related to cognitive processes and fixations: 1) if a person is
seeing an object (such as a word), he/she tries to understand
it; 2) a person fixates his/her gaze on an object until he/she
understands it. A fixation is an aggregation of gaze points
based on a specified area and time span. An Area of Interest
(AoI) is a part of a visual stimulus that is of special importance
Other important characteristics are a scan path, which is a
series of fixations that indicate the path and tendency of eye
movements, and a heat map, which identifies the focus of
© 2019 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0)
visual attention [
In this paper, we describe the results of gaze tracking study on evaluating and analysing the code reading skills of software programmers, specifically focusing on the ability to find errors in program code.
The study consisted of 4 tasks: a. In Task 1, the aim was to read the program source code with the aim of finding the result returned (printed) (Fig. 1).
b. In Task 2, the aim was to identify the purpose of the algorithm and discover the hidden error associated with the incompatibility of the variable types (Fig. 2).
c. In Task 3, the aim was to find three syntactic errors related to the incorrect use of variable names, types and basic methods (Fig. 3).
d. In Task 4, the aim was to determine whether the algorithm would perform the specified function, and to find a hidden semantical error (Fig. 4).
During gaze tracking we collect the number and location of fixations, which are gaze points that are directed towards a certain part of an image, which is labelled as Area of Interest (AoI). Fixations are indications of visual attention. Here we analyze the distribution of the number of fixations between and out of AoIs. The eye movements between fixations are known as saccades. However, we do not use the saccade data in this study. A scan path is a directed path created by saccades between eye fixations.
We assume that subjects are thinking about the object of interest when they are looking directly at it. Based on this assumption, we formulate the following research hypothesis:
H1: The number of fixations on Areas of Interest is larger than the number of fixations on other areas.
For testing of hypotheses we employ a statistical one-way analysis of variance (ANOVA) test. The test, which is a standard statistical method, confirms or rejects the equality of the averages of two or more samples by examining the variances of samples. ANOVA compares the variance between the data samples to variance within each particular sample. If the between-sample variance is much larger than the within-sample variance variation, the average values of different samples can not be equal.
Six participants (1 female and 5 male) were recruited for this study, ages between 20 and 25 with an average of 22.8 years. All participants had normal or corrected-to-normal vision. Participants were familiar with computers and had previous experience in using the internet and all of them were studying or working in programming sphere. An informed consent was obtained from subjects before the study.
All subjects were given the same laptop Dell which had an additional monitor used for experiment and the Tobii Eye Tracker 4C eye-tracking device used to record eye movements and gaze fixations. The eye tracker uses infrared corneal reflection to measure point of gaze with data rates of 90 Hz. A 24 inch screen was used to show the slides which consisted of programming source code. The eye tracker using instructions was mounted just below the visible screen area. The operating distance between the eye tracker and subjects’ eyes was between 70-75 cm. Efforts were made to ensure good lighting and a device calibrated before the test. For each subject the eye tracker was re-calibrated using an integrated 5-point calibration to achieve most accurate results.
Before the start of the experiment, the subjects were asked to fill in the Google Form questionnaire before the start of the study on their demographical characteristics (gender, education, age, level of programming skills). All responses were anonymized. After filling personal characteristics subjects had a chance to read some common information about tasks that they will face in this experiment, this way subjects were informed about some important rules, for example, no additional libraries or other extensions were used, also that some tasks were bug free and some had hidden bugs, the idea was to stimulate the subjects to be focused by not telling what tasks had bugs and what were bug free. After introducing tasks in common, the presentation with the slides containing the source code of tasks was opened, the observation session started at the start of each task and the session was stopped after the task was completed, each task had a separate observation session. 3 and 4 tasks had some brief information about given algorithms, for example, definition of palindrome and Armstrong’s number and examples of each case. To complete each task, 90 seconds were given. After the completion of each task, the participants were asked to provide the answers in a Google Form on what is the result of program execution (Task1), what is the purpose of an algorithm (Task 2), and is the program correct (Task 3, Task 4). option to choose screen resolution manually, which will allow to select concrete zones of interest.
The results of participants (number of fixations) are summarized according to tasks and subjects in Fig. 6.
Gaze monitoring system was used to measure the number and duration of fixations in the Areas of Interest (AOIs). The system consists of components listed below (see Fig. 5).
The Data Gathering Module reads the raw gaze data from the eye tracker device via USB.
The Data Preprocessing Module filters noise, calculates additional metrics and characteristics like saccades. The Data Persistence Module saves the acquired gaze data to CSV, XML or database.
The Data Post-processing Module maps persisted gaze data to AOIs and calculates additional data features such as the total and average number and duration of fixations. The Configuration Module configures how data is gathered and persisted in the system.
System offers four types of data stream which are used to gather fixations and saccades directly from gaze tracking device.
Unfiltered gaze Lightly filtered gaze Sensitive fixation Slow fixation
For this experiment, sensitive fixation type was chosen because of its accuracy and unnecessary noise reduction.
In addition, the system is running in the background and it has no effect on the stimulus, thus the subject's attention is concentrated only to source code.
Besides types of data stream, before starting gaze tracking session, user has an option to choose to record his screen, but for now it is only a prototype version, which needs to be improved for better accuracy, also session can have additional information about subject, for example name, age and other description, if it is not necessary, user can select anonymous session. In the near future, system will offer an
An example of the gaze path generated from gaze tracking data is presented in Fig. 7. The gaze path shows how and in what sequence the subject has read the code. Note the order of reading is clearly not linear.
An example of the heatmap generated from gaze tracking data is presented in Fig. 8. Note that most of attention was focused on and around the Area of Interest centred on code line 42 (see also Fig. 1).
In Fig. 9, the average gaze fixation numbers for AoI and Non-AoI areas is presented. We can see that for all tasks, the number of fixations on AoIs was larger, although the difference was not statistically significant for Task 2 (also see the results of statistical testing using ANOVA in Table I).
The results of statistical testing using ANOVA are presented in Table I. We found statistically significant differences in the number of fixations on the Areas of Interest (AoI) vs non-AoI for Tasks 1, 3 and 4. However, we did not find such differences for Task 2.
The study is based on the assumption that humans think
about objects when look at them, however we cannot be sure
that is assumption is correct. Our eye-tracking experiment
only explores the processing of cognitive response to visual
stimulus without considering the quality of responses.
Moreover, due to a small sample of subjects and gender bias
(all participants were male) we could not analyse the gender
and affective differences, which have been noted as significant
in other gaze tracking studies [
In three tasks of four performed we were able to confirm our research hypothesis. In one, task the hypothesis could not be confirmed. We think that we reason was in poor design of the task, which we hope to improve in our further research.
We have presented a study aimed at comprehending how programmers read and debug program code. Our results indicate that gaze tracking can be used successfully to follow and assess the cognitive behaviour of programmers as they correctly identify the errors embedded into the source code. The number of gaze fixations is a significant parameter when assessing the level of attention attributed to a particular Area of Interest.
Future work will focus on the methodological improvement of our research study and collection of a larger dataset of data from more subjects.