In the last decade, mobile eye trackers have become a commodity. With the decrease in their cost and their increased availability, a growing number of studies are exploring their potential in cultural heritage. In works on art fruition, the idea of using mobile eye trackers to understand how we observe artistic exhibits is becoming commonplace. Following this trend, the goal of this study was to explore the potential and to propose an eye-tracking method to analyse the behaviour of museum visitors while they observe artworks, using the Hecht Museum in Haifa as a case study. Recorded data from two diferent observation sessions, before and after a course in art history, were used to produce heat maps, which serve as a simple and efective tool for studying changes in visitors' gaze patterns when observing artworks.
• Human-centered computing → Heat maps; • Applied computing → Arts and humanities.
How people look at works of art involves a mixture of individual
factors ranging from cultural background to personal sensibility.
Thus there is no "correct" way to look at art. Works of art might
convey diferent meanings and transmit diferent emotions and
feelings to diferent observers. Nevertheless, we also know that
artists try to instill their ideas in their work, as is also the case,
for example, in architecture [
According to Rorty [
When we are interested in something, we first of all look at it and examine it. Hence, in light of the aforementioned studies and with the purpose of observing diferences between professionals and lay observers, we explored the potential of using a mobile eye-tracker for tracking museum visitors’ gazing and we present a method to analyse users’ behaviour while they look at paintings. The mobile eye-tracker allows us to follow their focus of attention as they observe and explore the artworks.
We first focus on related works (Section 2), discussing how eye tracking devices were adopted in art appreciation. Next, the Hecht Museum case study is introduced, and the data composition and the data acquisition protocol are explained (Section 3). In Section 4 we explore emerging issues and challenges and we describe the solutions implemented to analyse the data and generate the heat maps. Finally, in Section 5 we present a brief preliminary analysis, followed by our initial conclusions in Section 6. 2
For most people, vision is the most active of the senses, used in
daily life to identify and distinguish objects or persons. Especially in
visual art, our eyes are the most useful tool we have for perceiving
the messages conveyed by works of art. Considering the importance
of this sense, many researchers have investigated the possibility of
extracting information from eye-gaze patterns without the support
of other subjective tools such as questionnaires and interviews.
In recent decades, scholars have focused their attention on
eyetracking devices, and nowadays, mobile eye-trackers are prevalent
in many research areas [
Understanding user behaviour and processes of visual art
appreciation are interesting challenges for which mobile eye-trackers
are efective tools. The use of these devices in visual perception
evaluation is not new and examples can be found in art [
Quiroga et al. [
In other studies, eye-trackers are used to examine common
observation patterns. For example, starting from the idea that Caravaggio
consciously constructed a narrative path in his works, Balbi et al.
[
Other studies focus on people’s individual cognitive diferences,
assuming that they influence users’ experiences. Raptis et al. [
Following the path of the above studies, in this paper we propose a tool to visually represent gaze data acquired from a mobile eyetracker, in order to understand visitors’ behaviour while looking at paintings. 3
Here we discuss the data available to be processed using the method
that will be described in Section 4. The data was acquired in the
Hecht Museum in Haifa. Previous research (Kuflik et al. [
The data for the study was collected by the Haifa team during a one-semester introductory undergraduate level art history course. The data was collected using the Pupil-Lab1 Eye-Tracker. As shown in Figure 1, it has two cameras, one which records the eye movements (henceforth "eye camera"), and another which records the environment the user is looking at (henceforth "world camera").
The Haifa research team collected data from thirteen students in two sessions, one before the start of the course and one at the end. During the sessions the students were free to look at a set of predefined artworks as they liked and without time constraints. Therefore, the data is not homogeneous with regard to time, and it is highly variable with regard to how the students observe the artwork. While some users preferred to observe the work of art from close up, trying diferent observation angles in close examination of the details, others preferred to keep their distance while standing still, examining the artwork from one angle only. This kind of data poses non-trivial challenges. In fact, tracking gazes pointing at a specific stimulus is dificult without using some kind of marker. Typically, the objects to be tracked are identified by physical markers 1http://pupil-labs.com/pupil positioned on their boundaries, as shown in Figure 2. This solution is not acceptable in the case of art exhibits. First, it is not always possible to place markers on or next to an artwork. Second, visible objects overlaying part of the artwork could compromise the user’s experience. Therefore, we addressed this challenge by using computational vision algorithms, to be described in Section 4. The data set contains gaze data about statuettes and paintings, but in this work we focus only on paintings since they can be represented as 2D images. The problem of recognising 3D objects must be addressed diferently and is not discussed in this paper. In particular, for initial analysis, the paintings under discussion are Levona Benschen (Blessing the Moon) by Jakob Steinhardt and The Fisherman by Reuven Rubin (see Figure 3).
We obtained several data files for each participant but we considered only two that contain the information relevant to our goal.The ifrst is the video recorded by the eye-tracker world camera and shows the user’s point of view. The second contains time-stamped gazing information for each video frame. In particular, each row has the timestamp, the video frame, a level of confidence, 2D coordinates relative to the video and 3D coordinates relative to the eye-tracker. The way we use these files is discussed in the following sections. (a) Reuven Rubin, The Fisherman, 1926. (b) Jakob Steinhardt, Levona Benschen, Oil on Canvas. Courtesy of the Hecht 1920. Oil on Canvas. Courtesy of the Museum, University of Haifa, Israel Hecht Museum, University of Haifa, Israel
For each phase a pertinent python module was developed. Each module is independent, making it possible to reproduce only a single phase if necessary. Moreover, the modularity grants us better software maintainability. In the next sections we provide an indepth explanation of each phase.
(a) Original video light conditions. (b) The same frame after gamma and saturation were modified.
Here we discuss our three-step process of analysing and visualising users’ gaze patterns. The steps are: 4.1
The first phase consists of manipulating and filtering the data provided by the eye-tracker. It requires three input files: the video recorded by the world camera, the csv file containing the gaze, data, and a good quality image of the painting, from now on referred to as the reference image, to be matched in the video. During this phase, as the csv file is not suitable for the next elaboration step, we need to manipulate it by removing useless columns and switching the file format to tsv.
Moreover, we noticed that the video light conditions were poor, causing a low matching rate in the second phase. In order to solve this problem, we processed the video by modifying gamma and saturation, making it brighter and improving the performance of next phase. To this end we used the multi-platform tool FFmpeg 2 which provides several video manipulation functionalities. Figure 4 shows a comparison between an original and modified video frame. In the original version (Figure 4(a)), we can see that the shapes in the unlit areas of the painting are not well defined, in contrast to the modified version (Figure 4(b)). Obtained data will be used as input for the second phase. 4.2
Data manipulated in the previous step is used as the input to the
feature matching process described next. We first need to consider
the nature of our input files. We have a tsv file where coordinates
are expressed in pixels relative to the world camera video. This
means that the file contains more information than needed. In fact,
we only care about gaze coordinates falling within the painting
area. As people are left free to move during the recording session,
it is impossible to define the painting area in advance, but the
painting must be recognisable in each video frame. Thus the unneeded
information must be pruned from the gaze data. Moreover,
coordinates need to be translated from the video reference system to the
reference image system.
identifying the target stimulus in every frame of the recording and
mapping the gaze positions to a fixed representation of the stimulus.
It does this by identifying matching key-points between the
reference stimulus and each frame of the video. Key-points (see Figures
5, 6, 7) are obtained using the Scale Invariant Feature Transform
algorithm (SIFT [
Mobile Gaze Mapping [
The tool was tested for accuracy and precision by MacInnes et
al. [
(a) Before.
(b) After.
Here we discuss the heat maps that were obtained from the data set using our tool. Although ideally the heat maps should be analysed by art experts, here we try to interpret them numerically. Even a quick glance at the examples in Figures 11 and 12 shows noticeable diferences between the "before" and the "after". We assumed that these diferences could be analysed by examining three metrics: 3https://github.com/LumenResearch/heatmappy (a) Before.
(b) After. gaze distribution, gaze pathway and gaze focus on salient parts of the painting. Gaze distribution determines whether the user’s gaze is focused on a few points or distributed over many points, and it shows how this behaviour changes between the sessions. Gaze pathway is used to identify common patterns between the users in the pathways of the second recording session. We assume that the gaze pathway might change between sessions. Finally gaze focus helps us to understand whether the user’s gaze focuses on the salient parts of the painting as defined by art experts.
In this initial work we focused on the first metric - the gaze distribution. We assume that non-expert users focus their gaze only on a few points having a strong visual impact, while expert users are able to discern more details than non-experts. To estimate the gaze distribution we used the entropy measure. When it is applied to heat maps, it returns a higher value when there are many focal points and a lower value when there are few focal points. The results for all the subjects are presented in Figures 11 and 12. The y-axis shows the subjects participating in the experiment (S1,S2,...,S13) while the x-axis represents the entropy values. For each subject a comparison between the entropy found before the course (blue bar) and after the course (orange bar) is shown. Note that for The Fisherman we excluded three recordings because most of time the mobile eyetracker did not recognise the gaze, probably because of a calibration error. As already noted, Figures 9 and 10 present the heat maps generated while Subject 3 observed The Fisherman and Subject 5 observed Blessing the Moon. It can be seen that entropy indeed varies together with the gaze distribution. In particular, when the gaze distribution is low, we see contiguous and well-defined areas, but conversely, when the gaze is highly distributed, we see many and less-contiguous areas (see Figure 9(b)).
Entropy levels, in most of the cases, are higher in the session recorded after the course. For The Fisherman, the average entropy values are 4.31 (before) and 5.0 (after), and for Blessing the Moon they are 5.0 (before) and 5.55 (after), matching our assumption. Also interesting is overall gazing duration. Considering that the time parameter is important for the analysis, we discovered that, in most of the cases, users spent more time looking at the artworks in the second session than the first one. This may be considered as a sign of competence. While during the first session users probably looked at the artworks in a superficial manner, in the second one, after acquiring guidelines for art appreciation, they looked at the artworks more deeply, trying to discern the details. For Blessing the Moon, the average observation times for all participants were 30.61s before the course and 35.84s afterwards, while for The Fisherman, the average times were 27.4s before the course and 35.1s afterwards. Since this is a work in progress, we just show a preliminary and quite simple analysis based on entropy and observation time, but in the near future we plan to extend the study, together with art experts who will visually examine the heat maps visually in order to validate the current results, and we also plan to develop new analysis tools. Moreover, since in this study we did not consider the eventual influence of the first observation session on the second one, we planned to improve the analysis by involving a higher number of users and splitting them into two groups, one who will see the paintings before and after the semester, and one who will see the paintings only after the semester. By doing that, we will be able to make a statistical analysis of the two groups, evaluating the entity of the bias introduced by the first observation.
Understanding how people observe works of art, in our specific case paintings, is an open challenge and mobile eye-trackers are becoming the principal technology to address it. However, manual analysis of the data is impractical. For gaze_positions.csv for example, we had about 5000 rows for 43 seconds video. Therefore, an automatic tool that minimises the time spent on manual processing and exposes the data in a simple and intuitive format, could facilitate further studies in this field by allowing researchers to focus their efort on data interpretation. In this work we presented and implemented a pipeline to automatically filter, process and present mobile eye-tracker data. The data is presented through heat maps that can be easily interpreted by art experts. Moreover, we proposed a preliminary numerical analysis based on a calculation of heat map entropy calculation. In any case this is a preliminary work that poses new challenges to be addressed. In future works we aim to expand the tool’s features, introducing new metrics suggested by experts, and providing the option to extract and manage data from eye-tracking data recordings of 3D objects.
Heat maps and metrics provided by the tool could be used by museum curators and teachers to improve their work. For example, a teacher could understand if his students are acquiring the skills he explain and, if they are not, he could change the teaching approach. A museum curator could use the tool to understand if the painting position and the light condition emphasise the areas he wanted to. Finding in the heat maps that the the gaze rarely focuses on these areas could suggest that something needs to be changed in the artwork exposition.
The authors thank Menachem Merzel from the Information Systems Department at the University of Haifa for the data he collected, the students from the artwork analysis course of the Art History department who volunteered and contributed to the study, and the Hecht Museum for allowing us to perform the study. This work is funded by the Italian PRIN project Cultural Heritage Resources Orienting Multimodal Experience (CHROME) #B52F15000450001.