Artists value the ability to determine what parts of their composition is most appreciated by viewers. This information normally comes straight from viewers in the form of oral and written feedback; however, due to the lack of participation on the viewers part and because much of our visual understanding of artwork can be subconscious and difficult to express verbally, the value of this feedback is limited. Eye tracking technology has been used before to analyze artwork, however, most of this work has been performed in a controlled lab setting and as such this technology remains largely inaccessible to individual artists who may seek feedback. To address this issue, we developed a web-based system where artists can upload their artwork to be viewed by the viewers on their computer while a web camera tracks their eye movements. The artist receives feedback in the form of visualized eye tracking data that depicts what areas on the image looked at the most by viewers. We evaluated our system by having 5 artists upload a total of 17 images, which were subsequently viewed by 20 users. The artists expressed that seeing eye tracking data visualized on their artwork indicating the areas of interest is a unique way of receiving feedback and is highly useful. Also, they felt that the platform makes the artists more aware of their compositions; something that can especially help inexperienced artists. Furthermore, 90% of the viewers expressed that they were comfortable in providing eye movement data as a form of feedback to the artists.
Author Keywords eye-tracking; visual data; artwork analysis; heatmaps INTRODUCTION The intention of an art piece is often as important as the execution. As such, artists value the ability to determine what areas of their composition resonate most with their audience, as it can hint as to whether or not the artist’s intentions are evident in the execution. In this regard, knowing not only where a viewer first looks within a piece of art but also how their eyes travel through the piece and which parts of the image receive the most attention is crucial to an artist, as this feedback can inform the development of future pieces of art. Traditionally such feedback only comes in the form of oral and written feedback, which can be limited by difficulties in expressing opinions fully in words and/or by the viewer’s unwillingness to be overly critical. Furthermore, due to the qualitative nature of verbal feedback, it is difficult to collect and especially compare very large samples. Eye tracking technology has been used before to analyze artwork and provide feedback for the use of artists. However, most studies that utilize eye tracking as an art analysis tool are performed in a controlled lab setting and so far this technology is largely inaccessible to individual artists who may seek feedback. To rectify the lack of meaningful and authentic feedback, and the difficulty in acquiring specialized hardware for eye tracking, we developed VisualEYEze, a web-based solution that allows artists to upload their work and receive immediate, complete, and unbiased feedback from the viewers.
Our solution relies on feedback based on a viewer’s eye movements—tracked with a web camera—as they view the artwork. Specifically, we used eye tracking data to quantify where viewers look on a piece of art and how long they focus on different parts of the artwork. VisualEYEze provides this data to the artist by showing a heat map of the eye movements over the artwork. While creating VisualEYEze, we focused on the usability of the system and tried to accommodate the requirements of the artists and viewers. The artist first logs into the system to upload a set of artwork as images, and submits those images to be viewed by the viewers. Viewers can then see the submitted artwork on a web interface while the interface tracks their eye movements as they view the images. A central database stores both the images uploaded as well as the corresponding eye tracking data to generate the visualization. In this way, this system has the potential to positively impact numerous artists and the way their art is evaluated. Artists benefit from this system by being able to see how their art is interpreted. This will in turn help them create art in such a way that it is more likely to be perceived in the way that they intended. Furthermore, since this is a web-based system it is cost-effective as artists can easily reach a large number of viewers as they do not need to be physically present to view the work. Since a web camera is used for eye tracking, there is no need for specialized hardware, and the viewers can view the images and provide feedback at their own convenience.
In order to determine the usability of the system and how useful the artists may find the eye tracking data the system provides them, we conducted an evaluation through a multipart study. In the first part, we tested the system by having five artists upload their artwork and twenty users view the work while we gathered eye tracking data. In the second part, the artists viewed the eye tracking data superimposed on their artwork and provided feedback on the usefulness of such data. Overall, we found that our system could provide artists with a unique perspective on their work which could be useful in improving existing artwork or in beginning new projects. The system was determined to be easy to use although the viewer participation aspect could be improved with a reward system. Because of the very low barrier to entry, nearly any computer user could participate in a new community which focuses on facilitating a new type of feedback for artists. We believe that our cost effective and convenient solution allowing artists to receive meaningful, unbiased feedback will enable artists to reach a larger number of viewers and improve the quality of their work.
PRIOR WORK
Eye tracking has been used in a variety of application contexts
such as interaction and accessibility [
Analysis of Paintings with Eye Tracking
With regard to artwork, eye tracking has previously been used
to identify areas of interest as perceived by the viewer. This
involved studying how people view indeterminate art. In
a study conducted by Wallraven et al. [
Eye tracking has also been used to identify which aspects of an
artwork are most influential in viewing behavior. When human
subjects are present in the work, the subject matter itself seems
to have the larger impact on viewing behavior. However, when
the subject matter is a landscape, technical aspects of the
work seem to be the most important [
Research utilizing eye tracking has also been done to
determine what impact a change to a visual composition
element has on the eye tracking scan path. It has been
determined that a change in the size, shape, color, contrast,
proportions, or orientation of visual elements in a painting
causes a corresponding change in the viewer’s scan path across
the artwork. This is helpful information to provide to artists,
and we will allow artists to upload artwork for viewing so
they can see first hand how differences in versions of their
pieces change the outcome of the scan paths [
Another active area of research where eye tracking data is
commonly used is usability evaluation. Because gaze location
can help researchers identify the focus of an individual on
visual information, it is indispensable when it comes to
understanding the cognitive processes of users interacting with
graphical content. The processes for using eye tracking data
for usability evaluation sets a precedent for how to determine
the connection between raw data and a user’s thoughts. This
can be adapted to artwork as the idea of visual attention
and cognitive load is relevant to that space as well [
DESIGN MOTIVATION In order to make eye tracking technology available to artists, we wanted to design a system that allows artists to upload their work and receive instant feedback. To realize our goal, we created a web application that is accessible to any artist, and allows them to easily and instantly share their artwork with anyone and receive feedback. At any time an artist may check what feedback has been collected for the work they have uploaded. Users visit the same website to view artwork. While viewing artwork, eye-tracking is performed on the viewers using a web camera attached to their computers.
In our system, artists and viewers represent two separate entities: an artist is a user who uploads artwork on which they would like to receive feedback. Each artist has an account to manage their uploaded works and view their feedback. Thus, this account allows artists access to all the data collected on their work, and analyze it with a number of provided techniques. A viewer is a user who is looking at the artwork and providing the eye-tracking feedback. Viewers are not required to make an account and can view work freely. While artists and viewers represent separate roles in our system, any user can perform both roles if they so choose. To provide more details about the functionality of our system, below we present the available features by user role.
Create accounts to manage all of the work that they upload and the related feedback Upload artwork that they wish to receive feedback on to their accounts Set the defining information of their artwork such as the desired focal point and title of the work
Choose to share artwork for feedback by distributing a link to view that work directly View aggregate and individual heat maps generated from viewer’s gaze data View video progression of individual gaze data representing viewing order View fixation points of individual and aggregate gaze data View any written feedback provided by viewers who saw their artwork
Provide gaze data while viewing artwork by having a web camera enabled View their fixation points on each artwork after the slideshow is completed Optionally provide written feedback on individual works of art based on their fixation points
Creating a system where the roles of artist and viewer are separate allows the artists to have control over what work they would like feedback on, and gives them direct access to that feedback. In this way, a viewer is merely a subject of eye-tracking and as such they do not have to perform any complicated set-up or installation in order to contribute eye tracking data. It was the goal for our system to be intuitive in order to gather the most data. Because everything is accessed by web page and the eye tracking is performed with a standard computer camera, most personal computer users are able to participate either as an artist or viewer. Any artist who wishes to get information about their work can do so by uploading their art and waiting until it is viewed by the public. Furthermore, if an artist desires to speed up the process, they can manually share a link to their artwork to get instant feedback.
Architecture VisualEYEze mainly comprises of three modules: 1) Artist’s Interface, 2) Viewer’s Interface, and 3) Central Database. The interaction between these modules is shown in Figure 1. IMPLEMENTATION We used a number of tools and frameworks to develop our system. We used Ruby and Ruby on Rails to build a web framework and used Bootstrap and CSS for styling. To store the data from the artists and viewers, we used a PostgreSQL database. We had several tables, depicted in Figure 2, to store the data related to the artists, heatmaps, pictures from artwork uploads and feedback.
Open Source Components
We integrated several open source components into our system
to achieve the desired functionalities. The two primary open
source modules we used are Webgazer.js for eye tracking, and
Heatmap.js for visualization of eye tracking data.
Webgazer.js
Webgazer is an open source, Javascript application developed
by Brown University researchers. Webgazer is the basis of our
eye tracking system which gathers all of the gaze information
from the viewers. Webgazer utilizes a viewer’s web camera
that is built in to their laptop. It requires minimal calibration
from viewers (we use 9 calibration points on the screen) and
does not require any special lighting. Additionally, Webgazer
is scalable and since it is open sourced it allows our platform
to be extremely accessible to all users [
Heatmap.js
Heatmap.js is an open source, Javascript application that
creates heat map visuals from data points. This software allows
us to generate heat map visualizations of the eye tracking data
collected via Webgazer. Heatmap.js is extremely scalable and
each heatmap can hold 40,000+ data points, making it ideal
for our aggregated heatmaps that are displayed to artists [
Artist’s Interface The artist logs into VisualEYEze and is directed to the dashboard as shown in Figure 3.
This dashboard shows all of the images uploaded previously by the artist. When the artist clicks on an image, they are taken to a page that shows all of the relevant information for that artwork as shown in Figure 4.
Furthermore, the artist would be able to update the artwork’s title, delete an artwork and all associated data, and also set a desired focal area as shown in Figure 5. As shown in Figure 6, the artist can even view various composition frameworks on their art.
Once an artist receives feedback from viewers on a piece, they can compare the perceived focal point of the viewer to the desired one that they set (see Figure 7). The heatmap, fixation points, and the focal area are overlaid on to the artist’s image to assist in comparison.
The composition frameworks page allows artists to see different general rules for choosing where their subjects are located in the composition. Each composition framework displays a description of the framework and where the focal areas and objects should be located as well as overlays the framework on the image so artists can easily see how this is represented in their artwork. We include three of the most common frameworks: rule of thirds, bisection, and golden section.
The first heatmap that artists see on the detailed artwork page
is the aggregated heatmap (see Figure 4). This heatmap is the
combined data of all users viewing data on this artwork. Artists
will also see their set focal area overlaid on this heatamp.
Artists also have the option to hide the heatmap overlay on the
aggregated heatmap so they can see the image more clearly
if they desire. Furthermore, to enable the artist to see how
each individual has viewed the artwork, the page lists all
the individual heatmaps from all of the individual viewer
feedbacks (see Figure 4). Each of these heatmaps has an option
to view a video progression of the heatmap being created that
correlates to how the viewer viewed the art in real time, with
the first points the user looked showing up first.
are displayed as yellow dots overlaid on the artwork. In
eye tracking analysis it is common to classify raw gaze data
into fixations and saccades. Saccades represent rapid eye
movements while fixations are where the viewer focuses for
a period of time. In order to identify fixations in our own
data, we implemented a dispersion threshold algorithm. This
algorithm has been shown in research to be comparably robust
and accurate to other classification methods while remaining
efficient and simple to implement [
Viewer’s Interface The viewer’s interface is simple and does not require a login. Using the URL shared by the artists, the user visits the website and clicks on "Participate as User" on our main page to begin the viewing process (see Figure 8).
Furthermore, the user is guided through a sequence of instructions asking for the user’s consent to participate in the experiment, and how the camera needs to be set up for calibration. Figure 9 shows the consent form where the user is explicitly informed that the web camera will be used to track the eyes and no video of the user’s face will be recorded.
After consenting, the user will be shown a page with a set of instructions explaining the calibration process, and how the experiment will progress as shown in Figure 10.
Also on these individual heatmaps are the extracted fixation points of the user on the artwork (see Figure 7). These points To being calibrating the web camera for the user’s eyes, we ask the user to make sure that the camera can see their face (see Figure 11), and then we have the user calibrate the camera so it can accurately detect where the user is looking on the screen. To proceed with calibration, the user clicks on the "start" button (see Figure 12). During calibration, the user is shown 9 dots placed on the vertices of a 3*3 grid that covers the entire screen (see Figure 13). Once the calibration is complete, the user gets a countdown of 3 seconds and then the slide-show begins where the user is shown at most 10 images for 10 seconds each. After the user finishes viewing all the images, she is directed to a page that asks the user to provide additional, optional written feedback on the images. This written feedback page shows the images the user just saw (see Figure 14). On clicking on an image, the user is shown the points she fixated on the image so that the user can tell the artist why she fixated on these points through written feedback (see Figure 15).
In order to provide sufficient feedback for each work of art submitted, we needed to keep the distribution of feedback as uniform as possible. In order to achieve this, we designed an algorithm that selects artwork for the slide show based on how little feedback it has received. This is a greedy algorithm, because we prioritize the artwork that has received the least amount of feedback each time a viewer wishes to use the platform. In fact, the only heuristic used in this algorithm is the amount of feedback associated with each artwork. The algorithm first selects the artwork that has no feedback associated with it and adds it to the slide show. If the number of artworks that have received no feedback is greater than or equal to the amount of images shown in the slide show, then the algorithm can terminate at that point. Otherwise, we add images to the slide-show that have the least number of feedback until there is enough. The algorithm breaks ties arbitrarily. The purpose of this algorithm is to pull artwork from the database in such a way, so that the distribution of the artworks between artists is mostly uniform. This costs Q(nlogn) time, where n is the number of artworks in the database. Our algorithm uses a greedy strategy in order to allow artwork that has little to no feedback to be chosen for the slide show.
VALIDATION STUDY We began initial evaluation of our system by conducting viewing tests on the selected area of the artworks. This initial evaluation was conducted to test the accuracy of the gaze prediction software we were using. To do this, we selected a set of images and designated certain areas as the region of interest by drawing rectangles around those regions. These images were then showed to a set of participants as a slideshow, and the participants were instructed to only look inside the rectangular regions as we performed eye tracking. A total of 6 participants took part in the study, and these studies were conducted in a number of different settings with different lighting conditions in order to understand how robust the system was. Each participant was tested once in each setting. We viewed the results of this preliminary experiment by generating heat maps for each viewer and artwork and comparing the hot spots of the heatmap to the set region of interest on the artwork. The results show that the eye tracking software is fairly accurate under good conditions. However, there were some instances when the eye tracking was entirely off. We attribute this to the system having difficulty calibrating or locating the viewer’s pupils under bad lighting conditions. EXPERIMENT DESIGN Following the preliminary validation study, we conducted a more comprehensive study. To avoid the inaccuracies and inconsistencies we observed in the preliminary study, the final evaluation was conducted in a lab setting. All the participants took part in the study on the same computer and under the same lighting condition in order to maintain consistency. To fully test our system, we conducted evaluations for both the artist and viewer roles. To do this, we brought in artists to upload their work in the first phase, and the same artists were invited back to view the eye tracking feedback. Users were brought in to view the uploaded artworks and we tracked their eye movements while they viewed the artworks.
Artist Study - Part 1 Our artist participants consisted of 5 individuals (4 female, 1 male). All participants had over 5 years of experience in art and focused on various mediums including painting, photography, drawing, and linoprint. Each artist uploaded 2 to 4 works of art to our system. This resulted in a pool of 17 works of art in total. These works were what we used in the remainder of our studies. The artists were asked to create an account, upload their artworks on a live version of our website, and set the focal point for each work. They were then allowed to browse the dashboard leisurely. Once they were finished, they were asked to fill out a short survey regarding the usability of the system based on their interactions up to this point. Artwork Viewing Study Once all artists had uploaded their original artworks, users were brought in individually to view the work and provide eye tracking data. This was done using a live version of the website. In total, we had 20 viewers provide eye tracking data (5 females, 15 males). First, the users calibrated the camera for eye tracking which preceded the artwork viewing session. Once calibration was complete, each user viewed a timed slide show of ten images. Each image shown to the participant was viewed for ten seconds. In order to ensure each artist got enough feedback, only images with the least number of views, selected using the greedy algorithm, were shown in each slide show. While viewing these images, gaze data was collected on their viewing behavior. Once the entire slide-show of artwork was viewed, users were asked to look at their fixation points for each piece to rate accuracy of our system and were given the option to provide written feedback to the artist. At the end of the study, viewers were asked to take a short survey about the usability and accuracy of our system.
Artist Study - Part 2 After the viewers contributed the eye tracking data, we presented the data to the artists through our artist dashboard. The same five artists from our first artist study were asked to come back for a second session. They were asked to log in and review the data on their individual artworks. For each artwork the artists were asked to look through their aggregate and individual heat maps. They looked through any written feedback if present. They were also asked to compare the fixation points received to their set focal points. Finally, they were asked to look through the composition frameworks and compare those intersections with the perceived focal points and fixation data. After they completed these tasks we conducted a final interview. They were asked a few questions with the main focus of knowing whether they found the eye tracking information to be novel and useful. This included whether or not they received unexpected data, whether the platform seemed novel, what feedback features they found the most effective, and so on.
RESULTS We conducted surveys at each part of the artist study, as well as after the artwork viewing study. Furthermore, semi-structured interviews were conducted with the artists after completing the second part of the artist study. All the studies were focused on understanding if the system supports the expected usability by the artists and the viewers and if both artists and the viewers accept the accuracy of the gaze data visualization on the artworks.
Artist Studies The surveys following the first part of the artist study showed that all participants found it easy to navigate and upload artwork to the system. To get a better understanding of the utility of our system we asked artists more detailed questions about their experience after they were able to see the eye tracking feedback in the second part of the artist study. Four of the five artists interviewed mentioned that the system was user friendly. One artist suggested the system may be difficult to navigate at first while another thought that having more control over the feedback types might add to the usability. When asked what features artists found unnecessary or distracting, only one artist responded that they did not understand the purpose of the composition overlays. In order to understand if our tool stood out among existing methods in use, we asked the artists to describe anything they felt was unique to the system. Two of the artists described the whole system as unique while the others referred to the eye tracking in general, or heat maps and video progression features as being unique. We also asked artists to describe what they found useful about this platform. Each artist mentioned that they found seeing where the viewer had looked useful. One artist mentioned that the system let them get a different perspective on their art and let them see where viewers looked in what order to determine where they keep coming back to on the image.
Furthermore, all the artists mentioned that they got surprising results in some way. Some artists thought that while viewers generally looked where they thought, they also jumped to looking at other areas that weren’t intended to be looked at as often. Artists also commented that people seemed to be looking at an object in their art that wasn’t supposed to be the main focal area. We also wanted to get the artists’ opinions on how this platform could contribute to the art community. Two artists thought that the platform would make artists more aware of their compositions and make them think more about what actually draws viewers in. They thought this would help artists learn to direct the eyes of the viewer, something that would be especially beneficial to inexperienced artists. Another artist thought this would give a better mindset when you start developing artwork and that it would make you think more about how to set up a photograph or drawing. This artist also saw the potential to upload versions of a piece to see initially where people look then make a change and then check it again. We also asked the artists about some ideas that they thought could be added to the platform to improve it even more. One artist recommended an accuracy benchmark for the data so that they could know how accurate the eye tracking was for each viewer. Another artist thought it would be useful to implement a feature to predict focal points on the image. One other artist would like to let viewers flag the areas on the images they viewed that they found most interesting and provide comments on those flagged areas. We also received a request to make the images display for longer than 10 seconds each.
Viewing Studies Following the viewers participating in the eye-tracking portion of the platform, we conducted a short survey to get feedback on the system. One aspect of the system that we were concerned with was accuracy since we used a web camera instead of traditional infrared camera hardware packages. Users were shown the fixation points of their gaze data for each image that they viewed and were asked to rate the accuracy of the system on a five point scale (see Figure 16). The feedback showed that 11 participants thought the system was accurate to some degree, 7 thought it was inaccurate, and 2 were unsure. Another concern we had was whether or not people would be comfortable having eye tracking performed on them with this type of platform. No participants claimed to be uncomfortable with it, while 18 reported to be completely comfortable with it (see Figure 17). We also wanted to know if the viewers would use this type of platform on their own time to provide feedback to artists. These results were split, with half the participants stating that they would use and the other half stating that they wouldn’t. However, fifty percent is still quite high for this type of platform and would be a huge help to artists and the art community.
Because this would ideally be an improvement on existing forms of feedback, we asked participants about whether or not they thought this eye tracking feedback was preferable to providing written or oral feedback. Thirteen users said eye tracking was preferable, six where indifferent, and one thought the traditional methods were preferable (see Figure 18). When asked to rate the usability of the system on a five point scale (one being easy and five being difficult to use), 35% rated it at 1, 30% at 2, 10% at 3, 15% at 4, 10% at 5 (Figure 19). DISCUSSION From the user studies, a strong finding was that artists and viewers alike found the system easy to use. For artists, who were more involved with the system as they had to create accounts, upload work, and utilized a variety of visualizations and features, this was nearly unanimous. They found that all the system features were accessible and that to their knowledge, much of the system was unique, especially when it pertained to eye tracking data. Given that all the artists reported to have five or more years of experience and rated their ability level at or above average, this is a strong indicator that no other commonly used tools provide the same kind of insights as our eye tracking framework. Every artist interviewed mentioned the usefulness of seeing the viewer’s gaze data and many offered unique suggestions as to how that may be helpful. From verifying artistic choices, to choosing how to improve future work, or even comparing two versions of the same piece, it seemed that the participants could easily imagine how they might find this data useful in their work. This agrees with the results that each artist claimed they would use the system in their own time to get feedback if it were available. Overall, it seemed that the artists who participated in our study found the platform a novel and useful way to gather feedback about their art.
Based on the viewer survey, most participants found the system easy to use and preferable to traditional forms of providing feedback to artists. In fact, 50% of viewer participants reported that they would be willing to use this type of system on their own time. This indicates that it is feasible that the system could have a sizable viewer user base. This is of course important because the system would not be helpful to artists without anyone willing to provide eye-tracking data. One concern that needs further discussion is the accuracy of the system. Seven of the twenty viewers thought that the system did not accurately record their gaze data while two were uncertain. This information was based on the viewer’s opinion after they were able to see the fixation points calculated for their gaze data on each artwork. This could mean that either the gaze data was in fact inaccurate, the fixation points were not an intuitive reflection of their gaze data, or that the users themselves where not certain of their gaze patterns. More testing into the accuracy of our gaze prediction and fixation classification would be needed to improve these results. One artist also suggested that some form of an accuracy benchmark for each viewer be given to the artist so that they can understand how to interpret the data.
CONCLUSION AND FUTURE WORK VisualEYEze is one considerable effort in the right direction for the art community. The eye tracking feedback in our platform provides artists with the knowledge of what viewers are actually looking at and focusing on. This in turn can positively influence future work an artist creates. The service is also accessible for almost anyone to use, because most modern computers have a built in web camera, which is what our eye tracking software relies on. Overall, the system was positively received by both the artists and the viewers. The artists enjoyed using the platform and found it useful for eye tracking data to be recorded from viewers looking at their artwork. However, based on the results, the viewers don’t have as much incentive to use the platform as artists, which was an expected outcome.
As part of the future work, we would include ways to incentivize the viewers to view artwork. Prior applications of this nature offer ways for the viewer to follow certain artists, or view certain genres of artworks they enjoy. The concern with the latter is that it would skew the data towards popular artists getting more traction than newer artists. We would have to balance the features to make sure even smaller artists would get feedback through the random slide-show viewing that is currently in place. Additionally, users are often offered monetary compensation for their time. To counteract this, artists pay a fee to use the service. This was not the intention of our application and as such we didn’t offer this, but it would be interesting to see our application in this light.
Furthermore, we would like to implement increased gaze duration by offering longer periods for gazing data. This way the user would be able to choose the amount of time that fits their schedule. We could also have the users mark where they felt they looked at the most beyond the tracking data. Additionally we could add an admin portal for managing the accounts. Often we noted viewers misinterpreting the fixation points and to alleviate this we could show them the heat maps as well. The accuracy was a big concern on both ends. The compromise of using web-cameras is the accuracy hit that the data takes when compared to something like eye tracking hardware. This was a choice done explicitly to fit our goal of accessibility to all. We figured the accuracy hit would be outweighed by allowing artworks to actually be viewed multiple times by making the entire website more accessible.