In this paper, we describe two aspects of a study we conducted of faceted search in an online public access library catalog (OPAC). First, we describe how we used log data from a university OPAC to develop a set of grounded tasks. Then, we describe our use of eye-tracking in a controlled laboratory setting to examine user behaviors performing the grounded tasks. We discuss the challenges we encountered both in using the log data to develop tasks and in collecting and analyzing the eye-tracking data.
Many libraries have recently redesigned their online public access
catalogs (OPACs) to include faceted metadata as part of the
search interface. In these systems, metadata such as the Library of
Congress subject headings, time period, and region are displayed
as facets that can be used to explore and refine search results (see
Figure 1). There are many open research questions about how
people use facets in a search process and the library science
community is especially interested in how these redesigned
OPACs are being used. We designed a study to examine how
long and in what sequences searchers looked at the major
elements of a faceted OPAC interface [
This paper describes two types of challenges encountered along the way: developing exploratory search tasks and analyzing eye tracking data.
Our study needed search tasks that balanced two competing
needs: first, the tasks needed to induce an exploratory mode of
search instead of the directed mode used in many studies. Second,
the tasks needed to be constructed in a way that allowed us to
make comparisons between subjects. In addition, the tasks needed
to be appropriate for the catalog available on the test system,
which was based on the North Carolina State University (NCSU)
Libraries OPAC, reflecting real usage of that catalog. The online
library catalog for NCSU serves on average 7,824 search
transactions and 1,087 user sessions per day [
To develop the tasks, we extracted three days of anonymized log data from the servers. This extracted data included both keyword search terms and any facets used in the searches. Our goal was to use this log data to identify actual searches executed on the NCSU OPAC that made use of facets. We were especially interested in identifying exploratory searches (as opposed to directed or known-item searches) in the log data.
We manually looked through the extracted log data to identify situations in which the user appeared to be doing an exploratory search that also included the use of facets. We looked for log entries where it was clear that the user issued several searches with the same or related keywords and in which they interacted with the results. Our selection criteria required that the log file show that the searcher: 1) had looked through more than one page of results, 2) had selected more than one facet that was not identical to the search term, and 3) the selected facets were from the subject, time period, and region facets. The deployed NCSU OPAC has additional facets, but we decided to focus on only these three for our study.
To further define the tasks, we then conducted our own searches
using the topics that were extracted from the log files. If a single
keyword search could easily address the topic, it was either
rejected as too easy or modified to either broaden or narrow its
scope. Iterating this process, we developed a set of four
exploratory search tasks to use in the study. More details of our
task development and refinement process are given in [
We used a Tobii 2150 remote eye tracker (http://www.tobii.com) to collect the eye-tracking data. This system includes a 21” LCD monitor with embedded infrared cameras that sample at 50Hz. The monitor resolution was set to 1024x768.
For this study, we focused on how facets were used in the search process. The deployed NCSU OPAC includes many interface elements and features that are tangential to our current research interest. To keep the study focused, we developed a customized OPAC interface, shown in Figure 1. There were six major areas of interest (AOIs) in the interface: 1) a keyword query search box, 2) an area to display the facets, 3) a breadcrumb trail showing the current search terms and selected facets, 4) an area to display the results list, 5) a drop-down menu to select how to sort the results, and 6) a checkbox for each result so that the user could indicate which results they wanted to record as their “answers” for each task. This customized interface still accessed the full NCSU catalog of over 1.8 million records.
To facilitate collecting the eye-tracking data, we made several adjustments to this customized interface. First, we made sure that the interface used fixed-width elements when possible so that we could easily define a template for the areas of interest on each page. Second, we included 5 pixels of “padding” between interface elements to help increase the precision of gaze data collection for specific AOIs.
Data collection using the eye-tracker was a tricky process. First, we seated each participant at the computer with the eye-tracker and went through a calibration process. After the first and second tasks, while the participant was completing a post-task questionnaire, the experimenter would quickly skim a video that showed the eye-traces that were captured from the previous task to make sure that the eye-tracking was good. In cases where it had problems, we would either recalibrate the equipment and/or remind the participant to sit as they had been sitting when doing the calibration. For two participants, the equipment could not maintain tracking for more than a few seconds and we had to discard the tracking data.
We observed that changes of posture were often the cause of eyetracking failure. A typical example was that participants would sit in a neutral posture while doing the calibration, but then either slump or “lean in” while engaged in the tasks. We often had to gently remind participants during the tasks to resume their original posture. While we initially were reluctant to interrupt them to correct their posture, we believe that the negative impact of this interruption was very small compared to the gains in better eye-tracking. We often used wording to encourage the participant to help us, such as, “The equipment is being finicky today, could you just sit up a bit so it can track you better?” Other types of eye-tracking such as head-mounted units might not have these issues with posture causing a loss of tracking.
The challenge of maintaining tracking has encouraged us to consider using a secondary monitor that will display the tracking status in our subsequent studies. This will allow us to monitor the tracking in real-time during the tasks and to encourage the participants to adjust their posture if needed.
One other challenge encountered was caused by the automatic update feature of Microsoft Windows. During the course of data collection (which spanned a week), the system performed an automatic update which upgraded the Internet Explorer browser to version 7. This was not compatible with the Tobii eye tracker and forced us to reschedule several sessions while we downgraded back to IE6.
Tobii Clearview analysis software (v 2.7.1) was used to segment
each web page viewed into the areas of interest (AOIs). This was
a labor intensive step. Each web page viewed had to be
segmented by hand by defining a box around each AOI using a
GUI tool. Templates can be used to define the locations of fixed
size and fixed position AOIs. However, for each page, the AOIs
from this template had to be adjusted because some of the
interface elements were of variable size (both horizontal and
vertical). For example, the vertical size of the facet AOI
depended on the number and length of the facets. Cutrell et al.
[
We analyzed the raw eye-gaze data to extract fixations that had a minimum of 100ms duration within a radius of 30 pixels. Different domains use different fixation criteria. For example, for reading text, fixations may be more tightly defined than for imageoriented tasks such as visual search. For reading tasks, the manufacturer (Tobii) recommended a 20 pixel radius for 40ms. For image tasks, they recommend a 50 pixel radius for 200ms. Because our tasks involved both aspects of reading and visual search, we chose their recommendations for mixed content (30 pixel radius for 100ms duration).
After defining the AOIs and extracting the fixations, the Tobii software output a time ordered sequence of gaze data. We wrote scripts in PHP to convert and analyze this data. The scripts had to accumulate fixations across AOIs, tasks and individual page views.
In analyzing eye-tracking data, two measures have been widely
used for related studies: fixation counts and fixation times.
Fixation count is thought to be an indicator of the importance of
the item (or AOI) being fixated upon [
This work was supported in part by grants from the NSF/Library of Congress (ISS 0455970 and ISS 0812363) and a grant from the Catholic University Grant-in-Aid Committee. We thank Doug Oard for the use of the eye tracker, and Joseph Ryan and Jason Casden for their help in configuring the interface for this study.