Search algorithms in image retrieval tend to focus on giving the user more and more similar images based on queries that the user has to explicitly formulate. Implicitly, such systems limit the users exploration of the image space and thus remove the potential for serendipity. As a response, in recent years there has been an increased interest in developing content based image retrieval systems that allow the user to explore the image space without the need to type specific search queries. However, most of the research focuses on designing new algorithms and techniques, while little research has been done in designing interfaces allowing the user to actively engage in directing their image search. We present an interactive FutureView interface that can be easily combined with most existing exploratory image search engines. The interface gives the user a view of possible future search iterations. A task-based user study demonstrates that our interface enhances exploratory image search by providing access to more images without increasing the time required to find a specific image.
H.4 [Information Systems Applications]: Miscellaneous
In recent years, image retrieval techniques operating on metadata, such as textual annotations or tags, have become the industry standard for retrieval from large image collections, e.g. Google Image Search. This approach works well with sufficiently highquality meta-data, however, with the explosive growth of image collections, it has become apparent that tagging new images quickly and efficiently is not always possible. Secondly, even if instantaneous high-quality image tagging was possible, there are still many instances where image search by query is problematic. It might be easy for a user to define their query if they are looking for an image of a cat but how do they specify that the cat should be of a very particular shade of ginger with sad looking eyes.
A solution to this problem has been content based image retrieval
(CBIR) [
However, in spite of the development of new techniques to
support queryless exploratory image search, not much attention has
been devoted to the development of interfaces to support this type
of search [
Most image search systems still rely on search queries in order
to return to the user a set of images associated with a tag related to
the search query [
The main idea behind interactive interfaces used in most
queryless exploratory CBIR systems [
Our solution to this problem is an interface that provides the user with a “peek into the future". The FutureView interface, illustrated in Figure 1, is divided into two sections. The left-hand part of the screen is similar to a traditional interface, where the user can rate images by using a sliding bar at the bottom of each image. However, after rating one or more images, the user is not taken to the next search iteration but instead presented with the future view of the next iteration on the right-hand side of the screen. This allows the user to “try out" what impact providing feedback to different images will have on future iterations. When the user is satisfied with one of the future views, he clicks the “next" button in the right upper corner of the screen to confirm his choice and then is taken to the next search iteration. 4.
We conducted a comparative user study to evaluate the impact
of FutureView on three types of image search tasks: target,
category and open. The study included two conditions: 1) our
FutureView interface; 2) a version of our interface without the future
view, which from now on we will refer to as "single view". The
same backend system was used with both user interfaces. We used
as our backend an existing exploratory image search system, the
details of which can be found in [
We recruited 12 post-graduate students from our university to participate in the study (3 female). The average age of the participants was 24 years (from 20 to 30). Google image search is the most frequently used images search tool by all the participants. 4.2
We used the MIRFLICKR-25000 dataset with three types of
features: colour, texture and edge, as described in [
Target search - the user is looking for a particular image, e.g. a white cat with long hair sitting on a red chair.
Category search - the user does not have a specific image in mind and will be satisfied with any image from a given category, e.g. an image of a cat.
Open search - the user is browsing a collection of images without knowing what the final target may look like, e.g. looking for an illustration to an essay about “youth”.
We used a within subject design so that every participant performed three tasks covering all task types in both systems (six tasks in total = 3 (task types) 2 (systems)). We designed two tasks for each category to assign unique task for each system. The subject of the two tasks for target search are: red rose, and tall building. In category search, we asked the participants to find images from the following categories: city by night, seashore. In open search, we asked the participants to imagine they were writing a newspaper article on a given topic and they had to find an image to accompany their article. The topics of the articles were: (1) happiness; (2) gardening. We selected these topics because they are well covered in the MIRFLICKR-2500 dataset. We showed 12 images per iteration in the single view interface and in Futureview. After receiving feedback, FutureView shows the next 12 images on the right-hand side. 4.3
At the beginning of the experiment, we briefed the participants as to the procedure and purpose of the experiment before they signed the informed consent form. We then provided them with practice tasks to get them familiar with both systems. The participant would then proceed to perform six search tasks, divided into two groups of three tasks so that each participant would complete each different type of search task once with both systems. Before they started the target search tasks, we presented three example images and a short description of the image that they should look for. We did not provide any example images for category search and open search tasks. We randomized the order of tasks as well as the order of systems. After training, the eye tracker was calibrated.
We instructed the participants to finish each task when they find
the target image (in case of target search) or when they feel they
found the ideal image for the tasks from category search and open
search. In all the tasks, we limited the search to 25 iterations to
ensure that the participants did not spend an excessive amount of time
on any task. After finishing each task, the participants completed
the NASA TLX questionnaire [
Overall 12 users completed 72 tasks and all the participants completed all the tasks in fewer than 25 iterations. Figure 2 shows the average duration of a search session and the average number of images shown over a search session. On average, category searches were the shortest (104 seconds with single view and 109 seconds with FutureView), while open searches took the longest (145 seconds with single view and 140 seconds with FutureView). The Wilcoxon signed rank test indicates no significant difference in search session duration for any search type with the two interfaces (p > 0:6). In spite of the fact that no additional time is required to complete each type of search with FutureView, users are exposed to a much higher number of images – on average three times more than with single view. The Wilcoxon signed rank test shows that this number is significantly higher in open and target searches (p < 0:05) and marginally higher (p = 0:05) in category search with FutureView. These results indicate that FutureView supports more exploration.
Figure 3 shows the average scores of the NASA TLX questionnaire. In spite of the fact that with FutureView users were exposed to three times as many images as with the single view interface within the same period of time, users did not report feeling hurried, stressed or irritated. Similarly, users did not feel that FutureView made the task more mentally or physically demanding and they did not feel that they had to work any harder to achieve their goal. The Wilcoxon signed rank test indicates that there was not significant difference between the two interfaces in terms of scores for questions 1,2, 4, 5 and 6 (p > 0:2). The users, however, felt significantly more successful completing the task with FutureView (p < 0:04 according to Wilcoxon signed rank test).
The eye tracking results show that the participants spent similar amount of time looking at both the current search results and the future view. Out of the 12 participants, three had excessive amount of errors in the eye tracking data, so only nine participants were considered. On average, the users spent 41.8% of the time looking at the future section of the screen, with standard deviation of 11.8%.
The post-experiment interviews with the participants also indicate that they found the FutureView interface helpful and easy to use. Some of the comments include: “The FutureView is pleasant to use and play with"; “The FutureView helps in reaching target quicker than the single view"; “The FutureView is helpful for people whose job is to search for images". These comments are in striking contrast to the remarks the participants made in the prestudy questionnaire, where they stated that most existing image search engines are tiring and cumbersome to use. The participants also remarked that “Single View can be discouraging as the user has no idea what is coming next", “ once deviated from the actual path, there is no way to come back [in single view]". 6.
In this paper, we introduced the FutureView interface for queryless exploratory content based image search. It allows the user to see the effect of the relevance feedback on currently presented images on future iterations, which, in turn, allows the user to direct their search more effectively. Initial experiments show that users take advantage of the FutureView interface and engage in more exploration than in a system with a single view interface.
Our future plans include more extensive user studies with various types of image datasets and various image feature representations. Currently, the FutureView does not save the search history. We are planning to add this feature to our system to allow the user to branch out their searches using any point in the history as a new starting search point.
This work was supported by The Finnish Funding Agency for Innovation (projects Re:Know and D2I) and the Academy of Finland (the Finnish Centre of Excellence in Computational Inference).