=Paper=
{{Paper
|id=Vol-1175/CLEF2009wn-iCLEF-RuizEt2009
|storemode=property
|title=Users' Image Seeking Behavior in a Multilingual Tag Environment
|pdfUrl=https://ceur-ws.org/Vol-1175/CLEF2009wn-iCLEF-RuizEt2009.pdf
|volume=Vol-1175
|dblpUrl=https://dblp.org/rec/conf/clef/RuizC09a
}}
==Users' Image Seeking Behavior in a Multilingual Tag Environment==
https://ceur-ws.org/Vol-1175/CLEF2009wn-iCLEF-RuizEt2009.pdf
Users’ Image Seeking Behavior in a Multilingual Tag Environment
Miguel E Ruiz and Pok Chin
College of Information
Department of Library and Information Sciences
University of North Texas
1155 Union Circle 311068
Denton, Texas 76203‐1068
USA
Abstract:
This paper presents the results of a user study conducted in the framework of the Interactive Image
Retrieval task at CLEF 2009. The main goal of our research is to understand the way in which users
search for images that have been annotated with multilingual tags. The study is based on the
application of grounded theory to try to understand the challenges that users face when searching for
images that have multilingual annotations, and how they cope with these challenges to find the
information they need. The study includes two methods of data collection: an online survey and a face
to face interview that included a search task using Flickling. Because this was our first year
participating in the interactive image CLEF, we found that the most challenging aspect of conducting a
user centered evaluation in the context of CLEF is the short amount of time that is available from the
time the task is defined and the deadline for submitting results. User studies require the approval of
the research protocol by the Institutional Review Board of University before we can start gathering any
data and conducting user interviews. We were able to collect data for approximately three weeks (from
6/29/2009 to 7/17/2009) before the Flickling system was shutdown for server maintenance. During this
time we collected 12 responses to the online questionnaire and 6 face to face interviews. Our results
indicate that 67% of the users search for images at least once a week and that the most common
purposes for finding images are entertainment and academic. Our results from the user interviews
indicate that the users find the known-item retrieval task hard to do due to the difficulty in expressing
the contents of the target image using tags that could have also been assigned by the creator of the
image. The face to face interviews also give some feedback for improving the current Flickling
interface, particularly the addition of a spell checker mechanism and the improvement of the
multilingual translation of terms selected by users. Our results have limitations related to the number of
users that participated in the study as well as the fact that the users were recruited from only one of
the colleges at the University of North Texas. We would need to conduct these experiments in a larger
and more diverse population in order to derive a more general conclusion.
Introduction
Understanding the way people interact with information retrieval systems to fulfill their information
needs is one of the most crucial aspects that contribute to the acceptance of these systems. For this
reason we decided to participate in the CLEF interactive image retrieval track. For this year we
propose a user study that aims to understand the way in which American students search for images
and their needs for tools to support this task.
We present a study that follows an exploratory analysis method to identify the need for searching
images that American students have in general and to conduct an observation of the way in which
they approach the task of known-item search in a multilingual environment. Section 1 of this paper
presents some background on studies that explore user preferences in image searching. Section 2
describes the approach and methods used in the study. Section 3 presents the results and analysis of
our experiments. Section 5 presents our conclusion and future work.
1 Background
Research studies that explore user’s preferences on image retrieval are not that common. We present
here some of the studies that we consider are more related to the problem that we want to explore.
Goodrum (2000) suggests two opposite ends in user’s digital image searching: (1) focused specific
search, and (2) loose searching or browsing. According to Goodrum (2000), efficient image retrieval
systems may have to take into account various users’ need and image types available.
Many users employ the “browsing‐searching” combined techniques to find the images that fulfill their
information needs. The “browsing‐searching” model allows image seekers to refine their query using
different categories of images returned (Rasmussen, 1997). Some image retrieval systems, such as
Flickr, already support the “browsing‐searching” cycles by using techniques such as clustering and
tagging.
The Interactive CLEF track from 2008 started the use of a game setting to collect user interaction with
an interactive image retrieval system with multilingual support. The Flikling system is the same that is
being used in this year’s iCLEF and is described in more detail in the overview paper (Gonzalo,
Clough, & Karlgren, 2008). The work presented in the 2008 iCLEF paper inspired us to participate this
year in the conference.
2 Proposed Methodology
Although the main goal for the iCLEF conference was to use the game like setting to collect a large
number of interactions, in which we participated conducting our own searches, we recognize that we
needed to explore some more basic aspects related to the typical type of users that we expect to find
in the United States. We expect that most of our users are primarily fluent English with little or no
proficiency in other languages. This type of users is more likely to be very dependent on the
multilingual capabilities offered by the interface in order to be successful in finding images in Flikling.
There were two major questions that we wanted to explore in this research:
a) Finding more about the needs of predominantly English speakers when they search for
images. This includes the prevalence of their need for searching images (as opposed to other
type of information), the search tool that they prefer for conducting such searches, the type of
needs that they most recently have, and the context in which these information seeking
activities take place.
b) Observing these users while they perform their searches in Flikling to try to understand how
they use the multilingual capabilities and the major challenges that the users face.
For this purpose we created an online survey and recruited students in the College of Information at
UNT. We also recruited a reduced number of students to come and be part of a face to face section
that included some basic training on the use of Flikling, two sessions of searching images under
slightly different conditions and an exit interview to find out more about their experience with the
system and the challenges that they faced while performing the search task.
As a requirement of UNT we had to get approval from the Intitutional Review Board (IRB) before any
user was recruited and for this reason we had to come up with the full questionnaire and interview
protocol to ensure that our study complied with the human protection directives required in the USA.
This was a task that took some more time than we expected and reduced the amount of time available
for our research to about 3 weeks from June 29th (the first day when we were able to recruit users), to
July 17th (the last day when the Flikling system was still available).
The online questionnaire included 14 questions that gather the following information:
• Demographic information (4 questions): age range, gender, education, and language
proficiency
• Usage of search engines (3 questions): frequency of usage, preferred search engines, and
purpose for conducting searches (i.e. academic, entertainment, work related, etc)
• Data related to image searching behavior and preferences (7 questions): estimated frequency
of searches for images, purpose for conducting those searches, preferred search engines,
methods used for searching images (i.e. use of keywords, image descriptions and sample
images), eliciting an example of the latest image search that the user had conducted, the level
of satisfaction with the results obtained, and the ranking of importance of the features that
such search engines should have.
For the face to face interviews we follow a protocol that included a training session and three data
collection parts: A 20‐minute image searching without hint, A 20‐minute image searching with hint, and
an exit interview. The following section details the step‐by‐step procedure of this experiment:
1. Setup for experiment: All research subjects conducted the experiment using the same laptop
with the same system configuration.
2. Demographic data collection: The researchers collected demographic data from each
research subject before he/she performed the experiment.
3. Training: Before recording the experiment, the researchers trained each research subject
using the first and/or second images provided by Flickling. Research subjects learned the
Flickling functionalities by searching the first and/or second images. The training stop once the
research subject acknowledged to the researchers that he/she was comfortable with the
experimental procedures.
4. Part 1: After the training, the first part of this study allocated 20 minutes to measure the user’s
multi‐lingual image searching behaviors without using hints provided by Flickling. Each
research subject’s search session was recorded using TechSmith’s Camtasia Studio screen
recording software (http://www.techsmith.com). The screen recording of each research
subject’s activity was saved for further analysis.
5. Part 2: Part 2 of this experiment was almost identical to part 1. However, subjects were
allowed to use hints provided by Flickling in this part of the experiment.
6. Part 3: After completion of the part 2 tasks, the researchers conduct an interview with each
research subject. The researchers inform the subject that the entire interview session is
recorded using audio recording devices for transcription and subsequent analysis. Each
research subject is asked the same set of questions regarding the tasks he/she performed
during part 1 and 2 of the experiment.
3. Results and Analysis
Although we are still collecting data on the survey and plan to recruit more users during the Fall
semester which starts on August 27th, our preliminary results with 12 responses show some
interesting trends:
a) 75% of the users report to conduct online searches daily and 25% report to conduct web
searches at least once a week.
b) Users conduct online searches mostly for academic and professional purposes while
entertainment was ranked in third place.
c) Not surprisingly, Google was the most commonly used search engine and Yahoo ranked in
second place. A couple of users also reported using Bing (which was released shortly before
we started our survey) occasionally.
d) The majority of the users (67%) search for images at least once a week, 33% less than once
per month
e) The most cited purpose for searching images was entertainment, second academic and last
for professional or work related purposes.
f) The most commonly used search engine for finding images was Google, second Flicker and
third Yahoo.
We still need to do a complete analysis but these results show that there seem to be a clear difference
in terms of the motivations of the users to find images, as opposed to general search of information on
the Web. Entertainment seems to be the predominant purpose with academic in second place. Also
the frequency of searches for images was lower in comparison with general information search.
we were able to recruit 6 students to participate in the user observation and interviews. The protocol
took approximately one hour for each user. In general users had a very hard time finding images using
the Flikling interface. Most of the users were able to complete only a couple of searches during the
first 20 minutes when they were not allowed to use hints. In part 2 of the experiment, when they were
allowed to use hints, some users still seemed reluctant to use hints for fear to get a penalized and in
reported that they felt to be close to find the image that they were looking for. For most users the use
of hits was a relive and they usually had to get two hints (language and a term) before finding the
target image. The results from the interviews confirmed that the major barriers that the user faced
were: coming up with the appropriate terms for describing the target image, and finding the
appropriate translations because in many cases Flikling was not able to provide a translation for the
terms that they had selected and they had to use Google translations to find an appropriate term to
add to the multilingual query.
User’s also reported that some features that they would like the system to have such as a spell
checker mechanism so that they could find out whether their terms were properly spelled or available
in the tags. When asked about how they would rate their performance in the task on a scale from 1 to
10 (where 1 was poor and 10 was perfect) 2 of them scored themselves at 6 while the other 4 scored
themselves at 4.
4. Conclusions
We still need to complete a more in depth analysis of the data collected in this experiments but the
initial results seem to point towards some interesting facts that we can explore in future research. In
general it seems that users don’t find it easy to use the tags assigned to the images and find it hard to
come up with appropriate tags that could accurately describe these images. Also it seems from the
data gathered in the survey as well as from the user interviews that the known item search is not
probably a task that users perform frequently and seem to be rather unusual as a valid information
task. Most users report some sort of topic or theme based information need rather than the need for
finding exactly a specific image. This is a challenge that our community as a whole needs to rethink in
order to come up with a protocol that is more meaningful and closer to the real users’ needs.
We recognize that these results have some strong limitations related to the amount of users recruited,
as well as the recruiting pool which consisted only of university students. The survey and face to face
interviews would need to be conducted with other audiences to try to gain a better understanding of
the information needs of more general web users.
Bibliography
Gonzalo, J., Clough, P., & Karlgren, J. (2008). Overview of iCLEF 2008: Search Log Analysis for
Multilingual Image Retrieval. Working Notes for the CLEF 2008 Workshop. Aarhus, Denmark.
Goodrum, A. (2000). Image information retrieval: An overview of current research. informing Science,
3 (2), 63‐67.
Rasmussen, E. (1997). Indexing images. Annual Review of Information Science and Technology , 32,
169‐196.