=Paper=
{{Paper
|id=Vol-1175/CLEF2009wn-iCLEF-VassilakakiEt2009
|storemode=property
|title=Users' Perceptions of Searching in FlickLing
|pdfUrl=https://ceur-ws.org/Vol-1175/CLEF2009wn-iCLEF-VassilakakiEt2009.pdf
|volume=Vol-1175
|dblpUrl=https://dblp.org/rec/conf/clef/VassilakakiJHR09
}}
==Users' Perceptions of Searching in FlickLing==
https://ceur-ws.org/Vol-1175/CLEF2009wn-iCLEF-VassilakakiEt2009.pdf
Users’ Perceptions of Searching in FlickLing
Evgenia Vassilakaki, Frances Johnson, R.J. Hartley, and David Randall
Dept. Information & Communications, MMU
{evgenia.vassilakaki}@student.mmu.ac.uk,
{f.johnson,r.j.hartley,d.randall}@mmu.ac.uk
http://www.hlss.mmu.ac.uk/infocomms/
Abstract. This study aims to explore users’ image seeking behaviour
when searching for known, non-annotated images across languages in
FlickLing. In particular, the investigation reported in this paper aims to
demonstrate the value in focusing on user’s trust and confidence in the
exploration of seeking behaviour to reveal users’ perceptions of the tasks
involved when searching across languages. The task assigned to our 24
users was to search for three specific images described in Dutch, Ger-
man and Spanish consecutively. In this context, four different methods,
both qualitative and quantitative (questionnaires, observation, retrospec-
tive thinking aloud and interviews) were employed. Preliminary finding
regarding users’ switching between the two modes, users’ thinking of
languages and their answers to the two questionnaires on trust and con-
fidence are only reported here. It appears from our preliminary findings
that there is variation in our users perceptions of searching for images
across languages and their approach to using translations and that this
occurs regardless of the amount or type of help or guidance given.
Key words: Multilingual Information Retrieval, User Behaviour, User
Image Seeking Behaviour, Flickr, FlickLing, iCLEF
1 Introduction
The development of efficient and effective Cross Language Information Retrieval
(CLIR) systems depends not only on the implementation of the various tech-
niques for organizing linguistic resources but also on users’ adaptation and us-
age. The interactive CLEF (iCLEF) track in Cross Language Evaluation Forum
(CLEF) provides the participants with the necessary environment to run inter-
active experiments and focus on users’ image seeking behaviour in multilingual
environments. This is our second year participating in iCLEF track focusing on
users’ behaviour while searching across languages in FlickLing.
In the iCLEF2008 paper [4], the aim of the study was to explore users’ image
seeking behaviour when searching for the three known, non-annotated images
across languages in FlickLing. In particular, three research questions were ad-
dressed: a) identify the reasons that determined our users’ choice of a specific
mode (monolingual/ multilingual), b) examine if and/or to what extent users
2 E. Vassilakaki, F. Johnson, R.J. Hartley, D. Randall
were thinking about languages when searching and retrieving images and c) ex-
amine if and/or to what extent users were paying attention to translations when
searching and retrieving images. Four different methods, both qualitative and
quantitative were adopted to answer these questions. A questionnaire was used
to collect data on the users’ knowledge of foreign languages, previous experience
in searching for images, in searching across languages and previous experience
with Flickr. An observation sheet was designed to help the facilitator take notes
and to record comments and remarks made by the users on specific elements of
their search behaviour. These notes further aided the formulation of questions
during the interviews. In addition, retrospective thinking aloud was implemented
to shed light on the users’ actions and thoughts. Screen capture software was
used to record a search session and this was then played back to the user asking
them to describe what they were doing and what they were thinking. Finally,
semi-structured interviews were carried out with each participant focusing on
further explaining their specific actions, comments and remarks stated during
the whole study. For further information on the way these methods were em-
ployed see [4], [5].
The iCLEF2008 small study [4] showed that less than half of our ten users
appeared to consider identification of the language important in retrieving the
images. The majority either lacked confidence in using the different languages
or they were so focused on retrieving the images and completing the task that
they were not thinking about languages. Only four out of the ten users tried
to identify the language of the images and interact with the translations. These
users judged the translations as poor because either they were not retrieved, they
were not corresponding to users’ search terms or they were retrieving irrelevant
results. As a consequence, users were losing interest and trust in translations
resulting in no usage of them or not paying attention to them.
The iCLEF2008 study also revealed some insight into users’ trust in search-
ing in a multilingual environment such as FlickLing and a possible relationship
between trust and the use of translations, to search for the images in different
languages. In addition, it provided some indication that users’ confidence has an
impact when searching for images across languages and that confidence may or
may not be affected by users’ knowledge of foreign languages. In this context, a
follow up study was designed for iCLEF2009 drawing from the lessons learned
from the iCLEF2008 paper [4] and focusing on users’ trust and confidence as
revealed and/ or formulated in multilingual environments. As a consequence, the
iCLEF2009 paper again aims to explore users’ image seeking behaviour when
searching known, non-annotated images across languages. In particular, the in-
vestigation reported in this paper aims to demostrate the value in focusing on
user’s trust and confidence in the investigation of seeking behaviour to reveal
users’ perceptions of the tasks involved when searching across languages.
The remainder of this paper is structured as follows: a description of the
methods employed and the way the study was carried out are outlined in section
2. An initial analysis of the findings is given in sections 3. Finally, we conclude
by summarizing the current indications of users’ behaviour in section 4.
Users’ Perceptions of Searching in FlickLing 3
2 Method
In this section, further details on the test object, the task, the users and the
methods employed are illustrated.
Test Object: This study used the FlickLing multilingual search engine for
images as developed and provided by iCLEF organizers [4]. The same set up of
the interface was kept for this year’s participants to carry out their interactive
studies.
Task: As in the iCLEF2008 paper, our users were asked to search for the
first three known, non annotated images given after login in FlickLing. The users
did not know in advance in which of the six languages supported by FlickLing
(English, German, Dutch, French, Italian and Spanish) the three images were
described. As a result, users had to use all the features and different modes
(monolingual/ multilingual) provided by FlickLing to search the three images
across languages.
As opposed to last year’s study, the three images were pre-selected and pre-
sented to all 24 users so that the seeking behaviour and the thoughts recorded
for each user referred to the experience of searching for the same three images.
The three images were described in Dutch, German and Spanish consecutively.
Drawing from [4] during the selection process all English described images were
excluded as it was found that the majority of the users used English keywords
to search for the given images and only a landmark or a piece of writing would
trigger them to search across languages before going for the hints. As a result,
no English images were given in an attempt to encourage users to think about
languages and interact with the translations.
In addition, the images selected had both a ‘heading’ and at least three ‘tags’
describing them in order to provide our users with as many entry points to
these images as possible. This emerged from the iCFLEF2008 study [4] as users
complained when the images lacked either a heading or tags making their task
of finding it more difficult. Furthermore, all three images included a distinctive
landmark or a piece of writing that hinted at the language of the image. Again
the 2008 study highlighted the users behaviour in using details in the image
to identify the language of the image. Furthermore, there was no order of easy,
medium and hard to find images adopted and applied in both studies since there
was not such a distinction in FlickLing.
In this context, it should be also noted that during this year’s study, words
like task, or study were carefully avoided both from the written instructions read
to the users and during the process of recruitment. Again drawing from [4] it
was found that the users were obsessed with finding the images and complet-
ing the task “successfully” and that this competitive behaviour may have been
influenced by the request for users to take part in the FlickLing task.
User Sample: This study used a sample of twenty four (24) users, eighteen
(18) female and six (6) male ranging in age from 18 to 32 and only one stated
above 32. In addition, eleven out of the twenty four users were 1st year un-
dergraduate students, five 2nd year undergraduate students and the remaining
eight postgraduate students all studying at Manchester Metropolitan University
4 E. Vassilakaki, F. Johnson, R.J. Hartley, D. Randall
(MMU). Furthermore, twenty two users were English Native speakers and the
remaining two were Greek and Finnish native speakers. Eight were monolingual,
two bilinguals and sixteen users stated knowledge of one or more foreign lan-
guages (this includes the two bilingual users who stated knowledge of additional
foreign languages). In particular, the eight monolingual users were English na-
tive speakers, one of the bilinguals had Bengali as a native language, with an
excellent knowledge of English and a basic knowledge of French and Spanish
and the second user stated Urdu and English as native languages, a basic knowl-
edge for German, Dutch, Italian and Spanish, a good knowledge for French and
an excellent knowledge of Hindi. Finally, from the fourteen remaining users the
two non English native speakers stated an excellent knowledge of English, seven
users stated a basic and one very good knowledge of German, seven users stated
a basic, one good, one very good and one excellent knowledge of French, two
users stated a basic and one excellent knowledge of Italian, one stated a basic
knowledge of Dutch and finally five users stated a basic knowledge of Spanish
(see Table 1).
Language Basic Good Very Good Excellent
English 0 0 0 2
German 7 1 0 0
French 7 1 1 1
Italian 2 0 0 1
Dutch 1 0 0 0
Spanish 5 0 0 0
Table 1. Users’ Knowledge of non-native Languages
All the twenty four users stated their experience in searching for images. In
particular, five of them search for images very often, eight of them often, eight of
them sometimes and the remaining three users rarely. In addition, eight out of
the twenty four users have searched on the web for an image in a language other
than their native. These users employ mainly their own language skills (29%), an
online (24%) and hard copy dictionary (24%) consecutively, an online translator
(13%) and the help of a friend (10%). The main reason for doing so was for
school projects and assignments. The remaining sixteen users who are searching
images only in their native language explained that they “never needed to”.
In regards to previous experience with Flickr, eighteen out of the twenty four
have never used Flickr and explained “never heard of it”, ”never needed to”, “I
have not got around to using it” etc. Only six users have used Flickr in the past
mainly for searching images (57%), sharing images (29%) and “been directed to
it by Google” (14%).
Instructions: In iCLEF2008 study, the same written instructions were read
to all users yet it was observed that the majority of users would use the Help
instructions and ask for additional clarifications both regarding the task and
the various features of FlickLing. In the context of trying to investigate to what
Users’ Perceptions of Searching in FlickLing 5
extent instructions may affect or not users’ trust and confidence in FlickLing
and again retain an element of control in order to compare the findings, three
different texts of instructions were written and used in the iCLEF2009 study. In
particular, the users were divided in three different groups upon arrival. Each
group was provided with different level of instructions of the given task going
from ‘no’ to ‘detailed’ information regarding both the task and the functionality
of FlickLing’s features. In particular, the information given to the first group was
kept on a ‘need to know basis’ (see Appendix: Task Instructions). Users were
given the minimum information necessary to understand and complete the task
assigned. No explanation was given of the different fields and features employed
in FlickLing or how they could register and login. On the contrary, users were left
alone to discover the multilingual environment and form their search strategies
to retrieve the three given images.
The second group of users beside the general instructions concerning the
task were also informed about the languages that FlickLing supports and in
which the three images can be described (see Appendix: Task Instructions). In
addition, further information concerning the features of FlickLing were briefly
mentioned before describing them after login. This was a way of preparing (even
unconsciously) the users for what was to be described thoroughly afterwards,
on first login. Users were also instructed about the registration process but the
information given was kept to what each field stands for rather than how it
functions. In this context, information about the fields and features of FlickLing
was again provided after login but it was limited to what each field stands
for rather than disclosing their functionality. This element was left to users to
explore on their own while interacting with the interface.
The users in the third group were given the same instructions with the second
group with the difference that the functionality of the features in FlickLing was
also disclosed (see Appendix: Task Instructions). In particular, the languages
supported in FlickLing and in which the images could be searched across and
instructions concerning the registration process and how the various fields func-
tion were thoroughly described to these users. In addition, the fields found in
FlickLing and their functionality were also presented and thoroughly explained
to the users in a more technical way.
This group was given all the information available on the task and the Flick-
Ling interface, information focusing on both what is in there and how it func-
tions. These were explained in order to release users from the burden of exploring
it on their own. In addition, it was done to meet the purpose of providing dif-
ferent level of information between the three groups.
The assignment of users to each group was done based on users’ knowledge
of foreign languages. In particular, based on the number of foreign languages of
which they have some knowledge, the users were assigned to a specific group. This
was done in the context of equally assigning users with no knowledge of foreign
languages, knowledge of only one foreign language and knowledge of more than
one foreign languages across the three groups. In this way, homogeneity of the
6 E. Vassilakaki, F. Johnson, R.J. Hartley, D. Randall
user sample distributed to the three groups in regards of knowledge of languages
was enabled.
Data Collection: The same four methods, both qualitative and quanti-
tative (questionnaire, observation, retrospective thinking aloud and interviews)
were used for both studies. The adoption of the same methods was agreed on
the basis that these methods enabled us to draw insights on different aspects of
users’ characteristics and/or search behaviours. In addition, the defined order
of their implementation enabled us to gradually gain a better and a more com-
prehensive insight into users’ actions and thoughts and formulation of relevant
questions. Moreover, the way of combining the chosen methods contributed to
offset their weaknesses and draw on their strengths enabling a better and more
comprehensive understanding of users’ image seeking behaviour in multilingual
environments [5]. In particular, further description of the differences and simi-
larities between the way the four methods were designed and applied for both
studies is given below.
Questionnaires: In the iCLEF2008 study one questionnaire was adopted
and implemented inquiring on users’ native language, level of comprehension
for a series of foreign languages, previous experience in searching for images, in
searching across languages and in using Flickr and finally user demographics.
This questionnaire was also adopted in the iCLEF2009 study because it success-
fully enabled the definition of the user sample characteristics and users’ previous
experience in each of the sections of interest and the formulation of relevant ques-
tions during interviews [5]. In addition, one more question was included in this
questionnaire as a supplement regarding the means users employ to search across
languages. This question was derived from users’ comments during retrospective
thinking of the iCLEF2008 study when they were describing the way in which
they search across languages.
In addition to the first questionnaire, two more questionnaires were designed
and implemented inquiring on users’ trust and confidence which also emerged
from [4]. The second questionnaire consisted of a series of statements reflecting
on users’ trust and confidence in general when they are searching for images on
the web. A predefined six points scale (Don’t Know, Not at all, A little, So-So,
Very Much So, Totally) was provided for users to choose for each statement.
These statements did not exceed ten and they were implemented as a second
part of the first questionnaire. This was done both for presentation reasons and
in an attempt to relief users from the extra hassle of having to complete two
different questionnaires on arrival.
The third questionnaire had the same format as the second questionnaire.
It consisted of a series of statements with a predefined scale of six points an-
swers reflecting this time users’ trust and confidence in FlickLing and its various
features (monolingual/ multilingual mode, translations, hints etc). These state-
ments did not exceed twelve in number. These two questionnaires were used to
formulate an idea of each users’ trust and confidence both in general and in
particular in FlickLing and formulate relevant questions during the interviews.
Finally, all three questionnaires were employed to gather information on specific
Users’ Perceptions of Searching in FlickLing 7
aspects of this study. At the same time, they all contributed to the same goal to
explore users’ image seeking behaviour in multilingual environments.
Observation: This study adopted again the same observation sheet as de-
signed for the iCLEF2008 study [4]. It was kept the same because it proved
valuable for making notes on specific actions of users while performing the task
and formulating relevant questions during interviews. In addition, it enabled the
facilitator to note down specific user’s expressions while performing the task and
use those to engage users in conversation shading light on the specific actions
and thoughts of users reflecting on the reasons why [5].
Retrospective Thinking Aloud: This study also employed the retrospec-
tive thinking aloud method to reflect on users thoughts and action when search-
ing for images across languages. This method belongs to the Think Aloud Proto-
cols and it is a widely used method for usability testing of software and interfaces
[2]. It was adopted and applied in the same way and using the same screen cap-
ture software as in the iCLEF2008 study [4]. The same method was adopted
because during the iCLEf2008 study it enabled the facilitator to gain insights
on users’ information seeking behaviours, identifying the reasons why users were
behaving in this way and asking relevant questions during interviews [5].
Interviews: The same method of interview was also adopted in this study
as in the iCLEF2008 study [4] because it contributed to further clarifying users’
comments and sayings during retrospective thinking aloud, facilitator’s notes on
the observation sheet and users’ answers in the questionnaires. These interviews
were again semi-structured and kept as brief as possible because users were
already tired because of the retrospective thinking aloud and they were not
willing to spend more time on responding to additional questions [5].
In general, the same methods were adopted and applied in the same order
because they succeeded in gathering and verifying the validity and reliability
of the data on users’ image seeking behaviour in multilingual environments.
The weaknesses posed by every method was also considered and tackled by
employing a mixed methods research so that the weaknesses of one approach
were offset by the strengths of others [5]. Although the same methods was used
and implemented in the same order, minor changes and adjustments to the aim
of this year study was considered important in an attempt to focus more on users’
trust and confidence when searching for images in multilingual environments.
iCLEF2009 Study’s Experimental Procedure: This study consisted of
24 individual sessions which lasted on average one hour and a half each. Each user
on arrival was given the two questionnaires in the form of one to fill in. Based
on users’ number of foreign languages as they would define them in the first
questionnaire, the facilitator would assign them a specific group (e.g. 1st, 2nd or
3rd). In addition, a number (01, 02, 03 etc) was also assigned to each user based
on users’ arrival in order to ensure the anonymity of the data. Once the user
had the questionnaire completed, the facilitator read the relevant instructions
depending on which group the user was assigned. The users then started the task
while the facilitator observed and took notes. After the completion of the task,
users were asked to fill in the last questionnaire. Users were then asked, based
8 E. Vassilakaki, F. Johnson, R.J. Hartley, D. Randall
on the video capturing their search session, to describe their actions and reflect
their thoughts. At the same time, the facilitator took notes of users sayings
and scanned through the two last questionnaires focusing on specific patterns
regarding users’ trust and confidence. Finally, a brief interview followed enabling
the facilitator to ask questions based on the notes taken throughout the process
to further clarify users’ image seeking behaviour.
Data Analysis: The data gathered throughout the study were then ana-
lyzed to shed light on the different aspects of users’ actions and perceptions. In
particular, the questionnaires were processed in order to define the user sam-
ple’s characteristics (see Table 1) and users’ level of trust and confidence both in
general (see Table 5) and in FlickLing (see Table 6). In addition, the transcripts
based on retrospective thinking aloud and interviews were used to reveal users
justifications of their actions and thoughts as well as their answers in the two
questionnaires on trust and confidence. Furthermore, the videos based on users’
image seeking behaviour were used to identify the specific actions of users and
reflect back to users justifications and thoughts recorded during retrospective
thinking aloud.
The transcripts based on retrospective thinking aloud only were further an-
alyzed in regards to specific elements such as: a) users’ switching between the
modes (monolingual/ multilingual) throughout the three images (see Table 3
and 4), b) users’ thinking about languages throughout the task (see Table 2),
c) what triggered the users to think about the language of each image, d) the
reasons why they were giving up on each image, e) why they were using the
hints each time for each image and their actions after taking each hint, f) users’
comments regarding trust and confidence. In addition, the users’ explanations of
their answers in the two questionnaires on trust and confidence during interviews
were also identified and grouped. In particular, for each user specific comments
were identified regarding a) users’ trust in general, b) users’ trust in FlickLing
and c) whether their trust remained the same or changed throughout the task.
The same was done for users’ comments regarding confidence.
3 Findings
The small scale iCLEF2008 study laid the foundations and revealed the reasons
why users were switching between the two modes, the extend to which they were
thinking about languages and paying attention to translations consecutively. The
current study aims to extend and reveal users’ perceptions of the tasks involved
when searching across languages based on these behaviours and the expressed
justifications. In the context of a preliminary analysis and interpretation of the
data collected, we can report the following in regards to users’ perceptions when
searching for known, non-annotated images across languages. In particular, data
collected from the retrospective thinking analysis, the questionnaires on trust
and confidence and the videos recorded will be presented in an attempt to iden-
tify users’ perceptions when searching across languages.
Users’ Perceptions of Searching in FlickLing 9
One important element in the process of identifying users’ perceptions is the
extent to which they were thinking about languages throughout the three images.
In particular (see Table 2), on the first image thirteen users did not think about
languages before going for the first hint, six of whom were from the first group,
three from the second and four from the third group. In addition, five users were
thinking about languages from the beginning of the task, one of whom belonged
to the first group and two to the second and third consecutively. Furthermore, six
users started the task without thinking about languages but at some point before
taking the hint started realizing that they needed to search across languages. It
is also interesting to note that nineteen out of twenty four users started thinking
about languages after taking the hint. Four users misinterpreted the hint on
language and used it as a keyword. One user did not use the hints at all.
In terms of the second image, eight users started searching without thinking
about languages, five of whom were from the first group, one from the second and
two from the third. In addition, ten users were thinking about languages, three
of whom from the first and second group consecutively and four from the third.
Five users started without thinking about languages but at some point before
going for the hint realized that they would need to search across languages, four
of whom were from the second and one from the third group. Only one user
started with a perception of languages but at some point he realized that he was
actually trying to search only in English and he belonged to the third group.
On the second image, non of the users misinterpreted the hint and used it as a
keyword and again one user did not make use of the hints.
In regards to the last image, ten users again started without thinking about
languages, four of whom were from the first, five from the second and one from
the third group. In contrast twelve were thinking about languages, four of whom
were from the first group, three from the second and five from the third. Two
users started thinking about languages after spending sometime searching for
the image but before going for the hint, these belonged to the third group. On
the third image, again no users misintepreted the hint and this time four of them
did not make use of the hints.
It is interesting to point out that only three users were thinking about lan-
guages and trying to search for the three images across languages from the
beginning and throughout the task. The rest of the users either did not think
about languages throughout the three images and only the hint triggered them
to think about it or at some point either on the first or the second image started
to understand that they would need to search across languages.
Of particular interest is some of users’ remarks regarding the task of searching
across languages:
“ but I didn’t know what else to write or what I should do, because I didn’t know
any languages and things like searching in other languages”,
“even though I am writing in Dutch it doesn’t necessarily mean that it will bring
up different images or any more images”,
“but I didn’t think that it would affect, especially that it would affect the search
that you would use because you would have Dutch as a language to search so, I
10 E. Vassilakaki, F. Johnson, R.J. Hartley, D. Randall
just kept on typing English search”,
“ok it’s making me do more that just search, it’s making me use other languages
for me to search and then I was happier because I could search a Spanish word
which I obviously don’t speak”,
“searching in different languages seems, sounds like it would be much harder
than searching in English”,
“I just found that I had to think about a lot of things like you have to find the
image and try to translate it as well as find out if it is in English...it took me a
while to grasp, what I was meant to be doing for this one, like I realized it had to
be in a different language the image but actually I was looking for keywords...”.
Another aspect of users perception of searching across languages is the use
of the two modes (monolingual/multilingual) (see Table 3 and Table 4). Four
out of twenty four users stayed and used only one mode throughout the three
images. In particular, three used only the monolingual mode and one only the
multilingual mode throughout the three images. In terms of the rest twenty
users, on the first image ten were switching between the two modes, eight used
only the monolingual mode and the remaining two only the multilingual. On the
second image, sixteen users were again switching between the two modes and
four used only the multilingual mode. On the last image, seventeen users were
switching between the modes, two were using only the monolingual and one the
multilingual mode.
In the context of investigating users’ perceptions of the tasks involved when
searching for images across languages, users’ degree of trust in applications
and confidence in their searching and language skills were also monitored both
with the two questionnaires employed, the users’ comments during retrospective
thinking aloud and the individual interviews. In regards to the second ques-
tionnaire (see Table 5), the majority of the twenty four users have “Not at all”
confidence in their language skills to search for images across languages on the
web (S9), trust “A little” online translators to help them translate a text (S3),
for languages of which they have some knowledge, they trust “A little” provided
translations for searching on the web (S5) and they trust “A little” for use the
search terms suggested to them by search engines on the web (S6).
In addition, the majority of the twenty four users trust “So-so”1 applications
on the web to help them do the things they want to do (S1), the results that
search engines on the web give (S2), and they have “So-so” confidence in their
personal skills and abilities to search for images on the web (S8). Furthermore,
the majority of the twenty four users answered equally that for languages they
do not understand, they trust “A little” and “So-so” provided translations for
searching on the web (S4). The majority again equally stated that they trust “So-
so” and “Very much so” Help instructions to understand the use of interfaces
(S7). Finally, the majority of users who have experience in searching on the
1
A predefined six points scale (Don’t know!, Not at all, A little, So-so, Very much so,
Totally) was given to users to choose for each statement (see section 2: Question-
naires).
Users’ Perceptions of Searching in FlickLing 11
web across languages stated that they have “Very much so” confidence when
searching images across languages because of their relevant experience.
In terms of the third questionnaire (see Table 6), the majority of the twenty
four users stated they “Don’t know” for the statement “I trusted the help in-
structions to guide me through FlickLing (S8)”. In addition, the majority had
“Not at all” condence in their language skills to search for the images across lan-
guages in FlickLing (S11). The majority of users had “A little” condence in their
personal skills to search for the images in FlickLing (S10) and in their experience
that they were gaining in FlickLing for searching the images across languages
(S12). In addition, the majority “So-so” trusted the results that FlickLings both
monolingual and multilingual mode gave them (S1 and S2), trusted “So-so” the
translations FlickLing gives for languages they do not know (S4) and they are
“So-so” condent that FlickLing helped them search across languages for the im-
ages (S9). Finally, the majority trusted “Very much so” FlickLing to translate
their query (S3), the translations FlickLing gives for languages they understand
(S5), FlickLing to suggest search terms (S6) and the hints provided by FlickLing
to search for the images (S7).
The second goal of the analysis would be to identify the reasons why users
were or were not thinking about languages on each image and throughout the
task. In addition, the reasons why the four users used only one mode for searching
across languages and why the rest were either switching between the two or using
just one mode while searching for the three images are to be further analyzed.
In terms of users’ trust and confidence, further analysis is intended to combine
the data gathered from questionnaires, retrospective thinking aloud and inter-
views to shed light on the reasons why users provided the following answers to
the 2nd and 3rd Questionnaires and determine whether trust and confidence are
factors affecting users’ perceptions of the tasks involved when searching across
languages. Finally, to meet our aim to gain insight into users search behaviour
and use of translations further analysis will be carried out on data such as users’
comments regarding the three given images, their comments regarding their lan-
guage skills, and their interpretation and usage of the translation mechanism. It
appears from our preliminary findings that there is variation in our users per-
ceptions of searching for images across languages and their approach to using
translations and that this occurs regardless of the amount or type of help or
guidance given. Further analysis of the data collected on users perceptions of
the task and their explanation for their actions may provide further insight into
understanding the user in the design and development of multilingual retrieval
systems.
4 Conclusion
This study aimed at exploring users’ image seeking behaviour when searching for
known, non-annotated images across languages in FlickLing. In particular, this
paper aimed to demonstrate the value in focusing on user’s trust and confidence
in the investigation of image seeking behaviour to reveal users’ perceptions of
12 E. Vassilakaki, F. Johnson, R.J. Hartley, D. Randall
the tasks involved when searching across languages. The task assigned to our 24
users was to search for the three images given to them after first login. These
images were described in Dutch, German and Spanish consecutively. In this con-
text, four different methods, both qualitative and quantitative (questionnaires,
observation, retrospective thinking aloud and interviews) were employed to meet
its research question.
Preliminary findings regarding users’ switching between the two modes, users’
thinking of languages and their answers to the two questionnaires on trust and
confidence were only reported due to the inability to process the large amount
of data collected in such a short period of time. Nonetheless, it appears from our
preliminary findings that there is variation in our users perceptions of searching
for images across languages and their approach to using translations and that
this occurs regardless of the amount or type of help or guidance given. Further
analysis of the data will provide a better understanding of users’ behaviour
and perceptions so as to build cross-language information retrieval systems that
better serve users.
References
1. Gonzalo, J., Clough, P., Karlgren, J.: Overview of iCLEF 2008: Search Log Analysis
for Multilingual Image Retrieval. In Peters et al (eds ): Evaluating Systems for Mul-
tilingual and Multimodal Information Access 9th Workshop of the Cross-Language
Evaluation Forum, CLEF 2008, Aarhus, Denmark, September 17-19, 2008, Revised
Selected Papers, Springer LNCS.
2. Haak, M., de Jong, M., Schellens, P.J.: Retrospective vs. Concurrent Think-aloud
Protocols: Testing the Usability of an Online Library Catalogue. Behaviour & In-
formation Technology 22, 339–351 (2003).
3. Petrelli, D., Beaulieu, M., Sanderson, M., Demetriou, G., Herring, P., Hansen, P.:
Observing Users, Designing Clarity: A Case Study on the User-centered Design of
a Cross-language Information Retrieval System. JASIST 55, 923–934 (2004).
4. Vassilakaki, E., Johnson, F., Hartley, R.J., Randall, D.: A Study of Users Image
Seeking Behaviour in Flickling. In Peters et al (eds ): Evaluating Systems for Mul-
tilingual and Multimodal Information Access 9th Workshop of the Cross-Language
Evaluation Forum, CLEF 2008, Aarhus, Denmark, September 17-19, 2008, Revised
Selected Papers, Springer LNCS.
5. Vassilakaki, E., Johnson, F., Hartley, R.J. and Randall D., 2009. Users Image Seek-
ing Behaviour in Multilingual Environments: experience in combining qualitative
and quantitative data. In International Conference on Qualitative and Quantitative
Methods in Libraries, Chania, Greece, 26-29 May 2009.
Appendix: Task Instructions
This study used three different groups of users. These users received different
levels of information about the task to be carried out and the interface to be
employed. The full text of the instructions provided to each group of users (1st,
2nd and 3rd) is outlined here. The reader can find a detailed description of these
in the relevant chapter.
Users’ Perceptions of Searching in FlickLing 13
First Group Task Instructions
– The organizers would like to thank you for sparing some of your time to
participate in this study.
– I would like you first to fill in this short questionnaire on basic background
knowledge.
– This study uses FlickLing, a multilingual interface for searching images
across languages. You are asked to use this interface to search the first three
images given to you after login. The images given can be described in any
of the following six (6) languages: English, German, Dutch, French, Italian
and Spanish. You will not know in advance in which of these languages the
images provided are described. Therefore, you need to use the features sup-
ported by the FlickLing interface in order to search the three (3) images
across languages.
– At this point, the organizers feel the need to remind you that it is the system
and not you that is being evaluated and that you should perform as usual,
spending as much time as you would have normally spent on searching.
– You will be asked to fill in another short questionnaire after completing the
searches for the three images.
– Your search session will be system captured and played back to you. You
will then be asked to describe your thoughts and actions while searching for
the three images. The facilitator will encourage you to keep on describing
your thoughts and actions when going to silence. A digital recorder will be
used to tape your description.
– At the end, a short length interview will take place between you and the
facilitator that will be also taped.
– I would now ask you to register as a member of the MMU team and start
searching for the images.
Second Group Task Instructions
– The organizers would like to thank you for sparing some of your time to
participate in this study.
– I would like you first to fill in this short questionnaire on basic background
knowledge.
– This study uses FlickLing, a multilingual interface for searching images
across languages. You are asked to use this interface to search the first three
images given to you after login. The images given can be described in any
of the following six (6) languages: English, German, Dutch, French, Italian
and Spanish. You will not know in advance in which of these languages the
images provided are described. Therefore, you need to use the features (Help
instructions, monolingual/multilingual mode, translations, suggested terms,
give up button and hints) supported by FlickLing interface in order to search
the three (3) images across languages.
– At this point, the organizers feel the need to remind you that it is the system
and not you that is being evaluated and that you should perform as usual,
spending as much time as you would have normally spent on searching.
14 E. Vassilakaki, F. Johnson, R.J. Hartley, D. Randall
– You will be asked to fill in another short questionnaire after completing the
searches for the three images.
– Your search session will be system captured and played back to you. You
will then be asked to describe your thoughts and actions while searching for
the three images. The facilitator will encourage you to keep on describing
your thoughts and actions when going to silence. A digital recorder will be
used to tape your description.
– At the end, a short length interview will take place between you and the
facilitator that will be also taped.
– I would now guide you through registration for login to the FlickLing. You
need to click on ‘register’ button. First, you need to type in an ‘email account’
and provide a ‘password’. Choose your ‘native language’ from the menu and
the ‘language of the interface’ again from the ones in the relevant menu.
Determine your team from the scroll down menu as ‘MMU’ team. Choose
from the menu the ‘level of comprehension’ for the six languages supported
by FlickLing and then click on ‘ok’.
– You are now ready to use the FlickLing interface and search for the three
images. As you can see on your left, you have the first image. You can
click on it in order to bring it forward and see it in detail. Two modes are
provided, the monolingual and multilingual. On the monolingual, you write
in one language and you get results only in the language of your query.
On multilingual mode, you write your query in one language and you can
determine in which languages of the six supported by FlickLing you can
get results. On the right side of the search box, you will get each time the
translations. Each time, you can cross out or provide a new translation for
each desired language. You can also use the suggested terms provided by the
system and the tags found in the images of the results. At any time, you can
give up on a photo or make use of the hints provided by FlickLing.
– You can now start searching for the three images.
Third Group Task Instructions
– The organizers would like to thank you for giving some of your time to
participate in this study.
– I would like you first to fill in this short questionnaire on basic background
knowledge.
– This study uses FlickLing, a multilingual interface for searching images
across languages. You are asked to use this interface to search the first three
images given to you after login. The images given can be described in any
of the following six (6) languages: English, German, Dutch, French, Italian
and Spanish. You will not know in advance in which of these languages the
images provided are described. Therefore, you need to use the features (Help
instructions, monolingual/multilingual mode, translations, suggested terms,
give up button and hints) supported by FlickLing interface in order to search
the three (3) images across languages.
Users’ Perceptions of Searching in FlickLing 15
– At this point, the organizers feel the need to remind you that it is the system
and not you that is being evaluated and that you should perform as usual,
spending as much time as you would have normally spent on searching.
– You will be asked to fill in another short questionnaire after completing the
searches for the three images.
– Your search session will be system captured and played back to you after
completing the searches. You will then be asked to describe your thoughts
and actions while searching for the three images. The facilitator will en-
courage you to keep on describing your thoughts and actions when going to
silence. A digital recorder will be used to tape your description.
– At the end, a short length interview will take place between you and the
facilitator that will be also taped.
– I would now guide you through registration for login to FlickLing. You need
to click on ‘register’ button. First, you need to type in an ‘email account’
and provide a ‘password’. Choose your ‘native language’ from the menu and
the ‘language of the interface’ again from the ones in the relevant menu.
Determine your team from the scroll down menu as ‘MMU’ team. Choose
from the menu the ‘level of comprehension’ for the six languages supported
by FlickLing and then click on ‘ok’.
– You are now ready to use the FlickLing interface and search for the three
images. As you can see on your left side, you have the first image. You can
click on it in order to bring it forward and see it in detail. By clicking on
ok button, you go back to your previous screen. Two modes are provided,
the monolingual and multilingual. In the monolingual, you search in one lan-
guage and you get results only in the language of your query. In multilingual
mode, you can determine in which language you write the query from the
drop down menu and in which languages you get results by crossing out
the relevant language buttons. If you do not cross out a language button
then you get translations and consecutively results in all six languages. On
the right side of the search box, you will get with each query the relevant
translations. The system retrieves at the same time your search terms and
the provided translations, so there is no need to retype the translations in
the search box. Each time, you can cross out or provide a new translation
for each language by clicking on the translation. You can use the suggested
terms provided by the system after each search underneath your image by
clicking on the term and selecting one of the options provided (include only
this word, include this word to the rest of the query, include the translation
of this word to the query). You can also use the tags found in the images
of the results again by clicking on the tag and selecting one of the options
provided (include only this word, include this word to the rest of the query,
include the translation of this word to the query). The results are presented
underneath your image and are listed in pages. The number of the pages
varies according to the number of results. At any time, you can give up on
a photo by clicking on the give up button found on the bottom side of the
image. After clicking on the give up button, you can either make use of the
hints provided by FlickLing or decide to skip this image and go for the next
16 E. Vassilakaki, F. Johnson, R.J. Hartley, D. Randall
one. The system provides a number of hints, disclosing each time the lan-
guage in which the image is described and the terms that retrieve this image.
You can use as many hints as you wish by repeating the same steps (giving
up and going for the hint button). You can now start searching for the three
images.
Users’ Perceptions of Searching in FlickLing 17
Appendix: Analysis Tables
Group User ID TAL
Images 1st 2nd 3rd
BH AH BH AH BH AH
G1 01 N Y N Y N Y
G1 04 N Y N Y N Y
G1 07 N N Y Y N Y
G1 10 N Y N Y Y Y
G1 13 N Y N Y N Y
G1 16 Y Y Y Y Y Y
G1 21 N/Y Y Y Y Y Y
G1 23 N N N Y Y Y
G2 02 N Y N Y N Y
G2 05 Y Y Y Y N Y
G2 09 N/Y Y Y Y Y N/A
G2 11 N Y Y Y N Y
G2 14 N Y N/Y Y N Y
G2 17 Y Y N/Y Y Y Y
G2 19 N/Y Y N/Y Y Y Y
G2 22 N/Y Y N/Y N/A N N/A
G3 03 Y Y Y Y Y Y
G3 06 N Y Y Y N/Y Y
G3 08 N Y Y Y Y N/A
G3 12 N N Y/N Y Y Y
G3 15 N/Y Y N Y N/Y Y
G3 18 N/Y N N/Y Y N N/A
G3 20 Y Y Y Y Y Y
G3 24 N N/A N Y Y Y
Table 2. Users’ Thinking About Lan-
guages
Table 2
TAL= Thinking About Languages
BH= Before taking Hint
AH= After taking Hint
Y= Yes
N= No
N/Y= No at first and then Yes
Y/N= Yes at first and then No
N/A= Non Applicable
18 E. Vassilakaki, F. Johnson, R.J. Hartley, D. Randall
Group User ID SBM
Images 1st 2nd 3rd
G1 01 N N N
G1 04 N Y Y
G1 07 N Y Y
G1 10 N Y Y
G1 13 N N N
G1 16 N Y Y
G1 21 Y Y Y
G1 23 Y Y Y
G2 02 Y N Y
G2 05 Y Y Y
G2 09 N Y Y
G2 11 Y Y Y
G2 14 Y Y Y
G2 17 Y N Y
G2 19 N N N
G2 22 N Y N
G3 03 N N Y
G3 06 N N N
G3 08 Y N N
G3 12 N Y Y
G3 15 Y Y Y
G3 18 N Y N
G3 20 Y Y Y
G3 24 N Y Y
Table 3. Users’ Switching Between Modes
Table 3
SBM= Switching Between Modes
Y= Yes
N= No
Users’ Perceptions of Searching in FlickLing 19
Group User ID SBM
Images 1st 2nd 3rd
G1 01 OMN OMN OMN
G1 04 OMN M&M M&M
G1 07 OMN M&M M&M
G1 10 OMN M&M M&M
G1 13 OMN OMN OMN
G1 16 OMN M&M M&M
G1 21 M&M M&M M&M
G1 23 M&M M&M M&M
G2 02 M&M OML M&M
G2 05 M&M M&M M&M
G2 09 OMN M&M M&M
G2 11 M&M M&M M&M
G2 14 M&M M&M M&M
G2 17 M&M OML M&M
G2 19 OMN OMN OMN
G2 22 OML M&M OMN
G3 03 OML OML M&M
G3 06 OML OML OML
G3 08 M&M OML OML
G3 12 OMN M&M M&M
G3 15 M&M M&M M&M
G3 18 OMN M&M OMN
G3 20 M&M M&M M&M
G3 24 OMN M&M M&M
Table 4. Users Modes’ Usage
Table 4
SBM= Switching Between Modes
M&M= Both Monolingual and Multilingual Modes
OMN= Only Monolingual Mode
OML= Only Multilingual Mode
20 E. Vassilakaki, F. Johnson, R.J. Hartley, D. Randall
Answers S1 S2 S3 S4 S5 S6 S7 S8 S9 S10
Don’t know! 0 0 2 4 3 0 2 0 1 1
Not at all 0 0 2 1 0 2 2 0 7 0
A little 4 4 9 7 9 8 5 1 5 3
So-so 12 14 6 7 5 7 7 10 4 1
Very much so 7 6 5 5 7 7 7 9 5 5
Totally 1 0 0 0 0 0 1 4 2 0
Table 5. Users’ Answers to the 2nd Questionnaire
Table 5
S= Statement
S1= In general, I trust applications on the Web to help me do the things that I
want to do.
S2= In general, I trust the results that search engines on the web give me.
S3= In general, I trust online translators to help me translate a text.
S4= In general, for languages I do not understand, I trust provided translations
for searching on the web.
S5= In general, for languages of which I have some knowledge, I trust provided
translations for searching on the web.
S6= In general, I trust for use the search terms suggested to me by search en-
gines on the web.
S7= In general, I trust Help instructions to understand the use of interfaces.
S8= In general, I have confidence in my personal skills and abilities to search
for images on the web.
S9= In general, I have confidence in my language skills to search for images
across languages on the web.
S10= In general, I have confidence when searching images across languages be-
cause of my relevant experience.
Answers S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12
Don’t know! 1 3 2 2 2 1 0 10 2 0 1 3
Not at all 2 2 3 2 0 1 0 2 4 5 12 1
A little 5 4 0 1 4 3 4 4 4 10 4 9
So-so 10 8 9 10 5 8 5 3 7 5 3 6
Very much so 5 7 10 6 9 10 11 5 6 2 4 4
Totally 1 0 0 1 2 1 4 0 0 2 0 1
Table 6. Users’ Answers to the 3rd Questionnaire
Users’ Perceptions of Searching in FlickLing 21
Table 6
S= Statement
S1= I trusted the results that FlickLing’s monolingual mode gave me.
S2= I trusted the results that FlickLing’s multilingual mode gave me.
S3= I trusted FlickLing to translate my query.
S4= I trusted the translations FlickLing gives for languages I do not know.
S5= I trusted the translations FlickLing gives for languages I understand.
S6= I trusted FlickLing to suggest search terms.
S7= I trusted the hints provided by FlickLing to search for the images.
S8= I trusted the help instructions to guide me through FlickLing
S9= I am confident that FlickLing helped me search across languages for the
images.
S10= I had confidence in my personal skills to search for the images in FlickLing.
S11= I had confidence in my language skills to search for the images across lan-
guages in FlickLing.
S12= I had confidence in the experience that I was gaining in FlickLing for
searching the images across languages.