This study aims to explore users' image seeking behaviour when searching for known, non-annotated images across languages in FlickLing. In particular, the investigation reported in this paper aims to demonstrate the value in focusing on user's trust and con dence in the exploration of seeking behaviour to reveal users' perceptions of the tasks involved when searching across languages. The task assigned to our 24 users was to search for three speci c images described in Dutch, German and Spanish consecutively. In this context, four di erent methods, both qualitative and quantitative (questionnaires, observation, retrospective thinking aloud and interviews) were employed. Preliminary nding regarding users' switching between the two modes, users' thinking of languages and their answers to the two questionnaires on trust and condence are only reported here. It appears from our preliminary ndings that there is variation in our users perceptions of searching for images across languages and their approach to using translations and that this occurs regardless of the amount or type of help or guidance given.
The development of e cient and e ective Cross Language Information Retrieval (CLIR) systems depends not only on the implementation of the various techniques for organizing linguistic resources but also on users' adaptation and usage. The interactive CLEF (iCLEF) track in Cross Language Evaluation Forum (CLEF) provides the participants with the necessary environment to run interactive experiments and focus on users' image seeking behaviour in multilingual environments. This is our second year participating in iCLEF track focusing on users' behaviour while searching across languages in FlickLing.
In the iCLEF2008 paper [
The iCLEF2008 small study [
The iCLEF2008 study also revealed some insight into users' trust in
searching in a multilingual environment such as FlickLing and a possible relationship
between trust and the use of translations, to search for the images in di erent
languages. In addition, it provided some indication that users' con dence has an
impact when searching for images across languages and that con dence may or
may not be a ected by users' knowledge of foreign languages. In this context, a
follow up study was designed for iCLEF2009 drawing from the lessons learned
from the iCLEF2008 paper [
The remainder of this paper is structured as follows: a description of the methods employed and the way the study was carried out are outlined in section 2. An initial analysis of the ndings is given in sections 3. Finally, we conclude by summarizing the current indications of users' behaviour in section 4.
In this section, further details on the test object, the task, the users and the methods employed are illustrated.
Test Object: This study used the FlickLing multilingual search engine for
images as developed and provided by iCLEF organizers [
Task: As in the iCLEF2008 paper, our users were asked to search for the rst three known, non annotated images given after login in FlickLing. The users did not know in advance in which of the six languages supported by FlickLing (English, German, Dutch, French, Italian and Spanish) the three images were described. As a result, users had to use all the features and di erent modes (monolingual/ multilingual) provided by FlickLing to search the three images across languages.
As opposed to last year's study, the three images were pre-selected and
presented to all 24 users so that the seeking behaviour and the thoughts recorded
for each user referred to the experience of searching for the same three images.
The three images were described in Dutch, German and Spanish consecutively.
Drawing from [
In addition, the images selected had both a `heading' and at least three `tags'
describing them in order to provide our users with as many entry points to
these images as possible. This emerged from the iCFLEF2008 study [
In this context, it should be also noted that during this year's study, words
like task, or study were carefully avoided both from the written instructions read
to the users and during the process of recruitment. Again drawing from [
User Sample: This study used a sample of twenty four (24) users, eighteen (18) female and six (6) male ranging in age from 18 to 32 and only one stated above 32. In addition, eleven out of the twenty four users were 1st year undergraduate students, ve 2nd year undergraduate students and the remaining eight postgraduate students all studying at Manchester Metropolitan University (MMU). Furthermore, twenty two users were English Native speakers and the remaining two were Greek and Finnish native speakers. Eight were monolingual, two bilinguals and sixteen users stated knowledge of one or more foreign languages (this includes the two bilingual users who stated knowledge of additional foreign languages). In particular, the eight monolingual users were English native speakers, one of the bilinguals had Bengali as a native language, with an excellent knowledge of English and a basic knowledge of French and Spanish and the second user stated Urdu and English as native languages, a basic knowledge for German, Dutch, Italian and Spanish, a good knowledge for French and an excellent knowledge of Hindi. Finally, from the fourteen remaining users the two non English native speakers stated an excellent knowledge of English, seven users stated a basic and one very good knowledge of German, seven users stated a basic, one good, one very good and one excellent knowledge of French, two users stated a basic and one excellent knowledge of Italian, one stated a basic knowledge of Dutch and nally ve users stated a basic knowledge of Spanish (see Table 1).
All the twenty four users stated their experience in searching for images. In particular, ve of them search for images very often, eight of them often, eight of them sometimes and the remaining three users rarely. In addition, eight out of the twenty four users have searched on the web for an image in a language other than their native. These users employ mainly their own language skills (29%), an online (24%) and hard copy dictionary (24%) consecutively, an online translator (13%) and the help of a friend (10%). The main reason for doing so was for school projects and assignments. The remaining sixteen users who are searching images only in their native language explained that they \never needed to".
In regards to previous experience with Flickr, eighteen out of the twenty four have never used Flickr and explained \never heard of it", "never needed to", \I have not got around to using it" etc. Only six users have used Flickr in the past mainly for searching images (57%), sharing images (29%) and \been directed to it by Google" (14%).
Instructions: In iCLEF2008 study, the same written instructions were read to all users yet it was observed that the majority of users would use the Help instructions and ask for additional clari cations both regarding the task and the various features of FlickLing. In the context of trying to investigate to what extent instructions may a ect or not users' trust and con dence in FlickLing and again retain an element of control in order to compare the ndings, three di erent texts of instructions were written and used in the iCLEF2009 study. In particular, the users were divided in three di erent groups upon arrival. Each group was provided with di erent level of instructions of the given task going from `no' to `detailed' information regarding both the task and the functionality of FlickLing's features. In particular, the information given to the rst group was kept on a `need to know basis' (see Appendix: Task Instructions). Users were given the minimum information necessary to understand and complete the task assigned. No explanation was given of the di erent elds and features employed in FlickLing or how they could register and login. On the contrary, users were left alone to discover the multilingual environment and form their search strategies to retrieve the three given images.
The second group of users beside the general instructions concerning the task were also informed about the languages that FlickLing supports and in which the three images can be described (see Appendix: Task Instructions). In addition, further information concerning the features of FlickLing were brie y mentioned before describing them after login. This was a way of preparing (even unconsciously) the users for what was to be described thoroughly afterwards, on rst login. Users were also instructed about the registration process but the information given was kept to what each eld stands for rather than how it functions. In this context, information about the elds and features of FlickLing was again provided after login but it was limited to what each eld stands for rather than disclosing their functionality. This element was left to users to explore on their own while interacting with the interface.
The users in the third group were given the same instructions with the second group with the di erence that the functionality of the features in FlickLing was also disclosed (see Appendix: Task Instructions). In particular, the languages supported in FlickLing and in which the images could be searched across and instructions concerning the registration process and how the various elds function were thoroughly described to these users. In addition, the elds found in FlickLing and their functionality were also presented and thoroughly explained to the users in a more technical way.
This group was given all the information available on the task and the FlickLing interface, information focusing on both what is in there and how it functions. These were explained in order to release users from the burden of exploring it on their own. In addition, it was done to meet the purpose of providing different level of information between the three groups.
The assignment of users to each group was done based on users' knowledge of foreign languages. In particular, based on the number of foreign languages of which they have some knowledge, the users were assigned to a speci c group. This was done in the context of equally assigning users with no knowledge of foreign languages, knowledge of only one foreign language and knowledge of more than one foreign languages across the three groups. In this way, homogeneity of the user sample distributed to the three groups in regards of knowledge of languages was enabled.
Data Collection: The same four methods, both qualitative and
quantitative (questionnaire, observation, retrospective thinking aloud and interviews)
were used for both studies. The adoption of the same methods was agreed on
the basis that these methods enabled us to draw insights on di erent aspects of
users' characteristics and/or search behaviours. In addition, the de ned order
of their implementation enabled us to gradually gain a better and a more
comprehensive insight into users' actions and thoughts and formulation of relevant
questions. Moreover, the way of combining the chosen methods contributed to
o set their weaknesses and draw on their strengths enabling a better and more
comprehensive understanding of users' image seeking behaviour in multilingual
environments [
Questionnaires: In the iCLEF2008 study one questionnaire was adopted
and implemented inquiring on users' native language, level of comprehension
for a series of foreign languages, previous experience in searching for images, in
searching across languages and in using Flickr and nally user demographics.
This questionnaire was also adopted in the iCLEF2009 study because it
successfully enabled the de nition of the user sample characteristics and users' previous
experience in each of the sections of interest and the formulation of relevant
questions during interviews [
In addition to the rst questionnaire, two more questionnaires were designed
and implemented inquiring on users' trust and con dence which also emerged
from [
The third questionnaire had the same format as the second questionnaire. It consisted of a series of statements with a prede ned scale of six points answers re ecting this time users' trust and con dence in FlickLing and its various features (monolingual/ multilingual mode, translations, hints etc). These statements did not exceed twelve in number. These two questionnaires were used to formulate an idea of each users' trust and con dence both in general and in particular in FlickLing and formulate relevant questions during the interviews. Finally, all three questionnaires were employed to gather information on speci c aspects of this study. At the same time, they all contributed to the same goal to explore users' image seeking behaviour in multilingual environments.
Observation: This study adopted again the same observation sheet as
designed for the iCLEF2008 study [
Retrospective Thinking Aloud: This study also employed the
retrospective thinking aloud method to re ect on users thoughts and action when
searching for images across languages. This method belongs to the Think Aloud
Protocols and it is a widely used method for usability testing of software and interfaces
[
Interviews: The same method of interview was also adopted in this study
as in the iCLEF2008 study [
In general, the same methods were adopted and applied in the same order
because they succeeded in gathering and verifying the validity and reliability
of the data on users' image seeking behaviour in multilingual environments.
The weaknesses posed by every method was also considered and tackled by
employing a mixed methods research so that the weaknesses of one approach
were o set by the strengths of others [
iCLEF2009 Study's Experimental Procedure: This study consisted of 24 individual sessions which lasted on average one hour and a half each. Each user on arrival was given the two questionnaires in the form of one to ll in. Based on users' number of foreign languages as they would de ne them in the rst questionnaire, the facilitator would assign them a speci c group (e.g. 1st, 2nd or 3rd). In addition, a number (01, 02, 03 etc) was also assigned to each user based on users' arrival in order to ensure the anonymity of the data. Once the user had the questionnaire completed, the facilitator read the relevant instructions depending on which group the user was assigned. The users then started the task while the facilitator observed and took notes. After the completion of the task, users were asked to ll in the last questionnaire. Users were then asked, based on the video capturing their search session, to describe their actions and re ect their thoughts. At the same time, the facilitator took notes of users sayings and scanned through the two last questionnaires focusing on speci c patterns regarding users' trust and con dence. Finally, a brief interview followed enabling the facilitator to ask questions based on the notes taken throughout the process to further clarify users' image seeking behaviour.
Data Analysis: The data gathered throughout the study were then analyzed to shed light on the di erent aspects of users' actions and perceptions. In particular, the questionnaires were processed in order to de ne the user sample's characteristics (see Table 1) and users' level of trust and con dence both in general (see Table 5) and in FlickLing (see Table 6). In addition, the transcripts based on retrospective thinking aloud and interviews were used to reveal users justi cations of their actions and thoughts as well as their answers in the two questionnaires on trust and con dence. Furthermore, the videos based on users' image seeking behaviour were used to identify the speci c actions of users and re ect back to users justi cations and thoughts recorded during retrospective thinking aloud.
The transcripts based on retrospective thinking aloud only were further analyzed in regards to speci c elements such as: a) users' switching between the modes (monolingual/ multilingual) throughout the three images (see Table 3 and 4), b) users' thinking about languages throughout the task (see Table 2), c) what triggered the users to think about the language of each image, d) the reasons why they were giving up on each image, e) why they were using the hints each time for each image and their actions after taking each hint, f) users' comments regarding trust and con dence. In addition, the users' explanations of their answers in the two questionnaires on trust and con dence during interviews were also identi ed and grouped. In particular, for each user speci c comments were identi ed regarding a) users' trust in general, b) users' trust in FlickLing and c) whether their trust remained the same or changed throughout the task. The same was done for users' comments regarding con dence. 3
The small scale iCLEF2008 study laid the foundations and revealed the reasons why users were switching between the two modes, the extend to which they were thinking about languages and paying attention to translations consecutively. The current study aims to extend and reveal users' perceptions of the tasks involved when searching across languages based on these behaviours and the expressed justi cations. In the context of a preliminary analysis and interpretation of the data collected, we can report the following in regards to users' perceptions when searching for known, non-annotated images across languages. In particular, data collected from the retrospective thinking analysis, the questionnaires on trust and con dence and the videos recorded will be presented in an attempt to identify users' perceptions when searching across languages.
One important element in the process of identifying users' perceptions is the extent to which they were thinking about languages throughout the three images. In particular (see Table 2), on the rst image thirteen users did not think about languages before going for the rst hint, six of whom were from the rst group, three from the second and four from the third group. In addition, ve users were thinking about languages from the beginning of the task, one of whom belonged to the rst group and two to the second and third consecutively. Furthermore, six users started the task without thinking about languages but at some point before taking the hint started realizing that they needed to search across languages. It is also interesting to note that nineteen out of twenty four users started thinking about languages after taking the hint. Four users misinterpreted the hint on language and used it as a keyword. One user did not use the hints at all.
In terms of the second image, eight users started searching without thinking about languages, ve of whom were from the rst group, one from the second and two from the third. In addition, ten users were thinking about languages, three of whom from the rst and second group consecutively and four from the third. Five users started without thinking about languages but at some point before going for the hint realized that they would need to search across languages, four of whom were from the second and one from the third group. Only one user started with a perception of languages but at some point he realized that he was actually trying to search only in English and he belonged to the third group. On the second image, non of the users misinterpreted the hint and used it as a keyword and again one user did not make use of the hints.
In regards to the last image, ten users again started without thinking about languages, four of whom were from the rst, ve from the second and one from the third group. In contrast twelve were thinking about languages, four of whom were from the rst group, three from the second and ve from the third. Two users started thinking about languages after spending sometime searching for the image but before going for the hint, these belonged to the third group. On the third image, again no users misintepreted the hint and this time four of them did not make use of the hints.
It is interesting to point out that only three users were thinking about languages and trying to search for the three images across languages from the beginning and throughout the task. The rest of the users either did not think about languages throughout the three images and only the hint triggered them to think about it or at some point either on the rst or the second image started to understand that they would need to search across languages.
Of particular interest is some of users' remarks regarding the task of searching across languages: \ but I didn't know what else to write or what I should do, because I didn't know any languages and things like searching in other languages", \even though I am writing in Dutch it doesn't necessarily mean that it will bring up di erent images or any more images", \but I didn't think that it would a ect, especially that it would a ect the search that you would use because you would have Dutch as a language to search so, I just kept on typing English search", \ok it's making me do more that just search, it's making me use other languages for me to search and then I was happier because I could search a Spanish word which I obviously don't speak", \searching in di erent languages seems, sounds like it would be much harder than searching in English", \I just found that I had to think about a lot of things like you have to nd the image and try to translate it as well as nd out if it is in English...it took me a while to grasp, what I was meant to be doing for this one, like I realized it had to be in a di erent language the image but actually I was looking for keywords...".
Another aspect of users perception of searching across languages is the use of the two modes (monolingual/multilingual) (see Table 3 and Table 4). Four out of twenty four users stayed and used only one mode throughout the three images. In particular, three used only the monolingual mode and one only the multilingual mode throughout the three images. In terms of the rest twenty users, on the rst image ten were switching between the two modes, eight used only the monolingual mode and the remaining two only the multilingual. On the second image, sixteen users were again switching between the two modes and four used only the multilingual mode. On the last image, seventeen users were switching between the modes, two were using only the monolingual and one the multilingual mode.
In the context of investigating users' perceptions of the tasks involved when searching for images across languages, users' degree of trust in applications and con dence in their searching and language skills were also monitored both with the two questionnaires employed, the users' comments during retrospective thinking aloud and the individual interviews. In regards to the second questionnaire (see Table 5), the majority of the twenty four users have \Not at all" con dence in their language skills to search for images across languages on the web (S9), trust \A little" online translators to help them translate a text (S3), for languages of which they have some knowledge, they trust \A little" provided translations for searching on the web (S5) and they trust \A little" for use the search terms suggested to them by search engines on the web (S6).
In addition, the majority of the twenty four users trust \So-so"1 applications on the web to help them do the things they want to do (S1), the results that search engines on the web give (S2), and they have \So-so" con dence in their personal skills and abilities to search for images on the web (S8). Furthermore, the majority of the twenty four users answered equally that for languages they do not understand, they trust \A little" and \So-so" provided translations for searching on the web (S4). The majority again equally stated that they trust \Soso" and \Very much so" Help instructions to understand the use of interfaces (S7). Finally, the majority of users who have experience in searching on the 1 A prede ned six points scale (Don't know!, Not at all, A little, So-so, Very much so, Totally) was given to users to choose for each statement (see section 2: Questionnaires). web across languages stated that they have \Very much so" con dence when searching images across languages because of their relevant experience.
In terms of the third questionnaire (see Table 6), the majority of the twenty four users stated they \Don't know" for the statement \I trusted the help instructions to guide me through FlickLing (S8)". In addition, the majority had \Not at all" condence in their language skills to search for the images across languages in FlickLing (S11). The majority of users had \A little" condence in their personal skills to search for the images in FlickLing (S10) and in their experience that they were gaining in FlickLing for searching the images across languages (S12). In addition, the majority \So-so" trusted the results that FlickLings both monolingual and multilingual mode gave them (S1 and S2), trusted \So-so" the translations FlickLing gives for languages they do not know (S4) and they are \So-so" condent that FlickLing helped them search across languages for the images (S9). Finally, the majority trusted \Very much so" FlickLing to translate their query (S3), the translations FlickLing gives for languages they understand (S5), FlickLing to suggest search terms (S6) and the hints provided by FlickLing to search for the images (S7).
The second goal of the analysis would be to identify the reasons why users were or were not thinking about languages on each image and throughout the task. In addition, the reasons why the four users used only one mode for searching across languages and why the rest were either switching between the two or using just one mode while searching for the three images are to be further analyzed. In terms of users' trust and con dence, further analysis is intended to combine the data gathered from questionnaires, retrospective thinking aloud and interviews to shed light on the reasons why users provided the following answers to the 2nd and 3rd Questionnaires and determine whether trust and con dence are factors a ecting users' perceptions of the tasks involved when searching across languages. Finally, to meet our aim to gain insight into users search behaviour and use of translations further analysis will be carried out on data such as users' comments regarding the three given images, their comments regarding their language skills, and their interpretation and usage of the translation mechanism. It appears from our preliminary ndings that there is variation in our users perceptions of searching for images across languages and their approach to using translations and that this occurs regardless of the amount or type of help or guidance given. Further analysis of the data collected on users perceptions of the task and their explanation for their actions may provide further insight into understanding the user in the design and development of multilingual retrieval systems. 4
This study aimed at exploring users' image seeking behaviour when searching for known, non-annotated images across languages in FlickLing. In particular, this paper aimed to demonstrate the value in focusing on user's trust and con dence in the investigation of image seeking behaviour to reveal users' perceptions of the tasks involved when searching across languages. The task assigned to our 24 users was to search for the three images given to them after rst login. These images were described in Dutch, German and Spanish consecutively. In this context, four di erent methods, both qualitative and quantitative (questionnaires, observation, retrospective thinking aloud and interviews) were employed to meet its research question.
Preliminary ndings regarding users' switching between the two modes, users' thinking of languages and their answers to the two questionnaires on trust and con dence were only reported due to the inability to process the large amount of data collected in such a short period of time. Nonetheless, it appears from our preliminary ndings that there is variation in our users perceptions of searching for images across languages and their approach to using translations and that this occurs regardless of the amount or type of help or guidance given. Further analysis of the data will provide a better understanding of users' behaviour and perceptions so as to build cross-language information retrieval systems that better serve users.
This study used three di erent groups of users. These users received di erent levels of information about the task to be carried out and the interface to be employed. The full text of the instructions provided to each group of users (1st, 2nd and 3rd) is outlined here. The reader can nd a detailed description of these in the relevant chapter. First Group Task Instructions { The organizers would like to thank you for sparing some of your time to participate in this study. { I would like you rst to ll in this short questionnaire on basic background knowledge. { This study uses FlickLing, a multilingual interface for searching images across languages. You are asked to use this interface to search the rst three images given to you after login. The images given can be described in any of the following six (6) languages: English, German, Dutch, French, Italian and Spanish. You will not know in advance in which of these languages the images provided are described. Therefore, you need to use the features supported by the FlickLing interface in order to search the three (3) images across languages. { At this point, the organizers feel the need to remind you that it is the system and not you that is being evaluated and that you should perform as usual, spending as much time as you would have normally spent on searching. { You will be asked to ll in another short questionnaire after completing the searches for the three images. { Your search session will be system captured and played back to you. You will then be asked to describe your thoughts and actions while searching for the three images. The facilitator will encourage you to keep on describing your thoughts and actions when going to silence. A digital recorder will be used to tape your description. { At the end, a short length interview will take place between you and the facilitator that will be also taped. { I would now ask you to register as a member of the MMU team and start searching for the images.
Second Group Task Instructions { The organizers would like to thank you for sparing some of your time to participate in this study. { I would like you rst to ll in this short questionnaire on basic background knowledge. { This study uses FlickLing, a multilingual interface for searching images across languages. You are asked to use this interface to search the rst three images given to you after login. The images given can be described in any of the following six (6) languages: English, German, Dutch, French, Italian and Spanish. You will not know in advance in which of these languages the images provided are described. Therefore, you need to use the features (Help instructions, monolingual/multilingual mode, translations, suggested terms, give up button and hints) supported by FlickLing interface in order to search the three (3) images across languages. { At this point, the organizers feel the need to remind you that it is the system and not you that is being evaluated and that you should perform as usual, spending as much time as you would have normally spent on searching. { You will be asked to ll in another short questionnaire after completing the searches for the three images. { Your search session will be system captured and played back to you. You will then be asked to describe your thoughts and actions while searching for the three images. The facilitator will encourage you to keep on describing your thoughts and actions when going to silence. A digital recorder will be used to tape your description. { At the end, a short length interview will take place between you and the facilitator that will be also taped. { I would now guide you through registration for login to the FlickLing. You need to click on `register' button. First, you need to type in an `email account' and provide a `password'. Choose your `native language' from the menu and the `language of the interface' again from the ones in the relevant menu. Determine your team from the scroll down menu as `MMU' team. Choose from the menu the `level of comprehension' for the six languages supported by FlickLing and then click on `ok'. { You are now ready to use the FlickLing interface and search for the three images. As you can see on your left, you have the rst image. You can click on it in order to bring it forward and see it in detail. Two modes are provided, the monolingual and multilingual. On the monolingual, you write in one language and you get results only in the language of your query. On multilingual mode, you write your query in one language and you can determine in which languages of the six supported by FlickLing you can get results. On the right side of the search box, you will get each time the translations. Each time, you can cross out or provide a new translation for each desired language. You can also use the suggested terms provided by the system and the tags found in the images of the results. At any time, you can give up on a photo or make use of the hints provided by FlickLing. { You can now start searching for the three images.
Third Group Task Instructions { The organizers would like to thank you for giving some of your time to participate in this study. { I would like you rst to ll in this short questionnaire on basic background knowledge. { This study uses FlickLing, a multilingual interface for searching images across languages. You are asked to use this interface to search the rst three images given to you after login. The images given can be described in any of the following six (6) languages: English, German, Dutch, French, Italian and Spanish. You will not know in advance in which of these languages the images provided are described. Therefore, you need to use the features (Help instructions, monolingual/multilingual mode, translations, suggested terms, give up button and hints) supported by FlickLing interface in order to search the three (3) images across languages. { At this point, the organizers feel the need to remind you that it is the system and not you that is being evaluated and that you should perform as usual, spending as much time as you would have normally spent on searching. { You will be asked to ll in another short questionnaire after completing the searches for the three images. { Your search session will be system captured and played back to you after completing the searches. You will then be asked to describe your thoughts and actions while searching for the three images. The facilitator will encourage you to keep on describing your thoughts and actions when going to silence. A digital recorder will be used to tape your description. { At the end, a short length interview will take place between you and the facilitator that will be also taped. { I would now guide you through registration for login to FlickLing. You need to click on `register' button. First, you need to type in an `email account' and provide a `password'. Choose your `native language' from the menu and the `language of the interface' again from the ones in the relevant menu. Determine your team from the scroll down menu as `MMU' team. Choose from the menu the `level of comprehension' for the six languages supported by FlickLing and then click on `ok'. { You are now ready to use the FlickLing interface and search for the three images. As you can see on your left side, you have the rst image. You can click on it in order to bring it forward and see it in detail. By clicking on ok button, you go back to your previous screen. Two modes are provided, the monolingual and multilingual. In the monolingual, you search in one language and you get results only in the language of your query. In multilingual mode, you can determine in which language you write the query from the drop down menu and in which languages you get results by crossing out the relevant language buttons. If you do not cross out a language button then you get translations and consecutively results in all six languages. On the right side of the search box, you will get with each query the relevant translations. The system retrieves at the same time your search terms and the provided translations, so there is no need to retype the translations in the search box. Each time, you can cross out or provide a new translation for each language by clicking on the translation. You can use the suggested terms provided by the system after each search underneath your image by clicking on the term and selecting one of the options provided (include only this word, include this word to the rest of the query, include the translation of this word to the query). You can also use the tags found in the images of the results again by clicking on the tag and selecting one of the options provided (include only this word, include this word to the rest of the query, include the translation of this word to the query). The results are presented underneath your image and are listed in pages. The number of the pages varies according to the number of results. At any time, you can give up on a photo by clicking on the give up button found on the bottom side of the image. After clicking on the give up button, you can either make use of the hints provided by FlickLing or decide to skip this image and go for the next one. The system provides a number of hints, disclosing each time the language in which the image is described and the terms that retrieve this image. You can use as many hints as you wish by repeating the same steps (giving up and going for the hint button). You can now start searching for the three images.