This paper summarises activities from the iCLEF 2009 task. As in 2008, the task was organised based on users participating in an interactive cross-language image search experiment. Organizers provided a default multilingual search system (Flickling) which accessed images from Flickr, with the whole iCLEF experiment run as an online game. Interaction by users with the system was recorded in log les which were shared with participants for further analyses, and provide a future resource for studying various e ects on user-orientated cross-language search. In total six groups participated in iCLEF with di erent approaches, ranging from pure log analysis to speci c experiment designs using the Flickling interface.
interactive information retrieval, cross-language information retrieval iCLEF, Flickr, log analysis, multilingual image search, user studies, multilingual tag search iCLEF is the interactive track of CLEF (Cross-Language Evaluation Forum), an annual evaluation exercise for Multilingual Information Access systems. In iCLEF, Cross-Language search capabilities are studied from a user-inclusive perspective. A central research question is how best to assist users when searching information written in unknown languages, rather than how best an algorithm can nd information written in languages di erent from the query language.
Since 2006, iCLEF has moved away from news collections (a standard for text retrieval experiments) in order to explore user behaviour in scenarios where the necessity for cross-language search arises more naturally for the average user. We chose Flickr, a large-scale, web-based image database based on a large social network of WWW users sharing over two billion images, with the potential for o ering both challenging and realistic multilingual search tasks for interactive experiments.
Over the last years, iCLEF participants have typically designed one or more cross-language search interfaces for tasks such as document retrieval, question answering or text-based image retrieval. Experiments were hypothesis-driven, and interfaces were studied and compared using controlled user populations under laboratory conditions. This experimental setting has provided valuable research insights into the problem, but has a major limitation: user populations are necessarily small in size, and the cost of training users, scheduling and monitoring search sessions is very high. In addition, the target notion of relevance does not cover all aspects that make an interative search session successful; other factors include user satisfaction with the results and usability of the interface.
The main novelty of the iCLEF 2008 shared experience, which has been kept in 2009, was to focus on the shared analysis of a large search log from a single search interface provided by the iCLEF organizers. The focus is, therefore, on search log analysis rather than on system design. The idea is to study the behaviour of users in an (almost) naturalistic search scenario, having a much larger data set than in previous iCLEF campaigns. The search interface provided by iCLEF organizers is a basic cross-language retrieval system to access images in Flickr, presented as an online game: the user is given an image, and she must nd it again without any a-priori knowledge of the language(s) in which the image is annotated. Game-like features are intented to engage casual users and therefore increase the chances of achieving a large, representative search log.
The iCLEF 2009 task is the same as in 2008, the only di erence being the approach to select the target images (the topics for our task). In 2008 a large log was harvested, but in over half of the search sessions the user had active language skills in the target language, and the situations were the user has only passive or no abilities in the target language were underepresented. The reason was that many images in the target set had annotations in English (plus other languages in many cases), and the set of users (over 200 active searchers) tend to have English as a native or at least as a well-known language. Therefore, this year we explicitly avoided images annotated in English to increase the chances of having search sessions in unknown languages.
The structure of the rest of the paper is as follows: Section 2 describes the task guidelines (and can be skipped by readers familiarized with the iCLEF 2008 task); Section 3 describes the features of the search log distributed to participants. In Section 4 we summarize the participation in the track and give some conclusions about the experience. 2
The task is exactly the same as in 2008, and the di erences lie in the search log collected (target images, set of registered users, etc.) Readers which are familiarized with iCLEF 2008 can safely skip this Section. 2.1
First of all, the decision to use Flickr as the target collection is based on (i) the inherent multilingual nature of the database, provided by tagging and commenting features utilised by a worldwide network of users, (ii) although it is in constant evolution, which may a ect reproducibility of results, the Flickr search API allows the speci cation of timeframes (e.g. search in images uploaded between 2004 and 2007), which permits de ning a more stable dataset for experiments; and (iii) the Flickr search API provides a stable service which supports full boolean queries, something which is essential to perform cross-language searches without direct access to the index.
For 2008, our primary goal was harvesting a large search log of users performing multilingual searches on the Flickr database. Rather than recruiting users (which inevitably leads to small populations), we wanted to publicize the task and attract as many users as possible from all around the world, and engage them with search. To reach this goal, we needed to observe some restrictions:
The search task should be clear and simple, requiring no a-priori training or reading for the casual user.
The search task should be engaging and addictive. Making it an online game - with a rank of users - helps achieve that, with the rank providing a clear indication of success. There should be no need for manual judgements in order to establish the success of a search session, in order to avoid discouraging delays in the online game rankings.
It should have an adaptive level of di culty to prevent novice users from being discouraged, and to prevent advanced users from being unchallenged.
We decided to adopt a known-item retrieval search task: the user is given a raw (unnanotated) image and the goal is to nd the image again in the Flickr database, using a multilingual search interface provided by iCLEF organizers. The user does not know in advance in which languages the image is annotated; therefore searching in multiple languages is essential to get optimal results. Although the task is probably not the most natural one (thematic-based searches are probably more common than "stu I've seen before" search needs), it has the de nitive advantage of not requiring manual judgements, and that makes possible to keep an instantly updated user ranking.
Indeed the task is organized as an online game: the more images found, the higher a user is ranked. In case of ties, the ranking will also depend on precision (number of images found / number of images attempted). At any time the user can see the \Hall of Fame" with a rank of all registered users.
Depending on the image, the source and target languages, this can be a very challenging task. To have an adaptive level of di culty, we implemented a hints mechanism. At any time whilst searching, the user is allowed to quit the search (skip to next image) or ask for a hint. The rst hint is always the target language (and therefore the search becomes mono or bilingual as opposed to multilingual). The rest of the hints are keywords used to annotate the image. Each image found scores 25 points, but for every hint requested, there is a penalty of 5 points. The hint mechanism proved essential to engage users in 2008 and even more in 2009 (for reasons explained later).
Initially a ve minute time limit per image was considered, but initial testing indicated that such a limitation was not natural and had a deep impact on users' search behaviour. Therefore we decided to remove time restrictions from the task de nition. 2.2
We designed the so-called Flickling interface to provide a basic cross-language search front-end to
Flickr. Flickling is described in detail in [
Cross-language search is performed via term-to-term translations between six languages using free dictionaries (taken from: http://xdxf.revdanica.com/down).
A term-to-term automatic translation facility which selects the best target translations according to (i) string similarity between the source and target words; (ii) presence of the candidate translation in the suggested terms o ered by Flickr for the whole query; and (iii) user translation preferences.
A query translation assistant that allows users to pick/remove translations, and add their own translations (which go into a personal dictionary). We did not provide back-translations to support this process, in order to study correlations between target language abilities (active, passive, none) and selection of translations.
A query re nement assistant that allows users to re ne or modify their query with terms suggested by Flickr and terms extracted from the image rank. When the term is in a foreign language, the assistant tries to display translations into the user's preferred language to facilitate feedback.
Control of the game-like features of the task: user registration and user pro les, groups, ordering of images, recording of session logs and access to the hall of fame. Post-search questionnaires (launched after each image is found or failed) and nal questionnaires (launched after the user has searched fteen images, not necessarily at the end of the experience). 2.3
As in 2008, iCLEF 2009 participants can essentially adopt two types of methodology: (1) analyse log les based on all participating users (which is the default option) and, (2) perform their own interactive experiments with the interface provided by the organizers. CLEF individuals registered in the interface as part of a team, so that a ranking of teams is produced in addition to a ranking of individuals. 2.3.1
Generation of search logs Participants can mine data from the search session logs, for example looking for di erences in search behaviour according to language skills, correlations between search success and search strategies, etc. 2.3.2
Interactive experiments Participants can recruit their own users and conduct their own experiments with the interface. For instance, they could recruit a set of users with passive language abilities and another with active abilities in certain languages and, besides studying the search logs, they could perform observational studies on how they search, conduct interviews, etc. iCLEF organizers provided assistance with de ning appropriate user groups and image lists, for example, within the common search interface. Besides these two options, and given the community spirit of iCLEF, we were open to groups having their own plans (e.g. testing their own interface designs or using a speci c set of images) as long as they did not change the overall shared search task (known-item search on Flickr). 3
Search logs were harvested from the Flickling search interface between May and June 2009 (see [
Overall, 130 users registered for the task, for a total of 2527 search sessions, many of them ending in success (2149). There were 19 native languages in our user set, with this distribution: 46 Spanish, 38 Romanian, 10 English, 9 Italian, 4 Persian/Farsi, 4 German, 3 Chinese, 2 Finnish, 2 Catalan, 2 Basque, 2 Arabic, 1 Danish, 1 Vietnamese, 1 Malay, 1 Russian, 1 Greek and 1 Belarusian.
Apart from general users, the group a liation revealled two dominant user pro les: university researchers and students (most of them in Computer Science) and photography fans.
The 2008 search log was skewed towards "active" search sessions (where users had active skills in some of the languages used to annotate the image). Therefore this year we changed the methodology to select the target images, excluding those which had annotations in English, and reducing the number of images annotated in Spanish (because it was a well represented native language in our user base). The strategy was too successful: we harvested 1585 search sessions where the target language was unknown to the user, 18 where the user had passive abilities (i.e. could read results but not write queries), and none where the user had active skills in the target language. That makes this search log an excellent tool to study the behaviour of users searching in foreign language, but it can hardly be used to compare the three pro les. We also found that the combination of users and images is so di erent from the 2008 experience that merging the two search logs, even if the task is the same, is not advisable.
Overall, it has been possible to collect a large controlled multilingual search log, which includes both search behaviour (interactions with the system) and users' subjective impressions of the system (via questionnaires). This o ers a rich source of information for helping to understand multilingual search characteristics from a user's perspective. 4
Six sites submitted results for this year's interactive track: two newcomers (University of North Texas and Alexadru Ioan Cuza University, UAIC, in Romania) and four groups with previous experience in iCLEF: Universidad Nacional de Educacion a Distancia (UNED), the Swedish Institute of Computer Science (SICS), Manchester Metropolitan University (MMU), and the University of Alicante.
University of Alicante [
UAIC [
Manchester Metropolitan University [
UNED [
University of North Texas [
SICS has continued to investigate methods for how to study con dence and satisfaction of users. In previous years' studies, results have been somewhat equivocal; this year, some preliminary studies of the number of reformulations versus success rate have been performed. The SICS team found that the length of query sequences which eventually were successful were longer, indicating persistence when a search appears to be in the right direction. The number of query reformulations also correlate well with success: successful query sequences are a result of active exploration of the query space. But for users who persist in working with monolingual searches (search calls), the SICS team found that queries, rstly tended to be vastly less often reformulated to begin with, and that the successful sequences were more parsimonious than the failed ones (conversely from the clsearch calls): instead the number of scroll actions were much more frequent. This would seem to indicate that if users are fairly con dent of a well put query, they will persist by scrolling through result lists. The gures in Table 1 are all statistically signi cant by the Mann Whitney U rank sum test (p > 0:95). 5
Conclusions iCLEF 2009 has continued to run a large-scale interactive experiment as an online game to generate log les for further study. A default multilingual information access system developed by the organizers was provided to participants to lower the cost of entry and generate search logs recording user's interaction with the system and qualitative feedback about the search tasks and system (through online questionnaires). In addition, two groups have decided to replace (or extend) log analysis by recruiting their own set of users and employ the usual methodology (training, questionnaires, interviews, retrospective thinking aloud, observational studies) on them.
The search logs generated by the iCLEF track in 2008 and 2009 together are a reusable resource for future user-orientated studies of cross-language search behaviour, and we hope to see new outcomes in the near future coming from in-depth analysis of our logs. Researchers interested in this resource might contact the iCLEF organization (see http://nlp.uned.es/iCLEF) for details. This work has been partially supported by the Regional Government of Madrid under the MAVIR Research Network (S-0505/TIC-0267) and the Spanish Government under project Text-Mess (TIN2006-15265-C06-02).