In this paper we provide a full overview of GikiCLEF, an evaluation contest (track) that was speci cally designed to expose and investigate cultural and linguistic issues involved in multimedia collections and searching. In GikiCLEF, 50 topics were developed by a multilingual team with non-English users in mind. Answers should be found in Wikipedia, but not trivially, in the sense that the task should be difcult for human users as well. Crosslinguality was fostered and encouraged by the evaluation measures employed. We present the motivation and the organization process, the management system developed, dubbed SIGA, an overview of participation and results, concluding with what we have learned from the whole initiative.
It is often stated that multilinguality is just about coping with the same information coded in different ways,
and that natural languages are simply a hindrance in our way to to getting at the information (whatever the
language). This naive view does not take into consideration that the different members of different language
communities also have different views about the information itself. Furthermore, different information is
coded in different languages, as is a widely known in disciplines such as linguistics [
Now that everyone is aware of the need to process more than one language, one should be wary of
processing the same information in all languages, and should instead focus on the ability to look for,
and make special use of, different information encoded in different languages and cultures. In a similar
vein, we believe that systems should cater for different kinds of users and not expect the same user needs
overall [
GikiCLEF, a follow-up of GikiP [
Considering that Wikipedia is an information source widely consulted in many languages, GikiCLEF's
aim was to foster the development of systems that helped real users. These users can be loosely de ned
as everyone interested in knowledge already embedded in Wikipedia, but who cannot attain it easily, either
for lack of time or ingenuity, or simply for not being able to browse hundreds of pages manually.
Wikipedia is here being used as the source of freely available multilingual data, semistructured and with
some quality control, that is, as an invaluable resource to gather semantic data for natural language
processing, as advocated by many as a solution to the knowledge acquisition bottleneck. We are obviously
not the rst to see Wikipedia in this light, cf. [
To our knowledge, Wikipedia snaphsots are by far the largest (partially aligned) multilingual corpora
that have the highest number of crosslingual links. Most other Web pages have just one or two other
languages to which they are linked, as can be appreciated e.g. in [
This said, and given that Wikipedia is something that evolves daily, it is challenging to process something real (and therefore with inconsistencies and problems), rather than a formal model which is an idealization, and keeps us closed in a lab. 2
In GikiCLEF, systems need to answer or address geographically challenging topics, on the Wikipedia collections, returning list of answers in the form of Wikipedia document titles.
The geographical domain was chosen, not only on internal CLEF grounds (to maintain the tradition
started by GeoCLEF and continued with GikiP) or because it is a hot topic nowadays, but because it
displays a huge variety in natural language that current gazetteer compilers are often not aware of. We
believed it made sense to look at geographically-related queries in order to highlight what is or may be
different from language to language, or culture to culture. In fact, in other spheres of thought there have
been strong claims for different spatial conceptualizations in languages, see [
In practice, a system participating in GikiCLEF receives a set of topics representing valid and realistic user needs, coming from a range of different cultures and languages in all GikiCLEF languages, namely Bulgarian, Dutch, English, German, Italian, Norwegian both Bokmal and Nynorsk2 , Portuguese, Romanian and Spanish, and its output is a list of answers, in all languages it can nd answers.
This kind of output seems to be appropriate, considering that it would be followed by a output formatter module: For different kinds of human users, and depending on the languages those users could read, different possible output formats would lter the information per language, as well as rank it in order of preference. We are assuming here that people prefer to read answers in their native languages, but that most people are happier with answers (remember, answers are titles of Wikipedia entries) in other languages they also know or even just slightly understand, than with no answers at all.
Since we are aware that not all GikiCLEF participants have the resources and interest to process or give answers in the ten collections, we have added the option of languages of participation to the registration 1The exception is probably the Bible, but it is not so widely accessed as Wikipedia in our days.
2Norwegian has two written standards, and Norwegians therefore decided to maintain Wikipedia in two parallel versions, so GikiCLEF covers nine languages and ten collections. We have therefore created questions in and/or translated them into both written standards of Norwegian. process, that is, languages of the users the systems want to please. However, as will be explained presently, systems not tackling all languages will at once have a lower score.
The evaluation measures are then as follows for a given run, and for each language:
, etc.) The nal score of any system is given by the sum of the scores for each individual language. So, the more languages a system returns answers in, the better its scores. Furthermore, a language with no answers for a particular topic (C=0) will not contribute for the relative ordering of the systems.
Note that a score for a particular language is the sum for all topics, not the average of the scores per topic. This is in order not to penalize languages which have no information on a particular topic in their Wikipedia. 3
The Wikipedia collections for all GikiCLEF languages were released on 20 January, 2009, and correspond to the Wikipedia snapshots from June 2008. They were converted to XML with the WikiXML tool created by the University of Amsterdam, which is available from http://ilps.science.uva.nl/WikiXML/. Figure 1 presents their relative sizes. Later on, due to some problems in the conversion, we allowed participants to use the HTML versions as well.
This was the only task performed prior to the development of SIGA3, which we then developed in order to assist both organizers and participants in the GikiCLEF task. In fact, four distinct roles had to be implemented, with different access modes and privileges: participant, topic manager, assessor, and administrator.
Brie y, the different tasks involved in the several phases of GikiCLEF, in a loose chronological order, were: Topic management The process of developing topics, nding some answers (pre-determined, and mark if they were self-justi ed or required further information), translate the wording into other languages and provide a motivation for them (for topic managers); Participation Fetching the topics, submitting answers and validating them , getting nal individual scores (for participants); 3SIGA stands for SIstema de Gestao e Avaliac¸ ao do GIKICLEF, Portuguese for Management and Evaluation System of GikiCLEF. The word siga means Go on! (imperative of verb seguir, continue).
Answer pool creation The process of merging all answers from all runs, come up with a pool of unique answers to be assessed, and attribute them to different assessors, with some overlap per language (for administrators); Topic assessment The process of evaluating individual answers (and their justi cations) as well as discuss hard cases (for assessors); Con ict resolution Comparing the assessments done by different assessors and proceed to a nal decision (for administrators); Results computation For each run, propagate the justi cation to other languages, do another (crosslingual) con ict resolution, obtain individual scores, and provide aggregated results (for administrators).
A system helping during all these phases was necessary since GikiCLEF had a really large (and geographically distributed) organizer committee, and the same was even more true of the assessors and participants masses.
SIGA was developed in MySQL, Perl and PhP and its source code is available, under a Gnu license, from the GikiCLEF site. We will be presenting SIGA along with the description of the process followed in 2009. 4
The nal topics were released 15 May 2009, after a set of 24 example topics, displayed in Table 1, had been made available some months before.
Topic in English List the Italian places which Ernest Hemingway visited during his life.
Which countries have the white, green and red colors in their national ag? In which countries outside Bulgaria are there published opinions on Petar Dunov's (Beinsa Duno's) ideas? Name Romanian poets who published volumes with ballads until 1941.
Which written ctional works of non-Romanian authors have as subject the Carpathians mountains? Which Dutch violinists held the post of concertmaster at the Royal Concertgebouw Orchestra in the twentieth cen What capitals of Dutch provinces received their town privileges before the fourteenth century? Which authors are born in and write about the Bohemian Forest? Name places where Goethe fell in love.
What Flemish towns hosted a restaurant with two or three Michelin stars in 2008? What Belgians won the Ronde van Vlaanderen exactly twice? Present monarchies in Europe headed by a woman.
Romantic and realist European novelists of the XIXth century who died of tuberculosis.
Name rare diseases with dedicated research centers in Europe.
List the basic elements of the cassata.
In which European countries is the bidet commonly used? List the 5 Italian regions with a special statute.
In which Tuscan provinces is the Chianti produced? Name mountains in Chile with permanent snow.
List the name of the sections of the North-Western Alps.
List the left side tributaries of the Po river.
Which South American national football teams use the yellow color? Name American museums which have any Picasso painting.
Which countries have won a futsal European championship celebrated in Spain? Name Spanish drivers who have driven in Minardi.
Which Bulgarian ghters were awarded the Diamond belt? Which Dutch bands are named after a Bulgarian footballer? Find coastal states with Petrobras re neries.
Places above the Arctic circle with a population larger than 100,000 people Which Japanese automakers companies have manufacturing or assembling factories in Europe? Which countries have Italian as of cial language? Name Romanian writers who were living in USA in 2003.
What European Union countries have national parks in the Alps? What eight-thousanders are at least partially in Nepal? Which Romanian mountains are declared biosphere reserves? Name Romanian caves where Paleolithic human fossil remains were found.
Which Norwegian musicians were convicted for burning churches? Which Norwegian waterfalls are higher than 200m? National team football players from Scandinavia with sons who have played for English clubs. Which rivers in North Rhine Westphalia are approximately 10km long? Chefs born in Austria who received a Michelin Star.
Political parties in the National Council of Austria which have been founded after the end of World War II Austrian ski resorts with a total piste length of at least 100 km Find Austrian grape varieties with a vineyard area below 100 ha.
Find Swiss casting show winners.
German writers which are Honorary Citizens in Switzerland.
Which cities in Germany have more than one university? Which German-speaking movies have been nominated for an Oscar? Formula One drivers who moved to Switzerland.
Which Swiss people were Olympic medalists in snowboarding at the Winter Olympic Games in 2006?
Before disclosing the nal topics in Table 2, let us present the topic creation guidelines and our expectations (not necessarily met by the nal set, as will be hinted at in the nal discussion):
One should strive for realistic topics which can be answered in some Wikipedia covered by GikiCLEF, chosen with a conscious cultural bias so that not all Wikipedia would have that information. Ideal topics for GikiCLEF may require knowledge of culture to understand the way they should be answered (or better, what it is that is being sought). This requirement entails that translation into other languages may require lengthy explanations. For example, Spanish guitar is a technical term in music that is probably not the best way to translate viola o, the Brazilian (original) term. Also, to render the Norwegian oppvekstroman requires the clari cation that this is close, but not the same as what, in English, literature experts use the German (!) term Bildungsroman to express. Similary, Romanian balade is probably a false friend with Spanish ballada, and had to be translated by romance. Interestingly, this is again a false friend with Portuguese romance, denoting what in English is called novel.
Answers to the questions had to be justi ed in at least one Wikipedia (that is, the string may be found as a entry in all Wikipedias, but the rest of the information has to be found in at least one). So, we are not looking for absolute truth, we are looking for answers which are justi ed in Wikipedia. Questions may include ambiguous concepts or names, especially when translated. In that case, participants were warned that only answers related to the proper disambiguation will be considered correct e.g. Which countries did Bush visit in the rst two years of his mandate? will not be correctly answered by the singer Kate Bush's travels in whatever mandate she may have (had). Narratives4 should thus clearly specify and explain the exact user need.
In case there appear ambiguities in the topic formulation that have not been discussed or clari ed in the narrative, and which have more than one interpretation acceptable (with respect to the user model at stake), assessment will accept both. For example, in Award-winning Romanian actresses in international cinema festivals, one would have to accept not only those actresses actually receiving prizes, but also those just in the audience or even hosting the event, if that had not been made clear beforehand (in the Further clari cation text).
4In fact, the term Further clari cation was employed in SIGA instead. Participants did not have access to them during submission, only after their participation.
Different answers about the same subject are welcome, provided they have support in the material. Examples are Who is (considered to be) the founder of mathematics? or Name the greatest scienti c breakthroughs in the XIXth century, which are obviously open to different opinions.5 During the topic discussion phase, the topic creation group came up with 75 topics, from which the nal 50 were chosen according to the following additional criteria: avoid repetition, avoid quizz-like avour, avoid hard to interpret topics, and then removing randomly until the number 50 was reached.
As an integral part of topic choice and preparation, SIGA helped the topic managers to look for answers in titles of Wikipedia documents pertaining to the GikiCLEF collection, as illustrated by Figure 3.
We expected that this process of nding candidates by just looking in the titles would be of considerable help for topic managers, who would not need to deal with the large collections in order to list correct answers. However, we did not require that people stored the answers there during topic creation.
This was something we provided as a facility in order to avoid, later, much work during assessment. Interestingly, only half of the members of the topic group used this, and also for different topics and for different languages there were different policies. Some people did it for the topics they owned in almost all languages, others did it for all topics only in their language, some did no pre-storing at all, and the majority did just some and in some languages. In Figure 4 one can see the result of this process.
5For the record, no topic owner chose to do this kind of opinion questions in GikiCLEF 2009.
Systems were supposed to deliver as many answers (in as many languages) as possible, but answers had to be justi ed in at least one language. For an answer to be considered justi ed, it required simply that a person would accept the answer by reading it (the article) and further documents offered as additional material. Of course this is ultimately subjective, but all evaluation in information retrieval is. In order to ensure a maximum of fairness, guidelines for borderline cases had to be discussed among the assessors and normalized in the end, to the best of our abilities (and to the strain of the assessors, who had often to reassess their answers).
The Wikipedia page about the answer may be its own justi cation (the simplest case), but we imagined, and catered for, cases where other pages would have to be brought to bear (such as disambiguation or lists, or even images).
An answer without justi cation was never to be considered right.
Let us provide two examples in more detail:
Question In which places did Italo Calvino live during adulthood? would require a system to go to the page(s) devoted to this writer, nd that information, and get the places, namely e.g. Turin and Paris. In order to have these accepted as correct answers, the page about Italo Calvino which describes his life, e.g. http://en.wikipedia.org/wiki/ItaloCalvino, would have to be included as (further) justi cation for Paris and Turin.
Once there was a justi cation (in this example and to make it easier for the present paper, in English although the most complete is probably in the Italian Wikipedia), any answers like Turim in Portuguese or Parisj in Dutch would be considered correct: in other words once justi ed in a particular language, justi ed for all languages.
Now to a more complex example, to show how the GikiCLEF format allows arbitrary chains of reasoning which is not to say that we expected current systems to be able to do it. Take question Name American cities where people who killed presidents lived for more than one year. To answer it, in addition to the name of the city, systems would have to nd the names of presidents who were killed, and although in this particular case there is even a category in Wikipedia entitled United States presidential assassination attempts this might require that systems go through all pages concerning presidents and investigate the deaths, their causes and the names of the assassins, then check the assassins' pages, and nally extract the cities where they lived.
In order for an answer to be justi ed, let us say the answer Chicago, the justi cation would have to include the page of the assassin that mentions that place, and the name of the president killed as well if this is not mentioned in the assassin's page. So, in principle at least, one may have to include several pages in order to justify any given answer.
In GikiCLEF, as before in GikiP, only answers / documents of the correct type were considered correct. That is, if the question is about people, an answer of an organization is considered wrong, even if in that document whose title was an organization there is the person one would want as answer.
After pooling all answers returned by the participant systems, they were manually assessed by the assessors' group. SIGA's assessment interface, displayed in Figure 5, allows the assessors to judge the candidate answers, and check the correctness of their justi cations.
Prior to this, to ease the assessment task, an automatic process assesses the answer documents that were listed as correct answers during the topic preparation period, as well as eliminates invalid document answers (such as redirects).
Assessment in GikiCLEF proceeds in several phases: 1. All pre-stored correct answers which are self-justi ed are automatically classi ed. The ones which require a justi cation are marked as Correct, but are still presented to the assessors for them to assign a Justi ed (or Not Justi ed) verdict. 2. Assessors assess individual answers, assigning either Incorrect, Correct, or Unknown. If it is Correct, they have to indicate whether the individual answer they are assessing (which includes the justi cation chain) is Justi ed, or whether it is Not Justi ed. 3. A process of con ict resolution among different assessments of the very same answer is then run, which allows people to discuss and get aware of complications and/or mistakes or mistaken assumptions. Only after all con icts are resolved can one proceed to:
5. A new process of crosslingual con ict resolution then ensues, with the net result that positively con icting information for one topic brings about the inhibition of multilingual propagation: for those topics, only monolingually correct and justi ed answers will be considered correct.
It goes without saying that all these phases and checkpoints allowed us to nd problems, inconsistencies and even wrongly pre-de ned answers in the original topic set.
Figure 6 displays SIGA's assistance in con ict solving. The administrator can choose to send a question to the diverging assessors, or decide herself, if it is a straightforward case.
The nal scores are automatically computed after the assessment task and made available to the participants, who are granted access to several scores and the detailed assessment of their answers, as illustrated in Figure 7.
While this seems a complete enough description of the assessment process, one should document that a lot of other more speci c decisions and guidelines had to be decided during the process. By writing them down here we intend not only to illustrate the kinds of problems that arise, but also provide an initial set for further initiatives or GikiCLEF editions.
1. If the answer is already contained in the question, it is considered incorrect. For example, Italy is not a fair answer to a question starting by List the Italian places 2. If there is principled disagreement about vague, complex categories and different people have strong reasons for disagreement, for GikiCLEF we accept the union of all. 3. Speaking/writing poets in other languages than Romanian are Romanian poets? We decided for a yes. 4. Studying in a place, taking a short visit to another place and coming back in love to that place, does it qualify as a place where someone falls in love? Again, yes. 5. If a ciclist won the junior Tour de Flandres and then the adult one, is s/he considered a winner twice? We decided for yes, although this is a recurrent issue in sports questions. Often, without further speci cation, only the major competition is meant. 6. Very slight differences which very strongly convey the probability of yes are accepted, because we would expect most people (except lawyers and logicians) to accept that:
Eight thousanders accept a 50 m deviation (if a mountain is higher than 7950 m) Norwegian musicians convicted for burning (even if the article does not mention they burned churches) must be the ones looked for People wro wrote ballads and published a lot of volumes of poetry is expected to have published volumes with ballads although the article does not say so People who have two residences, one in Switzerland and another somewhere else, can be considered to have moved to Switzerland some time in their lives. 7. If two of three sisters died of tuberculosis and for the third the cause of death is not certain, is a page entitled the sisters Bronte¨ correct? We relaxed the strict requirement that writers should be people and not a group of people, because beforehand we did not expect groups of writers to stand as an article. So we accepted it as correct. 8. No longer existing Austrian parties, provided they were founded after the Second World War and had at some time people in the National Council of Austria, were considered correct. This brings about the often noted fact that most questions are not independent on time. 9. Finally, what are American museums? This expression should be interpreted according to the natural meaning of the corresponding word (american, amerikansk, americanos etc.) in the corresponding language at least this was how we asked people to translate the question. But apparently Canadian and Brazilian museums do not mention their Picassos in their Wikipedia pages (so, even if correct, these answers will turn out Not justi ed, hence Incorrect), and the only hits found corresponded to museums in the USA. In this case, it was probably the organization's fault not to emphasize to the participants that each language topics should be understood and answered in that language, even if the correct interpretation of the terms in the different languages turned out to be different.6 We do not want to convey the idea that anything goes, though. In fact there were several other cases which were negatively decided: 1. Gulag can be metonymically used for the places where people were imprisoned and most of them were above the Arctic. However, we did not consider it as a valid place. 2. Fictional countries were not considered as correct when asking for ctional works, even if they were created or presented in the scope of a written ctional work. 3. Cases where the expected answer type (EAT) was clearly different from the one returned were considered downright incorrect, notwithstanding our agreement that the answers could be useful. So, Flags were not accepted as answers to questions Which countries had a ag ...? Queens were not accepted as answers to questions Which countries have a queen...? Countries were not accepted as answers to questions Which national teams...?
This is in line with our belief that assessment would have become a nightmare if any answer, whatever its type, had to be investigated by the assessors to see whether it could be indirectly useful. However, we are also aware that different participants in GikiCLEF 2009 interpreted the task differently, which produced unwanted differences among the participants. Clearly, this issue has to be considered for future editions, and an intermediate solution could be that, for some topics, more than one EAT, previously agreed, could be accepted. 7
Although we had almost 30 registrations of interest for GikiCLEF, in the end only 8 participants were able to submit. For the record, they are displayed in Table 3 by registration order.
Ray Larson Sven Hartrumpf & & Johannes Leveling Iustin Dornescu TALP Research Center Gosse Bouma & Sergio Duarte Nuno Cardoso et al.
Adrian Iftene Richard Flemmings et al.
6This raises the problem, not yet satisfactorily solved, that culturally-laden questions are not exactly parallel, and that therefore the set of (multilingual) answers ultimately depends on the language the question was asked.
Figure 9 presents the participation detailed for each language. The last row indicates how many participants per language, and the last column the number of languages tried in that run. Eight runs opted for all (10) languages, four tried solely 2 languages, and ve one only.
While this seems a modest amount of work, in fact it produced a sizeable amount of material to deal with, as Table 4 shows.
The reason why there were considerably more manual assessments than manually assessed answers is due to the important fact that 2,131 answers had more than one assessor (often two, but they may have been assigned up to four different ones), to test the soundness and coherence of the assessment process. Note, anyway, that this does not include repeated assessments by the same assessor, nor assessments done by the organizers during con ict resolution, so that in practice the work involved was substantial, even with 29 assessors.
Turning now to the comparative weight and/or performance of the different languages involved, at face value, all languages participated in the answer gathering.
Figure 10 provides an overview of the total number of answers per language, while Figure 11 shows the distribution of only the correct answers.
The number of answers provided per language, as well as the amount of the correct ones, seems to demonstrate that the GikiCLEF systems could be used with the same level of success in all GikiCLEF languages. In Figure 12 also the precision per language is shown.
However, it does not say anything about whether there were languages which were necessary to check in order to have a (crosslingually) justi ed (and therefore correct answer). For this we tried to see if some languages had a large amount of correct answers due to other languages, that is, we wanted to check language dependence or interdependence. A language should be more authoritative the more answers it provided without requiring proof in other languages.
Figure 13 presents those numbers, as well as contrasting the number of correct answers per language, with the pre-assessed (correct) ones.
Interestingly, and contrary to our expectations concerning English, that gure again does not allow one to infer that English has more information or more detailed justi cations in pages written in that language. This must be an artifact of our topic choice, which was on purpose geared toward languages different from English. Still, and even with our initial guidelines, many of the topics chosen were more international than really national (even if they did relate to speci c individuals of a non-English-speaking nationality), and therefore one would expect that they would have equally developed pages in English as well.
A more thorough investigation of the different GikiCLEF topics regarding language spread should thus
take place, such as the one done by [
Also, it remains to be investigated which topics might be popular (or even asked at all) regarding different language populations. Of course our organizers' sample was not representative, and, in addition, and due to the random choice, some topic owners (proposers) received more topics than others. In fact, a cursory examination of the nal topics shows that language or culture distribution was quite skewed, with a predominance of Romanian and German topics, on the one hand, and a scarcity of Portuguese and Norwegian ones, on the other.
If we look at the topics per language, then the relative importance of English nally emerges: for the vast majority of topics the language with higher number of correct hits is English. Table 5 shows a selected sample of the topics per language. Most of the remaining ones did feature English as the decisive winner (the full table is available from the GikiCLEF site). One other thing that remains to be done is an
GC-2009-07 GC-2009-09 GC-2009-19 GC-2009-27 GC-2009-34 GC-2009-48 GC-2009-50
BG 8 3 0 1 15 1 0 investigation of how really different the several answers are, that is, are the answers relative to the same individuals or places, or rather different?
So, while GikiCLEF was able to demonstrate that there are systems that can answer (although still with poor performance) questions in these nine languages, the real utility for each language of processing also the other nine collections has not yet been established.
We present, for comparison, at the end of the paper, a description of the different monolingual GikiCLEFs, by presenting precision per topic for some of the languages. We would like to emphasize, however, that these views are somehow arti cial on several counts: not only they represent the joint performance of the several participants, but for some languages, such as for example Italian or Norwegian, their collections were not processed in any way, that is; the hits came from processing e.g. the German, the Dutch and the English collections... So these could be called, in GikiCLEF parlance, parasitic languages, and in fact it is interesting to note that they do attain better precision than other languages whose collections were processed, such as Spanish and Portuguese. 8
As a general opinion it is fair to say that GikiCLEF was universally considered too dif cult or ambitious, which resulted in that several prospective participants gave up and not even sent in results.
Many people strove hard to just be able to process the huge collections and minimally parse the topic renderings, and did not even consider cultural differences and/or crosslinguality. Our impression is that most participants did the bulk of processing in one main language, and then used naive and straightforward procedures to get answers in other languages. So, neither crosslinguality (differences in conveying related information) or multilinguality (the fact that different Wikipedias might produce different results) were really investigated by the rst GikiCLEF participants.
If we make a more detailed inspection of the topics and the systems' behaviors, we can identify the easiest and most dif cult topics, through the display, in Figures 14 and 15, of the number of answers and the conjoined precision (taking all participants together) attained.
Another feature we were expecting people to make use of was the justi cation eld, which could in a way display the reasoning done. However, very few participants (only two) used a justi cation eld, and apparently it was not very successful either, see Figure 16. In fact, the proportion of justi ed answers was only considered correct ca. 50% of the times. But we believe that if further justi cations had been given by the participants their score would increase.
Several other things did not work out as expected, and in particular we believe now that some quizzlike or relatively strained topics ended up in the nal topic list, while topic managers in general shied away from the formidable task to convey things peculiar to their own cultures to a set of foreigners, and decided for simpler topics to begin with.
Finally, we would have liked to see more practically oriented systems with a special purpose and an obvious practical utility to try their hand at GikiCLEF. Apparently most if not all participants were simply considering GikiCLEF too hard and had no independent system of their own to try out there. Again, this may prove the complexity of the task, or the fact that the audience was not appropriate. We hope that training with GikiCLEF materials, all of them made available on due course, may in any case help future systems to perform dif cult tasks with semi-structured multilingual material, which we still believe is something required out there.
All resources compiled under GikiCLEF, as well as collections and Web access to SIGA, can be reached from http://www.linguateca.pt/GikiCLEF.
We are very grateful to Nuno Cardoso for preparing the colections, to the remaining organizers So¨ren Auer, Gosse Bouma, Iustin Dornescu,Corina Forascu, Pamela Forner, Fredric Gey, Danilo Giampiccolo, Sven Hartrumpf, Katrin Lamm, Ray Larson, Johannes Leveling, Thomas Mandl, Constantin Orasan, Petya Osenova, Anselmo Penas, Erik Tjong Kim Sang, Julia Schulz, Yvonne Skalban, and Alvaro Rodrigo Yuste for hard work, supportive feedback and enthusiasm, and to the larger set of further assessors including the organizers and further Anabela Barreiro, Leda Casanova, Lu´s Costa, Ana Engh, Laska Laskova, Cristina Mota, Rosa´rio Silva, and Kiril Simov who helped with assessment. Paula Carvalho and Christian-Emil Ore helped in an initial phase by suggesting Portuguese and Norwegian-inspired topics, respectively.
Iustin Dornescu and Sven Hartrumpf deserve further mention, the rst for having performed an extremely large number of assessments, and the second for intelligent critical comments and revision throughout the whole process, as well as for pertinent discussions in all GikiCLEF lists. Finally, Alexander Yeh's testing and debugging of the Wikipedia collections was particularly useful.
The organization work, as well as the writing of this paper, were accomplished under the scope of the Linguateca project, jointly funded by the Portuguese Government, the European Union (FEDER and FSE), under contract ref. POSC/339/1.3/C/NAC, UMIC and FCCN. We also gratefully acknowledge support of the TrebleCLEF Coordination Action. ICT-1-4-1 Digital libraries and technology-enhanced learning (Grant agreement: 215231). and Multimodal Information Access 9th Workshop of the Cross-Language Evaluation Forum, CLEF 2008, Aarhus, Denmark, September 17-19, 2008, Revised Selected Papers. Springer, 2009.