GeoCLEF is an evaluation initiative for testing queries with a geographic specification in large set of text documents. GeoCLEF ran a regular track for the third time within the Cross Language Evaluation Forum (CLEF) 2008. The purpose of GeoCLEF is to test and evaluate cross-language geographic information retrieval (GIR). GeoCLEF 2008 consisted of two sub tasks. A search task ran for the third time and a Wikipedia pilot task (GikiP) was organized for the first time. For the GeoCLEF 2008 search task, twenty-five search topics were defined by the organizing groups for searching English, German and Portuguese document collections. Topics were developed also for English, German and Portuguese. Many topics were geographically challenging. Eleven groups submitted 131 runs. The groups used a variety of approaches, including sample documents, named entity extraction and ontology based retrieval. The evaluation methodology and results are presented in the paper.
The Internet has brought a large number of geographic services which range from map services to route planning and hotel reservation systems. Many queries for search engines are of geographic nature. The development and the evaluation of information retrieval systems which allow and optimize the geographically oriented access to information is of high practical relevance.
Geographical Information Retrieval (GIR) concerns the retrieval of information involving some kind of spatial awareness. Many documents contain some kind of spatial reference which may be important for IR. For example, to retrieve, rank and visualize search results based on a spatial dimension (e.g. “find me news stories about riots near Paris and their consequences”).
GeoCLEF is the first track at an evaluation campaign dedicated to evaluating geographic information retrieval systems. The aim of GeoCLEF is to provide the necessary framework in which to evaluate GIR systems for search tasks involving both spatial and multilingual aspects. Participants are offered a TREC style ad hoc retrieval task based on existing CLEF newspaper collections. GeoCLEF 2005 was run as a pilot track in 2005 and in since 2006, GeoCLEELF was has been a regular CLEF track. GeoCLEF has continued to evaluate retrieval of documents with an emphasis on geographic information retrieval from text. As such, searches with a geographic condition require the combination of spatial information and content based relevance into one result. Often, spatial reasoning is necessary to solve the search tasks.
Eleven research groups (13 in 2007) from a variety of backgrounds and nationalities submitted 131 runs (108 in 2007) to GeoCLEF.
For 2008, Portuguese, German and English were available as document and topic languages. As in previous years, there were two Geographic Information Retrieval tasks: monolingual (English to English, German to German and Portuguese to Portuguese) and bilingual (language X to language Y, where X or Y was one of English, German, or Portuguese).
GeoCLEF developed a standard evaluation collection which supports long-term research. Altogether, 100 topics including relevance assessments have been developed over the four last years (one pilot run and the three regular tracks). Another set of 26 CLEF ad-hoc topics with spatial restrictions has been identified and can also be used as a benchmark. It is intended to make the topics and the relevance judgment files publicly available 1 on the GeoCLEF website . Geographical Information Retrieval (GIR) concerns the retrieval of information involving some kind of spatial awareness. Many documents contain some kind of spatial reference which may be important for IR. For example, to retrieve, rank and visualize search results based on a spatial dimension (e.g. “find me news stories about riots near Paris and their consequences”).
Many challenges of geographic IR involve geographical references which are often vague, ambiguous
multi-lingually challenging [
The GeoCLEF track comprises two sub tasks. The main search task with newspaper collections is
described in the following sections. The GikiP task2 evaluates searches for Wikipedia entries which require
some geographical processing. It is describes in a separate overview paper [
GeoCLEF 2008 Search Task The geographic search task is the main task of GeoCLEF and it is developed like the CLEF ad-hoc task. The following sections describe the test design adopted by GeoCLEF. 2.1
The document collections for this year's GeoCLEF experiments areis identical to the ones used for GeoCLEF
20076,. It consists of newspaper and newswire stories from the years 1994 and 1995 used in previous CLEF
adhoc evaluations [
In all three collections, the documents have a common structure: newspaper-specific information like date, (optionally) page, issue, special filing numbers and usually one or more titles, a byline and the actual text. The document collections were not geographically tagged and contained no semantic location-specific information. Although the Portuguese collection can also be found in a fully parsed version, we have no information that this has been used by any participant. A total of 25 topics were generated for this year’s GeoCLEF (GC76 - GC100). Topic creation was shared among the Portuguese and the German groups, who created all topics utilizing the DIRECT System provided by the University of Padua. As in the past years, DIRECT included a search utility for the collections. Topics are meant to express a natural information need which a user of the collection might have.
Topic creation was performed in two stages. First, each group devised a set of candidate topics in their own language, whose appropriateness was checked in the text collection available for that language. The first choice was based on monolingual criteria These topic candidates were subsequently checked for relevant documents in the other collections. In many cases, it is difficult to find geographically interesting topics below the granularity of a country. Regional events with much coverage in one country do not often lead to many newspaper articles in other countries. As a consequence, the topics need to be partially modified or refined by relaxing or tightening the content or the geographic focus. Reasons driving this process were the absence of relevant documents in one of the languages, the complexity of topic interpretation and/or even translation into the other languages. For example, a candidate topic on fish on the Iberian Peininsula had relevant matches in the Portuguese collection. But not only did it lack matches in the other language newspapers but some of the species (e.g. "saramugo") were very hard to translate into German or English. The fish called saramugo seems to a species which lives only in Spanish and Portuguese rivers. Consequently, the spatial parameter (Iberian Peninsula) remained, but the subject was replaced by a matter more prone to attention from international mass media, (directly or indirectly) related to political and/or economical issues: in this case the state of agriculture. It should be stressed, however, that, in the majority of cases, the changes were not so extreme, corresponding to a simple topic extension. For instance, the initial candidate topic "Nobel Prize winners in Physics from Northern European countries" was replaced by a more general and simpler one: "Nobel prize winners from Northern European countries". In other cases we replaced the geographic term by other(s) involving a more difficult (but also more interesting) exercise of geographic reasoning and processing, as "Most visited sights in the capital of
http://www.linguateca.pt/CHAVE/ France" gave way to the topic: "Most visited sights in the capital of France and its vicinity", which is definitively much more challenging in the geographic point of view.
The final topic set was agreed upon after intensive discussion. Finally, all missing topics were translated into Portuguese and German and all translations were checked. The following section will discuss the creation of topics with spatial parameters for the track.
One goal of GeoCLEELF is the creation of a geographically challenging topic set. Geographic knowledge should be necessary to successfully retrieve relevant documents for most documents. While many geographic searches may be well served by keyword approaches, others require a profound geographic reasoning. We speculate that most systems and especially keyword based systems might perform better faces with a realistic topic set where these difficulties might occur less frequently.
In order to increase the difficulty of the topic set, several issues were explicitly included in the topics of GeoCLEF 20087. Some of them are the following ones: • • • • vague geographic regions (Sub-Saharan Africa , Western Europe ) geographical relations beyond IN (forest fires on Spanish islands) granularity below the country level (Industrial or cultural fairs in Lower Saxony) terms which do not occur in documents (Portuguese communities in demonstrations in German cities) other countries, We aimed at creating a set of topics representing different kinds of geographic queries that present different levels of complexity and may require different kinds of approaches to process them adequately and to successfully retrieve relevant documents. Rather than privileging specific geographical places, like a country or city, preference was given to reference to geographical regions, comprehending more than one physical or administrative place. Different kinds of regions were, then, considered, which may correspond, for instance, to a delimited geographical area of a given continent (e.g. Sub-Saharan Africa, Southeast Asia, Northern Africa, Western Europe) or country (e.g. Western USA, Lower Saxony, Spanish islands). Other interesting geoeconomic-political terms, such as OCDE countries, were also considered in the topic creation.
The majority of the GeoCLEF 2008 topics specify complex (multiply defined) geographical relations, a
property introduced in the GeoCLEF 2007 [
Proximity (e.g. Most visited sights in the capital of France and its vicinity); Inclusion (e.g. Attacks in Japanese subways);
Exclusion (e.g. Portuguese immigrant communities in the world).
Notice that the example illustrated in (i) also presents a relation of inclusion (established between sights and capital of France, and which is explicitly formalized by the preposition "in"). That relation can also be inferred in the NP Japanese subways occurring in the topic illustrated in (ii), which can be paraphrase by the expression "subways in Japan".
Contrarily to the topic creation performed last year, which contained explicit relations of exclusion (e.g. Europe excluding the Alps), such kind of relations were only implicitly represented in the topics of GeoCLEF 2008, as illustrated in (iii). This topic is indeed interesting because it refers simultaneously to an inclusion (communities from Portugal in the world) and exclusion (the generic geographical term world must be interpreted, in this context, as the entire world excluding Portugal).
We also chose to use vague geographic designations for certain topics, as done in previous GeoCLEF editions. For example, in the topic: Nuclear tests in the South Pacific, the geographical term South Pacific may refer both Australasia ("an an area including Australia, New Zealand, New Guinea and other islands including the eastern part of Indonesia") and Oceania ("a geographical (often geopolitical) region of many countries/territories (mostly islands) in the southern Pacific Ocean"). The interpretation of this geographical term (namely, in what concerns the determination if it refers to a land place or to the ocean) is only possible if we consider the full topic content.
A similar situation is observed in the topic "American troops in the Persian Gulf". In this case, the Persian Gulf does not stand for the gulf itself but to a Southwest Asian region, which is an extension of the Indian Ocean located between Iran and the Arabian Peninsula. Once again, the term disambiguation is only possible if taking into account the rest of the information described in the topic.
Another case of intended vagueness is the topic Environmental pollution in European waters, where the term waters can refer to rivers, lakes or sea
We repeat here that no markup of geographic entities in the topics was provided. Systems were expected to reveal that information be themselves from the topic which resembles a more realistic task. However, it was difficult to develop topics which fulfilled all criteria. For example, local and regional events which allow queries on a low level of granularity and which require local knowledge often do not result in newspaper articles outside the national press. This impedes made cross-lingual topic development hard. 2.3
The format of GeoCLEF 20087 is the same as that of 2006 and 2007. No explicit geographic information was given. Two examples for full topics are shown in table 3.
Tab. 3: Topics GC08958 and GC08475 <num>10.2452/89-GC</num> <title>Trade fairs in Lower Saxony </title> <desc>Documents reporting about industrial or cultural fairs in Lower Saxony. </desc>
<narr>Relevant documents should contain information about trade or industrial fairs which take place in the German federal state of Lower Saxony, i.e. name, type and place of the fair. The capital of Lower Saxony is Hanover. Other cities include Braunschweig, Osnabrück, Oldenburg and Göttingen. </narr> </top> <num>10.2452/84-GC</num> <title>Atentados à bomba na Irlanda do Norte </title>
<desc>Os documentos relevantes mencionem atentados bombistas em localidades da Irlanda do Norte </desc>
<narr>Documentos relevantes devem mencionar atentados à bomba na Irlanda do Norte, indicando a localização do atentado. </narr> </top> As can be seen, after the brief descriptions within the title and description tags, the narrative tag contains detailed description of the geographic detail sought and the relevance criteria. In some topics, lists of relevant geographic names are given. 2.4
In the last three editions of GeoCLEF, traditional ad-hoc retrieval approaches and specific geographic reasoning systems have been explored in parallel. Good success has often been achieved by ad-hoc techniques without any specific geographic knowledge or processing. These approaches have sometimes been developed as a baseline for more sophisticated systems. It has been observed that some of the traditional techniques may have effects which are beneficial for geographic search tasks. Blind relevance feedback can lead to a geographic term expansion necessary to solve a search problem. For example, a query for riots in German cities does not contain the name of any German city. A query including the term German may lead to documents which contain the word German as well as the names of some cities which can be included in subsequent optimized queries. As a result, geographic term expansion has been achieved without proper geographic knowledge being available for the system. This form of pseudo-geographic processing is obviously not very reliable, but the specific components often have a high error rate as well or do introduce too much noise. In GeoCLEF 2007, some systems tried combinations of the two approaches and the dedicated geographic systems had further matured. This year, new ideas have been introduced and an ontology based approach by the DFKI has been very successful for the most competitive task which is monolingual English. On the other hand, the University of Berkeley implemented a system designed like an ad-hoc system without any geographic elements and achieved excellent results for monolingual and bilingual tasks.
The participants used a wide variety of approaches to the GeoCLEF tasks, ranging from basic IR approaches to deep natural language processing (NLP) processing. The approaches include the use of full documents for ranking the result set, map based techniques and Wikipedia as a knowledge source. For more details we refer the reader to the description of the systems in the papers of the participants in this volume. 2.5
English assessment was shared by Berkeley and Hildesheim Universities. German assessment was done by the University of Hildesheim and Portuguese assessment by Linguateca. The DIRECT System already used for topic development, was also utilized for assessment. The system provided by the University of Padua allowed the automatic submission of runs by participating groups and supported assembling the GeoCLEF assessment pools by language. During the assessment process, the assessor tried to find the best collection of keywords – based on the detailed information in the narrative and his/her knowledge of the geographical concepts and subjects involved – and queried the DIRECT system. Some of the issues are reported in the following section for each language.
There were some topics which caused assessment difficulties: especially when the narrative required specific information. For example, given a sentence: Bonn ... former chancellor Willy Brandt ... Nobel Peace prize winner... Is this enough to infer that Willy Brandt was German? Opinions diverged but a consensus was reached. One topic that caused difficulties was the translation of Portuguese "monumentos" by English "sights" (or vice versa) when Euro Disney was to be assessed. It cannot by any means refer in Portuguese to such, but it can obvioulsy do that in the much more general English expression "sights". Also the German word Sehenswürdigkeiten (literally “things worth seeing”) can refer to more than “monuments” (Monumente in German). One could also subsume Euro Disney under this German term for “sights”. For the assessment, the more restrictive Portuguese interpretation was adopted for the pools in German and English. Discussing these issues among a geographically highly distributed group under serious time constraints which are set for the assessment is a challenge.
In assessments, topic drifts typically occur. This was also the case in the GeoCLEF 2008 assessment. Is a document about kidnapping of a French aid worker in Kenya relevant for "foreign aid in Sub-Saharan Africa"? On the one hand, his existence is a proof of the existence of foreign aid, but on the other hand, his kidnapping is not foreign aid.
Often, the assessment provides hints on why systems failed. This was the case for the German topic about fairs in Lower Saxony. The German word for fails (Messe) was matched against similar words which have a different meaning (e.g. angemessen -> appropriate, Messer -> knife). This may be due to inappropriate stemming rules or due to high n-gram similarity.
The English document pool also led to borderline cases for the assessors which needed to be discussed intensively. One topic required documents on natural disasters in Western states of the USA. Some documents only reported about the insurance costs caused by natural disaster overall. These were only considered relevant when the mentioned a geographically relevant place (sometimes they mentioned Los Angeles) even when they did not mention the disaster explicitly and directly.
The results of the participating groups are reported in the following sections. As shown in Table 6, a total of 11 groups from seven different countries submitted results for one or more of the GeoCLEF tasks. A total of 131 experiments (runs) were submitted. Five of these groups participated in GeoCLEF for the first time. Three different topic languages were possible for the GeoCLEF bilingual experiments. Again, the most popular language for queries was English; German took the second place. The number of bilingual runs by topic language is shown in Table 9. 3.2
Monolingual retrieval was offered for the following target collections: English, German, and Portuguese. Figures 1 to 3 show the interpolated recall vs. average precision for the top participants of the monolingual tasks. 3 2 12 17 The bilingual task was structured in four subtasks (X → DE, EN or PT target collection). The best system for each of the three bilingual sub-tasks was presented by the University of California at Berkeley who did not use any specific geographic reasoning or knowledge source. Figure 4 to 6 show the interpolated recall vs. average precision graph for the top participants of the different bilingual tasks.
The test collection of GeoCLEF grew of 25 topics each year. This is usually considered the minimal test collection size to produce reliable results. Therefore, statistical testing and further analysis are performed to assess the validity of the results obtained. The range of difficulties in the topics might have led to topics more difficult and more diverse than in traditional ad-hoc evaluations. To gain some insight on this issue, a topic performance analysis was also conducted. 4.1
Statistical testing for retrieval tests is intended to determine whether the order of the systems which results from
the evaluation reliably measures the quality of the systems [
We used the MATLAB Statistics Toolbox, which provides the necessary functionality plus some additional functions and utilities. We use the ANalysis Of VAriance (ANOVA) test. X X X X X X X X X X X X X X X X X
GeoCLEF has developed 100 topics and relevance judgments for geographic information retrieval. Another 26 topics with geographic specification were selected out of previous ad-hoc topics from CLEF. This test collection is the first GIR test collection available for the research community and it will be a benchmark for future research.
GIR is receiving increased notice both through the GeoCLEF effort as well as through scientific workshops on the topic. The wide availability of geographic systems on the Internet will further increase the demand for and the interest in geographic information retrieval.
For GeoCLEF 2009, a new GikIP track is again planned.
In addition, a query parsing and classification task is planned for GeoCLEF 2009. Such a task has been
part of GeoCLEF 2007 [