GeoCLEF ran as a regular track for the second time within the Cross Language Evaluation Forum (CLEF) 2007. The purpose of GeoCLEF is to test and evaluate cross-language geographic information retrieval (GIR): retrieval for topics with a geographic specification. GeoCLEF 2007 consisted of two sub tasks. A search task ran for the third time and a query classification task was organized for the first. For the GeoCLEF 2007 search task, twenty-five search topics were defined by the organizing groups for searching English, German, Portuguese and Spanish document collections. Topics were translated into English, German and Spanish. Several topics in 2007 were geographically challenging. Thirteen groups submitted 108 runs. The groups used a variety of approaches. For the classification task, a query log from a search engine was provided and the groups needed to identify the queries with a geographic scope and the geographic components within the local queries.
GeoCLEF1 is the first track in an evaluation campaign dedicated to evaluating geographic information retrieval systems. The aim of GeoCLEF is to provide the necessary framework in which to evaluate GIR systems for search tasks involving both spatial and multilingual aspects. Participants are offered a TREC style ad hoc retrieval task based on existing CLEF newspaper collections. GeoCLEF 2005 was run as a pilot track and in
2006, GeoCELF was a regular CLEF track. GeoCLEF has continued to evaluate retrieval of documents with an emphasis on geographic information retrieval from text. Geographic search requires the combination of spatial and content based relevance into one result. Many research and evaluation issues surrounding geographic monoand bilingual search have been addressed in GeoCLEF.
GeoCLEF was a collaborative effort by research groups at the University of California, Berkeley (USA) , the University of Sheffield (UK), the University of Hildesheim (Germany) and Linguateca (Norway and Portugal). Thirteen research groups (17 in 2006) from a variety of backgrounds and nationalities submitted 108 runs (149 in 2005) to GeoCLEF.
For 2007, Portuguese, German and English were available as document and topic languages. There were two Geographic Information Retrieval tasks: monolingual (English to English, German to German and Portuguese to Portuguese) and bilingual (language X to language Y, where X or Y was one of English, German or Portuguese). In the three editions of GeoCLEF so far, 75 topics with relevance assessments have been developed. Thus, GeoCLEF developed a standard evaluation collection which supports long-term research. Geographical Information Retrieval (GIR) concerns the retrieval of information involving some kind of spatial awareness. Many documents contain some kind of spatial reference which may be important for IR. For example, to retrieve, rank and visualize search results based on a spatial dimension (e.g. “find me news stories about bush fires near Sidney”).
Many challenges of geographic IR involve geographical references (geo-references). Documents contain georeferences expressed in multiple languages which may or may not be the same as the query language. For example, the city Cape Town (English) is also Kapstadt (German), Cidade do Cabo in Portuguese and Ciudad del Cabo (Spanish). Queries with names may require an additional translation step to enable successful retrieval. Depending on the language and the culture, translation may not helpful in some cases. For example, the word new within New York is often translated in Spanish (Nueva York) and Portuguese (Nova Iorque), but never in German. On some occasions, names may be changed and a recent modification may not be well reflected within a foreign collection. E.g. there were still references to the German city Karl-Marx-Stadt in Spain after it had been renamed to Chemnitz in 1990. Geographical references are often ambiguous (e.g. there is a St. Petersburg also in Florida and Pennsylvania in the United States).
The query parsing (and classification) task was offered for the first time at GeoCLEF 2007. It is dedicated to identify geographic queries within a log from the msn search engine. This task has been organized by Xie Xing from Microsoft Research Asia. A log of 800,000 real queries was provided. Out of these, 100 were labeled as training data and 500 were assessed as test data. The task required participants to find the geographic entity, the relation type and the non geographic content restrictions. The task attracted six participating groups. The task design, the data, participation and evaluation results are discussed in a separate overview paper [Li et al. 2007]. The task is of high practical relevance to GeoCLEF and the real log data is of great value for research.
Search is the main task of GeoCLEF. The following sections describe the test design adopted by GeoCLEF. 2.1
The document collections for this year's GeoCLEF experiments consists of newspaper and newswire stories from the years 1994 and 1995 used in previous CLEF ad-hoc evaluations [Braschler & Peters 2004]. The Portuguese, English and German collections contain stories covering international and national news events, therefore representing a wide variety of geographical regions and places. The English document collection consists of 169,477 documents and was composed of stories from the British newspaper The Glasgow Herald (1995) and the American newspaper The Los Angeles Times (1994). The German document collection consists of 294,809 documents from the German news magazine Der Spiegel (1994/95), the German newspaper Frankfurter Rundschau (1994) and the Swiss newswire agency Schweizer Depeschen Agentur (SDA, 1994/95). For Portuguese, GeoCLEF 2007 utilized two newspaper collections, spanning over 1994-1995, for respectively the Portuguese and Brazilian newspapers Público (106,821 documents) and Folha de São Paulo (103,913 documents). Both are major daily newspapers in their countries. Not all material published by the two newspapers is included in the collections (mainly for copyright reasons), but every day is represented with documents. The Portuguese collections are also distributed for IR and NLP research by Linguateca as the CHAVE collection2.
In all collections, the documents have a common structure: newspaper-specific information like date, page, issue, special filing numbers and usually one or more titles, a byline and the actual text. The document collections were not geographically tagged and contained no semantic location-specific information. A total of 25 topics were generated for this year’s GeoCLEF (GC 51 - GC75). Topic creation was shared among the three organizing groups, who all utilized the DIRECT System provided by the University of Padua. A search utility for the collections was provided within DIRECT to facilitate the interactive exploration of potential topics. Each group created initial versions of nine proposed topics in their language, with subsequent translation into English. Topics are meant to express a natural information need which a user of the collection might have. These candidates were subsequently checked for relevant documents in the other collections. In many cases, topics needed to be refined. For example, the topic candidate honorary doctorate degrees at Scottish universities was expanded to topic GC53 scientific research at Scottish universities due to an initial lack of relevant documents in the German and Portuguese collections. Relevant documents were marked within the DIRECT system. After intensive discussion, a decision was made about the final set of 25 topics. Finally, all missing topics were translated into Portuguese and German and all translations were checked. The following section will discuss the creation of topics with spatial parameters for the track.
The organizers continued the efforts of GeoCELF 2006 aimed at creating a geographically challenging topic set. This means that explicit geographic knowledge should be necessary in order for the participants to successfully retrieve relevant documents. Keyword-based approaches should not be favored by the topics. While many geographic searches may be well served by keyword approaches, others require a profound geographic reasoning. We speculate that for a realistic topic set where these difficulties might be less common, most systems could perform better.
In order to achieve that, several difficulties were explicitly included into the topics of GeoCLEF 2006 and 2007: • ambiguity (St. Pauls Cathedral, exists in London and São Paulo) • vague geographic regions (Near East) • geographical relations beyond IN (near Russian cities, along Mediterranean Coast) • cross-lingual issues (Greater Lisbon , Portuguese: Grande Lisboa , German: Großraum Lissabon) • granularity below the country level (French speaking part of Switzerland, Northern Italy) • complex region shapes (along the rivers Danube and Rhine) However, it was difficult to develop topics which fulfilled all criteria. For example, local events which allow queries on a level of granularity below the country often do not lead to newspaper articles outside the national press. This makes the development of cross-lingual topics difficult.
For English topic generation, topics were initially generated by Mark Sanderson and tested on the DIRECT system. Additional consultation was conducted with other members of the GeoCLEF team to determine if the topics had at least some relevant documents in the German and Portuguese collections. Those found to have few such documents were altered in order ensure that at least some relevant documents existed for each topic. The German group at Hildesheim started with brain storming on interesting geographical notions. Challenging geographic notions below the country granularity were procured. We came up with German speaking part of Switzerland, which is a vaguely defined region. A check in the collection showed that there were sport events, but not enough to specify a sport discipline. Another challenge was introduced with Nagorno-Karabakh which has many spelling variants.
The Portuguese topics were chosen in a way similar to the one suggested for the choice of adhoc topics in previous years [cf. Santos & Rocha 2005]. The tripartite division among international, European and national, however, was reduced to national vs. international because we did not consider European as a relevant category (given that neither Portuguese nor English language newspaper collections used in CLEF are totally based in Europe): so, we chose some culturally-bound topics (Senna, crime in Grande Lisboa), some purely international or global (sharks and floods) and some related to specific regions (because of the geographic relevance to GeoCLEF).
In all cases, but especially for those focusing on a particular region (inside or outside the national borders covered by any newspaper collection), we tried to come up with a sensible user model: either a prospective tourist (St. Paul's or Northern Italy) or a cub reporter (Myanmar human rights violation or casualties on the Hymalaia). In some cases, we managed to create topics whose general relevance could be either, although naturally the choices would be different for the different kind of users -- consider the case of navigation in the Portuguese islands, both relevant for a tourist and for a journalist discussing the subject.
We were also intent on trying some specifically known geographically ambiguous topics, such as St. Paul's or topics where the geographical names were ambiguous with non geographic concepts, such as Madeira (means wood in Portuguese and can also mean a kind of wine).
All the topics were then tried out in the CHAVE collection, encoded in CQP [Evert 2005] and available for Web search through the AC/DC project [Santos & Bick 2000] at http://www.linguateca.pt/ACDC/ in order to estimate the number of possible hits. In general, there were very few hits for all topics, as can be appreciated by the number of relevant documents per topic found in the Portuguese pool (see Table/figure X??). The translation of the topics leads to new challenges. One of the English topics about the Scottish town, St. Andrews, was judged to be challenging as it was more ambiguous than in English, because Santo André also denotes a village in Portugal and a city in Brazil. So this is a case where depending on the language the kind of results expected is different. While we are not defending a user model where this particular case would be relevant, we are showing that a mere topic translation (as might be effected by a crosslingual system) would not be enough if one were interested in the Scottish St. Andrews alone.
Another interesting remark is the use of the word "continent", which is very much context dependent and again therefore cannot be translated simply from "continent" to "continente", because depending on your spatial basis the continent is different. Again this requires some clever processing and/or processing for the translation. Finally, it appears that perto de X (near X, or close to X) carries in Portuguese the presupposition that X is not included, and this made us consider that we would have translated better "airports near to London" by "que servem Londres" (i.e., that are used to reach London). (Although we also used the phrase aeroprtos londrinos which may also include airports inside London). On the other hand, airplanes clashes close to Russian cities seemed more naturally translated by "na proximidade" and not included. We used perto for both, but this might have been a translation weakness. 2.3
The format of GeoCLEF 2007 differed slightly from that of 2006 as no markup of geographic entities in the topics was provided. Systems were expected to reveal that information to themselves from the topic. Two examples of full topics are shown in Figure 1. <num>10.2452/58-GC</num> <num>10.2452/75-GC</num>
<title>Travel problems at major airports near to <title>Violation of human rights in Burma</title> London</title> <desc>Documents are relevant if they mention actual <desc>To be relevant, documents must describe violation of human rights in Myanmar, previously travel problems at one of the major airports close to named Burma.</desc> London.</desc> <narr>This includes all reported violations of human <narr>Major airports to be listed include Heathrow, rights in Burma, no matter when (not only by the Gatwick, Luton, Stanstead and London City present government). Declarations (accusations or airport.</narr> denials) about the matter only, are not relevant.</narr> </top> </top> As can be seen, after the brief descriptions within the title and description tags, the narrative tag contains detailed description of the geographic detail sought and the relevance criteria. In some topics, lists of relevant regions are given.
A tentative classification for geographical topics was suggested at GIR 2006 and applied at GeoCLEF2006 [Gey et al. 2007]: 1 non-geographic subject restricted to a place (music festivals in Germany) [only kind of topic in
GeoCLEF 2005] 2 geographic subject with non-geographic restriction (rivers with vineyards) [new kind of topic added in
GeoCLEF 2006] 3 geographic subject restricted to a place (cities in Germany) 4 non-geographic subject associated to a place (independence, concern, economic handlings to favour/harm that region, etc.) Examples: independence of Quebec, love for Peru (as often remarked, this is frequently, but not necessarily, associated to the metonymical use of place names) 5 non-geographic subject that is a complex function of place (for example, place is a function of topic) (European football cup matches, winners of Eurovision Song Contest) 6 geographical relations among places (how are the Himalayas related to Nepal? Are they inside? Do the
Himalaya mountains cross Nepal's borders? etc.) 7 geographical relations among (places associated to) events (Did Waterloo occur more north than the battle of X? Were the findings of Lucy more to the south than those of the Cromagnon in Spain?) 8 relations between events which require their precise localization (was it the same river that flooded last year and in which killings occurred in the XVth century?) This year we kept topics of both kinds 1 and 2 as last year. The major innovation and diversity introduced in GeoCLEF 2007 were more complicated geographic restriction than at previous GeoCLEF editions. The following three difficulties were introduced: 1 by specifying complex (multiply defined) geographic relations: East Coast of Scotland; Europe excluding the Alps, main roads north of Perth, Mediterranean coast, Portuguese islands, and "the region between the UK and the Continent"; by insisting on as politically defined regions, both smaller than countries, such as French speaking part of Switzerland, the Bosphorus, Northern Italy, Grande Lisboa, or larger than countries: East European countries, Africa and north western Europe; by having finer geographic subjects, such as lakes, airports, F1 circuits, and even one cathedral as place. 2.5
The participants used a wide variety of approaches to the GeoCLEF tasks, ranging from basic IR approaches (with no attempts at spatial or geographic reasoning or indexing) to deep natural language processing (NLP) processing to extract place and topological clues from the texts and queries. Specific techniques used included: • Ad-hoc techniques (weighting, probabilistic retrieval, language model, blind relevance feedback ) • Semantic analysis (annotation and inference) • Geographic knowledge bases (Gazetteers, thesauri, ontologies) • Text mining • Query expansion techniques (e.g. geographic feedback) • Geographic Named Entity Extraction (LingPipe, GATE, etc.) • Geographic disambiguation • Geographic scope and relevance models • Geographic relation analysis • Geographic entity type analysis • Term expansion using Wordnet • Part-of-speech tagging 2.6
English assessment was shared by Berkeley and Sheffield Universities. German assessment was done by the University of Hildesheim and Portuguese assessment by Linguateca. The DIRECT System was utilized for assessment. The system provided by the University of Padua allowed the automatic submission of runs by participating groups and supported assembling the GeoCLEF assessment pools by language.
English relevance assessment was conducted primarily by a group of ten paid volunteers from the University of Sheffield, who were paid a small sum of money for each topic assessed. The English document pool extracted from 53 monolingual and 13 bilingual (language X to) English runs consisted of 15,637 documents to be reviewed and judged by our 13 assessors or about 1,200 documents per assessor.
15,637 documents • 14,987 not relevant • 650 relevant 25 topics
• about 625 documents per topic 27 out of 66 submitted experiments • monolingual: 21 out of 53 submitted experiments • bilingual: 6 out of 13 submitted experiments 13 assessors
• about 1,200 documents per assessor The box plot of figure 2 shows the distribution of different types of documents across the topics of the English pool. In particular, the upper box shows the distribution of the number of pooled documents across the topics; as it can be noted, the distribution is a little bit asymmetric towards topics with a higher number of pooled documents and does not present outliers. The middle box shows the distribution of the number of not relevant documents across the topics; as it can be noted, the distribution is a little bit asymmetric towards topics with a lower number of not relevant documents and does not present outliers. Finally, the lower box shows the distribution of the number of relevant documents across the topics; as it can be noted, the distribution is almost symmetric; with a median number of relevant documents around 20 per topic, but it present some outliers, which are topics with a large number of relevant documents.
While judging relevance was generally easier for the short news agency articles of SDA with their headlines, keywords and restriction to one issue, Spiegel articles took rather long to judge, because of their length and essay-like stories often covering multiple events etc. without a specific narrow focus. Many borderline cases for relevance resulted from uncertainties about how broad/narrow a concept term should be interpreted and how explicit the concept must be stated in the document. One topic required systems to find documents which report shark attacks. Documents telling the reader that a certain area is “full of sharks” were not judged as relevant. For other topics, implicit information in the document was used for the decision. For example, the topic sport events in German speaking Switzerland led to documents where the place of a soccer game was not mentioned, but the result was included in a standardized form which indicates that the game was played in the first city mentioned (e.g. Lausanne - Genf 0:2, has most usually been played in Lausanne). It was also assumed that documents which report that hikers are missing in the Himalayas are relevant for the topic casualties in the Himalayas. 15,488 documents • 14,584 not relevant • 904 relevant 25 topics
• about 620 documents per topic 24 out of 24 submitted experiments • monolingual: 16 experiments • bilingual: 8 experiments 8 assessors
• about 1,900 documents per assessor Many documents are at first identified as borderline cases and need to be discussed further. One topic requested topics on travel delays at London airports. One document mentioned that air travel had been delayed and some flight had to be directed to Gatwick. Because a delay at Gatwick is not explicitly mentioned, the document was regarded as not relevant.
The box plot of figure 3 shows the distribution of different types of documents across the topics of the German pool. As it can be noted the distribution of the pooled documents is almost symmetrical with no outlier; on the other hand, the distribution of not relevant documents is asymmetrical with a tail towards topics with a lower number of not relevant documents and does not present outliers; finally, also the distribution of the relevant documents is asymmetrical but towards topics with a greater number of relevant documents and presents outliers, which are topics with a great number of relevant documents.
In addition to the problem (already reported before) that some if the news articles included in the CHAVE collection are in fact a list of "last news" which concern several different subjects (and have therefore to be read in their entirety, making it especially tiresome), we had some general problems assessing topics, which we illustrate here in detail for the "free elections in Africa" subject: What is part of an election (or presupposed by it)? In other words, which parts are necessary or sufficient to consider that a text talks about elections: campaign, direct results, who were the winners, "tomada de posse", speeches when receiving the power, cabinet constitution, balance after one month, after more time... In fact, how far in time is information relevant? For example, does mention to the murder of the first democratically elected president in Ruanda qualify as text about free elections in Africa? And if elections took place and were subsequently annulated as in Argelia, do they count as elections or not? Also, how much indirectly conveyed information can be considered relevant? A text about the return of Portuguese citizens to Portugal after the (free) South African elections is about free elections in South Africa? And what to do if in the text no mention is made to whether the elections were free or not? Are we to assume anything? As in the case of a text about Uganda mentioning "voltou à Presidência no fim de 1980, pela via eleitoral" (X came back to presidency through the electoral path). Are either our knowledge or our opinions going to play a role on the relevance assessment, or we are supposed to just look at the document and not bring our own bias? Finally, how much difference of opinions is relevant to a topic? Consider the following piece of news "Savimbi considera ilegais as eleições consideradas livres e justas pela ONU..." (Savimbi considers illegal the elections considered free and just by UN). Are we to stand with UN or with Savimbi, as far as the elections in Angola are concerned? (In our opinion, this text is very relevant to the subject, anyway, since it mentions, and discusses, precisely the issue of "free elections in an African country".) Due to this (acknowledged) difficulty of assessing relevance for some topics, it would have been beneficial to have a pool of judges assessing the same documents and produce a relevance cline. Although this is currently not possible with the DIRECT system, it might make sense in the future, especially for more evaluative topics that involve complex issues. 15,572 documents • 14,810 not relevant • 762 relevant 25 topics
• about 623 documents per topic 18 out of 18 submitted experiments • monolingual: 11 experiments • bilingual: 7 experiments 6 assessors
• about 2,600 documents per assessor The box plot of figure 4 shows the distribution of different types of documents across the topics of the Portuguese pool. As it can be noted the distribution of the pooled documents is a little bit asymmetrical towards topics with a lower number of pooled document and presents both upper and lower outliers, i.e. topics with many or few pooled documents; on the other hand, the distribution of not relevant documents is almost symmetrical with an outlier, which is a topic with few not relevant documents; finally, also the distribution of the relevant documents is asymmetrical towards topics with a greater number of relevant documents and presents outliers, which are topics with a great number of relevant documents.
The results of the participating groups are reported in the following sections. As shown in Table 6, a total of 13 groups from 9 different countries submitted results for one or more of the GeoCLEF tasks. A total of 108 experiments were submitted. Table 8 provides a breakdown of the experiments submitted by each participant for each of the offered tasks.
DE EN PT
X2DE X2EN X2PT
Five different topic languages were used for GeoCLEF bilingual experiments: German, English, Indonesian, Portuguese, and Spanish. Differently from usual, the most popular language for queries was Spanish (11 experiments out of 28 bilingual experiments); English (7 experiments) and Indonesian (6 experiments) almost tied for the second place; German (2 experiments) and Portuguese (2 experiments) tied for the third place. The number of bilingual runs by topic language is shown in Table 9. Monolingual retrieval was offered for the following target collections: English, German, and Portuguese.
Fig. 6. Monolingual German top participants. Interpolated Recall vs. Average Precision. The bilingual task was structured in four subtasks (X → DE, EN, or PT target collection). Table 11 shows the best results for this task with the same logic of Table 7. Note that the top five participants contain both “newcomer” groups and “veteran” groups. Note that there is a significant improvement for Bilingual German since CLEF 2006, when it was 70% of the best monolingual system; Bilingual English shows a small improvement, with respect to the 74% of the best monolingual system in CLEF 2006; finally, Bilingual Portuguese is quite surprising since it outperforms the monolingual and it represents a complete overturn with respect to the 47% of CLEF 2006.
The test collection of GeoCLEF grew of 25 topics each year. This is usually considered the minimal test collection size to produce reliable results. Therefore, statistical testing and further reliability analysis are performed to assess the validity of the results obtained. The range of difficulties in the topics might have led to topics more difficult and more diverse than in traditional ad-hoc evaluations. To gain some insight on this issue, a topic performance analysis was also conducted. 4.1
Statistical testing for retrieval tests is intended to determine whether the order of the systems which results from the evaluation reliably measures the quality of the systems [Buckley & Voorhees 2005]. In most cases, the statistical analysis gives an conservative estimate of the upper level of significance [Sanderson & Zobel 2005]. We used the MATLAB Statistics Toolbox, which provides the necessary functionality plus some additional functions and utilities. We use the ANalysis Of VAriance (ANOVA) test. X X
X X X X X X X X X X X X X X X X X X X X X X
X X X X X X X X X X X X X X X X X
X X X X X X X X X X X X X X
X X X X X X X X X X X X X X
X X X X X X X X X X X X X
X
X X X X
X GeoCLEF 2007 has continued to create an evaluation resource or geographic information retrieval. Spatially challenging topics have been developed and interesting experiments have been submitted. The test collection developed for GeoCLEF is the first GIR test collection available to the GIR research community. GIR is receiving increased notice both through the GeoCLEF effort as well as due to the GIR workshops held annually since 2004 in conjunction with the SIGIR or CIKM conference. All participants of GeoCLEF 2007 are invited to actively contribute to the discussion of the future of GeoCLEF.
German assessment was based entirely on volunteer effort; it was carried out by Julia Jürgens, Theresa Märtl, Ben Heuwing, Frauke Zurek, Robert Stoldt, Thomas Mandl, Jens Plattfaut, and Rafael Hellmann of the University of Hildesheim. The English assessment was performed by Fredric Gey, Ray Larson, Mark Sanderson and the Sheffield student team (paid for by the EU 6th Framework project Tripod). The Portuguese documents were assessed by Paulo Rocha, Luís Costa, Luís Miguel Cabral, Susana Inácio, Maria Cláudia Freitas and Diana Santos, in the scope of the Linguateca project, jointly funded by the Portuguese Government and the European Union (FEDER and FSE) under contract ref. POSC/339/1.3/C/NAC. The topics were thoroughly checked by Sven Hartrumpf from the University of Hagen (Germany). The future direction and scope of GeoCLEF will be heavily influenced by funding and the amount of volunteer effort available.