Berkeley group 2 participated in the GeoCLEF pilot track of CLEF 2005 by performing English and German monolingual geographic information retrieval and bilingual geographic information retrieval using English topics to retrieve German documents and vice versa. Our results were substantially above the medians for all categories. We found that major improvements for German retrieval derived from successful blind feedback (between 53 and 72 percent improvement in MAP depending upon the run type) rather than decompounding of German compound words. Manual expansion of geographic references (e.g. Europe to individual country names) was disastrous to retrieval performance. Clearly, successful geographic information retrieval will require substantially more research to improve upon heuristic approaches.
GeoCLEF is a new track for the CLEF 2005 campaign and workshop. Geographic Information Retrieval (GIR) tries to exploit geographical references in text for targeted and improved retrieval. The topics presented in GeoCLEF differ from the usual ad-hoc topic layout by including additional tags customized for GIR. Each topic had a concept tag, a spatial relation tag and one or more location tags augmenting the usual title, description and narrative query structure. GeoCLEF provided English and German document collections and English, German, Portuguese and Spanish topic files. Berkeley group 2 participated in monolingual retrieval for both English and German as well as bilingual retrieval from English to German and vice versa. We experimented with the additional location-specific tags by including them into the query formulation and by manually manipulating the location tag to add more specific location names. For example, the location “Europe” was expanded to include the names of the individual countries in Europe. In all cases our retrieval was augmented with blind feedback. algorithm developed by Aitao Chen [2,3], adding 30 top-ranked terms from the top 20 ranked documents of an initial ranking. Thus the sequence of processing for retrieval is: query stopword removal (decompounding) stemming ranking blind feedback. For German runs, we used a decompounding procedure developed and also described by Aitao Chen in [2, 3], which has been shown to improve retrieval results. The decompounding procedure looks up document and query words in a base dictionary and splits compounds when found. We discuss the impacts of German decompounding and blind feedback in the results section below.
For monolingual retrieval, we submitted one title and description run, one run with title, description and narrative, one with title, description, concept and location tag and one with title, description, concept and the manually expanded location tag.
In English monolingual, adding the geographical tags (BKGeoE1) achieved the highest result with a MAP of 0.3936, but the manual expansion strategy did not improve the average precision (BKGeoE4 0.3550). The TDN run (BKGeoE3) outperforms the TD run (BKGeoE4) by 8% and improves from 0.3528 to 0.3840. In German monolingual retrieval, 4 topics did not retrieve any relevant documents overall. Additionally, our runs failed to retrieve any relevant documents for 3 more of the remaining 21 queries. Manually adding location information lowered the average precision score considerably. The TDN run (BKGeoD3) achieved the highest MAP with 0.2042 followed by the TD run (BKGeoD2) with 0.1608. The manual expansion strategy (BKGeoD4) achieved the lowest MAP (0.1112), whereas adding the tags achieved a MAP of 0.1545. Because a significant proportion of topics retrieved very few relevant documents from the German collection, this might explain these low precision scores.
For bilingual retrieval, we used the L&H Power Translator Pro to translate the topics from English to German and vice versa. In bilingual retrieval, adding the concept and location information improved the average precision score modestly. For English German, adding the concept and location tag improved precision from 0.1685 to 0.1788, a performance that is better than the same strategy in monolingual retrieval! For German English, adding the tags improved the average precision from 0.3586 (this TD run is even slightly better than the monolingual one) to 0.3715 in average precision.
Since our best results were considerably above an average of medians for both English and German monolingual and bilingual runs, we ran an additional set of experiments to see if we might isolate the effects of blind feedback and (for German) decompounding. What we found was that there was little effect of blind feedback on the English monolingual and German English bilingual results. Without blind feedback, English monolingual title-description (TD) run mean average precisions are virtually indistinguishable, while blind feedback titledescription plus concept-location is about 5% better (0.3936 versus 0.3737). The blind feedback results for English are summarized in Tables 1a (monolingual) and 1b (bilingual German English):
There is however, considerably greater impact of blind feedback on German monolingual and bilingual results, as Tables 2a and 2b show, on the order of 53 to 72 percent improvement:
To try to understand how blind feedback produced such stunning improvement in results, we need to make a more detailed examination of improvement produced for each topic. Table 3 presents MAP of our German monolingual runs for each topic, with Median, official TD and TD without blind feedback highlighted The four queries, where query expansion through blind feedback achieved the most improvement were 10 (Hochwasser in Holland und Deutschland, BF strategy improves by 1400%), 14 (Umweltschädigende Vorfälle in der Nordsee, BF improves by 650%) and 19 (Golfturniere in Europa, BF improves by 285%) and 13 (Besuche des amerikanischen Präsidenten in Deutschland, 168%). Query 12 is an example where blind feedback has a negative effect on the average precision scores (Kathedralen in Europa, -67%).
The blind feedback algorithm adds 30 terms to the query, which are weighted half compared to the original query terms in retrieval. “Good” terms to be added are terms that are relevant to the query and add new information to the search, for example synonyms of query terms but also proper names or word variations. The most improved queries seem to add mostly proper names and word variations and very few irrelevant words that won’t distort the search towards another direction.
For query 10, some of the words added by blind feedback were Hochwassergebiet (flooded area), Waal, Maas (rivers in Holland), Deich (levee) and Flut (flood) – all words that didn’t occur in the title and description tags of the original query but are eminently important words for the search.
For query 12, only a few original query words (after stopword removal) were fed into the blind feedback algorithm: Kathedrale, Europa, Artikel and einzeln, of which the last two don’t add relevant information to the search. Consequently, the suggested blind feedback terms don’t really fit the query (e.g. Besucherinnen (female visitors), kunstvoll (artful), Aussöhnung (reconciliation), Staatsbesuch (state visit), Parade). The more words are used to feed the blind feedback algorithm and the more distinctive they are in terms of occurrence in the collection and connectedness to a certain concept, the better the blind feedback algorithm will work. For example, the word Golfturnier doesn’t occur very frequently in the collection but it always co-occurs with articles that are related to golf, whereas Besucherinnen will be used in more frames (concepts) than just the European cathedrals. The combined queries 10, 13, 14, 19 account for almost all of the improvement in the average precision score between the run without blind feedback and the run with blind feedback. This is a thought provoking fact because for the rest of the queries the impact of the blind feedback terms in precision for each query centers around zero. We have found over and over again that blind feedback improves precision, but it seems to do so for only a particular kind of query.
Manual expansion of general geographic regions to individual country names was a clear losing strategy. For topics 2 and 4, expanding the German location name “Europa” to Frankreich, Griechenland, Grossbritannien, Holland, Italien, Irland, Malta, Österreich, Portugal, Russland, Skandinavien, Dänemark, Finnland, Norwegen, Schweden, Spanien, Tschechien, Ungarn, Zypern, Belgien, Bulgarien, Deutschland, Großbritannien, Kroatien, Liechtenstein, Litauen, Monaco, Niederlande, Polen, Schweiz and Slowenien turned reasonable retrievals go to zero precision for those topics. Similarly poor results were obtained from equivalent English monolingual expansion of “Europe.” For topic 3, “Latin America” was expanded to 42 country names with equally dismal results. This does not bode well for using a geographic thesaurus to automatically obtain such expansions.
Berkeley Group 2 participated in the GeoCLEF track with a focus on the German and English languages for both documents and topics. We utilized standard information retrieval techniques of blind feedback and German complex word decompounding. Query translation used commonly available machine translation software. Blind feedback was particularly impressive in improving German monolingual and bilingual English German results. Our venture into geographic location resolution by manually expansion of the general terms “Europe” and “Latin America” into a list of individual country names resulted in a considerably diminished performance effectiveness. It seems that successful geographic expansion will only occur in the context of requiring the concept (e.g. “Golf Tournaments”) to also be present in the documents. This may argue for a generalized Boolean approach to retrieval.
Thanks to Aitao Chen for implementing and permitting the use of the logistic regression formula for probabilistic information retrieval as well as German decompounding and blind feedback in his MULIR retrieval system.