This paper describes the participation of the IICS group at the GeoCLEF task of the CLEF campaign 2006. We describe different retrieval experiments using a separate index for location names and identifying and indexing of metonymic location names differently. The setup of our GIR system is a modified variant of the setup for GeoCLEF 2005. We apply a classifier for the identification of metonymic location names for preprocessing the documents. This classifier is based on shallow features only and was trained on manually annotated data from the German CoNLL-2003 Shared Task corpus for Language-Independent Named Entity Recognition and from a subset of the GeoCLEF newspaper corpus. After preprocessing, documents contain additional information for location names that are to be indexed separately, i.e. LOC (all location names identified), LOCLIT (location names in their literal sense), and LOCMET (location names in their metonymic sense). To obtain an IR query from the topic title, description, and narrative, we employ two methods. In the first method, a semantic parser analyzes the query text and the resulting semantic net is transformed into database query. The second method uses a Boolean combination of a bag-of-words (consisting of topical search terms) with location names. The results of our experiments can be summarized as follows: excluding metonymic senses of location names improves mean average precision (MAP) for most of our experiments. For experiments in which this was not the case, a more detailed analysis showed that for some topics the precision increased. Our experiments show that the additional use of topic narratives decreases MAP. For almost all experiments with the topic narrative, lower values for MAP and for the number of relevant and retrieved documents were observed. However, query expansion and the use of separate indexes improves the performance of our GIR application.
There are several essential tasks a geographic information retrieval (GIR) application has to perform, for
example the identification and disambiguation of location names (see [
This paper presents an application of a classifier for literal and metonymic senses of location names. The classifier is trained on manually annotated data and uses shallow features from the textual context of location names only. We create different indexes corresponding to different senses of location names and investigate in a baseline experiment if utilizing a separate index containing location names will improve performance in GIR. The focus in our experiments lies on metonym identification in documents, because the GeoCLEF query topics did not contain any metonymic location names in topic titles, descriptions, or narrative. 2
Our GIR system is based on the same system that was developed for GeoCLEF 2005 (see [
For the identification of metonymic location names, we employed a classifier trained on manually
annotated data. The data consists of a subset of the German CoNLL-2003 Shared Task corpus for
LanguageIndependent Named Entity Recognition [
Two methods were employed to obtain an IR query from a topic title, description, and narrative: In the
first method, the WOCADI parser is applied to perform a deep linguistic analysis of the query text and the
resulting semantic net is transformed into a database independent query representation (DIQR, see [
<DOC> <DOCID> ... </DOCID> <TEXT>At the meeting of France and Germany in Lisbon last year, Paris vetoed the decision. ... </TEXT> <LOCALL>France Germany Lisbon Paris</LOCALL> <LOCLIT>Lisbon</LOCLIT> <LOCMET>France Germany Paris</LOCMET> </DOC>
In our baseline experiment (FUHddGNNNTD), the standard IR model (tf-idf ) was utilized without additions or modifications. In experiments not accessing the separate name index, the location names were searched within the index for the TEXT field; in experiments using the separate index (e.g. FUHddGYYYTDN) location names were looked for in the index for the LOC field. For experiments with metonymy identification, the index for location names corresponds to the field LOCLIT, i.e. only location names in their literal sense were searched for. 3
The parameters used in our GeoCLEF experiments can be described as follows:
• BOP: Boolean operator for the combination of topic and location subquery
• LI: use separate location index (LOC or LOCLIT, as described in Section 2) vs. use TEXT index
• LA: apply deep linguistic analysis (WOCADI parser [
Using a separate index for location names leads to a better performance in our experiments. Additional experiments confirmed that the query expansion and the provided background knowledge have significant influence on results. As a preparation for the GeoCLEF 2006 experiments, we analyzed data from GeoCLEF 2005, in particular (but not exclusively) those topics for which we did not find any of the relevant documents. We modified the knowledge base containing our background knowledge to include some missing facts1 and performed manual searching. For example, for topic GC019 (“European golf tournaments”), 1 For the GeoCLEF 2006 experiments, the unmodified knowledge base from GeoCLEF 2005 was used.
BOP OR OR OR OR OR OR AND AND AND AND BOP OR OR OR OR OR OR
LI N Y Y N Y Y Y Y Y Y LI N N Y Y Y Y
N Y Y N Y Y Y Y Y Y N N Y Y Y Y
N Y N N Y Y N N Y Y N N Y N Y Y
N N N N N Y N Y N Y N N N N N Y we found that with only slight differences in query formulation the MAP as well as the number of relevant documents increased. For topic GC019, 41 of 61 relevant documents were found after modifying the query. As there is now more data (topics and relevance assessments) available, we hope to identify regularities for search failures more easily.
As we observed from results of the GeoCLEF 2005 experiments, query expansion with meronyms leads to significantly better precision (0.1466 MAP vs. 0.1608 MAP; 0.1718 MAP vs. 0.1953 MAP) for most monolingual German runs, although recall is slightly worse. A detailed analysis of the runs FUHddGYYYTDN an FUHddGYYYMTDN shows that in this case, the metonymy identification task added this year improves the results for some topics.
Results for the bilingual experiments were found to be lower in general. There were a few errors in the translated topic titles, descriptions, and narratives. For some topic titles, there does not seem to be enough textual context to provide an adequate translation (topic titles and descriptions were translated separately).
One hypothesis to be tested was that the additional information in topic narratives (runs with QEX=TDN instead of TD) would improve results. We can not confirm this assumption because with our setup, MAP and relevant and retrieved documents are almost always lower for runs using the topic narrative than for runs with topic title and description. 5
We expected to find a significant increase in precision for all GIR experiments excluding metonymic senses
of location names for a search. This assumption holds for most experiments, conforming results of earlier
experiments [
A different explanation for a low performance might be that in our setup for GeoCLEF 2006, the location name index does not contain terms representing adjectives, language, or inhabitants for a given location name (e.g. the terms like Dutch, Spanish, or Spaniard do not occur in the location name index). Furthermore, instead of removing all metonymic senses for location names from the index, the corresponding terms should be indexed differently (with a lesser weight or with a different sense).
Additional information from the topic narratives did not improve precision or recall in our experiments,
although one might have expected a similar effect as for query expansion with meronyms. We plan to rerun
experiments with a state-of-the-art database management system (DBMS), such as Cheshire 3 [