This paper describes the system developed by the Language Technologies Laboratory of INAOE for the Geographical Information Retrieval task of CLEF 2008. The presented system focuses on the problem of ranking documents in accordance to their geographical relevance. It is mainly based on the following hypotheses: (i) current IR machines are able to retrieve relevant documents for geographic queries, but they can not generate a pertinent ranking; and (ii) complete documents provide more and better elements for the ranking process than isolated query terms. Based on these hypotheses, our participation at GeoCLEF 2008 aimed to demonstrate that using some query-related sample texts it is possible to improve the final ranking of the retrieved documents. Experimental results indicated that our approach could improve the MAP of some sets of retrieved documents using only an average of two sample texts. These results also showed that the proposed approach is very sensitive to the presence of irrelevant sample texts as well as to the ambiguity of geographical terms.
Geographic Information Retrieval (GIR) is aimed at the retrieval of documents based not only on
conceptual keywords, but also on spatial information (i.e. geographical references) [
Recent development of GIR systems [
The proposed method was evaluated in the Monolingual English exercise of the 2008 GeoCLEF task. In particular, the purposes of our experiments were twofold: first, to confirm that traditional IR machines can achieve high recall levels, and second, to probe that using some query-related sample texts allow improving the original ranking of the retrieved documents.
The following section describes the general architecture of our GIR system as well as the main functionality of each one of its components. Then, Section 3 shows the results from a subset of the official submitted runs to GeoCLEF 2008. Finally, Section 4 presents some conclusions and describes ongoing work. 2
The core module of the proposed system is the information retrieval (IR) machine. In our architecture, this module is used two times: in a first iteration, it retrieves a set of relevant documents using the original given query, then, in a second iteration, it retrieves a larger set of relevant documents considering an expanded query.
For the experiments, the IR machine was implemented using the Lemur open source search engine1, and the documents and query were preprocessed by applying stopword removal and suffix stripping. 2.1.2
The objective of this module is to select some relevant items from the set of retrieved documents. We call these items sample texts, and use them for two different purposes. On the one hand, to modify the original query and perform a second IR process, and on the other hand, to re-rank the set of retrieved documents.
The implementation of this module was based on the blind relevance feedback (BRF) technique
[
This module takes as input the set of sample texts and extracts from them a set of relevant terms. Then, it uses these terms to expand the original query. The idea of is to provide more information to the IR machine in order to improve its performance, in particular, to improve its recall rate.
Similar to the previous module, in this case we also implemented this module following the
suggestions by Chen [
The expanded query is sent to the IR machine, and a new set of documents is retrieved. Finally, this new set of documents is analyzed by the re-ranking process and the output of the system is generated. 2.1.4
This module is the main contribution of our system. Its goal is to re-rank the set of retrieved documents using the information (thematic and geographic) contained in the query-related sample texts. Figure 2 shows the main processes of this module.
Before describing each process it is important to mention that, in theory, all sample texts are relevant to the query; nevertheless, in practice, this condition not always happens. Because of that, the final process of this module considers the integration of evidence from different sample texts. Geo-Expansion. The objective of this process is to expand the geographical terms (geo-terms) contained in the sample texts. Mainly, it adds to each geo-term its two nearest ancestors. For instance, if the term Madrid occurs in a sample text, then the document is expanded using the terms Spain and Europe. By doing this geographic expansion, it is expected to identify a major number of query-related documents.
In order to perform the geographic expansion we employed the Geonames database2. Similarity Calculation. In this process the set of retrieved documents is compared against each one of the sample texts, generating, in this way, several different rankings of the former documents. The comparison of the documents considers both, their thematic and geographic information. In particular, the similarity between two documents is computed as indicated by formula 13.
SQ(s, r) = (λ × SQthematic(s, r)) + ((1 − λ) × SQgeographic(s, r)) (1) where s represents a sample text, r represents a document from the set of retrieved documents, and λ is a weighting value.
In order to be able to compute the thematic and geographic similarities, it was necessary
to identify all geographical named entities from the given documents. For this purpose, in
our experiments we used the LingPipe tool4. It is also important to mention that both
similarities were computed using the cosine formula and that the λ coefficient was set to
0.7 in order to give more importance to the thematic similarity. This decision was based
on evidence from previous GeoCLEF tracks [
Information Fusion The goal of this process is to merge, into one single list, all the information
contained in the different ranking proposals. In order to generate this single ranking we
employed techniques of information fusion [
This section describes the results from a subset of our experiments evaluated at GeoCLEF 2008. For all of them, we preprocessed the documents and queries as described in Section 2.1.1, and used the following evaluation measures: Mean Average Precision (MAP), R-Precision, Precision at the first 5 documents (P@5) and Overall Recall.
Table 1 shows the results from our four baseline runs. The first two rows correspond to the results of the first IR iteration (refer to Figure 1). In this case, the run inaoe-BASELINE1 employed the title and description fields, whereas the run inaoe-BASELINE2 used all available information: title, description and narrative. As it can be seen, adding the information from the narrative did not help the IR process, on the contrary, the best performance was achieved by exclusively using the title and description fields.
The third and fourth rows of Table 1 indicate the results achieved in the second IR iteration. For these two experiments, we expanded the original query using the K most frequent terms from the top N documents retrieved by the inaoe-BASELINE1 run. We named these experiments as inaoe-BRF-N-K. As expected, the query expansion process allows both configurations to obtain better results than the BASELINE1, especially for the case of the recall rate.
Table 2 shows the results achieved by the proposed method. The results correspond to the
following experiment configurations: if the name of the experiment contains the tag RRBF, it
means that we re-ranked the retrieved documents making no distinction between the thematic and
geographic parts; if the name contains the tag RRGeo, it means that we re-ranked the retrieved
documents making a distinction between the thematic and geographic parts, but without applying
2http://www.geonames.org/
3Some research groups have used similar ways for computing the similarities among documents [
From the results of Table 2, we can observe that the distinction between the thematic and geographic parts allowed to obtain a better performance. It is also possible to notice that the MAP differences between the experiment tagged as RRGeo and the one tagged as RRGeoExp was not very significant. We believe that this performance happened due to the noise introduced by our nave geo-expansion process as well as by the errors from the incorrect selection of the sample texts. The goal of these additional experiments was to evaluate the effect of using only relevant sample texts in the re-ranking process. In order to accomplish this evaluation, these experiments considered the manual selection of the sample texts. In particular, we defined as sample texts only those relevant documents among the first five positions from the first IR iteration (i.e., from the first five documents of the inaoe-BASELINE1 run). Table 3 shows the results from these experiments. It is important to point out that in all cases the average number of sample texts was of two. Therefore, Table 3 can be read as:“By taking only two sample texts, and making no distinction between thematic and geographic parts, we could reach a MAP of 0.306”. These results show that the proposed method works well, but also indicate that it is very sensitive to the presence of incorrect sample texts. The results from our participation at GeoCLEF 2008 showed that the use of query-related sample texts allows improving the original ranking of the retrieved documents. Nevertheless, they also showed that the proposed method is very sensitive to the presence of incorrect sample texts, and that it is also affected by the incorrect expansion of the geographical terms.
Our current work is mainly focused on tackling these drawbacks. In particular, we are working in a new sample-text selection method which is based on the idea of keeping only those documents that have a high density levels among its geographic and thematic terms, and in a new strategy for geographic expansion that considers on the one hand the inclusion of a more precise disambiguation strategy, and on the other hand, including the use of geographical coordinates. Acknowledgements. This work was done under partial support of CONACyT (scholarship 165545), SNI and INAOE.