This paper describes the rst participation of the SINAI (Intelligent Systems of Access Information) group of the University of JaØn in GeoCLEF 2006. We have developed a system made up of three main modules. The rst one is the translation subsystem, that works with queries into Spanish, Portuguese and Deutsche. The second one is the query expansion subsystem, that integrates a Named Entity Recognizer, a Gazetteer, a Thesaurus expansion module and a Geographical information module. The last subsystem is the Information Retrieval module, that works with collections and queries into English, and returns the result le. We have made several runs, that combines these modules to resolve the monolingual and the bilingual tasks. The results obtained shown that the use of geographical and thesaurus information for query expansion does not improve the retrieval, but this is the rst step to try to improve the system in the future.
The main objective of our rst participation in GeoCLEF have been the study of the problem of this task, and the development of a system that returns relevant documents.
The Cross-Language Geographic Information Retrieval (GIR) system developed at the University of JaØn has been designed to retrieve relevant documents that contain geographic tags.
For this reason, our system consist of several modules: Translation, Named Entity RecognitionGazetteer, Geographical Information Subsystem and Thesaurus Expansion Subsystem.
This paper is organized as follows: section 2 describes the whole system and each module of the system in detail. Then, in the section 3 experiments and results are described.
Finally, the conclusions about our participation in GeoClef 2006 are expounded, and some future work. 2
We propose a Geographical Information Retrieval System that is made up of ve subsystems: ² Translation Subsystem: is the query translation module. This subsystem translates the queries into the other languages. ² Named Entity Recognition-Gazetteer Subsystem: is the query geo-expansion module.
This subsystem uses the geographical information from Geographical Information module. ² Geographical Information Subsystem: is the module that stores the geographical data.
This information has been obtained from Geonames1 gazetteer. ² Thesaurus Expansion Subsystem: is the query expansion module using an own Thesaurus. ² IR Subsystem: is the Information Retrieval module. We have used the LEMUR IR system2.
The combination of these subsystems gives the results of the di erent runs, and will be shown in section 3. 2.1
The purpose of the Translation Subsystem is to translate the queries or topics that are not in English.
This module is used for the following bilingual tasks: Spanish-English, Portuguese-English and German-English.
For the translation we have used an own module, called SINTRAM (SINai TRAnslation Module), that works with several online Machine Translators, and implements several heuristics. In this case we have used an heuristic that joins the translation of a default translator (the one that we indicate, depending of the pair of languages) with these words that have another translation (using another translator).
SINTRAM works with the following online translators: ² Epals: available at http://www.epals.com ² Prompt: available at http://translation2.paralink.com ² Reverso: available at http://www.reverso.net ² Systran: available at http://www.systransoft.com 1http://www.geonames.org 2http://www.lemurproject.org The main goal of NER-Gazetteer Subsystem is to detect and recognize the entities in the queries, in order to expand the topics with geographical information. We are only interested in geographical information, so we have used only the locations detected by the NER module. The location term includes everything that is town, city, capital, country and even continent.
As we will see in the next section, the information about locations is loaded previously in the Geographical Information Subsystem, that it is related directly to the NER-Gazetteer Subsystem.
Figure 1 describes the system architecture with this relation. The NER-Gazetteer Subsystem recognizes the entities and provides this information to the Geographical Information module.
We have used the NER system that GATE3 provides.
The basic operation of NER-Gazetteer module is the following: ² The rst step is the preprocessing phase. Each query is preprocessed using a tokenizer, a sentence splitter and a POS tagger. The NER system we have used needs this information in order to improves the named entity detection and recognition. ² The second step is the extraction of the title of each topic and the submission to the NER.
The result is saved to another labelled topic le with the location entities labelled. ² The last step is the detection of the geographical places, that the NER module have not detected. For this proposal we have use a Gazetteer, included also in GATE. We also
include this information in an expanded topic, using an XML label.
The NER-Gazetteer Subsystem generates some labelled topics, base on the original one, adding the locations. 2.3
The objective of this module is to expand the locations of the topics, using geographical
information. We have used automatic query expansion [
Geonames integrates geographical data such as names, altitude, population and others from various sources.
Some examples of queries that receives this module are: ² Find the capital of a country whose population is greater than X. ² Find ve cities from a country whose population is greater than X. ² Find the country name of a city. ² Find the latitude and longitude from a location.
When a location is recognized by the NER subsystem we look for in the Geographical Information Subsystem.
In addition, it is necessary to consider the spatial relations found in the title ( near to( , within X miles of( , north of( , south of( , etc.). Depending on the spatial relations, the search in the Geographical Information subsystem is more or less restrictive.
We also have to verify if the location is a city, a country or a continent. Depending on the location type, the expansion will become of the following way: ² If the location is a continent, we expand with the capitals of countries that belong to that continent, and with capitals if the population is greater than a number of habitants (one of ours parameters). The expansion is not very large in order to avoid noise in the recovery process. ² If the location is a country, we expand with the ve most important cities (with greater population) of that country. ² If the location is a city or capital, rst we veri ed if there is some spatial relation in the topic. If exists we use the latitude and longitude information to nd other relevant locations and we expand the topic with them. If there is not some spatial relation, we expand the topic only with the name from the country to which city belongs.
Finally we add the locations given back by the Geographical Information Subsystem to the topic title.
To adjust the parameters of this module, for instance the number of habitants to consider a relevant capital or the number of cities to expand, we have made experiments with the GeoCLEF 2005 framework. 2.4
A collection of thesauri was generated from the GeoCLEF training corpus. We were looking for words with a very high rate of document co-location. These words will be treated like synonyms and added to the topics.
For that, we generated an inverse le with the entire corpus. The le has a row for each
di erent word of the corpus. Following each word appear all the current word frequencies for each
corpus le. These rows can be treated with the standard TF.IDF [
We probe this method with the GeoCLEF 2005 corpus and we found that a cosine similarity great than 0.9 between words was the rate that obtain best precision/recall results.
The same procedure was applied to the 2006 corpus. The thesauri collection was generated for all the names of the topics and all the thesauri words were added to its topic. The Figure 2 shows the calculated thesauri for the topics GC033 and GC034. We can see the topic code and the pairs word-similarity. The English collection dataset has been indexed using LEMUR IR system. It is a toolkit4 that supports indexing of large-scale text databases, the construction of simple language models for documents, queries, or subcollections, and the implementation of retrieval systems based on language models as well as a variety of other retrieval models.
The English collection include a variety of topics and geographical regions form news stories between 1994 and 1995.
Previously the English collection provided for GeoClef 2006 have been preprocessed, using the
English stopwords list and the Porter stemmer [
Each topics set, the monolingual expanded and the bilingual translated and also expanded, is run to LEMUR.
One parameter for each experiment is the weighting function, such as Okapi [
² We use the original English topics set ² We preprocess each topic (stopper and stemmer) ² Topics without expansion (geographical or thesaurus) ² Information Retrieval without PRF 4The toolkit is being developed as part of the Lemur Project, a collaboration between the Computer Science Department at the University of Massachusetts and the School of Computer Science at Carnegie Mellon University.
sinaiEnEnExp1 (best result) sinaiEnEnExp2 sinaiEnEnExp3 sinaiEnEnExp4 sinaiEnEnExp5 SINAI has participated in monolingual task with ve experiments and the o cial results are shown in Table 1.
Best results were obtained when using our baseline experiment, without adding no type of query expansion (experiment sinaiEnEnExp1 ), only we preprocess each topic with stopper and stemmer and information retrieval process uses Okapi with feedback like weighting function. We considered all tags (title, description and narrative) in information retrieval process. The experiment sinaiEnEnExp2 is the same that the previous one but considering only title and description tags from topics for information retrieval. We use Okapi with feedback like weighting function in all experiments.
In the experiment sinaiEnEnExp3 we expand only the title of topic with geographical information and considered only title and description tags from topics for information retrieval with LEMUR system.
In the experiment sinaiEnEnExp4 we expand the title and the description of topics with thesaurus information, considering only title and description tags in information retrieval process.
In the experiment sinaiEnEnExp5 we expand the title and the description from topics with geographical and thesaurus information. We considered only title and description tags for information retrieval. 3.2
In bilingual task we have participated with a total of ve experiments: two experiments for German-English task and three experiments for Spanish-English task. The o cial results are shown in Table 2.
For German-English task we submit the experiment sinaiDeEnExp1 in which we have not expanded the title or the description of topics, only we have translated the topic and we preprocess it with stopper and stemmer. The retrieval process uses Okapi with feedback like weighting function too. We considered all tags (title, description and narrative) for information retrieval process in this experiment. We submit the experiment sinaiDeEnExp2 for German-English task too. It is the same that previous one but only considering the title and description tags for information retrieval.
For Spanish-English, in the experiment sinaiEsEnExp1, we have not expanded the topics, only we have translated it and we preprocess it with stopper and stemmer. We considered all tags (title, description and narrative) for information retrieval process in this experiment. We submit the experiment sinaiEsEnExp2 for Spanish-English task too. It is the same that previous one but only considering the title and description tags for information retrieval.
Finally, in the experiment sinaiEsEnExp3, we expand the title of topics with geographical information, considering only the title and description tags for information retrieval. 4
In this paper, we have presented the experiment carried out in our rst participation in the GeoCLEF campaign. We have only tried to verify if the topic expansion with geographical and
sinaiDeEnExp1 sinaiDeEnExp2 sinaiEsEnExp1 sinaiEsEnExp2 sinaiEsEnExp3
Spanish Spanish thesaurus information increases the e ectiveness of the information retrieval process. Evaluation results show that the use of geographical and thesaurus information does not improve the retrieval. But this is the rst step for improving the system in the future.
Several reasons exist to explain the worse results obtained with the expansion of topics: ² The NER used sometimes did not work well, because in various topics some entities are recognized and other no. For the future we will try with another NERs. ² In topics, sometimes, appear compound locations like New England, Middle East, Eastern Bloc, etc. that are not in Geographical Information Subsystem. Would be interesting to create rules to control this. ² Depending on spatial relation in topic, we could improve the expansion, testing so that cases work better to add more locations or less.
Therefore, we will try to improve the NER-Gazetteer Subsystem and the Thesaurus Expansion Subsystem to obtain one better query expansion. 5
This work has been supported by Spanish Government with grant TIC2003-07158-C04-04.