This paper describes the third participation of the SINAI research group from University of Ja¶en in GeoCLEF track. We have tried to improve the system proposed last year in GeoCLEF 2007. The main developments are related to the use of query reformulation, keywords recognition, hyponyms extraction and query geo-expansion. On the other hand, new rules have been applied in the Validator subsystem in order to ¯lter the documents recovered by the IR subsystem. We have run several experiments, combining these developments in order to resolve the monolingual and bilingual tasks. The results obtained shown that ¯ltering does not reach yet to improve the baseline case. However, the use of keywords and hyponyms in the re-ranking process seems to improve the ¯ltering results. On the other hand, the use of query reformulation and geo-expansion does not improve the baseline case either.
GeoCLEF is a cross-language Geographic Information Retrieval (GIR) task at the Cross-Language
Evaluation Forum (CLEF) campaign since 2005. The aim of GeoCLEF is to evaluate GIR systems.
Given a multilingual statement describing a spatial user need (topic), the challenge is to ¯nd
relevant documents from target collections using queries into several languages [
This paper describes the approaches taken by the SINAI1 research group from the University of
Ja¶en for the main GeoCLEF 2008 subtasks: mono and bilingual retrieval. In 2006 [
Next section describes the whole system. Then, in the section 3, each module of the system is explained. Following, results are described and ¯nally, the conclusions about our participation in GeoCLEF 2008 are expounded. 2
Our SINAI-GIR system is made up of ¯ve main subsystems: Translator, Collection Preprocessing subsystem, Query Analyzer, Information Retrieval subsystem and Validator. We make use of the Geonames gazetteer2 as geographic knowledge base for the whole system. As Information Retrieval (IR) index-search engine we have used Lemur3.
Each translated query is preprocessed and analyzed by the Query Analyzer, identifying their
geo-entities and spatial relations. This module also applies query reformulation based on the query
parsing subtask [
As translation module, we have used SINTRAM (SINai TRAnslation Module), our Machine
Translation system which works with di®erent online machine translators and implements several
heuristics to combine di®erent translations [
In our architecture we only worked with English documents collection4 and we have applied a
o®-line preprocess to this one. In this preprocess we have applied the Porter stemmer [
During the preprocessing, two indexes are generated: ² The locations index. This index stores all location entities detected and recognized by the NER in each document of the collection. This year we have used LingPipe5 as NER in our architecture. All entities classi¯ed as a location by the NER are checked using the GeoNames gazetteer. This locations index will be used later by the Validator in order to ¯lter the documents recovered by the IR subsystem.
2http://www.geonames.org/. Geonames is a geographic database which contains over eight million geographical names and consists of 6.3 million unique features whereof 2.2 million populated places and 1.8 million alternate names.
3http://www.lemurproject.org/. Lemur is a open-source IR toolkit which is being developed as part of the Lemur Project, a collaboration between the Computer Science Department at the University of Massachusetts and the School of Computer Science at Carnegie Mellon University.
4The collection consists of 169,477 documents composed of stories from the British newspaper Glasgow Herald (1995) and the American newspaper Los Angeles Times (1994).
5http://alias-i.com/lingpipe/. LingPipe is a suite of Java libraries for the linguistic analysis of human language which includes a NER module. ² The keywords index. This index stores the keywords of each document of the collection.
We have considered as keyword the nouns that appearing more than once in the document. We have decided to consider only the nouns because they have more meaning than verbs or adjectives. This keywords will be used by the Validator in order to re-rank the documents recovered by the IR subsystem. 3.3
The Query Analyzer module is responsible for preprocessing of English queries as well as the
generation of di®erent query reformulations. This analyzer is also made up of several components:
Preprocessing module. This module removes guidance information from the topics such us
\Relevant documents contain information about ", \Find documents describing " or \To be
relevant, documents must describe", and descriptions about irrelevant documents. The
default query (Q1) is formed by the preprocessed content of the title, description and narrative
labels of the topics, applying the Porter stemmer [
Geo-Relation Finder module. This module is used to ¯nd the spatial relations in the queries.
It is based on manual rules and the entities detected by the NER module. This module makes use of several text ¯les that store all geo-relations that can be detect. Some examples of these spatial relations are: in, near, north of, next to, in or around, in the west of... Query Reformulation module. This module parses only the title of the query, detecting the three components on which it is usually composed: \what ", \geo-relation" and \where". Before the query parsing, we apply a particular translation of sentences like \capital of <entity>" or \<entity>'s capital ", replacing the entire sentence by the corrected location using the Geonames gazetteer. By example, the sentence \the capital of France" would be replaced by \Paris". This module generates three query reformulations: ² Q1: it is formed by the preprocessed content of the topic labels, depending on the experiment. Some experiments consider only the content of the title and description from the topics. Others consider the content of all labels (title, description and narrative). ² Q2: it is formed only by the concatenation of \what " and \where" components detected in the title of the topic. ² Q3: it is the same as the previous one, but adding it the expanded locations found in Geonames gazetteer, related to the \where" component and depending on the georelation detected in the title of the topic.
Keywords-Hyponyms extractor module. The aim of this module is to detect the keywords
only in the title of the queries. As in the Collection Preprocessing subsystem, we have
considered as keywords only the nouns because they have more importance than verbs or
adjectives in the re-ranking process. We have used the Brill POS tagger [
The index-search engine that we have used in the experiments is Lemur. It is a open toolkit
that supports indexing of large-scale text databases, the construction of simple language models
for documents, queries, or sub-collections, and the implementation of retrieval systems based on
language models. Lemur supports several weighting functions such as Okapi [
The aim of this module is to retrieve the most relevant documents for each query reformulation. As we can see in the previous section, the Query Analyzer generates three queries for each topic (Q1, Q2 and Q3). The list of retrieved documents for the Q1 query is considered as baseline case in our experiments because we do not apply it any ¯ltering or re-ranking process. 3.5
The aim of the Validator process is to ¯lter the lists of documents recovered by the IR subsystem, establishing what of them are valid, depending on the locations and the geo-relations detected in the query. Another important function of this process is to establish the ¯nal ranking of documents, based on manual rules and prede¯ned weights. These prede¯ned weights allow to assign a score to each document, in addition to the score provided by the IR subsystem.
In order to validate each document recovered, the Validator applies di®erent manual rules, making use of geographical data detected in the topic. Some examples of these manual rules are: ² If the entity of the topic is a country and its geo-relation associated is \in the north of ", the Validator will accept the document if it has any location situated in the north of that country. Using the Geonames gazetteer, the module obtains the maximum and minimum latitudes of all locations that belong to that country. Then, it calculates half of the latitude from the maximal and minimal latitudes to estimate the north of the region. ² If the entity of the topic is a city and its geo-relation associated is \near to", the module will consider that a location is near to another one when it is at a distance of less than 50 kilometers around. We have tried several distances in the experiments and the distance of 50 kilometers provided us the best results. In order to measure the distance in kilometers between two locations we have used the simple formula for calculating the distance with geographic coordinates:
d = p(x2 + y2) where: x = 110:56 ¤ abs(lat2 ¡ lat1) y = 84:8 ¤ abs(lon2 ¡ lon1) lat2, lat1, lon2 and lon1 are the latitudes and longitudes from location 2 and 1 respectively.
If the d value is less than 50, the location 1 will be considered near to location 2. ² If the entity of the topic is a continent or a country and its geo-relation associated is \in", \of ", \at ", \on", \from" or \along", the module will accept the document recovered if a location exists in the document that belongs to that continent or country.
In order to re-rank each valid document recovered, the Validator makes use of several prede¯ned weights that are added to the score provided by the IR subsystem, depending on the manual rules which the document complies. By example, there is a prede¯ned weight for the ¯rst manual rule previously explained (the entity type is country and its geo-relation associated is \in the north of "). If any document complies with that rule, the prede¯ned weight for that rule is added to the IR score of the document. In the experiments carried out in this paper we have tried with several weights, allowing us for an optimal adjustment of the system. 4
SINAI has participated in monolingual and bilingual tasks with a total of 15 experiments. In some experiments we have considered the content of all the topics labels (title, description and narrative), identi¯ed them as TDN, and for others experiments we have considered only the title and description labels, identi¯ed them as TD.
Our baseline experiment consists of the retrieval of documents more relevant by means of IR subsystem using the Q1 as query for each topic, without applying any ¯ltering or re-ranking process. This baseline experiment has been applied in the monolingual and bilingual tasks, using TDN and TD labels from topics.
Other experiments carried out in this paper consist of applying the ¯ltering and re-ranking processes to the di®erent lists of relevant documents retrieved by the IR subsystem. In some experiments, the ¯ltering and re-ranking processes have been applied to the fusion list of the documents recovered by the Q1, Q2 and Q3 queries for each topic. This fusion list is generated by adding the documents from Q2 and Q3 lists that are not in Q1. In other experiments, we have considered the use of keywords and hyponyms in the re-ranking process.
With respect to the weighting function used in the IR subsystem, in the experiments carried out in this paper we have always used Okapi with PRF because after trying several weighting functions, it o®ers the best results. 4.1
In monolingual task we have participated with a total of 9 experiments. Three of them are considered as baseline experiments because we have not applied any ¯ltering or re-ranking process to the list of recovered documents. Other experiments combining the fusion list and the use of keywords and hyponyms. The results for monolingual task are shown in Table 1.
Labels
TD TDN TDN TD TDN TDN TD TDN TDN TDN
Fusion no no yes no no yes no no no yes
Filtering no no no yes yes yes yes yes yes yes
Keywords no no no no no no yes yes yes yes
Hyponyms no no no no no no no no yes no In bilingual task we have participated with a total of 6 experiments. We have used the English collection and the topics in Portuguese and German languages. Three of the experiments are considered as the baseline case because we have not applied any ¯ltering or re-ranking process to them. Other experiments combining the ¯ltering and re-ranking process without the use of keywords or hyponyms. The results for bilingual task are shown in Table 2. 4.3
The analysis of results based on MAP values shows that the ¯ltering and re-ranking process does not improve the baseline case, because the re-ranking process is not performing well yet. There are valid documents which do not rise enough in the ¯nal ranking. Moreover, the number of documents recovered by the IR subsystem normally is around 3000 and the size of the ¯nal list returned by the Validator is 1000, so some valid documents are left outside for some topics.
However, in some experiments in which we apply ¯ltering, the use of keywords improves the result obtained without using keywords. Instead, the use of hyponyms does not improve in any case the results.
On the other hand, surprisingly we have obtained best results using only the content of the title and description labels from the topics (TD), unlike what happened in the 2007 experiments, where we reached the best results using the content of all labels (TDN ).
In reference to the results obtained for the bilingual task, we can a±rm that the Translator module works better with Portuguese. As in the monolingual experiments, the best results have been achieved without the use of ¯ltering or re-ranking process, although the results using ¯ltering almost equal the baseline experiments.
In this paper we have presented the experiments carried out in our third participation in the GeoCLEF track, following the basic architecture used in the 2007 experiments. We have tried to improve the ¯ltering and re-ranking process introduced the previous year, adding new developments related to several techniques such as query reformulation, keywords and hyponyms extraction and even query geo-expansion. Moreover, we have established prede¯ned weights for each manual rule in Validator in order to improve the ¯nal score of the valid documents. However, we still get the best result without applying any of these techniques. This is because we have not used an optimal method to raise valid documents in the ¯nal ranking, depending on the geo-information that recovered documents and topics have in common.
About the new developments employed in the experiments, only the use of keywords in the reranking process seems to improve the ¯ltering results in some cases. Instead, the use of hyponyms does not improve the results. Therefore, the proper use of keywords for the re-ranking process could be interesting in the future.
With respect to the experiments in which we have used the fusion list, the results obtained indicates that the query reformulation does not seem to work well in this ¯eld, although in some topics the Q2 and Q3 query types add valid documents to the ¯nal list which have not been found by the IR subsystem using the default query (Q1). 6
This work has been supported by a grant from the Spanish Government, project TIMOM (TIN200615265-C06-03), and the RFC/PP2006/Id 514 granted by the University of Ja¶en.