This paper describes the second participation of the SINAI group of the University of Ja¶en in GeoCLEF 2007. We have developed a system di®erent from the one presented in GeoCLEF 2006. Our architecture is made up of ¯ve main modules. The ¯rst one is the Information Retrieval Subsystem, that works with collections and queries in English and returns relevant documents for a query. The queries that are not in English are translated by the Translation Subsystem. All the queries are ¯ltered by the Geo-Relation Finder Subsystem, that ¯nds any spatial relation in the topic, and NER (Named Entities Recognition) Subsystem, that looks for any location in the topic. The most important module is the Geo-Relation Validator Subsystem, it applies some heuristics to ¯lter documents recovered by the IR Subsystem. We have made several runs, combining these modules to resolve the monolingual and the bilingual tasks. The results obtained show that the heuristics applied are quite restrictive and therefore it must be generated new heuristics and to improve the de¯nition of new rules to ¯lter recovered documents.
The most important one is the Validator module. The list of relevant documents is ¯ltered using it: if a document doesn't pass the validation test it is removed from the list. Next section describes the whole system. Then, in the section 3, each module of the system is explained. Later on, results are described and ¯nally, the conclusions about our participation in GeoCLEF 2007 are expounded. 2
We propose a Geographical Information Retrieval System that is made up of ¯ve related subsystems. These modules are explained in detail in the next section.
In our architecture we only worked with the English collection1 and we have applied a o®-line
preprocess to it. This preprocess consists in using the English stop-words list, a named entity
recognizer (NER) and the Porter stemmer [
The translated query or topic proposed is indexed using the IR Subsystem too. If the language of topic is di®erent from English, then it is translated by means of the Translation Subsystem. For each translated query to evaluate we labeled it with NER and geo-relation information. The GeoRelation Finder Subsystem (GR Finder Subsystem) extracts spatial relations from the geographic query and the NER Subsystem recognizes named entities.
From the original English query the documents are recovered by the IR Subsystem that previously indexed the data collection. The NER information and the geo-relation components in relevant documents, and the NER locations from the geographic query are the input for GeoRelation Validator Subsystem (GR Validator Subsystem), the most important module in our architecture.
In the GR Validator Subsystem we eliminate those relevant documents previously retrieved that do not agree with several rules. These rules are related to all the information that handles this module (locations and spatial relations from documents and geographic queries) and are explained in section 3.4. Figure 1 shows the proposed system architecture. 3 3.1
As translation module, we have used SINTRAM (SINai TRAnslation Module)[
The main goal of NER Subsystem is to detect and recognize the entities appearing in the queries. We are only interested in geographical information, so we have just used locations detected by this NER Module. We have used the NER module of the GATE2 toolkit. The location terms includes everything that is town, city, capital, country and even continent. The NER module adds entity labels to the topics with the found locations. An entity label example recognized in the title of the topic follows:
< en title position = "15" type = "LOC" > USA < =en title > 1English Los Angeles Times 94 (LA94) and English Glasgow Herald 95 (GH95) 2http://gate.ac.uk/ where position is the position of the entity in the phrase. This value is greater than or equal to zero and we used it to know at any moment what locations and geo-relations are related to each other by proximity.
The basic operation of NER Subsystem is the following: ² The ¯rst step is the preprocessing phase. Each query is preprocessed using a tokenizer, a sentence splitter and a POS tagger. The NER Subsystem we have used needs this information in order to improves the named entity detection and recognition. ² The second is the detection of the geographical places. For this proposal we have use a
Gazetteer, included also in GATE.
The NER Subsystem generates some topic labels, based on the original ones, adding the locations. These topic labels will be used later by the GR Validator Subsystem. 3.3
The Geo-Relation Finder Subsystem is used to ¯nd the spatial relations in the geographic queries. This module makes use of four text ¯les to store the geo-relations identi¯ed. Four geo-relations ¯les exist because our system detects spatial relations of four words at the most. Some geo-relations examples are: in, near, north of, next to, in or around, in the west of...
The GR Finder module adds geo-relation labels to the topics with the found spatial relations. A geo-relation label example recognized in the title of the topic would be the following one: < gr title position = "43" > near < =gr title > where position is the position of the spatial relation in the phrase.
In this module we controlled a special geo-relation named between. For this case, the GR Finder Subsystem adds the two entities that this preposition relates. An example of the label that this module adds for description label "To be relevant documents describing oil or gas production between the UK and the European continent will be relevant " is:
< gr desc position = "9" > between the; U K; European < =gr desc > where we can see how both entities (UK and European) are added after the preposition, separated by a semicolon.
The basic operation of GR Finder Subsystem is the following: ² For each topic label (title, desc or narr ) the subsystem looks for some spatial relation. It makes use of the text ¯les that store the geo-relations that can detect. ² For each found spatial relation we veri¯ed that the word that comes next is an entity. For that reason it is necessary that the NER Subsystem is executed before.
Like NER Subsystem, the GR Finder Subsystem also generates topic labels, based on the original topic, adding the spatial relations. These topic labels (entities and geo-relations) will be used later by the GR Validator Subsystem. In the Figure 2 we can see an example of the text that generates this subsystem. 3http://www.geonames.org/. Geonames geographical database contains over eight million geographical names and consists of 6.3 million unique features whereof 2.2 million populated places and 1.8 million alternate names ² Find the country name of a city. ² Find the latitude and longitude for a given location. ² Check if a city belongs to a certain country. ² Check if a location is to the north of another one. ² Calculate the distance from a location to another one.
Many heuristics can be applied with the former check points to make the validation of a document recovered by the IR Subsystem. The GR Validator Subsystem receives external information from IR Subsystem (entities from each document recovered) and from GR Finder and NER Subsystems (entities and spatial relations from each topic). This year we have used the following heuristics in our experiments: 1. For every entity appearing in query without an associated geo-relation, the system checks if this entity is present in documents recovered by IR Subsystem. The module discards a document whenever the number of entities found in topic with no associated geo-relation and not appearing in that document exceeds the ¯fty percent of the total of topic entities. 2. If the entity appearing in the topic has associated some spatial relation, the module checks if location is a continent, a country or a city. Depending on this location type for the query, the heuristics which we have followed in our experiments are the following ones: (a) If the location from a query is a continent or a country and its associated geo-relation is in, on, at, from, of or along, then the module checks if most of the entities of the document belong to that continent or country (at least ¯fty percent). (b) If the location from a query is a city and its associated spatial relation is near, north of, south of, east of or west of, the subsystem obtains the latitude and longitude information from Geonames Gazetteer about all locations from the document to be validated. The module will check if the geographic situation of each location is valid or not depending on the topic space relation.
For each heuristic to check, the system is adding or reducing points of a ¯nal score, depending on the result of that validation. A recovered document will be considered valid when the sum of all the scores when applying the heuristics to each entity of the document is greater than zero. 3.5
The information retrieval system that we have employed is Lemur4. It is a toolkit that supports indexing of large-scale text databases, the construction of simple language models for documents, queries, or sub-collections, and the implementation of retrieval systems based on language models as well as a variety of other retrieval models.
Previous to the index step, the English collection provided for GeoCLEF have been
preprocessed, using the English stop-words list for meaningless terms removal, and the Porter stemmer
[
One parameter for each experiment is the weighting function, such as Okapi [
4http://www.lemurproject.org/. The toolkit is being developed as part of the Lemur Project, a collaboration between the Computer Science Department at the University of Massachusetts and the School of Computer Science at Carnegie Mellon University.
Experiment Sinai ENEN Exp1 fb okapi
Sinai ENEN Exp1 fb t¯df Sinai ENEN Exp1 simple okapi Sinai ENEN Exp1 simple t¯df
Sinai ENEN Exp2 fb t¯df
SINAI5 has participated in monolingual and bilingual tasks for GeoCLEF 2007 with a total of 26 experiments. In all experiments we have considered all tags from topics (title, description and narrative) as source for the information retrieval process.
Our baseline experiment consists in the Lemur retrieval on preprocessed collections (stopper and stemmer ) without applying heuristics on relevant documents retrieved. This experiment has been applied in the monolingual and bilingual tasks.
The second experiment that we have made, consists in applying the heuristics that has been explained in section 3.4, on relevant documents retrieved by the IR Subsystem. This experiment also has been used in the monolingual and bilingual tasks. 4.1
In the monolingual task we have participated with a total of 8 experiments: four about baseline experiment (Exp1 ) and other four about second experiment applying the heuristics introduced previously in this paper(Exp2 ). Some results are shown in Table 1.
Experiments named against an ending "fb okapi " we have used Okapi with feedback as weighting function in the information retrieval process. Those ending with "fb t¯df " indicate that we have applied TF.IDF with feedback. Also we have run experiments with Okapi but without feedback ("simple okapi ") and with TF.IDF but without feedback ("simple t¯df "). 4.2
In the bilingual task we have participated with a total of 18 experiments: twelve about the baseline case (Exp1 ) and six applying our heuristics (Exp2 ). Some results are shown in Table 2.
For naming the experiments we have followed the same convention described in the previous section (see section 4.1). For German-English task we submitted six experiments in total. The string "GEEN " identi¯es them. For Portuguese-English task we submit six experiments too, identi¯ed by the string "PTEN ". For Spanish-English task we submit six experiments in total (string "SPEN "). 5
In this paper we have presented the experiments carried out in our second participation in the GeoCLEF campaign. The philosophy followed in this second experimental study has changed with respect to the approach presented last year. This year we have introduced a very restrictive system: we have tried to eliminate those documents recovered by the IR Subsystem that do not satisfy certain validation rules. However, the previous year we were centered in increasing the queries, expanding them with entities and thesauri information in order to improve retrieval e®ectiveness.
The results obtained the previous year showed that query expansion does not improve in general the quality of the information retrieval process. The results of this year shown that the documents that have been recovered are valid but the GR Validator Subsystem has ¯ltered some ones that must not have eliminated.
For the future, we will try to add more heuristics to the GR Validator Subsystem making use of Geonames Gazetteer. Also we will de¯ne more precise rules so that the system is less restrictive for the selection of recovered documents. Finally, we will also explore a larger number of retrieved documents by the IR Subsystem, in the aim of providing a larger variety of documents to be checked by the GR Validator Subsystem. 6