This paper describes GeoTALP-IR system, a Geographical Information Retrieval (GIR) system. The system is described and evaluated in the context of our participation in the CLEF 2005 GeoCLEF Monolingual English task. The system architecture has two phases that are performed sequentially: Topic Analysis and Document Retrieval. The Topic Analysis phase extracts and analyzes the relevant keywords from the topic. This phase uses a Keyword Selection algorithm based on Linguistic and Geographical Analysis of the topics. A Geographical Thesaurus has been build using a set of publicly available Geographical Gazetteers. The Document Retrieval system is based on Lucene and uses a modified version of the Passage Retrieval module used by the TALP Question Answering (QA) system at the CLEF 2004 and TREC 2004 QA evaluation tasks. The results of our experiments show that the use of a Geographical Thesaurus for Geographical Indexing and Retrieval has improved the performance of our GIR system.
The GIR system is based on Lucene, uses a modified version of the Passage Retrieval module
used by the TALP Question Answering (QA) system at CLEF2004 [
In this paper we present the overall architecture of GeoTALP-IR and describe briefly its main components. We also present an evaluation of the system used in the GeoCLEF 2005 evaluation. 1.1
GeoCLEF is a cross-language geographic retrieval task at the CLEF 2005 campaign. The goal of the task is to find as many relevant documents as possible from the document collections, using a topic set. Topics are textual descriptions with the following fields: title, description, narrative, location (e.g. geographical places like continents, regions, countries, cities, etc.) and a geographical operator (e.g. spatial relations like in, near, north of, etc.). See below an example of a topic: <num> GC001 </num> <orignum> C084 </orignum> <EN-title> Shark Attacks off Australia and California </EN-title> <EN-desc> Documents will report any information relating to shark attacks on humans. </EN-desc> <EN-narr> Identify instances where a human was attacked by a shark, including where the attack took place and the circumstances surrounding the attack. Only documents concerning specific attacks are relevant; unconfirmed shark attacks or suspected bites are not relevant. </EN-narr> <EN-concept> Shark Attacks </EN-concept> <EN-spatialrelation> near </EN-spatialrelation> <EN-location> Australia </EN-location> <EN-location> California </EN-location> 2 2.1
The system architecture has two phases that are performed sequentially (as shown in Figure 1): Topic Analysis (TA) and Document Retrieval (DR). Angeles Times 1994 (LAT94) (i.e. 169,477 documents). We pre-processed the whole collection with linguistic tools (described in the next sub-section) to mark the part-of-speech (POS) tags, lemmas and Named Entities (NE). After this process the collection is analyzed with a Geographical Thesaurus (described in the next sub-section). This information was used to built an index that contains the following fields for each document: • Form Field: the original text with Named Entity Recognition that is retrieved when a query succeeds on the Lemma field. • Lemma Field: this part is built using the lemmas of the words, POS and the results of the Named Entity Recognition and Classification (NERC) module and the Geographical Thesaurus. This text is then indexed and used in the IR module. • Geo Field: contains all NEs classified as LOCATION or ORGANIZATION that appear in the Geographical Thesaurus. This part has the geographical information about these NE: including geographical coordinates and geographical relations with the corresponding places of its path to the top of the geographical ontology (i.e. a city like ”Barcelona” contains its state, country, sub-continent and continent). If a NE is an ambiguous location, all the possible ambiguous places are stored in this field.
See below an example of a sequence of an indexed document:
for#IN a#DT weekend#NN in#IN Barcelona#NNP#LOCATION#city ,#, return#VBD to#TO London#NNP#LOCATION#capital on#IN monday#NNP ,#,
• A Maximum Entropy based NERC, a Named Entity Recognizer and Classifier that
identifies and classifies NEs in basic categories (person, place, organization and other). This
NERC has been trained with the CONLL-2003 shared task English data set [
Spanish) and actor-action relations (e.g. write-writer). 2.3.2
Geographical Analysis The Geographical Analysis is applied to the Named Entities provided by the location tag (<ENlocation>), and the Named Entities from the Title and Description tags that have been classified as LOCATION or ORGANIZATION by the NERC module. This analysis has two main components: • Geographical Thesaurus: this component has been built joining three gazetteers that contain entries with the places and their geographical class, coordinates, and other information: 1. GEOnet Names Server (GNS)2: a gazetteer covering worldwide excluding the United States and Antarctica, with 5.3 million entries. Each gazetteer entry contains a geographical name (toponym) and its geographical coordinates (latitude, longitude), language of the geographical name and other features (e.g. country, first administrative division). 2. Geographic Names Information System (GNIS)3, contains information about physical and cultural geographic features in the United States and its territories. This gazetteer has 2.0 million entries, but we used a subset (39,906) of the most important geographical names. 3. GeoWorldMap4 World Gazetteer: a gazetteer with approximately 40,594 entries of the most important countries, regions and cities of the world.
sea capital continent sub−continent country
summit state city
county
Each one of these gazetteers have a different set of classes. We have mapped this sets to
our set of classes (see Figure 3), which includes the most common classes and the most
important (i.e. country is not common, but important)). The resulting thesaurus contains
approximately 3.7 million of places with its geographical class. This step is similar to the [
Topic Keywords Selection We designed an algorithm for extracting the most relevant keywords of each topic. These keywords are then passed to the Document Retrieval phase. The algorithm is applied after the Linguistic and Geographical analysis and has the following steps: 1. Initial Filtering. First, all the punctuation symbols and stopwords are removed from the analysis of the title, description and geographical tags. 2. Title Words Extraction. All the words from the title tag are obtained. 3. Description Chunks Filtering. All the Noun Phrase base chunks from the description tag that contain a word with a lemma that appears in one or more words from the title are extracted. 4. Description Words Extraction. The words that pertain to the chunks extracted in the previous step and haven’t a lemma appearing in the words of the title are extracted. 5. Join Title, Description and Location Words Analysis. The words extracted from the title and description and the geographical tag are joined.
issues#issue#NNS region#region#NN scottish#Scottish#NNP#misc#location(”Scotland”)
The main function of the Document retrieval component is to extract relevant documents that
are likely to contain the information needed by the user. Document retrieval is performed using
the Lucene Information Retrieval system. Lucene uses the standard tf.idf weighting scheme with
the cosine similarity measure, and allows ranked and boolean queries. The document retrieval
algorithm uses a data-driven query relaxation technique: if too few documents are retrieved, the
query is relaxed by discarding the keywords with the lowest priority. The reverse happens when
too many documents are extracted. Each keyword is assigned a priority using a series of heuristics
fairly similar to [
The main options of the Document Retrieval phase are: • Query types: • Geographical Search Mode: – Boolean queries: all the keywords must appear in the documents retrieved. Lucene allows boolean queries and returns a score for each retrieved document. – Ranked queries: Lucene does ranked queries with tf·idf and cosine similarity. – Boolean+Ranked queries: this mode joins documents from boolean and ranked queries, giving priority to the documents from the boolean query. – Lemma field: this search mode implies that all the keywords that are Named Entities detected as LOCATION are searched in the ”Lemma” field part of the index. – Geo field: this search means that the NEs tagged as LOCATION and detected as keywords will be searched at the ”Geo” index field. • Geographical Search Policy: – Strict search: this search policy can be enabled when the ”Geo” Field search is running, and serves to find a LOCATION with exactly all this ontological path and coordinates search for the following classes: country and region. As an example the following location (”Australia”) with its ontological information and coordinates: Oceania#Oceania#Australia#-25.0 135.0
Then ”Australia” is searched in the index with exactly this form. – Relaxed search: this search policy can also be enabled when the ”Geo” field search is running, instead of the case of countries and regions, this mode searches without coordinates. The previous example was the search of the location ”Australia” with its information: Oceania#Oceania#Australia In this case, the search is more flexible and all the cities and regions inside Australia will be returned, as an example this location will be found:
Oceania#Oceania#Australia#Western Australia#Perth#-31.966 115.8167 2.5
This component joins the documents provided by the Document Retrieval phase. If the query type is boolean or ranked it returns the first 1000 top documents with their Lucene score. In the case of a query mode boolean+ranked, first gives priority to the documents retrieved from the boolean query and holds their score. Then, the documents provided by the ranked query are added to the list of relevant documents, but their score is then re-scaled using the score of the last boolean document retrieved (the document with lower score of the boolean retrieval). Finally, the first 1000 top documents are selected. 3
We designed a set of four experiments that consists in applying different query strategies and tags to an automatic GIR system (see Table 1). Two baseline experiments have been done: the runs geotalpIR1 and geotalpIR2. These runs differ uniquely in the query type used: a boolean+ranked retrieval in geotalpIR1 run and only ranked retrieval in geotalpIR2 run. These runs used the Title and Description tags, and they use the ”lemma” index field. The third run (geotalpIR3) differ from the previous ones in the use of the Location tag (uses Title, Description and Location) and uses the ”Geo” field instead of the ”lemma” field. The ”Geo” field is used with a Strict search policy. This run also uses a boolean+ranked retrieval. The fourth run (geotalpIR4) is very similar to the third run (geotalpIR3), but uses a Relaxed search policy.
In these experiments we can expect to see a considerable difference between the two first runs and the last ones, because the two last ones used an index with geographical knowledge. The Run geotalpIR1 geotalpIR2 geotalpIR3 geotalpIR4 fourth run is expected to be better than the third, due to the use of a relaxed search policy, that can increase the recall. In the other hand, we omitted the use of the operation tag (e.g. south, in, near, around,...) because our system is not prepared to deal with this information. Finally, the use of the location tag in the last runs is not so relevant, because our NERC and Geographical Thesaurus are able to detect all the locations from the Title and Description tags with high performance. 4
The results of the GeoTALP-IR system at the GeoCLEF 2005 Monolingual English task are summarized in Table 2. This table has the following IR measures for each run: Average Precision, R-Precision, Recall, and the increment over the median average precision (0.2063) obtained by all the system that participated in the GeoCLEF 2005 Monolingual English task.
The results show a substantial difference between the two first runs and the two last ones, specially in the recall measure: 49.51% and 49.22% in the first and second run (geotalpIR1 and geotalpIR2) and 62.35% and 66.83% in the third and fourth run (geotalpIR3 and geotalpIR4). The recall is also improved by the use of Geographical Knowledge and a relaxed search policy over the ”Geo” Field as is seen in the fourth run (geotalpIR4). Finally, in the last run (geotalpIR4) we obtained a results about +8.14% better than the median average (0.2063) obtained by all the runs from the participants in the GeoCLEF-2005 task.
Run geotalpIR1 geotalpIR2 geotalpIR3 geotalpIR4
Tags TD TD TDL TDL
AvgP. This is our first participation in a IR task. Our approach is based in a QA-based IR system for Document Retrieval and a Linguistic and Geographical Analysis of the collections and topics. We conclude that the use of a Geographical Thesaurus for Geographical Indexing and Retrieval has improved the performance of our GIR system.
As a future work we propose the following improvements to the system: i) analysing the topics using WordNet, ii) the use of the spatial operator and narrative tags, iii) improving the boolean IR strategy, and iv) the resolution of geographical ambiguity problems. 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Interpolated Recall
Acknowledgments 0 5
500 1000
CLEF2005 GC-Monolingual-EN - Interpolated Recall vs Average Precision geotalpIR4 geotalpIR3 geotalpIR2 geotalpIR1 This work has been partially supported by the European Commission (CHIL, IST-2004-506909) and the Spanish Research Dept. (ALIADO, TIC2002-04447-C02). TALP Research Center is recognized as a Quality Research Group (2001 SGR 00254) by DURSI, the Research Department of the Catalan Government.