The main objective of the designed experiments is testing the effects of geographical information retrieval from documents that contain geographical tags. In the designed experiments we try to isolate geographical retrieval from textual retrieval replacing all geo-entity textual references from topics with associated tags and splitting the retrieval process in two phases: textual retrieval from the textual part of the topic without geo-entity references and geographical retrieval from the tagged text generated by the topic tagger. Textual and geographical results are combined applying different techniques: union, intersection, difference, and external join based. Our geographic information retrieval system consists of a set of basics components organized in two categories: (i) linguistic tools oriented to textual analysis and retrieval and (ii) resources and tools oriented to geographical analysis. These tools are combined to carry out the different phases of the system: (i) documents and topics analysis, (ii) relevant documents retrieval and (iii) result combination. If we compare the results achieved to the last campaign's results, we can assert that mean average precision gets worse when the textual geo-entity references are replaced with geographical tags. Part of this worsening is due to our experiments return cero pertinent documents if no documents satisfy de geographical sub-query. But if we only analyze the results of queries that satisfied both textual and geographical terms, we observe that the designed experiments recover pertinent documents quickly, improving R-Precision values. We conclude that the developed geographical information retrieval system is very sensible to textual georeference and therefore it is necessary to improve the name entity recognition module.
• • • • •
This paper is organized as follow: first of all, we briefly present the main components of our geographical information retrieval system, next we describe the experimental runs and results, and finally, we expound the conclusions and future works for GeoCLEF.
Our geographic information retrieval system consists of a set of basics components organized in two categories:
In this section we describe the main geographical tools of our approach. They make up the Named Entity
The geo-entity recognition system developed involves a lexicon consisted of a gazetteer list of geographical resources and several modules for linguistic processing, carrying tasks such as geo-entity identification and tagging.
For lexicon creation we have coalesced two existing gazetteers: the Geographic Names Information System
(GNIS) gazetteer of the U.S. Geographic Survey [
Each one of them uses its own scheme of conceptualization and characterization of the resources. We have defined our own geographical ontology and we had completed and joined the referred gazetteers to obtain a more accurate and flexible gazetteer. The defined ontology allows locating resources based on its geographical areas as well as on other types of relationships like its language (Latin America, countries Anglo-Saxon) or religion (catholic, protestant, Islamic ...).
The gazetteer we have been finally working with has 7,323,408 entries, each one characterized by several
features, such as unique identifier, continent, country, county/region, longitude, latitude, name, etc.
The information retrieval engine used for indexing and searching the gazetteers has been Lucene [
The named geo-entity identify process involves several stages: text preprocessing by filtering special symbols and punctuation marks, initial delimitation by selecting tokens with a starting uppercase letter, token expansion by searching possible named entities consisting of more than one word and eliminating tokens that do not match exactly any gazetteer entry.
The identifier associates each identified named geo-entity with a list of matched resources of the gazetteer.
For the geographical entity tagging we have chosen an annotation scheme that allows us to specify the geographical path to the entity. Each one of the elements of this path provides information of its level in the geographical hierarchy (continent, country, region…) as well as an unique identifier that distinguishes it from the rest of geographical resources of the gazetteer.
Description of the MIRACLE experiments
The baseline approach to processing documents is composed of the following sequence of steps: Extraction: ad-hoc scripts are run on the files that contain particular documents collections to extract the textual data enclosed in XML marks. We have used HEADLINE and TEXT marks. The contents inside these marks were concatenated to feed the followings steps.
Remove accents: all document words are normalized by eliminating accents in words. In spite of this process provides better results running it before the stemming step, we have had to do in this order because our gazetteer consists of normalized entity names.
Geo-entity Recognition: all document collections and topics are parsed and tagged using the geo-entity recognition and tagging tool introduced in the previous section.
Tokenization: this process extracts basic text components, detecting and isolating punctuation symbols.
Some basic entities are also treated, such as numbers, initials, abbreviations, and years. For now, we do
not treat compounds, proper nouns, acronyms or other entities. The outcomes of this process are only
single words and years that appear as numbers in the text (e.g. 1995, 2004, etc.)
Lowercase words: all document words and tags are normalized by changing all uppercase letters to
lowercase.
6. Filtering: all words recognized as stopwords are filtered out. Stopwords in the target languages were
initially obtained from [
Conversion to structured topic: this phase parses the original topic in a structured XML topic consisting of two main entities. The first one (text-query) only contains the associated textual query with the topic, without textual geo-entity references. The second one (geo-query) is a complex XML entity that describes the geo-entity and geo-spatial references extracted from the topic. This second XML entity tries to disambiguate the geo-entities using only information contained in the original topic.
Geo-query expansion: this phase parses the geo-query XML entity and returns a textual document with geographical tags related to the query. The expansion tool developed for this purpose consists of three functional blocks:
Geo-entity Identifier: identifies geographic entities using the information stored in the gazetteer. Spatial Relation Identifier: identifies and qualifies spatial relationships supported by a regular expression based system.
Expander: tags and expands the topic according to the identified spatial relationships and the geoentities related to them. This block uses a relational database system to compute the points located in a geographic area whose centroid is known.
The expansion made by the algorithm is determined by the type of geographic resource (continent,
country, region, county, city…) and the associated spatial relation [
Text-query processing: the goal of this phase is to apply to the textual query similar linguistic processing that has been applied to the collection documents: punctuation marks and accents elimination, conversion to lowercase letters, filtered of stopwords, and stemming.
When all the documents and topics have been processed, they are fed to an ad-hoc front-end of the retrieval trie
engine to search the built document collection index. Only OR combinations of the search terms were used. The
retrieval model used is the well-known Robertson’s Okapi BM-25 [
The textual and geographical results from topics can be combined applying different techniques: union (OR), intersection (AND), difference (AND NOT), and external join (LEFT JOIN) based. Each of these techniques reranks output results computing new relevance measure value from input results.
For this campaign we have designed several basic experiments where the documents for indexing and the topic queries for retrieval are processed using the combination of the steps described in the previous section. They are differentiated mainly by the topic processing as well as by the results combining.
We have used in the run identifier the following letters to denote the fields used in the text-query (t) and geoquery (g) processing: - N: title, and description fields.
- A: title, description, and narrative fields.
AA
AO AtLg (mandatory run) NA NtLg
Combination t AND g t OR g t LEFT JOIN g t AND g t LEFT JOIN g 5.
We tried a wide set of experiments, running several combinations of the variants described in the previous section.
For each of the monolingual tracks, we show a table with the run identifier, the precision at 0 and 1 points of recall, the mean average precision, the R-precision and the percentage deviation (in mean average precision and R-precision) from best one obtained. The last two rows show the two best results when the queries that satisfied textual and geographical sub-query are taking into account.
GCenAtLg
GCenNA GCenNtLg GCenAO
35,00% ) (%30,00% s eg25,00% a e20,00% r v nA15,00% o i is10,00% c e rP 5,00% 20,00% 10,00% 0,00% 0,16 0,12
0,16 0,06 0,51 0,27 0,68
0,32 0,06 0,05 70,00% 60,00% 50,00% ) (%40,00% n o ii s c e rP30,00% R 20,00% 10,00% 0,00% 26 Results for Monolingual Spanish 90,00% 80,00% 70,00% 60,00% ) % (n50,00% o ii s c re40,00% P R30,00% 20,00% 10,00% 0,00%
GCesAtLg GCesAA GCesNA GCesAO
60,00% ) (%50,00% s e ag40,00% r e v30,00% A n io20,00% s i c re10,00% P
If we compare the results achieved to the last campaign’s results, we can assert that mean average precision gets worse when the textual geo-entity references are replaced with geographical tags. Part of this worsening is due to our experiments return cero pertinent documents if no documents satisfy the geographical sub-query. If we only analyze the results of queries that satisfied both textual and geographical terms, we observe that the designed experiments retrieval pertinent documents quickly, improving R-Precision values. Therefore we can conclude that the underlying idea of designed experiments apparently produces good results.
The advantage of basing the experiments on the combination of independent tools allows analyzing the efficiency of each phases of the recovery process and to work on which are weaker. The fact that the designed system presents great sensitivity to textual geo-references shows that one of the weak points of the system is the Named Entity Recognition, reason why our efforts will have to focus on exploring and evaluating different technique-based NER systems.
This work has been partially supported by the Spanish R+D National Plan, by means of the project RIMMEL (Multilingual and Multimedia Information Retrieval, and its Evaluation), TIN2004-07588-C03-01; and by the Madrid’s R+D Regional Plan, by means of the project MAVIR (Enhancing the Access and the Visibility of
Special mention to our colleagues of the MIRACLE team should be done (in alphabetical order): Ana María García-Serrano, Ana González-Ledesma, José Mª Guirao-Miras, José Luis Martínez-Fernández, Paloma