This paper reports on the results of our experiments in the Monolingual English, German and Portuguese tasks and the Bilingual Spanish ! English, Spanish ! Portuguese tasks. We also present initial results on the recognition, extraction and categorization of web-based queries for the Query Parsing task. Twenty-three runs were submitted as o cial runs, 16 for the monolingual task and seven for the bilingual task. We used the Terrier Information Retrieval Platform to run experiments for both tasks using the Inverse Document Frequency model with Laplace after-e ect and normalization 2 and the Ponte-Croft language model. Experiments included topics processed automatically as well as topics processed manually. Manual processing of topics was carried out for the bilingual task using the transfer approach in machine translation. Topics were pre-processed automatically to eliminate stopwords. Results show that automatic relevance feedback with 5 terms and 20 documents performs better, in general. The initial approach used in the Query Parsing task is a pattern-based approach. Due to the ungrammaticality, multilinguality and ambiguity of the language in the 800,000 web-based queries in the collection, we started by building a list of all the di erent words in the queries, similar to creating an index. Next, a lookup of the words was done in a list of countries to identify potential locations. Because many locations were missed, we further analyzed the queries looking for spatial prepositions and syntactic cues. Queries were processed by combining search in gazetteers with a set of patterns. Categorization was also based on patterns. Results were low in terms of recall and precision.
Geographical Information Retrieval (GIR), Query Processing, Information Extraction 1
Geographic Information Retrieval (GIR) is aimed at the retrieval of geographic data based not only on conceptual keywords, but also on spatial information. Building GIR systems with such capabilities requires research on diverse areas such as information extraction of geographic terms from structured and unstructured data; word sense disambiguation, which is geographically relevant; ontology creation; combination of geographical and contextual relevance; and geographic term translation, among others.
Research e orts on GIR are addressing issues such as access to multilingual
documents, techniques for information mining (i.e., extraction, exploration and
visualization of geo-referenced information), investigation of spatial
representations and ranking methods for di erent representations, application of machine
learning techniques for place name recognition, development of datasets
containing annotated geographic entities, among others. [
The tasks in GeoCLEF 2007 were Cross-language GIR and Query Parsing. The focus of the rst task was on experimenting with and evaluating the performance of GIR systems when topics include geographic locations such as rivers, regions, seas, continents. Collections of documents and topics in di erent languages were available to carry out monolingual and bilingual experiments. We ran monolingual experiments in English, German, and Portuguese; for bilingual retrieval, we worked with topics in Spanish and documents in English and Portuguese.
The query parsing task consisted of parsing queries to recognize and extract georeferences. The output was structured as a frame, which included geographical such as \where", \geospatial relation" (e.g., in, west, ...), type of geographical query (information, map, yellow page) \latitude-longitude".
In this paper we describe experiments in the cross-language monolingual and bilingual task. We used the Terrier Information Retrieval (IR) platform to run our experiments. This platform has performed successfully in monolingual information retrieval tasks in CLEF and TREC. In addition, we ran initial experiments in the query parsing task. We initially applied pattern-based parsing that did not generate accurate results. We are currently working on inferring a grammar to improve recognition and extraction of geographical references in web-based queries.
The paper is organized as follows. In Section 2 we present our work in the monolingual task including an overview of Terrier. Section 3 describes our setting and experiments in the bilingual task. Pattern-based parsing applied to web-based queries is discussed in Section 4. Finally, we present conclusions and current work in Section 5. 2
In this section we present Terrier (TERabyte RetRIEveR) an information retrieval (IR) platform used in all the experiments. Then we describe experiments and results for monolingual GIR in English, German, and Portuguese. The nal subsection includes the experiments and results for bilingual GIR with topics in English, Portuguese and Spanish.
Terrier is a platform for the rapid development of large-scale Information
Retrieval (IR) systems. It o ers a variety of IR models based on the Divergence
from Randomness (DFR) framework ([
Both indexing and querying of the documents in English, German, and
Portuguese was done with Terrier using the InL2 term weighting model. This
model is the Inverse Document Frequency model with Laplace after-e ect and
normalization 2. The InL2 model has been used in experiments in the past,
GeoCLEF2006 and GeoCLEF2005[
The risk of accepting a term is inversely related to its term frequency in the
document with respect to the elite set, a set in which the term occurs to a relatively
greater extent than in the rest of the documents. The more the term occurs
in the elite set, the less the term frequency is due to randomness. Hence the
probability of the risk of a term not being informative is smaller. The Laplace
model is utilized to compute the information gain with a term within a
document. Term frequencies are calculated with respect to the standard document
length using a formula referred to as normalization 2 shown below.
tf n = tf:log(1 + c
sl
dl
)
tf is the term frequency, sl is the standard document length, and dl is the
document length, c is a parameter. We used c = 1.5 for short queries, which is the
default value, c = 3.0 for short queries with automatic query expansion and c =
5.0 for long queries. Short queries in our context are those which use only the
topic title and topic description; long queries are those which use the topic title,
topic description and topic narrative. We used these values based on the results
generated by the experiments on tuning for BM25 and DFR models done by He
and Ounis [
Additionally, we queried the documents in all the collections using the
PonteCroft language model ([
The document collections indexed were the LA Times (American) 1994 and the Glasgow Herald (British) 1995 for English, publico94, publico95, folha94 and folha95 for Portuguese, and der spiegel, frankfurter and fr rundschau for German. There were 25 topics for each of the languages tested. Documents and topics in English were processed using the English stopwords list (571 words) built by Salton and Buckley for the experimental SMART IR system [1], and the Porter stemmer. Stopwords lists for German and Portuguese were also used. No stemming was applied to the German and Portuguese topics and collections, 2.2
We submitted 6 runs for English, 6 runs for German, and 4 runs for Portuguese. Queries were automatically constructed for all the runs. Results for the monolingual task in English, German and Portuguese are shown in Table 1, Table 2 and Table 3, respectively. The third column shows the model used in each experiment, InL2 or LM (Language Model). The fourth column indicates whether the experiment was run with relevance feedback. For relevance feedback we choose 15 terms and 20 documents to expand the query. This choice was arbitrary and more experiments are needed to nd the combination that yields the best performance. Results for experiments querying the collection with the language model option are not accurate because we did not index the collection using the language model. Therefore we cannot compare the results between the two models as originally planned.
Comparison of the results using the InL2 model shows, for the three languages, that relevance feedback with 15 terms and 20 documents improves performance retrieval. For the bilingual task we worked with Spanish topics and English and Portuguese documents. We translated the topics applying the transfer approach in machine translation using rules to map from the source language to the target language. All the information in the topics within the title, description and narrative was translated. Topics in English, Spanish, and Portuguese were preprocessed by removing diacritic marks and using stopwords lists. Diacritic marks were also removed from the stopwords lists and duplicates were eliminated. Plural stemming was then applied.
Automatic and manual query construction was carried out with the aid of the Spanish Toponymy from the European Parliament [7], and the Names les of countries and territories from the GEOnet Names Server (GNS) [8]. 3.1
Eight runs were submitted as o cial runs for the GeoCLEF2007 bilingual task. In Table 4 we report the results on runs with topics in Spanish and documents in English and in Table 5 the results on runs with Spanish topics and documents in Portuguese.
Run Id
Rel.
Fb.
MAP
Prec.
Rel. Ret. Runs 1 and 4 are the same. We had problems uploading the correct run and deleting the duplicate experiment. Similar to the monolingual task, comparison of the results using the InL2 model shows that relevance feedback with 15 terms and 20 documents improves performance retrieval.
Documents were indexed with InL2 only. Therefore, the results for experiments querying the collection with the language model option are not accurate and a proper comparison with the parametric-based InL2 model could not be made.
Run Id
Prec.
Rel. Ret.
GEOBIESPT1 title, desc. InL2 no 0.05 0.06 GEOBIESPT2 title, desc. InL2 yes 0.05 0.06 GEOBIESPT3 title, desc., narr. InL2 no 0.05 0.06 GEOBIESPT4 title, desc., narr InL2 yes 0.05 0.06
Table 5: Spanish!Portuguese Retrieval Performance Unlike the monolingual runs and the Spanish !English run, relevance feedback did not improve performance retrieval. No querying was done with the language model option.
Information Extraction (IE) has traditionally involved manual processing in the
form of rules or tagging training examples where the user is requied to
specify the potential relations of interest ([
Current work on query processing for retrieving geographic information on the
Web has been done by Chen et. al ([
In our case we are working with a collection of heterogeneous queries and no documents. The task as de ned by the organizers comprises three subtasks: 1) recognize geographic web-based queries; 2) extract the geographical location, the latitude and longitude, geographical relations; and 3) categorize the queries into three types, namely \map", \information" and \yellow page". The initial approach used in the Query Parsing task combines information extraction and patterns.
Due to the ungrammaticality, multilinguality and ambiguity of the language in the 800,000 web-based queries in the collection, we started by building a list of all the di erent words, similar to creating an index, excluding stopwords. Next, a lookup of the words was done in a list of countries, main cities and states to identify potential locations. The list was created from the GEOnet Names Server database ([8]). One problem were multiword georeferences. Because many locations were missed, we selected those queries where spatial prepositions such as \in", \near" and syntactic cues, such as \lake", \cayo", \street", \piazza", \hotel", \accommodation", were present. We have considered these as good heuristics for recognizing multiword expressions as georeferences and create pattern-based rules to further process potential candidates. Extraction of geographical information such as latitude and longitude was done as follows. We created a new list of words identi ed as potential geographic references. Latitude and longitude information was looked up in the GNS database. A problem that we found is related to ambiguity since a geographic reference may refer to a city, state, park, and the same geographic entity may be in different continents, countries, states, and cities.
Finally, categorization was done using patterns. If the only information available was the name of a place, the query was categorized as of type \Map". If words such as \college", \airport", \studio" were present, the query was categorized as of type \Yellow Page". If the query included words such as \ ight", \survey", \company", the query was categorized as of type \Information". Results were low in terms of recall and precision. We are currently working on inferring a grammar and eventually a language model that would improve the performance our initial system. 5
In this paper we presented work on monolingual and bilingual geographical information retrieval. We used Terrier to run our experiments, and an independent translation component built to map source language (Spanish) topics into target language (English or Portuguese) topics. In general, performance retrieval was improved with automatic relevance feedback using the InL2 model. Further experiments indexing the collection with the language model option and querying with this option will allow us to compare parameter-based vs. language-based models. Parsing of web-based queries is a di cult task because of the nature of the data. Further investigation and application of classical and statistical language processing techniques is needed to improve the performance of the approach presented. [1] http://ftp.cs.cornell.edu/pub/smart/. [7] European Parliament. Tools for the External Translator.
http://www.europarl.europa.eu/transl es/plataform/pagina/toponim/toponimo.htm [8] http://earth-info.nga.mil/gns/html/index.html