This paper reports the participation of the XLDB Group from the University of Lisbon at the 2007 GeoCLEF task. We adopted a novel approach for GIR, focused on handling geographic features and feature types on both queries and documents, generating geographic signatures with multiple geographic concepts as a scope of interest. We experimented new query expansion and text mining strategies, relevance feedback approaches and geographic score metrics. In the paper we introduce the new approach, discuss the experiments and analyse the obtained results.
This paper presents the participation of the XLDB Group from the University of Lisbon at the 2007
GeoCLEF task. We experimented novel strategies for geographic query expansion, text mining, relevance
feedback and geographic score metrics in a renewed GIR system. The motivation for this work derived from
the results obtained in last year’s participation, which revealed limitations on our previous GIR model [
This year, we decided to challenge some of the underlying assumptions of the GIR model used in the
previous year, and tested a new approach. We introduced significant changes in the assembled GIR system,
both on the query and on the document sides, to see if they could effectively tackle the limitations detected
on the past GIR system. The improvements have been introduced at three levels:
Query Processing: We have rebuilt the query processing modules so that all geographic information
present on a query is captured and subject to proper geographic query expansion. We gave special attention
to feature types and spatial relationships, as guides for the geographic query expansion [
Geographic Ranking: As the new text mining approach generates a geographic signature for each document (DSig), and the geographic query expansion module generates a geographic signature for the query (QSig), the geographic ranking step now has the burden of evaluating relevance considering queries and documents that contain multiple geographic concepts as a scope. In 2006, our similarity metric compared the (single) scope of a document against the (single) scope of a query. This year, we had to handle each of the features in the geographic signatures as part of a scope and compute a metric accounting for all concepts in the geographic signatures. We made some preliminary experiments to assess new combination metrics for computing relevance based on geographic signatures.
The rest of this paper is organised as follows: Section 2 depics our assembled GIR system, and describes in detail each module of our prototype. Section 3 presents our experiments and Section 4 analyses the results. Section 5 ends the paper with conclusions and directions for future work. 2
The geographic ontology is a central component of our GIR system, providing support for geographic
reasoning for all modules. It models both geographic concepts and the relationships between concepts in
Island
Airport
River
Mountain
Lake
Circuit
Region
Continent
Names
Features
Feature Types
an hierarchical scheme. The geographic data come from several public sources, and include names for
places and other geographic features, feature types, adjectives, relationships between concepts (adjacent
and part-of ), demographic data, spatial coordinates and bounding boxes [
The improvements made to the ontology for this year’s participation were twofold: i) update of the GKB
conceptual model to directly support multilingual names for geographic references, and ii) the addition of
new features that we found missing after inspecting the GeoCLEF topics for 2007. The GKB 2.0 model
now supports relationships between feature types, a better property assignment for features and feature
types, and a better control of information sources [
On the query side, we developed a new geographic query parsing module, QueOnde. The geographic query
expansion module, QuerCol, introduced for last year’s participation [
QueOnde automatically converts GeoCLEF topic titles into <what, spatial relationship, where> triplets
with the help of the geographic ontology and a set of manually-crafted context rules for capturing and
disambiguating spatial relationships, features and feature types. QueOnde also participated on the 2007
GeoCLEF Query Parsing subtask [
The QuerCol module is able to expand the thematic (what) and the geographic (where) parts of a query
separately. The what is expanded through blind relevance feedback (RF) [
When feature types are present in the query, they may mean two things: i) the user is disambiguating the geoname, because it can be associated to other geographic concepts (e.g., City of Budapest and Budapest Airport); or ii) the user is designating a set of concepts as a scope of interest (e.g., Airports of Hungary). In case i), the feature type is disambiguating the geographic concept given by the feature Budapest as the scope of interest, while in case ii), the feature type is designating a group of geographic concepts of the scopes of interest, requiring additional geographic reasoning to obtain the corresponding concepts.
The geographic query expansion step of our GIR system is now guided according to the spatial relationship, features or feature types specified on the query. For instance, in the CLEF topic #74, Ship traffic around Portuguese islands, QuerCol considers in as the spatial relationship, Portugal as a feature name and islands as a feature type, and it reasons that the scope of interest is all geographic concepts of type island that are part of Portugal: São Miguel, Santa Maria, Formigas, Terceira, Graciosa, São Jorge, Pico, Faial, Flores, Corvo, Madeira, Porto Santo, Desertas and Selvagens. 2.3
Faísca The text mining module Faísca parses the documents for geonames, generating geographic signatures for each document. Faísca relies on pattern matching from a gazetteer generated from the geographic ontology, containing all concepts represented by their names and respective feature types. Consider the following (fictional) example for the geoname Lisbon, which is associated to multiple geographic concepts in the ontology. The gazetteer would have the following pattern entries: city $ Lisbon: 1 Lisbon city: 1 district $ Lisbon: 2 Lisbon district: 2 Street $ Lisbon: 3 Lisbon Street: 3 (...)
Lisbon: 1,2,3,(...) The left size of these entries contains the text patterns to be matched, in [<feature type> $ <feature>] and in [<feature> <feature type>] formats (being the former one more common for Portuguese texts, and the latter one for English texts), while on the right side there is an identifier of the corresponding geographic concept in the ontology. The character $ means that an arbitrary term or group of terms is allowed to be present between the feature and the feature type, in order to avoid different stopword and adjective patterns. This approach immediately captures and grounds all geonames into their unique concept identifiers, without depending on hard-coded disambiguation rules. In the end, we have a catch-all pattern, which is used when the geoname found in the document does not contain any kind of external hints on its feature type. For these cases, we assign all identifiers of geographic concepts that are associated with the geoname Lisbon.
The geographic signatures (DSig) generated by Faísca consist on a list of concept identifiers and a
corresponding confidence measure (Con f Meas) normalized to [
To retrieve documents, Sidra5 first uses the what part of the query and the term index to retrieve the top
1000 documents. Afterwards, the DSig of each document is retrieved with the help of the geographic index.
Finally, the document score is obtained by combining the Okapi BM25 text score [
GeoSim(s1, s2) =0.5 × OntSim(s1, s2) + 0.2 × DistSim(s1, s2)+
0.2 × PopSim(s1, s2) + 0.1 × Ad jSim(s1, s2) Mean: GeoScore is the average GeoSim values computed between a (query, doc) pair. (1) (2) (5) Ranking(query, doc) = 0.5 × NormBM25(query, doc) + 0.5 × GeoScore(query, doc)
The calculation of GeoScore begins with the computation of the geographic similarity GeoSim for each
pair (s1, s2), where s1 in QSig and s2 in DSig, through a weighted sum of four heuristic measures (discussed
in our 2006 GeoCLEF participation [
Having geographic signatures with multiple concepts requires adding aggregation metrics to GeoScore for handling the different GeoSim values that a (query, doc) pair can generate. We experimented four metrics: Maximum, Mean, Boolean and Null.
Maximum: GeoScore is the maximum GeoSim value computed between a (query, doc) pair.
GeoScoreMaximum(query, doc) = max (GeoSim(s1, s2) × Con f Meas(s2)) , s1 ∈ Qsig ∧ s2 ∈ Dsig (3) GeoScoreMean(query, doc) = avg (GeoSim(s1, s2) × Con f Meas(s2)) , s1 ∈ Qsig ∧ s2 ∈ Dsig (4) Boolean: GeoScore equals 1 if there is a commom concept in a (query, doc) pair, and equals 0 otherwise.
GeoScoreBoolean(query, doc) = (1 if ∃ s1 = s2, s1 ∈ Qsig ∧ s2 ∈ Dsig 0 otherwise Null: GeoScoreNull is always 0, turning off the geographic scores. This is used as a baseline metric for comparing results obtained with the other metrics.
Geographic QE before RF. Classical text retrieval.
Geographic QE before RF, GIR with Mean geoscore.
Geographic QE before RF, Maximum geoscore.
Geographic QE after RF, Mean geoscore.
Geographic QE after RF, Maximum geoscore.
The computation of the four GeoScore metrics is illustrated in Figure 2, which presents a fictional query (Hungary), and two document surrogates, along with the GeoSim × Con f Meas values and final GeoScore values. 3
• evaluate if the current GeoIR approach of treating geonames in a separate geographic ranking obtains better results than treating geonames as terms in a standard IR approach; • determine which GeoScore combination metrics is best. We experimented the GeoScoreMean and GeoScoreMaximum on our runs. The GeoScoreBoolean and GeoScoreNull metrics were later included in post-hoc experiments; • measure the importance of the geographic query expansion before or after the relevance feedback step.
We generated initial queries from the topic titles to obtain initial runs for the RF. We used 32 top-k
terms and 20 top-k documents as parameters for the blind relevance feedback [
The Terms only experiment (run 1) uses early geographic reasoning to generate a QSig. Yet, it uses the names of geographic concepts as standard terms in the generation of the initial and final runs, meaning that this run uses only classical text retrieval.
The other runs use the text and geographic scores for ranking documents: Geographic QE before RF experiments (runs 2 and 3) considers the QSig as the where part of the initial query, for initial run and final run generation, while the Geographic QE after RF experiments (runs 4 and 5) use only the captured concepts on the topic title as the where part for the initial run generation, and the QSig on the final run generation. The Terms/GIR runs on these experiments differ by the use of the initial run generated in the Terms only experiment. 4
Unfortunately, the runs submitted to GeoCLEF were hampered by programming errors in our GIR prototype, and so the obtained poor MAP values did not allow us to draw any early conclusions regarding our experiments. After some code revision, we managed to obtain more significative MAP values and conducted additional experiments with the fixed GIR prototype. The MAP values presented on Table 3 refer only to the post-hoc experiments.
We observed that the GeoScoreMean produces poor MAP values, because long document signatures tend to cause query drifting. GeoScoreMaximum and GeoScoreBoolean revealed to be much more robust, and the GeoScoreBoolean metric has the best MAP values for Portuguese. This is explained in part because the GeoScoreMaximum is highly dependent on the heuristics used, and these are dependent on the quality of the geographic signatures and the quality of the ontology, while the GeoScoreBoolean metric is more straightforward on assigning maximum scores for geographically relevant documents. This difference also Initial run
0,175 0.086 0.089
0.093 0.104 0.166 00..103413 00..103454
0.081 0.087 b) Results for the English monolingual subtask. 0.175 0.218 0.044 0.204 0.208 means that there are more irrelevant documents that are being scored higher than relevant documents being scored lower by the GeoScoreMaximum.
Regarding the geographic query expansion before or after the RF, we found that early geographic expansion results in a better generation of initial runs (0.126 versus 0.084), meaning that more relevant documents are present on the top-k docs, thus improving the results from the RF step.
Using geonames as terms on the term index instead of geographic concepts still gets better results in the initial run (0.210 versus 0.126). The final run obtained without performing geographic ranking improves the MAP value to 0.233. We were intrigued with the consistent better results obtained with the Terms Only experiment. The good MAP value obtained by its initial run (0.210) suggested an experiment with this initial run, followed by a term and geographic expansion to generate a final query with a geographic signature, and ending in a GIR retrieval just like the other experiments. This Terms/GIR experiment obtained an MAP of 0.268 for the GeoScoreBoolean metric, the highest MAP value of our post-hoc experiments.
Regarding the English experiments, we observe similar trends as in the Portuguese experiments. The slightly lower values are consequence of the quality of the ontology, which is more complete with Portuguese feature names. Also, we observe that the GeoScoreMaximum outperformed the GeoScoreBoolean geoscope values for the Terms/GIR experiment, which prompt us to make further analysis on the meaning of the observed differences between these two metrics. 5
This year’s participation was a deception in terms of results for official runs, but we accept it as the consequence of deciding to develop a totally renewed and untested GIR system. Yet, the post-hoc experiments drew some interesting results for understanding why the GIR approaches are still outperformed by classic IR approaches. Our Terms/GIR experiments manage to obtain the highest MAP values, which might shed some light on this problem and suggest that there may be more efficient ways to introduce geographic reasoning in a GIR system.
The approaches of this year and last year’s participations are both very dependent of the quality of the geographic ontology. 25% of the relevant documents contained geonames that were not in our ontology, and we found that we have poor results when handling queries with unknown geonames. In addition, the ontology is not comprehensive on coordinates and population data to serve the geographic heuristics. We need to make further experiments with a more complete ontology, in order to better evaluate the fitness of the geographic similarities.
We also believe that our results could be improved with a more robust term query expansion module, as the current query expansion through blind relevance feedback is basic and does not produce significant improvements. We are also aware that some of the blame may be on the query construction step, as the readaptation for the MG4J syntax was overlooked. Our post-hoc experiments used RF parameters of eight top-k terms and five top-k docs, and used different logic operations for query construction. These changes resulted in significant improvement of the results, showing that we still have some tuning to do in the term query expansion step.
Finally, we conclude that this new GIR approach has its merits, and may be further improved to produce good results. Yet, it is still on its early steps, so our next work is to mature the approaches and develop a stable GIR prototype for further experiments.
We thank Joana Campos, for developing the text mining module, and Catarina Rodrigues for managing the geographic data. Our participation was jointly funded by the Portuguese government and the European Union (FEDER and FSE) under contract ref. POSC/339/1.3/C/NAC (Linguateca), and partially supported by grants SFRH/BD/29817/2006 and POSI/SRI/40193/2001 (GREASE) from FCT (Portugal), co-financed by POSI.