This paper reports the participation of the XLDB team from the University of Lisbon at the 2008 GeoCLEF task. We focused on developing a better text annotation tool for geo-parsing the documents, handling both explicit geographic evidence (as given by placenames) and implicit geographic evidence (as given by monuments, for example). The query processing and geographic ranking approaches were redesigned to handle thematic and geographic criteria of each search in a non-segregation way. We detail the GIR system, describe the optimisation procedure that preceded the run generation, and dissect the results.
Capturing additional geographic evidence from documents. People describe places of their interest in
several other ways, other than explicit placenames. Entities such as “Big Apple”, “Kremlin” or “UE
Headquarters” are easily connotated to their respective locations, and these entities might have a
decisive role on the defining the geographic scope (that is, the geographic area of interest) of the
document. Our shallow text mining approaches often failed to capture essential geographic evidence
to geo-reference many documents, and this naïve text mining approach was reflected on poor retrieval
results for some geographically challenging topics [
Smoothing the effects of text and geographic query expansion. Query expansion (QE) is known to
improve IR performance in most queries, but often at the cost of degrading the performance on other
queries [
For this year’s participation, we addressed these topics on the main improvements made in our GIR
system, namely:
Query Processing: We now handle placenames as both geographic criteria and as plain query terms. In
contrast to our initial ideas, placenames revealed to be in fact good retrieval terms, and they were
frequently selected as the top ranking terms in the blind relevance feedback (BRF) process [
Text mining: We developed a new named entity recognition module, REMBRANDT, and used it as a text
annotation tool to identify and classify all kinds of named entities in the CLEF collection [
Document Processing: To cope with the new approaches on query processing, we needed a simple
ranking model that elegantly combined the text and geographic subspace models, eliminating the need
for merging text and geographic ranking scores, while still allowing us to assign a weight for each
model on the retrieval. Therefore, we extended MG4J to suit our requirements for this year’s
experiments [
The rest of the paper is organised as follows. Section 2 outlines our GIR prototype and describes in detail each module. Section 3 presents the optimising steps and the configurations selected for the submitted runs. Section 4 dissects both the official results in GeoCLEF and our post-hoc evaluation results, and Section 5 concludes the paper with insights drawn from this participation. 2
Figure 1 describes the architecture of our GIR prototype. In a nutshell, the CLEF topics are pre-processed by QuerCol, generating query strings in MG4J syntax. The CLEF documents are geo-parsed by REMBRANDT, a named entity recognition module, that plays the role as a text annotation tool and identifies GeoCLEF topics
QuerCol 2008
Query expansion Geographic Ontology REMBRANDT Text annotation tool
MG4J Indexing & Ranking
named entities that have geographic evidence, generating the geographic document signatures (Dsig). Afterwards, the text and Dsig of the documents are indexed by MG4J. The document retrieval uses an optimised BM25 weighting scheme and receives the query strings from QuerCol, generating results in the trec_eval format. The geographic ontology assists only QuerCol in its geographic term expansions, as REMBRANDT and MG4J use other geographic knowledge resources, as described further in this section. 2.1
REMBRANDT REMBRANDT is a language-dependent named-entity recognition (NER) system that uses Wikipedia as a raw knowledge resource, and explores the Wikipedia document structure to classify all kinds of named entities in the text. By using Wikipedia, REMBRANDT obtains additional knowledge on every named entity that can be useful for understanding the context, detecting relationships with other named entities, and use this information to contextualise and classify surrounding named entities in the text.
One example of this additional knowledge in practice is the use of the Wikipedia page categories to derive implicit geographic evidence for each named entity. REMBRANDT handles category strings as text sentences and searches for place names in a similar way as it is performed on normal texts, generating a list of captured place names that are considered as implicit geographic evidence for the given named entity.
REMBRANDT currently classifies named entities using the 9 main categories and 47 sub-categories
defined by the second edition of HAREM, a NER system evaluation contest for Portuguese [
The REMBRANDT classification strategy relies on mapping each named entity to a Wikipedia page and
subsequently analysing its document structure, links and categories, searching for suggestive evidences.
REMBRANDT also relies on manually crafted rules for capturing internal and external evidence of named
entities for both Portuguese and English texts, as suggested by McDonald [
The classification is best illustrated by following how the example named entity, “Empire State Building”, is handled: the english Wikipedia page of the Empire State Building (en.wikipedia.org/wiki/ Empire_State_Building) is labelled with 10 categories, such as “Skyscrapers in New York City” and “Office buildings in the United States”. With this information, REMBRANDT classifies the named entity as a PLACE/HUMAN/CONSTRUCTION. In the hypothetical case that this named entity could not be mapped to a Wikipedia page, internal evidence rules, such as the presence of the term “Building” in the end, can classify the named entity as a PLACE/HUMAN/CONSTRUCTION. Finally, external evidence rules check the context on
PLACE/PHYSICAL/ISLAND PLACE/PHYSICAL/WATERCOURSE PLACE/PHYSICAL/WATERMASS PLACE/PHYSICAL/MOUNTAIN PLACE/PHYSICAL/REGION PLACE/PHYSICAL/PLANET PLACE/HUMAN/REGION PLACE/HUMAN/DIVISION PLACE/HUMAN/STREET PLACE/HUMAN/COUNTRY
EVENT/PASTEVENT EVENT/ORGANIZED EVENT/HAPPENING PLACE/HUMAN/CONSTRUCTION ORGANIZATION/ADMINISTRATION ORGANIZATION/INSTITUTION ORGANIZATION/COMPANY
THING/CLASS THING/CLASSMEMBER THING/OBJECT THING/SUBSTANCE PLACE/VIRTUAL/MEDIA PLACE/VIRTUAL/ARTICLE PLACE/VIRTUAL/SITE PERSON/POSITION PERSON/INDIVIDUAL PERSON/INDIV.GROUP PERSON/POSIT.GROUP PERSON/MEMBER PERSON/MEMBERGROUP PERSON/PEOPLE VALUE/CURRENCY VALUE/CLASSIFICATION VALUE/QUANTITY
ABSTRACTION/DISCIPLINE ABSTRACTION/STATE ABSTRACTION/IDEA ABSTRACTION/NAME MASTERPIECE/WORKOFART MASTERPIECE/REPRODUCED MASTERPIECE/PLAN TIME/GENERIC TIME/DURATION TIME/FREQUENCY TIME/HOUR TIME/INTERVAL TIME/DATE which the named entity is inserted, ensuring that the named entity is not referred in another context (for example, as an hypothetical movie, street or restaurant name). For the detection of implicit geographic evidence, the categories “Skyscrapers in New York City” and “Office buildings in the United States” are handled by REMBRANDT as additional text, and the place names “New York City” and “United States” are captured and listed as implicit geographic evidence associated to the named entity “Empire State Building”. From REMBRANDT annotations to geographic document signatures Each CLEF document annotated with REMBRANDT contains a list of named entities that might convey explicit or implicit geographic evidence. We can now generate rich geographic document signatures Dsig by adding named entities that have explicit geographic evidence, together with the placenames that were associated as implicit geographic evidence for other named entities. We divide the 47 sub-categories of named entities into 3 levels of eligibility, as depicted in Table 1: 1. Sub-categories that have explicit geographic evidence: all sub-categories under the main category PLACE, with the exception of the sub-categories PLACE/HUMAN/CONSTRUCTION and
main, but the PLACE/VIRTUAL/* sub-categories span virtual places such as web sites, newspaper articles or TV programs, and therefore are not eligible for inclusion in the geographic signatures. In HAREM, the subcategory PLACE/HUMAN/CONSTRUCTION is included in the PLACE main category, precisely because of its strong geographic connotation, but it is not an explicit geographic entity. As such, the subcategory PLACE/HUMAN/CONSTRUCTION is handled in the next level. 2. Sub-categories that have implicit geographic evidence: the categories ORGANIZATION, EVENTS and sub-category PLACE/HUMAN/CONSTRUCTION. The category ORGANIZATION spans institutions and corporations, such as city halls, schools or companies, which are normally related to a defined geographic area of interest. The category EVENTS spans organised events that normally take place in a defined place, such as olympic games, rock concerts or conferences. 3. Sub-categories that have no geographic evidence: categories considered to have no significative contribution for the geographic signatures. It spans the categories PERSON, THING, ABSTRACTION, MASTERPIECE, TIME and VALUE, and sub-categories PLACE/VIRTUAL/*.
“Tall buildings in the USA” l o C r e u Q
Terms Tall, Buildings, USA
Ontology Geo:Terms Expansion USA
BRF Expansion tsokwysecrsra,peemrsp,irbeu.rj, biggest, tallest, united states, america california, seattle, washington, los angeles, chicago, san francisco, (...) text:tall{1.0} | text:buildings{1.0} | text:usa{1.0} | text:skyscraper{0.9} | text:burj{0.8} | text:towers{0.7} | text:empire{0.6} | text:america{0.5} | local:usa{1.0} | local:california{0.5} | local:(los angeles){0.333}...
This eligibility table of named entity classifications into Dsig signatures is a simplification exercise, and it is far from consensual. It is questionable whether categories such as PERSON can also convey a significative geographic evidence to define the document scopes. For instance, the named-entity “Nelson Mandela”, as processed by REMBRANDT, is associated to “South Africa” as its implicit geographic evidence because the Wikipedia page of Nelson Mandela (en.wikipedia.org/wiki/Nelson_Mandela) contains the category “Presidents of South Africa”. Yet, this geographic evidence may cause the drift from the document scope, because not all documents mentioning “Nelson Mandela” have the South African territory as their geographic scope.
On the other hand, we are assuming that all captured geographic evidence is relevant for the document scope, but this is not always true. Take for instance the named entity example “Empire State Building”; while it conveys an implicit location when it is addressed, for example, in a context of office headquarters, it is not important for the document scope when it is addressed on a context of its architectural style. 2.2
QuerCol
QuerCol’s query reformulation has two different procedures: first, it uses blind relevance feedback (BRF)
for selected terms, and secondly, it performs geographic query expansion for geographic terms, by
exploring the relationships between geographic concepts on a geographic ontology [
Figure 2 illustrates the two different expansion procedures of QuerCol, for the example query “Tall
buildings in the USA”. First, QuerCol removes the stopwords from the query, and recognises the geographic
terms with the help of REMBRANDT. Afterwards, the non-stopwords tall, buildings and usa are expanded
through BRF, using the wt (pt -qt ) algorithm to weight terms in a normalised scale of [
On the other hand, the geographic term “USA” is grounded to the geographic concept ‘United States of America (country)’, triggering the ontology-driven geographic query expansion that searches for other geographic concepts known to be contained within the USA territory. The new geographic terms are then re-weighted according to the ontology node distance between the root node and the leaf node by the formula 1 . For the given example, USA generates 50 states with a weight of 21 and several cities with weight 13 n 1 (considering that the node distance in the ontology between states and countries is 1, and between cities and countries is 2). In the end, terms are labelled as search terms for the explicit local and implicit local index fields. These list of terms is desigated as the query geographic signature, the Qsig. 2.3
MG4J indexing and ranking MG4J is responsible for the indexing and retrieval of documents. MG4J indexes the text of CLEF documents into a text index field, while the Dsig of the documents is divided in two geographic indexes: the explicit local and implicit local index fields, according to each type of geographic evidence. Figure 3 presents an example of REMBRANDT’s annotation and subsequent MG4J indexing steps.
Source documents I visited the Empire State Building, on my trip to the USA.
t d n a r b m e R
Tagged documents I visited the <CONSTRUC TION>Empire State Building</CONSTRUCTION> <LOCALITY>New York </LOCALITY>, on my trip to the <LOCAL>USA<LOCAL>.
J 4 G M
text Indexes i, building, empire, my, on, state, the(2), to, trip, usa, visited expl.local
usa impl.local new york
We define term similarity as the similarity between query subjects and document subjects, and computed with the use of BM25 on the text index field only, and geographic similarity as the similarity between geographic signatures of queries and documents (Qsig and Dsig), computed with the use of BM25 on the explicit local and implicit local index fields. MG4J allows us to dynamically select the indexes to be used in the retrieval, and to change the weight of a field before retrieval.
Unfortunately, we were not able to extend the BM25 implementation on MG4J to support term weight, so all the terms weights were set to the defalut value of 1 for all the generated runs. 3
The run generation procedure is depicted in Figure 4. In an initial step, QuerCol processes the topics and performs only a geographic query expansion, generating an ontology expanded query. The ontology expanded query is submitted to MG4J, generating the initial run. The initial run with the best men average precision (MAP) value is chosen as the source of relevance for the blind relevance feedback step, performed by QuerCol on the ontology expanded query. In the end, QuerCol generates a final BRF + ontology expanded query, which is submitted to MG4J to generate the final run.
GeoCLEF topic
Ontology
QuerCol Expansion Ontology Expanded
Query
Initial Run
QuerCol BRF + Ontology Expanded
Query
BRF Expansion
Final
Run M G 4 J
M G 4 J We thoroughly optimised the parameters of our GIR prototype, so that we could minimise the effect of a detuned GIR system on the MAP values, and increase our confidence that the results are a direct consequence of the approaches being evaluated. Regarding the document retrieval, we optimised the b and k1 parameters of BM25, and we experimented different weights for the text, explicit local and implicit s e u l a v P A M Default BM25 parameters: k1=2.0 b=0.75 Default index weights: 1.0, 0.0, 0.0
Optimised BM25 parameters
Optimised index weights Generate final queries with: 4, 8, 12, 16 top-k-terms 5, 10, 15 top-k-docs Optimised BM25 parameters
Optimised index weights
Optimisation step local index fields. For the QE step, we experimented with different blind relevance feedback parameters: the number of terms added for the final query, top-k-terms, and the number of top documents considered relevant, top-k-docs.
Figure 5 describes the optimisation procedure performed on our GIR prototype with the help of the 2007 GeoCLEF data. We started with the default values for the BM25 parameters, and using only the text index field, we searched for the best BM25 values that generated the optimal MAP values. Then, with these BM25 parameters, we then optimised the index weight values according to the MAP values. The best initial run fed the blind relevance feedback process, and the optimisation procedure was repeated for the final run stage, using several combinations of top-k-terms and top-k-docs. 3.2
Configuration of the official runs
#1, #2, #3 0.4 0.9 {2.0, 2.5, 3.0} 0.25 0.0 - - - - - - #4, #5, #6 0.4 0.9 2.5 0.25 0.0 8 5 0.95 0.3 {2.0, 2.5, 3.0} 0.25 0.0 #7, #8, #9 0.4 0.9 2.5 0.25 0.0 8 5 0.65 0.35 {2.0, 2.5, 3.0} 0.25 0.0 #10,#11, #12 0.4 0.9 2.5 0.25 0.0 8 5 0.65 0.5 {2.0, 2.5, 3.0} 0.25 0.0
#1, #2, #3 0.65 1.4 {1.5, 2.0, 2.5} 0.5 0.0 - - - - - - #4, #5, #6 0.65 1.4 2.0 0.5 0.0 8 15 0.65 1.4 {1.5, 2.0, 2.5} 0.5 0.0 #7, #8, #9 0.4 0.9 2.5 0.25 0.0 8 10 0.65 0.35 {1.5, 2.0, 2.5} 0.5 0.0 #10,#11,#12 0.4 0.9 2.5 0.25 0.0 8 5 0.65 0.5 {1.5, 2.0, 2.5} 0.5 0.0
In 2008, GeoCLEF allowed up to 12 official runs to be submitted, for each of the monolingual subtask. We submitted a total of 12 runs for each subtask, using the most promising parameters from the optimisation procedure, with a sligh variation on the index weights. Table 2 resumes the parameter values used for the official runs.
The official runs are composed by initial runs (#1 to #3) and final runs (#4 to #12). We experimented different ratios of text / explicit local index weights, by increasing and decreasing the text index weight by 0.5.
During the optimisation, we observed that the BM25 optimisation for the Portuguese subtask presentes many local optimal MAP values, so we decided to submit runs with three BM25 configurations from different areas, to increase the odds of standing near a global optimal BM25 parameter. For the English optimisation, we observed that the BRF parameters had more influence on the optimal MAP values than the BM25 parameters, so we submitted runs with different BRF parameter values. Also worth mentioning is the fact that the implicit local index field did not improved MAP values in any optimisation scenario, and thus it was turned off on all official runs. 4
b k1 MAP text exp.l. imp.l. MAP terms docs PT3 0.4 0.9 0.2222 2.5 0.25 0.0 0.2234 - EN6 0.65 1,4 0.2519 2.0 0.5 0.0 0.2332 8 15
b k1 MAP text exp.l. imp.l. MAP - - - - - - 0.65 1.4 - 2.5 0.5 0.0 0.2755
Table 3 presents the best GeoCLEF official runs (top part) and the best optimisation values for the GeoCLEF evaluation data from 2006 to 2008 (top part), for the Portuguese and English monolingual subtasks. We observe that our best Portuguese run was in fact an initial run (with a MAP of 0.2234), and the post-hoc optimisation corroborated the fact that the best MAP values for Portuguese are achieved by initial runs (with the best MAP value of 0.2301), which is somewhat unexpected. For the English subtask, the best run was indeed a final run, achieving a MAP value of 0.2755, that could be pushed further up to 0.2814 with optimised parameters.
The results show that the use of explicit local index field on the retrieval process improves the results in all evaluation scenarios, while the explicit local index field does not contribute at all to the improvement of the retrieval results. This fact proves that the GIR prototype is able to outperform a classic IR system in a consistently way, but it also contradicts our initial beliefs that implicit geographic evidence would have an important role on the Dsig. In fact, we only observe that the explicit local index field takes part on the best MAP values for GeoCLEF 2006, which we think that it might be related to the geographically generic topics used in that year (mostly about countries and continents), which favours the implicit geographic evidence (that is also normally given by countries and continents).
Another topic of interest of the results is that we were not so far away from the optimal MAP values as we initially expected to be, and thus we believe that we did not over-fitted our GIR system with the 2007 data. Nonetheless, the post-hoc optimisation revealed that the English topic sets are quite balanced, where we consistently achieved MAP values around 0.28, while the difficulty of the Portuguese topic sets is more unpredictable. This reveals how important it is to tune up a system according to the collections and topics, as the default parameter values rarely produce good results. This year, we participated in GeoCLEF with the purpose of maturing the ideas first coined on last year’s participation, namely: i) the document geographic signatures must be more comprehensive, extraction all kinds of geographic evidence that can be derived from all named entities in the text, and ii) the thematic and geographic facets of each search are not antagonic, and could be used together to retrieve documents in a common weighting scheme that can elegantly combine term and geographic index fields.
The results showed that our GIR prototype is consistently better when using the geographic indexes on the retrieval, meaning that our GIR approach outperforms a classic IR retrieval in every GeoCLEF evaluation scenario since 2006. For future work, we plan to improve REMBRANDT’s strategy for capturing implicit geographic evidence, as we believe that its naïve approach generated noisy signatures and it was responsible for the futility of the implicit local index field. We also want to develop a new adaptive strategy for QuerCol, as the optimal QE parameters vary for each topic, and using the same configuration set for all topics generates sub-optimal expanded queries. We also plan to rebuild the BM25 implementation of MG4J, so that term weights can be properly used on document retrieval.
Acknowledgements We thank David Cruz and Sebastiano Vigna for the modifications made to MG4J according to the experiments, and to Marcirio Chaves for updating the geographic ontology.