In this paper we describe the geographic information retrieval system developed by the Multimedia & Information Systems team for GeoCLEF 2006 and the results achieved. We detail our methods for generating and applying co-occurrence models for the purpose of place name disambiguation, our use of named entity recognition tools and text indexing applications. The presented system is split into two stages: a batch text & geographic indexer and a real time query engine. The query engine takes manually crafted queries where the text component is separated from the geographic component. Two monolingual runs were submitted for the GeoCLEF evaluation, the first constructed from the title and description, the second included the narrative also. We explain in detail our use of co-occurrence models for place name disambiguation using a model generated from Wikipedia. The paper concludes with a full description of future work and ways in which the system could be optimised.
In this paper we detail the MMIS team’s entry for GeoCLEF 2006. We have two objectives with our entry: to test the accuracy of our co-occurrence model generated from Wikipedia and to test whether the use of large scale co-occurrence models can aid the disambiguation of geographic entities.
We begin with a discussion of disambiguation methods, followed by a full outline of the system, we then present our experimental runs and results, concluding with an analysis and future work.
Methods of place name disambiguation can generally be split into three categories:
• Rule-Based methods, which use a series of hand crafted heuristic rules [
We use a rule-based approach to annotate how places occur in Wikipedia (taking advantage of structure and meta-data). This annotated corpus is then applied as a co-occurrence model using a data-driven method to annotate the GeoCLEF data. 2
Our geographic information retrieval system is split into two parts: the indexing stage and the querying stage. The Indexing stage requires the corpus and some external resources to generate the geographic and text indexes (a slow task). The querying stage requires the generated indexes and the queries; it runs in real time.
The Indexing stage consists of four separate applications: WikiDisambiguator is first used to
build a co-occurrence model of how place names occur together in Wikipedia [
The Querying stage consists of our Query Engine, which queries the text index and geographic index separately, combining the results (Figure 1). 2.1
Wikipedia is being used more and more in geographic information retrieval, it is extremely useful as
a resource due to its size, variation, accuracy and quantity of hyper-links and meta-data. Anyone
can contribute articles to Wikipedia meaning the diversity of articles is huge: to date there are
over 2 million articles and stubs (short articles) [
WikiDisambiguator is the application designed to build our co-occurrence model. The data gathered (collected from a crawl of every Wikipedia article1) takes the form of three database tables: links believed to be places and the order in which they occur; links believed to be nonplaces and the order in which they occur and a mapping of Wikipedia articles to TGN unique identifiers.
WikiDisambiguator uses rule-based methods of disambiguation. It is made up of two parts, the disambiguation framework and the method of disambiguation itself. Using Wikipedia as the corpus solves two problems: the problem of synonyms (multiple words referring to a single entity) is resolved as we can record how multiple anchor texts point to the same page; and the problem of polynyms (a single word referring to multiple places) can be solved with our disambiguation system. 2.1.1
The disambiguation framework is a simple framework to allow independent disambiguation methods to be slotted in.
The framework is outlined as follows: 1: The Wikipedia articles are loaded from the database 2: for each Wikipedia article all the links are extracted 3: for each link 4: if it has already been disambiguated as not a place 5: then add an entry to the db and continue 6: if the page pointed to has already been disambiguated as a place 7: then add an entry to the db and continue 8: else attempt to disambiguate using the Method of Disambiguation specified 9: end for 10: end for
The disambiguation methods is passed: • a list of candidate places • a list of names of places related to this link • the text making up the article that this link points to • the article title • how the link appeared in the text
The candidate places are taken from the Getty Thesaurus of Geographical Names. The candidate places for an article are places matching either the article’s title or the anchor text linking to the article. 2.1.2
Based on the results observed by running a series of simple disambiguation methods on test data,
we designed a disambiguation pipeline that could exploit the meta-data contained in Wikipedia
and strike a balance between precision and recall [
Each disambiguation method is called in turn (Figure 2). A list of candidate places is
maintained for each article, an article is denoted as unambiguous when this list contains one or zero
elements. Each method of disambiguation can act on the candidate places list in the following
1Our copy of Wikipedia was taken 3rd Dec 2005
ways: remove a candidate place; add a candidate place; remove all candidate places (disambiguate
as not a location); or remove all bar one candidate places (disambiguate as a location).
1. Disambiguate with templates – The template data in Wikipedia is highly formatted data
contained in name-value pairs. The format of the templates is as follows {{template name |
name = value | ...}}. The template name is used initially for disambiguation, for example
“Country” will indicate this page refers to a location of feature type nation or country.
Templates are also used to identify non-places, for example if the template type is “Biographic”
or “Taxonomic.” The name-value pairs within a template are also used for disambiguation,
e.g. in the Coord template a latitude and longitude are provided which can be matched to
the gazetteer.
2. Disambiguate with categories – The category information from Wikipedia contains softer
information than the template information [
News articles have a large number of references to named entities that quickly place the reader into the context of the news piece. Sometimes the same named entity is referred to in different ways 2In our implementation importance is based on the feature type recorded in the gazetteer. (e.g. “British prime minister”, “Mr. Blair”, “Tony Blair”). Thus, the detection of references to all named entities is the problem that we addressed in this part of the system. This part receives as input the GeoCLEF news articles and outputs the named entities of each news article, which will be used by the Disambiguator.
Named entity recognition systems rely on lexicons and textual patterns either manually crafted
or learnt from a training set of documents. We used the ESpotter named entity recognition system
proposed by Zhu et al. [
ESpotter uses a database to store the lexicon and the textual pattern information. It can be easily customised to recognise any type of entities one might be interested in by adding new lexicon and textual patterns. The database we used is the one supplied by Zhu et al., we did not create a database of GeoCLEF based lexicon and patterns. 2.3
The news article corpus was indexed with Apache Lucene 2.0 [
To allow the returned results to be pruned geographically the data needs to be geographically indexed. We take the named entities tagged as locations output by the Named Entity Recogniser and disambiguate them based on how they co-occur in our co-occurrence model.
Having the corpus indexed with place names, we could apply our co-occurrence model to disambiguate the places to distinct locations as listed in the Getty TGN. Our method is a Na¨ıve Bayesian approach designed to maximise speed and implemented in the application Disambiguator. 1: for all documents 2: for each adjacent tuple of place names 3: for each possible location for either place name 4: disambiguate as the places that most often appear together 5: end for 6: end for 7: end for
Possible locations are defined as any location appearing in our co-occurrence model that has
been referred to by the same toponym as the named entity extracted from the corpus. The
geographic index is then stored in a Postgres database and indexed with an R-Tree (to allow
efficient processing of spatial queries) [
The queries are manually split into a text component and a geographic component. The text query is handled normally by Lucene, the geographic query is manually split into a tree of conjunctions and disjunctions. Once the news articles are indexed with Lucene, the query terms will be extracted in the same way that the document terms were, a similarity measure is taken between the query’s terms and all indexed documents. The similarity function is given by the following expression: score(q, d) =
Pt∈d tft(d) · idf2(d ∋ t, D) · norm(d)
qPt∈d tft2(d)
,
where Pt∈d tft(d) is the t term frequency for the given document d (in our case is 0 or 1),
idf(d ∋ t, D) is the frequency of documents d containing the term t in the D collection, and
norm(d) is a normalization constant given by the total number of terms in document d. See [
The query trees are constructed by hand. The nodes of the tree are either conjunctions or disjunctions while the leaves of the tree are (spatial-relation, location) pairs see Figure 3. For each document that matches the text query we check whether it refers to a place matching the geographic query – any documents not matching the geographic query are removed (Figure 4). 1: fetch all documents that satisfy the text query from Lucene 2: for all documents 3: prune results against the geographic query 4: end for 5: return remaining documents 3
We entered two runs for GeoCLEF 2006: Both were mono-lingual, English queries on an English corpus with manually constructed queries. Our first run used queries constructed from the title and description, the second run also took into account the narrative. As far as was possible we attempted to add no world knowledge, the query trees we produced resembled what could be produced with a query parser. Our runs appeared between the 25% quantile and the median for mean average precision (see table below). The run consisting of queries constructed from Title, Description and Narrative (TDN) generally out performed the run constructed from Title and Description (TD); in Figure 5 we compare the average precision of our runs for each query against the maximum average precision achieved by any system. Our system as presented here uses a simple approach to the application of co-occurrence models for place name disambiguation, text indexing and the combination of text and geographic queries.
The system gave results appearing slightly below the median MAP; this shows the system model is valid; however, there is significant room for improvement. Without further tests we cannot comment on specific parts of the system; each of the five applications will have to be tuned independently.
With respect to our objectives we can conclude that the co-occurrence model accuracy agrees
with the previous experiments conducted in [
We are currently exploring whether we can improve our results by applying co-occurrence models in more sophisticated ways. The three methods currently being worked on are: • Using a generalised Jaccard co-efficient to produce a co-occurrence index • Learning a hierarchical decision list • Applying Latent Semantic Indexing to build place-name neighbourhoods
We hope after a study of these methods to evaluate the suitability of using co-occurrence models for place-name evaluation and to identify the optimal method.
ESpotter utilises an Access database for Named Entity Recognition; we would like to see if it is possible to optimise this database for use with the GeoCLEF corpus and general place name recognition.
Lucene was applied in the default configuration and the text part of the queries were not altered in any way. We plan to experiment with suitable query weights for Lucene and try alternative configurations of the index. Ultimately we would like to combine the geographic and text indexes so that they can be searched and applied simultaneously.
We also plan to implement a query parser to allow the queries to automatically be parsed into query trees; this would require a level of natural language processing.