This paper summarizes the work done at the State University of New York at Buffalo (UB) in the GeoCLEF 2006 track. The approach presented uses pure IR techniques (indexing of single word terms as well as word bigrams, and automatic retrieval feedback) to try to improve performance of queries with geographical references. The main purpose of this work is to identify the strengths and shortcomings of this approach so that it serves as basis for future development of a geographical reference extraction system. We submitted four runs to the monolingual English task, 2 automatic runs and two manual runs, using the title and description fields of the topics. Our official results are above the median system (auto=0.2344 MAP, manual=0.2445 MAP). We also present an unofficial run that uses title description and narrative which shows a 10% improvement in results with respect to our baseline runs. Our manual runs were prepared by creating a Boolean query based on the topic description and manually adding terms that are consulted from geographical resources available on the web. Although the average performance of the manual run is comparable to the automatic runs, a query by query analysis shows significant differences among individual queries. In general, we got significant improvements (more that 10% average precision) in 8 of the 25 queries. However, we also noticed that 5 queries in the manual runs perform significantly below the automatic runs.
Details about the GeoCLEF document collection are note discussed in this paper but the reader is referred to the GeoCLEF overview paper. Our document collection consists of 169,477 documents from LA Timas and The Glasgow Herald. Processing of English documents followed a standard IR approach discarding stop words and using Porter’s stemmer. Additionally we added word bigrams that identify pairs of contiguous non-stop words to form a two word phrase. These bigrams allowed a stop word to be part of the bigram if they included the word “of” since it was identified as common component of geographical names (i.e. “United_Kingdom” and “City_of_Liverpool” would be a valid bigrams). Documents were indexed using the vector space model (as implemented in the SMART system) with two ctypes. The first ctype was used to index words in the title and body of the article while the second ctype represented the indexing of the word bigrams previously described.
To process the topics we followed the same approach described above (using stop words, stemming, and adding word bigrams). Each query was represented using two ctypes. The first ctype for single word terms extracted from the parts that will be used in the query (i.e. title and description). For our official runs we only use the title and description.
We designed a way to identify geographical features and expand them using geographical resources but due to the short time available for developing we could not include it in our official runs. For this reason we submitted results using a pure IR approach for this year and work on the development of the geographical feature extraction for next year. Our results should be considered as baseline results. One of the authors created a manual version of the queries using geographical resources available on the internet and writing a Boolean query. This manual run was included in the official results. We also explore automatic retrieval feedback of both automatic and manual queries.
We use a generalized vector space model that combines the representation of the two ctypes and weights the r r contribution of each part in the final similarity score between document di and query q . The final score is computed as the linear combination of ctype1 (words) and ctype2 (bigrams) as follows:
r r r r r r sim(d i , q) = λ * simwords (d i , q) + μ * simbigrams (d i , q) Where λ and µ are coefficients that control the contribution of each of the two ctypes. The values of these coefficients are computed empirically using the optimal results in the GeoCLEF 2005 topics. The similarity values are computed using pivoted length normalization weighting scheme[4] (pivot=347.259, slope= 0.2). We also performed automatic retrieval feedback by retrieving 1000 documents using the original query and assuming that the top n documents are relevant and the bottom 100 documents are not relevant. This allows us to select the top m terms ranked according to Rocchio’s relevance feedback formula[2]: wnew (t) = α ∗ worig (t) + β ∗ i∈Re l
| Re l | ∑ w(t, d i )
∑ w(t, d i ) − γ ∗ i∈¬ Re l | ¬ Re l | Where α, β, and γ are coefficients that control the contribution of the original query, the relevant documents (Rel) and the non-relevant documents (¬Rel) respectively. The optimal values for these parameters are also determined using the CLEF 2005 topics. Note that the automatic query expansion adds m terms to each of the two ctypes. We first tested our baseline system using the GeoCLEF2005 topics. We used the title, description and geographic tags. Table 1 shows the performance values for the baseline run and for the best run submitted to GeoCLEF 2005[1] (BKGeoE1). The mean average precision for this baseline run is 0.3592 which is pretty good and would have been among the top 3 systems in GeoCLEF 2005. This certainly indicates that a pure IR system was enough to answer most of the topics proposed last year.
UB
Baseline A query by query analysis reveals that the IR approach performs well in many topics but there are a few that could be improved (See Table 2). The system did not perform well in topics 2, 7, 8, 11 and 23. After analyzing these topics we conclude that most of them could have performed better if we had use some sort of expansion of continents using the countries located in them (i.e. European countries).
We submitted four official runs: two using automatic query processing and two using manual methods. As expected our results (both automatic and manual) performed above the median system. Results are presented in Table 3.
The automatic runs perform slightly above the median system which indicates that the set of topics for this year where harder to solve using only IR techniques. After taking a look to the official topics we realize that we could have used a better expansion method using the geographical resources (i.e identifying queries that have specific latitude and longitude references to restrict the set of retrieved results).
On the other hand, the manual queries perform in average similarly to the automatic runs but a query by query analysis reveals that there are quite a few queries that outperform significantly the automatic runs. However, at the same time there are two queries that perform significantly below the automatic systems. Note that the first manual run (UBGManual1) does not use automatic feedback while the second manual run (UBGManual2) uses automatic retrieval feedback. This merits further analysis to identify those strategies that are successful in improving performance.
Run Label UBGTDrf1 (automatic feedback)
We also noted that our best run (not submitted) performs quite well with respect to our baseline official runs. This run uses title, description and narrative, and conservative retrieval feedback parameters (n=5 documents and m=5 terms). It is also encouraging that this run, when compared to the manual run, captures several of the good terms that were added manually.
1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1
0 -0.1 -0.2 -0.3 -0.4 -0.5 -0.6 -0.7 -0.8 -0.9 -1 1 2 3 4 5 6 7 8 9
Figure 2 Comparison of best manual run and best automatic run using our system This paper presents an IR based approach to Geographical Information retrieval. Although this is our baseline system we can see that the results are competitive, especially if we use the long topics (title description and narrative). We still need to do more in depth analysis of the reasons why some manual queries improved significantly with respect to the median system and the problem presented in 5 queries that did perform significantly below the median. We plan to explore way to generate automatic geographic references and ontology based expansion for next year.