This paper describe the joint participation by the Universidad Politecnica de Valencia and the Universidad of Jaen to the GeoCLEF task. This activity has been carried out within the framework of the Spanish TextMESS project (Intelligent, Interactive and Multilingual Text Mining based on Human Language Technologies). The method employed for the participation is a result merging algorithm based on the fuzzy Borda voting scheme. This method takes as input the two document lists returned by the two systems developed by the participating groups and creates a document list where the documents are ranked according to the fuzzy Borda voting scheme. The results obtained are better than the individual systems, and also ones of the best ones of the task (second as group). However, the best result was obtained with a run which combined the baseline systems. The analysis of the results showed that the best runs were those in which only title and description were used, and unfortunately we chose to submit only a run of this type, with the base systems. The results con rm the e ectiveness of the fuzzy Borda scheme for the combination of di erent systems.
In this paper we describe the joint participation of the groups of the Universidad Politecnica de Valencia and Universidad de Jaen to GeoCLEF 2008. This participation has been carried out within the framework of the Spanish project TextMESS (Spanish acronym for Intelligent, Interactive and Multilingual Text Mining based on Human Language Technologies).
We previously investigated various possibilities for the integration of our systems, focusing on two possible choices:
Identify some features in the topics that allow to determine which system is going to obtain the best result over a determined topic (system selection);
Combine the output of the di erent systems in a unique output (output merging ).
We carried out some preliminary experiments with the GeoCLEF topics from 2005 to 2007 and the
systems presented by the two groups in GeoCLEF 2007, in order to check whether the rst option
was feasible or not. These results proved that it was possible to use bag-of-words features to select
the best system for a given topic. However, the experiments carried out with the new systems
(those developed for GeoCLEF 2008) did not provide us with the same conclusion. Therefore, we
chose to participate with an output merging algorithm based on the fuzzy Borda voting scheme
[
In Sections 2 and 3 we describe brie y the systems of each group (a more complete description can be found in the corresponding report of the CLEF Working Notes). In Section 4 we describe the fuzzy Borda ranking method, and nally we present the results and a brief discussion. 2
The SINAI-GIR system is made up of ve main subsystems: Translator, Collection Preprocessing
subsystem, Query Analyzer, Information Retrieval subsystem and Validator. Each translated query
is preprocessed and analyzed by the Query Analyzer, identifying their geo-entities and spatial
relations and making use of Geonames gazetteer2. This module also applies query reformulation based
on the query parsing subtask [
The main features of each subsystem are:
Translator. We have used SINTRAM (SINai TRAnslation Module), our Machine
Translation system which works with di erent online machine translators and implements several
heuristics to combine di erent translations [
Collection Preprocessing Subsystem. During the collection preprocessing, two indexes
are generated (locations and keywords indexes). We apply the Porter stemmer [
Query Analyzer. It is responsible for preprocessing of English queries as well as the generation of di erent query reformulations.
Validator . The aim of this subsystem is to lter the lists of documents recovered by the IR subsystem, establishing what of them are valid, depending on the locations and the georelations detected in the query. Another important function is to establish the nal ranking of documents, based on manual rules and prede ned weights. 3
The system is built around the Lucene5 open source search engine, version 2:1. The Stanford
NER system based on Conditional Random Fields [
During the indexing phase, the documents are examined in order to nd location names (toponym) by means of the Stanford NER system. When a toponym is found, the disambiguator determines the correct reference for the toponym. Then, a modi ed lucene indexer adds to the geo index the toponym coordinates (retrieved from GeoWordNet); nally, it stores in the wn index the toponym together with its holonyms and synonyms. All document terms are stored in the text index. The indices are then used in the search phase, although the geo index is not used for search: it is used only to retrieve the coordinates of the toponyms in the document.
3.2
The architecture of the search module is shown in Figure 2.
The topic text is searched by Lucene in the text index. The toponyms extracted by the Stanford
NER are searched for in the wn index with a weight 0:25 with respect to the content terms. The
result of the search is a list of documents ranked using the Lucene's weighting scheme. At the
same time, the toponyms are passed to a module named GeoAnalyzer that creates a geographical
constraint that is used to re-rank the document list. The GeoAnalyzer may return two types of
geographical constraints:
a distance constraint, corresponding to a point in the map: the documents that contain
locations closer to this point will be ranked higher;
an area constraint, correspoinding to a polygon in the map: the documents that contain
locations included in the polygon will be ranked higher. The polygon is obtained by
calculating the convex hull of the points associated to the toponyms using the Graham algorithm
[
WordNet is used by the GeoAnalyzer module in order to extract the meronyms of the toponyms in the topic. These meronyms allow to improve the precision of the area constraint.
The objective of the GeoFilter module is to re-rank the documents retrieved by Lucene, according to geographical information. If the constraint extracted from the topic is a distance constraint, the weights of the documents are modi ed according to the following formula: w(doc) = wLucene(doc) (1 + exp( min d(q; p))) p2P (1)
Where wLucene is the weight returned by Lucene for the document doc, P is the set of points in the document, and q is the point extracted from the topic.
If the constraint extracted from the topic is an area constraint, the weights of the documents are modi ed according to formula 2: w(doc) = wLucene(doc) 1 + jPqj (2) jP j where Pq is the set of points in the document that are contained in the area extracted from the topic. 4 4.1
In the classical (discrete) Borda count each expert gives a mark to each alternative, according
to the number of alternatives worse than it. The fuzzy variant [
Let R1; R2; : : : ; Rm be the fuzzy preference relations of m experts over n alternatives x1; x2; : : : ; xn. Each expert k expresses its preferences by means of a matrix of preference intensities: where each rikj = Rk (xi; xj ), with Rk : X X ! [0; 1] is the membership function of Rk. The k number rij 2 [0; 1] is considered as the degree of con dence with which the expert k prefers xi to xj . The nal value assigned by the expert k to each alternative xi is the sum by row of the entries greater than 0:5 in the preference matrix, or, formally: 0 r11 k k B r21 B@ : : : k rn1 k r12 k r22 : : : k rn2 : : : r1kn 1 : : : r2kn C : : : : : : CA
k : : : rnn rk(xi) = n
X
j=1;rikj>0:5
k
rij
r(xi) =
m
X rk(xi)
k=1
(3)
(4)
(5)
The threshold 0:5 ensure the relation Rkto be an ordinary preference relation [
The fuzzy Borda count for an alternative xi is obtained as the sum of the values assigned by each expert: 4.2
In our approach each system is an expert: therefore, there are two preference matrices. The size of these matrices is variable: the reason is that the document list is not the same for the two systems. Therefore, the size of a preference matrix is Nt Nt, where Nt is the number of unique documents retrieved by the two systems (i.e. the number of documents that appear at least in one of the lists returned by the systems) for topic t.
The systems ranks the document with weights that are not in the same range. Therefore, the output weights w1; w2; : : : ; wn of each expert k are transformed to fuzzy con dence values by means of the following transformation: rikj =
wi wi + wj
This transformation ensure that the preference values are in the range [0; 1]. In order to adapt the fuzzy Borda count to the merging of the results of IR systems, we had to determine the preference values in all the cases where one of the systems does not retrieve a document that has been retrieved by the other one. We decided to set the preference values of these documents to 0:5. This corresponds to the idea that the expert is presented an option on which it cannot express a preference.
We submitted a total of 9 runs. In Table 1 we show the detail of each run in terms of the two systems combined and the topic elds used.
In Table 3 we show the Mean Average Precision (MAP) obtained for each run, together with the MAP obtained by its composing runs.
The obtained results show that the use of the fuzzy Borda merging method always allows to improve the results of the best system. The improvement is greater if the two systems have a similar performance (see TMESS06) and the UPV system does not use map ltering. This behaviour is not observed when the UPV system uses map ltering. We suppose that a key feature for obtaining greater improvements by means of fuzzy Borda is that the systems share as few as characteristics as possible. 6
We combined two di erent systems by means of the fuzzy Borda voting scheme. The implemented method allowed to improve the results of the combined systems, although the improvement was limited. We suppose that the best results with the fuzzy Borda merging can be obtained if the two systems share the same level of accuracy. Further work will be aimed to verify this hypothesis and to the integration of more than two systems.
We would like to thank the TIMOM (TIN2006-15265-C06-03) and TIN2006-15265-C06-04 research projects for partially supporting this work.