This paper reports on the GikiP pilot that took place in 2008 in GeoCLEF. After providing motivation from both organizers and participants, it presents the task description, detailing topic choice and evaluation measures. Results are reported together with assessment difficulties and issues. Each participant system is described in detail, and the paper concludes with remarks on the current venue as well as ideas for improvements for future editions of GikiP or similar evaluation contests.
This paper introduces GikiP1, an evaluation contest on retrieving information from Wikipedia [30] in the
form of a list of answers (corresponding to articles) that have some geographical component. We start by
reporting different kinds of motivation that led us to propose, organize or participate in GikiP:
the first is the often voiced dissatisfaction of GeoCLEF participants (and people involved in GIR in
general) with continuing querying a collection of old newspapers – this has been a constant in
GeoCLEF [
Our view is that traditional evaluation tasks tend to create artificial divides where ultimately what is at stake is the satisfaction of user needs concerning access to information and knowledge through automatic means (or automatic helpers). It is not the form of the question or of the answer that should ultimately define the limits of what information access can offer.
Clearly, CLEF [
However, the need to capitalize on the evaluation setup already created, as well as the upsurge of scientific communities around a particular task, may, in the long run, cause separation of otherwise similar concerns and interests, as is, in our opinion, what happens with QA@CLEF and GeoCLEF (or, for the sake of the argument, also WebCLEF and Ad-Hoc CLEF).
GikiP is an attempt to bring back together the two communities/tasks, by merging – albeit in a very specific context – the two forms of information request (questions or topics) and the two kinds of expected answers (factoids or documents), which are typically the hallmark of QA and IR, respectively.
Another concern of GikiP is to encourage multilingual and cross-lingual processing. In fact, in CLEF this has – understandably – not been a priority for participants. Even though the organizers take pains to treat all languages equally well, one might call CLEF (or some of CLEF tracks, at least) a set of monolingual evaluation campaigns in disguise. This is not a criticism. Rather, it reflects an important reality: In most research groups, most if not all resources are devoted to the processing of one’s language (as it should be). Nevertheless, GikiP provides a task where it might be comparatively easy to satisfy other languages’ needs (and/or make use of other languages), given that it has as target one of the most genuinely multilingual resources available: Wikipedia.
Let us acknowledge that CLEF has already witnessed an interesting pilot with Wikipedia: WiQA
2006 [
On the contrary, GikiP has a very straightforward and understood task (answer open list questions),
and a very broad intended user community: any user/reader of Wikipedia might be interested in asking
questions to it and get a list of articles. Systems that can perform successfully in this task may help
harnessing the wealth of information that is included in Wikipedia. (And in fact, GikiP could be just the
beginning of a contest where one would also look for images [
But let us go back to the 2008 pilot in the remainder of this paper. We first describe the task in Section 2, with Section 3 devoted to topic choice, translation and assessment. Section 4 describes actual participation and results, while Sections 5 to 7 present each participant’s approach in detail. Section 8 sums up and concludes. 2
We defined the particular specific task to be solved by this year’s participants:
Find Wikipedia entries (i.e. articles) that answer a particular information need which requires geographical reasoning of some sort.
In order to guarantee a common evaluation ground, participants were required to use the Wikipedia collection(s) already used in the QA@CLEF main track2, henceforth called the GikiP 2008 collections.
Fifteen topics were made available on the 2th June 2008 from the GikiP site, where eight example topics in the three languages had already been published. Below is topic GP4 in the English version: Participants had 10 days to return the results, in the form of a list of Wikipedia articles, by providing their results as a list including the title of Wikipedia web pages. A (reduced) example of results for the GP4 topic is: 2.1
Only answers / documents of the correct type were requested (and therefore assessed as correct). In other words, if a topic concerned painters or scientists, results should be (lists of) names of people (painters and scientists), and not names of boats or countries. Conversely, if the question was about countries, the type of results should be places of the country type, and not wars or kings.
The maximum number of documents returned per topic was set to 100, but systems were strongly encouraged to try to return only the right ones (which, as the organization was at pains to emphasize, were typically much less than that number). Initially, we had decided on a maximum of two runs per system, but 2This was also a way to diminish the threshold for QA participants, who already had to process those collections anyway. However, in the end it apparently scared away other participants who had other Wikipedia versions at their disposal, and produced some problems for the organisers as well since some runs did provide out-of-collection answers.
Which waterfalls are used in the film “The Last of the Mohicans”? Which Vienna circle members or visitors were born outside the Austria-Hungarian empire or Germany? Portuguese rivers that flow through cities with more than 150,000 inhabitants Which Swiss cantons border Germany? Name all wars that occurred on Greek soil.
Which Australian mountains are higher than 2000 m? African capitals with a population of two million inhabitants or more Suspension bridges in Brazil Composers of Renaissance music born in Germany Polynesian islands with more than 5,000 inhabitants Which plays of Shakespeare take place in an Italian setting? Places where Goethe lived Which navigable rivers in Afghanistan are longer than 1000 km? Brazilian architects who designed buildings in Europe
French bridges which were in construction between 1980 and 1990 later on we accepted up to six runs from the one group who suggested this alternative. And we believe that this, coupled with a restriction like “as long as the total number of different topic-document pairs does not exceed 1500 (i.e. the number of assessments needed for 1 dumb run that delivers 100 documents for each of the 15 topics)”, would be a better task definition for further editions. 2.2
Evaluation was devised in order to emphasize diversity and multilinguality, ensuring that the systems which were able to retrieve most cases and in most languages would be considered best by the GikiP scores.
We delivered (and assessed) topics in English, German, and Portuguese, and therefore, an additional bonus was computed for multilinguality, mult, which is 1, 2 or 3 depending on the number of languages tried out by the systems.
System results are thus evaluated according to the following formula mult N N total where mult rewards multilinguality, N is the number of correct hits, and N total is precision. The system’s final score is given by the average of the individual scores. 3
Considerable care was put in obtaining a balance between topics more naturally covered in each of the three GikiP languages, as well as providing also other topics equally alien, in principle, to any of the languages or cultures (such as those on French bridges, Afghan rivers or Polynesian islands). This is not necessarily a true description of what is or can be found in the particular Wikipedia language versions available, but this was a guideline for topic choice.
Although, by definition, a geographic flavour had to be associated, we also strived to create topics quite
different from those used in the GeoCLEF main task – in order to cover higher levels of the typology
described in [
All topics were initially devised in Portuguese, and then roughly translated into English and from there
to German. Then a thorough revision of the three versions took place. Although it was not intended, it
appeared that some topics in Portuguese were hard to translate into the two Germanic languages, so we
note this here, as a counterpart to the usual difficulty to translate some English topics into Portuguese
(reported for example in [
Another issue was the difficulty of expressing something that overlapped temporally with a particular decade in English – bringing the awkward formulation “whose construction started, continued or ended in or between 1980 and 1990”. Likewise, the expression of “the place of the plot” in German was not direct, either, and in fact “Welche Stu¨ cke Shakespeares spielen in Italien?” can also indicate that a theater company is doing a Shakespeare play in some part of Italy.
Of course we are not saying that either of these questions is impossible or even too difficult to express in German or English. We are just pointing out that they are probably more naturally come upon and formulated by Portuguese speakers. This is the sort of information that is interesting to amass in a crosslingual context: some questions are easier to answer (and more natural to pose) in different languages. 3.3
In order to provide an idea of the assessment work, we describe briefly the pool obtained, by listing, for each topic, the number of different answers from all participants, as well as number of the correct answers, in Table 2.4
As far as assessment matters are concerned, as usual some decisions had to be made: As a general rule, we considered as wrong answers all cases where the human assessor was not able, to the best of her ability, to verify the truth of the answer. For example, in topic GP14 (Brazilian architects who designed buildings in Europe), several Brazilian architects where no mention to works in Europe could be found were deemed 3And possibly also in German. Here, opinions diverge.
4Note that the answers themselves may not necessarily be different, they just need to correspond to a different Wikipedia article. incorrect, because we expect that non-verifiable information is not useful for any user. If the user wanted to assess himself based on the information that only he had, then he would have asked for Brazilian architects only.
Of course, other issues were more difficult to assess. For example, does the mention that a particular river was strategic for military operations imply that it is navigable? – we assumed a negative answer in topic GP13 (Which navigable rivers in Afghanistan are longer than 1000 km?).
We acknowledge that our role as human assessors may not guarantee perfect knowledge, but this is in no way different from any other evaluation contests which involve human judgements.
Rather more interesting was the problem of different language versions having different answers (for example as far as population of the Tahaa island, for GP10 (Polynesian islands . . . ) is concerned). Here, we assumed a very liberal procedure. If a positive answer could be found in any version, all of them were deemed correct.
Also, some answers to GP7 (African capitals . . . ) referred to places that no longer exist or have changed name or status (such as Salisbu´ ria or Abidjan). We considered them correct since no temporal restriction was explicitly mentioned in the topic, but this is obviously an issue that has to be better dealt with in real-life information access.
In fact, the temporal interaction is even more complex: if we take a closer look at the GP7 topic, three ways to interpret it arise: (i) a user might be interested in African capitals with a given population at any time (this is the broadest possible interpretation) or (ii) only with such population when they were capitals (even if they are no longer capitals), or even (iii) only cities with such population, even though no longer capitals (provided they had been once).
Yet another interesting issue also raised in connection with the GP7 topic is the status of capital itself: in some countries, it is distributed among different cities, as is the case of South Africa, with three capitals (administrative: Pretoria, legislative: Cape Town and judicial: Bloemfontein). Any of these cities was considered a correct hit if it also satisfied the other topic requirement(s).
During assessment, we also found interesting language differences concerning cognate proper names:
While Salisbu´ ria in Portuguese refers unambiguously to a previous (temporal phase of a current) African
capital, now named Harare, the “corresponding” Salisbury entry in the English Wikipedia points to a place
in the United Kingdom, more specifically in the English county of Wiltshire. (This reminds us of the issues
with last year’s GeoCLEF topics on St. Paul’s Cathedral and St. Andrews, whose “translation” into Sa˜ o
Paulo and Santo Andre´ brought up different and more prominent places in Brazil and Portugal as well [
At another level, it was not always possible to maintain the level of detail as far as specific objects or concepts were concerned: for example, ra´ pidos in Portuguese is a subtype of waterfall (namely one which is navigable), which apparently has to be rendered in English as waterfall, which is undeniably much more generic.
Finally, some features of Wikipedia itself caused unexpected problems for assessment: While a question for a city population seems to be natural from a user’s point of view, we found that for some cities there are up to three different numbers (city, urban, metropolitan)! This implies that interaction with the user would be needed to identify which of these concepts s/he had in mind. For the time being, we accepted answers as correct provided one of the numbers satisfied the restriction stated in the topic (two topics concerned city populations: GP3 and GP7). 4
As usual, a pilot task gets more expressions of interest than actual participations, and this happened in GikiP, with 13 groups reporting interest but only three participants in the end. One probably not irrelevant consideration is that GikiP is a hybrid task between QA and GIR, but was fully deployed – and therefore only conveniently publicized – under the GeoCLEF umbrella.5 These are the participating systems: 5This should by no means be read as a critic of QA@CLEF organizers. On the contrary, they gave us at once access to their data and considered the task interesting. It just happened that the task was located at GeoCLEF because we were involved in GeoCLEF organization and not in QA@CLEF at the time.
GIRSA-WP RENOIR WikipediaListQA@wlv Human WikipediaListQA@wlv, represented by Iustin Dornescu, Research Group in Computational Linguistics (CLG) at the University of Wolverhampton (UK) Curiously, we had one participant per country where one of the three languages is spoken, and participation was divided equally between GeoCLEFers and QA@CLEFers (given that the IICS group is known to participate in both). We have also kept non-official submission to GikiP open until later (30 June) in order for people busy with the GeoCLEF main task to be able to try GikiP, although non-officially, but we received no further submissions.
The participating systems are shortly described in Table 3, together with a fully manual run based on the current Wikipedia, which was provided by Paulo Rocha. The point of requesting this run was to be able to, later on, compare human performance to automatic answers. Also, we wanted to assess how much Wikipedia information had changed regarding the particular topics, from the official collections to the June 2008 date. Each system will be fully described in the corresponding section in the paper.
The results obtained by the systems can be found in Table 4. For the record, the maximum number of documents returned per topic was 23 (for GP7), and the minimum was zero. It is also interesting to note that the human participant was not able to find any results for topics GP2 (Which Vienna circle members . . . ) and GP15 (French bridges . . . ), contrarily the automatic systems, which together managed to find 7 and 5 correct hits, respectively. 5
GIRSA-WP (GIRSA for Wikipedia) is a fully-automatic, hybrid system combining methods from question
answering (QA) and geographic information retrieval (GIR). In particular, it merges results from InSicht,
an open-domain QA system [9], and GIRSA, a system for textual GIR [
In comparison with the two underlying basic systems, GIRSA-WP applies a semantic filter on the article titles (which are encoded in the answers in GikiP) to increase precision. This semantic filter ensures that the expected answer type (EAT) of the topic and the title of a Wikipedia article are compatible. This technique is widely known from QA for typical answer types such as PERSON, ORGANIZATION, or LOCATION. In our system, a concept (a disambiguated word) corresponding to the EAT is extracted from the topic title or description. Then, this concept and the title of a candidate article are parsed by WOCADI [8], a syntactico-semantic parser for German text. The semantic representations (more specifically, the ontological sort and the semantic features, see [12] for details) of the semantic heads are unified. If this unification succeeds, the candidate article is kept; otherwise it is discarded. For example, from topic GP4 (Which Swiss cantons border Germany?), the extracted concept is canton, which is an artificial geographical entity denoting a kind of regional institution.
The major differences to InSicht and GIRSA are that GIRSA-WP does not merge streams of answers and does not include a logical answer validation. In contrast to GIRSA, the retrieval is based on documents indexed on a per-sentence basis of Wikipedia articles. In addition, the documents from Wikipedia had not been geographically annotated at all.
For the GikiP experiments, the topic title and description were analyzed and sent to GIRSA and InSicht. In GIRSA, the top 1000 results were retrieved and scores were normalized in the interval from 0 to 1. For results returned by both GIRSA and InSicht, the maximum score was chosen. Results whose score was below a given threshold were discarded and the semantic filter was applied to the remaining results. To obtain multilingual results, the German article names were translated to English and Portuguese using the Wikipedia linking between languages. Note that this linking was the only non-textual information we used from Wikipedia; for example, categories and inter-article links were completely ignored.
In InSicht, the semantic representation of the query and the semantic representations of document
sentences are compared. To go beyond perfect matches, InSicht uses many techniques, for example intratextual
coreference resolution, query expansion by inference rules and lexicosemantic relations, and splitting the
query semantic network at certain semantic relations. InSicht employed a special technique called query
decomposition (first tried in GeoCLEF-2007, [
Due to time constraints, the Wikipedia articles had not been fully processed and some methods have been applied to the topics only although they should have been applied to the documents, too. We performed six runs with the following experiment settings: run 1: topics and documents (Wikipedia sentences) are processed with full word forms (no stemming and no stopword removal); results are filtered by applying a threshold score of 0.01 run 2: same setting as in run 1; location names are identified and normalized (for topics only) run 3: same setting as in run 2; German noun compounds are identified and split into their constituents (for topics only) run 4–6: same settings as in run 1–3; results are filtered by applying a threshold score of 0.03 5.3
Several reasons account for the relatively low performance of GIRSA-WP:
In comparison with topics from the QA@CLEF task at CLEF, GikiP topics are more difficult to answer and aim at a wider range of EATs.
In comparison with topics from the GeoCLEF task, topics are at least as difficult. They include complex geographic relations (GP2: outside, GP4: on the border), restrictions on measurable properties (GP3: more than, GP13: longer than), and temporal constraints (GP9: Renaissance, GP15: between 1980 and 1990).
Indexing sentences instead of complete Wikipedia articles was meant to ensure a high precision.
However, the fallback strategy (GIRSA) does not work well when applied on document sentences.
Geographic entities were not annotated at all in the documents for GIRSA. Thus, the high precision and recall observed with GIRSA for the news collection could not be observed with the Wikipedia articles.
For InSicht, the main problems were (1) that important information is given in tables (like inhabitant numbers), but the syntactico-semantic parser ignores these parts of articles and (2) that the semantic matching approach forming the basis of QA is still too strict for the IR oriented parts of GikiP queries (similar problems occurred for GeoCLEF experiments).
The system’s multilingual approach is too simple because it relies only on the Wikipedia of one
language (German) and adds results by following title translation links to other languages. Therefore
for questions that have no or few articles in German, relevant articles from English or Portuguese
cannot be found.
@ @= : , , , , , and . REMBRANDT participated
RENOIR is an interactive tool where query procedures are executed, generating partial and final results for
each GikiP topic. RENOIR makes extensive use of REMBRANDT [
REMBRANDT classifies NEs according to the following 9 main categories: , ,
on the second HAREM [
We indexed the GikiP 2008 collection with MG4J [
As such, we chose to participate with one semi-automatic run using query procedures as retrieval input, instead of query terms. We define query procedures as a group of pipelined actions that express each GikiP topic. The generation of query procedures was entirely manual, and the execution of each action varies from automatic, semi-automatic and manual. 5.4
Future work will include tackling some of the problems discussed in Section 5.3, enabling the annotation of geographic entities and geo-inferences, and preferring special regions of Wikipedia articles (for example, the introductory sentences). 6
6.1
What is RENOIR 2008 collection. As such, RENOIR also allowed us to perform basic actions to match the documents from the Wikipedia dumps to its corresponding GikiP 2008 documents, to cope with GikiP’s submission format.
The RENOIR actions used for the query procedures are described as follows. The actions are labelled as automatic (RENOIR performs the action alone), semi-automatic (the action is supervised), or manual (the action is made manually).
Automatic Automatic Automatic Automatic Automatic Automatic Manual
Performs a simple term query search in the GikiP 2008 collection, and returns a list of Wikipedia documents. Searches the Wikipedia dumps for documents with the given Wikipedia category, and returns a list of Wikipedia documents.
Automatic
Searches the Wikipedia dumps for documents that link to a given Wikipedia document. part in the GikiP 2008 collection.
2008 collection.
Semi-automatic
Maps a document from the Wikipedia dump to its counterSemi-automatic
Maps a NE to its corresponding document in the GikiP Annotates selected
Wikipedia document(s) with REMBRANDT, generating lists of NEs for each document. Invokes REMBRANDT to classify the title of a given Wikipedia document, generating the respective NE. erating a subset of NEs. given term/pattern Filters a list of NEs of a given classification category, genFilters a list of Wikipedia documents by having (or not) a Filters a list of Wikipedia document by evaluating a condition for a given subset of NEs. For instance, if the document has a number NE greater than 1000, or if it has a place name NE within Europe. 6.2
The query procedures were formulated in a simple modular and pipelined approach. This allowed us to “divide and conquer” the complex task of translating the GikiP topics into a machine-understandable way. So, the actions that could be made automatically were therefore implemented, while the more complex actions performed in GikiP with human intervention (so far) were also kept simple in order to be possible to extend RENOIR to perform them automatically in the future. Anyway, while RENOIR can not perform (yet) all actions automatically, the methodology devised to handle the GikiP task can already be tested, and we can get acquainted with the difficulties that will face us during further development of RENOIR. for Portuguese had the same actions, with small exceptions discussed later. The query procedures are best explained by following the example topic GP7 (Capitals of Africa . . . ), which has a query procedure pattern similar to other topics.
Docs1 - The expected answers for this topic is a filtered list of Wikipedia pages that are likely to belong to a same category, “Capitals of Africa”.
“last mohicans film”
Docs2, NE2; “Vienna Circle”
Docs2, NE2;
NEn represents lists of NEs. “Rivers of Portugal” NE2 *,- ’Austria’, ’Hungary’, ’Germany’./ NE2 contains ‘war’ “Mountains of Australia”
Docs 1; DoNcEs21;;& Docs 1;
NE1;
So, we first search for all Wikipedia pages that contain that category, and obtain the list of Wikipedia documents Docs1. This step is done automatically.
Docs1
Docs1
NE1 - Afterwards, REMBRANDT annotates the documents in Docs1, generating a tagged version of these same documents, Docs1, and their corresponding named entities, NE1. This step is done automatically.
NE1 - We narrow the NE list to the entities that were classified by REMBRANDT as VALUE/QUANTITY, generating the subset NE1. This step is done automatically.
Other over-simplifications involve, for instance, topic GP14 (Brazilian architects . . . ), where we assume that a Wikipedia page that contains at least one NE referring to a place in Europe is enough to consider that the targeted Brazilian architect page is relevant, or in topic GP11 (Shakespearean plays . . . ) where reference to a place in Italy is enough to consider that a given Shakespearean play actually happens in Italy. Last but not least, knowing whether the place names were Italian or European was manually done. Later on, we will try to access geographic ontologies in order to implement such restrictions, to fully verify the
Also worth mentioning is the bypass we made to the problem of selecting the correct categories for the first pool of Wikipedia documents, as done in the query procedures that started with the action, by doing it manually. For instance, in topic GP9 (German composers . . . ), it turned out that the best category to start with was “German composers”, while in topic GP15 (French bridges. . . ) the best category is “Bridges in France”. We tentatively concluded that the first form is typically used to cluster persons by searching for values greater than 2,000,000. When found, the respective document is therefore added to Docs2. This step is done manually.
Docs2 - We now evaluate the NE subset NE1, Docs2
Docs2 - Finally, since we started with a pool of Wikipedia documents from the SQL dumps because of the
action, we need to map the documents in Docs2 to their corresponding GikiP 2008 documents, generating Docs2, which is the final result. The URLs of Docs2 are then added to the submission file. The mapping step is done semi-automatically. Our approach for GikiP has some obvious faults and over-simplifications that are clearly compensated by lation count is easy for a human, but it is not an easy task for an automated system. Yet, one of the best features of Wikipedia (from an IE system developer’s point of view) is the clustering of such important features into infoboxes, which are easily machine-interpretable and thus this problem can be mitigated by feasibility of the approach.
NE1
NE1
Results and discussion country and occupation, and the second is used for non-person entities. developing a robust Wikipedia infobox parsing module.
Nonetheless, one of the requirements for an automated query procedure builder module is to figure out that “Composers of Germany” and “French bridges” are not Wikipedia categories, and thus a domain identification step as performed by WikipediaListQA@wlv (see Section 7) is quite important. Another problem arises for topics that could be started with a category, but there is no such category, as in topic GP5 (Name all wars . . . ). We expected a category such as “Wars of Greece” to exist, but it did not, so we had to choose a different query procedure that involved heavy processing: we selected Wikipedia pages with outlinks to Greece, resulting in an initial pool of over 20,000 documents linking to the Wikipedia page “War” in the title were finally selected.
Lastly, we report the differences between the Portuguese and English query procedures, which were related to the Wikipedia categories used in both snapshots. Take for instance the topic GP6 (Which Australian mountains . . . ). It is well bootstrapped in Portuguese by the category “Montanhas da Austra´lia”, but, in the English Wikipedia snapshot, the category “Mountains of Australia” has a bundle of Wikipedia pages pointing to mountains and also subcategories that group mountains in Australian regions, such as corresponds to a popuplus the term New South Wales, South Australia, Northern Territory or Tasmania. Clearly, the two topic formulations, although apparently similar, may refer in English to a whole continent while in Portuguese only to a country (the continent being named Oceaˆ nia).
On the other hand, in Wikipedia there are some categories which have subcategories before we arrive to the nodes that are real answers. To handle this we included new queries for the subcategories. For example, for topic GP11 (Shakespearean plays . . . ), the English Wikipedia category “Shakespearean plays” is divided into several subcategories that group the adaptations, comedies, tragedies, histories and apocrypha. So all these subcategories had to be visited. In this case, there was a difference between the English and the Portuguese Wikipedia since the latter had the subcategories but these were not yet filled, so the procedure had to be different. The GikiP experiments allowed us to take a first glance at the difficulties that expect us when dealing with more complex queries with specific geographic criteria. As our goal is to research retrieval approaches that can profit from a comprehensive semantic layer over queries and documents, we found the GikiP exercise to be interesting and innovative.
The next obvious step is to implement the automatic generator of query procedures, dealing with the problems that were mitigated by using human reasoning. At the same time, future work includes the improvement of the Wikipedia mining approaches, namely extracting information from infoboxes. 7 The participation in this pilot task was motivated by our interest in using Wikipedia as a backbone in QA. In addition, the way the task is setup required us to rely on the information inherent in the Wikipedia article link graph and the relation between entities, rather than developing accurate textual answer extractors.
For example, if we try to find out information about the cities that the Douro river flows through, we can extract all the links present in the article describing the Douro river, creating a list of entities that are related to the river. If we only select the articles from this list that describe a settlement, a town or a city, the filtered list would likely contain localities that the river is passing through. By examining the infobox of each candidate, we can determine the population of each locality. We may apply a selection function to this list. We can compute the average population size, number of towns with population larger than 150,000 inhabitants, etc. discovering new facts.
For example, in solving the GP3 topic, the system identifies the Wikipedia category Category:Rivers of Portugal. All the articles directly linked to this category are likely to be rivers, thus the list of candidate answers is created. As previously described, for each river the system extracts the list of links to articles and attempts to extract the population size from each one. The maximum value encountered is computed. If this value is smaller than the threshold, the river is discarded.
In order to navigate the Wikipedia link graph we had to transform the Wikipedia SQL dump and index it with Lucene [11] because loading the data in MySQL would have taken too long (more than 200 million links in English Wikipedia alone). The index already contained a cleaned version of Wikipedia (April 2008). Thus our results had to be mapped to the November 2006 version. 7.1
We propose a simple model for topic interpretation. It exploits relationships between entities that may not be expressed in the article text, but are implied by the links between the articles.
Our system starts by identifying a domain category that comprises candidate articles, and then filters
out the ones that do not correspond to the topic filter. Thus two parts are identified in each topic: a) the
domain of the candidate answers, and b) the filters to apply in order to select the correct ones.
Domain identification We had to identify a Wikipedia category that would describe the candidate
articles. We used the Connexor FDG parser [
For the purpose of this pilot only very simple filters were implemented. There are two main filter categories: entity filters and factoid filters.
The entity filters match documents that mention or have a link to a given entity. 8 of the 15 topics had such a constraint. In 2 cases there was a list of entities that should have matched. Most notably the temporal restriction identified in topic GP15 between 1980 and 1990 was expanded to the list of 10 years in the range. This behaviour was adapted from our QA system.
The factoid filters match documents in which the identified fact can be extracted, and the value corresponds to the selection criterion. The facts were extracted using components from our question answering system (infobox look-up, regex patterns). The facts were: population (3 topics), nationality (2 topics), height (1 topic) and length (1 topic). Articles from which the fact could not be extracted were dismissed. The selection criterion was applied to the extracted fact: greaterThan (5 topics - numeric facts), inList (1 topic), not inList (1 topic).
The method proposed has the advantage that it can easily be adapted to a cross-lingual task. Only the English topics were analyzed. The results of the analysis can be directly mapped to any language version of Wikipedia. Firstly the corresponding category describing the domain of the candidate articles has to be identified by using inter-wiki links. Secondly the filter has to be “translated”. This either means translating the entities, or having the necessary language dependent fact extractors. Cross-wiki answers can be combined and re-ranked, aggregating the results from all the languages by using the inter-wiki links. This allows an English-speaking user, for example, to exploit the fact that the Portuguese Wikipedia has much richer content regarding Brazil and Portugal since most Portuguese Wikipedia contributors live in these two countries. This year we only searched the English Wikipedia and the results were mapped to all the three languages by using the inter-wiki links because we did not have time to create the fact extractors 2 der to achieve a good accuracy (e.g. for German and Portuguese. 7.2
The accuracy of the system is limited due to the ambiguity of links. Not all articles that pertain to
Category:Abidjan refer to an African capital. Category relations were not classified: hypernymy vs. meronymy
vs. similarity. When searching all the articles that have a certain hyper-category, due to link type ambiguity,
very large article sets might be extracted (the system did not return any results for three topics, because the
list of domain articles was too large). This can be avoided by using resources that map Wikipedia articles
to WordNet (e.g. Yago [
The current system is a simple model which has proven to have a good precision of finding the answer. Its main advantage is that – using a small set of filters – very complex data can be queried from Wikipedia. Its greatest disadvantage lies in the complexity of correctly identifying (combined) filters in natural language questions. Given an appropriate user interface, this method can become an alternative to Wikipedia search, allowing users to access information that is not textually present in the encyclopedia. 8
We think that the results presented in this pilot are encouraging, both for the possibility of automating the particular task, and for its general interest as another way of reaching the information in Wikipedia. GikiP has shown that there are interesting kinds of non-trivial questions that have a retrievable answer in Wikipedia, and which can be quickly assessed by users, often without even having to visit the page.
Interestingly, participating systems took a wide range of different approaches, as well as different main collections (GIRSA-WP used German and WikipediaListQA2wlv English as main answering sources; while RENOIR did a parallel process for English and Portuguese). We intend, in a following publication, to study results per language and per topic to see whether results might have been heavily influenced by this choice of main collection to investigate. While one participant took a preliminary semi-automatic approach, although making use of several automated procedures, the two others (including the winner) used fully automatic systems from start to end. Much work remains to be done for issues of redundancy removal, choice of which language / answer to present first (or only), as well as how to present a compound set of pages to justify a particular answer.
Also, the task needs a more reflected and precise definition.
For example, the fact that natural language is fundamentally vague may lead to a set of answers at different levels, which, in order to satisfy a human user, should be presented in a more appropriate way: Consider the case of topic GP12 (Places where Goethe lived), where also names of countries were considered correct. Obviously, a more adequate answer would structure or make use of the hierarchical relations between the answers and not present all of them alphabetically as a flat list.
Most information needs require a lot of conceptualization. This is illustrated by topic GP5 (Name all wars that occurred on Greek soil). After starting assessment, we found that the relation of battles or sieges (which occur in particular places) as belonging to wars may require considerable theorizing and historical knowledge. Wars, especially perhaps when one is considering ancient wars, are more difficult to pin down to particular places. So, assessment turned out to be quite tricky, and we fear that the topic did not illustrate a realistic user need.
Finally, the mult factor, which was assigned 2 to RENOIR and 3 to the other two systems, was a fairly gross attempt to incentivize results in the three languages. However, different topics, as discussed in Section 3.1, would be easier or more productive in different languages and in fact answers which could only be found in one vs. two or three languages might be considered more difficult and therefore rewarded with another kind of multilinguality-related factor. This is something that must be investigated in the future. Namely, how to increase the score of particularly difficult topics. We hope that further editions of GikiP and similar tasks will help research and development along those lines.
Acknowledgements We are grateful to Ross Purves for checking the English rendering, to Sven Hartrumpf for analysing and debugging the German and English versions with the WOCADI parser, to Anselmo Pen˜as for making the Wikipedia collections available to GikiP participants, and most particularly to Paulo Rocha for providing a fully manual run in two days.
The organization work was done in the scope of the Linguateca project, jointly funded by the Portuguese Government and the European Union (FEDER and FSE) under contract ref. POSC/339/1.3/C/NAC.
The development of the WikipediaListQA@wlv system was partly supported by the EU funded project QALL-ME (FP6 IST-033860).
The development of RENOIR was supported by grant SFRH/BD/29817/2006 from FCT (Portugal).
[10] Sven Hartrumpf. Semantic decomposition for question answering. In Malik Ghallab, Constantine D.
Spyropoulos, Nikos Fakotakis, and Nikos Avouris, editors, Proceedings of the 18th European Conference on Artificial Intelligence (ECAI), pages 313–317, Patras, Greece, July 2008. [11] Erik Hatcher and Otis Gospodnetic. Lucene in Action (In Action series). Manning, Greenwich, CT,
USA, 2004. [6] Fredric Gey, Ray Larson, Mark Sanderson, Kerstin Bishoff, Thomas Mandl, Christa Womser-Hacker, Diana Santos, Paulo Rocha, Giorgio Di Nunzio, and Nicola Ferro. GeoCLEF 2006: the CLEF 2006 Cross-Language Geographic Information Retrieval Track Overview. In Carol Peters, Paul Clough, Fredric C. Gey, Jussi Karlgren, Bernardo Magnini, Douglas W. Oard, Maarten de Rijke, and Maximilian Stempfhuber, editors, Evaluation of Multilingual and Multi-modal Information Retrieval: 7th Workshop of the Cross-Language Evaluation Forum, CLEF 2006. Revised selected papers, volume 4730 of LNCS, pages 852–876. Springer, Berlin, 2007. [9] Sven Hartrumpf. Question answering using sentence parsing and semantic network matching. In Carol Peters, Paul Clough, Julio Gonzalo, Gareth J. F. Jones, Michael Kluck, and Bernardo Magnini, editors, Multilingual Information Access for Text, Speech and Images: 5th Workshop of the CrossLanguage Evaluation Forum, CLEF 2004, volume 3491 of LNCS, pages 512–521. Springer, Berlin, 2005. [7] Danilo Giampiccolo, Pamela Forner, Anselmo Pen˜as, Christelle Ayache, Dan Cristea, Valentin Jijkoun, Petya Osenova, Paulo Rocha, Bogdan Sacaleanu, and Richard Sutcliffe. Overview of the CLEF 2007 Multilingual Question Answering Track. In Alessandro Nardi and Carol Peters, editors, Working Notes for the CLEF 2007 Workshop, 19–21 September 2007. [8] Sven Hartrumpf. Hybrid Disambiguation in Natural Language Analysis. Der Andere Verlag, Osnabru¨ck, Germany, 2003. [12] Hermann Helbig. Knowledge Representation and the Semantics of Natural Language. Springer,
Berlin, 2006. [15] Valentin Jijkoun and Maarten de Rijke. WiQA: Evaluating Multi-lingual Focused Access to Wikipedia. In The First International Workshop on Evaluating Information Access (EVIA), Tokyo, Japan, May 15 2007.