This report presents the outcomes of the activity carried out at FBK-irst for the participation in the CLEF-2007 Main QA track. Both the major improvements over last year's version of the DIOGENE system, and the results achieved in the evaluation exercise are reported. The main novelties in this year's setting of the Main QA Task at CLEF are represented by: Introduction of topic-related questions. Questions, possibly referring to each other through anaphoric expressions, are organized into clusters related to a speci c topic. Extended answer search space. Besides the past years document collection, Wikipedia articles were added as a possible answer source.
This year the dataset provided by the organizers included a dump of the Wikipedia articles. The resulting new dataset posed new problems that had to be addressed, including:
Wikipedia articles contain di erent types of texts: information about a certain topic, formulas, lists, tables etc. We considered as a processable unit any text paragraph inside an article, apart form the Wikipedia links. Thus, we didn't process any other information that is contained in the other parts of the Wikipedia articles. For each processable unit we cleaned the text, using regular expressions, to remove the following text formatting information:
As a result, the clean processable units were considered as potential answer sources. The open
source search engine Lucene [
We decided to integrate the Wikipedia document index inside the document retrieval component of DIOGENE. The system uses a document retrieval technique based on query relaxation loops [3]. Such technique is designed to output a limited set of ranked documents (at least 30, at most 100). The Wikipedia document collection, however, is only considered as an auxiliary information source due to the noisy documents it contains. Often, in fact, our rst implementation of the cleaning procedure does not return fully reliable processable units. This is due to the large amount of unremoved tags, special symbols, or other XML annotations. As a result, Wikipedia documents are considered as a less reliable information source and are accessed only if an insu cient number of articles (less than 30) is returned by the MG search engine accessing the news document collection. 3
Dealing with Topic-Related Questions The other new problem that we had to address was handling a set of questions which share the same f ocus. To handle this problem the f ocus of the rst question has be recognized. For this purpose, we adopt the following simple heuristic, which de nes the f ocus of a question as the rst noun phrase or multi-word after the main verb of the question, if it is capitalized, or the second if the rst one is in lower case.
Examples of the focus identi ed for some CLEF-2007 questions are the following:
(In what year Flashdance came on the screen? ) Focus { Flashdance
(What are the Great African Lakes? )
Focus { Grandi Laghi africani 3. Question { Chi e l'autore del libro \Giorni giapponesi"? (Who wrote the book \Giorni giapponesi"? )
Focus { libro \Giorni giapponesi"
Once the f ocus of the input question Q1 is identi ed, it is added as a keyword (or a conjunction of keywords) to the search queries of the following questions Q2, ..., Qn in the cluster, unless it is already present among their terms.
Apart from these slight modi cations to the system's architecture, our submission to this year's
edition of the CLEF QA task (results are reported in Table 1) has been obtained with the same
system's components described in our previous participation in CLEF [
task Italian/Italian
Overall (%) 11.50
Def. (%) 2.36
List (%) 0.00
Factoid (%) 15.17
Temp. (%) 12.50
A preliminary analysis of the results achieved focused on the impact of the adaptations of the system to this year's task.
As for wikipedia articles, potential answer candidates have been extracted from such additional resource only for 9 questions (for a total of 38 candidates). Out of them, the nal answer returned by DIOGENE came from Wikipedia in 6 cases, but only in one case it was the correct one (i.e. Q-0134: \Quanto dista Dunleary da Dublino" - \How far is it from Dunleary to Dublin ").
As for topic-related questions, our focus extraction heuristic has been applied for 67 questions. The focus has been correctly added to the search keywords of a question in 42 cases, leading to 5 questions correctly answered. In 1 case it is not clear what the focus actually is, making a decision about its correctness rather di cult. This is:
(What is the capital of Rhode Island? ) Q-0114 { Dove si trova? (Where is it located? ) 5
In this report we presented our adaptations of the FBK-irst DIOGENE QA system, made to
participate in the CLEF-2007 Main QA track. Such improvements addressed the problems posed
by the two novelties of this year's edition of the task, namely the introduction of Wikipedia articles
to extend the document collection, and the introduction of topic-related questions. The results
achieved by the system show that our basic procedures dealing with such problems need to be
re ned. In particular, as a rst step, the cleaning procedure designed to extract reliable processable
units from Wikipedia articles will be improved, allowing for a more e ective exploitation of such
resource. As for topic-related questions, future improvements will address the focus selection
strategy, either with re ned heuristics, or with supervised approaches as proposed in [