We present the second version of the Prodicos query answering system which was developed by the TALN team from the LINA institute. We have participated to the monolingual evaluation task dedicated to the French language. We firstly present the question analysis step which makes it possible to extract many features from the questions (question category, question type, question focus, answer type, ...). For this new campaign, new features are extracted from the questions in order to improve the passage selection process (named entities, noun phrases and dates). We also determine four different strategies that will be used during the answer extraction step (entity named strategy, numerical entity strategy, acronym definition strategy, pattern-based strategy). We also take into account a new category of question (lists). We then present the passage selection process whose goal is to extract from the journalistic corpora the most relevant passages which answer to the question. This year, we present a new strategy applied to definitional queries. We use external knowledge (Wikipedia encyclopedia) to add information to these kinds of questions. Then, we discuss, in details, the major improvements made on our system at the answer extraction module level. According to the strategies determined during the question analysis stage, we present the 4 different strategies applied to this step. We present, in details and independently, each strategy and their use context. Afterwards, for the passage selection and answer extraction modules, the evaluation is put forward to justify the results obtained.
In this paper, we present the second version of the Prodicos query answering system which was developed by the TALN team from the LINA institute. It was our second participation to the QA@CLEF evaluation campaign. We have decided to participate to the monolingual evaluation task dedicated to the French language. This campaign enables us to analyse the performances of our system. Firstly, we present the various modules constituting our system and for two of them (passage extraction module and answer extraction module) , the evaluation is put forward to justify the results obtained. 2
The Prodicos query answering system is divided into three parts (figure 1): • question analysis module; • passage extraction module (extracts passages which might contain the answer); • answer extraction module (extracts the answer according to the results provided by the previous module).
The modules of the Prodicos system are based on the use of linguistic knowledge, in particular
lexical knowledge coming from the EuroWordnet thesaurus [
The system has participated to the QA@CLEF 2006 evaluation campaign for the monolingual
query answering task dedicated to the French language. This campaign enables us to make an
evaluation of the system. We present, in the next sections, in greater detail, the various modules
which belong to the Prodicos system and the linguistic tools used to implement them. In parallel,
we analyse in detail the results for the passage extraction module and the answer extraction
module. The question analysis module was been evaluated last year during the QA@CLEF 2005
campaign [
The question analysis module aims to extract relevant features from questions that will make it
possible to guide the passage selection and the answer search. We extract many features from the
questions [
The question category is determined according to specific syntactic rules. The main feature
which comes from the question analysis is then the question type. It will not only help to
determine the strategy to perform an answer search but also it will make it possible to select rules to
extract other important features from questions (answer type, question focus). We defined twenty
question types which correspond to a simplified syntactic form of the question 1 (for example the
type QuiVerbeGN). The question type makes also it possible to verify the answer type that will
be retrieved. The answer type may be a named entity (Person, Location-State, Location-City,
Organization...), or a numerical entity (Date, Length, Weight, Financial-Amount...). The
question focus corresponds to a word or a word group involved in the question. Its main particularity
is that, generally around it, the answer is present within the passages which may contain the
answer. These different features are extracted by using the TreeTagger tool and then,
according to the part-of-speech tags, by building some rules to determine the question chunks (noun
phrase, adjective phrase, adverb phrase, prepositional phrase, verb phrase). Then, according to
the previous syntactic chunks, we have written rules which make it possible to extract, from the
questions, information like question focus, principal verb,... For determining answer type, we use
semantic knowledge (EuroWordnet Thesaurus). We build lists of words which are hyponyms of
some predefined words which are considered like categories and we use them in order to generate
the answer type [
For this new campaign new features are extracted from the questions in order to improve the
passage selection process (named entities, noun phrases and dates). For example, for the queries:
“Qui est Boris Becker ?” (the 90th question: “who is Boris Becker?”) and ”Qu’est-ce que l’effet de
serre ?” (189th question: “what is the greenhouse effect?”), we now consider “Boris Becker” and
“effet de serre” as a single entity. We also determine a new feature which is the strategy to use to
search the right answer. It is determined according to the question focus and the question type.
These strategies are either an entity named strategy, either a numerical entity strategy, either an
acronym definition strategy or a pattern-based strategy (Figure 2). For example, if we take into
account the first case, this means that the answer extraction module must use a named entity
recognizer in order to extract the answer ...
• one answer type. For example “Qui est Boris Becker ?” ("Who is Boris Becker?"),
• precise number answer type (often extracted from the noun phrase corresponding to the
question focus). For example, for the question: “Qui sont les deux principaux responsables
1excepted for definitional questions [
First passage
coef=1 & not first passage coef < 1
Not present in any passage Not in Wikipedia
Wrong tagged Not present in corpus
Other cases
Total them. Indeed, for French language, some words constituting these named entities are ambiguous and represent either an adjective or a common noun. In table 1 , we can see that for 67% of the definitional questions, the passages containing the right answer take the value 1. Moreover, for 83% of them the answer belongs to a passage selected during this step. This are good results and this shows that the use of encyclopedic knowledge helps the selection passage process. The nature of the resource (Wikipedia) is also very interesting because of the recurrent problem for French language to have such kind of resource at one’s disposal. The multilingual property can also be used in a cross-language evaluation context. 5 5.1
This step comes at the end of our process. After the question analysis and the passage selection, we have to extract correct answers corresponding to questions. To this end, we use on the one hand elements coming from the question analysis like, for instance, the question’s category, the strategy to use it, the number of answers, and so on (see figure 2 for an example of a part of such an analysis, element shown are used in this step) and, on the other hand, a list of passages selected and evaluated by our previous step according to this question.
The goal of this step is to find the precise answer(s) to a question. An answer is built with the answer itself, the passage used to answer and, a trust value. This ending process can be divided into 4 local steps (figure 3): 1. according to the question’s strategy, the convenient entity extraction module is selected, 2. candidate answers are detected and selected by the previous selected module, 3. answers are evaluated and the answer(s) with the highest trust coefficient is (are) kept, 4. passages where each answer has been found are also associated to the selected answer.
The question’s analysis can give 4 groups of categories which correspond to 4 possible strategies: numerical entities extraction, named entities extraction, acronym definitions extraction and pattern-based extraction (the default one). Now, we will present processes associated to each strategy and the build of final answers. 5.2
For locating numerical entities, we use a set of dedicated regular expressions. These expressions
make it possible to the system to extract numerical information namely: dates, duration, times,
periods, ages, financial amounts, lengths, weights, numbers and ratios. It uses the MUC (Message
Understanding Conference) categories ("TIMEX" and "NUMEX") to annotate texts. For example,
lets take the 13th question: «En quelle année la catastrophe de Tchernobyl a -t-elle eu lieu ?» (the
year of the Tchernobyl’s nuclear explosion). Our numerical extraction tool gives results as for the
7th sentence of "LEMONDE95-041936" shown in figure 4.
For locating named entities, NEMESIS tool [
For acronym’s definition search, we use a tool developed by E. Morin [
3In this example "SIGLE" describes an acronym and "DEF" (with the same identifier "no") its definition For the pattern-based answer extraction process, we developed our own tool. According to question categories, syntactic patterns were defined in order to extract answer(s) (see figure 7 for an example of a pattern set associated with the question category called "Definition"). These patterns are based on the question focus and makes it possible to the system to extract the answer. Patterns are sorted according to their priority, ie answers extracted by a pattern with an higher priority are considered as better answers than the ones extracted by patterns with a lower priority. <regle> <Categorie> Definition </Categorie> <patron> GNRep GNFocus </patron> <patron> GPRep GNFocus </patron> <patron> GNFocus GNRep </patron> </regle>
As a result, for a given question, patterns associated with the question category are applied to all selected passages. Thus, we obtain a set of candidate answers for this question. For example, lets take the 2nd question: «Qu’est ce que Hubble ?» ("What is Hubble?"). This question corresponds to the category "Definition" (see figure 7). Pattern-based extraction process gives a set of candidate answers as the one presented in figure 8.
Patterns (syntactic patterns) are based on the noun phrase that contains the focus of the question. Therefore, the first step consists in selecting only passages which could contain the answer and which contain the focus of the question. To apply syntactic patterns, passages are parsed and divided into basic phrases such as noun phrase (GN), adjectival phrase (GA), adverbial phrase (GR), verb phrase (NV), etc. We use a parser which is based on TreeTagger tool for annotating text with part-of-speech and lemma information. Subsequently, passages are studied to detect the focus noun phrase and to apply each pattern of the question’s category. The figure 9 gives an example of an annotated sentence for the question 2 («Qu’est ce que Hubble ?»). In this case, all patterns, for the category "Definition" (see figure 7), are applied. The pattern "GNRep GNFocus" can be applied (the answer focus is "Hubble"). Thus, the noun phrase "E-3G10" is a candidate answer.
For the "Definition" category, the pattern strategy gives good results. Nevertheless, for more complex questions, a semantic process could improve the answer search. Indeed, sometimes the build of powerful patterns is a difficult task (such as for the question «Dans quel lieu des massacres de Musulmans ont-ils été commis en 1995 ?») knowing our patterns are based on the question’s focus and the focus is not always easy to find. In addition, the answer type is not always easy to define without semantic information. Another improvement can take into account verb categorization. Indeed, the verb in the question is quite important. 5.6
When the answer type was been determined by the question analysis step, the process extracts, from the list of passages provided by the previous step, the candidate answers. Named entities, acronym definitions or numerical entities closest to the question focus (if this last is detected) are supported. Indeed, in such cases, the answer is often situated close to the question focus.
The answer selection process depends on the question category. For numerical entities, named entities and acronym definitions, the right answer is the one with the best frequency. This frequency is weighted according to several heuristics such as: the distance (in words) between this answer and the question focus, the presence in the sentence of named entities or dates from the question, etc. For answers extracted by the pattern-based selection, two strategies are used according to the question category: • the selection of the first selected answer obtained by the first applicable pattern, • the selection of the most frequent answer (the candidate answer frequency). Most of the time, the first heuristic is the better one. Indeed, the selected answer is the first one obtained by the first applicable pattern (patterns sorted according to their convenience) and into the first passage (sorted by the passage selection step according to their convenience). Nevertheless, for definitionnal questions such as the 90th question «Qui est Boris Becker ?» ("Who is Boris Becker?") or the first question «Qu’est ce qu’Atlantis ?» ("What is Atlantis?"), we noted that the better strategy is the candidate phrase frequency. Indeed, for this question category where the number of question’s terms is low, the passage selection step does not make it possible to the system to select with precision passages containing the answer. Therefore, the frequency-based strategy generally selects the right answer. For example, for the second question, the answer «télescope» is selected for the definitionnal question (figure 9) because of its frequency. Table 2 presents all results of our run according to question types.
Named entities extraction Numerical entities extraction Acronym definition search Pattern-based answer extraction
43 questions (21.5%) was considered as named entities extraction strategy. Our process find at least 15 right answers. 81 questions (40.5%) was considered as named entities extraction strategy. Our process find at least 23 right answers. 71 questions (35.5%) was considered as pattern-based answer search type. Our process finds at least 16 right answers.
5 questions (2.5%) were analyzed as acronym definition search: question 28 (OMS), question 48 (OUA), question 95 (RKA), question 129 (KMT) and question 145 (TDRS). Our system found 4 good answers (28 with "Organisation Mondiale pour la Santé", 48 with "Organisation de l’ unité africaine", 129 with "Kouomintang" and 145 with "Tracking and Data Relay Satellite"). Only "RKA" (for "Agence Spatiale Russe") was not found. This is a specific case. In fact, the definition does not contain the letter "K". Actually, "RKA" is based on the russian definition that does not appear in the corpus.
For 4 questions (5 questions awaiting a list were undetected), a list of answers is awaited: • question 88 (Pattern-based search strategy), «Citer le nom de tous les aéroports de Londres , en Angleterre .» which demands an unlimited list, • question 92 (Named entities extraction strategy), «Qui sont les deux principaux responsables de l’ attentat d’ Oklahoma City ?», waiting for 2 answers, • question 100 (Named entities extraction strategy), «Donner le nom des neuf planètes qui constituent le système solaire .», 9 answers awaited. • question 117 (Named entities extraction strategy), «Quels sont les sept pays les plus industrialisés du monde ?», 7 answers awaited.
In such cases, our process does not produce satisfactory results. For the first "question" (88), a process problem due to the unlimited list causes no answer ! For the second one (92), we give the right answers (with the same associated sentence). For the third one (100), we give only one wrong answer. For the last one (117), we have found 3 of the seven answers, ie (Canada, Russia, Bosnia, Italy, USA, Ukraine, Uruguay) instead of (USA, Canada, Japan, United Kingdom, France, Germany and Italy). In this experiment report, we have studied our second version of the Prodicos QA system on QA@CLEF2006 question set. The comparison between this version and the first one studied on QA@CLEF2005 is not an easy task. Indeed, question types are quite different. For instance, in the CLEF’2005’s session, 21 questions were acronym definition search. Conversely, in the CLEF’2006’s session, only 5 questions were acronym definition search. Consequently, we have not presented our result relatively to the preceding system.
The hard result of the evaluation of the 2006 session is a rate of good answers (overall accuracy) of 29% (14.5% at the 2005 session). We regard this result as encouraging (although definitely perfectible). In addition, acronym definition questions are less numerous while our tool is more powerful. The rate of good answers is definitely low but some improvements were made. For instance, the pattern-based answer extraction found 16 answers whereas it found only 2 answers last year. In addition, in this process, 11 answers are "inexacts" (int the set "X"). As a result, it is obvious that the syntactic parser has to be improved to reduce the set "X" and, consequently, to increase the set "R".
Positive points are: (1) a good study of the question type, (2) a correct passage search and (3) an improvement of the answer extraction process. Nevertheless, some improvements have to be done concerning: (1) the question focus identification, (2) the use of semantic resources in French language for all process steps and (3) the answers extraction processes. Furthermore, we have to improve our French semantic ressource. Indeed, EuroWordnet in its French version has some defaults like the lack of definitions for concepts, relations between some concepts are unavailable, etc.