strategy is a riteria use to sear h the right answer. It is determined a ording to the question predened words whi h are onsidered like ategories and are used in order to generate the answer from EuroWordnet Thesaurus [1℄ is used. Lists of words are build whi h are hyponyms of some It is generally lo ated lose to the answer within the passages whi h may ontain the answer. The ? , "Abagelard de Paris" is onsidered as a single entity. important features from questions (answer type, question fo us). The answer type may be a features are extra ted from the questions in order to improve the passage ranking pro ess (named 1. Question type: QUAND The rst and main feature whi h omes from the question analysis is the question type. Twenty entities, noun phrases and dates). For example, for the query Quand Abagelard de Paris est-il nØ to perform an answer sear h but also it will make it possible to sele t rules to extra t other Length, Weight, Finan ial-Amount...). For determining answer type, semanti knowledge oming For example, if the question is Quand Abagelard de Paris est-il nØ ? , the analysis of this question is: type [4℄. The question fo us orresponds to a word or a group of word involved into the question. fo us and the question type. The strategies available are either an named entity strategy, either a question types are dened whi h orrespond to a simplied synta ti form of the question 1 (for named entity (Person, Lo ation-State, Lo ation-City, Organization...), or a numeri al entity (Date, example the type QuiVerbeGN). The question type will not only help to determine the strategy numeri al entity strategy, either an a ronym denition strategy or a pattern-based strategy. Other 4. Strategy: Numeri al Entity 3. Question Fo us: Abagelard de Paris 2. Answer type: DATE 5. Named Entity: "Abagelard de Paris" 4 Passage ranking module answer type • strategy • on the other hand, a list of passages evaluated by our previous step a ording to this question [4℄. questions. To this end, we use on the one hand elements oming from the question analysis like, for instan e, the question’s ategory, the strategy to use it, the number of answers, and so on and, This module is divided into four steps: After the question analysis and the passage ranking, we have to extra t answers orresponding to sept as being September whereas it orresponds rather to the integer 7 (the name of the With the pre eding example, we obtain then: First of all, we have a system allowing to re ognize the numeri al values in form literal. Then, On this result, we an make a some remarks. First of all, are not regarded as "dates" only re ognizes September but also sept. or sept . Consequently sept indi ates an integer To fa ilitate the omparison of date, we hose to al ulate the elements of date (and hour) in the referen es in luding a day, a month and a year. In the ontrary ase, this referen e is labelled This arti le gives information on erning the "dire tion of time" ompared to the obje t of the the ISO 8601 form. For that, we have al ulated the numeri al values of the years, the months "duration" (durØe in fren h). Moreover, we also labelled the arti le pre eding this referen e. but also September. television hannel ; sept means sev en in fren h). This is produ ed by two dieren t pro esses. the months whose name is long are often shortened. Also, we parameterized the system so that it and the days. Then, we built the ISO form of the date. We made the same thing with hours. senten e. Our system of temporal labelling still has some imperfe tions. In this example, it labels <duree type="date"> <mod-pre type="eq">En</mod-pre> <mois type="car">mars</mois> <annee type="num">1989</annee> </duree> , La <mois type="car">Sept</mois> devient la Société européenne de programmes de télévision et reçoit du CSA l’ autorisation d’ émettre sur le satellite TDF 1 <duree type="date"> <mod-pre type="eq">en</mod-pre> <mois type="car">avril</mois> <annee type="num">1989</annee> </duree> . Elle commence à diffuser ses programmes <date type="lin"> <mod-pre type="eq">le</mod-pre> <no-jour type="num">30</no-jour> <mois type="car">mai</mois> <annee type="num">1989</annee> </date> ; <duree type="date" iso8601="1989-03"> <mod-pre type="eq">En</mod-pre> <mois type="car" val="03">mars</mois> <annee type="num" val="1989">1989</annee> </duree> , La <mois type="car" val="07">Sept</mois> devient la Société européenne de programmes de télévision et reçoit du CSA l’ autorisation d’ émettre sur le satellite TDF 1 <duree type="date" iso8601="1989-04"> <mod-pre type="eq">en</mod-pre> <mois type="car" val="04">avril</mois> <annee type="num" val="1989">1989</annee> </duree> . Elle commence à diffuser ses programmes <date type="lin" iso8601="1989-05-30"> <mod-pre type="eq">le</mod-pre> <no-jour type="num" val="30">30</no-jour> <mois type="car" val="05">mai</mois> <annee type="num" val="1989">1989</annee> </date> ; <date-time type="lin" iso8601="1994-01-17T13:31"> <date type="lin" iso8601="1994-01-17"> lundi <no-jour type="num" val="17">17</no-jour> <mois type="car" val="01">janvier</mois> <annee type="num" val="1994">1994</annee> </date> <time type="lin" iso8601="T13:31">
<mod-pre type="eq">à</mod-pre> <heure type="num" val="13">13</heure> h 5.3 A ronym denition extra tion 5.4 Pattern-based answer extra tion 5.2 Named entities extra tion 5.5 Answer sele tion The answer sele tion pro ess depends on the question ategory. For numeri al entities, named Indeed, in su h ases, the answer is often situated lose to the question fo us. quen y is weighted a ording to several heuristi s su h as: the distan e (in words) between this When the answer type was determined by the question analysis step, the pro ess extra ts, from answer and the question fo us, the presen e in the senten e of named entities or dates from the denitions or numeri al entities losest to the question fo us (if this last is dete ted) are supported. question, et . For answers extra ted by the pattern-based sele tion, two strategies are used a entities and a ronym denitions, the right answer is the one with the best frequen y. This frethe list of passages provided by the previous step, the andidate answers. Named entities, a ronym ording to the question ategory: this AVE 2008 evaluation, few questions suggested were based on the temporal aspe t. Moreover, temporal elements in the questions as well as in the sele ted passages. Unfortunately, on erning the pro essing was undoubtedly in omplete. It thus did not produ e signi an t results.
This system of labelling fun tions but is to be improved. The obje tive is better to lo ate the For the pattern-based answer extra tion pro ess, we developed our own tool. fo us of the question. rst step onsists in only sele ting passages whi h ould ontain the answer and whi h ontain the to all passages. Thus, we obtain a set of andidate answers for this question. Patterns (synta ti answer. Patterns are sorted a ording to their priority, i.e. answers extra ted by a pattern with lower priority. an higher priority are onsidered as better answers than the ones extra ted by patterns with a As a result, for a given question, patterns asso iated with the question ategory are applied patterns) are based on the noun phrase that ontains the fo us of the question. Therefore, the These patterns are based on the question fo us and makes it possible to the system to extra t the A ording to question ategories, synta ti patterns were dened in order to extra t answer(s). a temporal validation • The validation module is divided into two steps: 6 Validation module In the AVE 2008 task, our temporal validation module obtains the following results: If there are temporal elements but highly ontradi tory (not same year in the question If there is no temporal element in the passage : s ore = 0 2 If there are temporal elements but not ompleted (same year, but days or months not For the rst time, our goal is to use the temporal oe ien t to hoose the best passages for and in the passage): s ore = -2 If there are temporal elements but ontradi tory (year not spe ied in the question or the passage): s ore = -1 spe ied in the question or the passage): s ore = 1 the question : the passages whi h have a temporal oe ien t equal to 1 or 2 are sele ted.
If there are temporal elements in the question : • If there are exa tly same temporals elements in the question and the passage : s ore = temporal oe ien t = 2 : 9 passages • in the evaluation. The human evaluation results are given in table 1. A rst evaluation of the noting that the "unknown" value given by a human expert to an answer is not taken into a ount equal to 0.56 and a re all rate equal to 0.46. results obtained by Prodi osAV are given in table 2. 43 answers were validated by Prodi osAV and among them 24 are validated too by human judges. 136 answers were reje ted by our system All answers (199) given by the system are analysed a ording to human judgment. It is worth validated by Prodi osAV only 5 of them do not orrespond to the human judgment. Therefore, and among them 109 are reje ted too by human experts. Our system obtains a pre ision rate We made an other evaluation on erning the type of question and results obtained (table 3 and table 4).
For denitional question, the test set ontains 46 answers for 29 questions. For 16 answers but the onden e oe ien t is de reased If the Prodi os answer in luded in the passage’s answer: the passage’s answer is validated • oe ien t is more than de reased The Prodi os answer is not ompletly in luded in passage’s answer (depends on the number • of present words of this answer) : the passage’s answer is validated but the onden e 43 136 Prodi osAV System 6 22 Validated 7 16 by Prodi osAV 16 by human 22 8 7 Validated Reje ted 20 47 by human 15 44 Numeri al entity Denition Other queries Named entity Reje ted Validated 11 2 8 Prodi osAV and human Validated by 3 by Prodi osAV 21 Reje ted 30 50 47 system should have obtained is 52% (a ording to the number of reje ted response the humans 23 responses whi h are well validated. Human experts gives a negative response for 56 questions, The se ond group of measures aims at omparing QA systems performan e with the potential maximum that the system should have obtained is 48% (a ording to the number of validated response the humans found). The qa-rej-a ura y rate obtained is 35%. The maximum that the found). Our system obtains a not satisfa tory estimated-qa-performan e rate equal to 29%. The among them, our system gives 38 negative responses. The qa-a ura y rate obtained is 22%. The gain that the parti ipant Answer Validation systems ould add to them. The test set in ludes 108 questions. Human experts nd a response for 52 of them. Among them, our system gives maximum that it should have obtained is 73%. This shows that Prodi osAV System has not a the question fo us doesn’t dete t in a lot of passages or from question analysis problem or from questions, the test set ontains 56 answers for 27 questions. The system pre ision and rappel are For numeri al entity question, the test set ontains 28 answers for 16 questions and for named Win hester) est une Ø rivain") or from a ronym extra tion problems (the meaning of the a ronym equivalent : only 2 answers of 6 answers validated by Prodi osAV and of 7 answers validated by problem (for example question 188: "Vasa" tagged as verb), from pattern extra tion problems (for system while 23 of them should have been. The problems ome mainly from question analysis is translated in the passage what implies that a ronym’s letters are ompletely independant of it). validated by Prodi osAV orrespond to the human judgment. And the system re all is also worse, example question 159: "Jane Austen (16 dØ embre 1775, Steventon, Hampshire - 18 juillet 1817, the human judgment. The problems ome mainly from the sele tion of the andidate passages : entity question, the test set ontains 69 answers for 36 questions. Only 11 answers of 29 answers the system pre ision is high but its re all is worse. Indeed, only 16 answers were validated by our referen e’s absen e. indeed only 11 answers were validated by our system while 24 of them should have been. For other 24 Human expert 109 Other queries Numeri al entity Denition Named entity 7.2 System analysis at questions level Table 3: Prodi osAV evaluation Table 2: Prodi osAV evaluation Table 4: Prodi osAV evaluation [1℄ Vossen P. : EuroW ordNet: A Multilingual Database with Lexi al Semanti , editor Networks Piek Vossen, university of Amsterdam, 1998. franais, These en informatique, Nantes, LINA(2004) [3℄ Fourour, N. : Iden ti ation et atØgorisation automatiques des entitØs nommØes dans les textes [6℄ S. Tellex S., Katz B., Lin J., Fernandes A., Marton G.: "Quantitative evaluation of passage in information retrieval, pages 4147. ACM Press, 2003. retrieval algorithms for question answering", In SIGIR onferen e on Resear h and development niques", PhD Thesis, UniversitØ de Nantes, LINA, De’e embre 1999. http://www.s ien es.univ[8℄ E. Morin, "Extra tion de liens sØmantiques entre termes Æ partir de orpus de textes te hnantes.fr/info/perso/permanents/morin/arti le/morin-these99.pdf 8 Con lusion and Prospe ts Referen es good ability to a knowledge the identi ation of questions with a set of answers in whi h no orre t one has been found.
The results are not satisfa tory, be ause we only re over 24 orre t answers on 53 orre t answers. strategies of extra tion module and not only the best. We an also take into a ount the another improve our validation module by taking into a ount all the answers proposed by the various pattern extra tion problem (spe ialy the absen e of semanti s and oreferen e). We an also The problems ome mainly from question analysis problem (spe ialy the word’s labeling), from external informations as the passage’s date, et . un systŁme de question - rØponse, These en informatique, Paris Sud, ORSAY, LIMSI (2003) [2℄ Mon eaux L. : A daptation du niveau d’analyse des interventions dans un dialogue - appli ation