This paper presents the work carried out at ANLP Research Group for the CLEF-QA4MRE 2012 competition. This year, the Arabic language was introduced for the first time on QA4MRE lab at CLEF whose intention was to ask questions which require a deep knowledge of individual short texts and in which systems were required to choose one answer from multiple answer choices, by analyzing the corresponding test document in conjunction with background collections. In our participation, we have proposed an approach which can answer questions with multiple answer choices from short Arabic texts. This approach is constituted essentially of shallow information retrieval methods. The evaluation results of the running submitted has given the following scores: accuracy calculated overall all questions is 0.19 (i.e., 31 correct questions answered correctly among 160), while overall c@1 measure is also 0.19. The overall results obtained are not enough satisfactory comparing to the top works realized last year in QA4MRE lab. But as a first step at the roadmap of the evolution of the QA to Machine Reading (MR) systems in Arabic language and with the lack of researches investigated in the MR and deep knowledge reasoning in Arabic language, it is an encouraging step. Our proposed approach with its shallow criterion has succeeded to obtain the goal fixed at the beginning which is: select answers to questions from short texts without required enough external knowledge and complex inference.
The QA4MRE lab this year was aimed to evaluate machine reading systems which require a deeper level of text understanding to answer questions with multiple answer choices in a set of seven languages1. The way proposed to assess the understanding of a text is the ability to answer a set of questions about it. This evaluation manner is similar to reading comprehension tests designed to evaluate how well a human reader has understood a text.
This paper is structured into 6 sections. Section 2 presents a short overall of the state of the arts of Machine Reading Evaluation. Section 3 details our proposal approach to deal with QA4MRE lab in Arabic language. Section 4 presents the experiment carried out. Section 5 discusses the obtained results, and finally, Section 6 presents a general conclusion and perspective.
Our proposed approach is constituted of a shallow process to understand texts based on Information Retrieval (IR) and it requires inferring from texts. Four steps are involved in the shallow method after the preprocess of the corpus based on the anaphoric resolution. The first step is about the question analysis where the stop words in the question were removed and the rest of words are saved as the question focus. The second step is the research of passages containing the focus question. In the third step, the passages collected are aligned with the multiple answer choices for the respective question. The answer which is included at these passages is selected as the correct answer from the multiple answer choices. In case, where there is no answer included in passages. We introduce a list of inference constituted of pair of words generated from the background collection according to a list of inference rules. In the alignment process, any word from the answer option that does not exist in the passage is replaced by its respective inference word. If after using the inference there is not an answer included in the collected passages, the question is considered as unanswered.
This approach is proposed for dealing exactly with non complex question types such as the factoid question type, and questions whose answers are selected from their respective single short texts. Table1 illustrates an example of these questions introduced in the data set of QA4MRE lab 2012. This part of questions is similar to the part of 76 questions (63%) of the 120 questions used in QA4MRE lab in CLEF 2011 and that do not require extra information from the background collection in order to be answered. They require just information presented in a paragraph or a sentence [2].
When India began to recognize the patented pharmaceutical products? The proposed approach was implemented in a system following the architecture shown in Figure 1. The module ‘preprocess of the corpus’ where the anaphoric resolution and the construction of inference list by topic according to inference rules were not integrated. The evaluation of our system was carried out using the data given by the CLEF organization. The test set is composed of 16 test documents, 160 questions with a set of five answer choices per question. The test documents are related to 4 topics (i.e. “Aids”, “Climate change”, “Music and Society”, and “Alzheimer”). Each topic includes 4 reading tests and each reading test is composed of one single document with 10 questions related to it. A background collection consisted of un-annotated documents related to the 4 topics in 7 languages are given to the participants to acquire the reading capabilities and the knowledge needed to fill in the gaps required to answer a test on the topic. Our system was required to answer these 160 questions by selecting one answer from the five alternative answers. There is always one and only one correct answer for each question. We have the option to leave a question unanswered if there is no confidence about the correctness of its response. The measure c@1 is applied to encourage systems to leave some questions unanswered in order to reduce the amount of incorrect answers [3]. The results of the evaluation measures are as follows: the overall accuracy is equal to 0.19 (i.e. we have succeeded to answer correctly to 31 questions from 160 questions), and C@1 is equal also 0.19 where: nr: is the number of correctly answered questions nu: is the number of unanswered questions n: is the total number of questions 5.
The overall results obtained are not comparable to the top performance obtained last year for the English language. Nevertheless as a first step at the roadmap of the evolution of the QA to MR evaluation systems in Arabic language and with the lack of researches investigated in deep knowledge reasoning in Arabic language, it could be considered as an encouraging step. Our proposed approach with its shallow criterion has succeeded to obtain the goal fixed at the beginning which is: selecting answers to questions from short texts without required external knowledge and complex inference. 6.
The QA4MRE lab was focused this year on the evaluation of Machine Reading systems. The goal behind it was to push researches towards a deeper understanding of a single text using inference deduced from document collection. We have participated at this QA4MRE lab which has included Arabic language for the first time. In our work, we have proposed an approach which did not require a deep reasoning and inference. We have succeeded to a certain degree to obtain an overall accuracy of 0.19. We plan in the future to improve this result by investigating further in MR research deep knowledge reasoning in Arabic language.
Acknowledgments. The European Commission as part of the WIQ-EI IRSES-Project (grant no. 269180) within the FP 7 Marie Curie People Framework has partially funded the work of the third author. His work was carried out also in the framework of the MICINN Text-Enterprise (TIN2009-13391-C04-03) research project and the Microcluster VLC/Campus (International Campus of Excellence) on Multimodal Intelligent Systems.