This paper describes the Entrance Exams task at the CLEF QA Track 2014. Following 2013 edition, the data set has been extracted from actual university entrance examinations including a variety of topics and question types. Systems receive a set of Multiple-Choice Reading Comprehension tests where the task is to select the correct answer among a finite set of candidates, according to the given text. Questions are designed originally for testing human examinees, rather than evaluating computer systems. Therefore, the data set challenges human ability to show their understanding of texts. Thus, questions and answers are lexically distant from their supporting excerpts in text, requiring not only a high degree of textual inference, but also the development of strategies for selecting the correct answer. As a novelty this year, data sets originally in English were manually translated into Russian, French, Spanish and Italian.
Following 2013 edition, the Entrance Exams task at CLEF QA Track 2014 is focused on solving Reading Comprehension tests of English examinations. Reading Comprehension tests are routinely used to assess the degree to which people comprehend what they read, so we work with the hypothesis that it is reasonable to use these tests to assess the degree to which a machine “comprehends” what it is reading. Despite the difficulty of the challenge, we believe we are building a real benchmark that will serve to measure real progress in the field during the next years.
With this goal in mind, CLEF and NTCIR started collaboration in 2013 around the idea of testing systems against University Entrance Exams, the same exams humans have to pass to enter University. The data set was prepared and distributed by NTCIR, while other organization efforts, including announcements, collecting and evaluating submissions, etc. were managed by CLEF. The success of this coordination also owes to the standard data format and evaluation methodology followed in past editions.
Participant systems are asked to read a given document and answer a set of questions. Questions are given in multiple-choice format, with several options from which a single answer must be selected. Systems have to answer questions by referring to "common sense knowledge" that high school students who aim to enter the university are expected to have. Another important difference is that we do not intend to restrict question types. Any type of reading comprehension questions in real entrance exams will be included in the test data. 3
Japanese University Entrance Exams include questions formulated at various levels of complexity and test a wide range of capabilities. The challenge of "Entrance Exams" aims at evaluating systems under the same conditions that humans are evaluated to enter the University.
The data set is extracted from standardized English examinations for university admission in Japan. Exams are created by the Japanese National Center for University Admissions Tests. Original examinations include various styles of questions, such as word filling, grammatical error recognition, sentence filling, etc.
One of such styles is reading comprehension; a test provides a text that describes some daily life situation, and questions about the text are asked. As in the previous edition, we reduced the challenge to these Reading Comprehension exercises contained in the English exams, leaving other types of exercises available for future tasks.
For each examination, one text is given, and five questions on the given text are asked. Each question has four choices. For this year campaign, we reused as development data 12 examinations from last year campaign. Besides, we provided new 12 documents, 60 questions and 240 candidate answers to be validated. 3.2
As a novelty this year, data sets for development and testing originally in English were manually translated into Russian, French, Spanish and Italian. They are parallel translations of texts, questions and candidate answers.
In addition to the official data, we collected four more unoffcial translations
into French. Despite they preserve original meaning, each translation has its
particularities that produce different effects on systems performance: text simplification, lexical
variation, different uses of anaphora, overall quality, etc. This data is extremely useful
to get insights about systems and their level of inference. Synapse [
EVALUATION Scoring of the output produced by participant systems was performed automatically by comparing the answers of systems against the gold standard collection with annotations made by humans. No manual assessment was performed.
Each test receives an evaluation score between 0 and 1 using c@1 [
A test is considered to be passed if a c@1 score above 0.5 is reached. Then, the proportion of tests that are passed is given as a global score. 5
csgs-2 cicnlp-2 cicnlp-3 cicnlp-4 cicnlp-1 cicnlp-6 DIPF-5
DIPF-3 LIMSI-4-HR
LIMSI-7 LIMSI-4 DIPF-6
Random LIMSI-2-Inv
DIPF-4 cicnlp-5 LIMSI-2 DIPF-2
DIPF-1 LIMSI-1-dude1
LIMSI-3
LIMSI-5 LIMSI-1-dude
LIMSI-6
According to Table 2, the system with higher score (Synapse for French [
Table 3 shows results for the reading perspective. First column corresponds to systems run id, second column to the overall c@1 obtained, third column shows the number of tests that the systems have passed if we consider the threshold of 0.5, and the rest of columns correspond to the c@1 value for each particular test. The results observed under the reading perspective are very encouraging. The three top systems were able to pass at least half of tests. As observed in Table 4, each test has a different degree of difficulty for the systems. There are three main reasons for that: the way the questions are formulated, the lexical gap between the text and the candidate answers, and the inherit difficulty of some questions for which wrong candidate answers seems to be closer to the supporting text in a light reading.
An overall overview of systems' descriptions shows the importance given to answer ranking over validation. In fact, all participant systems relied on ranking methods except the LIMSI-CNRS group, which applied an approach based on validation.
We found two different approaches regarding the use of documents for finding the correct answer: on one hand, some systems work with the whole document while on the other hand, some systems select a set of promising text snippets using retrieval techniques. We do not see observation about the best performance of one approach over the other.
Some participants create hypotheses combining questions and candidates, and trying to match these hypotheses with the document excerpts.
All participants except [
DIPF system [
Synapse Dèveloppement [
CICNLP system [
CSGS system [
LIMSI-CNRS system [
Last year exercise experience suggested the need to develop strategies to reject answers more than strategies to accept answers. One system started to develop this strategy but results aren’t yet among the top performers. All systems except Synapse’s for French select more incorrect answers than correct ones. This is really a measure of progress in systems development. However, at the reading perspective evaluation, we have already three systems (two teams) able to pass at least half of reading tests.
Again, the Entrance Exams task shows that Question Answering is a task far from being solved. However, it provides a real benchmark able to assess real progress in the field along future years.
The collaboration has been developed in the framework of Todai Robot Project in Japan, and the CHIST-ERA Readers project in Europe (MINECO PCIN-2013-002C02-01). The Todai Robot Project is a grand challenge headed by NII, and aims to develop an end-to-end AI system that can solve real entrance examinations of universities in Japan integrating heterogeneous AI technologies, such as natural language processing, situation understanding, math formula processing or vision processing.