English. In this paper, we present DARCIT, a new reading comprehension dataset for the Italian language aimed at identifying 'question-worthy' sentences, i.e. sentences in a text which contain information that is worth asking a question about1. The purpose of the corpus is twofold: to investigate the linguistic profile of questionworthy sentences and to support the development of automatic question generation systems.
Reading comprehension (RC) can be defined as “the process of simultaneously extracting and constructing meaning through interaction and involvement with written language” (Snow, 2002). Such a definition emphasizes that RC is a complex human ability that can be decomposed into multiple operations, such as coreference resolution, understanding discourse relations, commonsense reasoning 1The corpus will be made publicly available for research purposes at the following link: http://www.italianlp.it/resources/
2Il corpus sara` messo a disposizione liberamente per scopi di ricerca al seguente indirizzo: http://www.italianlp.it/resources/ and reasoning across multiple sentences. In educational scenarios, student’s comprehension and reasoning skills are typically assessed through a variety of tasks, going from prediction tasks (e.g. cloze test) to retellings generation and question answering, which are costly to produce and require domain expert knowledge. Given also the challenges posed by the broad diffusion of distance learning programs, such as MOOC (Massive Open Online Courses), the automatic assessment of RC is becoming a rapidly growing research field of Natural Language Processing (NLP). While much more work has been done on developing Automated Essay Scoring (AES) systems (Passonneau et al., 2017), recent studies have focused on the automatic generation of questions to be used for evaluating humans’ reading and comprehension (Du and Cardie, 2017; Afzal and Mitkov, 2014). This is not a trivial task, since it assumes the ability to understand which concepts in a text are most relevant, where relevance can be here defined as the likelihood of a passage to be worth asking a question about. The availability of large and high-quality RC datasets containing questions posed by humans on a given text thus becomes a fundamental requirement to train data-driven systems able to automatically learn what makes a passage ‘question-worthy’. In this regard, datasets collected for other NLP tasks, Question Answering above all, provide a valuable resource. One of the most widely used is the Stanford Question Answering Dataset (SQuAD), (Rajpurkar et al., 2016). It contains more than 100,000 questions posed by crowdworkers on a set of Wikipedia articles, in which the answer to each question is a segment of text from the corresponding reading passage. More recently, other large RC datasets have been released: it is the case of the ‘TriviaQA’ dataset (Joshi et al., 2017), which is intended to be more challenging than SQuaD since it contains a higher proportion of complex questions, i.e. questions requiring inference over multiple sentences. The same holds for RACE (Lai et al., 2017), which is also the only one specifically designed for educational purposes. Indeed it covers multiple domains and written styles and contains questions generated by domain experts, i.e. English teachers, to assess reading and comprehension skills of L2 learners. While all these datasets are available for the English language, to our knowledge, no similar RC datasets exist for the Italian language. In this paper we introduce a new corpus for Italian specifically conceived to support research on the automatic identification of question-worthy passages. In what follows, we first describe the typology of texts it contains and the annotation process we performed on them. We then carry out a qualitative analysis based on linguistic features automatically extracted from texts with the aim of studying, on the one hand, which features mostly discriminate question-worthy sentences from other sentences and, on the other hand, whether the two classes of sentences have a different profile in terms of linguistic complexity. 2
The first step in the process of corpus construction was the selection of appropriate materials. As noted by Lai et al. (2017), a major drawback of many existing RC datasets is that they were either crowd-sourced or automatically-generated thus paying very little attention to the intended target user; this makes them less suitable to be used in real educational scenarios. To prevent these limitations, we relied on a corpus of reading comprehension tests designed by the National Institute for the Evaluation of the Education System (INVALSI), which is the Italian institution in charge of developing standardized tests for the assessment of numeracy and literacy skills of primary, middle and high school students.
To create the corpus, we focused only on tests designed to assess students’ competences in the Italian language. We thus collected a total of 86 Italian tests administered between 2003 and 2013, of which 31 targeting primary school’s pupils of the second, third and fifth grade, 29 targeting students of the first and third year of middle school and 26 targeting students of first, second and third grade of high school. To each text a number of questions is associated, which aim to deeply assess student’s ability of reading and understanding. As documented by the last available technical report provided by the Institute3, the INVALSI Italian test has been designed to cover seven main aspects underlying text comprehension, namely: understanding the meaning of words; identifying explicit information; inferring implicit information; detecting elements conveying cohesion and coherence in text; comprehending the meaning of a passage by integrating both implicit and explicit information; comprehending the meaning of the whole text; generating a meaningful interpretation (e.g. understanding the message, the purpose etc.). With respect to their form, questions can be either multiple-choice (typically with 3 or 4 options, see example (1)) or, more rarely, open-ended questions (example 2).
Example (1): Dove abita il ragno del racconto? (Where does the spider of the story live?) A. In un albero del bosco. (On a forest tree) B. Sopra un fiore del bosco. (Upon a forest flower) C. In una siepe del bosco. (In a forest hedge) Example (2): Dopo aver letto il testo, qual e` secondo te il messaggio che vuole dare l’autore? (After reading the text, what do you think is the message the author wants to give?) For the purpose of our study, we selected only the first type of questions, thus obtaining a total of 354 questions. Table 1 reports some statistics about the final corpus collected from the INVALSI tests.
SchoolGrade 2nd Primary 4th Primary 5th Primary 1st Middle 3rd Middle 1st High 2nd High 3rd High TOT For each question of the corpus, the annotation process was meant to identify the sentence (or a sentence span) containing the corresponding answer. This information was marked on text by enclosing the relevant text span in opening and closing xml tags with a letter R in upper case.
The outcome of the annotation process was a tabular file with the following information reported in separate columns: i) the text segmented into sentences; ii) a binary value 1 vs 0 (1 if the sentence contains the answer to the question and 0 if not); iii) the corresponding question; iv) the answer provided by the annotator. Table 2 gives an example of the dataset structure.
A qualitative inspection of the corpus allowed identifying different typologies of ‘questionworthy’ sentences: sentences that were the target of one question only (this is the case of the second sentence reported in Table 2); sentences that were the target of multiple questions, such as (4), and sentences that only partially answered the question (i.e. the whole information required to give the answer is spread across multiple sentences), such as (5).
(4) Question-worthy sentence: Leo decide di aiutare gli animali della giungla (Leo decided to help the jungle animals)
Corresponding questions:
Qual e` la cosa piu` importante per Leo? (What is the most important think to Leo?) Multiple choice answers: A. Essere un bravo cacciatore. (To be a good hunter); B. Diventare il piu´ coraggioso di tutti. (To become the bravest of all); C. Rendersi utile agli altri. (To make himself useful to others); D. Fare nuove esperienze. (To make new experiences).
Cosa sceglie di fare Leo nella giungla? (What does Leo choose to do in the jungle?) Multiple choice answers: A. Giocare con tutti. (To play with everybody); B. Dormire e mangiare. (To sleep and eat); C. Aiutare chi e` in difficolta`. (To help people in need); D. Nuotare nell’acqua del fiume (To swim in the river water) (5) Question-worthy sentences: “Io faro` il postino!” Disse uno. “Io faro` il maestro!” Disse un altro. “E io faro` lo chef!”. Urlo` un terzo e sal`ı sul vagone delle marmellate. (I’m going to be a postman! One said. I’m going to be a teacher! Another said. And I’m going to be a chef! Shouted a third one and went up on the wagon of the jams).
Corresponding question: A che cosa pensano i bambini quando vedono gli oggetti sul treno? (What do children think when they see the items on the train?)
Multiple choice answers: A. Ai giochi che potranno fare. (To the plays they can do); B. A cose utili che si possono vendere. (To useful things that can be sold); C. Ai regali che vorrebbero ricevere. (To the presents they would like to receive); D. Ai lavori che faranno da grandi. (To the trades they will do as adults.) 3
As a result of the annotation process, we obtained 398 ‘question-worthy’ sentences and 2059 ‘nonquestion’ worthy sentences. Starting from this classification we carried out an in-depth linguistic analysis based on a wide set of features capturing properties of a sentence at lexical, morpho– syntactic and syntactic level. The aim of this analysis was to understand whether there are some linguistic features that mostly allow predicting the ‘likelihood’ of a sentence to be the target of a question. To allow the extraction of linguistic features, all sentences were automatically tagged by the part-of-speech tagger described in (Dell’Orletta, 2009) and dependency parsed by the DeSR parser described in (Attardi et al., 2009).
Table 3 shows an excerpt of the first 20 features (of 177 extracted ones) for which the average difference between their value in the ‘questionworthy’ and ‘non question-worthy’ class was highly statistically significant using the Wilcoxon rank sum test4. As it can be seen, sentences on which a comprehension question was asked are on average much more longer. This could be expected since the longer the sentence the higher the probability that it is more informative and thus containing concepts that are worth asking a question about. This is also suggested by the higher distribution of proper nouns [10], most likely referring to relevant semantic types (e.g. person, location) which typically occur in Narrative, i.e. the main textual genre of the Invalsi tests. The higher sentence length of ‘question-worthy’ sentences has effects also at morpho-syntactic and 4All significant features are shown in Appendix (A). Sentence La lucciola si preparo` e, quando calo` la sera, ando` all’appuntamento.
Entro` nel bosco scuro e raggiunse la siepe dove viveva il ragno.
Class 0 1
Tag Question Answer Entro` <R>nel bosco scuro e raggiunse la siepe dove viveva il ragno.<nR> Dove abita il ragno del rac- In una siepe del conto? bosco. syntactic level, as shown e.g. by the higher proportion of conjunctions introducing subordinate clauses ([7] Subord. conj: 1.63 vs 1.50) and by the presence of longer syntactic relations in which the linear distance between the ‘head’ and the ‘dependent’ is higher than 10 tokens ([20] Max link: 11.30 vs 8.30).
Question NoQuestion Features Avg (StDev) Avg (StDev)
Raw Text features [1] Sentence length* 29.00 (16.11) 20.00 (13.75)
Morpho–syntactic features [2] Punctuation* 4.74 (2.82) 7.70 (6.23) [3] Negative adv* 1.23 (2.82) 1.19 (3.13) [4] Coord. conj* 3.50 (3.40) 3.20 (3.81) [5] Poss. adj* 0.96 (2.10) 0.89 (2.33) [6] Relative pron* 1.14 (2.00) 1.12 (2.32) [7] Subord. conj* 1.63 (2.80) 1.50 (2.90) [8] Prepositions* 7.90 (5.01) 7.60 (6.20) [9] Determiners* 9.13 (5.00) 9.00 (6.20) [10] Proper nouns* 2.05 (3.90) 2.00 (4.30) [11] Numbers 0.66 (1.87) 0.64 (2.25) [12] Verbs 15.98 (6.32) 16.97 (8.18) [13] Indicat. mood* 57.00 (30.70) 60.00 (33.82) [14] Particip. mood 7.13 (14.22) 6.34 (14.88) [15] 3rdpers. verb* 55.15 (39.50) 45.20 (42.62) [16] Conjunctions 5.1 (4.35) 4.34 (4.66)
Syntactic features [17] Clause length* 8.63 (4.34) 7.90 (4.24) [18] Verbal heads* 4.00 (2.30) 3.00 (2.03) [19] Postverb Subj* 13.60 (27.00) 15.70 (32.00) [20] Max link* 11.30 (7.06) 8.30 (6.80)
A further analysis was meant to investigate the profile of question-worthy sentences with respect to linguistic complexity. To this end, we exploit READ-IT (Dell’Orletta et al., 2011), a generalpurpose readability assessment tool for Italian, which combines traditional raw text features with lexical, morpho-syntactic and syntactic information to operationalize multiple phenomena of text complexity. READ–IT assigns different readability scores using the following four models: 1) Base Model, relying on raw text features only (e.g. average sentence and word length); 2) Lexical Model, relying on a combination of raw text and lexical features; 3) Syntax Model, relying on morpho-syntactic and syntactic features; 4) Global Model, combining all feature types (raw text, lexical, morpho-syntactic and syntactic features).
Results are reported in Table 4. As it can be noted, question-worthy sentences have a higher complexity with respect to all models. Especially at syntactic level, this could be expected given the higher values obtained by features related to syntactic complexity which turned out to be significantly involved in discriminating these sentences.
READ-IT Base READ-IT Lexical READ-IT Syntactic READ-IT Global
Question 59,9% 98,9 % 69,3% 100%
NoQuestion 21,1% 66,4% 37,5% 95% We presented DARC-IT, a new reading comprehension dataset for Italian collected from a sample of standardized evaluation tests used to assess students’ reading and comprehension at different grade levels. For each text, we annotated ‘question-worthy’ sentences, i.e. sentences which contained the answer to a given question. A qualitative analysis of these sentences showed that the likelihood of a sentence to be ‘questionworthy’ can be modeled using a set of linguistic features, which are especially linked to syntactic complexity. We believe that this corpus can support research on the development of automatic question generation systems as well as question answering systems. Current developments go into several directions: we are carrying out a first classification experiment to automatically predict ‘question-worthy’ sentences and evaluate the impact of linguistic features on the classifier performance. We are also planning to enlarge the corpus and to investigate more in-depth the typology of questions and answers it contains, in order to study what characterizes sentences answering, for instance, to factual vs non-factual questions. 5
The work presented in this paper was partially supported by the 2–year project (20182020) SchoolChain – Soluzioni innovative per la creazione, la certificazione, il riuso e la condivisione di unita` didattiche digitali all’interno del sistema Scuola, funded by Regione Toscana (BANDO POR FESR 2014-2020).
Naveed Afzal and Ruslan Mitkov. 2014. Automatic generation of multiple choice questions using dependency-based semantic relations Soft Computing, 18 (7), 1269–1281.
Giuseppe Attardi, Felice Dell’Orletta, Maria Simi, Joseph Turian. 2009. Accurate dependency parsing with a stacked multilayer perceptron. In Proceedings of EVALITA 2009 - Evaluation of NLP and Speech Tools for Italian 2009, Reggio Emilia, Italy, December 2009.
Felice Dell’Orletta. 2009. Ensemble system for partof-speech tagging. In Proceedings of EVALITA 2009 - Evaluation of NLP and Speech Tools for Italian 2009, Reggio Emilia, Italy, December 2009. Felice Dell’Orletta, Simonetta Montemagni and Giulia Venturi. 2011. READ-IT: assessing readability of Italian texts with a view to text simplification. Proceedings of the Second Workshop on Speech and Language Processing for Assistive Technologies (SLPAT 2011), Edimburgo, UK: 73–83.
Xinya Du and Claire Cardie. 2017. Identifying Where to Focus in Reading Comprehension for Neural Question Generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Copenhagen, Denmark.
Mandar Joshi, Eunsol Choi, Daniel Weld and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, 1601–1611.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, Eduard H. Hovy. 2017. RACE: Large-scale ReAding Comprehension Dataset From Examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Copenhagen, Denmark. Catherine Snow. 2002. Reading for understanding: Toward an RD program in reading comprehension. Rand Corporation.
Features Sentence length*** % Basic Italian Vocabulary (BIV)* % Fundamental BIV** % ‘High Usage’ BIV* Lexical density* % Adjectives* % Articles*** % Conjunctions** % Coordinat. conj*** % Demonstrative determiners*** % Indefinite pronouns % Interrogative determiners* % Interjections* % Numbers** % Negative adverbs*** % Ordinal numbers* % Possessive adjectives*** % Prepositions*** % Proper nouns** % Punctuation*** % Relative pronouns*** % Subordin. conj*** % Verbs** % Verb Participial mood** % Verb Indicative mood*** % Verb Conditional mood** % Verb Past tense** % Verb Imperfect tense** % Verb Present tense* % 3rdpers. verb*** % 2ndpers. verb* TTR ratio (first 100 lemmas)** Clause length (in tokens)*** Avg verbal heads/sentence*** Avg prep. links length* Max link length*** Verb arity % Postverbal subject*** % Preverbal objects* % DEP Root** % DEP Mod rel*** % DEP Copulative Conj** % DEP Determiner*** % DEP Disjuntive Conj % DEP Locative Compl* % DEP neg*** % DEP conj** % DEP concatenation* Question-worthy sentences Average (StDev)
Raw Text features