Calor-Dial is an enriched version of the Calor corpus, collected from French encyclopedic data in order to study Information Extraction on domain specific data. The corpus was initially annotated in semantic Frames (Calor-Frame ) and enriched with a first set of questions for Machine Reading Question Answering (Calor-Quest ). The new Calor-Dial version presented here addresses the scope of conversational Question Answering. The main originality is that diferent types of questions are annotated, including more challenging configurations than in classical QA corpora. This paper describes the corpus and proposes some baseline results obtained with models trained on the FQuAD corpus.
dificulty that questions may contain linguistic phenomena such as coreferences and ellipsis
or implicit references to past turns. Existing corpora are available in English and usually their
conversations refer to short and simple paragraphs such as excerpts of Wikipedia, children
stories, web search or news. [
phenomenon such as coreference resolution, ellipsis or paraphrase.
written questions where the answers can be contained in distinct spans of the document (i.e Multihop QA). In other words, answers can gather disjoint evidence sentences. Besides annotating the spans containing the answer, Calor-Dial also provides annotations on question rewriting and answer paraphrasing. Calor-Dial contains 234 dialogues and 1663 questions with their answers.
Calor is a corpus collected for Information Extraction studies2 and regularly enriched with
annotations at various levels. It gathers French encyclopedic documents annotated with semantic
information (Calor-Frame ) following the Berkeley Framenet paradigm described in [
for the French language with two main purposes. The first one was to have a large amount of annotated examples for each Frame with all their possible Frame Elements, with the deliberate choice to annotate only the most frequent Frames. As a result, the corpus contains 53 diferent
corpus was built by gathering encyclopedic articles from two thematic domains (WW1 for First
the time the paper will be published if it is accepted. the Vikidia encyclopedia for children and CT for the Cliotext collection of historical documents), resulting in the 4 subcorpora that will be further described in Table 1.
on a document, each question containing a reference to a previous question in the sequence. The main originality of Calor-Dial is the labels attached to each question. The annotators had to qualify every question they wrote according to 4 dimensions: 1. in-context vs. out-of-context → does the question need to access to the conversational context in order to be found?
the answer or is it more abstract?
or is it self suficient?
the question?
• to write an out-of-context version for each in-context question • to write two versions of each answer, a short one containing the smallest word sequence containing the answer and a long version containing the context of the question.
• Q0 : Quels sont les trois noms d’Hammourabi?
Q0 : What are the three names of Hammourabi? – type: paraphrase-self-simple – short answer: Hammourabi, Hammurabi ou Hammurapi. – answer with context: Les trois noms d’Hammourabi sont : Hammourabi,
Hammurabi ou Hammurapi. – word span supporting answer in document: Hammourabi , ou Hammurabi ou encore Hammurapi • Q1 : Qui est-il?
Q1 : Who is he? – out-of-context question: Qui est Hammourabi? Who is Hammourabi? – type: litteral-coreference-multihop – short answer: Le vrai fondateur du premier empire de Babylone et créateur du code d’Hammurabi. – answer with context: Hammourabi est le vrai fondateur du premier empire de Babylone et créateur du code d’Hammurabi. – word span supporting answer in document: le vrai fondateur du premier empire de Babylone [. . . ] célèbre pour le code d’ Hammurabi • Q2 : Qu’est-ce que ce code?
Q2 : What is this code? – out-of-context question: Qu’est-ce que le code d’Hammurabi? What is the
Hammurabi code? – type: litteral-coreference-simple – short answer: Un recueil de lois. – answer with context: Le code d’Hammurabi est un recueil de lois. – word span supporting answer in document: • Q3 : Sur quel support a-t-il été écrit?
Q3 : On which support was it written? – out-of-context question: Sur quel support a été écrit le code d’Hammurabi? On which support was the Hammurabi code written? – type: paraphrase-coreference-simple – short answer: Sur une stèle. – answer with context: Le code d’Hammurabi a été écrit sur une stèle. – word span supporting answer in document: sur une stèle • Q4 : Où a-t-elle été découverte ?
Q4 : Where was it discovered? – out-of-context question: Où a été découverte la stèle supportant le code d’Hammurabi? Where was the stele supporting Hammurabi code discovered? – type: paraphrase-ellipse-simple – short answer: À Suse. – answer with context: La stèle supportant le code d’Hammurabi a été retrouvée à Suse.
– word span supporting answer in document: à Suse • Q5 : Où est-elle exposée aujourd’hui ?
Q5 : Where is it exhibited nowadays? – out-of-context question: Où est aujourd’hui exposée la stèle supportant le code d’Hammurabi? Where is the stele supporting the Hammurabi code exhibited nowadays? – type: paraphrase-coreference-simple – short answer: Au musée du Louvre à Paris. – answer with context: La stèle supportant le code d’Hammurabi est aujourd’hui exposée au musée du Louvre à Paris.
– word span supporting answer in document: au musée du Louvre à Paris
conversations have been annotated for a total amount of 1663 questions. The questions are spread in the four subcorpora as described in Table 1.
experiments can be run by using only out-of-context questions (including the first questions of each conversation, as well as the following questions in their full out-of -context reformulation). For this experimental setup, it is possible to analyse the results along with 4 levels of dificulty refering to both the formal similarity between the question and the paragraph (Question: literal vs paraphrase) and to the level of analysis that must be performed within the paragraph to retrieve the answer (Paragraph: simple vs multihop):
answer to answer with context) or question rephrasing (from in-context question to out-of-context question).
setup. To this purpose we fine-tune the transformer model CamemBERT (large version, 335M parameters) 3 on the FQuAD corpus [11].
The results obtained on the Calor-Dial corpus are given in table 4. We use the following metrics: exact-match and F-score between the word spans expected in the reference annotations and the prediction by the MRQA model. As we can see the results obtained are much lower that those that can be obtained on the FQuAD or the SQuAD test corpora. This can be explained by the fact that the specific topics in the Calor-Dial corpus are quite diferent from those in FQuAD. We can also verify that paraphrase and multihop are two complexity factors that afect greatly the performance of the MRQA model. Each level of dificulty has an impact of roughly 10 points of F-measure compared to the previous one. This advocates the need for more sophisticated model to be able to handle properly dificult phenomena such as paraphrases and multihop.
dificulties corresponding to the diferent dimensions characterizing each questions. It can also be used for evaluation text generation models such as Answer Generation models (from in-context to out-of-context answers), Question Rewriting models (paraphrasing in-context question into out-of-context questions) and Question generation. The corpus is publicly available on the following archive: https://gitlab.lis-lab.fr/calor/calor-dial-public semantic annotations, in: MRQA: Machine Reading for Question Answering-Workshop at EMNLP-IJCNLP 2019-2019 Conference on Empirical Methods in Natural Language
[11] M. d’Hofschmidt, W. Belblidia, Q. Heinrich, T. Brendlé, M. Vidal, FQuAD: French question answering dataset, in: Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online, 2020, pp. 1193–1208. URL: https:// aclanthology.org/2020.findings-emnlp.107. doi: 10.18653/v1/2020.findings-emnlp. 107.