We report in this paper the coreference annotation process of the CSTNews corpus as part of a collective task of the IberEval 2017 conference. The annotated corpus is composed of 140 news texts written in Brazilian Portuguese language and counts with several annotation layers, including annotations in the morphosyntax/syntax, semantics, and discourse levels. The annotation, focused on nominal references, was conducted in a semi-automatic way by ve teams, achieving satisfactory annotation agreement results.
Coreference resolution is the task of nding linguistic expressions in a text that
refer to the same entity [
At least 17 people died after the crash of a passenger plane in the
Democratic Republic of Congo. According to an ONU spokeswoman, the
airplane was trying to land in the Bukavu airport in the midst of a
storm. It failed to reach the runway and fell in a forest 15 kilometers
away from the airport.
Coreference resolution and its subtasks have been investigated for a long time
in the Natural Language Processing (NLP) area. There are approaches based on
heuristics [
Coreference is a very important information for NLP systems. For instance, it is essential for summarization systems to produce coherent and cohesive summaries, allowing the systems to properly \glue" together text passages; it is useful to track, on the web, entities of interest; it may be necessary for information extraction about entities of interest and for performing the related inference; it may help simplifying texts, allowing changing pronominal anaphoras by nominal antecedents, as certain anaphoric mentions cause di culties for people with cognitive disabilities; and so on. Therefore, research e orts in such task are very relevant. Although some coreference annotated corpora and automatic coreference annotation softwares are available for English (the interested reader may refer to the shared task in CoNLL), similar initiatives for other languages are still rare, including Portuguese, which is the focus of this paper.
We report here the coreference annotation process of a corpus of news texts
written in Brazilian Portuguese language - the CSTNews corpus [
In what follows (Section 2), we present some basic concepts regarding coreference. In Section 3, we introduce the corpus we annotated. Section 4 reports the annotation process and the achieved results, describing some challenges of the task. Some nal remarks are presented in Section 5. 2
Detecting coreferent elements and composing coreference chains is a task that demands sophisticated linguistic knowledge. Coreference mainly happens at the semantics/discourse interface, as it requires the identi cation of the meaning of the elements (i.e., to what they refer to) in the same sentence or across sentences in a text. There are also initiatives for detecting coreference relations across di erent texts, which is relevant for multi-document processing purposes.
As presented in [
Referring expressions may happen in several forms in a text, usually
syntactically realized as noun phrases. For instance, we may use pronouns (e.g., \it"),
common nouns (\airplane") and proper nouns (\William Shakespeare"),
optionally presenting determiners, pre- and/or post-modi ers (e.g., \the beautiful girl")
and having high size variance (e.g., \the beautiful and charming girl that was
looking at me"). In [
To the best of our knowledge, the only manually annotated corpus in
Brazilian Portuguese that is speci cally focused on coreference chains is the Summ-it
corpus [
In such scenario, producing coreference annotated corpora and developing and/or improving coreference resolution systems for Portuguese are key issues to be pursued. The annotation e ort reported in this paper is a step towards new advances in the NLP area for Portuguese. In what follows, we introduce the corpus that was annotated. 3
The CSTNews corpus [
In the years following its creation, the corpus received several linguistic
annotation layers (mainly of discourse nature), according to the uses it had. The
corpus currently includes:
{ manual multi-document discourse annotation, according to the Cross-document
Structure Theory (CST) [
Structure Theory (RST) [
Except for the last one, which was automatic, all of these annotations were manually carried out in systematic and controlled way. Satisfactory annotation agreement results were obtained (when applicable), which allows to infer that the data is reliable.
With such annotations, the corpus has subsidized several research e orts,
including traditional multi-document summarization [22{24], the more recent
update summarization [
The coreference annotation reported here constitutes a new annotation layer in the CSTNews corpus. The annotation process is described in the following section. 4
The coreference annotation of the CSTNews corpus was carried out as a collective task, named \Collective Elaboration of a Coreference Annotated Corpus 3 https://tac.nist.gov//2010/Summarization/Guided-Summ.2010.guidelines.html for Portuguese Texts"4, of the IberEval 2017 (Evaluation of Human Language Technologies for Iberian Languages) conference5.
In the collective task, annotation teams with at least 3 members should submit their corpus for annotation. If selected to participate, the members of the team would receive pre-processed texts to annotate. Part of the texts was from the corpus submitted by the team (4 of them were used to compute the annotation agreement of the team), and another part was from the corpora submitted by other teams. Each member should then individually annotate the coreference chains in his/her texts. Only nominal referring expressions were intended to be annotated.
As said above, the texts were presented in a pre-processed form. They were
automatically annotated by the CORP coreference resolution system [
Figure 1 shows the CorrefVisual interface with an annotated text loaded. The text is in Portuguese, as it is in the corpus. It is about an accident in an airport, where an airplane crashed into a building. One may see the text at the left of the panel, the coreference chains in the middle, and the remaining referring expressions that did not form any chain, called \unique mentions", on the right side. Above these unique mentions, there is an auxiliary panel to help dealing with the chains. In the interface, each chain is associated to a di erent color, and, every time that an element is selected, its occurrence in the text is highlighted in the corresponding color.
Correcting a chain consisted, therefore, in moving the referring expressions among the windows in the tool, e.g., removing an expression from a chain and adding it to another one, incorporating a unique mention to some existing chain, creating new chains, etc. If necessary, the auxiliary panel would help in grouping and moving the elements across the windows. The tool also allowed to edit the referring expressions, by adding or removing words next to it (to the left or to the right of the expression). This is an important step, as such expressions were automatically detected and some errors probably occurred (in fact, during the annotation, we have noticed that such errors were very frequent). Unfortunately, the tool does not allow to use new expressions that were not previously detected by the tool itself. The edition of the target expressions had also been explicitly and highly discouraged by the organizers of the collective annotation task. Such 4 http://ontolp.inf.pucrs.br/corref/ibereval2017/ 5 http://nlp.uned.es/IberEval-2017/ decision bene ts the annotation agreement results, but is also based on the idea that we should make use of the available state of the art preprocessing tools, with their current limitations and potentialities.
Finally, one may also notice in the interface that a semantic category should
be assigned to each chain, indicating the type of the entities of the chain. In
the interface, the category appears above each chain, in capital letters. The
available semantic categories in the tool are based on the named entity typology
of the REPENTINO gazetteer [
In order to fully annotate the CSTNews corpus, 5 annotation teams were assembled: one with four members and four with three members. Each team was leaded by its more senior member, usually a researcher with some experience in NLP and corpus annotation. All the members were native speakers of Portuguese.
Initially, all the teams read the material made available by the organizers of
the collective annotation task (regarding the task instructions and how to use the
CorrefVisual tool) and also studied a didactic reference paper on coreference for
Portuguese [
After the training with a single text, the task organizers considered that the task was well understood and provided the teams with the actual texts to annotate. In average, each annotator had to annotate from 11 to 13 texts, in a period of a month and a half. In general, most of the teams managed to nish the task in 2 to 3 weeks, with each member trying to annotate one text per day.
Some issues were very challenging to deal with. Several members had severe problems with the annotation tool, which apparently has inconsistent functionalities in di erent operating systems. Given the di culties with the tool, some annotators preferred to manually annotate the texts (in paper or in a di erent electronic edition application) before replicating the process in the tool. More serious than the inconsistencies of the tool were its limitations regarding the inclusion of referring expressions and the guideline for avoiding editing the expressions. Such problems are the main cause of some very poor annotations, with wrong referring expressions in the chains (e.g., see in Figure 1 the expression a aeronave que in the rst chain; in English, \the airplane that"), non-nominal elements (as que; \that"), and expressions in the text that were not detected by the tool and, therefore, could not be included in any chain.
The task organizers provided the annotation agreement results to the teams.
Agreement was computed over all possible pairs of referring expressions that
formed coreference chains, and, for each pair of expressions, it was indicated
how many annotators said that the pair was in fact coreferring. Kappa measure
[
Table 1 shows the kappa results for each team, including the general average. As said before, for each team, 4 texts were used for computing kappa. One may see that, excepting team 2, all the other teams reached an agreement of at least 0.50. In average, the teams presented an agreement of 0.54, which we consider satisfactory given the annotation conditions (limited training during a short time period, and di culties and limitations with the annotation tool). Overall, including the other participants of the collective annotation task, the minimum agreement value was 0.41, and the maximum was 0.64 (achieved by our team 4).
Some very interesting cases of disagreements may be found in the data, which may be partially explained by the limited training, and partially by di erent principles regarding the task and the high level of subjectivity in some cases. For instance, to cite a few (related to the text shown in Figure 1): { while one annotator has built a coreference chain with all the elements that refer to the airport (that is, Congonhas - which is the name of the airport - and o aeroporto - \the airport", in English), another one has divided this chain in two, one for the sense of airport and another one for the sense of the location of the airport, being this di erence very di cult to realize (and this may explain why the semantic categories \organization" and \place" have been joined by the task organizers); { one annotator has considered that the expressions hipotese (\hypothesis") and falha mec^anica (\mechanical failure") were coreferent (as the \hypothesis" for the accident with the airplane was a \mechanical failure"), but another one considered that these expressions are in di erent generalization levels and built di erent chains for them; { some annotators have built chains for time expressions that refer to the same event (in such cases, some inference was necessary to determine the correct time that was referred to), while others did not, considering that time is not a proper element to a coreference chain (the attentive reader probably realized that the semantic categories in the annotation tool did not include \time", which is a traditional class in named entity recognition tasks); { some annotators have included in the chains the occurrences of relative pronouns (e.g., que; \that/which", in English), as they refer to some previous entities and were detected by the annotation tool, but other annotators did not consider such items because they are not strictly nominal expressions (which were the focus of the annotation).
Di erences as the previous ones may result in broad variations in the nal annotation. For instance, for the text of Figure 1, the three annotators produced from 32 to 46 coreference chains (with unique mentions varying from 49 to 86 referring expressions).
The annotation task required a lot of attention and dedication from the annotators, which had to read several times each text and its referring expressions in order to nd all the chains. In many situations, background and domain knowledge was necessary for correctly identifying the chains. Usually, annotators consulted the web (mainly Wikipedia) to solve these cases. In average, the annotators took above 1 hour to annotate each text.
We present some nal remarks in the next section. 5
All the annotated data is in an XML format, which is a traditional way of marking and making data available. It shall be available in the SUCINTO project website, as it constitutes an additional linguistic annotation layer of the CSTNews corpus.
We expect that the produced coreference annotation fosters other research
initiatives on discourse processing tasks. For the short term, the new data may
help to improve summarization models, speci cally those involving coherence
and cohesion evaluation, for which the occurrence and distribution of referring
expressions are very important features (see, e.g., the entity-based model
proposed in [
For future work, concerning the CSTNews corpus, the task of pronominal anaphora resolution remains to be done, as it was not directly tackled in the reported annotation e ort.
The authors are grateful to FAPESP, CAPES and CNPq for supporting this work.