Cross-document event coreference resolution (CDCR) is the task of detecting and clustering mentions of events across a set of documents. A major bottleneck in CDCR is a lack of appropriate datasets, which stems from the diculty of annotating data for this task. We present the first scalable approach for annotating cross-subtopic event coreference links, a highly valuable but rarely occurring type of cross-document link. The annotation of these links requires combing through hundreds of documents - an endeavor for which conventional token-level annotation schemes with trained expert annotators are too expensive. We instead propose crowdsourcing annotation on sentence level to achieve scalability. We apply our approach to create the Football Coreference Corpus (FCC), a corpus of 451 sports news reports, while reaching high agreement between NLP experts and crowd annotators in the process.1 Events, i.e. actions of participants happening at a specific time and place [CV14], lie at the core of news reporting. Event mention detection is the task of finding spans in text which mention such events. The goal of event coreference resolution is to cluster these event mentions so that each cluster contains mentions referring to the same event. Cross-document event coreference resolution (CDCR) is an extension to multiple documents, producing coreference links within and across document boundaries. Knowing which text passages corefer boosts performance in multi-document downstream tasks such as question answering [Mor99; PIV18] and enables applications such as news timeline generation [Min+15]. CDCR datasets generally consist of a set of pre-clustered documents with added coreference annotations. We follow the terminology of Cybulska and Vossen [CV14] and define a subtopic as a cluster of documents reporting about the same event, for example “France beats Croatia to win the FIFA World Cup 2018”. To increase diversity, datasets may provide several subtopics from the same domain, referred to as a topic. This way, a second set of documents reporting about the event “Croatia beats England in the semifinal of the FIFA World Cup 2018” would form another subtopic. Together with the previous example, the two subtopics are part of the topic “football matches”. The event reported in a document which determines its (sub)topic is referred to as the seminal event [BH14].
Subtopic: Croatia beats Russia in Quarterfinal
Topic: Football Matches
Croatia vs.
England
Subtopic:
France beats Croatia in Final event mention cross-subtopic link within-subtopic link within-document link Croatia converted its last three chances to set up the showdown with England.
Croatia had done it the hard way to reach this final, requiring extra time to beat England in the semi-final.
In 2008, Bejan and Harabagiu [BH08] introduced the EventCorefBank as a dataset for CDCR which was later extended by Lee et al. [Lee+12]. Another augmentation e↵ort by Cybulska and Vossen [CV14] then led to the ECB+ dataset. With the goal of improving lexical diversity, they added a second set of documents focusing on a di↵erent seminal event for each topic, thereby introducing the notion of subtopics. The augmentation e↵ort of adding 502 documents took four months with two expert annotators. Despite the addition of further subtopics, the lexical variety is still low enough that a lemma-matching baseline is a strong contender [Upa+16].
As a consequence, Vossen et al. [Vos+18] developed the Gun Violence Corpus, a corpus consisting of a single topic and 241 subtopics to achieve high lexical diversity. The authors propose the data-to-text annotation methodology in which coreference links are established by linking event mention spans in multiple documents to the same entry in a knowledge base. The corpus was annotated by two annotators over the course of six weeks. 3
This section explains the principles of our annotation approach. The application of these principles for annotating the FCC follows in Section 4.
Annotation Unit. As mentioned previously, cross-subtopic event coreference links occur sparsely throughout a set of documents. We therefore propose to annotate event mentions entirely at the sentence level. In doing so, we trade mention span granularity for an increased density of the coreference phenomenon we seek to annotate, which in turn ensures an a↵ordable annotation. Another implication is that the annotation of sentences becomes a multi-label task, because a single sentence can mention multiple events. The definition of what triggers an event mention within a sentence remains a separate question and is una↵ected by the change of the annotation unit. A sentence mentioning an event may depend on participants, time or location mentioned elsewhere in the document. Therefore, annotating and resolving mentions requires the full document for context.
Annotation Objective. We frame the annotation task as a variation of event linking. Given a query sentence, several preceding sentences for context and a predefined set of events, annotators need to select the subset of events which are explicitly mentioned in the query sentence. In the end, all sentences linked to the same event e will form an event coreference cluster corresponding to e. Note that the set of events is mainly an aid for annotation and needs not to be provided to coreference resolution systems at training or test time.
Conceptually, the only assets required in our approach are a set of events and a loosely related set of documents. In domains where the set of to-be-annotated events is known prior to the annotation, such a set of events either already exists or is easy to construct. Furthermore, given the complexity of annotating cross-document event coreference, the opportunity of scaling up the annotation far outweighs the comparably small e↵ort of defining such a set.
Past work on event coreference annotation [BH08; Hov+13b; Vos+18] demonstrated that structuring events in a hierarchical fashion o↵ers benefits, for example by separating events that are distinct but conceptually related, which leads to more precise annotations if the hierarchy is relayed to human annotators. We therefore impose a hierarchical structure on the set of to-be-annotated events via the subevent relation. We follow the definitions of Hovy et al. [Hov+13b] and define that an event e1 is considered a subevent of another event e2 if its respective action, participants, time and location are subsets of the corresponding properties of e2. Note that the annotation of a list of unrelated events remains possible with our proposed approach, since a list of events can be reformulated as a flat hierarchy.
Annotation Workflow. For each document, we provide annotators with one predefined event hierarchy. Given a query sentence, we first ask annotators via binary yes/no question whether the sentence mentions events from the event hierarchy. If it does, annotators are supposed to select the subset of events from the hierarchy which are mentioned in the sentence explicitly or by a subevent.
Annotation Aggregation and Agreement. We aggregate the annotations of the binary question into a gold standard using MACE [Hov+13a]. We obtain a multi-label annotation for each sentence and each annotator for the event linking annotations. Based on this information, we need to find the gold set of events E⇤ which is most representative of all annotations.
Identifying this set is challenging because one needs to distinguish between cases where high variance in the annotations stems from disagreeing annotators or from a genuine case of a sentence deserving multiple labels. Conceptually, any mention of an event implicitly also functions as a mention for all its super events (i.e. its ancestors in the event hierarchy). We therefore require that no two events standing in an ancestor relation to each other in the event hierarchy may be present in the gold set of events of a sentence. We framed the search for the optimal set E⇤ as a constraint optimization problem based on this condition.
We compute inter-annotator agreement using Krippendor↵’s Alpha [Kri04]. Because our labels are sets of events, we follow the recommendations of Artstein and Poesio [AP08] and use the Jaccard index [Jac12] as a distance metric. 4
We applied our annotation approach to create the Football Coreference Corpus (FCC), the first corpus which focuses on cross-subtopic event coreference relations. Due to the sparsity of this event coreference phenomenon, one needs to draw a large sample of articles from the same topic to obtain suciently many and suciently large coreference clusters. We therefore decided to annotate documents from the sports domain, specifically match reports and related articles about football (soccer) tournaments. As we will explain in the following, our annotation approach can be implemented without requiring domain-specific annotation guidelines which boosts its generalizability. 4.1
We annotated news articles for five tournaments
The annotation e↵ort was distributed to crowdworkers on the Amazon Mechanical Turk platform. We developed an annotation interface implementing the approach outlined in Section 3 (see figure 2). Apart from the query sentence, crowdworkers were additionally shown several preceding sentences, the article headline and its publication date to establish the document context. If the seminal event of a document was part of the event hierarchy, we additionally displayed the properties of this event (action, participants, location, date).We kept textual explanations to a minimum and instead primed annotators through a series of examples on how to complete the assignments correctly. In particular, we did not specify rules on what defines an event mention but relied on the annotators’ commonsense to make this decision. The only domain-dependent elements in our annotation setup are the examples (which were chosen from the same domain as the input documents) and minor help texts explaining the process of football tournaments. 4.2
In order to ensure the validity of our study and to determine the minimum number of crowd annotators that provide sucient quality, we compared crowdworker annotations to those of experts. 121 sentences from three randomly picked documents were independently annotated by 8–9 crowdworkers and two NLP experts. Between the two experts, an agreement of 0.76 Krippendor↵’s Alpha was reached. We manually created a gold standard from the expert annotations. We bootstrapped the crowdworker annotations to simulate annotation studies with 2 to 8 crowdworkers per instance. Each bootstrapped set of crowdworker annotations was aggregated and compared to the experts’ work. Figure 3 shows the resulting agreement distributions for the three tasks in our annotation scheme (mention detection for seminal events, mention detection for other events, annotating the set of referenced events). With five crowd annotators, we reach a mean Krippendor↵’s Alpha of 0.734, 0.719 and 0.677, allowing tentative conclusions [Car96]. This constitutes a suitable tradeo↵ between quality and costs, leading us to annotate the main portion of the dataset with five crowd annotators. 4.3
Using time measurements from pre-studies as a reference point, we paid annotators according to the US minimum wage of $7.25/hour. Overall, 451 documents were annotated over the course of three weeks, amounting to $2800 in total.
Table 1 shows the properties of our dataset alongside a comparison to existing CDCR datasets. The contrast in the overall number of annotated events is a result of di↵erent annotation strategies: In ECB+, all mentions of a document’s seminal event as well as any other events mentioned in the same sentence were annotated. For the Gun Violence Corpus (GVC), only mentions of a given seminal event and a predefined set of its subevents were annotated. In our dataset, 311 coarse-grained events were available for annotation across all event hierarchies. In the end, 217 of these events were annotated by crowd annotators. Given their higher granularity compared to those annotated in ECB+ and the GVC, these events are less frequent by nature. Most notably however, our proposed annotation scheme resulted in a dataset with a large number of cross-subtopic event coreference clusters. While the annotation of this type of coreference is technically possible with traditional token-level annotation schemes, ours is markedly faster and does not require complex or domain-dependent annotation guidelines or trained annotators, which to the best of our knowledge makes it the first scalable technique for annotating cross-subtopic event coreference links.
Table 2 shows exemplary results from our annotation. A strong point of our approach relying on commonsense
is that crowdworkers also linked future events and expressions with quantifiers triggering multiple events (“three
previous matches”) without us having to provide detailed annotation guidelines on how to handle these cases.
We manually analyzed a number of annotated documents. In some cases, multiple events are mentioned via
non-countable quantifiers (“few”, “every”, “more than two”, etc.). This caused annotators to agree on the
presence of a mention but caused disagreement for the linking step. In case annotators reach no consensus with
respect to the set of mentioned events, the ⌧ parameter in our aggregation returns an empty gold set of events.
Some sentences cause disagreement because an event mention leaves room for interpretation as to which event is
being referenced. For example,
We are, to the best of our knowledge, the first to tackle cross-subtopic event coreference, a salient but rarely occurring coreference phenomenon which is underrepresented in other datasets. To capture these links a↵ordably and with sucient density in text, we developed a novel sentence-level crowdsourcing annotation scheme, which produces reliable results when compared to NLP experts. We created the Football Coreference Corpus (FCC), the first CDCR corpus specifically targeting cross-subtopic event coreference which consists of 451 football news reports. Our work o↵ers several possibilities for follow-up work: Since our proposed annotation scheme does not require domain-specific annotation guidelines, future work may add further topics with relative ease.
The authors would like to thank the anonymous reviewers for their helpful insights. This work was supported by the German Research Foundation under grant №GU 798/17-1. [AP08]
Cosmin Adrian Bejan and Sanda Harabagiu. “Unsupervised event coreference resolution”. In: Computational Linguistics 40.2 (2014), pp. 311–347. doi: 10.1162/COLI\_a\_00174.
Steven Bird, Ewan Klein, and Edward Loper. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O’Reilly Media, 2009. [CV14] Agata Cybulska and Piek Vossen. “Using a Sledgehammer to Crack a Nut? Lexical Diversity and Event Coreference Resolution”. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14). Reykjavik, Iceland: European Language Resources Association (ELRA), May 2014. isbn: 978-2-9517408-8-4.