We present an analysis of a high-level semantic task, the construction of cross-document event timelines from SemEval 2015 Task 4: TimeLine, to trace down errors to the components of our pipeline system. Event timeline extraction requires many di erent Natural Language Processing tasks among which entity and event detection, coreference resolution and semantic-role-labeling are pivotal. These tasks yet depend on other low-level analysis. This paper shows where errors come from and whether they are propagated through the di erent layers. We also show that performance of each of the subtasks is still insu cient for the complex task considered. Finally, we observe that there is not enough semantics and inferencing within the standard NLP techniques to perform well.
Textual interpretation requires many analyses ranging from tokenization, PoStagging to high-level semantic tasks such as entity-detection or semantic-rolelabeling (SRL). In Natural Language Processing (NLP), such tasks are usually considered in isolation. However, the output of low-level analyses is used to achieve higher levels of interpretations and it is well-known that their errors propagate to higher levels. Furthermore, di erent modules may make incoherent claims on the same text elements. A Named-Entity Recognition and Classi cation (NERC) system may say that Ford is an entity of the type person, whereas the entity linker nds the URI to the company and the SRL module assigns the role of vehicle. Typical NLP architectures allow dependencies in one direction only and do not have any mechanism to reconcile inconsistency. Therefore most high-level modules are not designed to detect inconsistency and simply ignore competing results from other modules. Especially when high levels of semantic processing and interpretation are the goal, it becomes very di cult to relate the performance of the system to the performance of each of the sub-modules and to improve it.
In this paper, we present an error analysis of a complex semantic task: the SemEval 2015 Task 4: TimeLine: Cross-Document Event Ordering1 to learn more about the dependencies between modules and missed opportunities. The most relevant NLP sub-tasks for timeline extraction are entity and event detection, coreference resolution, semantic-role-labeling and time expression recognition and normalization. Timelines are created by combining the results of 10 modules carrying out subtasks. This pipeline is an excellent case for demonstrating the complexity of the process and the (non-)dependencies between the modules. Our analysis starts from the errors creating the timelines and drills down to lower levels that provide the necessary elements. We show that error propagation from lower levels occurs, but its impact remains limited. Errors from high level tasks piling up form the main cause of overall low performance. Our analyses reveal that several errors can be avoided if information from various modules is integrated better. However, only full comprehension of the context could yield high results on this task.
Our study di ers from previous work in that we quantify error dependencies
and shortcomings of a pipeline system rather than providing users with tools for
error tracking or component integration. To the best of our knowledge, the most
similar work we have identi ed in literature is Clark et al. (2007) [
The paper is further structured as follows. In Section 2 we describe the task and the NLP pipeline that we used to participate. We give a detailed trace-back analysis for the errors of our system in Section 3. We discuss these errors further in Section 4 after which we conclude in Section 5. 2
The SemEval 2015 Task 4: TimeLine: Cross-Document Event Ordering aims
at identifying all events in a set of documents where a given entity plays the
PropBank [
Figure 1 illustrates our pipeline indicating the dependencies among the di erent modules.2 The pipeline modules mostly represent state-of-the-art systems, some developed by third parties, and integrated in the NewsReader project.3 The rest of this section describes the main NLP modules of our system. 2.1
Event extraction and entity-linking is based on the output of the SRL module of
our pipeline. The predicates identi ed by the SRL module form candidate events.
1 http://alt.qcri.org/semeval2015/task4/
2 A detailed description can be found in Agerri et al. (2014) [
The SRL module we use is based on the MATE-tools [
The next step is to identify which events have an Arg0 or Arg1 that corresponds to the target entity of a timeline. Three modules are involved in this sub-task: a) NERC to identify and classify names, b) a NED to disambiguate named entities by means of DBpedia URIs and c) coreference resolution to identify which other expressions in the text have the same referent.
The NERC module is a supervised machine learning system trained on the
CoNLL 2002 and 2003 shared tasks [
The coreference resolution module is a reimplementation of Stanford's Multi
Sieve Pass system [
The event coreference module uses the predicates from the SRL layer. Predicates
with the same lemma in the same document are considered coreferential as well
as predicates with high similarity in WordNet [
We use FBK TimePro to identify time expressions in text. This machine learning
system is trained on TempEval3 data [20]. Time expressions are consequently
normalized using the timenorm library [
Each of the above modules generates an independent stand-o layer of
annotation in the NLP Annotation Format (NAF, [
Overall, performance is very low and varies a lot across the 4 entities. Although it is a very di cult task, a deep error analysis is required to nd the main causes for this performance, which will be presented in the next section. 3
To get more insight in the problems for this task, we want to answer the following questions: { Which modules are responsible for most errors? { How are errors propagated to other modules in the pipeline? { To what extent do di erent modules make di erent statements on the same interpretation?
All of our modules have been benchmarked against standard datasets obtaining state-of-the-art performances.5 However, what these gures do not tell us is how errors propagate and whether a certain number of errors in di erent modules make it impossible to derive the high-level structure of a timeline.
We perform an in-depth error analysis of four target entities (Airbus,
Airbus380, Boeing and Boeing 777) by reversing the steps in the pipeline to nd the
module that breaks the result. We used false negatives (FN) and false positives
(FP) errors as starting point.
4 More details on the system can be found in Caselli et al. (2015) [
The analysis of events involved a total of 178 events for the FNs of all four target entities and 209 events for the FPs of Airbus and Airbus A380. These two entities are paradigmatic cases, i.e., the set of detected errors will apply to the other entities as well.
We identi ed 21 error types: 9 error types are due to speci c annotation layers (e.g. SRL or Event detection), 9 are due to a combination of annotation layers (e.g. SRL + Entity coreference), and 3 are not related to the pipeline (e.g. Gold data or spelling errors). Table 2 provides an overview of the errors.
False Negatives: The analysis has highlighted three major sources of errors all related to high-level semantic tasks, namely Event detection (39 events); Entity coreference (36 events) and SRL (23 events).
Error propagation from low-level annotation layer a ects event detection for only 8 cases (i.e. wrong PoS tagging). The majority of cases is directly a ected by the SRL layer with 11 instances of partial detection (i.e. multi-tokens events) and 24 cases of events realized by nouns.
Entity coreference a ects event extraction indirectly as the target events are detected but they cannot be connected to the target entities.
Finally, the SRL module mainly introduces two types of errors: wrong role assignment or implicit arguments in the predicate structure. a It contains 1 case due to wrong PoS tagging. b It contains 2 cases of partial detection and 1 case due to wrong PoS tagging. c It contains 6 cases of partial detection and 3 cases due to wrong PoS tagging. d It contains 3 cases of partial detection and 3 cases due to wrong PoS tagging. e All cases include also errors with Final Entity Extraction.
False Positives: These errors point to issues related to over-generation of events and of event entity linking. Most errors concern Event coreference (39 cases, plus 10 cases in combination with other modules) and Event detection (15 cases, plus 43 in combination with other modules).
The event coreference errors point to both a limit of the speci c annotation layer and to a propagation of errors from the SRL module.
The SRL module responsible for event detection does not only miss most
nominal events, it also overgenerates them. This is due to the implementation
of the SRL module which labels all entries in NomBank [
The Final Entity Extraction refers to errors in entity matching for the nal timeline. The high number of errors for Airbus A380 (77) is caused by the substring method used as a back-o strategy. 3.2
We analyzed the SRL for 60 events in the gold timeline for the entity Airbus. Table 3 provides an overview of our ndings.
Events in gold timeline (gold events) 60 Gold events in the SRL layer 54 Gold events in out-of-competiton timeline 25 Gold events not in out-of-competiton timeline 29 Correct SRL assignment in SRL layer 41 Correct SRL assignment not in out-of-competiton timeline 19 Correct SRL assignment in out-of-competiton timeline 22 Incorrect SRL assignment in SRL layer 15 Incorrect SRL assignment not in out-of-competiton timeline 10 Incorrect SRL assignment in out-of-competiton timeline 5
Events in out-of-competiton timeline 103
The SRL layer contained 54 out of 60 gold events. From these 54 events, 41 had a correct SRL assignment. However, only 22 out of these 41 events ended up in the system's timeline. We distinguish two groups among the 19 cases of correct SRL that did not appear in the timeline:
NED system failed to link the named entity Airbus to the correct DBpedia URI. In 2 cases, Airbus was not recognized as the head of the NP. In the other 5 cases, the error is caused by the coreference module.
that are not instantiated in the clause, but can be recovered from the context. In 4 cases, the role could be inferred from information in the same sentence, and 4 cases required previous context. These 8 errors clearly originate in the SRL layer but are not errors of the SRL module, since the SRL assignment is correct according to the state-of-the-art tools. 3.3
Named entities can be mentioned explicitly in text or implicitly through coreferential chains. We analyzed both cases by inspecting the entities which appear in the roles Arg0 and Arg1 of the event predicates. Explicitly mentioned named entities are analyzed using the following steps: 1. Did disambiguation (NED) fail or recognition (NERC)? 2. If disambiguation failed, is the entity recognition incorrect? This includes both missed entities and incorrect spans (e.g. Airbus to instead of Airbus) 3. If recognition failed, are there problems with the PoS, lemmas or tokens?
For each coreferencial mention, we traced back the correct coreference chain(s).
We identi ed two types of errors: missing coreference link and wrong coreference
link. For each case, we also inspected whether the coreferential problem is caused
by a wrong PoS tag. Overall, 16 cases were identi ed where the anaphoric
mention was realized by a pronoun not linked to the correct antecedent. In 20 cases,
the anaphoric element was realized by a Full De nite Noun Phrase (FDNP) of
the form \the + (modi er) + noun". An in-depth analysis of these cases, in
the line of Poesio and Vieira (1998) [
Results are reported in Table 4. The numbers for coreference relations only concern direct links between a Named Entity and an anaphoric mention. Cases of implicit arguments, or zero anaphora, are not reported in the table. Normalized time expressions are needed to anchor and order events on the timeline. We analyzed the system's performance on the events in the gold timelines as well as the events identi ed by our pipeline for Airbus and Airbus 380.
The gold standard of Airbus and Airbus A380 contains 103 events that need to be linked to a speci c date. The system has no mechanism for disambiguation and extracted 208 links. There were 21 extracted events linked to a speci c date and 91 links found. Unsurprisingly, time anchoring has decent recall and low precision: Recall was 59.2% on gold events and 71.4% on the extracted events. Precision was 30.5% on gold and only 16.4% on the extracted set.
Table 5 provides an overview of the outcome of our analysis. The rst column provides the number of correctly identi ed dates. We found ve di erent errors and a handful of cases where the date should be identi ed by event coreference (i.e. not a time anchoring error). The rst two errors are related to identifying the correct date. FBK TimePro identi ed the wrong span leading to 32 errors. These errors occur despite the constituents being correct. The error propagates up to timenorm which provides the wrong interpretation of the date. Timenorm furthermore systematically places underspeci ed dates in the past, so that all future events are placed in the wrong year (37 errors).
The next two errors concern the correct link between a date and an event. Finally, there are interpretation errors, where event and date are indeed connected, but the relation is not an anchoring one (but, e.g., a preceding relation). Among them, 12 cases can only be solved by world knowledge or interpretation of context. Multiple errors can occur around the same date or event, e.g., a date may be placed in the wrong year and be wrongly linked to the event.
Overall, the system is quite good at identifying time expressions and interpreting them. The main challenge lies in linking events with the correct date. This is also re ected by 13 cases where the Gold Standard seemed incorrect. With everything piling up, our system performs much worse with only 15 out of 108 events identi ed and anchored correctly. 4
The analyses show that error propagation is an issue which mainly starts from high-level tasks such as entity detection and linking, event detection and SRL. The impact of error propagation from low-level layers appears to be minimal in our data. For example, we noted that the PoS-tagger correctly detects proper nouns, but the NERC module still fails.
Another conclusion is that modules require more semantic analysis and inferencing. This speci cally applies to coreference and implicit arguments of events. Possibly, this requires richer semantic resources and broadening the context of analysis rather than limiting it to local and structural properties. In many cases, participants of events are mentioned in other sentences than the one where the event occurs, which is missed by sentence-based SRL. Semantics can also exploit word-sense-disambiguation (WSD) for coreference and event detection. For event coreference, the highest similarity is currently taken across all meanings of predicates, connecting the same predicates in di erent meanings across the text. Restricting the similarity to dominant meanings appears to eleminate false positives of event coreference links with higher f-measures. The opposite holds for nominal coreference that fails to nd bridging relations across NPs and entities (false negatives) due to lack of semantic connectivity. Widening reference to capture metonymy, e.g. Airbus A380 implies Airbus, requires more elaborate semantic modeling. With respect to event detection, WSD can eliminate false positive nominals that are not events in the dominant meaning.
Finally, temporal processing must be improved, with a particular focus on
temporal anchoring. Time expressions are sparse, but they are the most
reliable source of information for anchoring events and provide partial ordering.
Additional research based on the notion of temporal containers [
We provided a detailed error analysis of a NLP pipeline system that took part in the SemEval 2015 Task 4: TimeLine. We have shown how the system's poor performance is in part due to errors in di erent modules of the pipeline (e.g. SRL, NERC and NED) and partially to lack of rich semantic models in current state-of-the-art systems.
An interesting aspect is the relatively low error propagation from low-levels to
high-levels. This shows that most basic tasks such as tokenization, PoS-tagging
and lemmatization achieve reliable performance and could almost be considered
as solved. We say almost, because errors in these layers do propagate to high-level
layers (also see Manning (2011) [
Semantic tasks, such as NERC, NED, event detection, SRL and temporal processing, are far from being solved. The mismatch in performance between benchmark datasets and the task points out that more research and new solutions are required. We found evidence that some sub-tasks are deeply interrelated (e.g. event detection and SRL; NERC, NED and entity coreference) and require better integration to boost performance. Furthermore, some errors can only be avoided by rich semantic models interpreting context.
The complex task of timeline construction will rst of all greatly bene t from dedicated modules for event detection: the presence of an argument structure does not necessarily correlate with event readings. Secondly, coreference resolution needs to be improved through more elaborate semantic modeling. Finally, no module uses the output of the WSD module despite evidence that a more optimal selection of senses will help their task. Future work will concentrate on these pending issues. 20. UzZaman, N., Llorens, H., Allen, J., Derczynski, L., Verhagen, M., Pustejovsky, J.: SemEval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations. In: Proceedings of SemEval-2013. pp. 1{9 (2013)