News agencies and other news providers or consumers are confronted with the task of extracting events from news articles. This is done i) either to monitor and, hence, to be informed about events of speci c kinds over time and/or ii) to react to events immediately. In the past, several promising approaches to extracting events from text have been proposed. Besides purely statistically-based approaches there are methods to represent events in a semantically-structured form, such as graphs containing actions (predicates), participants (entities), etc. However, it turns out to be very di cult to automatically determine whether an event is real or not. In this paper, we give an overview of approaches which proposed solutions for this research problem. We show that there is no gold standard dataset where real events are annotated in text documents in a ne-grained, semantically-enriched way. We present a methodology of creating such a dataset with the help of crowdsourcing and present preliminary results.
News agencies and other digital media publishers publish each day news articles in the magnitude of dozens of thousands. They also process the news for further business tasks such as trend prediction and market change detection. This is still mainly done manually today. Even if knowledge workers at news agencies have access to all this information, it is infeasible for them to read all the news and to determine, whether the articles contain information which is not only interesting for people in their domains, but which contain real events and, hence, have a signi cant, immediate impact on business such as nancial operations (shares) and political happenings. Consider for example the rst sentence of a news article: \Apple may acquire Beats Electronics next week." (1) ? This work was carried out with the support of the German Federal Ministry of Education and Research (BMBF) within the Software Campus project SUITE (Grant 01IS12051).
Here, it remains unclear whether Apple is really going to acquire Beats (and does not cancel it in the last minute) or whether this is just a rumor. The sentence \Apple con rmed that it acquired Beats Electronics on Wednesday." (2) in contrary, reveals that the acquisition already happened (besides the con rmation which is an event per se). This demonstrates the di erentiating characteristic between real events and events in general. As humans we can estimate that the rst article is not a trigger for immediate shifts in the stock market (besides psychological e ects), but maybe the second mentioned article. Machines, in contrast, have their di culties in distinguishing real events from other events.
We envision building a decision support tool for agents like stockbrokers. The aim of the system is to inform the user quickly and automatically when some detected event has really happened and hence might in uence the invested assets of the user. The user should also have the possibility to store purely real events in his database. For such purposes, an event extraction system would consist of two steps: i) It extracts events in a structured, semantically enriched representation and ii) determines based on linguistic cues whether the event is real or not.
Research on real event detection has been very limited so far. In this paper, we present an approach to de ne events and real events in a setting as described. Since no suitable gold standard for evaluating a real event detection system exists, we present our setting of creating one using crowdsourcing. Preliminary results regarding this gold standard are presented, as well as challenges which we came across.
The remainder of this paper is organized as follows: First we present de nitions of event detection in Section 2, before considering de nitions of real event detection in Section 3. After discussing our setup of creating a gold standard for real event detection in Section 4, we conclude in Section 5. 2 2.1
We can distinguish between the following classes of event representation (see also Fig. 1 for examples): 1. Something happened : In this event representation, events are only roughly covered. There are no types and deeper meanings gathered, only what topic the document/sentence is about. This topic is often characterized by the words occurring in the document (bag-of-words model) and/or by the set of recognized named entities. 2. This happened : For this representation, the event type of the event is detected. The event type can be quite generic such as earthquake. The number of events which can be detected is often very limited. Events may
Wednesday
Beats acquired Electronics
Apple
confirmed (a) Event Representation Class 1 "Wednesday"
Event type Participant Participant (b) Event Representation Class 2 :time :subevent :acquire
:agent :confirm :agent
:Beats Electronics :patient
:Apple Inc.
(c) Event Representation Class 3 have attributes or slots which are pre-de ned for the single event types. Instead of prede ned entity types such as earthquake or accident sometimes only the entity types Per, Loc, Org, and Misc are used. 3. This happened to these objects in this way : If we use this representation format, we have a deeper understanding in the actual event. Events of this class are quite speci c and include not only speci c actions, but also participants, and maybe time, place, and manner of the action. Often linguistic theories such as Semantic Role Labeling provide the basis for event representations of this class.
Related work using event de nitions corresponding to the rst event
representation class do not de ne events at all [
Approaches using the second event de nition have in common that
coarse-grained events such as accidents or earthquakes are represented. Each
event has therefore an event type. Property-value-pairs can be assigned to the
events, whereas the assignable properties are pre-de ned for all event types.
Often templates are used for storing the information about events [
In case of event representations of the third kind, structural representations
of ne-grained events are extracted from text { here, typically from single
sentences or clauses. Research based on this event class usually does not
introduce a new de nition of events, but instead either uses linguistic de nitions
of events where events consist of happenings with agents, locations, time, etc.
[
In this paper we focus on the detection and semantically-structured representation of real events of the third-mentioned event class, which is the most expensive one. More speci cally, an event in our scenario is characterized by { speci c participants (agents or objects) { situations (events or states) which are described within the event { taking place at a speci c place and/or time { being not a state.
States are hereby de ned as lasting for an inde nite period of time and which are not really observable. Given the example sentence 2 in Section 1 we can extract two events from it: i) The event that Apple con rmed something (which is an event itself) and ii) the event that Apple acquired Beats Electronics.
Fig. 1c shows how these events can be represented as a semantically-structured graph. Hereby, Event ii) can either be part of Event i) (as depicted in the gure) or be stored as a separate graph. Nodes in each event graph can be either predicate nodes (representing actions), entity nodes (representing participants), or literal nodes (representing the time, etc.). Predicate and entity nodes can be linked to entries in knowledge bases such as DBpedia (for entities) and WordNet (for predicates). This enables having unique identi ers for resources and to resolve ambiguities. The edges in these event graphs arise from the semantic roles assigned by a Semantic Role Labeling tool. In the depicted gure, the semantic roles are grounded as RDF predicates.
We de ne real event detection as the task of determining whether a given event expressed in text is real. Real events are events according to the de nition in Section 2.2 and have already happened or are happening. Thus, the de nition of events is extended by this aspect. We can split the task of real event detection therefore into two subtasks: 1. Determining if the situation described in the text is about an event according to our de nition. 2. Determining if the event already happened or is currently happening.
Regarding the rst subtask, we can refer to two areas of linguistic work:
i) The distinction of events from states, and ii) the identi cation of factuality
of events. In the following, we amplify these two areas with respect to our goal
of real event detection. We hereby use the term situation as a generic concept
which encompasses both event and state (cf. [
Ad i) The classi cation of situations can be traced back to Aristotle who
distinguished between verbs that have a de ned end or result, and others that
do not [
To determine which aspectual class a
given situation belongs to, we can di er
between telic, dynamic, and durative Table 1: Vendler's four-way
situations (see Table 1). Telic situations distinction between verbs based
always have a culmination point beyond on their aspectual features [
Dynamic situations consist of internal sub-events which change over time and are, hence, intrinsically heterogeneous.
For instance, walking consists of several alternating subevents. Durative situations (e.g., eating) last for a speci able period in time and are not punctual.
In our case we want to distinguish events from states. But how can we
determine which aspectual class holds for a given situation? For Vendler [
Class Telic Dynamic Durative state - - X activity - X X accomplishment X X X achievement X X
Moens and Steedman [
Ad ii) Other researchers have focused on determining the factuality of events,
i.e. to recognize whether events are presented in the sentences as corresponding
to real situations in the world, as situations that have not happened, or as
situations of uncertain status. The focus is, hence, the trustfulness of events in
text. Factuality can be characterized by two dimensions: Polarity and epistemic
modality. Polarity { more concrete: polarity on actuality and not subjective
polarity { is a discrete category and can be either positive or negative. Epistemic
modality, in contrast, expresses the speaker's degree of commitment to the
truth of a proposition [
How is factuality expressed in the text? This is done by lexical markers as well as syntactic markers. Lexical modal markers are modal auxiliaries (e.g., \could", \may", \must"), as well as clausal/sentential adverbial modi ers (e.g., \maybe", \likely", \possibly"). Examples of lexical polarity markers are adverbs (e.g., \not", \until"), quanti ers (e.g., \no", \none"), and pronouns (e.g., \nobody"). Syntactic constructs are necessary to consider since often one clause is embedded in another. Considerable are in this context especially relative clauses and that-clauses as in the example sentences.
What are the challenges to determine the factuality? Factuality markers
interact with each other. The local modality and polarity operators (e.g., of
the current clause) are therefore not enough. Instead, a global consideration is
necessary. For instance, in case of that-clauses, the factuality of the inner event
is dependent on the factuality of the outer event. Furthermore, what makes the
factuality much more complex is the fact that the source of an event is often not
only the author. These additional sources are introduced by means of predicates
of reporting (such as \say" or \tell"), knowledge and opinion (such as \believe",
\know"), psychological reaction (such as \regret"), etc. Sauri and Pustejovsky
[
According to our event de nition in Section 2.2 and the additional aspect of factuality addressed in Section 3.1 we can list the following requirements a gold standard dataset for the evaluation of a real event detection system must ful ll: 1. Each mention of an action within an event (e.g., \wrote") is annotated. 2. There is a distinction between events and states, so that all events in the strict sense are annotated. 3. There is no restriction to speci c event types. 4. The factuality of the event is annotated (being positive or negative). 5. All participants and participating objects are annotated. 6. All participants and participating objects are linked to prevalent knowledge bases. 7. Subevents of events are annotated and linked. 8. Mentions of place and time of each event are annotated.
This gold standard is also suitable when it comes to extracting real events according to the Event Representation Classes 1 and 2 (see Section 2.1). In these cases, the information about the structural representation of events can be neglected. Additional ltering can achieve that only events of speci c types such as accidents are detected. 3.3
In the following, we review existing corpora where event factuality was annotated to some degree.
The Multi-Perspective Question Answering (MPQA) corpus [
The Penn Discourse TreeBank (PDTB) [
TimeBank 1.2 [
FactBank [
ACE 2005 [
In summary, we can state that none of the mentioned corpora contains semantically-structured representations of events to the extent it is needed to evaluate a real event detection system where events are de ned as in Section 2.2. Thus, in the following section we provide experiments on how to build a gold standard which ful lls all our requirements. 4
Very rst crowdsourcing experiments revealed that letting users annotate real events as described in Section 3.2 at once is too complex for any crowdsourcing job. Therefore, we arranged subtasks where the following questions are answered separately for each event: 1. Which are the actions/predicates inducing a real event? 2. Which are the participating objects? 3. What is the time and place? 4. Which sub-events are contained? In the following we present our approach regarding the rst subtask, namely identifying real events and naming the central predicates of them. We performed two crowdsourcing jobs which di er in their methodology.1
Run 1 The crowd was asked to read a given sentence, to look for real events (as de ned above), and to enter the action verbs of these events as written in the sentence. 1 The crowdsourcing job descriptions and evaluation data is available online at http: //www.aifb.kit.edu/web/Toward_Real_Event_Detection Run 1: "Find real actions" 187 sentences, 8 test questions, 12¢ per task, 5 users per judgment
Our gold standard: 205 verbs inducing real events 224 verbs judged by crowd as inducing real events 152/224 (67.9%) of verbs judged as inducing real events are correct
Run 2: "Find observable and non-observable predicates" 187 sentences, 9 test questions, 12¢per task, 5 users per judgment
Our gold standard: 205 action verbs 354 observable 185 non-observable predicates predicates 133/205 (64.9%) of predicates judged as observable are corecct 285/334 (85.3%) of predicates judged as non-observable are correct 205 verbs classified by crowd as observable 334 predicates classified by crowd as non-observable
Run 2 For this second run, the crowd was asked to read each given sentence, look for all verbs, and categorize them into either observable or not-observable.
Observable events/facts were de ned as follows:2 An observable fact can be an occurrence (e.g., "arrive\, "destroy\), a reporting (e.g., "report\), or an immediate action (e.g., "approve\). Observable facts are characterized by the fact that they could be observed or con rmed by third persons directly (e.g., in case of "say\) or indirectly (e.g., in case of "con rm\). Non-observable facts describe states which characterize persons or objects, but which are not observable by other persons than the persons involved. Such non-observable facts are states which last for an inde nite/unspeci ed period of time (e.g., "be happy\), immediate states (e.g., "believe\, "worried\), aspects (e.g., "start\, "continue\), or perceptions (e.g., "feel\). The categorization into observable vs. non-observable facts is here done independently of the fact whether the event has happened (or the state is) for sure or not. The categorization into the past/presence or future is performed in a separate crowdsourcing task.
As dataset we used all rst sentences of news articles which were published on one day (2014/05/28) by the news agency Bloomberg and where the news articles contained some information about Apple Inc. In total we manually annotated 187 sentences to assess the performance of our crowdsourcing tasks. Crowd sourcing was performed on the platform Crowd ower.3 In Run 1 (Run 2), users had to answer 8 (9) quiz test questions before entering the actual task. In both runs, users got 12 cent per task consisting of 4 questions each. For each question we gained results from 5 users and took the answers where there was an inter-rater agreement of at least 50%.
The results of our crowdsourcing annotation experiments are summarized in Fig. 2. It became apparent that completing the crowdsourcing tasks requires high cognitive e orts in comparison to other crowdsourcing tasks. A considerable amount of users did not pass the test questions at the beginning. Even if we 2 The de nition is based on the TimeBank annotation guidelines. 3 http://crowdflower.com admit only users who worked on our job in the past su ciently well, creating a big annotated corpus is tricky. As Run 2 shows, already the distinction between observable events, i.e. events showing up in the real world, and not-observable events is hard to perform. Although we put much e ort in re ning the task descriptions the question arises whether a better approach to annotating the factuality of events is achievable. 5
If events are extracted from text in a ne-grained manner, huge amounts of events are gathered, but only a fraction of them represent real events and, hence, are worthwhile to process further on. In this paper, we gave an overview of existing linguistic work about the detection of real events. In order to evaluate a proposed system which extracts semantically-structured, real events from text, we de ned requirements and proposed a methodology to create a gold standard dataset. Preliminary experiments with crowdsourcing showed that the annotation of text with factual information is non-trivial. Still, we believe that the creation of such a dataset is necessary for many event detection systems in the future.