Introduction

Event extraction is one of the most important tasks of knowledge discovery. It may be regarded as the core of knowledge-based systems that aim at providing the public (people, organizations, government agenda etc.) with condensed and filtered information concerning events. These events are described in texts written in natural language, thus the posted problem is related to the issue of information extraction (IE). Particularly, the task is to extract data concerning the described action (the event) and its arguments (called event roles). To implement the considered task different approaches are applied. They can be classified according to the provenance of the approach (pattern-based linguistic ones vs. classifier-based (statistical) methods) or to the 'openness' of it (fully open extraction vs. trained with the use of corpora one). The next important classification criterion is the nature of the context of the extraction, namely locality (one sentence only) or a larger context that takes into account consecutive sentences (a discourse). In many cases the hybrid methods are used that combine the different approaches. The open extraction systems (operating across one sentence context) scale well to the open corpora [1,3], especially that acquitted from the Web. But the most accurate IE systems are domain-specific, that use linguistic patterns and are somewhat trained with the aid of statistics. Our work follows the latter approach in that we use a training domain-specific corpus. Let us characterize it briefly.

Due to a series of DARPA Message Understanding Conferences (MUCs), significant progress in pattern-based (NLP based) extraction technologies has been achieved. In this work we capitalise on the results of MUC-3 and MUC-4 ( [10] that were held in 1991-1992) conferences, which used news reports corpus (MUC corpus) on terrorist activities in Latin America. MUC Conferences developed standards for evaluation, e.g. the adoption of metrics like precision and recall.

The goal of MUC was to extract from texts an information concerning 7 classes of terrorist events: Attack, Kidnapping, Hijacking, Bombing, Arson, Robbery and Forced Work Stoppage, plus several variations on each (for accomplished, threatened and attempted incidents). The process of extraction was augmented by the knowledge frames (event templates) generation. Every such template consisted of 24 attributesslots. A document (a multi-sentenced message concerning an event) could be labeled with more than one template type. The MUC-4 corpus consists of 1700 documents, from which 1300 (DEV) were used in MUC-4 for training, 200 documents (TST1+TST2) were used as a tuning set, and the last 200 documents (TST3+TST4) were applied as the test set. The resulting knowledge base frames are called "key templates". We filter out messages concerning one event type only, namely the kidnapping. Also, from among 24 slots we consider the two of them: a perpetrator and a person_target.

The main contributions of the presented paper are:  a method of comparing events to check whether a given two events are in fact identical or whether they are different, on the basis of semantic typing (semantic classes) of event's arguments; it relies on using several types of rules, namely atomic, filling thematic role rules and whole events comparing rules; the method may be also used in coreference resolution  an implementation of a corpus crawling tool that looks for words/phrases that lexicalize the kidnapping event  additional lexical rules related to identification of victims and perpetrators.

The paper is organised as follows. Section 2 contains some notes concerning related works. In Section 3 our extraction method is presented. Section 4 describes a prototype implementation of the Word-statistics tool and its use. Section 5 demonstrates our information extraction results. In section 6 we give the concluding remarks and mention on our future work.

2

Related works

The main drawback of open information extraction [3] is that it uses the natural language features which do not classify (semantically type) arguments of an extracted relation. Additionally, in such methods the syntactic patterns (for example, regular expressions) do not match verb arguments that are distant from the verb phrase in a sentence. These are the features having the great negative impact on the ability to compare events (whether they are identical or not) described in the different sentences. In our work we avoid this drawback.

Authors of [4] use the language resources (dictionaries) to obtain sets of words that are relevant to the semantic class (a type of a verb argument). Having such extensionally defined types (semantic classes) they use them in the extraction process. In this work it is also shown how to apply such classes in the process of events comparison.

The method of event's comparison is also described in [5]. Here, the authors compare them (and extract their arguments) on the basis of head parts of noun phrases. For example, the events described in the following two sentences:

1) A customer in the store was shot by masked men.

2) The two men used 9mm semi-automatic pistols. are in fact the same due to the fact that they use the same word "men". In our approach the events may be unified (or differentiated) on the basis of the membership (non-membership) of two used ("linking") words to the same semantic class. Also, it is not known, which pairs of sentences should be analyzed according to the event (we describe this problem later on).

3

The extraction method

Preliminary definitions

At first, let us give some definitions of the terms used in the paper. They are as follows.

Event (denoted by E n , where n stands for event's name) is an entity representing the event (conceptually it is an occurrent that plays the central role in some situation, which represents a state of affairs) described in the text. The event is connected with a syntactic phrase (a verb phrase) that helps to identify it in a sentence, which is called an anchor. Also, there are some participants in the event  we identify them via thematic roles that are arguments of an anchoring phrase. Anchor (marked as A k , where k stands for an anchor name) is a verb or a verb phrase, which appearance in a derivation (i.e. a syntactically parsed sentence) triggers the process of recognition of an event (such as, for example, the kidnapping).

Thematic role (a semantic role label, marked as R m , where m is a role name) is an entity representing an argument of a verb or a verb phrase (an anchor) denoting the event. For example, there may be such roles as Agent (in our considerations, a perpetrator), Patient (a victim), Instrument, Location, Time and others.

Role filler is a text phrase that instantiates a thematic role in the text (marked with the symbol R p F v , where p is a role name and v identifies a filler).

Syntactic similarity. Let us assume that the two argument function of syntactic similarity simsyn (W 1 , W 2 ), while given two words (or phrases) as arguments returns a binary value true or false. The function will return the true value if W 1 and W 2 have the same syntactic properties (i.e. number and gender), otherwise it returns false.

Semantic class (denoted by C s , where s is a class name) is defined as an entity that is expressed by all of its verbalizations. For example, the verbalizations of the semantic class concerning kidnapping are C kidnapping ={kidnap, seize, abduct, capture, intercept, take hostage}. It should be noted that we do not use all the meanings of the listed words, but only these fitting to a specific context.

Atomic formula is a triple of the form <sub, pred, obj>, where sub means the subject of the sentence (and semantically it may play a thematic role R m ), pred means the predicate (represents an event in terms of a certain semantic class C s ) and obj means the object (semantically playing a role R p ). An atomic formula could be considered as a rule representing a fact.

Let us illustrate the introduced notions with the exemplary message from DEV-MUC3-0018 (the text in this corpus is given in an upper case). We decorated the text with roles, role fillers, events and anchors. One of the considered sentences is: OQUELI, LEADER OF THE NATIONAL REVOLUTIONARY MOVEMENT (MNR) AND HILDA FLORES, A GUATEMALAN SOCIAL DEMOCRATIC LEADER(R victim F 1 ) WERE ABDUCTED(E kidnapping A kidnapping1 ) AND KILLED IN JANUARY(R time F 1 ) BY UNIDENTIFIED INDIVIDUALS(R perpetrator F 1 ) IN GUATEMALA CITY(R location F 1 ) AS THEY WERE HEADING TO THE LA AURORA AIRPORT.

Assume that there exists another sentence concerning the same event but with the new fillers for the victim and perpetrator roles:

IT TURNED OUT THAT POLITICIANS(R victim F 2 ) WERE KIDNAPPED(E kidnapping A kidnapping2 ) BY URBAN TERRORISTS OF FARABUNDO MARTI NATIONAL LIBERATION FRONT(R perpetrator F 2 ).

After decorating the two sentences we are to check, whether two pairs: E kidnapping A kidnapping1 and E kidnapping A kidnapping2 concern the same event. We will show how to approach this issue in section 2.3.

We are motivated by VerbNet (VN) [1] thematic/semantic role methodology. VerbNet verb classes are organized according to the syntactic behavior of verbs. VerbNet uses 109 verb classes and 29 semantic role labels for arguments of the <sub, pred, obj> triple pattern (which resembles our atomic formulae). We adhere to VerbNet semantics rather than to ontologies, because we are not aware of any publicly available ontology with adequate expressive power and rich verbalization of classes (ontological entities). We are in the process of using our CATIE ontology for the general extraction of facts from MUC-4 corpus [6].

We are interested in such event specifying verbs as: kidnap, abduct, seize (VN index/vn/steal-10.5.php#steal-10.5; sense number 3: take or capture by force or authority) belonging to class steal-10.5. However, instead of a role Agent [

Basic rules for identifying thematic roles

The next type of rules (besides the earlier described atomic formulae that represent facts) says that as the direct anchors we use all the interesting verbs (C kidnapping ) in the past tense forms. Using a special function that retrieves a predicate of a given triple, namely predicate_of(<s,p,o>) = p, we denote such rules as triples of the form: <predi-cate_of(<s,p,o>), tense_of, "Past">. We assume that tense_of is a built-in predicate representing verb tenses, i.e. "Past" and "Past Participle". Another built-in predicate, named voice_of, represents voice of a verb phrase, namely "active_voice" and "pas-sive_voice". The third built-in predicate, named plays, represents a fact concerning the deduced thematic role of a subject and an object of some triple (as it was assumed we only consider the agentive role (a perpetrator) and the patientive (beneficiary) role  a victim). Now we are ready to give the rules to identify thematic roles of a predicate given in the past tense form. We are concerned with predicates expressed by verbs being members of a C kidnapping semantic class.

The first rule states that for a given triple if its predicate is in the past tense and in the active voice then the subject plays the agentive thematic role of a perpetrator while the object plays the patientive thematic role of a victim (a kind of a per-son_target). The rule (1) is as follows: <predicate_of(<sub,pred,obj>), tense_of, "Past">  <predicate_of(<sub,pred,obj>), voice_of, "active_voice">  <sub, plays, "agentive_role"> <obj, plays, "beneficiary_role">

The second (2) rule differs in the voice specification only that influences the order of the atomic formulae in the conclusion. The rule is as follows: <predicate_of(<sub,pred,obj>), tense_of, "Past">  <predicate_of(<sub,pred,obj>), voice_of, "passive_voice">  <sub, plays, " beneficiary_role">  <obj, plays, " agentive_role">.

(2)

Rules for event identification

In many cases information about certain roles and events is included in several sentences. Thus, matching different phrases to one thematic role constitutes one of a key tasks. We define a set of rules to identify such cases and eventually we either unify different events or differentiate them (the are_different predicate). One of these rules bases on two sentences with a verb phrases denoted as two pairs containing an event and an anchor, E n1 A m1 , E n2 A m2 . Each of these sentences contains a phrase that represents a filler of the same role, namely R p1 F k1 , R p1 F k2 . To activate such a rule we need to find at least two sentences with these role fillers and event anchors. If we happen to find more than two sentences of such a kind, we need to analyze them in pairs. To describe such a rule, we need to define two predicates. The "belongs_to" predicate is used if a given phrase belongs to a certain semantic class (this means that the main word in the phrase is a member of the considered class). The "is_equal_to" predicate decides whether either two semantic classes contain the same set of elements or role fillers are syntactically equivalent.

The process of analysis starts with searching of described pair of sentences. Let us denote the anchor and the role filler that were found in the first sentence as R 1 F 1 and E 1 A 1 , and the anchor and the role filler found in the second sentence as R 1 F 2 and E 2 A 1 . Once we have found these pairs we need to decide whether the described event anchors belong to the same semantic class (denoted as C 1 ). This is formalized as:

<E 1 A 1 , belongs_to, C 1 >  <E 2 A 1 , belongs_to, C 1 >.

This basic condition should be considered as preemptive and its result decides if we are going to consider a pair of sentences as worth of executing this rule on.

The second part of the analysis starts with determining if role fillers belong to classes that are different, but there exists some relation between those classes. Furthermore we need to check if role fillers have the same syntactic properties. If those conditions are true, we can assume that phrases describe the same event. Additionally, there exists some relation among semantic classes, which may also be projected on role fillers (in particular it may be a subsumption). Let us formalize these considerations in the form of rule (3). In this rule, we mark "some relation" as a variable "?rel".

<R 1 F 1 , belongs_to, C 2 >  <R 1 F 2 , belongs_to, C 3 >  <C 2 , ?rel, C 3 > <C 2 , is_equal_to, C 3 >  simsyn(R 1 F 1 ,R 1 F 2 )  are_the_same (E 1 ,E 2 )  (R 1 F 1 , ?rel, R 2 F 2 )(3)

However, if role fillers belong to the same class, but are different or role fillers have different syntactic properties, it is necessary to classify two events as different ( 4):

(<R 1 F 1, belongs_to, C 4 ><R 1 F2, belongs_to, C 4 ><R 1 F 1 , is_equal_to, R 1 F 2 >) simsyn(R 1 F 1 ,R 1 F 2 ))  are_different(E 1 , E 2 ). (4

) We illustrate that rule with the following examples.

Example 1

There are two consecutive sentences in the message: 1) John Smith (R victim F 1 ) has been kidnapped (E kidnapping1 A 1 ).

2) President (R victim F 2 ) was taken hostage (E kidnapping2 A 2 ) by unknown perpetrators.

The preemptive constraints are: <"kidnap", belongs_to, C kidnapping >  <"take_hostage", belongs_to, C kidnapping >.

The following rule activation captures lexical associations between two neighboring sentences by pairing as similar each noun in the role of a victim (person_target). This is similar to lexical bridge features used in [5]. The rule for those sentences goes as following:

<"John Smith", belongs_to, C Person >  <"President", belongs_to, C Politician >  <C Person , represents, C Politician >   <"John Smith", is_equal_to, "President">  simsyn("President", "John Smith")  are_the_same(E kidnapping1 ,E kidnapping2 ).

As the result we obtain a fact (an atomic formula) of the form: <"John Smith", represents, "President">.

The confidence of this rule could be measured in distance between the considered sentences (thus the distance is measured in the number of sentences). In particular this rule may be used only to analyze consecutive sentences.

Example 2

We have three sentences, not necessarily in one document.

1. Ricardo Alfonso Castellar, mayor of Achi,(R victim F 1 )who was kidnapped(E kidnapping1 A 1 ) on 5 January, apparently by Army Of National Liberation guerillas, was found dead.

2. Castellar(R victim F 2 )was kidnapped(E kidnapping2 A 1 ) by a group of armed men.

3. A politician condemned kidnapping(E kidnapping3 A 1 ) of mayor of Achi(R victim F 3 ).

In this case we need to process sentences in pairs. First, we take sentences 1 and 2. We execute the rule and as a result we get the unification of E kidnapping1 and E kidnapping2 . This means that unification of E kidnapping3 event, with both of the previous events would be redundant and we just need to clarify if E kidnapping3 could be unified with any of those events. However, if E kidnapping1 and E kidnapping2 would not be unified, all events need to be compared separately. In this case we get three fillers of the victim role, furthermore the relation between those fillers is quite specific. That relation could be marked as "is_substring_of". The left-hand side argument of this relation is always less expressive then its right-hand side and thus we could find the most expressive filler -"Ricardo Alfonso Castellar, mayor of Achi".

Our method of unification is conceptually more powerful than the so far used for coreference resolution (for example in [11,9]). But so far it is used only for establishing the agreement of semantic classes and also the noun-pronoun agreement features, that means features 2-3 and 8 out of 12 features proposed in [11].

Additional lexical rules

The examples shown in the previous subsection illustrate the need for rules that go beyond search of sentences with verb phrases corresponding to event related semantic class. To make the task of identifying event easier for the annotators, it is necessary to use the secondary semantic class containing words that are in a fuzzy relation to the core event term. We introduce a class:

C fuzzy_kidnapping = {disappear, release}

Following the Automatic Content Extraction (ACE) Programme guidelines: An event trigger refers to the term within the event mention that most clearly expresses the occurrence of the event instance and is based on direct anchorcorresponds to C kidnapping .

An event mention refers to the sentence within which an event instance is reportedcorresponds to C fuzzy_kidnapping . An event can have multiple mentions associated with it.

Apart from the sentence that initially reports the event, other coreferring sentences that contain anaphors of events (such as pronouns and definite descriptions of previously mentioned events) are taggable mentions of that event [9]. In general there always exists a direct connection between roles of events corresponding to C 1 and C fuzzy_1. For example a victim of kidnapping directly corresponds to a subject of releasement or disappearance. To measure the confidence of fuzzy classes we look at the statistics of all words/stems in various part-of-speech forms, which directly or indirectly could indicate an event of kidnapping. They are words corresponding to C kidnapping and C fuzzy_kidnapping classes  verbs for kidnap (heads of verb phrases) in the past tense or attributive kidnapped, verbs in the past tense, verbs (infinitve, -ing form for a verb, gerund), nouns related to an act of kidnapping or a perpetrator, namely: kidnap, kidnapping, kidnapped, kidnapper stem seiz, seized, seizing, abduct, abducted, abducting, stem captur, capturing, captured, intercept, intercepting, intercepted, stem releas, released, releasing, disappear, disappeared, disappearing, take/hold hostage. Finally, we apply coreference rules for both C fuzzy_kidnapping and C kidnapping semantic classes.

Example 3:

1. Ricardo Alfonso Castellar(R victim F 1 ), mayor of Achi, was released(E 1 A 1 ) on 15 January. 2. Kidnapping(E 2 A 1 ) of Castellar(R object F 1 ) was a brutal act.

Even though events E 1 and E 2 belong to different semantic classes we can unify specific role fillers within those events.

Word Statistics Tool

The process of designing pattern-based linguistic rules is a very tedious work, what constitutes the main disadvantage of such methods. To alleviate a burden we implemented a MUC Word Statistics Analyzer (Figure 1). The tool realizes several useful functions:

1) it presents graphically statistics of words across a document or a corpus 2) and it displays in two separate panels fragments of text pertaining to this statistics.

The considered in the paper extraction method relies on the quality of verb argument's typing (semantic classes). To obtain good results concerning the extensions of semantic classes C kidnapping and C fuzzy_kidnapping we designed and implemented a statistic tool. It estimates the frequency of words (exactly, their stems) occurrences in the message or in the whole corpus. The tool also enables the analysis of sentences (or message) across which the stems appear. In the upper right corner of the screen given in Figure1 the histogram is located that depicts the number of a word (stem) occurrences in the message and in the sentence. The exemplary message is shown in the lower left corner. In the bottom panel the list of sentences is located in which the stems with different endings appear, for example: a stem kidnap, end words kidnapped, kidnapper or kidnapping. Summing up, by the quick inspection of the frequency of appearance of words and their correlation and varying the trigger term lists we can assess effectiveness of linguistic features.

Results

There are five overall IE related tasks that evolved from MUC.  Named entity (NE) aims to extract all instances of persons, organisations, locations, dates, times, percentages and monetary entities.  Coreference (CO) given a set of entities, this task aims to generate a set of entity coreference chains, such that mentions that coreference to the same entity appears in the same chain.  Template element (TE) aims to extract all entity attributes. As an example, for the entity mention \ Castellar ", the aim is to extract its name (\Ricardo Alfonso Castellar, "), type (\PERSON") and descriptor (\the mayor of Achi").  Template relation (TR) aims to extract all well-defined facts from each newswire text. In MUC-4 this was related to the knowledge frame (24 slots) of 8 terrorist type of events. In MUC-7 the facts were limited to relationships with organisations: employee of, product of and location of.  Scenario template (ST) aims to extract pre-specified event information from anywhere in the given text, and relate it to the particular organisation and person entities etc. involved in the event. The figures presented in this table are based on the performance levels of systems participating in the MUC evaluations. More detailed figures can be found in Table 1. For many years these results have not been significantly improved. Only recently a significant progress [9,11] has been made.

There appear 159 events resolved as kidnappings out of 1700 documents as a result of assessment of the MUC-4 community [7].

We define the following numbers or word occurrences: The detailed analysis of these results will be presented at the Challenge event. We used the Sundance and AutoSlog systems for syntactic parsing and extraction patterns generation [12] together with Name Entity Extraction (with slightly modified dictionaries). Then we applied the semantic role filler and event resolution tool SRL Master.

In Table 3 the meaning of symbols is the following: EN= event name (e.g. kidnapping, crime, etc.)there are anchors, in all other patterns VP are anchors, NP = noun phrase, VP = verb phrase, PVP = passive verb phrase, AdjP=adjective phrase, PP= prepositional phrase starting with specific prepositions, Pron= noun phrase represented by a pronoun, Perp=perpetrator. Effectiveness of our system is due to several factors:  Our patters are mostly triples, whether most previous works were based on syntax patterns consisting of 2 elements, see e.g Fig. 1 of [13].  Non-triple patterns are more likely to generate extraction of nonrelevant patterns. For a pattern to be relevant we need to have at least either of two: location, date sentence part (first sought in a simple sentence, then in the complex sentence, and finally in adjacent sentences.  One of the main contributions of this work is the introduction of VP(S) = supplementary verb phrase (particularly effective involving NP=EN are: take place, claim responsibility, be responsible for, carry out. To a lesser degree this helps to identify perpetrators and victims.

The correctness of extraction in this paper is providing all of the following kidnapping event roles (recall): perpetrator individuals, perpetrator organizations, hu-man_target/victim, location and date. These roles are narrower than 24 slots of the MUC-4 contest.

Table 4 presents the recall for the kidnapping events (here the same events in different documents are counted separately, similarly as for MUC-4 evaluation). The recall numbers are significantly higher than in the MUC-4 contest (where the best contribution achieved around 60% for both precision and recall), but achieved for the easier task and for only one type of a terrorism event. They are also higher than in Table 3 of [5].

The system is presented at http://draco.kari.put.poznan.pl/ruleml2013_Extraction.

Conclusions

The recent wave of methods [11,9,8,3,4] is capable of significant improvement of extraction measures. The MUC Conferences provided benchmarks that decrease arbitrariness of a given method evaluation. For example open extraction system ReVerb gives a good precision but a poor recall [3]. We plan to apply against the full MUC-4 benchmark. The MUC Word Statistics Analyzer would be helpful for this task. There are improvement possibilities in using the probable better syntax parser, Named Entity Recognition and using a wider set of coreference comparison.

Our choice of anchor words can be more optimal. In general, our patterns presented in Table 3 are more compatible with ontology-driven extraction than purely linguistic methods. Rather than use one general dictionary as used by most MUC related works, we can have lexicalization specific to ontology element. We are working in this direction.