In this paper we have described a neural network based approach for Event extraction(EE) task which aims to discover different types of events along with the event arguments form the text documents written in Indian languages like Hindi, Tamil and English as part of our participation in the task on Event Extraction from Newswires and Social Media Text in Indian Languages at Forum for Information Retrieval Evaluation (FIRE) in 2018. A neural netork model which is a combination of Convolution neural network(CNN) and Recurrent neural network(RNN) is employed for the Event identi cation task. In addition to event detection, the system also extracts the event arguments which contain the information related to the events(i.e. when[Time], where[Place], Reason, Casualty, After-effect etc.). Our proposed Event Extraction model achieves f-score of 39:71; 37:42 and 39:91 on Hindi, Tamil and English dataset respectively which shows the overall performance of Event identi cation and argument extraction task in these three language domain.
A huge number of news on events, occuring in different corners of the world, are reported each moment in online and printed media. To keep track of those news, it is very important to identify relevant events and extract the spatio-temporal aspects about those events. Understanding events and their descriptions in raw text is the key factor in automatic event extraction which is an important and challenging task in Natural Language Processing(NLP) and Information Extraction(IE). It is also essential in practical applications like news summerization, information retrival and knowledge base construction. Event extraction is an important and challenging task in Information extraction(IE) which aims to detect, from the text, the occurrence of events of speci c types, and to discover the arguments(event participants or attributes) that are associated with the event. Event Arguments are basically represents the event related information i.e. capturing who does what to whom, how, when and where. For example,
{ S1: Mild earthquake has been found in Indonesia's Sulawesi island.
A typical automatic Event extraction system will identify from S1 that the sentence depicts a Earthquake event and it also extract some important information related to the earthquake such as, Place of the event: Indonesia's Sulawesi island.
Current event extraction systems rely on examining event trigger words to
check if an event occurs or not and if occurs then what is the type of the event.
An event trigger is a word or phrase that clearly express the occurrence of
an event. For example, in S1, earthquake is the trigger word as it is the most
salient clue in the sentence to identify the Earthquake. The Understanding and
identifying event triggers in raw text is an important and challenging problem
as the same event may be expressed by various trigger expressions and an
speci c event trigger might be used to represent different events in different context.
From the previous research it is evident that Event detection task is not a simple
trigger-token searching task. And Contextual feature information is very much
important in identi cation and classi cation of event triggers. Event argument
extraction is another challenging task. Unlike entity identi cation task, event
argument extraction task deals with discovering the speci c entities which are
relevant to the events appeared in the document. So typical NER systems are not
sufficient to solve the problem. Previously, we have worked on detecting the
segment of the text which contains some event related information from twitter[
A plethora of works have already been done in event extraction from text
documents. However supervised methods are dominant in this problem which
requires annotated training data to employ machine learning based approaches.
Previously, the researchers have designed various feature (lexical, contextual,
syntactic) based approach for Event Detection[
Now neural networks are widely applied in NLP tasks including Event
Extraction(EE) as they automatically extract underlying features. Some of the
papers apply Bidirectional recurrent neural networks(Bi-RNN) as it is capable
to capture both preceding and following context information of each token in the
sentence[
Here, we are interested in news events depicting man-made or natural
disasters. In the disaster domain, maximum research works are centered around
extracting events from user generated content such as social media texts.
Extracting event from social media text consists of two subtask: Identi cation of
informative tweets and detecting events from those informative tweets[
Event extraction are also represented as a natural language understanding
problem. Previously, natural language understanding task was highly
dependent on script knowledge[
Here, we focus on the sentence level Event extraction task. Given a news document we want to extract the events inside that document and represent the events in structured format. Our work covers the events which depicts the occurrence of any disasters caused by nature (e.g. Flood, Earthquake etc.) or manmade (e.g. terrorist attack, accident, riot etc). The term Event covers the type of the 1 https://tac.nist.gov/2017/KBP/Event/index.html event and the event participants or arguments which depicts the informations regarding that event(i.e. when[Time], where[Place], cause, effect etc). So while extracting events rst we have to identify the type of the extracted event based on the trigger word found in the text. Event argument identi cation is the another crucial module of our system as arguments are realy important to describe the events. Each event has some general attributes like place(where), time(when) and some speci c attributes like cause, after-effect, magnitude. Our interest is to identify these attributes for each events which are present in the document. And if the same event has more than one argument of same type then we have to aggregate the argument information accordingly such that no super uous information exist in nal outcome. Table 1 depicts sentence level event extraction output where each row indicates one event mention with corresponding argument information.
This section describes the dataset that have been used in this work. The training and the test data have been made available to the task participants by the organizers. The training dataset consists of annotated documents where event trigger words as well as event arguments are tagged. The testset contains news documents without any annotations. The dataset statistics is represented in Table 2.
An event extraction problem can be further subdevided into two subtasks: (1)Event Detection, the task of nding the event mentions of prede ned event types and (2)Event argument extraction , the identi cation of event related information like time, place, cause, effect which are relevant to the event mentions. We formalize the Event Detection(ED) problem as a multi-class classi cation problem via combination of convolution neural network(CNN) and bidirectional Long short term memory (Bi-LSTM). We expect that the Bi-LSTM and CNN would capture syntactic and semantic information from the text so their combination would help to enhance the overall ED performance. Let, we have k numbers of prede ned event types E1; E2; :::; Ek. Now, given a sentence S = w1w2w3:::wn, where n is the sentence length, for each word wi in the sentence, we want to predict whether the current token is an valid event trigger. And if it is an event trigger then with which event type Ej the word wi will be matched. So, the current word along with its sentential context constitute an event trigger candidate which will be input to our classi cation model. We have xed the context window size in order to feed the trigger candidate to the CNN.
If the window size is k then trigger candidate for word W would be represented as [w k; w k+1; :::; w0; :::; wk 1; wk] where current word is positioned in the middle position (i.e. w0). Now these (2k +1) size trigger candidates are taken as input to the CNN and Bi-LSTM models.
Before entering the CNNs and Bi-LSTMs, each word is converted into a real valud vector by looking up a embedding table with an intution that these real valued vectors will capture various semantic characteristics of the words. So the input to our model is a matrix X of size (2k + 1) jV j where jV j is the dimension of the real valued vectors and X = [x k; x k+1; :::; x0; :::; xk 1; xk] where xj is the vector representation of word wj . Now the two dimensional representation of each word is feed to a convolution layer followed by a max-pooling layer.
In convolution layer we have used a set of lters F1; F2; ::Fp and each lter Fi has a window size mi and can be represented as a matrix of size mi jV j. After employing max-pooling on the convolution layer output we get a hidden vector representation of size jF1j + jF2j + ::: + jFpj where, jFij represents the number of instances of lter type Fi. The hidden vector representation which we get from CNN model is named as FCNN .
Besides CNN, we have also used Bi-directional Long Short Term Memory(BiLSTM) in this work. The matrix representation of each word W which is represented as X = [x k; x k+1; :::; x0; :::; xk 1; xk], is also feed to the Bi-LSTM model. The hidden states of Bi-LSTM are computed both in forward and backward ways at each time step. We denote the outputs of the forward and backward LSTM as !ht and h′t respectively. At each time step t, hidden vector hi is computed based on current input vector xt and previous hidden vector ht 1. !ht = LST M!(ht!1; wt) h′t = LST M (h′t 1; wt) (1)
Then the output at time t is ht = [!ht; h′t]. Here we have taken the hidden representation of x0 i.e. concatination of hidden vector representation of forward LSTM h!0 And hidden vector representation of h′0 as the output of the Bi-LSTM. Formally, FBi LST M = [h!0; h′0].
Now to combine the output of CNN and Bi-LSTM we have concatenated the representation vector FCNN and FBi LST M and feed the concatenated vector to a fully connected layer followed by a softmax layer to get the proper event type of the current word W (see Fig. 1). The gradients are calculated using back-propagation. We have also implemented regularization by dropout.
For Event-Argument Extraction we have applied the same model which is used for Event Trigger classi cation. Only the number of output classes is different in these two models. We have used BIO annotation scheme for argument identi cation as most of the arguments contain more than a single token.
Now after Event-Trigger and Event-Arguments have been identi ed, our next task is to link the event arguments (which are identi ed by the EventArument extractor) to their corresponding event trigger words so that we can identify the event participants for a speci c event trigger. For Event TriggerArgument linking we have taken a simple heuristic based approach where, For each argument phrase we have identi ed the nearest event trigger word and link those argument- trigger pair. The distance between trigger and argument is calculated by the number of sentences between them and if a tie is there then decision has been taken by number of tokens in between the pairs. Sometimes, a sentence contains useful argument information regarding an event but there exist no event-trigger word in that very sentence. In that situation the argument is mapped to nearest event-trigger present in the previous or next sentence. 6
We have used the same parameters for the neural network models used in trigger identi cation and argument extraction. In Convolution neural network we have used lter size of 2; 3; 4 and for each size we have taken 200 lters to generate feature maps from the convolution operatons. The nal output of the CNN is represented as FCNN which has size 600. The number of hidden units used in Bi-LSTM is 200 which leads to the size of nal outcome of the Bi-LSTM layer FBi LST M is 400. Regarding embedding, we have used pre-trianed word embeddings of size 300 which we have taken from the fastText toolkit2. Finally, we have trained the neural network models using adam optimizer with suffled minibatches, dropout rate=0.5, backpropagation for gradient calculation and parameter modi cation. 7
According to the FIRE 2018 official evaluation metodology the performance of the system is measured by the Precision and Recall and F-score. For example, let there is an Event-mention E1 and there are six elds such as Event Type, Location, Time, Event-Participants, Causes, Effects for that event. Now for that event E1, if all these elds are identi ed correctly then the system gets full score of 1 else according to the identi ed elds the score will be calculated. And nally macro-average of the Precision and Recall is calculated for all events in all documents. And the performance of our system is depicted in 3. From the result, we can conclude that we have to improve the Recall score to enhance the system performance. In all the three languages the precision score is quite acceptable but due to low Recall, overall performance of the system is not so satisfactory. 2 https://github.com/facebookresearch/fastText/blob/master/pretrained-vectors.md Our event extraction system has three steps. Firstly we have ident ed the event trigger words. Then, we have extracted argument phrases. Lastly we have linked the arguments with their nearest trigger words. We have split the training data in the the ratio of 80:20 and use the rst partition(80%) for training and remaining 20% as development set. In this section we will try to analyze the system's performance on development set and examine the shortcomings of the system. 8.1
While identifying the trigger words, system sometimes fail to get the contextual information which has an important role in event identi cation. For example: { S2: Nepalese Maoists will not attack on ordinary people and politicians any more.
The system tagged attack as an event trigger of type Terrorist-Attack. But from the context it is clear that this sentence does not describe any terrorist attack event.
{ S3: It is reported that 56 people were also wounded in the bombing in the neighborhood of Dasht-e-Barchi.
The event identi cation module sometimes failed to correctly classify the event types present in the documents. We have noticed that the system is confused among related event types like terrorist-attack, Shoot-out, Suicide-attack or Vehicular-Collision, Transport-hazard and Accident. For example, in S3, system classi ed trigger word bombing as Shoot-out type event but it is actually an event of type Suicide-attack.
{ S4: Officials blame the Islamic State group for this attack.
In some of the cases system failed to detect the correct event triggers from the sentences. For example, in S4, system failed to identify attack as an event trigger word. 8.2
We have checked the argument extraction output and identi ed some errors, which we will dicuss below.
Symantic Variety: Based on development set, have noticed that our system performance is not so satisfactory for some frequent arguments like Place, Time. The possible cause is the symantic variety. Some arguments are represented in a very speci c way. For example, { S5:The magnitude of the rst quake was 4.9. { S6:After a short span there is an another earthquake whose magnitude is 4.1. { S7:The Governor of California was injured in a motorcycle accident near the house on Sunday afternoon.
The magnitude of quake is represented very speci c way (example S5, S6) and it is always a numeric value. But arguments like place is not all time a place entity which can be detected by typical Named entity recognizer(NER) systems. For example in S7, near the house is not a typical location entity, but is the place of event type VEHICULAR COLLISION.
Token missing in phrase: Most of the event arguments are more than one word phrase. But our system is failed to identify all the words in the phrase which leads to missing important information regarding the event. For example, in the S7 our system has extracted Sunday as time argument whereas the accurate time information will be Sunday afternoon.
Reason and After-effect: Our system has failed to identify arguments like Reason and After-effect for any event instance. Most of the time reason and effect of any event is an event itself and are represented by a long sequence of words or phrase. But as there is no such speci c pattern in these two arguments the system is failed to identify them. The cases where there is a semantic clue present in the sentence then system can identify it. For example: { S8:US Airways has been forced to land in Tehran due to mechanical disruption on its way to Amsterdam with 255 passengers from Mumbai. The system has accurately identi ed the reason of the AVIATION HAZARD type event as: due to mechanical disruption. The presence of the term due to may help in this case. But in most of the cases this type of clues is not present. And the proposed system accuracy is miserably bad for these two attributes. Some smarter technique have to be employed to improve our performance. 8.3
In this mdule, we have linked the event trigger words with its corresponding arguments using a heuristic approach stated in Section 5.
Incorrect assignment It is noticed that the heuristic approach which takes nearest token for linking, may causes wrong output. Some times due to complex syntatic structure of the sentence a valid argument may appear far away from the trigger word and lots of other token are exist between that argument and the valid trigger word. But our heuristic function has selected an incorrect triggerargument pair. For example,
{ S9: Translation: Earthquake in Indonesia. { S10: Mild earthquake has been found in Indonesia's Sulawesi island. { S11: Meanwhile, one Indian has died from the four oors of Manama Hotel in Dubai.
In the sentence S11, there is no trigger word of our interest but our EventArgument extractor has identi ed One Indian has died as a casualty argument. Our linking module will wrongly map this with EARTHQUAKE event in the previous sentence(S10), though they are totaly unrelated.
Mutual exclusiveness It is highly probable that one single argument may be attribute of more than one different event instance. But according to our heuristic function one argument can only be linked to a specif event trigger word. These problem leads to missing important information in the extracted events and directly affect system performance. 9
The main problem that impede the designing of Event extraction model for Indian language news document texts is scarcity of annoatated corpus. The second hinderance is unavailability of syntactic and semantic feature extractor for Indian languages. In this task we have relied on automatically learning effective features from data, without using language speci c resources. So our hybrid neural network model incorporates both bidirectional LSTM and CNN to capture sequencial and structural information from text. Our main insight from is that, the sentence level event extraction is not effective in case of news documents. When we take a news article into consideration, we have noticed that the detected events may have their arguments distributed along the length of the document. Sometimes event arguments and corresponding event trigger may appear in two different sentences. And it is evident that the event span(portion of the document where event related information exist) may be a single sentence or multiple sentence or even a whole document. So, identifying events on sentence level will narrow down the event span which leads to a substantial loss in acquired event related information. Eventually the sentence level event extraction system is inadequate to get the document level view of the events.To cope up with this problem, detection and aggregation of coreferring event mentions could be an useful remedy. As future work we would like to explore more sophisticated models for Event extraction and try to incorporate background knowledge and other language resources to improve the system performance.