In this paper we describe the IIT KGP team's participation in the Event Extraction task at FIRE 2017. We have developed an event extraction system which can extract event-phrases from tweets written in Indian language scripts along with Roman script. We designed our system on Hindi language and then used the same system for Malayalam and Tamil languages. We have submitted two systems one uses pipelined architecture another uses non-pipelined architecture. In case of pipelined architecture we first identify the tweets which contain event inside it and then extract the eventphrase from those tweets. In case of non-pipelined system all the tweets are directly pass to the event extraction system. Though conceptually simple, non-pipelined approach gives better result than pipelined approach and achieves F1-score of 50.01, 48.29 and 51.80 on Hindi, Malayalam and Tamil dataset respectively. 1https://tac.nist.gov/2017/KBP/ To overcome the dificulties of complicated feature engineering and domain dependency, researchers use neural network approach for event classification [ 2, 11, 14]. But all these works deal with English language and principle objective of these tasks is to detect the trigger word in the text which indicate an event. Some of these papers also identify the arguments related to these event trigger and their corresponding roles in the events [2, 14, 18].
Event Extraction from unstructured text is one of the most
important and problematic task in Information extraction and natural
language processing. Event extraction deals with automatic
extraction of events depicting accidents, crime, natural disasters, political
events etc. from various newswires, discussion forums, social media
texts. Most of the existing event extraction systems [
Many approaches have been taken to extract events from text.
Judea and Strube,2015 formulated the event extraction problem
as frame-semantic parsing [
Dataset contain tweets written in both Indian languages and Roman script. Three Indian languages are: Hindi, Malayalam and Tamil. Training dataset contains two file for each language. One file contains all the tweets obtained using the Twitter API. Another annotation file contains event phrases extracted from tweets present in previous file. Each line in the annotation file contains: tweet-id, user-id, Event phrase of the tweet, index where this phrase starts in the tweet string, string length of the event phrase. Test file contains only the tweets with corresponding tweet-id and user-id. The details of the training and test dataset is depicted in the Table 1.
402 674 1109
In this section we describe our event extraction system. We have experimented with two types of event extraction systems: 1. Nonpipelined approach 2. Pipelined approach. We have used neural networks as the main technique in both the cases. 5.1
The training file contains tweets which are written in mainly Indian language script with some Romanized script. Some of the tweets are ending with urls. To avoid data sparseness problem we have replaced all the urls with a unique token. Event annotation file contains some event phrases which are taken from same tweets and indicate same event and the words contained in those eventphrases are more or less same. We have omitted those redundant event-phrases. 5.2
In case of non-pipelined approach we have formulated the event
extraction problem as sequence labelling problem. For every token
in the input tweet we have tagged the word with ’0’ or ’1’ i.e. ’outside
event-phrase’ or ’inside event-phrase’ respectively. And for this
task we have used a combination of convolution neural network [
W1
W2
W3
Wn
Input Convolution layer
Two back to back convolution layer Word Embedding Feature vector representation
Forward LSTM LSTM
LSTM
LSTM
LSTM
LSTM
It is noticed in the training corpus that approximately 40% of the
tweets contains event phrases. So it is vacuous to check all the
tweets for extracting event phrase. From this intuition we have
employ ed an event classification module before the event extrction
module, depicted in non-pipelined approach. In case of pipelined
approach first events are classified as event-tweet and without
event-tweets. Tweets which are classified as event-tweets by our
classification module are fed to the event extraction module. Other
tweets which are classified as without event-tweets are discarded
and are not fed to the event-extraction module. The classification
module is similar to [
Preprocessing input tweets
Cleaned Tweets
Tweet classification
Tweets containing events output:Run2
event extraction model 5.3.1 Tweet Classification. Here we have used a convolution nueral network(CNN) based architecture for tweet classification. As the tweets are of diferent length so padding is applied to make them of fixed size. Now these padded sequences are fed to an embedding layer to convert the tokens into a fixed size real-valued vectors. Then the sequences of fixed size vectors are fed to a convolution layer followed by maxpooling layer. The internal representation again fed to a combination of convolution layer followed by a pooling layer. The model uses multiple filter size to get multiple features. Now the output is fed to a fully connected softmax layer which gives the probability distribution over two classes: event-tweet or without event-tweet. The performance of Tweet-classification module is reported in Table 2.
Conv layer
Pooling Layer
Concatination W1 W2 W3 W4 W5 P
P P: padding token Input Tweet
Filter size=3,4,5 Feature maps
Contextual feature vectors
Conv and Pooling layer
Eventually the tweets classified as event-tweets are fed to the event extraction module described in non-pipelined section. The architecture of event extraction module in pipelined approach is same as non-pipelined approach. The only diference is that in case of pipelined approach at the training time we use only those tweets which contains events. Tweets which contains no event are discarded from training data.
Event extraction module will give the event span(i.e. the event phrase) within the tweets.
The event phrase which depicts events inside a tweet consists of cosecutive word sequences. So after sequence tagging if there exist ’0’s inside sequence of ’1’s then first ’1’ is taken as the strating point of event-phrase and the last ’1’ in the sequence indicates the ending of event-phrase. All the tokens inside the boundary are cosidered as event-pharase. We use this heuristic to maintain the constraint that all the event-phrases consists of consecutive tokens.
Event extraction model used in pipelined and non-pipelined approach uses same architecture and hyperparameters. Regarding embeddings we have used 100 dimensions for word embedding in the word embedding layer. The first convolution layer uses filter size of 3 and number of filters used mf = 30. In second convolution layer the filter size is 4 and number of filters mh = 20. The bidirectional LSTM layer uses one hidden layer with hidden layer size 60. For event classification we have used CNN based classification approach which uses word embedding of size 100. These vectors are randomly initialized and fed to the embedding layer. We have employed filter size of {3, 4, 5} with 20 filters for each filter size for the convolution operation. Finally, we have trained the neural network models using adam optimizer with sufled minibatches, dropout rate=0.5, backpropagation for gradient calculation and parameter modification.
Table 3 shows the performance of event extraction in all three languages using both pipelined and non-pipelined approach. While examining the result in each languge we have found that nonpipelined system has given better F-score than pipelined approach. In Hindi dataset Pipeline system acquire F-score of 40.35 but in non-pipelined approach the F-score is 50.01. For Malayalam the Fscore in Pipelined and non-pipelined approach are 47.17 and 48.29 respectively which are comparable. But in Tamil non-pipelined system whose F-score is 51.80 beats pipelined system (F-score: 44.01). Error propagation in pipelined approach may be responsible for this low performance of pipelined system. The performance of tweet-classification module directly influenced the event extraction system in pipelined approach. It is also obvious from the Table. 3 that the precision is very much low in both pipelined and nonpipelined system. We will investigate on our model to improve the precision score.
We have taken two strategies for event extraction. In case of
nonpipelined approach we have classified each word with tag ’0’ or ’1’
indicating inside event phrase or outside event-phrase. But there
are many tweets which do not indicate any event. So in pipelined
approach first we have detected those tweets which contain any event
and then identify the span of the event inside the tweet. The
accuracy of the pipelined approach depends on accuracy of the tweet
classification module. So we will try to improve the performance
of tweet-classification module. In our experiment the number of
training tweets are very low. If more training data could be used
the event extraction accuracy may increase. In future we will try to
increase the performance of the event extraction system by using
more training data and other advanced strategies [