Social Media tools popularized the digital devices among masses making information dissemination easier and faster. Exchange of text is most popular efective means of communication across social media users. It has become necessity to process, understand the semantics of messages communicated as the messages have wide efect across the users. Event extraction refers to understanding the events across streams of social media messages. Event extraction helps in taking quicker corrective actions in case of natural calamities and hence possibly save lives of people. The main objective of the task is, drawing specific knowledge to predict the events( incidents) specified in digital text. We proposed two step procedure to extract events. First phase consists of applying a binary classifier to identify the messages, containing the event. Second phase consists of applying a sequence labeling technique, conditional random ifelds(CRF), to extract the event from the message. As social media text is noisy, it is a challenge to develop learning algorithms for these tasks. We use Parts of Speech (POS) tags of the words to address some of the issues in this challenge.
• Information systems → Structured text search; ACM proceedings, LATEX, text tagging
Twitter1 and Facebook2 messages provide up-to-date information about the current events. In todays world with proliferation in the usage of social media, extracting current events from unstructured tweets and posts has gained ample attention. Social media data consists unusual characteristics like short length, stylistic variations, acronyms, noisy and unstructured forms. This makes event extraction a challenging problem in Natural Language Processing (NLP). Numerous tools have been developed [14], [8] in the recent past for enabling short-text processing for various tasks such as POS Tagging, Chunking, Named Entity Recognition. These tools use special techniques to account for the out-of-vocabulary terms in text collected from twitter. Ritter et al. [14] uses brown’s clustering on a huge corpus of english tweets to cluster similarly used words such as ’2morrow’, ’tmrrow’, ’2mar’. In comparison to english, the resources available for remaining languages are quite less even for formal text. This shared task pertains to processing text written in ∗Work done as part of Ph.D 1http://www.twitter.com 2http://www.facebook.com Indian languages namely ’hindi’, ’tamil’, ’malayalam’ to obtain the events from the social media messages.
Event Extraction is one of the most valuable tasks in Natural Language and Information Extraction. For example, accurate selection of news messages will improve the performance of news systems [5]. Furthermore, by detecting the occurrence of events, as early as possible, the performance of risk analysis systems Capet et al. [4], trafic monitoring systems Kamijo et al . [7] can be improved and forecasting civil unrest [13].
Early works, most of the methods Allan et al. [1] [6] Yang et al. [17] for event extraction have focused on news articles, which is the only best source of information for current events. With the ability of social media tools to virally popularize news items and their acceptance across masses, numerous media agencies have been relying on twitter, facebook feed pages to disseminate their news highlights. Twitter feeds for hindi 3,4, tamil 5,6 and malayalam 7,8 are few examples of social media forums continuously posting the news items. Among the posts made by these feeds, only a small fraction of tweets contain events.
Alen Ritter Allan et al. [1] developed the first open-domain event extraction tool (TWICAL) for twitter data. Extraction of NASDAQ100 listed companies information from RSS feed using StockWatcher was proposed by [10]. Hermes Borsje et al. [3] is news interpreter that supports the decision making process to filter the relevant news using Semantic Web technologies. Using specific features related to the natural disasters, Sakaki et al. [15] proposed a method to detect the earthquake-related tweets. Benson et al. [2] presented a relation extractor to predict the artists and venues from tweets. 2
Social media text written in Indian languages has received much lesser attention compared to english. The multiplicity of languages in India and usage by comparatively lesser population can be thought of as the possible reasons for this observation. Tokenization is first step in processing the social media text written in Indian languages. Efective tokenization helps in segregating meaningful features from noise. Feature extraction involves conversion of text into lemma form and morphological analysis. POS tagging involves attributing POS tags for the text, observed as a sequence of words. We depend on the tools available 9 [12] for POS tagging of various Indian language sentences. The task is to identify event carrying 3https://twitter.com/aajtak?lang=en 4https://twitter.com/bbchindi?lang=en 5https://twitter.com/news7tamil?lang=en 6https://twitter.com/thatstamil?lang=en 7https://twitter.com/manoramanews?lang=en 8https://twitter.com/beatsofkerala?lang=en 9http://ltrc.iiit.ac.in/download.php messages and then extract relevant portions of the events from such carrying text. The main phase of the approach consists of building a classifier to distinguish messages based on whether they carry event or not. Further post-processing is applied to extract the actual event from the event text. This is done through sequence tagger trained from the provided training data. Figure 1 indicates the overview of the our approach followed for the shared task. We use mallet 10 [9] the toolkit for the various machine learning algorithms applied in this work. 2.1
Given a collection of sequence of words D where each sentence is of form w1, w2...wn , identify the sequences carrying events. For the identified sequences a label sequence l1, l2...ln is predicted where each tag li ∈ B, I , O. The tags B,I,O indicate the beginning, inside and outside of event in event carrying text. 2.2
Tokenization for our approach follows that of twokenizer 11. This was extended to handle the unicodes of Indian languages. The indian languages hindi,tamil,malayalam have unicode ranges of 0x0900-0x097F, 0x0B80-0x0BFF, 0x0D00-0x0D7F respectively. Emoticons, urls, hashtags, userids are various other special tokens.
The POS taggers used from 12 have their own tokenizers and
lemmatizers to do the morphological analysis. These tools are designed
for news wire text which is supposed to be much more cleaner than
the text seen on social media. We modified the respective tokenizers
to get process the social media text.
10http://mallet.cs.umass.edu/index.php
11https://github.com/brendano/tweetmotif
12http://ltrc.iiit.ac.in/download.php
The critical component of our solution is the classifier which detects
the events. We analysed the detection capabilities of 3 classifiers
namely naive Bayes, logistic regression and semi-supervised naive
Bayes EM. Naive Bayes (NB) algorithm models the classifier as an
outcome of generative algorithm. If D is a training dataset of N
examples x1, x2, ...xN and y1, y2...yN are corresponding labels, NB
model is expressed in the equation (
N maximize X loд(p (xi, yi ))
i=1 p (xi, yi ) = p (yi )p (xi |yi ) p (xi, yi ) =
F
Y p (wj |yi )xi j
j=1
(
Event extraction is performed using CRF [16]. If (xi, yi) is the i’th
example where xi is a feature sequence of lenght T and yi is
corresponding label sequence, the CRF models the maximization of joint
conditional likelihood in Equation (
Naive Bayes EM (NBEM) is a semi-supervised algorithm [11]
which makes use of test data also along with the training data to
infer the classifier. If the dataset training and testing portions of the
dataset are designated as Dl and Du respectively, the NBEM learns
the classifier as a maximizer of (
Maximum entropy or logistic regression (MaxEnt) is a
discriminative approach for building the classifier and hence does not get
much benefit of the EM setting. MaxEnt learns the model which
maximizes the objective in equation (
N maximize X loд(p (yi |xi ))
i=1 p (yi |xi) = PL
y=1 µ yt .xi)
exp (µ yi t .xi)
in Equation (
p (yi |xi) = exp (µ t .f (xi, yi) + ηt g(yi))
Z (xi)
(
We used CRF++ 13 as our sequence labeller. CRF++ allows feature templates to be given for learning. The features being used for CRF++ are unigrams with a window of 5 words from the current word. As unigram features are extremely noisy as they are mostly seen very few times in the corpus and test data unigrams mostly are seen for the first time, we use POS tag based features also as second set of features to help the model inferred by CRF in mimicking event extraction process. We use similar 5 tag window for POS tags also. We used label based bigram features as label based features. The dataset provided for the shared task contains the starting ending positions of the event for each matched event text with in the text. We have converted this format of the input to the B, I , O based tagging to reflect the input for CRF++. As the tokenization employed adds spaces to reflect the tokenization process, the output of CRF is remapped to original to reflect the positions of the event as expected by the shared task. 3
The datasets taken for the task are summarized in Table 1. The preprocessing replaces all urls with keyword URL, ’@’ mentions are replaced with USER and hash-tags are all preserved as they sometime contain the event specific tags such as #BombBlast #EarthQuake. The test set for hindi is 4-times that of trainset while tamil and malayalam are relatively better in this ratio. Number of unique words is lesser for malayalam. 13https://taku910.github.io/crfpp/
Hindi Hindi
Hindi Malayalam Malayalam Malayalam Tamil Tamil Tamil
NB MaxEnt
NBEM MaxEnt + POS MaxEnt + POS
NBEM
NB MaxEnt NBEM
NB MaxEnt NBEM We used mallet library for building the binary classifier for event detection. The training dataset consisted of event-text file and annotation file. The events text file consisted of one message for each line with additional details such as user-id, message-id. The annotation file shows message-id and event index for each event carrying message given in events file. We prepared a binary classifier treating the missing messages of annotation file as labelled ’no’, while matched events as ’yes’. The performance of classifier is measured using two metrics namely Accuracy and F1-Score. They are defined in equations
Accuracy = Precision =
Recall = F 1 − Score =
N oo f CorrectPredictions T otalnoo f Predictions
T ruePositives T otalPositivePredictions
T ruePositives T otalPositiveexamples 2 ∗ Precision ∗ Recall
Precision + Recall
(
We performed a 5 fold cross-validation to detect events. The classification accuracies and F1-Scores of diferent classifiers NB, ME, NBEM are reported in table 2
We can observe that MaxEnt classifier performs better than NB in all cases significantly, asserting the superiority of discriminative approaches. Adding POS tag features has improved the classification accuracy of MaxEnt and NB by 4.4% and 6.4% respectively. NBEM is the semi-supervised approach which is consistently better than the other two methods with and with-out POS tags, as it uses the test portion of the data for learning its model. 3.2
The event extraction module contains the tags B, I, O indicating
the beginning, inside, ending of a event. The accuracy and F1-Score
of Equations (
F-measure 43.77