Today communication has become very fast and is happening in real time. An event that happens in any part of the world gets communicated in few seconds/minutes to the rest of the world. For example the recent twin bomb blasts in Damascus, Syria was known to the world within few minutes. This event was broadcasted in various media channels. . The penetration of smart phones, tabs etc., has significantly changed the way people communicate. The information about events or happenings in real time is very valuable to the administration for disaster management, crowd control, public alerting. These information which is used in the development of recommender systems adds value for the growth of business enterprises. Thus there is a great need to develop systems which can automatically identify various events such as bomb blasts, floods, cyclone, fires, political events etc., reported in various Newswires, Social Media text. This is the 2nd edition of the track. The first edition of this track was conducted last year at FIRE 2017. In that edition the task was to identify only the event and event span given in the data. Thus further going ahead in this track, along with the identification of event and its span, it is necessary to identify the cause and effects of a given event. The actual real time applications will be benefited only if the full information related to the event is identified. For example for a bomb blast, it will be required to know where it has occurred, when it has occurred, who and what all got effected, what are the causalities etc. In this edition of the track we propose to provide data annotated with the cause and effect details of an event and participants are required to identify these details along with event identification. And as in the last year, the focus is on Indian languages text. This paper presents the overview of the task “Event extraction in Indian languages”, a track in FIRE 2018. The task of this track is to extract events and all other associated arguments or information such as locations, cause, and its effects from the text. Though event extraction from Indian language texts is gaining attention among Indian research community, however there is no benchmark data available for testing the systems. Hence we have organized this track in the Forum for Information Retrieval Evaluation (FIRE). The paper describes the corpus created for two Indian languages, viz., Hindi, and Tamil and present the overview of the approaches used by the participants.
1
Over the past decade, Indian language content on various media types such as websites, blogs, email, chats has increased significantly and it is observed that with the advent of smart phones more people are using social media such as twitter, facebook to comment on people, products, services, organizations, governments, etc. Thus it is seen that content growth is driven by people from non-metros and small cities who generally are comfortable with their own mother tongue rather than English. The growth of Indian language content is expected to increase by more than 70% every year. Hence there is a great need to process these data automatically. This requires natural language processing software systems which extracts events, entities or the associations of them. Thus an automatic Event extraction system is required. The objectives of the evaluation are:
Creation of benchmark data for Event Extraction in Indian language Social Media text.
To encourage development of Event extraction systems for Indian language Social Media text.
Event extraction has been actively researched for over last decade. Most of the
research has, however, been focused on resource rich languages, such as English,
French and Spanish. The scope of this work covers the task of event recognition and
extraction in newswire, social media text such as facebook for Indian languages. In
the past there were events such as Workshop on NER for South and South East Asian
Languages [
The research in analyzing the social media data is attempted in English through
various shared tasks. Language identification in tweets (tweetLID) shared task held at
SEPLN 2014 [
Named Entity recognition was explored in twitter through shared task organized by Microsoft as part of 2015 ACL-IJCNLP, a shared task on noisy user-generated text, where they had two sub-tasks namely, twitter text normalization and named entity recognition for English. The ESM-IL track at FIRE 2015 came up with the named entity annotated benchmark data for the social media text. And CMEE-IL Track of 2016 came for named entity annotation detection for code-mixed data. The task of Event identification in Indian languages is at nascent stage. EventXtraction track at FIRE 2017 is the first step towards creating benchmark data and boosting the research in Indian language event extraction. This edition of the EventXtraction track at FIRE 2018 focusses on complete extraction of events and their associated arguments.
The paper is organized as follows: section 2 describes the challenges in event extraction on Indian languages. Section 3 describes the corpus annotation, the tag set and corpus statistics. In section 4 the overview of the approaches used by the participants are described and section 5 concludes the paper. 2
discourse markers in English have explicit words whereas it is not so in Indian languages, it is expressed as inflection markers. 3
The corpus was collected in two different time periods. The training partition of the corpus was collected during June 2018. And the test partition of the corpus was collected during Aug 2018. In this present initiative the corpus is available for two Indian languages Hindi and Tamil along with English. 3.1
Here we have followed HTML style of annotation in this work. The general syntax for the event tagging is as given below.
<EVENT ID=”number” TYPE=“abc” SUBTYPE_1=“xyz” SUBTYPE_2=“def”>Event Trigger</EVENT>
Here, this event tag has attributes: i) ID -- This is a number which will be unique for each event in a given document. ii) iii)
TYPE – This is the type of the event such as “manmade disaster” SUBTYPEs – These are the subtype category names of the particular event.
Event arguments such as participants, time of occurrence, and location of occurrence are also annotated using HTML style. For example the “time of occurrence” attribute of an event will be annotated as follows: <TIME-ARG REF-ID=”eventID”> abc </TIME>
Each argument tag of an event will have the attribute “REF-ID”, which is a number that refers to the event ID of the event to which the argument belongs.
The different types of event argument tags are as follows: a) <TIME-ARG> b) <CAUSE-ARG> c) <CAUSUALITIES-ARG> d) <PLACE-ARG> e) <EFFECTS-ARG>
On 29 December 2017 a massive fire broke in Kamala Mills, Mumbai the capital of Maharastra, killed at least 14 people and injured several.
On <TIME-ARG REF-ID=”1”>19 JULY 2018</TIME-ARG>, a massive <EVENT TYPE= “Manmade Disaster” ID=”1” SUBTYPE_1= “Accident” SUBTYPE_1.1= “Fire Accident”>fire broke</EVENT> in <PLACE-ARG REFID=”1”>Kamala Mills, Mumbai the capital of Maharastra</PLACE-ARG>, killed <CAUSALITIES-ARG REF-ID=”1”>at least 14 people and injured several</ CAUSALITIES-ARG>.
The participants were provided the data with the above explained annotation markup in a separate file called annotation file. The participants were also instructed to provide the test file annotations in the same format as given for the training data. The dataset statistics is as follows:
The data has events from different types such as cyclones, floods, accidents, disease outbreak and political events. And the majority of the types were the disasters and political events such inaugurations/opening ceremonies by political leaders. Also the data had events on movie or audio release functions. 4
A total of 10 teams registered for participating in the track. The final submissions were done by 2 teams among the 10 teams. They submitted their test runs for evaluation. A total of 5 test runs were submitted for evaluation. Only 1 team had participated for all the three languages. Another team had participated for English and Hindi.
We had developed a base system without using any pre-processing and lexical resources. The base line system was developed using a CRF classifier which will mark if a phrase is an event phrase or not. The baseline system performed with a Precision of 0.4521 and Recall of 0.6522 for event identification. The different methodologies used by the teams are summarized in Table 2.
Team cleaning and Tokenization
NIL
NLTK Tool kit Evaluation metrics used are the well measures precision, recall and f-measure. All the systems have been evaluated automatically by comparing with the gold data. We define:
Precision, P= (No. Correctly identified Events by the system)/ (Total No. of Events identified by the system)
Recall, R= (No. Correctly identified Events by the system)/ (Total No. of Events identified in the Gold)
F-measure= (2*P*R)/ (P+R)
The methodology for calculating the Precision and Recall will be field based average score. For example, for an Event E1, if there are 6 fields such as Event Type, Event Location, Event Date, Event Actors/Participants, Causes, Effects. Then for that event E1, if all these fields are identified correctly then the system gets full score of 7/7 else according to the identified fields the score will be modified. And finally micro and macro-average of the Precision and Recall will be calculated and final score is arrived at. The results obtained for the system runs is presented in Table 3. 5
The main objective of creating benchmark data representing a few of the popular Indian languages has been achieved. And this data has been made available to research community for free for research purposes. The data is user generated data and online Newswire data. Efforts are still going on to standardize this data and make it perfect data set for future researchers. We observe that the results obtained are encouraging but scores are low and need lots of improvement for real time use. We aim to provide a more standard and corrected data for these languages. We hope to see more publications in this area in the coming days from these different research groups who could not submit their results. Also we expect more groups would start using this data for their research work.
This EventXtract-IL track is one of the first efforts towards creation of Event annotated user generated data for Indian languages. In this edition of the track we have provided data which can be used to develop a complete Event extraction engine, so that real time systems can be developed in near future. We plan to add few more other languages data.
Team
We thank the FIRE 2018 organizers for giving us the opportunity to conduct the evaluation exercise. We also thank the Language Editors in our CLRG lab, AU-KBC Research Centre for annotating the data.