The goal of FIRE 2020 EDNIL track was to create a framework which could be used to detect events from news articles in English, Hindi, Bengali, Marathi and Tamil. The track consisted of two tasks: (i) Identifying a piece of text from news articles that contains an event (Event Identification). (ii) Creating an event frame from the news article (Event Frame Extraction). The events that were identified in Event Identification task were Man-made Disaster and Natural Disaster. In Event Frame Extraction task the An event is defined as an occurrence happening in a certain place during a particular interval of time with or without the participation of human agents. It may be part of a chain of occurrences or an outcome or efect of preceding occurrence or a cause of succeeding occurrences. An event can occur naturally or it can be because of human actions. An event can have a location, time, agents involved (causing agent and on which the efect of the event is felt) etc.
In this task the participants had to identify a event given a news article. The events were of two type: Natural disaster and Manmade disaster.
In this task the participants had to form an event frame given a news article.The event frame consists of the following fields: 1. Type: Detect the type of the event. There are two type of events a) Natural disaster b) Manmade disaster 2. Subtype: It is the event which is subtype of Natural or Manmade disaster.
The subtypes of Natural disaster are forest fire, hurricane, cold wave, tornado, storm, hail storms, blizzard, avalanches, heat wave, cyclone, drought, heavy rainfall, limnic erruptions, floods, tsunami, land slide, volcano, earthquake, rock fall, seismic risk, famine, epidemic and pandemic.
The subtypes of Manmade disaster are crime, riots, aviation hazard, accidents, train collision, vehicular collision, transport hazards, industrial accident, fire, normal bombing, terrorist attack, miscellaneous, shoot out, surgical strikes, suicide attack and armed conflicts. 3. Casualties: Number of people injured or killed and Damage to properties. 4. Time: When the event took place 5. Place: Where the event took place 6. Reason: Why and how the event took place
Shared tasks on event detection have also been proposed earlier, such as TAC-KBP 2016 Event
Nugget track [
The contribution of EDNIL is that it provides an annotated dataset for event detection from ifve Indian languages i.e. English, Hindi, Bengali, Marathi and Tamil.
The dataset was created as part of the project ”A Platform for Cross-lingual and Multilingual Event Monitoring in Indian Languages” 1. The dataset consists of news articles in English, Hindi, Bengali, Marathi and Tamil languages which have been collected from diferent news agencies. The statistics of the dataset documents is shown in Table 1.
1https://imprint-india.org/knowledge-portal-5592-a-platform-for-crosslingual-and-multilingual-eventmonitoring-in-indian-languages
News article of each language is annotated manually by annotators from IIT Kharagpur (Bengali), IIT Bombay (Marathi), IIT Patna (Hindi), AU-KBC (English and Tamil). The annotation has been done at word level and the news articles after annotation are stored in XML format. The description of the XML tags are given below and the statistics of the XML tags is shown in Table 2.
<MAN_MADE_EVENT ID = ” number ” TYPE = ” s u b t y p e ” > E v e n t T r i g g e r </MAN_MADE_EVENT> <NATURAL_EVENT ID = ” number ” TYPE = ” s u b t y p e ” > E v e n t T r i g g e r </NATURAL_EVENT> Here MAN_MADE_EVENT and NATURAL_EVENT tag is related to Manmade disaster and Natural disaster event respectively, contains the event trigger and has the following attributes: 1. ID : A number which is unique for each event/tag in a given document. 2. TYPE : Represents subtype of the particular event (Manmade disaster or Natural disaster).
The event Manmade disaster has subtypes crime, riots, aviation hazard, accidents, train collision, vehicular collision, transport hazards, industrial accident, fire, normal bombing, terrorist attack, miscellaneous, shoot out, surgical strikes, suicide attack and armed conflicts. Language wise details statistics of subtypes of man made event XML tag shown in Table 3. The event Natural Disaster has subtypes forest fire, hurricane, cold wave, tornado, storm, hail storms, blizzard, avalanches, heat wave, cyclone, drought, heavy rainfall, limnic erruptions, lfoods, tsunami, land slide, volcano, earthquake, rock fall, seismic risk, famine, epidemic and pandemic. Language wise details statistics of subtypes of natural disaster event XML tag shown in Table 4.
The event arguments are casualties, reason, time of occurrence of event and location of event. The XML tags wrt each event argument is given below: 1. <CAUSUALITIES-ARG> : This tag contains the words that are casualties that have occurred due to an event. 2. <TIME-ARG> : This tag contains the words that are time at which the event has occurred. 3. <PLACE-ARG> : This tag contains the words that is the place at which the event has occurred. 4. <REASON-ARG> : This tag contains the words that are the reason due to which the event has occurred.
For example, the “casualties” attribute of an event is annotated as follows: <CASUALTIES−ARG ID = ” number ” > c a s u a l t i e s <CASUALTIES−ARG> Each argument tag of an event has the attribute “ID,” which is an unique number for each tag in a given news article.
An example, of annotation of man-made event news ”The accident occurred around 6.30 pm at Manathoor Church junction on the Pala-Thodupuzha State Highway.” is shown in Fig. 1 and an example annotation of natural event news ”An earthquake measuring 5.5 on the Richter
In both task 1 and task 2 the evaluation metrics that was used was F1-score. The F1-score was calculated separately for all the five languages in both Task 1 and Task 2. For Task 2 the F1 score was calculated separately for each argument in the event frame and then the score was averaged out. While evaluating the arguments in the event frame only exact string match of the values was considered. Eg: If the PLACE argument in test article is New Delhi and the output of the PLACE argument for test article given by the participant’s method is Delhi then it was not be considered as a match.
For the first task of Event Identification in English language, we received seven runs from five teams. For Hindi language we received five runs from three teams. For Bengali language we received six runs from four teams. In Marathi and Tamil language, for each we received two runs from two teams.
For the second task of Event Frame Extraction in English language, we received three runs from three teams. In case of Hindi, Bengali, Marathi and Tamil languages for each language we received one run from one team. The submission statistics are shown in Table 5. The results for all the five languages shown from Tables 6,7,8,9.
Team 3Idiots [
Team BUDDI_SAP 2 ranked second in both task in English language. They used DistillBERT based word embedding, POS tags based embeddings and character level embeddings which were then concatenated together to represent a word. This was then passed through Bi-LSTM the output of which passed through fully connected layer which was used to predict the words associated with an argument. Two separate models were trained for Task 1 and Task 2.
Run number 3,2 and 1 of team ComMA [
NLP@ISI 1 1 SR NO
Team Name
Run
Precision for text representation of all three languages, which was then fine tuned for Task 1.And in run number 3 BERT was used for text representation of all three languages, which was then fine tuned for Task 1.
Team MUCS [
Team NLP@ISI [
The FIRE 2020 EDNIL track was successful in releasing a multilingual dataset of Indian languages for event detection. As can be observed from the result tables for Task 1 barring English there is still lot of scope to improve the F1 scores for other languages. And for Task 2 there is still a huge scope for improvement in all languages. In the future we plan to extend the task by introducing event linking which will link one event to another if they are related to each other. For evaluation we intend to evaluate partial matching strings along with full matching strings. We also plan to introduce a summarization of event task wherein a summary of events within a particular time period will be generated and a short description of the events will be generated. However for this task annotators will be required who can create a gold standard dataset of event based summaries, which may require significant amount of time.
The track organizers thank all the participants for their interest in this track. We also thank the FIRE 2020 organizers for their support in organizing the track. We thank the Principal Investigator, Co-Principal Investigators and Host Institute (IIT Kharagpur) of ”A Platform for Crosslingual and Multilingual Event Monitoring in Indian Languages” for providing us with this opportunity of using the dataset in the track. We also thank Ministry of Electronics and Information Technology (MeitY) and Ministry of Human Resource Development, Government of India for providing this opportunity to develop the dataset and other resources.