In countries like the US, European countries, Australia and Japan, user-generated content from microblogging sites is extensively used for crowdsourcing actionable information during disasters. However, there has been limited work in this direction in India. Moreover, there has been a limited attempt to verify the credibility of the information extracted from microblogs from other reliable sources. To this end, the FIRE 2018 Information Retrieval from Microblogs during Disasters (IRMiDis) track focused on the identi cation of factual or fact-checkable tweets and supporting news article for each fact-checkable tweets. The data consists of around 50; 000 microblogs (tweets) from Twitter and 6; 000 news articles, that were posted during the Nepal earthquake in April 2015. There were two tasks. The rst task (Task 1) was to identify factual or fact-checkable tweets and the second task (Task 2) was to identify supporting news articles for fact-checkable tweets.
Microblogging sites like Twitter are increasingly being used for aiding relief
operations during various mass emergencies. However, critical actionable information
is often immersed in the deluge of insigni cant conversational contents. Hence,
automated methodologies are needed to extract the important information from
microblogs during such an event [
fact-checkable ibnlive:Nepal earthquake: Tribhuvan International Airport bans landing of big aircraft [url] @mashable some pictures from Norvic Hospital *A Class Hospital of nepal* Patients have been put on parking lot. @siromanid: Many temples in UNESCO world heritage site Bhaktapur Durbar Square have been reduced 2 debris after recent earthquake [url]
non-fact-checkable
Table 1. Examples of fact-checkable and non-fact-checkable posted during a recent disaster event (2015 Nepal earthquake). techniques are needed to identify, process and verify the credibility of information from multiple sources.
To address the aforesaid issues, we organized the FIRE 2018 IRMiDis task. The track had two tasks, as described below.
Task 1: Identifying factual or fact-checkable tweets: Here the participants needed to develop automatic methodologies for identifying fact-checkable tweets. This is mainly a classi cation problem, where tweets are classi ed into two classes fact-checkable tweets and non-fact-checkable tweets. However, apart from classi cation, the problem of identifying fact-checkable tweets can also be viewed as a pattern matching problem or an IR problem. Table 1 shows some examples of fact-checkable tweets and non-fact-checkable tweets from thedataset that consists of about 50,000 tweets posted during the 2015 Nepal earthquake (details of dataset given in Section 2 ).
Task 2: Identi cation of supporting news articles for fact-checkable tweets: A fact-checkable tweet is said to be supported/veri ed by a news article if the same fact is reported by both the media. In this task, the participants were asked to develop methodologies for matching fact-checkable tweets with appropriate news articles. Table 2 shows some examples of fact-checkable tweets and extracts from news article that veri es/supports the fact reported in the tweet, posted during the 2015 Nepal earthquake from the dataset that was made available to the participants (described in the next section).
For each fact-checkable tweet, participants should report - (i) the supporting news article id, and (ii) the particular sentence in the news article, which supports the fact-checkable tweet. It should be noted that many of the fact-checkable tweets might not have supporting news articles in the dataset. Examples of Fact- Headline of Extract from supporting Url of news site checkable tweet news article news article ibnlive:Nepal earth- President Sirisena Tribhuvan International Air- http://news rst.lk/english quake: Tribhuvan expresses condo- port in Katmandu, Nepal has /2015/04/president-sirisenaInternational Airport lences to earth- been closed for all commercial expresses-condolences-tobans landing of big quake victims in ights. Only ights carrying earthquake-victims-inaircraft [url] Nepal relief are allowed into its run- nepal/91798 ways. #Nepal #Earthquake Protests over Four days after the deadly http://www.businessday four. Slowly in poor relief as quake, more shops opened standard.com/article/newsthe capital valley In- Nepal toll crosses here and tra c returned to ians/protests-over-poorternet and electricity 5,000 (Roundup) Kathmandu's roads. Author- relief-as-nepal-tollbeing restored. A re- ities also restored electric- crosses-5-000-rounduplief for at least some ity while telephones began to 115042901022 1.html ones function in more areas.
Many temples in UN- Nepal earth- Historical monuments such http://www.businessESCO world heritage quake: Over 1,800 as Dharhara and Basanta- standard.com/article/newssite Bhaktapur Dur- dead pur Durbar Square and Patan ians/nepal-earthquakebar Square have been Durbar Square have been over-1-800-deadreduced 2 debris af- completely destroyed by the 115042600075 1.html ter recent earthquake tremors [url] @MSF canada: DATE: We're sending 8 to Nepal ing highly emergency teams[url]
UP- US Pledges $1 Doctors Without Borders sent http://www.breitbart.com now Million, Relief eight medical teams and four /nationalteams Teams to Nepal arrived the same day. The security/2015/04/26/usinclud- After Earthquake teams include a surgical team pledges-1-million-reliefskilled composed of eight highly teams-to-nepal-aftersurgical skilled MSF sta members earthquake/ to set up surgical units and mobile clinics In this track, our motivation was to develop a test collection containing microblogs for evaluating{ { Methodologies for identifying speci c type of actionable situational information { factual or fact-checkable tweets, and { Methodologies for identi cation of supporting news articles for fact-checkable tweets The detail description of the test collection development procedure of IRMiDis track is described in this section. 2.1
In the present task, we included both microblogs (tweets) and news articles in our
dataset. We reused the tweet collection of 50; 018 English tweets related to the
Nepal earthquake that occurred on 25th April 20155 developed and released as
part of the same track in FIRE 2016 [
Additionally, we introduced a collection of 6; 000 news articles, that were posted during the Nepal earthquake in April 2015. We used Radian6 tool6 to search for news articles posted during the two weeks after the earthquake, using the query term `nepal'. The dataset contains tweets/articles in English only. 2.2
We employed pooling for the current task. We pooled top 100 results from each run and involved a set of three human annotators having pro ciency in English, who are regular users of Twitter, and had previous experience of working with social media content posted during disasters. We asked the annotators to judge the fact-checkability of the tweets and independently. Annotators observed that there were di erent types of fact-checkable tweets, based on how de nite the factual information reported in the tweet is. Hence we decided to adopt a graded gold standard. The graded gold standard development process is as described below { Annotators were asked to grade the fact-checkable tweets in three levels and scores were assigned as 1, 2, 3 depending on the de niteness of facts reported in the fact-checkable tweets.
{ Grade 1: Grade `1' depicts tweets are containing factual information but without Nepal-related information i.e., about some location outside Nepal { Grade 2: Grade `2' signi es fact-checkable tweets containing Nepal-related information. However, the factual information is generic and very de nite i.e speci c resource name, quantity, organizations are not reported by the tweet { Grade 3: Grade `3' signi es highly fact-checkable tweets having speci c a reference of source, location, organization, quantity, resource name etc. The grade of the rest of the tweets is assigned as 0, that signi es tweets are nonfact-checkable. The nal set of graded tweets in di erent categories was decided through a mutual agreement among all three annotators.
The summary of the numbers of graded fact-checkable tweets present in the nal gold standard is reported in Table 3 along with the example of each category of tweets. 2.3
In Task-2 of IRMiDis track this year, only one run was submitted. Thus, pooling was employed on only one run to create the gold-standard. By checking the overlapping with the gold standard developed as a part of Task 1 (Identifying fact-checkable tweets) it was noticed that the supporting new-articles were reported against 40 correctly identi ed fact-checkable tweets (according to the gold standard of Task 1) only. Hence, human assessors were employed and asked
of Count Examples Category tweets
to manually inspect each of the relevant fact-checkable tweets and corresponding matching news-article reported by the run and decide whether matching is correct or not. Accordingly, the evaluation metrics were calculated. 3
Task 1: Identifying fact-checkable tweets In IRMiDis track this year 6 teams participated in Task 1 and nine automatic and three semi-automatic runs were submitted. The di erent methodologies developed by the participating teams are summarized and described in the following sub-section. 3.1
{ MIDAS: This team from Indraprastha Institute of Information Technology Delhi (IIIT-D) submitted the one semi-automatic and one automatic run. For both the runs, tweets were pre-processed by removing punctuations, stop-words and emojis. Hence, pre-processed tweets were POS tagged.
MIDAS 1 (automatic): The run used proper nouns and numbers present in the tweets as features. Factuality score was calculated by the average of two scores namely PROPN and NUM, where PROPN is the proper noun count in each tweet and NUM is the count of numbers present in each tweet. The scores were normalized by dividing each of the counts by maximum count of the corresponding features across the dataset. MIDAS 2 (semi-automatic): Around 1500 tweets were manually labeled to train the classi er, features were extracted using cbow and bi-gram models then fastText classi cation algorithm was used to classify the tweets. Tweets were ranked according to the con dence score provided by the classi er. { FAST-NU: This team partook from, FAST National University Karachi Campus, Pakistan. It formulated the task as an Information Retrieval problem and submitted three automatic runs. It used the set of 6000 news articles introduced as a part of the dataset in the current track. It considered both string similarity( ) and cosine similarity( ) to rank the tweets. It used AhoCorasick algorithm to compute string similarity( ). Details of runs were illustrated as below:
FAST NU Run1 : Computed between the tweet and news headlines and between news content and tweets FAST NU Run2 : This run computed between hashtags extracted from tweets and news headlines FAST NU Run3 : Combination of previous two approaches as described above. A tweet was considered as a factual tweet if it has a supporting news article both from FAST NU Run1 and FAST NU Run2. { UEM-DataMining-CSE : This team from University of Engineering and Management, Kolkata, India, submitted two automatic runs. Tweets were pre-processed and POS tagged 7to extract proper nouns from the tweets. Both the runs were generated by using SVM classi er with linear kernel to classify the tweet. Used bag-of-words as feature extraction algorithm.
UEM DataMining CSE run1: SelectKbest feature selection algorithm was used to select top perfoming 10000 proper nouns as features.
UEM DataMining CSE run2 : Used T dfVectorizer algorithm to select
top-ranked 6000 proper nouns (according to tf*idf score) as features .
{ iitbhu irlab irmidis hm: This team is from Indian Institute of Technology
(BHU) Varanasi, India. It submitted the one automatic run described as
follows:
iitbhu irlab irmidis hm r1: This run trained word2vec model with 50; 000
pre-processed tweets in default settings. Created a tf*idf based ranked
list of terms from 84 ground truth tweets provided as a part of the
dataset in present task. Hence, a weighted function of tf*idf score and
word-embedding was used to rank the tweets.
{ DAIICT-Hildesheim: This team participated from, Hildesheim
University, Germany and Dhirubhai Ambani Institute of Information and
Communication Technology, Gujrat, India. It submitted threeautomatic runs.
Tweets were pre-processed by removing @string value, RT, and URLs. Among
these rst two runs (DAIICT-Hildesheim-mod1-sif,
DAIICT-Hildesheim-mod1nosif): used Recursive Neural Network based approach to obtain the
semantic label of the tweets using Stanford semantic analysis library [
DAIICT-Hildesheim-mod1: Sentence vector was computed by taking sum of all the term vectors present in a sentence and then dividing by the length of the sentence to take the average. Hence, rst principle component is multiplied with each sentence vector.
DAIICT-Hildesheim-mod1: Sentence vector is calculated in the same way as of the rst run( DAIICT-Hildesheim-mod1-sif).
DAIICT-Hildesheim-mod3 (semi-automatic): This run used a Convolution Neural Network based classi er. CNN was intialized with the GloVe pre-trained vectors. The classi er was trained with 1700 tweets labeled as subjective/non-fact-checkable tweets and 2000 tweets labeled as objective/fact-checkable tweets with 10-fold cross validation. { iitbhu irlab irmidis ab: This team participated from Indian Institute of Technology (BHU) Varanasi, India. It submitted one Semi-Automatic run described as follows: iitbhu irlab irmidis ab 2 (semi-automatic): This run trained a doc2vec model for representing each tweet as a 50 dimension vector. Manually observed the datasets and randomly labeled few fact-checkable tweets. Used crystallization of the dataset. SVM Classi er is used to classify the tweets. 3.2
The performance of the methodologies submitted to the Task 1 of FIRE 2018 IRMiDis track are illustrated in this section. We considered NDCG as the primary measure for evaluation. Ranking of runs are based on NDCG scores. However, we noted the following measures as well to evaluate the performance { (i) Precision at 100 (Precision@100): what fraction of the top ranked 100 results are actually relevant according to the gold standard, i.e., what fraction of the retrieved tweets are actually fact-checkable tweet (ii) Recall at 1000 (Recall@1000): fraction of relevant tweets (according to the gold standard) that are in the top 1000 retrieved tweets (iii) NDCG at 100 (NDGC@100): considering ranking upto top 100 retrieved tweets (iv)) NDCG (NDCG Overall): considering the full retrieved ranked list Table 4 reports the retrieval performance for all the submitted runs in Task 1. Each of the measures (i.e. Precision@100, Recall@1000, NDCG@100, NDCG Overall) are reported.
It is observed that simple NLP and classi cation based approaches performed better than the other methodologies based on word-embeddings as is evident from the Table 4. 4
Task 2: Identi cation of supporting news articles for fact-checkable tweets In Task 2, one team participated and one semi-automatic run was submitted. Description of the run is as follows{ Run Id
Type MIDAS 1 Automatic 0.8800 0.1292 0.5649 0.6835 POS tagging, Normalized sum of proper noun count (PROPN) & number count (NUM) FAST NU Run2 Automatic 0.7000 0.0885 0.5723 0.6676 String similarity between hashtags & news headlines , Cosine similarity, AhoCorasick algorithm UEM DataMining Automatic 0.6800 0.1427 0.5332 0.6396 POS tagging, T dfVectorizer, CSE run2 SVM classi er (linear kernel) UEM DataMining Automatic 0.6400 0.1069 0.5237 0.5276 POS tagging, SelectKbest feature selecCSE run1 tion algorithm, SVM classi er (linear kernel) iitbhu irlab irmidisAutomatic 0.9300 0.1938 0.8645 0.4532 tf*idf score & hm r1 FAST NU Run1 FAST NU Run3 DAIICTHildesheim-mod1 DAIICTHildesheim-mod2 DAIICT- SemiHildesheim-mod3 Automatic MIDAS 2 iitbhu irlab irmidis ab 2
SemiAutomatic SemiAutomatic
Crystallization,SVM Classi er
Methodology iitbhu irlab irmidis hm : This team is from Indian Institute of Technology (BHU) Varanasi, India. It submitted one Semi-Automatic run described as follows: It utilised Apache Lucene, a open source Java-based text search engine library8.News articles and tweets were pre-processed by stopwords, hashtags and addressing removal, stemming (porter stemmer) and case-folding. Then, headline and the rst three sentences of each news article were combined to form the test documents and each pre-processed tweet was used as a query. Tweets were categorized according to the score returned by Lucene search engine. 4.2
Only one run was submitted in Task 2 and the run could retrieve only 40 factcheckable tweets according to the gold standard developed for Task 1. We employ pooling, though on only one run. Thus the human annotators only checked the relevance of the news articles retrieved for these 40 tweets. Thus, for each of the 40 fact-checkable tweets identi ed, total how many news articles were identi ed, and out of that how many were judged to be correct (i.e, the news article sentence that was retrieved actually veri ed the information contained in the tweet) needed to be evaluated. Hence, we have evaluated the run according to the measure Precision@N described as below: Precision@N: for each fact-checkable tweet, out of N retrieved supporting articles, how many are correctly identi ed.
The performance of the submitted run is evaluated as 0.9378 (Precision@N). It is evident that term-based matching could produce good result. However, it is to be noted that the result is evaluated only for 40 fact-checkable tweets. Hence, it may be concluded that those fact-checkable tweets were easy to match. For rest of the tweets other methodologies needs to be explored. 5
Conclusion and Future Directions The FIRE 2018 IRMiDis track successfully created a benchmark collection of fact-checkable tweets posted during disaster events with graded relevance. The track also compared the performance of various methodologies in identifying fact-checkable tweets and matching the fact-checkable tweet with supporting news articles. We hope that the test collection developed in this track will help the research community in the development of a better model for retrieval and matching in future. Moreover, Task 2 did not have much participation this year. Hence, the problem of matching the fact-checkable tweet with supporting news articles needs to be explored more in subsequent years.
Acknowledgements The track organizers thank all the participants for their interest in this track. We acknowledge our annotators for their sincere e ort in developing the ground truth. We also thank the FIRE 2018 organizers for their support in organizing the track.