hTe FIRE 2017 Information Retrieval from Microblogs during Disasters (IRMiDis) track focused on retrieval and matching of needs and availabilities of resources from microblogs posted on Twitter during disaster events. A dataset of around 67,000 microblogs (tweets) in English as well as in local languages such as Hindi and Nepali, posted during the Nepal earthquake in April 2015, was made available to the participants. There were two tasks. The first task (Task1) was to retrieve tweets that inform about needs and availabilities of resources; these tweets are called need-tweets and availability-tweets. The second task (Task2) was to match needtweets with appropriate availability-tweets.
•Information systems →Query reformulation;
Various important information is posted on online social media
like Twiter at the times of disaster events such as floods and
earthquakes. However, this important information is immersed within
a lot of conversational content such as prayers and sympathy for
the victims. Hence automated methodologies are needed to extract
the important information from the deluge of tweets posted
during such an event [
Task 1: Identifying need-tweets and availability-tweets: Here the participants were asked to develop methodologies for identifying need-tweets and availability-tweets. Note that this task can be approached in diferent ways. It can be approached as a retrieval or search problem, where two types of tweets are to be retrieved. Differently, the problem of identifying need-tweets and availabilitytweets can also be viewed as a classification problem, e.g., where
In this track, our objective was to develop a test collection containing code-mixed microblogs for evaluating
Methodologies for extracting two specific type of actionable situational information – needs and availabilities of various types of resources (need-tweets and availabilitytweets), and Methodologies for matching need-tweets and availabilitytweets In this section, we describe how the test collection for both the tasks of IRMiDis track was developed. 2.1
As part of the same track in FIRE 2016, we had released a collection
of 50; 018 English tweets related to the devastating earthquake
that occurred in Nepal and parts of India on 25th April 20151 [
hTe gold-standard for both tasks was generated by ‘manual runs’.
To develop the gold standard set of need-tweets and
availabilitytweets, a set of three human annotators having proficiency in
English, Hindi and Nepali were involved. Additionally, annotators
were a regular user of Twiter, and had previous experience of
working with social media content posted during disasters. The
gold standard development involved similar three phases as
described in [
hTe summary of the number of need-tweets and availabilitytweets present in the final gold standard corresponding to three diferent languages is reported in Table 2. 2.3
To develop the gold standard for matching, the same human annotators were involved. The annotators were asked to inspect the gold standard for need-tweets and availability-tweets, and to manually find out the set of need-tweets for which at least one matching availability-tweet exists. The annotators were also asked to ifnd matching availability-tweets for each need-tweet. Additionally, pooling was used over the participant runs to identify relevant matches which the annotators might not have found. 3
11 teams have participated in Task1 and 18 runs were submited. A summary of the methodologies used by each team is given in the next sub-section. 3.1
We now summarize the methodologies adopted in the submited runs.
iitbhu_fmt17: This team participated from Indian Institute of Technology (BHU) Varanasi, India. It submited the following two Automatic (i.e. no manual step involved) runs. Both the runs used google translator API to convert the code-mixed tweets.
– iitbhu_fmt17_task1_1: It used Apache Lucene, a open source Java-based text search engine library2. Training data was indexed using Standard Analyzer and frequency of each token is training set recorded. Query is generated by the disjunction of tokens with frequency more than or equal to a threshold value. Tweets are categorized according to the score return by Lucene search engine. – iitbhu_fmt17_task1_2: It treated the task as a classification task, and used SVM algorithm. Undersampling was employed. A threshold of 0.2 in the predicted score by the SVM classifier was set to classify a tweet as relevant.
DataBros: This team participated from Indian Institute of Information Technology, Kalyani, India. It submited one automatic run described below: – iiests_IRMiDis_FIRE2017_1: The bag of words model was used with TfidfVectorizer to collect the features including unigram and bigrams. Recursive Feature Elimination (RFE) algorithm with LinearSVM was used to compute the ranking weights for all features and sort the features according to weight vectors. In addition, Decision Tree Classifier is applied to classify the data.
Bits_Pilani_WiSoc: This team participated from Birla Institute of Technology and Science, Pilani, India. It submitted two automatic runs. Both the runs were generated by using word embeddings and then fastText classification algorithm to classify the tweet to its appropriate category. hTe fastText classifier was trained on the labeled data and the previously created word embeddings.
– BITS_PILANI_RUN1: Created word embeddings using Skip-gram model. – BITS_PILANI_RUN2: Created word embeddings using CBOW model.
Data Engineering Group: This team participated from Indraprastha Institute of Information Technology, Delhi, India. It submited one automatic run – DataEngineeringGroup_1 described as follows: – DataEngineeringGroup_1: This run used Stanford CoreNLP library 3 for the POS tagging along with the lemma identification of all the words in the tweet set. Features were constructed using both the words present in the tweets and its POS tag. Logistic Regression model was used for this classification task.
DIA Lab - NITK: This team participated from, National Institute of Technology, Karnataka, India. It submited one automatic run described as follows: – daiict_irlab_1: This run used Doc2vec model to transform tweets into embedding vectors of size 100. To convert the code-mixed tweets, the ASCII transliterations of unicode text (tweet) was used. The frequency of each token available in a tweet is also used as the feature. These embeddings was the input for multilayer preceptron (a feed forward Artificial Neural Network model) for classification and w-Ranking Key Algorithm was used to rank the tweets.
FAST-NU: This team participated from, FAST National University Karachi Campus, Pakistan. It submited one automatic run described below: – NU_Team_run01: This run extracted textual features using tf*idf scores. All non -English tweets are translated using Google Translator API into English equivalent text. The logistic regression based classifier is used for classification.
HLJIT2017-IRMIDIS: This team participated from Heilongjiang Institute of Technology, China. It submited three automatic runs. The task was viewed as a classification task in all the runs and the feature selection was based on logistic regression method.
– HLJIT2017-IRMIDIS_task1_1: SVM of Liner kernel classifier was used. – HLJIT2017-IRMIDIS_task1_2: AdaBoost classifier was used. 3https://nlp.stanford.edu/software/tagger.shtml – HLJIT2017-IRMIDIS_task1_3: SVM of Nonlinear kernel classifier was used.
HLJIT2017-IRMIDIS_1: This team participated from Heilongjiang Institute of Technology, China. The task was viewed as a classification task in all the runs used words as a feature.
– HLJIT2017-IRMIDIS_1_task1_1: LibSVM classifier was used. – HLJIT2017-IRMIDIS_1_task1_2: LibSVM classifier was used. – HLJIT2017-IRMIDIS_1_task1_3: Linear Regression model was used.
Iwist-Group: This team participated from, Hildesheim University, Germany. It submited one automatic run Iwist_task1_1 that is described as follows. Pole-based overlapping clustering algorithm was used to measure the degree of relevance of the tweet. For ranking the tweets Euclidean distance was used as a similarity measure and the object closer to a pole was ranked higher.
Radboud_CLS Netherlands: This team participated from Radboud University, the Netherlands and submited the following two semi-automatic runs described as follows. Code-mixed tweets were preprocessed and translated to English using Google translator.
– Radboud_CLS_task1_1: A lexicon and a set of handcrafted rules were used to tag the relevant n-grams. hTen the class labels were automatically assigned to the tagged output. The output was initially ranked using combined score of human-estimated confidence of specific class label and tag patern. However, the ifnal ranking was generated by ordering the tweets within these ranked sets according to their tweet ID. – Radboud_CLS_task1_2: This run used a tool Relevancer for initial clustering of the tweets tagged as English or Hindi. English clusters were annotated and used as training data for the support vector machines (SVM) based classifier.
Amrita CEN 1: This team participated from Amrita school of Engineering, Coimbatore, India. It submited one semiautomatic run AU_NLP_1 described as follows. The training data was tokenized. Classifier was trained using the word count as feature. For ranking the tweets cosine similarity was used.
We now report the performance of the methodologies submited to the Task1 of FIRE 2017 IRMiDis Track. We consider the following measures to evaluate the performance – (i) Precision at 100 (Precision@100): what fraction of the top ranked 100 results are actually relevant according to the gold standard, i.e., what fraction of the retrieved tweets are actually need-tweets or availability-tweets, (ii) Recall at 1000 (Recall@1000): fraction of relevant tweets (according to the gold standard) that are in the top 1000 retrieved tweets, and (iii) Mean Average Precision (MAP) considering the full retrieved ranked list.
Run Id iitbhu_fmt17_task1_2 iiests_IRMiDis_FIRE2017_1
DataBros : This team participated from Indian Institute of Information Technology, Kalyani, India. It submited one automatic run. This run used POS (Parts of Speech) tagging and matching-score was obtained from the number Team Id
DataBros
0.2482 0.2081
of overlapping of common nouns between Need-tweets and Availability-tweets.
Data Engineering Group: This team participated from Indraprastha Institute of Information Technology, Delhi, India. It submited two automatic runs described as follows: – Both the runs used POS tag of nouns and similarity between Need-tweets and Availability-tweets were measured by cosine similarity. However, for the first submited run the similarity threshold was set as 0.7 as inferred on the basis of experimentation. Thus, brute force approach was followed in searching. – In the second submited run, greedy approach was followed and the search stopped as soon as it finds the first five or lesser availability tweets with a cosine similarity score greater than our set threshold of 0.7. HLJIT2017-IRMIDIS: This team participated from Heilongjiang Institute of Technology, China. It submited three automatic runs. The task was viewed as an IR task. All the runs used the open source retrieval tool Indri language model based on the Dirichlet smoothing for retrieval and KL distance as the sorting model.
HLJIT2017-IRMIDIS_1: This team participated from Heilongjiang Institute of Technology, China. It submited three automatic runs. The task was viewed as an IR task. Needtweets used as a query set and Availability-tweets used as a collection of documents. All the runs used Indri opensource retrieval tool and the Dirichlet smoothing language model to solve the matching problem. However, the three runs submited by this team difer in preprocessing step.
Radboud_CLS Netherlands: This team participated from, Radboud University, Netherlands, and submited the semiautomatic run Radboud_CLS_task1_1. This method used the tagged output obtained in the processing the tweets for Task 1 using a linguistic approach. For every Needtweet all the word n-grams were tagged as identifying a resource; the approach atempt to find an exact match in the Availability-tweets and ranked the Availability-tweets accordingly. 4.2
hTe runs were evaluated against the gold standards generated by manual runs. Additionally, the annotators (same as used to develop the gold standard) checked many of the need-availability pairs matched by the methodologies (after pooling), and judged whether the match is correct.
We have used the following IR measures to evaluate the runs. (i) Precision@5: Let n be the number of need-tweets correctly identified (i.e., present in the gold standard) by a particular matching methodology. For each need-tweet, we consider the top 5 matching availability-tweets as matched by the method. The precision of a particular matching methodology is the fraction of pairs that are matched correctly by the methodology (out of the 5 n pairs). (ii) Recall: The recall of matching is the fraction of all the needtweets (present in the gold standard) which a methodology is able to match correctly. (iii) F-Score: F-score of a matching methodology is the harmonic mean of the precision and recall.
Table 4 shows the evaluation performance of each submited run, along with a brief summary. For each type, the runs are arranged in the decreasing order of the F-Score. It is evident that the methods which considered noun overlapping or cosine similarity between need-tweets and availability-Tweets to obtain matchingscore (post POS tagging) outperformed the other methodologies. 5 CONCLUSION AND FUTURE DIRECTIONS hTe FIRE 2017 IRMiDis track successfully created a benchmark collection of code-mixed microblogs posted during disaster events. hTe track also compared the performance of various methodologies in retrieving and matching two pertinent and actionable types of information, namely need-tweets and availability-tweets. We hope that the test collection developed in this track will help the research community in the development of a beter model for retrieval and matching in future.
In this year’s track we considered a static collection of codemixed microblogs. However, in reality, microblogs are obtained in a continuous stream. The challenge can be extended to retrieve relevant microblogs from the live streaming of microblogs dynamically. We plan to explore this direction in the coming years.
hTe track organizers thank all the participants for their interest in this track. We also thank the FIRE 2017 organizers for their support in organizing the track.