IIT BHU at FIRE 2017 IRMiDis Track - Fully Automatic Approaches to Information Retrieval

Abstract

hTis paper presents the work of the team of IIT (BHU) Varanasi for the IRMiDis track in FIRE 2017. The task involved classifying tweets posted during a disaster into those expressing need and availability of various types of resources, given some tweets from the Nepal 2015 earthquake. We submited two runs, both of which were fully automatic.

Keywords

• Information retrieval
• microblogs
• disaster
• Lucene
• query generation

INTRODUCTION

With the increasing impact of social media, websites like Twiter which provide microblogging services have become increasingly popular. Apart from acting as a window to the outside world, these also serve as an important means to communicate and collect information, especially in times of emergency/disaster. The IRMiDis track in FIRE 2017 [5] posed a challenge to work on such data collection and analysis purposes. Specifically, the task was to develop IR methodologies to classify tweets as:

Need-tweets: Indicating the need or requirement of some specific resource such as food, water, medical aid, shelter, to name a few. Tweets pointing to scarcity or nonavailability of some resources also qualify for this category.

Availability-tweets: Informing about the potential/actual availability of resources. The former may be speaking about resources being transported, or food packets being delivered.

A tweet may be both a need-tweet and an availability-tweet.

We submited two runs, both of which were fully automatic i.e. no retrieval step involved manual intervention. Details of the runs are given in the subsequent sections.

DATA

hTe data contained around 70,000 microblogs (tweets) from Twiter that were posted during Nepal earthquake 2015, some of which were code-mixed, i.e., contained diferent languages and/or scripts. Around 20,000 of these were provided for development/training purpose and the remaining 50,000 for testing and evaluation. 3

OUR METHODOLOGY - RUN 1 hTe run is fully automatic in both query generation and searching. It makes use of Apache Lucene, a open source Java based text search engine library [1]. The run can be divided into the following steps: ( 1 ) Cleaning and Tokenization:

hTe tweets in the training data are first cleaned to remove hashtags, numbers, addresses (of the type @…) and URLs. These objects are not deterministic of the category (Nepal-Need/Nepal-Avail) a tweet falls in. Many hashtags (like #earthquake, #nepal, #NepalEarthquake) can appear in tweets of any category. Following this, the cleaned tweets are tokenized using the Standard Analyzer, which indexes documents after converting each token to lowercase, and removing stopwords and punctuations, if any. [4] The frequency of each token in the training set is then recorded. ( 2 ) Query Generation:

hTe token set of each category is modified to its set diference with the other token set. Then the queries for the 2 categories are generated as follows:

Nepal-Avail: Disjunction of tokens with frequency more than or equal to 3 given weight of their respective frequencies divided by 3.

Nepal-Need: Disjunction of tokens with frequency more than or equal to 2 given weight of their respective frequencies divided by 2.

hTe threshold frequencies are set in accordance with the number of tweets of each category present in the training set. ( 3 ) Searching and Scoring: hTe test set is also pre-processed and indexed like the training set in step 1. The test index is then searched for the queries generated in step 2. The scores are computed by Lucene. This scoring uses a combination of the Vector Space Model (VSM) of Information Retrieval and the Boolean model to determine how relevant a given document is to a user’s query. [3] ( 4 ) Categorization: hTe scores returned by Lucene are normalized to ( 0,1 ) and tweets having scores >=0.1 and >=0.2 are considered appropriate for the categories Nepal-Avail and Nepal-Need respectively. It is seen in our experiment that since tokens for Nepal-Avail are selected for a greater threshold frequency, the corresponding search query gives suitable tweets even on a lower score, hence the above diference. 4 OUR METHODOLOGY - RUN 2 hTis run is also a fully automatic one, i.e., no retrieval step required manual intervention.

hTe task was treated as a classification task, and SVM algorithm was applied, as implemented in the scikit-learn machine learning library [6]. hTe preprocessing included removal of tokens like ”RT”, URLs, and tokens starting with ”@” or ”#”.

Besides the provided code-mixed training data for this task, the gold standard from the FIRE Microblog Track 2016 was also used.

Undersampling was employed, i.e., only as many non-relevant tweets were given as input to the machine learning classifier as relevant tweets (since relevant tweets were much less as compared to irrelevant ones).

For dealing with code-mixed tweets, Google Translate [2] was used to convert tweets in other languages to English. Specifically, if the language field of the tweet metadata was ”hi” (which denotes Hindi) or ”ne” (for Nepali), the tweet was translated from its original language to English. For tweets in any other non-English language, they were assumed to be in Nepali (the most common non-English language) and were translated to English.

A threshold of 0.2 in the predicted score by the SVM classifier was set to classify a tweet as relevant.

MAP 0.3786 0.057

MAP 0.4986 0.1241

Average MAP MAP 0.4386 0.0906

5 RESULTS

hTe results of our runs based on several metrics are given in Table 1.

Our run with Run ID iitbhu_fmt17_task1_2 was the best-performing run in this task.

References

[1] [n. d.]. Apache Lucene Core. ([n. d.]) . https://lucene.apache.org/core/.

[2] [n. d.]. Google Translate . ([n. d.]). https://translate.google.com/.

[3] [n. d.]. Lucene Scoring . ([n. d.]). https://lucene.apache.org/core/3_6_0/scoring. html.

[4] [n. d.]. Lucene Standard Analyzer. ([n. d.]) . https://www.tutorialspoint.com/ lucene/lucene_standardanalyzer.htm.