This paper presents details of the work done by the team of IIT (BHU) Varanasi for the IRMiDis track in FIRE 2018. The task involved classifying tweets posted during a disaster into those which are fact-checkable or factual and which are not, and also match these tweets to relevant news articles. Methodologies had to be developed in context of the 2015 Nepal Earthquake.
The methodology for the rst sub-task i.e. identi cation of fact-checkable tweets is fully automatic in both, query generation and searching. The key steps are indicated as follows: 1. Pre-processing of all tweets by lower-casing, removal of stopwords, hashtags and addressing and nally stemming using porter stemmer. The term tweet hereafter refers to the pre-processed tweet. 2. Creating a TF-IDF based ranked list of terms in the reference set of 84 tweets. Only those terms are considered that occur in the reference set more than once. We call this set of terms R with the TF-IDF score function being T . 3. A word2vec word embedding model is trained on the entire set of 50,000 tweets. 4. Each test tweet is now attributed to its corresponding feature vector that is formed by an arithmetic mean of the sum of the individual terms embeddings. 5. To form the reference feature (V ) vector against which they will be matched, we use the following weighted mean:
V =
PjiR=j1 T (Ri)E(Ri)
PjiR=j1 T (Ri)
(
The methodology for the second sub-task is manual in query generation and automatic in searching, scoring, using the Java based text search library Lucene. Details of the constituent steps are as follows: 1. The news articles are pre-processed in the same way as tweets are in subtask 1. 2. The headline and rst 3 sentences of each news articles are combined.
This creates one testing document for each news article. 3. Now each pre-processed tweet is used as a query to match with the testing documents of the news articles. This done using Lucene and the score of the best matching document is seen for each tweet. 4. If this score is more than 0.30, the corresponding news article is said to be matching the tweet, otherwise no relevant news article is said to be found for the tweet. 5. To nd the matching sentence, the tweet as a query is matched with each sentence of the relevant news article. The sentence with the highest score is returned as the answer. 3
The results on the two sub-tasks, based on di erent metrics are indicated in Table 1 and 2. 4
Depending on the kind of data, a sentiment analysis module can be augmented in the classi cation pipeline. However since such system should be ready to use for a disaster when it happens, the weight of such an additional module can be found as a hyperparameter by studying data from such incidents that have already occurred.