Microblogging sites like twitter are increasingly playing an important role in real time disaster management. There are many miscreants who would want to derail the rescue and relief operation by spreading rumors and thereby creating panic. Therefore, it becomes imperative to correctly identify such rumors and nip them in the bud. This paper describes our approach on “Identifying factual or fact checkable tweets” as an attempt in the shared task of the Microblog Track at Forum for Information Retrieval Evaluation (FIRE) 2018 [6]. Our approach uses a version of Stanford's POS Tagger [1] trained especially on tweets to extract features from the tweets for training the classifier. The system was evaluated on the Twitter dataset consisting of 50000 odd tweets provided by the FIRE 2018 shared task. Our system showed encouraging performance. We had submitted two separate runs, each using a different approach. The performance in each case is separately mentioned and explained.
Social media has become increasingly important in disseminating real-time
information in case of disaster outbreaks. Efficient processing of information from social
media websites such as Twitter can be challenging due to the noisy nature of the
tweets, but if pursued properly can be very helpful in disaster management. A lot of
research regarding extraction of situational information from microblogs during
disasters already exists [
A set of fifty thousand tweets were given and the task is to classify each tweet as either fact checkable or non-fact checkable. The tweets given in the task were posted during the Nepal earthquake in April 2015. 2.1
This section describes the dataset and resources provided to the shared task participants. The organizers provided a text file containing 50,068 tweet identifiers that were posted during the Nepal earthquake in April 2015. A Python script was provided to download the tweets using the Twitter API into a JSON encoded tweet file, which was processed during the task. A set of 80 fact checkable tweets was also given for testing the model. 3 3.1
The raw tweets from jsonl file were taken into a separate file. All the tweets were pos tagged using a special version of Stanford's POS Tagger trained just on tweets. These tweets were broadly classified into four files one containing only the tweets with Retweets(RT) in them, one with tweets containing numerical values in them, one with tweets containing more than 2 proper nouns and other containing the rest of the tweets. The first three files were carefully and minutely examined to further filter out the redundancies and repetitions.5000 tweets were selected from this corpus with 1500 from the first file,2000 from the second file and 1500 from the third. By this stage, we had a corpus of 5000 fact-checkable labelled data for training. We separately examined the fourth file with the same objective. 5000 tweets were handpicked from this generating a corpus of non-fact checkable tweets.
The stopwords were filtered out from each of the tweets using NLTK (Natural Language Toolkit).
A Bag of Words model was created with the 26000 proper-nouns(obtained by POS tagging of the tweets) as features present among all the 50000 tweets. Top performing 10000 features were selected using the SelectKbest class in the sklearn library and a "linear" SVM model was trained with these 10000 features. The model made predictions on the corpus of 80 tweets provided with 80% accuracy. The following results were obtained when the model was tested on all the 50,000 odd tweets.
A Bag Of Words model was created with 6000 features obtained using the TFIDF vectorizer available in the sklearn library. A linear SVM model was trained using these features. The model made predictions on the corpus of 80 tweets provided with 93% accuracy. The following results were obtained when the model was tested on all the 50,000 odd tweets.
Automatic 0.6400 0.1069 0.0340 0.0767 0.5237 0.5276
Automatic
In this paper, we presented a brief overview of our system to address the issue of fact check ability of microblogging data. As a future work, we would like to explore more sophisticated techniques to classify the microblogs according to their fact check ability so that we can minimize the menace of fake news and false rumors in catastrophic situations.