This paper presents the methodology employed by the author for IRMiDis Track in FIRE 2018. The task comprised of two subtasks, rst, classifying whether a tweet is fact check-able or not and second, to identify the news article that would support or verify the tweet. Two runs were submitted for the rst subtask only. { Fact Uncheckable tweet: We humans need to come up with a strong solution to create earthquake proof zone's ? Supported by IR-Lab, Dept. of Computer Science and Engineering, IIT (BHU), Varanasi
Micro-blogging platforms, such as Twitter, have become in recent years, an important medium for information disbursement. The advantage of such platforms is that the source is omni-present, the people all around the world. It doesn't depend on any organisation to release the information. Due to this fact and its widespread availability, such platforms can be leveraged in times of emergency to gather information critical to relief/rescue operations.
The downside of this platform is that people also use it to spread rumours or false and misleading information. If rescue operatives are depending on a system that retrieves information from this platform, it becomes imperative to ensure that the information is genuine and free from bias.
To this end, this year's IRMiDis Track in FIRE 2018 [
Subtask 1 - Identifying fact check-able tweets This subtask aimed at separating the given 50,000 tweets into two classes, if viewed as a classi cation problem or as an Information Retrieval problem that would generate a ranked list, with higher ranked tweets being more fact checkable. As an example: { Fact Checkable tweet: #Nepal #Earthquake day four. Slowly in the capital valley Internet and electricity beeing restored . A relief for at least some ones
Subtask 2 - Identi cation of supporting news article This subtask aimed at identifying the news article(s) that would act as evidence for the tweet deemed as fact checkable. The dataset contained 6000 news articles for the same event. 1.2 2
The dataset consisted of 50,000 tweets (microblog postings from Twitter) posted during the Nepal Earthquake of 2015 and 6,000 news articles published during the same time. A separate but small set of sample fact checkable tweets was also provided for training. The tweets were all in English. More training data was generated from within this dataset as part of the methodology applied. 3
The task was treated as a PU (positive-unlabelled) classi cation problem (as
described in [
Each line of the new le corresponds to a tweet as below: tweet id : cleaned tweet text (b) Using more tweet data from unrelated dataset and the gensim library for python, a doc2vec model was trained to represent each tweet as a 50 dimension vector (c) Manual intervention - manually observed all the provided tweets and randomly added few apparently fact checkable tweets from JSONL le to given small positive set 2. Calculated mean vector for given sample fact checkable tweet (SFCT) dataset - also calculated the max. euclidean distance any of these vectors had with the mean (max euc) and the max. angle in radian any of these vectors had with the mean (max angl). 3. Using thresholds max euc and max angl, did an ad-hoc classi cation of the JSONL dataset; out of 50,000 tweets - around 1400 tweets were declared as reliably negative (had distance and angle greater than the thresholds from the SFCT mean.)
Measuring Fact Check-ability In Microblogs 4. Used crystallization (involved: original SFCT as seed and adding new tweets to the set in rst iteration, then using new new SFCT as new seed in next iteration and so on) of the SFCT dataset using an angle threshold of 0.05166 over 7 iterations to grow the SFCT dataset - this will act as reliably positive (around 1500 tweets). 5. Using positive and negative examples generated at 4 and 3 above, an SVM Classi er was trained; accuracy of around 57.16% was obtained. (Accuracy of the classi er was measured on the original SFCT dataset.) 6. Using the above classi er, labels were predicted for the entire JSONL dataset.
Two les were generated, one for fact checkable (fc) and another for fact uncheckable (fuc). 7. The fc le was put through the crystallization process for 5 iterations to grow its size, since a very small number of tweets were classi ed as positive by the classi er. The fuc le was simply populated as the complement of fc le w.r.t. the entire JSONL dataset. 8. For generating the nal ranked result (positive tweets ranked higher than negative): (a) Both les were sorted separately on score and appended in as: rst positive then negative nal result (b) Score was the angle in radian; each tweet in fc and fuc les make some angles with all the tweets in the SFCT dataset; the least angle amongst these was taken as the score for the concerned tweet.
Contribution: In this semi-automatic method, the step that was introduced and apparently not generally used is the crystallization step at 4 and again used at 7. Since, the available set was extremely small for good training, this step was used to increase the size of training data. 4
This submission was a semi-automatic approach as discussed above. The results for this methodology may be viewed in Table 1 for Run ID iitbhu irlab irmidis ab 2. The `NDCG Overall' was considered as the nal measure of performance during evaluation.