In this paper, we explore three di erent ad hoc retrieval approaches to rank veri ed claims, so that those that verify the input claim are ranked on top. In particular, we deploy DPH Divergence from Randomness (DFR) term weighting model to rank the veri ed claims. In addition, we deploy the Sequential Dependence (SD) variant of the Markov Random Fields (MRF) for term dependence to re-rank documents (veried claims) that have query terms (input claim) in close proximity. Moreover, we deploy LambdaMART, which is a learning to rank algorithm that use machine learning techniques to learn an appropriate combination of features into an e ective ranking model.
Information posted on social media platforms such as Twitter is not often
factchecked by an authoritative entity before being published [
In this section, we present a brief but essential background on the di erent adhoc retrieval approaches used in our investigation. In particular, we start by providing a description of the DPH term weighting model in Section 2.1. This is followed by a description of the learning to rank techniques in Section 2.2. 2.1
For all our experimental investigation, we used the parameter-free DPH term
weighting model from the Divergence from Randomness (DFR) framework [
Learning to rank techniques are algorithms that use machine learning
techniques to learn an appropriate combination of features into an e ective
ranking model [
In this work, we deploy LambdaMART [
X t(fd) t2T (2) 3
FAQ Retrieval Platform: For all our experiments, we used Terrier-4.23 [8], an open source Information Retrieval (IR) platform. All the documents used in this study were rst pre-processed before indexing and this involved tokenising the text and stemming each token using the full Porter stemming algorithm [10]. We indexed the collection using blocks in order to save positional information with each term.
Training Learning to Rank Techniques: For our learning to rank approach,
we used the Terrier-4.2 Fat4[
T2-EN-UB ET-DPH: For all our runs, we used the parameter-free DPH
Divergence From Randomness term weighting model in Terrier-4.2 IR platform to
score and rank the documents (veri ed claims)
T2-EN-UB ET-DPH LTR: We used T2-EN-UB ET-DPH as the baseline
system. As improvement, we deployed a learning to rank technique. For our
3 http://terrier.org/
4 http://terrier.org/docs/v4.0/learning.html
learning to rank technique, we used the training and development tweets with
their qrels for training and validation. We used the Terrier-4.2 Fat framework to
retrieve 1000 documents for each topic (tweet) using the DPH term weighting
model, and then calculated several additional query dependent features in
Table 1. Using these features, we used Jforests to learn a LambdaMART model. We
then applied this learned model on the test tweets to generate a nal ranking.
T2-EN-UB ET-DPH MRF: We used T2-EN-UB ET-DPH as the baseline
system. As improvement, we deployed the Sequential Dependence (SD) variant
of the markov random eld for term dependence [
T2-EN-UB ET-DPH T2-EN-UB ET-DPH LTR T2-EN-UB ET-DPH MRF 0.843 0.818 0.838 0.868 0.862 0.865 0.873 0.864 0.869 0.840 0.300 0.185 0.815 0.307 0.186 0.835 0.300 0.184
Table 2 presents our evaluation results. The o cial evaluation measure for Task 2: Claim Retrieval is MAP@k, where k = 5. We also present Precision@k. The results of this study suggests that ad-hoc retrieval approaches such as term weighting models, proximity (Dependence) models and learning to rank techniques can be used to rank veri ed claims for a given check-worthy tweet. Overall, our primary submission T2-EN-UB ET-DPH LTR ranked third out of 10 submissions. It is worth noting that an attempt to improve the retrieval performance using a learning to rank technique resulted in a degradation in performance. An examination of the data revealed that for a majority of checkworthy tweets, there was a single veri ed claim. This lack o su cient training data could have resulted in the degradation in retrieval performance. For example, after performing a rst-pass retrieval with DPH and attempting to improve the ranking with our learned ranking model, some veri ed claims that verify the input claim ranked lower than in the previous ranking. 6
In this paper, three di erent ad hoc retrieval approaches were explored to determine their e ectiveness in raking veri ed claims so that those that verify the input claim are ranked on top. The results of this study suggests that term weighting models such as DPH can be used to rank veri ed claims for a given check-worthy tweet. In our attempt to improve the retrieval e ectiveness using a learning to rank technique, we noticed a degradation in retrieval performance. In future, we will explore using su cient training data in our learning to rank technique coupled with additional query dependent and query independent features. Similarly, re-ranking the veri ed claims using markov random eld for term dependence resulted in the degradation in performance. In our experiments, default parameter settings were used. Further research could usefully explore using different parameters settings such as varying the window size in order to improve the retrieval performance. 8. I. Ounis, G. Amati, Plachouras V., B. He, C. Macdonald, and Johnson. Terrier Information Retrieval Platform. In Proceedings of the 27th European Conference on IR Research, volume 3408 of Lecture Notes in Computer Science, pages 517{519, Berlin, Heidelberg, 2005. Springer-Verlag. 9. V. Plachouras and I. Ounis. Multinomial Randomness Models for Retrieval with Document Fields. In Proceedings of the 29th European Conference on IR Research, pages 28{39, Berlin, Heidelberg, 2007. Springer-Verlag. 10. M.F. Porter. An Algorithm for Su x Stripping. Readings in Information Retrieval, 14(3):313{316, 1997. 11. Shaden Shaar, Alex Nikolov, Nikolay Babulkov, Firoj Alam, Alberto BarronCeden~o, Tamer Elsayed, Maram Hasanain, Reem Suwaileh, Fatima Haouari, Giovanni Da San Martino, and Preslav Nakov. Overview of CheckThat! 2020 English: Automatic identi cation and veri cation of claims in social media.