In recent years, we witnessed an incredible amount of misinformation spread over the Internet. However, it is extremely time consuming to analyze the veracity of every claim made on the Internet. Thus, we urgently need automated systems that can prioritize claims based on their check-worthiness, helping fact-checkers to focus on important claims. In this paper, we present our hybrid approach which combines rule-based and supervised methods for CLEF-2019 Check That! Lab's Check-Worthiness task. Our primary model ranked 9th based on MAP, and 6th based on R-P, P@5, and P@20 metrics in the o cial evaluation of primary submissions.
Check-Worthiness Learning-to-Rank
With the fast developing information retrieval (IR) technologies and social media
platforms such as Twitter and Facebook, it is very easy to reach any kind of
information we are looking for and share it with other people. Therefore, any
information can be popular worldwide by the incredible sharing behavior of the
Internet users. As a worrisome nding, a recent study [
Obviously, false news can have many undesired consequences and be very
harmful for our daily life. Many researchers and journalists are trying to
minimize the spread of misinformation and its negative impacts. For example,
factchecking websites, such as Snopes1, investigate the veracity of claims spread
on the Internet and publish their ndings. However, these valuable e orts are
not enough to e ectively combat the false news because fact-checking is a very
time-consuming task, taking around one day for a single claim [
Considering the vast amount of claims spread on the Internet on a daily basis and high cost of fact-checking, there is an urgent need to develop systems that Algorithm 1 Claim Ranking Algorithm 1: Input: Training Data TrD 2: Input: Test Data TD 3: FT rD = extract f eatures(T rD) 4: model = train M ART (FT rD) 5: FT D = extract f eatures(T D) 6: ranked claims = test(model; FT D) 7: ranked claims = apply rules(ranked claims; RU LES) 8: return ranked claims assist fact-checkers to focus on the important claims instead of spending their precious time for less important claims.
In this paper, we present our method for CLEF-2019 Check That Lab!'s[
We propose a hybrid approach which uses both rule-based and supervised methods to rank claims based on their check-worthiness. Now we explain our approach in detail. 2.1
We rst rank claims using a supervised method and then rerank the claims
using a rule-based method. Algorithm 1 shows the steps in our proposed hybrid
approach. For the supervised ranking phase, we investigated logistic regression
(LR) and various learning-to-rank (L2R) algorithms including MART [
In the training dataset, we observed that many sentences contain phrases that
are not likely to be a part of a check-worthy claim such as thanks. In addition,
speaker of many statements are de ned as system and these statements contain
non-claim phrases such as applause indicating the actions of the audience during
the debate. Thus, in rule-based reranking phase, inspired from [
The features we use can be categorized into ve categories: 1) Named entities, 2) part-of-speech tags, 3) topical category, 4) bigrams, and 5) speakers. Now we explain our features used in our supervised methods.
Named Entities: Claims about people and institutions are likely to be check-worthy because people and institutions can be negatively a ected by those claims. In addition, a check-worthy claim should be also veri able. The location information, numerical values in claims are easy to verify because they provide unambiguous information. Therefore, detecting whether a statement is about a person or an institution, and existence of numerical values, location, and date information might be good indicators for check-worthy claims. Thus, we identify the named entities using Stanford Named Entity Tagger2 which tags entities such as person, location, organization, money, date, time and percentage. We use a binary vector of size 7 representing the presence of each named entity type separately.
Part-of-speech (POS) Tags: Sentences without informative words are less likely to be check-worthy. POS tags of words might help us to detect the amount of information in a sentence. For instance, a sentence without any noun such as "That is great!" does not contain any information that can be fact-checked. Thus, we detect POS tags in the statements using Stanford POS toolkit3. For this set of features, we again use a binary feature vector of size 36 in which each feature represents existence or absence of a particular POS tag in the corresponding statement.
Topical Category: Topic of a claim might be an e ective indicator for
check-worthy claims [
Bigram: Instead of using a large list of bigrams, we use a limited but powerful set of bigrams. In this set of features, we use bigrams that appear at least N times only in check-worthy or only in not check-worthy claims in the training set. We set N to 50 empirically based on our initial (not-reported) experiments, yielding 47 features. Some of these bigrams are "All right", "I know", and "go 2 https://nlp.stanford.edu/software/CRF-NER.html 3 https://nlp.stanford.edu/software/tagger.html 4 https://www.ibm.com/watson/services/natural-language-understanding/
Speaker List: A claim's check worthiness may depend on the person who makes it. Claims of in uential people are more important than claims of random people. In addition, some people might make check-worthy claims more frequently than others. For instance, in the training dataset, we found that 3.8% of Donald Trump's statements and 2.6% of Hilary Clinton's statements are labeled as check-worthy, showing that Donald Trump is more likely to make check-worthy claims than Hilary Clinton. Therefore, we use the 33 speakers appearing in the training data as binary features. 3
In this section, we present experimental results on both training and test data using di erent sets of features and machine learning algorithms.
The training and test datasets contain 19 and 7 les at varying lengths, respectively, where each le corresponds to transcription of a debate for the US 2016 presidential election. We perform 19-fold cross validation on the training data where each fold is a separate debate, and calculate mean average precision (MAP). We use Ranklib5 library in our L2R approach and Scikit toolkit6 for LR. We set the number of trees and the number of leaves parameters of MART to 50, and 2, respectively based on our initial experiments with di erent parameter con gurations. We use default parameters for LR.
We rst investigate the impact of each feature group using LR and MART. In particular, for each feature group we use all other features to build the model and observe how the performance would change without the corresponding feature group. Table 1 shows MAP scores for various feature sets. 5 https://sourceforge.net/p/lemur/wiki/RankLib/ 6 https://scikit-learn.org/ category features cause around 2.96%, and 2.28% decrease in the performance of MART, respectively. However, for LR, POS tags seem the most e ective features while named entities and topical categories have very small impact on the performance. Nevertheless, we achieve the best results when we use LR with all features (0.2198) while LR with All-fNamed Entitiesg, MART with All-fSpeaker List g, and MART with all features have also similar results.
For our primary submission, we selected MART algorithm with all features except speaker list. This is because its performance is very close to our best performing model and it does not use any speaker list which makes it more debate-independent model. For our contrastive-1 and contrastive-2 submissions, we elected MART and LR using all features.
For the evaluation on test data, we train our selected models with all training data. Table 2 presents o cial performance scores of all our submissions based on various evaluation metrics. 12 groups submitted 25 runs in total. Our primary submission ranked 9th among primary models based on MAP which is the o cial evaluation metric of the task. Our primary submission is also ranked 6th based on R-P, P@5, and P@20 metrics, among primary models. In this section, we take a deeper look into our results and conduct a qualitative analysis. We manually investigate the top 10 claims in our primary system's output in each test le. Table 3 shows a sample of claims with their labels and ranks in our system's output.
Comparing Row 1 and 2, while both of the statements are about the change in unemployment, one of them is labeled as check-worthy while the other one is not. We also observe a similar issue in statements shown in Row 3 and 4 where both statements are about the amount of trade and use similar words with similar sentence structure but their labels are di erent. This might be due to many reasons. First, check-worthy claims can be subjective, making the task even more challenging. Second, we might need background information about the claims because a claim which has been already discussed among people is more check-worthy than the one which is not discussed by anyone. The requirement of background information suggests that context the claim made and external resources such as web pages, social media posts about the claim can be useful to detect check-worthiness of claims.
Regarding the claims in Row 5 and 6, the statements contain veri able claims and their topic is about economics which can be considered as an important topic. In addition, the statement in Row 5 has also an explicit accusation regarding to an administration. However, they are both labeled as not check-worthy. These samples provide further evidence that detecting check-worthiness of a claim requires utilizing external resources and its context instead of just using the claim itself.
Regarding the claims in Row 7, 8, and 9, we can see that all three claims are about trade but only claim in Row 9 is labeled as check-worthy. This might be because the amount of money mentioned in Row 7 and 8 are not exact numbers, making them hard to verify. On the other hand, the claim in Row 9 mentions an exact number about the trade de cit. While our features take account whether a claim has a numerical value or not, this suggest that we have to handle special cases where numbers are not exact.
In this paper, we described our methods for CLEF-2019 Check That! Lab's Check-Worthiness task and discuss results conducting a qualitative analysis. We proposed a hybrid approach which combines rule-based and supervised methods together. We rst rank claims using MART algorithm, and then change the scores of some statements based on our hand-crafted rules. We investigated various features including topical category, POS tags, named entities, bigrams, and speakers of claims. Our primary model ranked 9th based on MAP, and 6th based on R-P, P@5, and P@20 metrics, among primary models. In our qualitative analysis we discussed that detecting check-worthiness is a subjective task and linguistic analysis of claims are not enough, requiring utilization of other resources to capture the background information about claims.
In the future, we plan to investigate linguistic features to capture the contextual information about the claims, and develop more e ective hand-crafted rules.