Since information can be created and shared online by anyone, a lot of time and e ort are required to manually fact-check all the information encountered by users everyday. Hence, an automatic factchecking process is needed to e ectively fact-check the vast information available online. However, gathering information related to every single claim can also be redundant, as not all sentences or articles are checkworthy. In this paper, we propose an e ective approach for retrieving check-worthy sentences within American political debates, which relates to the rst task of the CLEF CheckThat! 2019 Lab. To rank sentences based on their check-worthiness, we propose to represent each sentence using their mentioned entities using a TF-IDF representation. We use a SVM classi er to predict the check-worthiness of each sentence. Our approach ranked 4th out of 12 submissions. Our experiments show that the pronouns and coreference resolution pre-processing procedure we use as part of our approach does improve the e ectiveness of sentence checkworthiness prediction. Furthermore, our results show that entity analysis features provide valuable evidence for this task.
Nowadays, information is easily accessible online, from articles by reliable news agencies, to reports from independent reporters, to extreme views published by unknown individuals. Such amount of information may create di culties for information consumers as they try to distinguish fake news from genuine news. Indeed, users may not be necessarily aware that the information they encounter is false, and may not have the time and e ort to fact-check all the claims and information they come across online. Moreover, social media outlets are becoming increasingly important in everyday life, where users can obtain the latest news and updates, share links to news and information they want to spread, and post comments with their own opinions. With the amount of information that is created daily, it is not feasible for journalists and users to manually fact-check every news article, sentence or tweet online. Therefore, an automatic fact-checking system that extracts the most check-worthy claims from articles and debates could allow journalists to focus on manually checking suspicious but worthy claims, thereby reducing the workload required for the task.
The task of predicting the check-worthiness of each sentence in the text is
the objective of Task 1 of the CLEF CheckThat! 2019 Lab [
The focus of this paper is to e ectively address Task 1 of the CLEF
CheckThat! 2019 Lab. To do so, we build upon recent developments to improve a
chatbot's understanding in a conversation (a debate is form of conversation [
The rest of the paper is organised as follows. We brie y introduce related work in Section 2. Section 3 describes our proposed approach. We provide the experimental setup in Section 4, followed by the results and analysis in Section 5. Finally, we draw the main conclusions from this paper in Section 6. 2
Previous studies that focused on the task of predicting the check-worthiness
of a sentence are limited. ClaimBuster [
In last year's CLEF CheckThat! Lab, aside from the above mentioned
methods, team Prise de Fer [
However, the above mentioned approaches did not pay much attention to automatic pre-processing, in order to unify the pronouns and references. Moreover, although these approaches used named entities as features, none of these approaches used external resources to analyse the entities mentioned in the text. Our work focuses on these two parts of analysing sentences, to predict their check-worthiness. 3
The aim of the check-worthiness task is to rank the sentences, such that those sentences estimated most likely to be check-worthy are ranked rst. In addressing this task, we use a classi er based on several groups of features to estimate the check-worthiness of each sentence. Sentences are then ranked based on the classi er's con dence about the check-worthiness of each sentence. Our classi cation approach makes use of a pre-processing of the text that addresses pronouns and coreference resolution (described in detail in Section 3.1 below), as well as several groups of features, including some that consider the presence of entities within the sentences (Section 3.2).
Debate
D
First person
Pronouns
Resolution
American political debates usually consist of two or more participants, and one
or more moderators, where each debate has di erent participants. In this case,
it is not explicitly apparent to the system which participants are referenced
by which pronouns. Similarly, implicit pronouns can also be used to identify
a speci c person or a particular thing previously mentioned or known, leading
to a possible confusion. To combat the above mentioned challenges in implicit
references, we propose a two-step procedure to resolve the implicit references
found in the debates, namely, rst-person pronouns resolution, and coreference
resolution, as illustrated in Figure 1. Detailed examples can be found in Table 5.
1. First-person pronouns resolution: In this step, we simply change all the
rst-person pronouns in each sentence sn into the current speaker's name pn.
2. Coreference resolution: Coreference resolution is the task of nding the
entity expression that a pronoun refers to within a piece of text. In our proposed
procedure, we use coreference resolution to replace implicit mentions to one of
the previously stated real-world entities. Speci cally, we use Lee et al. [
Check-worthiness Estimation After the pre-processing procedure, we obtain an ordered (based on the order of the debate) list of sentences for each debate, where most pronouns are replaced with the person's name and/or entities. Note that the coreference resolution method cannot achieve a perfect accuracy, as only a subset of the actual pronouns are resolved.
Next, to obtain a ranking of sentences, we extract features from each sentence, and use these features as input to a classi er that is trained to estimate the check-worthiness of each sentence. Figure 2 shows the overall architecture of our proposed approach. Below, we describe two sources of features that we use to assist the check-worthiness estimation, namely TF-IDF sentence representation and entities analysis.
1. TF-IDF sentence representation: We calculate the TF-IDF score of each term in each sentence, where the IDF values are calculated over all of the Sentence
Entity extraction using Dbpedia spotlight Tokenization
Sentence, entity1, … entityn
Calculate similarity & relevance
of entities
TF-IDF representation
Combined, SVM estimate
Check-worthiness score training set. In particular, we use Sklearn's T dfVectorizer1 to extract features for each sentence. We do not discard any terms from the dictionary.
2. Entity analysis: Our second group of features concerns the entities that
appear in each sentence, obtained through entity linking occurrences in each
sentence to a knowledge graph (KG), namely Wikipedia. In particular, Wikipedia
is a large-scale online encyclopedia, where users can create articles related to
speci c entities, and can edit existing articles. The crowd-sourcing nature of
Wikipedia allows the entities' information to be updated quickly, which means
that the information is kept up-to-date. Wikipedia also contains structure
relationships where one or more entities are linked together through hyperlinks, such
as Polysemy (disambiguation pages), Synonymy (redirect pages) and
Associative relationships (hyperlinks between Wikipedia articles). Ciampaglia et al. [
2(a). Similarity of entities: We follow the method described by Zhu and
Iglesias [
2(b). Relatedness of entities: We extract the relatedness between two
entities using the method described by Witten et. al., [
log(jW j) log(min(jAj; jBj)
where a and b are two entities, and A and B are the sets of all the concepts that are linked to a and b. W is the whole set of concepts that appear in all of Wikipedia. jxj is the number of concepts that a given set x contains.
2(c). Count of entities: Finally, we also count the non-repeated entities that appear in each sentence, and use the number of the entities as a feature.
As some sentences contain more than two entities, we need to aggregate the similarity and relatedness of each pair of entities into sentence-level features. Therefore, we calculate the mean and max of the similarity and relatedness scores for the pairs of entities within each sentence. Overall, in addition to TFIDF term features, we therefore have additional 5 features for each sentence, as shown in Table 2. 4
In this section, we describe the used dataset, the settings of each component our approach, as well as the evaluation metrics.
Dataset: We use the training and test data provided by the CLEF CheckThat! 2019 lab as the training and test datasets, respectively. In the following, we describe in detail the experimental setup we used for the components of our approach:
First-person pronouns resolution and coreference resolution: As
mentioned in Section 3, we simply change all the rst-person pronouns (i.e.,
I, we, us, etc) to the current speaker's name. We use Lee et al.'s coreference
resolution package2 to nd the entity that a pronoun is referring to. All the
parameters are set to their recommended settings [
Tokenisation and TF-IDF: We use the Sklearn's T dfVectorizer3 to tokenise and calculate the TF-IDF features. All the parameters remain at their default settings.
Entities extraction: In our experiments, we use DBpedia Spotlight4 to
extract entities from each sentence, with the con dence set to 0.3 following [
Entities analysis: In our experiments, we use the Sematch [
Classi er: We tune the SVM classi er's hyperparameters on the training set. In particular, we use the RBF kernel, a C penalty of 10, and a of 0.1 in our tuned SVM classi er. Sentences are ranked in descending order by their distance from the classi er's hyperplane.
Evaluation metrics: To evaluate the e ectiveness of our approach at highly ranking check-worthy sentences, we use the evaluation metrics suggested by the CheckThat! lab organisers, namely Mean Average Precision (MAP), reciprocal rank (RR), and precision at k (P@k, k=f1,3,5,10,20,50g). 5
In this section, we address the usefulness of including the pre-processing procedure, as well as the e ectiveness of our classi cation model. In particular, we report and discuss the results of our sentence check-worthiness prediction experiments. Table 3 shows the e ectiveness of the top 5 ranked groups that participated in the Lab. Out of a total of 12 groups, our classi er was placed fourth group (ranked by MAP).
Next, to answer whether the pre-processing of data bene ts the check-worthiness prediction task, we conduct an ablation study, whereby we remove some of the components of our approach and assessed their resulting performance. Table 4 presents the results using three di erent variants of our approach: simple TF-IDF model with an SVM classi er, using the pre-processing procedure to
0.0826 0.2000 0.0000 0.0000 0.2000 0.2000 0.3500 0.1571 0.0956 0.2000 0.1667 0.1875 0.1471 0.1587 0.0985 0.0874 0.1263 0.3254 0.2857 0.2381 0.2000 0.2000 0.1287 0.0915 process the debate before using the TF-IDF features and the SVM classi er, and our full approach. The results show that the pre-processing procedure to address pronouns and perform coreference resolution improves MAP performance by 16% (0.0826 ! 0.0956). Furthermore, adding the entities features enhances MAP by a further 32% (0.0956 ! 0.1263). Thus, we conclude that the pre-processing procedure, as well as our entity-based features are promising and do improve the performance of our approach.
However, as the coreference resolution method cannot achieve a perfect accuracy, the results of our pre-processing procedure are not completely satisfactory. Table 5 shows a clip of one debate, where some sentences' pronouns and coreference resolutions are correct, some are missing, and some are incorrect. Moreover, we do not consider the types of entities in our entity analysis. Such type information may actually be informative, as the entity \the United States " may be less informative than \immigration" in an American political debate. 6
In this paper, we addressed a task that can be seen as the rst step towards factchecking internet content e ectively, as de ned by the CLEF CheckThat! 2019 Lab. In particular, we designed a pre-processing procedure, as well as a checkworthiness prediction model, to predict the check-worthiness of each sentence in a given debate. Our experiments showed that the pre-processing procedure, with pronouns resolution and coreference resolution, does improve the performance of the prediction system. Moreover, when using entities extracted and analysed using existing knowledge base tools, the performance of our prediction approach improved further. These ndings suggest that pre-processing can be bene cial when analysing text for check-worthiness prediction. They also show that entities analysis might be bene cial in the general fake news detection tasks. In the future, we propose to compare more machine learning methods, and enrich the language processing choices.
The rst authors acknowledges the support of the China Scholarship Council.