The 2018 CLEF CheckThat! is composed of two tasks: (1) Check-Worthiness and (2) Factuality. We participated to task (1) only which purpose is to evaluate the check-worthiness of claims in political debates. Our method to achieve this goal is to represent each claim by a vector of ve computed values that correspond to scores on ve criteria. These vectors are then used with machine learning algorithms to classify claims as check-worthy or not. We submitted three runs using di erent machine learning algorithms. The best result we achieved using the o cial measure MAP ranks our run that uses non linear SVM the 12th over the 16 submitted runs. Our run that uses linear SVMis ranked 2nd with the Mean Precision@1 measure.
The CLEF CheckThat! rst task aims at predicting which claims in political
debates should be prioritized for fact-checking. All the background and detailed
information about the task are available on the task description paper provided
by the organizers of the task [
To achieve this goal, the task organizers released several textual transcripts of political debates with each sentence being annotated according to whether it is check-worthy or not.
This paper describes the participation of the Universite de Toulouse team (o cial name RNCC) at CLEF 2018 CheckThat! pilot task for check-worthiness.
We preprocessed the data by representing each sentence corresponding to a transcription of what a speaker said in the debate by a vector containing the score of this sentence for ve di erent criteria. We then trained three classi ers using these vectors to submit three di erent runs.
The remaining of this paper is organized as follows: Section 2 gives a description of the pilot task. Section 3 details the model we developed and the submitted runs. Then Section 4 details the results we obtained. Finally, Section 5 concludes this paper. 2 2.1
The Check-Worthiness task aims to predict which statements in a political
debate should be fact-checked. Indeed, nowadays, information objects are
spreading faster and faster on the Internet and especially on social networks. This
spreading is named the virality of the information [
During a political debate, any of the statements made by the participants can be reused without checking its factuality and it even can become viral. CheckThat! aims at providing journalists with a list of statements members of the debate made that should be checked before they are reused by others. 2.2
There are two datasets : one to train the model and one to test it. Both sets consist of political debates transcribed into texts.
They are annotated so that each row indicates the sentence number, the
speaker, the transcription of the sentence that the speaker said. The training
dataset includes in addition a label that indicates whether this sentence is to be
fact-checked or not. The training set contains three political debates while the
test set contains seven debates [
The task has been evaluated according to di erent measures. The o cial measure
is MAP which calculates the usual mean of the average precision. Then, other
measures were used as Mean Reciprocal Rank which allows to obtain reciprocals
of rank of the rst relevant document as well as Mean Precision at x which
performs the average of x best candidates. Details on the measures used can be
found in the task overview [
Evaluations are carried out on primary and contrastive runs. Primary run corresponds to the results le of the participant's main model ; the decision of the main run was the participant's decision. Contrastive runs match the secondary models the participant used. 3
We computed ve of the criteria from the Information Nutritional Label for
Online Documents proposed by [
(X posW ordV alue + X jnegW ordV aluej)=totalN umberW ords
{ Technicality : Determines the degree of technicality in a text. We count
the number of domain-speci c terms in the text. For that, we use NLTK9 [
https://spacy.io/ 6 https://en.wikipedia.org/wiki/Wikipedia:List of controversial issues 7 https://en.wikipedia.org/wiki/Anchor text 8 http://www2.imm.dtu.dk/pubdb/p.php?6010 9 Natural Language ToolKit, https://www.nltk.org/ 10 Regular Expression, https://docs.python.org/3/library/re.html keep those which appear more than once. We then calculate the ratio of the number of these NPs over the number of words in the text.
(X N P s)=totalN umberW ords
We decided to use only these criteria as features because our goal was to test the Information Nutritional Label on a concrete task. 3.1
Each of our three runs uses its own model to compute a check-worthiness score. For each of our models, we preprocessed the data using the criteria previously described. We computed the ve features for each sentence that has to be evaluated for check-worthiness. These sentences are then represented by a vector containing ve features, one for each criterion score.
For our INL SVM RBF (primary run) and INL SVM Lin ( rst contrastive) runs, we decided to use the Support Vector Machine in sklearn 11 with the probability setting set to "True". We used a RBF kernel for INL SVM RBF run and a linear kernel for the INL SVM Lin run. For our INL RF (second contrastive) run, we used the random forest classi er in sklearn.
To train our models, we used the three annotated debates provided by the clef2018-factchecking github repository12.
To obtain a score of check-worthiness, we computed the probability for each sentence to be check-worthy using the classi ers. The score of a sentence was then normalized by the highest score obtained for this sentence divided by the highest probability computed, so that the scores are between 0 and 1. 4
Seven teams submitted runs to this task for a total of 16 runs.
Table 1 presents the results of our three runs and the best submitted run
according to the MAP measure, which is from the Copenhagen team [
Overall, the INL SVM Lin run obtained better results than the INL SVM RBF run; that was somehow unexpected since non linear kernel have been shown to work better in other information retrieval applications. The INL SVM Lin run has been ranked twelfth according to the main measurement (Mean Average Precision), but obtained better rank when considering other measures: it is ranked fth according to the Mean Reciprocal Rank and second according to the Mean Precision@1. These ranks mean that our INL SVM Lin run would be good if the purpose of the task was nding the most check-worthy claim instead of nding all the check-worthy claims. However, we need to deeper analyse the results to understand why.
Post-hoc experiments showed that the least important criterion is Technicality. This may be due to the fact that the method we use to compute this feature was meant to work with large texts and it is not appropriate for a single sentence. The most important criterion is Emotion. We can assume that a claim has greater chances to be check-worthy if it is highly emotional. The speaker thinks less about what he says and it is more likely that his claims are not fully accurate. We will check this hypothesis in future work.
Table 2 presents the weight of the 5 features for our INL SVM Lin model. The weights of the features for our INL RF model are similar.
In this paper we proposed three models to solve the CLEF2018 CheckThat!
challenge (task 1 Check Worthiness) which deals with the evaluation of the
check-worthiness of statements in political debates. We used random forest and
support vector machine to learn models that make use of the Information
Nutritional Label features [
We are currently working on better calculation of the ve features. We would
like to complete the representations of the texts by using content-based
components like it is done in [
Finally, we will test these models on other datasets such as social networks. For example, we will consider a Twitter-based dataset where each tweet would have a score indicating its worthiness for fact-checking taking into account hashtags and tweet sources.