With the uncontrolled increasing of fake news, untruthful claims, and rumors over the web, recently di erent approaches have been proposed to address this problem. In this paper, we present a credibility detector of factual claims in presidential debates. Our approach captures the distribution of the results from the search engines to infer the credibility of the claims. We participated in the CLEF-2018 Check That lab for Task 2, obtaining acceptable results.
A massive amount of information is spread on the web. One of the main
disadvantages of this data growth is the uncontrollability of their veracity. The
existence of social media networks has helped the increase of untruthful news.
Recently, di erent attempts have addressed this issue to propose solutions for
fact checking, for instance for presidential debates. During the debates, multiple
claims are made by the candidates about previous facts. Some of these claims
could be untruthful: the claimer makes it in an attempt to weaken the other
candidate. These untruthful claims pose a real risk on the elections results. In
this paper, we present our approach for detecting the factuality or of claims
in the presidential debates, where in [
This task concerns with investigating claims veracity in presidential debates. Therefore, a set of presidential debates from the US presidential elections are presented. In our approach, we used results from search engines to infer the veracity of the claims. The idea behind our approach is to infer the veracity by measuring the similarity between the claims and the search engines results, with extracting the results' sources reliability. Also, we modeled the distribution of these features in each search engine. In the following section, we present an overview of the literature. In Section 3, we present our approach, with giving a view on the task. The experiments with the results are presented in Section 4. Finally, in Section 5, we draw some conclusions. 2
Researchers in the literature studied several features from di erent aspects to
address this research problem. These features addressed di erent characteristics
of the claims on the web. Authors in [
As we mentioned, this task concerns with the claims from the presidential
debates. The claims that are unworthy for checking have not been annotated and
kept in the debates to keep the context. Factual claims have been tagged as True,
False, and Half-True. These debates are provided in two languages: English and
Arabic, where the Arabic text is translated from the original English debates.
The dataset that was provided is imbalanced, where the total number of factual
claims is 81; claims as True: 19, Half-True: 22, False: 41. The task goal is to
detect the credibility of the provided claims. The macro F1 score was used as
the performance measure. More details of the task are mentioned in [
Hypothesis: Factual claims have been discussed and mentioned in online news agencies. In our approach, we used the distribution of these claims in the search engines results. Furthermore, we supposed that truthful claims have been mentioned more by trusted web news agencies than the untruthful ones. Therefore, our approach depends on modeling the returned results from search engines using similarity measures and with extracting the reliabilities of the results' sources (dependent features). Also, we captured the distribution of these previous features from each search engine (independent features).
At the beginning, we started to reformulate each claim into a query. We fed this query to the Google and the Bing search engines to obtain a set of results. In our approach, we use only the returned snippets and we do not investigate more the original web pages. Given the search engine results, we used in our approach the rst N results for the feature extraction. Next, we built the representation of the features: Independent features: For each returned result, we extracted the following three features:
Cosine over embedding: we used pre-trained Google News word2vec
embedding to measure the cosine similarity between each snippet and the
query. We used the main sentence components, discarding the stopwords.
In the same way, we built this feature for the Arabic language, but we
used fastText pre-trained embedding [
AlexaRank : For each result, we used Amazon Alexa Rank to retrieve the rank of its site. The sites that have lower values are the sites that have higher reliability.
Text similarity : we used another text similarity measure, but using the
full sentence components (similarity over tokens), and without text
embedding. For the English part, we used the Spacy python library4, while
for the Arabic language, since this library is not available, we implemented
the text similarity approach that used in [
Dependent features: We extracted a set of features based on the previous independent features. These features model the distribution of the previous feature set, that has been extracted using Average (Avg) and Standard Deviation (Std).
Avg and Std of AlexaRank feature: We computed both Avg and Std features for the Alexa values that were extracted for the rst N results. Avg and Std of the Cosine over embedding feature: Similarly, we computed also the Avg and the Std features for the cosine similarities values that were extracted. 4 https://spacy.io/, visited in May 2018 At the end, our representation has (3 N) + 4 features.
All of these previous dependent and independent features were extracted twice, one from the Google and another one from the Bing search engines. In the following section, we will investigate their importance. 4
As we mentioned in the previous section, we built our features based on the rst N results from the search engines. Experimentally, we found that choosing the rst 5 results (N=5) has produced the highest results. Based on that, our feature vector length is 38 features.
In Figure 1, we show the information gain of these features for each search engine. From these results, we can infer that the features that were obtained by the Google search engine are more important than the Bing features. Based on that, we can notice that the Google results can improve the performance more than the Bing results. At the beginning of our experiments, we tried also to combine Yahoo results, but unfortunately, in all of our experiments, Yahoo results had a lower performance.
Since our approach is search engines-based, for the Arabic task, we found that these claims did not exist because they were written originally in English and translated into Arabic for this task. Therefore, we translated back these claims into English to retrieve results. After that, the results and the query were translated back to Arabic.
During our experiments, many classi ers were tested. We found that the Random Forest classi er achieved the highest results. By using the K-Fold strati ed technique, we achieved 0.34 of macro F1 score. The chosen value of K is 5, where we have a small number of data instances and an imbalanced dataset. Thus, higher values of K may lead to absence of some classes in one of the training/testing cycles. We tried also to build a di erent type of queries, using main sentence components, or phrase queries, but we found that when we changed the query, the results were a ected negatively, especially when we used the phrase query, we noticed that the search engines snippets became meaningless (phrases appeared in the snippets but as small text clips connected using "..." characters and combined into the main snippet, where the semantic meaning of the main snippet became biased). For this reason, we passed the queries without any modi cation, letting the search engines to retrieve the most appropriate results for each one.
For the o cial testing phase the Mean Absolute Error (MAE) was used as performance measure. In Table 1, the results of the task are shown. We have presented our simple approach for detecting the veracity of claims in the presidential debates. As we mentioned, our approach uses search engine results to infer the claims veracity. In our approach, we extracted two di erent types of features, dependent and independent, to model the distribution of claims' results. In general, the results of the task are low, knowing that our approach during the tuning phase has achieved good results comparing to the o cial one. Also, we can conclude that our feature set has improved the results with respect to the other participants approaches and to the baseline. 6
The authors acknowledge the SomEMBED TIN2015-71147-C2-1-P MINECO research project. The work on the data in Arabic as well as this publication were made possible by NPRP grant #9-175-1-033 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the last two authors.