Misinformation is a growing problem around the web. The spread of such a phenomenon may impact public opinions. Hence fake news detection is indispensable. The rst step for fact-checking is the selection of check worthy tweets for a certain topic, then ranking sentences from related web pages according to the carried evidence. Afterward, the claim will be veri ed according to evident sentences. At CLEF2020 { CheckThat! lab, three tasks run in Arabic, namely check-worthiness on tweets, evidence retrieval, and claim veri cation that corresponds respectively to task1, task3, and task4. We participated in the three tasks. We integrated manual sentiment features as well as named entities to detect fake news. The integration of sentiment information in the rst task caused result degradation since there may be an overlap between check worthy and not check worthy tweets. For the second task, we explored the e ect of sentiment presence and we used cosine similarity as a similarity measure between the claim and a speci c snippet. The third task is a classi cation task based on sentiment and linguistic features to compute the overlap and the contradiction between the claim and the detected check worthy sentences. The results of task1 and task3 leave large room for improvement, whereas the results of task 4 are promising since our system reached 0.55 of F1-measure.
Interaction with others online through social media has become indispensable. Social media are not only a way to communicate but also a warehouse of news used by people who seek news and information from social media rather than news organizations. People publish their opinions as if they were facts which can mislead the orientation of the public opinion and have negative e ects on the psychology of the people. A high proportion of people is exposed to misleading or false claims. For example during coronavirus pandemic, claims about COVID-19 appeared without trustful reference. Many stories have been found such as the theory that the spread of COVID-19 is caused by 5G technology. The spread of such claims a ected people understanding of the pandemic.
Fake news has known explosive growth in recent years especially on social media where a large amount of data is uncontrolled. The extensive spread of this phenomenon may impact individuals and society negatively.
In recent years, fake news appear to mislead the orientation of public opinion for
commercial and political purposes. Facts are ignored when shaping public
opinion, since appealing to emotions works better as it has a potential impact on the
person. Social media publish fake news to a ect reader psychology and hence
increase readership. With o ensive and deceptive words, social media users can
get a ected by these fake news easily, which brings tremendous e ects on society.
The identi cation of fake news is hard and time consuming. To improve
information trustworthiness, we should build systems to detect fake news in real time.
Thus, many studies addressed the automation of fake news detection process to
facilitate their veri cation among which [
In the following, we describe our participation in CLEF2020 { CheckThat! lab. The paper is organized as follows, we present previous works, afterward we dene the tasks in which we participated, then we describe the external resources used by our system as well as the system approach and we give the obtained results for the classi cation task (task 4). 2
This paper investigates the principal approaches used to de ne news factuality. In the following, we present some previous works that aimed to detect fake news.
Hansen et al. [
Shu et al. [
Zafarani et al. [11] presented a paper that introduces the characteristics of fake news that di erentiate it from similar concepts such as misinformation to present fake news detection strategies systematically.
Atanasova et al. [
Hasanain et al. [
CLEF2020 { CheckThat! lab proposed many tasks among which three tasks that run in Arabic. We have participated in the three tasks, namely task1, task3 and task4. In the following, we describe each task according to its presentation by the lab organizers. 3.1
The organizers gave a set of topics and their corresponding potentially-related tweets. This task aims to verify whether a tweet is check worthy. A tweet is considered check worthy if it carries harmful content or it is of interest to a large audience. This task is a ranking task that aims to rank the tweets according to their check-worthiness for the topic. The o cial measure used for evaluation is P@30 for the Arabic dataset. 3.2
For this task, the organizers presented a set of topics and the corresponding claims and a set of text snippets extracted from potentially-relevant webpages. The task aims to return for a given claim a ranked list of evidence snippets that support or refute the claim, namely the ones that are useful in verifying it. 3.3
The task presents a dataset that contains 201 check-worthy claims related to 12 topics. For these topics, a set of potentially related web pages is given. The task aims to use the data to predict claims veracity. The task is a classical binary classi cation task that uses true or false tags to tag a speci c claim according to its veracity. Precision, recall, and F1-measure are used as evaluation measures and the macro-averaged F1 is used as the o cial measure. 4
This section aims to describe the resources used by our system besides the ones presented by the task organizers. We constructed four lexicons, namely sentiment, o ense, sarcasm, and named entities lexicons. The lexicons are described in the following: Sentiment lexicon: the lexicon contains 9858 sentimental terms. This lexicon is a combination of many resources which are: Lexicon1(SemEval 1): a lexicon that contains sentimental terms and their corresponding sentiment intensity.
Lexicon2 (MPQA 2: the Arabic version of the original MPQA lexicon that contains sentimental terms.
Lexicon3 (ENGAR): is an English sentiment lexicon created by [
Lexicon4: a sentimental lexicon extracted from a corpus collected from Hespress 3 Facebook page.
The lexicons have been veri ed semantically by the authors of this paper. We give in Table 1 the statistics of sentiment lexicons. Lexicon Lexicon1 Lexicon2 Lexicon3 Lexicon4 Statistics 980 4166 3504 1778 Total 10428
Total unique 9858
O ense lexicon: the o ensive lexicon is sharper than the negative sentiment lexicon. We constructed a lexicon by extracting o ensive terms from the o ensive corpus collected by the organizers of the o ensive language detection shared task 4. The lexicon contains 1120 o ensive terms.
Sarcasm lexicon: the lexicon contains 148 sarcasm indicators extracted manually
from the ironic corpus that was created by [
Named entities lexicon : the lexicon contains the names of religions, countries and 1 http://www.saifmohammad.com/WebPages/SCL.html 2 http://www.purl.org/net/ArabicSA 3 https://fr-fr.facebook.com/Hespress 4 https://sites.google.com/site/o ensevalsharedtask/ known personalities. The terms of this lexicon were collected by google queries and then were expanded by the authors of this paper. a) Religion lexicon: contains 9 religions which give 104 terms of the corresponding adjectives and nouns. b) Nationality lexicon: contains 194 countries. We enhanced the names of the countries by the corresponding nationalities. c) Named entities: contains named entities extracted from the o ensive corpus. The terms target religions (ÕÎÓ), countries (Q¢¯), backgrounds (Q¯), sports teams (½ËAÓP), political parts (úGñk), genders (à@ñ ), famous personalities (úaeJ). The lexicon contains 216 terms.
We give in Table 2 examples of the mentioned lexicons. 148 éêë
Named entities 514
The presence of sentiment and sarcasm terms within an expression may indicate that the expression is an opinion not a fact. The cause behind using o ense lexicon is that it may de ne harmful expressions. We use named entities to dene trustful sources since a text that contains named entities tends to be more factual. The lexicons have been created for sentiment analysis purposes, they have not been made publically available yet. We use them as external resources that cover a large set of sentiment terms. 5 5.1
We extract texts of tweets, claims, and snippets from the given JSON object using regular expressions. For task 4, instead of using Jsoup that is a java library that parses HTML documents as in [10] to extract the content of relevant web pages, we use the text snippets that were extracted from these pages. We give a standard representation to the extracted text, we preprocessed the claims and the text snippets extracted from potentially relevant webpages by removing all characters other than the Arabic letters. We tokenize each text into terms using space delimiter. Hence, each claim and text snippet will be represented by a set of terms. 5.2
In this task, we aim to rank the top 500 tweets related to each topic according to their check-worthiness. In the following, we use the resources created by our system to rank tweets besides other features. We use di erent features namely the title and the description of the topic, sentiment, o ense and named entities features. We weigh each feature using a weight that represents its importance. F1: represents the weighted intersection between the topic title and tweet text. We give the title the weight 3 as it is the most important part of the content. F2: is the weighted intersection between the topic description and the tweet text. We give the description the weight 2 since it contains important information. F3: represents the occurrence of named entities in the tweet text. Texts that contain named entities are check worthy as they represent a trustful source of information. We give this feature the weight 1.
F4: represents the occurrence of o ense lexicon terms in the tweet text. The o ense lexicon is an indicator of the presence of harmful content. We give this feature the same weight as named entities.
F5: represents the weighted occurrence of sentiment terms in the tweet text. The text that contains sentiment lexicon tends to be an opinion not a fact. This feature is given a negative weight -1.
All the features are given a positive weight except sentiment features since checkworthy tweets tend to be facts rather than opinions, hence we give a negative weight to the present sentiment terms. We give positive weight to o ense feature since from the de nition, check-worthy tweets are the ones that carry harmful content. Since the title and description are related to the topic, then each tweet is given a score based on the product of the mentioned features weights and their intersection with the tweet text. In other words, whenever a topic title or description term matches tweet text term, we increment the value by 3, 2 or 1 according to the corresponding weight for each feature. The same for other features. The score is then divided by the sum of feature sizes which gives a normalized score.
Table 3 gives an example of initial values. Using Formula (1) we compute the normalized score for each tweet. Li is the length of each lexicon. The statistics of each lexicon are given in Table 2.
Feature Size 12 Weight 3 Intersection2 with the tweet
This approach showed degraded results in comparison to the approach [10] that uses only document parts as features to rank web pages. In the o cial results it reached only 0.28 using P@30 which was under the baseline. The reason for this may be the intersection between check worthy and less check worthy terms that match the extracted features which means that they may characterize both of which or rather the selected features match more less check worthy tweets which made the ranking di cult. 5.3
We aim to rank a list of text snippets based on their evidence namely their usefulness for fact checking. We extract a text snippet and compare it with a tweet text. Then we a ect a score to each snippet based on the results of (2) that gathers cosine similarity between the tweet and the text snippet and weighs using a negative weight the intersection between a speci c snippet and the sentiment lexicon. In other words if ve sentiment terms are present in the text snippet, then the intersection is ve. The negative weight is given to di erentiate between facts and opinions. We rank the top 100 evidence text snippets corresponding to each tweet based on the relation (2) and also using only on the cosine similarity score. When multiple snippets have equal cosine similarity score we break the tie by considering both of which if they are ranked with the top 100. Using cosine similarity only shows better results than adding sentiment information. This may be explained by the fact that sentiment terms may appear in a factual text without the intention of the holder to express an opinion. However the obtained results for this task were degraded by reaching 0.05 only using P@10 metric.
Score = cosineSimilarity 0:5 intersection (2) 5.4
In this task, we aim to classify claims as true or false. We compare each claim with the text snippets extracted from the relevant web pages. We calculate factuality based on a snippet information using two values identical and opposite, we de ne each of which by: Identical : represents the concordance between the claim terms and the text terms.
Opposite: is a negative value that represents the number of claim terms which opposite appear in the text.
In order to de ne the negated terms we use a list of negation words. If a claim term matches a text term then based on table 4, we can de ne whether they are identical.
Yes * *
Text term Negated
No * *
Concordance Identical Opposite Opposite Identical
We use snippets extracted from potentially relevant Web pages. For a given
Web page, if a snippet supports the claim as none of its terms are negated,
whereas, a second snippet contradicts the claim as one or many of its terms
among the ones that match claim terms are negated. Then the claim is false
at the current Web page level. According to the relation (3), if the factuality
is greater than 0, then the claim is true. Unless, the factuality will be negative
which means the presence of snippets opposite to the claim. Thus, wherever our
system nds contradicting snippets, it tags the claim as false, otherwise, it gives
it true tag. This threshold has been chosen since according to Baly et al. in [
F actualityInitial = Identical opposite (3) In table 5 we give the results of Task4 using the mentioned criteria.
The second test is based on the following criterion, if the factuality is false according to the aforementioned criteria or a sarcasm indicator is present in the text, then the claim is false. The presence of sarcasm features augments the probability of the analyzed sentence to be fake. The second test uses a list of negation terms that contains 189 terms. Adding sarcasm features doesn't generate any improvement, which may be due to the weak intersection between sarcasm lexicon and the analyzed text and hence we don't report the results of the corresponding test. 6
In this paper, we presented our participation in CLEF2020 { CheckThat! lab. We aimed to build a sentiment aware fake news detection system. We participated in three tasks, we based approaches on various mathematical dependencies. We enhanced the used approach by adding sentiment features to de ne the impact of it on the detection of fake news. The challenge of this paper wasn't the integration of sentiment features only, but also we aimed to base our system on an unsupervised approach to overcome the di culties of datasets collection and annotation and also to reduce the time of fact-checking taken when building supervised models. The obtained results in the classi cation task are promising, however, there is a large room for improvement when it comes to ranking tasks. 10. Touahri, I., Mazroui, A.: Automatic veri cation of political claims based on morphological features. In: CLEF (Working Notes) (2019) 11. Zafarani, R., Zhou, X., Shu, K., Liu, H.: Fake news research: Theories, detection strategies, and open problems. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. pp. 3207{3208 (2019)