This paper addresses the problem of automatically detecting fake news spreaders in social networks such as Twitter. We model the problem as a binary classification task and consider several groups of features, including writing style, word and char n-grams, BERT semantic embedding, and sentiment analysis, which are computed from a set of tweets each user authored. Our proposed approach is evaluated on the dataset made available by the PAN at CLEF 2020 shared task on profiling fake news spreader, which provided labeled data in both English and Spanish. Experimental results show that we can detect fake news spreaders with an accuracy of 0.73 in English and 0.77 in Spanish when our approach is evaluated with 10-fold cross-validation on the provided training set, and with an accuracy of 0.71 in English and 0.76 in Spanish when the model is trained on the whole training set and tested on the provided test set. We also investigate the role of psycho-linguistic (LIWC) and personality features to detect fake news spreaders and find out that personality features do have a significant impact in user sharing behavior, achieving an accuracy of 0.72 in English and 0.80 in Spanish when evaluated with 10-fold cross-validation on the provided training set.
Fake news sharing has become a concerning problem in online social networks in
recent years. Research has found that fake news is more likely to go viral than real news,
spreading both faster and wider [
Thus, being able to identify fake news spreaders in social networks is one of the
key aspects to effectively mitigate misinformation spread. Examples of strategies that
could be implemented include assisting fake news spreader with credibility indicators
to lower their fake news sharing intent [
This paper describes the approach we implemented and submitted to profiling fake
news spreaders on Twitter as part of the PAN at CLEF 2020 shared task [
Furthermore, the paper also investigates the role of psycho-linguistic (LIWC) and personality features (including Big Five personality traits, needs, and values) in detecting fake news spreaders. Our additional experimental results on the provided training set show that we can achieve an accuracy of 0.72 with LIWC or personality features on the English dataset, while personality features achieve an accuracy of 0.80 on the Spanish dataset, outperforming our submitted approach on this specific language. These results suggest that psycho-linguistic and personality features are valuable characteristics to consider when profiling fake news spreaders in social media. 2
Several studies have been conducted to understand the characteristics of regular
(nonmalicious) users that correlate with fake news spreading behavior in social networks.
Vosoughi et al. [
We carried out our experiments on the dataset provided by the PAN’20 shared task for
Profiling Fake News Spreaders on Twitter [
This section presents the features we considered for detecting fake news spreaders. As the dataset files are in raw XML format, we parsed and formatted them by using the xml.etree.ElementTree1 library. After extraction, we pre-processed the content (user tweets) as per requirement for each feature we considered as there are some features that require cleaned texts and some other that require the text as it is for incorporating underlying details of the author’s writing. Basically, four different types of features were considered to address this task, as explained in detail below.
Style. This first set of features captures the writing style of the set of tweets authored by the same user. Specifically, we computed the average number of certain words, items, 1 https://docs.python.org/3/library/xml.etree.elementtree.html and characters per user tweet, which includes the average number of (1) words, (2) characters, (3) lowercase words, (4) uppercase words, (5) lowercase characters, (6) uppercase characters, (7) stop words, (8) punctuation symbols, (9) hashtags, (10) URLs, (11) mentions, and (12) emojis and smileys. Also, we considered the (13) percentage of user tweets that are a retweet and (14) the percentage of user tweets that are a sharing of breaking news.
N-grams. The second group of features includes TF-IDF based n-grams for both words and characters. We concatenated all the tweets by each user to form a single document per user. For each document, we removed words like ’RT,’ ‘Via,’ and ‘&’ and replaced emojis and smileys with the corresponding English words. Next, each document was converted to lowercase, stop words and punctuation symbols were removed, and the remaining words were lemmatized into root words the using the NLTK toolkit.2 Finally, we computed the TF-IDF vector representation for both word and char n-grams.3 We experimented with different parameters, including the number of terms and the length of grams, from uni-grams to tri-grams. We obtained the best results for both chars and words with uni-grams and by including all the terms (max_df = 1.0).
Tweet embedding. The third group of features includes the embedding of tweets as
computed by using the BERT state-of-art NLP model. Specifically, we used the pre-trained
multilingual model provided by SBERT [
Sentiment analysis. As people express their emotions, appraisals, and sentiments
towards any news or article through the choice of words in their tweets, we leveraged
sentiment analysis as another feature. Also in this case, we pre-processed each tweet to
remove ‘RT,’ ‘Via,’ and ‘&.’ However, we did not replace emojis and smileys with
the corresponding English text for this feature since these emojis adds to the sentimental
values of the text. Then for each user, we measured the average sentiment across all their
tweets. We used the Valence Aware Dictionary and sEntiment Reasoner (VADER) [
3 https://scikit-learn.org/stable/modules/generated/sklearn.
feature_extraction.text.TfidfVectorizer.html 4 https://scikit-learn.org/stable/modules/generated/sklearn.
decomposition.PCA.html 5 https://pypi.org/project/sentiment-analysis-spanish/ We built our classification model as an ensemble of classifiers. Specifically, we considered each group of features separately and, by performing 10-fold cross-validation on the training set, we chose the best classifier for that group of features among Support Vector Machine (SVM) with linear kernel, Logistic Regression (with default parameter), Random Forest and Extra Trees (both with 500 estimators and minimum sample leaf parameter equal to 1).
According to the results shown in Tables 1a and 1b, for English, we chose Extra Trees as the best classifier for the style features, SVM for both n-grams and tweet embedding, and Logistic Regression for sentiment analysis. Likewise, for Spanish, we chose SVM as the best classifier for the style features, Extra Trees for n-grams, Random Forest for tweet embedding, and Logistic Regression for sentiment analysis.
We see that, for English Twitter users, the best performing set of features is n-grams (accuracy of 0.72), followed by BERT tweet embedding (accuracy of 0.69), sentiment (accuracy of 0.66), and style-based features (accuracy of 0.64). For Spanish Twitter users, style-based features and n-grams are equally the best sets of features (both with an accuracy of 0.75), followed by BERT tweet embedding (accuracy of 0.74), sentiment (accuracy of 0.57). Overall, the different performance of the features across English and Spanish may highlight a different cultural behavior in the use of Twitter in general, and in sharing news in particular.
For the final model, we first trained each group of features with the corresponding
selected best classifier from Tables 1a and 1b. Then, we used the prediction
probabilities of all the four selected best classifiers as input features to a majority voting classifier
that combined the predictions of four base estimators, namely SVM, Logistic
Regression, Random Forest, and Extra Trees. Table 2 reports the resulting accuracy values of
the final majority voting classifier in two cases. First, when we evaluate our model with
10-fold cross-validation on the training set (i.e., we trained on 90% of the training set
and tested on the remaining 10%, repeated the experiment 10 times and averaged the
results), we achieve an accuracy of 0.73 and 0.77 for the English and Spanish dataset,
respectively. As we can see, the accuracy value of the features combined by our final
model improves over the accuracy values achieved by each group of features
individually, as reported in Tables 1a and 1b. Second, when we train on the whole training set
and test on the test set, we achieve an accuracy of 0.71 for English and 0.76 for Spanish
(run executed on TIRA [
After the submission of our run, we performed some additional experiments to investigate how other groups of features, such as psycho-linguistic and personality features, would perform at detecting fake news spreaders. Specifically, we considered the set of psycho-linguistic features computed by the Linguistic Inquiry and Word Count (LIWC) tool and personality features extracted by using the IBM Watson Personality Insights service6. To compute these two sets of features, tweets were pre-processed in the same way as for the tweet embedding features from Section 4. These features are described in detail here below.
Linguistic Inquiry and Word Count (LIWC). LIWC is a transparent text analysis tool
that counts words in psychologically meaningful categories. The tool works for
different languages, including English and Spanish. We used LIWC2015 for English (93
features) and LIWC2007 for Spanish (90 features). LIWC computes different measures
for analyzing the cognitive, affective, and grammatical processes in the text. The LIWC
features can be divided into four main categories [
gory of features also includes negations as well as part-of-speech (Adjective, Noun, Verb, Conjunction) frequencies.
Punctuation features include the occurrences of Periods, Commas, Question, Exclamation, and Quotation marks, etc. in the text.
Psychological features target emotional, social process, and cognitive processes. The affective processes (positive and negative emotions), social processes, cognitive processes, perceptual processes, biological processes, time orientations, relativity, personal concerns, and informal language (swear words, nonfluencies) can be used to scrutinize the emotional part of the text.
Summary features define the frequency of words that reflect the thoughts, perspective, and honesty of the writer. It consists of Analytical thinking, Clout, Authenticity, Emotional tone, Words per sentence, Words more than six letters, and Dictionary words under this category.
To compute the LIWC feature set for each user in the dataset, we first computed the
LIWC features for each tweet and then averaged the features for the same user.
Personality Features. The IBM Watson Personality Insights service uses linguistic
analytics to infer individuals’ intrinsic personality characteristics, including Big Five
personality traits, Needs, and Values, from digital communications such as social media
posts. The tool is able to work for different languages, including English and Spanish.
In our case, we concatenated all the tweets of a given user in a unique document to
compute their personality characteristics. The features computed by this service are detailed
in the following (we considered the raw scores provided by the service):
Big Five The Big Five personality traits, also known as the five-factor model (FFM)
and the OCEAN model, are a widely used taxonomy to describe people’s
personality traits [
Values These features describe the motivating factors that influence a person’s decision making. They include self-transcendence, conservation, hedonism, self-enhancement, and open to change.
Tables 3a and 3b report the accuracy achieved by the four considered classifiers with input LIWC and personality features for detecting fake news spreaders. We were able to evaluate these features only on the provided training set and reported results are the averaged values of 10-fold cross-validation. As we can see, for the English dataset, LIWC and personality features achieve both the best accuracy of 0.72 with Random Forest, which is the same as the accuracy achieved by n-grams features (cf. Table 1a) and slightly lower than the accuracy of 0.73 achieved by our final submitted approach on the training set (cf. Table 2). In the case of the Spanish dataset, LIWC achieves the best accuracy of 0.78 with Random Forest, while personality features perform with the best accuracy of 0.80 with Extra Trees. In this case, both LIWC and personality features outperform our submitted approach on the training set, which achieves an accuracy of 0.77 (cf. Table 2), with personality features having the best ever achieved accuracy among all the group of features we tried for this shared task. 7
We addressed the problem of automatically detecting Twitter users keen at spreading fake news as part of the PAN at CLEF 2020 shared task on profiling fake news spreaders in two languages, namely English and Spanish. We proposed a first approach leveraging several groups of features, including writing style, word and char n-grams, BERT semantic embedding, and sentiment analysis, which are computed from the Twitter feed provided for each user. This approach achieved an accuracy of 0.73 in English, resp. 0.77 in Spanish, when evaluated with 10-fold cross-validation on the provided training set, and an accuracy of 0.71 in English, resp. 0.76 in Spanish, when evaluated on the provided test set. We also investigated the role of psycho-linguistic (LIWC) and personality features on the same task. We showed that personality traits are important characteristics to consider when modeling user sharing behavior, as they achieved an accuracy of 0.72 in English, resp. 0.80 in Spanish, when evaluated with 10-fold cross-validation on the provided training set. Overall, the task of detecting fake news spreaders turned out to be very challenging. Even if we tried several sets of features, we could not achieve an accuracy value higher than 0.80. One possible motivation could be that some users keen to spread fake news do not do it intentionally; hence they are hard to differentiate from users who never shared fake news. Also, the accuracy gap between English and Spanish may indicate users have different news spreading behaviors across different cultures.
Future work will be devoted to analyzing the role of additional sets of features for detecting fake news spreaders in social networks, including behavioral features describing the user activity on Twitter and social network features.
Acknowledgements This work has been partially supported by the National Science Foundation under Awards no. 1943370 and 1820685.