The authors of fake news often use facts from verified news sources and mix them with misinformation to create confusion and provoke unrest among the readers. The spread of fake news can thereby have serious implications on our society. They can sway political elections, push down the stock price or crush reputations of corporations or public figures. Several websites have taken on the mission of checking rumors and allegations, but are often not fast enough to check the content of all the news being disseminated. Especially social media websites have offered an easy platform for the fast propagation of information. Towards limiting fake news from being propagated among social media users, the task of this year's PAN 2020 challenge lays the focus on the fake news spreaders. The aim of the task is to determine whether it is possible to discriminate authors that have shared fake news in the past from those that have never done it. In this notebook, we describe our profiling system for the fake news detection task on Twitter. For this, we conduct different feature extraction techniques and learning experiments from a multilingual perspective, namely English and Spanish. Our final submitted systems use character n-grams as features in combination with a linear SVM for English and Logistic Regression for the Spanish language. Our submitted models achieve an overall accuracy of 73% and 79% on the English and Spanish official test set, respectively. Our experiments show that it is difficult to differentiate solidly fake news spreaders on Twitter from users who share credible information leaving room for further investigations. Our model ranked 3rd out of 72 competitors.
Author profiling uses information of people’s writing style to determine specific
characteristics such as the author’s gender, age, personality, or cultural and social context, like
mother tongue and dialects [
Fake news poses a serious threat to our society. They can destroy reputations of corporations and public figures, can push down the stock price and manipulate peoples opinions and therefore also their actions. Social media has become an ideal place for fake news propagation as user-generated content reaches very quickly a broad audience. Fraudsters use those networks to deceive users and shape specific opinions by making the reader believe a certain political or social agenda. The sheer mass of false information spread on the internet has reached new heights and cannot be handled by manual fact-checking alone. However, automatic recognition of fake news is a challenging task. Knowledge-based and context-based approaches to combat fake news can be applied, but only after the fake in the news has been verified by experts. This is often not fast enough as fake news spread very quickly and reach a broad audience, especially on social media websites.
Style and content-based approaches are a viable alternative [
In the following, we describe our approach for the author profiling task at PAN 2020. After a review of related work in Section 2, Section 3 details the Twitter data that was provided by the PAN organizers and shows some key statistics observed in the corpus. The preprocessing steps and features used to train our models are detailed in Section 4. Our models and classification results are discussed in Section 5. We also provide some information about our alternatively tested methods (Section 6) and conclude our work in Section 7. 2
Potthast et al. [
Liu and Wu [
Zhou et al. [
To train our system, we used the PAN 2020 author profiling corpus7 proposed by Rangel
et al. [
EN and ES True News Tweets EN and ES Fake News Tweets “RT #USER#: Best dunk of the contest no doubt “Jay-Z Must Give Beyonce $5 Million Per about it. Aaron Gordon robbed again #URL#” Child They Have Together Due to Crazy
Prenup. . . #URL#” “RT #USER#: Sure would be an interesting day “RT #USER# #USER# When Obama was tapto read a book that examines Trump’s obsession ping my phones in October, just prior to Elecwith the king-like powers of his offic. . . ” tion!” “A Data-Driven Approach Aims to Help Cities “Why Trump lies, and why you should care Recover After Earthquakes #URL#” The Boston Globe #URL#” “Javier Cámara ya es el líder más valorado de “Dictadura pura y dura toma tasas y todos felilos españoles por delante de Pedro Sánchez, cices #URL#” según una encuesta #URL# #URL#” “Me gusta la foto. Una foto con variedad, diver- “GANAR DINERO AHORA ES FACIL – sidad. Me da la impresion que con más sonrisas Google te paga 15 dólares por contestar encuesque otras. #URL#” tas #URL# #URL#” “Navidad en RD: son 3 días gozando, luego 362 “Ortega Smith: ‘VOX expulsará de España a tollorando y deseando mal a los demás. Dejen su dos los inmigrantes ilegales’ #URL#” hipocresía !!” As can be seen in Table 1, the Twitter specific tokens hashtags, URLs and user mentions were replaced by the providers with the following placeholders: #HASHTAG#, #URL# and #USER#. Prior to the feature engineering, we analyzed the distribution of different tokens. Additionally, we determined the sentiment of each tweet (positive, negative, or neutral) using TextBlob8. For recognizing the named entities (NER), we used the Python library spaCy. Table 2 shows some key insights for both languages.
The observations of the corpus content were the following: – Fake news spreaders: mention other Twitter users less often (#USER#9). utilize fewer hashtags (#HASHTAG#). re-post fewer tweets (RT).
share slightly more URLs (#URL#). – Spanish speaking authors use more emojis than English speaking Twitter users. – Half of the English tweets are based on factual information and most of the Spanish tweets (90%) are free of emotions.
– Fake news tend to be more often negative. – Tweets of true news spreaders tend to be more often positive. – By counting the named entities no significant difference between the classes could be established. – Fake news spreaders tend to tweet slightly more often about other people. – Uppercased tokens are shared equally by true news and fake news spreaders. – Spanish fake news spreaders make more often use of capitalized phrases. 4
The preprocessing pipeline was performed for both languages (EN and ES) basically. The steps for cleaning and structuring the data were performed as follows: 1. First, we extracted the text from the original XML document of each user and concatenated all 100 tweets to a single text. 2. White space between tokens were normalized to a single space. 3. URLs, hashtags and user mentions were left untouched as they are already replaced by placeholders by default. 4. Numbers and emojis were replaced by the placeholders #NUMBER# and #EMOJI#. 5. Irrelevant signs, e.g. “+,*,/,” were deleted. 6. Sequences of repeated characters with a length greater than three were normalized to a maximum of two letters (e.g. “LOOOOOOOOL” to “LOOL”). 7. Words with less than three characters were ignored. 8. Stopwords were deleted by using the NLTK (Natural Language Toolkit) library10 for each language separately. 10 https://www.nltk.org/ 9. From the NLTK library we additionally used the TwitterTokenizer to tokenize the words. The tokenizer is suitable for Twitter and other casual speech that is often used in social networks. Additionally, TwitterTokenizer contains different regularization and normalization features. We made use of the lowercaser.
After the Twitter texts were preprocessed, we tested different vectorization techniques with manual hyperparameter tuning, and by employing scikit-learn’s grid search function. The hyperparameters were tuned separately for English and Spanish, but the features we used were mainly language-independent which means that the same set of features can be used in multi-language domains. The selected features were presented in Section 3 (e.g. counts of tokens or named entities). The only language dependant feature we experimented with was the sentiment polarity calculated separately for every tweet (whether it is positive, negative, or neutral). Besides the handcrafted features, we also experimented with automatically learned features i.e. term frequency distribution (tf) and character and word n-grams. Additionally, we made use of Feature Union11 to experiment with feature concatenation. To convert the tokens to a numerical matrix in order to build a vector for each language, we made use of: (1) Scikit-learn’s term frequency-inverse document frequency (TF-IDF) (2) GloVe12 (Global Vectors for Word Representation) word vectors pre-trained on
Twitter data as well as custom trained word2vec13 word embeddings (3) Scikit-learn’s Count Vectorizer
We defined the author profiling task as a binary problem predicting whether a tweet was composed by a fake news spreader or a reliable Twitter user. For each language (EN and ES) a separate classification model was trained. As mentioned before, for training and testing, we split the data in the ratio 70/30. We tested different features, vectorization techniques and dimensionality sizes in combination with a Support Vector Machine 11 https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.FeatureUnion.html 12 https://nlp.stanford.edu/projects/glove 13 https://radimrehurek.com/gensim/models/word2vec.html (SVM) and Logistic Regression of which we report the best performed ones. For the final SVM, we used a linear kernel with default hyperparameter values14. Logistic Regression was also trained by utilizing default hyperparameters15.
The performance of the fake news spreader author profiling task was ranked by
accuracy. Table 4 shows the scores for our final system performed on the official PAN
2020 test set on the TIRA platform [
LinearSVC.html 15 https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.
LogisticRegression.html
In this Section, we report our experiments with alternatively tested feature selections and representation techniques which were not able to keep up with the systems described above in terms of performance (see Section 5). Besides character n-grams, we also experimented with word n-grams in the range of [1;7]. Other selected features comprised counts of emojis, uppercase tokens and phrases, hashtags, user mentions, URLs and retweets. Additionally, we incorporated sentiment analysis in our vector by using TextBlob. The selected features we presented in Section 4 and Table 3.
Besides TF-IDF, we tested term frequencies (tf) and word embeddings as feature representations. Therefore, we utilized GloVe word vectors pre-trained on Twitter data as well as custom trained word2vec word embeddings. To combine the different features in one vector, the inner product space of two vectors was required. First, all texts of the fake news spreaders were concatenated and vectorized. Then, the cosine similarity of this vector and every twitter user was determined. The resulting vector comprising a varying number of features was standardized (using StandardScaler 16). The final vector was then forwarded to train the SVM and Logistic Regression models. Our aim was to test whether emotions and sentiments, emojis, or uppercase tokens in fake news could improve the classification performance. The training results showed that none of those features or feature combinations could improve the performance in both languages. The accuracy has even slightly decreased. 7
In this paper, we described our participation in the author profiling task at PAN 2020. The goal was to develop a system for profiling fake news spreaders on Twitter as a first step towards preventing the propagation of fake news among online users. For our experiments, we used the PAN 2020 author profiling corpus provided by the organizers. We conducted different learning experiments from a multilingual perspective, namely English and Spanish. We evaluated different features, most of them languageindependent. The features were extracted and had their importance evaluated in the detection task. We provided some corpus statistics that showed that there are differences between fake and true news spreaders. We experimented with different features, vectorization techniques and dimensionality sizes.
For the English language, our model performed best using SVM with TF-IDF weighted character n-grams with range [1; 3] and top 3,000 features. For the Spanish language, the best results were achieved using Logistic Regression employing a feature union of TF-IDF weighted character n-grams with range [1; 3] and top 5,000 features and a vector consisting of character n-gram counts with range [3; 7] and top 50,000 features. The submitted models achieve an overall accuracy of 73% and 79% on the English and Spanish corpus, respectively. Our model ranked 3rd out of 72 competitors.
The results showed that it is challenging to detect fake news spreaders in Twitter data. It was challenging in two ways. First, not every tweet of a fake news spreader is 16 https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html false but a mixture of true and false information. Second, Twitter data is short, noisy and incorporates platform-specific features (such as user mentions and retweets). The biggest challenge is the orthography. The tweets are strewn with spelling mistakes and grammatical errors. Word-level based approaches perform poorly compared to approaches based on character n-grams.
In the future, we first want to experiment with style-based approaches in order to determine whether fake news spreaders can be identified by the writing style alone. Finally, we plan to experiment with different standardization and pre-processing techniques as our submitted system does not consider misspelled words.
This work was supported by the German Federal Ministry of Education and Research and the Hessen State Ministry for Higher Education, Research and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE and under grant agreement "Lernlabor Cybersicherheit" (LLCS) for cyber security research and training.