In this notebook, we summarize our work process of preparing a software for the PAN 2020 Profiling Fake News Spreaders on Twitter task. Our final software was a stacking ensemble classifier of five different machine learning models; four of them use word n-grams as features, while the fifth one was based on statistical features extracted from the Twitter feeds. Our software uploaded to the TIRA platform achieved an accuracy of 75% in English and 80.5% in Spanish. Our overall accuracy of 77.75% turned out to be a tie for the first place in the competition.
The aim of the PAN 2020 Profiling Fake News Spreaders on Twitter task [
We used an ensemble of different machine learning models to provide a prediction for each user. All of our sub-models handle the Twitter feed of a user as a unit and determine a probability for each user how likely they are to be fake news spreaders. For the final predictions, these sub-models are combined using a logistic regression.
In Section 2 we present some related works on profiling fake news spreaders. In Section 3 we describe our approach in detail together with the extracted features and models. In Section 4 we present our results. In Section 5 we discuss some potential future work and in Section 6 we conclude our notebook.
Using word n-gram variables for author profiling has been shown to be effective
[
SVMs [
The corpus for the PAN 2020 Profiling Fake News Spreaders on Twitter task [
We developed our software using the Python language (version 3.7). To build our models we mainly used the following packages: scikit-learn1, xgboost2, emoji3, lexical-diversity4, pandas5 and numpy6. Our codes are available on GitHub7. 1 https://scikit-learn.org/ 2 https://xgboost.readthedocs.io/ 3 https://pypi.org/project/emoji/ 4 https://pypi.org/project/lexical-diversity/ 5 https://pandas.pydata.org/ 6 https://numpy.org/ 7 https://github.com/pan-webis-de/bolonyai20
We experimented with a number of machine learning models based on word ngrams extracted from the text. Precisely, we investigated the performance of regularized logistic regressions (LR), random forests (RF), XGBoost classifiers (XGB) and linear support vector machines (SVM). For all four models, we ran an extensive grid search combined with five-fold cross-validation to find the optimal text preparation method, vectorization technique and modeling parameters. We tested the same parameters for the English and Spanish data. We investigated two types of text cleaning methods for all models. The first method (M1) removed all non alphanumeric characters (except #) from the text, while the second method (M2) removed most non alphanumeric characters (except #) but kept emoticons and emojis. Both methods transformed the text to lower case. Regarding the vectorization of the corpus, we experimented with a number of parameters. We tested different word ngram ranges (unigrams, bigrams, unigrams and bigrams) and also looked at different scenarios regarding the minimum overall document frequency of the word n-grams (3, 4, 5, 6, 7, 8, 9, 10) included as features. Table 1 describes the tested model hyperparameter values during the training phase of our models.
For the early bird testing phase conducted through TIRA [
LR RF SVM XGB
Regularization coefficient (C)
Number of boosting rounds (B) Minimum number of cases on each leaf
(min_samples_leaf) Regularization coefficient (C)
Learning rate (eta): Number of estimators (n_estimators) Maximum depth of a tree (max_depth)
Subsample ratio (subsample) Subsample ratio of columns (colsample_bytree)
Values {0.1,1,10,100,1000} {100,300,400} {5,6,7,8,9,10} {1,10,100,1000} {0.01,0.1,0.3} {200,300} {3,4,5,6} {0.6,0.7,0.8} {0.5,0.6,0.7} EN ES
LR RF SVM XGB LR RF SVM XGB
M1 M2 M1 M1 M1 M1 M1 M1
N-grams uni- and bigrams uni- and bigrams uni- and bigrams uni- and bigrams bigrams uni- and bigrams bigrams uni- and bigrams compared to the cross-validation results (79% vs. 83% for the Spanish dataset and 69% vs. 76% for the English dataset), so we used a different approach during the final testing phase.
The ensemble method we used for the final version of our software (described in detail in Section 3.2.3) required the best text cleaning and vectorization parameters and hyperparameters for each model. These hyperparameters are summarized in Table 2. 8 Parameter names in the relevant Python package/function. Detailed description in Table 1.
Apart from the n-gram based models, we constructed a model based on statistical variables describing all hundred tweets of each author, thus giving one more prediction per author. The variables used in this model are as follows: • the mean length of the 100 tweets of the authors both in words and in characters; • the minimum length of the 100 tweets of the authors both in words and in characters; • the maximum length of the 100 tweets of the authors both in words and in characters; • the standard deviations of the length of the 100 tweets of the authors both in words and in characters; • the range of the length of the 100 tweets of the authors both in words and in characters; • the number of retweets in the dataset by each author; • the number of URL links in the dataset by each author; • the number of hashtags in the dataset by each author; • the number of mentions in the dataset by each author; • the number of emojis in the dataset by each author; • the number of ellipses used at the end of the tweets in the 100 tweets of the authors; • a stylistic feature, the type-token ratio to measure the lexical diversity of the authors (in the dataset each author has 100 tweets thus the number of tokens per author does not differ as much that it would cause a great diversity in the TTRs).
This gives a total of 17 statistical variables. Since we used an XGBoost classifier, we did not normalize the variables and the linear correlation between the variables posed no problem.
To find the best hyperparameter set, we used a five-fold cross-validated grid search and finally refitted the best model on the whole data. The cross-validated accuracies achieved this way are 70% and 74% for the English and Spanish data respectively. Table 3 contains the best hyperparameters found.
Column sample by node Column sample by tree gamma Learning rate
Max depth
Min child weight Number of estimators alpha Subsample EN 0.9 0.2 1 2 2 4 200 0.1 0.8 ES 0.8 0.8 4 0.3 3 5 100 0.3 0.8
After identifying the best hyperparameters for the five mentioned models with cross-validation, we had to find a reliable ensemble method. To avoid overfitting this ensemble model to the training set, we did not train it using the predictions of the five final trained models. Instead, we wanted to create a dataset that represents the predictions that are produced by our models. To do this, we refitted the five submodels with the cross-validated hyperparameters five times on different chunks of the original training data (each consisting of tweets from 240 users). The predictions given by these five models to the 60 remaining users were appended to the training data of the ensemble model, thus this training set consisted of predictions given to all 300 users in the training data, but these predictions were given by five different models in case of each model type. The sample created this way can be interpreted as an approximation of a sample from the distribution of the predictions of the final five models on the test set. We created a test set with the same method but with a different split of the training data.
We then used these constructed training and test sets to find the best ensemble from the following three methods: majority voting, linear regression of predicted probabilities (this includes the simple mean), and a logistic regression model. The best and most reliable results were given by the logistic model; therefore, we used this model as our final ensemble method. Table 4 summarizes the logistic regression coefficients for the probabilistic predictions of each model for both languages.
The validity of this method is backed by the fact that our results on the training sets (an accuracy of 75% and 81% for the English and Spanish set respectively) were only slightly better than the final test results.
As mentioned in Section 3, we tested two versions of our software. For the early bird testing, we used the single best n-gram models based on our cross-validated grid search (a random forest classifier for the English set and a support vector machines classifier for the Spanish set). Using these models, we experienced a significant decrease in the accuracy of the models compared to their cross-validated performance, so this was one of the reasons why we decided to incorporate a number of different models for our final software. As Table 5 shows, relying on a number of different models and a statistically based ensemble method proved to be a good solution. First, the cross-validated accuracies of our final models were almost the same as their accuracies on the test set, and second, our final software was able to reach a higher accuracy in both languages than our early bird solution.
One of the unanswered questions that emerged during this project is concerning the reasons behind the fact that our models are better at identifying fake news spreaders that tweet in Spanish. This is true about all of our individual models regardless of the features they used, and about the final ensemble model as well. We assume that it would be beneficial to conduct some qualitative research about the tweets in the dataset to better understand why fake news spreaders that tweet in Spanish are more distinguishable from regular users than those that tweet in English.
Another promising direction for achieving higher accuracy in profiling fake news
spreaders is to develop a software that is able to determine whether a single tweet
should be considered as fake news. It is reasonable to assume that even those that are
labelled as fake news spreaders only post some tweets that can be considered as fake
news, while some of their posts are just regular tweets. Therefore, from the
perspective of our approach, the current dataset is likely to contain a lot of noise. If
we were able to identify fake news on the level of tweets, we could build a model
relying on this information that would allow us to give predictions for each tweet.
This approach was unfortunately not executable with the PAN20 Fake News
Spreaders dataset [
In this notebook, we summarized our work process of preparing a software for the
PAN 2020 Profiling Fake News Spreaders on Twitter task [