In this document we present our approach for profiling fake news spreaders. The model relies on semantic features, part-of-speech tag related features and other simple features. We have reached an accuracy of 0:697 and 0:810 for English and Spanish, respectively, on validation data. Test accuracies using these same models reach 0:690 and 0:725 for English and Spanish data. We believe that this is a simple and robust model that could potentially be used as a baseline for this task.
In this paper, we present our method for the Shared Task on profiling fake news
spreaders [
We expect that the topics (or meaning) that a fake news spreader covers will differ from the ones that other users cover. Besides, we expect that the used part-of-speech (POS) tags will be good predictors, as this is a common source for author profiling. We also included the average tweet length in characters and also the uppercase/lowercase letter ratio, with the expectation to capture fake news spreaders.
This document is structured as follows. First, we mention the resources that we have employed. Then, we give more information about how the representation of each user is built. We continue mentioning the classifiers that we have tested. Finally, we discuss the results and provide some insights for possible future directions.
We have trained our models on the data published by the organizers of the Shared Task
on Profiling Fake News Spreaders2 [
In order to build the required user representations, we employed a number of resources that are presented below.
Currently, semantic representations are built using word embeddings, for which we
employed a collection trained on Twitter [
For the part-of-speech (POS) tagger, we decided to build our own POS tagger, as
commonly used POS taggers may not work so well with the language from Twitter
because of shortened words, specific slang, and so on. We built a Hidden Markov Model
POS tagger [
Following the expectations mentioned at the beginning of the article, we assume that using the average word embedding representation from all words that a user has written, we get an approximation of the semantic content that they published. Hence, we represent a user as an average embedding (200 dimensions). We also include the standard deviation of each dimension. We do not do any further lemmatization, stemming or preprocessing to the tweets.
We include a bag-of-pos, which encodes the frequency of each part-of-speech, but we normalize it by dividing with the most frequent part-of-speech frequency, and hence, all numbers are at the range 0 : : : 1. While in the English version this bag contains 53 different tags, the Spanish tagger can capture 18 different tags.
We then add some commonly used simple features, such as, the average length of tweets in characters, and also the ratio of uppercase letters. We calculate this last number by just counting the uppercase letters and dividing them by the sum of the uppercase and lowercase letters. 4
We decided to use two linear classifiers, especially Logistic Regression and Linear
SVM, as they can be trained very fast, and they give a good insight of how a set of
features work. We further decided to include a non-linear model, such as the Multilayer
Perceptron and the Random Forest, because of its popularity in text classification tasks.
The MLP classifier was trained with three hidden layers of size 50. All other classifiers
were trained using the default parameters from the Scikit-learn package [
In the table below we can see the results of the different classifiers. We validated the models using Stratified K-Fold Cross-Validation with K = 5.
Logistic Regression Linear SVM Multilayer Perceptron Random Forest
English Spanish 0:500 0:500 0:677 0:720 0:503 0:550 0:677 0:703 0.697 0.810
Considering these results, we decided to use the Random Forest model as our final model for testing. We got a test accuracy of 0:690 for English data and 0:725 for Spanish.
Further experiments We further experimented including grammatical errors. We included information about misspellings by using the Python package pyspellchecker6 to detect whether there are misspelled words, and afterwards, we control which letter becomes which letter. Therefore, if the alphabet has 27 letters, we create a vector of 272 numbers and we save how often each variation happens. The goal of this representation was to capture systematic errors that a user may make, hoping that this would be representative.
The results of this experiment can be seen below, using the same classifiers as before, and also validated under the same conditions (Stratified K-Fold Cross-Validation with K = 5).
English Spanish 0:577 0:720 0:570 0:677 0:600 0:693 0.720 0.773
Unfortunately, these last test was done out of competition time, and therefore, we could not test this models performance on test data. 6
In this paper, we presented a model that could potentially be used for capturing fake news spreaders. Considering average accuracy, our model ranked 31st out of 66 participants in the competition. The ranking also includes five different baselines: A model
that was used for language variety identification [
In the box plot below, we illustrate how our model performs compared to the other participants. Note that outliers such as authors that have not participated in specific language configurations, have been discarded.
The presented model is relatively simple and efficient, but we believe that results can still be improved. We mention here some possible future directions.
As the character-based SVM model performs better than our model, we believe that adding character-aware representations can boost our performance. This could be done either using character n-grams or using a character-based Recurrent Neural Network to build representations.
Apart from that, we have not done any preprocessing in this work. Considering
the language use on Twitter [
In the current work, we trained a very simple Hidden Markov Model for POS
tagging. This model may fall short because of the high number of misspellings in
social media language. This effect could be reduced by training a POS tagger using a
character-level tagger, such as a BiLSTM+CRF model [
The author would like to acknowledge all participants from the course Language Processing 2 (spring semester on 2019/2020) at the M.S. program in IT & Cognition at the University of Copenhagen, as the majority of the ideas presented here flourished in the discussions during class.