Fake news is an emerging problem in online news and social media. Efficient detection of fake news spreaders and spurious accounts across multiple languages is becoming an interesting research problem, and is the key focus of this paper. Our proposed solution to PAN 2020 fake news spreaders challenge models the accounts responsible for spreading the fake news by accounting for different types of textual features, decomposed via sparse matrix factorization, to obtain easy-to-learn-from, compact representations, including the information from multiple languages. The key contribution of this work is the exploration of how powerful and scalable matrix factorization-based classification can be in a multilingual setting, where the learner is presented with the data from multiple languages simultaneously. Finally, we explore the joint latent space, where patterns from individual languages are maintained. The proposed approach scored second on the 2020 PAN shared task for identification of fake news spreaders.
The notion of fake news refers to distortions of news with the intention to affect the
political landscape and to create confusion and divisions in society. Even if the
phenomenon of fake news is not new, the scale and impact of fake news has never been so
important than today, which can be attributed to the digital transformation of the news
industry, and especially to the rise of social media as a news distribution channel. [
One of the crucial problems is the recognition of fake news spreaders. For example,
Twitter bots (fake accounts) are capable of generating fake information and propagating
it through their follower networks, which can impact real-life entities such as stock
markets and possibly even elections [
This paper is structured as follows. In Section 2 presents related work, Section 3, we discuss the problem addressed in this work. Next, in Section 4, we discuss the proposed method, followed by empirical evaluation and discussion. 2
A critical mass of fake news can have serious, real-life consequences, and can for
example impact election process [
Twitter fake news spreaders can be captured in their own social bubbles, which was
shown to be an efficient defense tactic [
Traditional classifiers with extensive feature engineering seem to be pervasive in
the literature about distinguishing between bots and humans but there was also some
attempts to tackle the task with neural networks. In the recent work, [
Provided a timeline of chosen tweets of ground truth labeled data consisting of fake news spreaders and non-spreaders, the goal is to decide if a new author is a spreader of fake news or not. Formally, we are given a decision problem which states: Given an author A who tweets in language L 2 fEnglish _ Spanishg and from the collection of tweets C, given a subset of tweets CA (of an author A),
CA = t1; t2; : : : ; tn
where ti represents a tweet content, find a decision function that maps f : CA 7! author reliability, hence
(0 a non fake-news spreader; f (C(A)) =
1 a fake-news spreader; This decision problem is specialization of the problem of author profiling. It requires learning a representation from CA, suitable for approximating f . The provided data consists of tweets by 300 English and 300 Spanish authors respectively, respectively.
For each author 100 tweets are provided making a total of 300000 English and
300000 Spanish tweets. The balance of classes is consistent for both languages, both
having 150 negative and 150 positive samples, as shown in Table 1.
The following section includes description of the proposed method with the
corresponding intermediate steps.
1. From the original data punctuation is removed
2. URL and hashtags are removed from the result of step (1)
3. stop-words are removed from the output of step (2).
For each author’s collection of tweets we initially define a collection of candidate n
features from the pre-processed data which are iteratively selected and weighted, similarly
to Martinc et. al. [
At this we have prepared word and character features from each author’s collection of tweets, ready to be used in the feature selection step. 4.3
Next, we perform sparse singular value decomposition (SVD)1[
M = U
V T : The final representation (embedding) E is obtained by multiplying back only a portion of the diagonal matrix ( ) and U , giving a low-dimensional, compact representation of the initial high dimensional matrix. Note that E 2 RjDj d, where d is the number of diagonal entries considered. The obtained E is suitable for a given down-stream learning task, such as classification (considered in this work). Note that performing SVD in the text mining domain is also commonly associated with the notion of latent semantic analysis. 4.4
Classification model we aimed for in this task was to be robust yet highly flexible,
one that will score well on the prepared data without using many features or extensive
processing power. Following this goal we conducted a series of experiments, trying
different representations with corresponding linear models as presented in Section 5. The
classifiers used were the following (from scikit-learn [
Considering the size of the dataset and the distribution of the data within the dataset, we preformed a series of experiments. All of them aimed to test the pipeline described in the Section 4. The experiments conducted can be divided into two main categories, based on the language considered by a given model: 1. Multilingual - Both languages’ data is fused together and is subject to the same feature construction and representation creation steps. 2. Monolingual - For each language in the dataset, English and Spanish, we create a separate pipeline, that is also executed exclusively on the data from a given language.
For both approaches we performed extensive grid search over parameter space to find best hyper-parameter configuration with the help of Scikit’s Learn GridSearchCV function. By doing 10-fold cross validation, the grid consisted of reducing the dimensions parametrized by k in the following interval:
k 2 [128; 256; 512; 640; 768; 1024] and the number of generated n features from the interval
n 2 [2500; 5000; 10000; 20000; 30000]: 1 https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html Monolingual variant was based on splitting the data from each language separately into training 90% and 10% validation set, obtaining 270 training examples Ctraining and 30 validation examples Cvalidation. Such splits were obtained for each language. Only training data was used for feature construction and dimensionality reduction. Multilingual variant merged the data from both languages and after that the same approach as previously was applied. Merging the data from both languages potentially reduces the computational load required to train two separate models. Data was split into training 90% and 10% validation set, obtaining 540 training examples Ctraining and 60 validation examples Cvalidation. In each iteration we generated n features in R540xn, reduced them to dimension k obtaining a matrix from the space R540xk. g(Ctraining, n features) : RNxn SV!D RNxk where g denotes the 4.3 process.
Once constructed, the feature space was subject to learning. We experimented with both logistic regression and linear SVMs and in initially some experiments were conducted with RandomForest model, of which hyperparameters we optimized in 5-fold cross validation considering the size of the dataset. Finally, we tested the performance on the Cvalidation set.
We visualise the distribution of the dataset reduced to 2 dimensions using UMAP
[
We constructed two baselines one that was based on TF-IDF on Logistic Regression (LR) with L1 regularization and the second was doc2vec modeled with RandomForest (RF) as classifier. The array of experiments conducted yielded the results presented in Table 2, and the outcomes of our final submission in Table 3.
As discussed in Section 5 all training was conducted by using Ctraining data and the validation was done on Cvalidation set. The next presented Table 2 shows the model results as measured on TIRA training evaluation on the whole Cvalidation [ Ctraining data.
The final un-official evaluation as reported on TIRA’s page is presented in Table 3.
The Model column in Table 2 refers to the classifiers used, such that if two classifiers are present the model is monolingual - the first classifier is for English and the second one for Spanish and in case the model is multilingual only one classifier is used. The type column discriminates between the number of languages the model is trained on. Name column consists of vectorizer used and is followed by dimension size or type of tokinizer used or, dimensions column denotes the number of dimenstions SVD reduces to.
As it can be seen the highest evaluation score on our training data was obtained by the multilingual model tfidf_large, with the following hyper-parameters: k = 768 dimensions, n = 5000 features, Logistic Regression classifier with 2 = 0.002 and fit_intercept= F alse.
Monolingual model that preformed best is tfidf_cv which for English is paramatrized as SVM model with the following hyper-parameters: = 0:001, 1 = 0.8 while penalizing elastic-net, loss-function = hinge and power_t = 0.5 and for Spanish of SVM model with hyper-parameters: = 0:0005, 1 = 0.25 while penalizing elastic-net, lossfunction = hinge and power_t = 0.9 .
The more detailed insight into the performance of the best performing models over the inference of the number of word and char n-grams and the accuracy on the 5fCV of the models is also given in Figures 5 and 6. The figures show the performance of the best mono and multilingual models – the confidence intervals indicate the variability obtained when repeating the experiments The code and the pilot experiments are freely available at https://gitlab.com/skblaz/pan2020. 8
The series of experiments conducted as a part of this work indicates, n-grams for the
task of Author Profiling are still sufficient and method compared to more complex
methods as transformers and word2vec [
Further on we can focus on exploring the possibility for detecting fake news profiles across different languages by first considering Latent Semantic Analysis across different language settings, further segmenting the semantic space prior to learning. 9
The work of the last author was funded by the Slovenian Research Agency through a young researcher grant. The work of other authors was supported by the Slovenian Research Agency (ARRS) core research programme Knowledge Technologies (P2-0103), an ARRS funded research project Semantic Data Mining for Linked Open Data (financed under the ERC Complementary Scheme, N2-0078) and European Unions´ Horizon 2020 research and innovation programme under grant agreement No 825153, project EMBEDDIA (Cross-Lingual Embeddings for Less-Represented Languages in European News Media). 25. Škrlj, B., Martinc, M., Kralj, J., Lavracˇ, N., Pollak, S.: tax2vec: Constructing interpretable features from taxonomies for short text classification. Computer Speech & Language p. 101104 (2020)