With the diffusion of Web and Social Media, automatic user profiling classifiers applied to digital contents have become extremely important in application contexts related to social and forensic studies. In many research papers on this topic, an important part of the work is devoted to a costly manual "feature engineering" phase, where the semantic, syntactic, and often languagedependent features need to be accurately chosen to be relevant for profilation task. Differently from this approach, in this work we propose a Twitter user profiling classifier which exploits deep learning techniques to automatically generate user features being a) optimal for user profilation task, and b) able to fight covariance shift problem due to data distribution differences in training and test sets. In the best configuration found, the built system is able to achieve very interesting accuracy results on both English and Spanish languages, with an average final accuracy of more than 0.83.
Since the 19th century, authorship identification (AI) analysis have been used in several
contexts [
Previous editions have shown a prevalence of software solutions based on traditional
machine learning approaches, based on costly manual feature engineering phases.
Except for some works, the usage of language modeling and deep learning techniques for
textual features encoding has been quite limited and the level of accuracy reachable by
using only these techniques is unclear. In this work we thus propose a software
solution strongly based on these recent approaches able to a) perform textual encoding in a
language-agnostic way and with almost no manual feature engineering, and b)
attenuate the data covariate shift [
The rest of this paper is organized as follow. In Section 2 we briefly introduce related works on this topic, and in Section 3 we show how we performed text encoding. After having introduced the covariate shift problem of challenge datasets in Section 4, we describe in details the design of the proposed solution in Section 5, the experimental results in Section 6, and the conclusions on Section 7. 2
Until 2017, "Author profiling" task in PAN challenges was focused on age and gender
identification [
Two main different types of feature representation have been used in past works. A
first one is a traditional approach with features (content-based or style-based) selected
through a manual "feature engineering" phase. Many works [
The classification approaches used by many works [
A common approach to encode textual contents is to use a NLP (Natural Language
Processing) processing pipeline [
The first type of text encoding used (called W2V) is based on the usage of a language
modeling algorithm very popular in the last years: word2vec [
Given a raw tweet, in order to obtain a vector representation to be used as text encoding with machine learning methods, we proceed as following. We first convert the text to lowercase, next we replace each distinct URL with the word "url" and we remove all punctuation characters. If present, we remove also all the emoticons occurring in the text. The remaining textual content is then tokenized into distinct tokens (eventually appearing more than 1 time) and the final encoding vector is computed as Vtweet =
N i=1 1 XN we(token(i)) (1) where Vtweet is the final tweet vector, N is the total number of tokens belonging to final tokenized tweet, token(i) is the function returning the token at position i in the tokenized vector, and we(x) is the function returning the word vector associated to token x.
For both target languages, we used precomputed 300-dimensional embedding models built on big data collections: GoogleNews for English2 and SBWCE for Spanish3. 3.2
The W2V encoding described previously has the main drawback in trying to only capture semantic information from Twitter data, skipping all the remaining knowledge related to write-style and syntax used by the different type of users to write tweets. In 1 The meaning of this hypothesis is that words appearing in similar contexts often have a similar meaning. 2 Available at http://tiny.cc/0vhz6y 3 Available at https://bit.ly/2Q9DBzU Syntax tweet encoding: encoder Chars tweet encoding: encoder learning (2) learning order to extract syntax-related information from textual data, we proposed an encoding method based on deep learning technology, as shown on Figure 1a.
The original tweet text is tokenized before being fed to the encoding neural network. The tokenization process is performed in the following way. We replace all distinct URLs with the word "url" but we do not remove neither the punctuation nor the emoticons found on text. Each distinct user mention (e.g. @realdonaldtrump) and hashtag (e.g. #politic) is replaced respectively by the words mention and hashtag. Each remaining word is transformed into one of these 3 words: uppercase_w for a completely uppercase word, lowercase_w for a completely lowercase word, and mixed_w for a word including both uppercase and lowercase characters.
In order to learn a tweet encoder using the architecture proposed in Figure 1a, we
used only training data split and transform the dataset of our original problem
(proposed to classify users into profiles) into a new dataset suitable to classify tweets into
user profiles. For each user profile in training data, we thus take its tweets and its
original assigned label (bot, female, or male) and we extract 100 different tweets where each
one has assigned the label of the user. At the end of this process we have produced 2
new training datasets, one for English and one for Spanish, consisting in respectively
288,000 and 208,000 training documents, which can next be used to learn the two
profile classifiers. The proposed neural network uses two sub-networks to learn different
feature types. A first part exploits a GRU net [
The syntax classifiers so obtained are not interesting in themselves, what it does matter for us is the possibility to take the trained nets and extract, as tweet encoding vector, the output produced by "Encoder layer" layer. The resulting vector should be in fact an effective representation of the syntax information convey by the original tweet. 3.3
This encoding, differently from W2V method, want to exploit data at sub-word level in order to find interesting recurrent patterns and potentially useful information for user profile classification. The tokenization process of original tweets is simpler than the others two seen encoders. In this case, we have defined the set of valid symbols in the characters dictionary as (A Za z0 9), the punctuation symbols, and all the distinct emoticons found on the training data split. The tokenization process thus simply consists in removing all invalid characters from input raw text. In the same manner as described for SYN encoder, starting from training data split, we create two new different training datasets to solve a tweet ) user_profile classification task. As shown on Figure 1b, the neural network architecture used to learn the encoder is slightly different from that used in SYN encoder. The network only use a CNN sub-network for encoding the tweet’s data and the layer taken as final tweet encoder is the CNN layer. 4
The first attempt to build a user’s profile classifier for the "Bots and Gender Profiling"
task was to take all the data included in train and validation splits, merge them together,
and then try to perform a k-fold evaluation [
For each language, we designed a single-label multiclass classifier [
We tested three different types of textual encodings:
Architecture of final user’s profile classifier Bot
Male
Female Tweet enc 1 TE Tw1eet
SVM classifier User Encoder
Tweet Tweet enc ... 1e0n0c 2 TE
TE
Tw2eet ... T1w0e0et Classifier architecture
BUE encoder CBUE DUE encoder BUE encoder CDUE Masked DUE vector DUE encoder BUE encoder CmaskDUE
W2V
W2V-SYN
W2V-SYN-CHARS
In the remainder on the section, we will describe the two proposed core user encoders (BUE and DUE encoders) used in tested configurations, the strategy used to mask most discriminating features, and how we built the final user’s profile classifier. ncoding user profiles (BUE ncoder) e training data ly. code users with tor of size 512 ted various ers DL subnets: M, M+CNN, GRU, U+CNN, In this section we describe a first type of user encoder (called BUE, Base User Encoder) which is able to process user timelines and provide good vector representations of the user profiles. In order to auto-learn very informative features for the final task, we designed a classification system based on a deep learning architecture (see Figure 3a) and trained it using only train data split. The final built classifier, thanks to CNN sub-network4, learn how to encode properly the sequence of tweets submitted by the user by identifying the space-invariant inter-tweets patterns most relevant to solve the classification problem. In particular, the trained network can be used to compute a good user vector representation starting from its timeline. We tuned the network to have 512 as vector size for user encoding layer (layer "User encoding layer"), while for CNN we used 512 different filters with a window size set to 10. 5.2
User encodings produced by BUE encoder are optimal user representations for training data split and final classification task, but as we have seen before are suboptimal for encoding and classify test datasets due to different characteristics of data. The main aim of this step is thus to start from BUE vector representations and try to find similar user representations, suitable for classification task but where the features high discriminative for training and test distributions are highlighted and easily recognizable. In order to tackle this issue, we propose a discriminative user encoder (called DUE, Discriminative User Encoder) based on the neural network architecture shown on Figure 3b. 4 We tested other variants of deep learning architectures (e.g., using sub-nets based on LSTM, GRU, or a combination of them with CNN) but we finally choosed CNN because it was able to return back slightly better user vectors encoding (giving better effectiveness on final classifier). This encoder is trained on a new dataset composed by both train and validation data but where each user has different target labels respect to original task: "True" if the user was originally included in validation split, "False" otherwise. The neural network structure is very simple. It takes as input a user vector representation computed by BUE encoder and then in the first hidden dense layer (Discriminant user encoding) try to learn a user representation optimal for this new binary classification task. To ensure that the net would learn a user representation suitable also for original "Bots and Gender Profiling" task, we introduce a constraint of what the net can learn at this level. In particular, we force the net to minimize two different losses in gradient descent optimization: the loss on labels related to data distribution (Ldistribution), and the loss due to the difference from learned encoding and the encoding produced by BUE encoder (Luser). With the imposed constraints we aim to obtain a trained net which will produce user vectors of dimension 512 very similar to those produced by BUE encoder but containing some specific feature resulting very discriminative in recognizing the original data distribution of a user. 5.3
DUE encoder provides user vector representations containing some high informative
features for identifying the origin of the data distribution of a user. A feature is high
discriminative for data distribution if, after having built a binary classifier using only
the considered feature, we are able to distinguish quite well the distribution origin of
a user. To weight the discriminative power of a feature we can use ROC-AUC [
On Figure 4 we show a typical distribution of the features related to their ROC-AUC values emerged from the tests. The majority of features have a ROC value around 0.55 but a long queue exists on the right with few features having high discriminative power (up to more than 0.8). These last type of features are the those we want to mask in the final user encoding vector. Indeed, without these features the user vector should be less sensible to distribution origin and therefore, in the successive learning phase, we force the classifier to concentrate on the other distribution-invariant features to solve the final task. 5.4
The final user’s profile classifier has been choosed and tuned using remaining train and validation data not included in dataset Dmask. We tested two different classifiers (Random Forest and SVM) but we finally adopted SVM because it performed better in terms of accuracy in all tested configurations. The classifier has been optimized by fine-tuning model parameters through the use of a k-fold validation. For both languages and independently from the used text encoding, the best found configuration uses an RBF kernel and standard parameters "gamma" and "C" set to respectively 0.01 and 0.1 . The best threshold determining the ROC minimal value over which we apply 3) Mask most important discriminant features ▷ RandomForest classifier ▷ Mask features with
ROC-AUC value >= chosen threshold. feature masking depends on the adopted text encoding and it will be reported in the experimental section.
After found the best model parameters and chosen a threshold for ROC value, we built the final optimized classifier using train split as training data and using validation split as test data to measure the accuracy of a specific solution. The final models submitted on early test dataset for evaluation have instead been built using all the data available (the union of data in training and validation splits). Detailed results of the tested configurations will be presented in the experimental section. 6
In this section we will report all the results obtained in our experimentation. A detailed
description of the dataset used in the challenge is available in [
For each language, on Table 1 we report the system global accuracy on validation dataset obtained a) using the three different strategies seen above for encoding tweets and b) using three ways to encode the user vector before building the final classifier. In particular, we report the 3 types of user encoder configurations introduced in Section 5, together with cut threshold for masking features ("Masking threshold" column). This first type of experiment has the main aim in identifying, for each language and each tweet encoding, the best user encoding which next will be used to test the system 5 Available at https://keras.io/ . 6 Available at https://www.tensorflow.org/ . 7 Available at https://scikit-learn.org/stable/ . performance on early test dataset. For this reason, for each language and each tweet encoding tested, we also report in bold the best user encoding found.
On both languages, the results for W2V-SYN and W2V-SYN-CHARS configurations show that the better results are obtained by using CmaskDUE configuration, while for W2V the best ones make use of CDUE without using feature masking. The optimal threshold identified for the various configurations is very similar and is always comprised between 0.65 and 0.70. It is also interesting to note that the accuracy for W2VSYN always increases while passing from CBUE to CmaskDUE configurations. W2VSYN-CHARS instead, thanks probably to a richer representation of input data, seems instead to be a strong competitor also using the base CBUE user encoding.
On Table 2, we report a comparison between the accuracy obtained for both languages on validation data and on the early test dataset provided by the challenge organizers. The reported results are referred to the best configurations identified on Table 1 and show the accuracy obtained on the 2 sub-tasks defined in the competition ("Type task" and "Gender task" columns) together with the averaged global accuracy of the system ("Global" column). For each type of dataset and each language, we have also reported in bold the best results.
Globally, the results show clearly that W2V-SYN is the best performer, in particular, W2V-SYN with CmaskDUE on early test dataset and for both languages, it always provided the best accuracy. W2V-SYN-CHARS seems to be competitive only on validation dataset but on early test dataset the accuracy is not very good, especially on "Gender task". A possible explanation of this worst performance respect to W2V-SYN is that the addition of features generated by CHARS encoder confuse the classifier while learning the best parameters to build a classification model invariant to data distribution. W2V is instead the worst performer on both datasets, probably a sign that this type of text encoding has a limited expressivity in representing the data of the classification task.
If we observe the results from the point of view of languages and classification subtasks, we can note the the proposed configurations always perform better on English, with a remarkable difference in terms of accuracy between the two classification subtasks. In particular, solving the "Type" task seems a lot easier than solving the "Gender" task. This difference is probably due from one side to the greater number of training examples available for "Type task", from the other side for the greater difficulty8 in identifying the differences in the write style between males and females.
Finally, on Table 3 we show the accuracy results obtained by our best configuration (W2V-SYN with CmaskDUE encoder) on the test dataset used for final evaluation of system performance in the challenge. In addition, on the table we report also the accuracy obtained by various baseline methods proposed by organizers. Our method outperforms all the baselines, in particular seems more effective especially on Spanish language. On both languages, the global accuracy results obtained by our algorithm on final test dataset are quite similar to those obtained on early test dataset (0.8409 and 0.8366 vs. 0.8523 and 0.8250). An interesting observation is that in final test dataset the difference in terms of accuracy between the two sub-tasks "Type" and "Gender" is very similar for both languages, suggesting that the proposed solution is a good balanced system and not sensible to the target language. 8 If we manually inspect a timeline of a human user of the dataset, it is not always so easy identify its gender. Generally, a bot user is instead easily recognizable because certain types of writing patterns occurred frequently in the timeline.
In this work we tackled the "Bots and Gender Profiling" task of PAN 2019 challenge by proposing a software solution making extensive use of state of the art language modeling and deep learning techniques. In particular, these type of algorithms have been used in order to automatically performing feature encoding of textual contents, avoid almost completely the usual costly and manual "feature engineering" phase. In addition, deep learning has been also exploited to find textual features invariant to train and test datasets, helping fighting data covariate shift which usually has remarkable negative effect on final classifier accuracy. The experimental results show that, among all tested configurations, the solution using W2V-SYN textual encoding and combined with user encoder CmaskDUE is globally the best performer on both validation and early test datasets. In particular, on the final test set, such configuration allow to obtain a balanced multilingual classifier able to obtain very interesting accuracy results on both languages, with a global combined accuracy of 0.8387.
Possible future works could try to use more advanced language modeling techniques like BERT and Elmo for textual encoding. These methods provide contextual embeddings for text representation, by automatically disambiguate both the terms and the used syntax in analyzed contents. A comparison with the textual encoding methods proposed here could thus be very helpful in order to address future research directions. 27. Veenhoven, R., Snijders, S., van der Hall, D., van Noord, R.: Using translated data to improve deep learning author profiling models. In: Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018) (2018) 28. Yule, G.U.: On sentence-length as a statistical characteristic of style in prose: With application to two cases of disputed authorship. Biometrica 30, 363–390 (1939)