This paper describes an approach to bot and gender author profiling that makes use of a weighted ensemble of convolution (CNN) and recurrent (RNN) neural networks based on char and word n-gram models alike. The proposed ensemble model is shown to outperform the use of its individual classifiers alone, and it was submitted for participation at the PAN-2019 author profiling shared task.
Author profiling is generally understood as the computational task of inferring an
individual’s demographics from the text that they have written. The present work addresses
two particular instances of author profiling - gender and bot versus human recognition
- as developed in the context of the PAN-2019 bot and gender profiling task [
The tasks proposed at PAN-2019 consist of determining (in supervised fashion) whether the author of a given piece of text is a bot or a human and, in case of human authorship, to determine their gender. To this end, two datasets were provided, containing 412.000 tweets in English and 300.000 tweets in Spanish, respectively. Both datasets are labelled with author type (human or bot) and, in case of human authors, with gender information (male or female).
Central to our current work, we notice that deep learning methods, although
generally successful in many fields, have been applied to author profiling tasks with varying
degrees of success. In particular, we notice that the two best-performing systems at
PAN-2017 author profiling task [
Based on these observations, the present work proposes a new take on deep learning methods for author profiling by presenting a combined approach that makes use of a weighted ensemble of convolution (CNN) and recurrent (RNN) neural networks based on char and word n-gram models alike. In doing so, our goal is to assess whether the combination of these resources may outperform the use of their individual classifier components alone, and whether the ensemble method may produce competitive results in bot and gender profiling as proposed at the PAN-2019 shared task.
In this section we briefly review a number of recent studies that address popular author profiling tasks such as age and gender recognition by making use of deep learning methods.
The work in [
The use of deep learning methods is also the focus of the study in [
The work in [
Finally, the PAN-CLEF 2018 shared task in [
Our current work is motivated by the assumption that different learning strategies and text representations may provide multiple contributions to the tasks at hand, namely, bot and gender author profiling. To this end, the proposed model - hereby called CNN+RNNchar-word - combines four neural models into an ensemble setting: char- and word-level convolutional networks (CNN), and char- and word-level recurrent networks (RNN). To this end, both CNNs follow a multichannel architecture for variable-length n-gram representations, and both RNNs follow a long short term (LSTM) architecture with self-attention. The ensemble architecture is illustrated in Figure 1.
The weights assigned to each of the individual classifiers are computed by searching
for an optimal confidence estimate for each model so that the combined output error
is minimised. Optimization proper is performed by making use of Simplex [
w~ = min X fi(x) i wi (1) one-hot char unigram one-hot word unigram
x Pin=1 fi(x) w~i y~
In this equation, for each model i, we vary the voting weights w 2 R between 0 and 1 so as to minimise the error rate of the combined models f given by the factor w~, which represents the relative weight of the model in the overall results.
Input texts are individually transformed into one-hot character- and word-level vectors as required by each model. Vectors are taken as initial weights for the input layers in both CNN and RNN modules, and subsequently optimised through back-propagation.
The CNN modules make use of 1D convolutional layers with max pooling, using filters of size 3 and 4, both of which with mappings of size 64, and using ReLU as an activation function, a 0:003 L2 regularization max pooling of size 2. This is followed by a fully-connected layer containing 1024 neurons using ReLU and a 0.35 drop-out regularization and a softmax output layer. Training was performed in mini batches of size 32 with RMSProp optimization and using entropy loss as a cost function. Validation was performed over a 20% portion of the data until convergence with Early Stopping.
The RNN models make use of a self-attention mechanism and memory of size 64 with a 0:12 drop-out regularization. This is followed by a hidden layer conveying 1024 neurons using ReLU as an activation function, and a softmax output layer. Training is performed with AdaDelta optimization and using entropy loss as a cost function. Validation is performed over a 20% portion of the data until convergence with Early Stopping.
Results from the four models are combined using a weighted voting strategy based on a confidence estimate for each individual model according to their results during the training stage. 4
The combined CNN+RNN-char-word ensemble model described in the previous section was tested against its individual components, namely, CNN-char, CNN-word, RNN-char and RNN-word. In addition to that, a simple logistic regression baseline - hereby called LogReg - using a bag-of-words model with LibLinear optimization, L2 regularization and =100 was also implemented.
The ensemble voting mechanism is based on the weights assigned to each individual
classifier during training by performing Simplex [
From these optimal weights we notice that the bot recognition problem is more generally captured by the RNN-char model, whereas gender profiling seems to be best captured by the RNN-word model. 5
In this section we report results of the proposed Ensemble approach and baseline
systems applied to the bot and gender profiling tasks at PAN-2019, using the evaluation
tool provided [
Bot Recognition This paper presented an approach to bot and gender profiling that makes use of a weighted ensemble of convolution (CNN) and recurrent neural networks (RNN) based on char and word n-gram models alike. The Ensemble approach outperformed a number of baseline alternatives making use of single network and language models, but a certain accuracy loss was observed in the final validation.
As future work, we intend to investigate how the individual components of the model interact, and extend the current architecture by making use of word and character embeddings. The authors acknowledge support by FAPESP grant 2016/14223-0 and from the University of São Paulo.