This paper describes our approach for the Author Profiling Shared Task at PAN 2017. The goal was to classify the gender and language variety of a Twitter user solely by their tweets. Author Profiling can be applied in various fields like marketing, security and forensics. Twitter already uses similar techniques to deliver personalized advertisement for their users. PAN 2017 provided a corpus for this purpose in the languages: English, Spanish, Portuguese and Arabic. To solve the problem we used a deep learning approach, which has shown recent success in Natural Language Processing. Our submitted model consists of a bidirectional Recurrent Neural Network implemented with a Gated Recurrent Unit (GRU) combined with an Attention Mechanism. We achieved an average accuracy over all languages of 75,31% in gender classification and 85,22% in language variety classification.
Social media has become an important platform for communication and exchange of information. In contrast to classical letters and emails, the language on social media is much more personal. This raises the question whether the text style and content allows to draw conclusions about demographics traits of its author, such as age, gender, or language variety. Such insights can be used in various applications, such as forensics, security, or marketing. For instance, on the basis of such profiles it would be possible to determine which users could be interested in a new product or campaign, how urgent a complaint is, or if a profile in an online forum might be a fake profile.
The Author Profiling Shared Task of the PAN shared task aims to answer these
question by extracting information about authors based on their linguistic style of
writing [
We have implemented a solution that is based on a bidirectional recurrent neural network (bi-RNN) using gated recurrent units (GRUs) in combination with an attention mechanism.
The paper is structured as follows. In Section 3, we give a short overview of related work. Then, in Section 4, we describe our model, and Section 5 compares the different attempts and their results on test data. Conclusions are drawn in the last section.
PAN
PAN is a series of different digital text forensics tasks. It organizes shared task
evaluations. Shared Tasks are computer science events of a specific problem of interest. This
paper is the result of our participation at the Author Profiling Shared Task of 2017.
Author Profiling includes gender and language variety predictions of an author of a given
Twitter document. To solve this problems, training and test datasets are available [
For each of the language varieties, there are 600 Twitter profiles. In each language there are the same number of male and female profiles. The dataset includes exactly 100 tweets for each author.
Language Variety. Language Variety is defined as a specific variation of an author’s
native language. For instance, one has to identify whether an English author has a
language variation from Australia, Canada, Great Britain, Ireland, New Zealand or the
United States.
TIRA. TIRA is an evaluation-as-a-service platform [
Evaluation. The performance measure of the submissions at PAN 2017 is done with accuracy. The individual accuracy for gender and variety identification was calculated for each language as follows:
correct predicted accuracy : (1)
total
The joint accuracy is calculated when both gender and variety are properly predicted together. The final ranking is calculated with the averaged accuracy over all four languages.
In this chapter we provide an overview of the most relevant works for the Author Profiling Task with neural networks.
Neural Networks. Neural networks have achieved great results in natural language
processing in the past few years. In many tasks like machine translation [
RNNs and CNNs. The recent success of RNNs are achieved through long short-term
memory networks (LSTM) and gated recurrent unit networks (GRU) [
In this chapter we describe the technical solution. Main focus is on the system architecture of the neural networks. 4.1
Every single tweet was preprocessed by converting them to lower-case. We replaced
URLs and usernames with a standardized token. We converted hashtags to regular
words and used the TweetTokenizer from NLTK [
Each token in a tweet is represented by pretrained word embeddings [
For Portuguese and Arabic we used pretrained embeddings from [
In this section we describe our model, which consists of a bi-RNN with GRUs followed
by an attention mechanism.
Embedding Layer. The embedding layer is used to map the token-IDs with their
vector representation. The token-ID is used to lookup the word-vector in the embeddings.
Those vectors get concatenated and are passed to the next layer. This results in an
output matrix S P Rd n, where d stands for the dimension of the word vector and n for the
size of the input. To determine n, we took the tweet with the biggest amount of tokens
from our training dataset and rounded the number up to the next 10. This resulted in a
maximum input size of n 60. Shorter inputs were padded with zeros to match that
size. To reduce the effect of unknown and padded words we used masking [
GRU Layer. This layer consists of two GRUs with u number of units. We used a GRU for each direction, which resulted two matrices RF P Ru n and RB P Ru n. Finally both matrices were concatenated and resulted a matrix R P R2u n. For our model we used u 50. 1 http://wikipedia.org Attention Layer. This layer is used to weight the most important parts of the GRU encoded input and deliver a more simplified matrix of the input. The output-matrix R of the previous layer, the weight-matrix Wa P R2u 2u and the bias b P R2u is used to calculate a hidden state ht: The hidden state ht and the weight-vector Wu P R2u used to calculate the final attention a for each word by ht a tanhpWaR
bq:
softmaxphtWuq:
(2)
(3)
The attention-vector a is then multiplied with R and the result summed together. This
results a summarized representation of the sentence as a vector sa P R2u.
Softmax Layer. As the final layer we used a fully connected layer with softmax as
the activation function. The number of output nodes were depending on the number of
classification possibilities. For gender prediction were 2 nodes required, for language
variety predictions were between 2 and 7 nodes required, depending on the language.
Dropout. Dropout drops individual nodes during training with a probability of p and
is therefore used to reduce overfitting [
Optimization. Our model is trained using the AdaDelta optimizer [
Author Prediction. Our model is trained to classify single tweets. To get the classification of an author, his tweets are classified separately. The outputs of our model, which is the output of the softmax layer, is then summed together and the class with the highest value is the final prediction. For example, if we want to predict the gender of an user u who has three tweets t1; t2; t3, we first classify the tweets separately. This could result following predictions: t1 r 0:4; 0:6s; t2 r 0:3; 0:7s; t3 r 1:0; 0:0s. The first number of each output indicates the probability that the tweet is written by a female and the second number indicates the probability that the author is a male. The outputs of the tweets t1; t2; t3 are summed together and results r1:7; 1:3s. In this example, user u would be predicted as a female. 4.4
To train our models for submission we used 90% of the training data and the remaining 10% were used as validation set. The validation set was used to select a model checkpoint during training. For more details in model checkpoints, see Section 5.1. 5
We distinguish between the evaluation during development, and the benchmark measured on actual test data on TIRA. The results during the development phase were achieved on the provided training corpus with cross validation.
tp f n
The harmonic mean of this two scores is called F1 score. The F1 score is calculated as follows:
F 1 2 precision recall precision recall : (4) (5) (6) tp f p
Recall is the ratio between correct classified data (tp) to the number of total data in the corresponding class (tp f n): precision recall
tp tp : : Cross Validation. Our models were trained with 10-fold cross validation. We used cross validation to calculate a representative score for the model. The data in each fold was used as follows: 80% training data, 10% validation data and 10% test data. The evaluation on the test data does not influence the training and is only used to evaluate the model. We used a validation set in combination with model checkpoints to prevent overfitting. Model checkpoints will be explained in the following section. F1 Score. During the training phase we used F1 score to find the best model. The F1 score considers both precision and recall to compute the score. We used the F1 score, because it penalizes one-sided predictions of a model. The abbreviations tp, f p, f n indicate in the following calculations true positives, false positives and false negatives. Precision is the ratio between correct predicted (tp) to all classified data of this class (tp f p): 5.1
The accuracy and F1 score of the model were measured during training. The scores were evaluated on a validation and a test dataset. If the model achieved a higher F1 score on the validation data than a previous one, the model (and its weights) was saved. An example of the measured scores is shown in Figure 2.
The goal is to select the best weights for a model during the training phase. Figure 2 shows that our model performs very similar on validation and test data. That means by choosing the best weights on the validation set, the chances are high that the model performs equally on the test set. This makes our model very stable and predictable. 5.2
While working with attention mechanism we developed a tool to represent how the different words in a tweet are weighted. This tool helped us to understand which words are more important for our model. An example on language variety is shown in Figure 3, where multiple tweets of British and American authors are compared.
In Figure 3 the attention of the words are highlighted. As we can see some typical
American English and British English words are marked. For example, in the first tweet
is the word "color" and in the third tweet "Walmart" marked as very important, which
are common words in American English. In the second and fourth tweet are the words
"bloody" and "cheeky" marked as significant for British English, which are common
words in British English.
During our preparation for the PAN shared task several models were tested and
compared. Our baseline was a CNN model [
The experiments have shown that the bi-GRU+Attention model has the best performance on both classification tasks (gender, variety). The measured scores of both models are shown in Table 2 and Table 3. We trained two distinct models for each language: one for gender and one for variety. These models were uploaded to the virtual machine and were evaluated on the actual test dataset. In Table 4 the results obtained on the PAN 2017 Author Profiling test dataset are shown.
The highest score on gender prediction was achieved in English. Portuguese gender prediction follows with 0.075% less accuracy. The gender predictions in Spanish and Arabic are lower than the others. We assume that this issue is related to the worse vocabulary usage: For both languages Spanish and Arabic, the vocabulary coverage is below 80%, in contrast to around 90% coverage of the vocabularies in English and Portuguese.
In general, good scores are achieved for variety prediction. Outstanding is the variety accuracy of 91,43% for the Spanish language, which consists of seven language variations. The score dropped only in English and Arabic below 80%. The lowest score 76,88% is achieved for variety prediction on Arabic, due to low vocabulary coverage. The exact vocabulary coverage of the used embeddings is shown in Table 5.
The results in Table 5 seems to imply that the accuracy for gender prediction correlates with vocabulary coverage. 6
In this paper, we presented deep learning models to predict gender and language variety of Twitter profiles. We described a bidirectional RNN with GRU and an attention mechanism. We compared the average accuracy of our models over all languages with a previously developed CNN model. The RNN exceeds the CNN in gender prediction by 1,45% and in variety prediction by 2,69% on average over four languages on PAN 2017 training data.
For future work, we would like to see if a combination of several high-quality
solutions for Author Profiling with a random forest could even outperform each of the
subsystems. This has been done successfully for sentiment analysis [