This paper describes neural network models that we prepared for the author profiling task of PAN@CLEF 2017. In previous PAN series, statistical models using a machine learning method with a variety of features have shown superior performances in author profiling tasks. We decided to tackle the author profiling task using neural networks. Neural networks have recently shown promising results in NLP tasks. Our models integrate word information and character information with multiple neural network layers. The proposed models have marked joint accuracies of 64-86% in the gender identification and the language variety identification of four languages.
Researches to automatically extract author profile traits from social media have been
done to empower activities such as advertisement, forensic, marketing, personalization,
and security. PAN tasks have focused on traits like gender, age, and personality type in
the past series. This year’s author profiling task was to identify a gender and a language
variety of a Twitter user [
Statistical models using a machine learning method like support vector machine
have shown effectiveness to identify profile traits in past PAN series. Various
features were introduced to these models including word n-grams[
Word Embedding
Character Embedding
words
characters
consisting of a recurrent neural network layer, a convolutional neural network layer, and
an attention mechanism[
In the following section of this paper, we first describe our neural network models in Section 2. Data used in the models are explained in Section 3 following Section 4 with the details of an experiment to confirm the performances of the models. Finally, Section 5 concludes the paper with some future directions. 2
We propose two models that consist of multiple layers to classify a profile trait with neural networks. The architectures of the two models share most of their layers but differ in the fusion strategies of word information and character information. The first model NeuralNet-FusionTweet (NN-FT) combines the two kinds of information with a tweet-level fusion. The second model NeuralNet-FusionUser (NN-FU) performs a fusion process in user-level. 2.1
input … uT … g2 h2 … h2 … gT hT hT u1
u2
g1
ferent representations of same tweet texts. The words and the characters are
embedded with embedding layers and are processed with a recurrent neural network (RNN)
layer, convolutional neural network (CNN) layers, attention mechanism[
(W z xt + U z ht 1 + bz ) rt = (W rxt + U rht 1 + br) ~ ht = tanh (W hxt + U h (rt ⊙ ht 1) + bh) ht = (1 zt) ⊙ ht 1 + zt ⊙ h~ t (1) (2) (3) (4) where zt is an update gate, rt is a reset gate, h~ t is a candidate state, ht is a state, W z ; W r; W h; U z ; U r; U h are weight matrices, bz ; br; bh are bias vectors, is a logistic sigmoid function, and ⊙ is an element-wise multiplication operator. The bidirectional GRU outputs !h and h are concatenated to form g where gt = ht∥ht and ! are passed to Attentionw. CNN Layer
Max features CNN features
c1 filter width h input o … max over time
cL-h+1 c2
… where v is a weight vector, W is a weight matrix, and b a bias vector. ut is an attention context vector calculated from gt with a single FC layer (Eq. 7). ut is normalized with softmax to obtain t as a probability (Eq. 6). character processes Figure 3 illustrates the overview of character processes by CNNC and MaxPoolingC. The input characters are embedded to kc dimension character embeddings with character embedding matrix Ec to obtain s with si 2 Rkc . s is then passed to CNNC to obtain c with:
ci = f (W csi:i+h 1 + bc)
where f ( ) is a non-linear function, W c is a weight matrix, h a convolution window
size, and bc a bias vector. We used rectified linear unit for f ( ). c is further processed
with max-over time process[
FCFU1 AttentionFUW AttentionW
RNNW
AttentionFUC MaxPoolingC
CNNC Word Embedding
Character Embedding words characters The weights in the proposed models require some data to be trained. We used two datasets to train the proposed models with two different objectives. Gender Labels
English The first dataset we used to train the proposed models is the official PAN@CLEF 2017 Author Profiling Training Corpus. The dataset consists of 11; 400 Twitter users in four languages with the gold labels of gender and language variety. The languages, gender labels, and language variety included in this dataset is summarized in Table 1 This dataset is used to train the models to minimize an empirical loss between predictions and gold labels.
We divided this dataset into train8, dev1, and test1 with a stratified sampling by ratio of 8:1:1. These subsets were made so that we can empirically decided some parameters of the models. We will describe the detail of parameter selection in Section 4.2. 3.2
The second dataset we used to train the proposed models is tweets collected by Twitter
Streaming APIs1. We collected these tweets to pre-train the word embedding matrix
Ew of the models. Neural network models are known to perform better when word
embeddings are pre-trained by a large-scale dataset[
APIs during the period of March–May 2017. 1 https://dev.twitter.com/streaming/overview
Parameter Size word embedding dimension 100 character embedding dimension 25
RNNW units 100 CNNC units 50
CNNWC units 300 CNNC filter sizes 3; 6 CNNWC filter size 1 AttentionW units 200 AttentionFT units 300 AttentionFUW units 200 AttentionFUC units 100
FCFT1 units 150 FCFU1 units 150 FCFT2 units #label
FCFU2 units #label 2. Retweets are removed from the collected tweets. 3. Tweets posted by bots2 are deleted from the collected tweets.
Text Processor We applied a unicode normalization, a Twitter user name
normalization, and a URL normalization for text pre-processing. Pre-processed texts were
tokenized with the two kinds of tokenizers. Twokenizer[
Initialization of Embeddings We pre-trained word embeddings using streaming tweets
of Section 3.2 by fastText[
Convolution Filter Sizes, Layer Unit Sizes, and Word Embedding Sizes Table 3
summarizes the sizes of various parameters included in the proposed models. In CNNC,
two values are listed since we used the multiple filters approach[
Spanish Portuguese 84.17
Arabic
Optimization Strategy We used cross-entropy loss as an objective function of the
models. l2 regularization was applied to the RNN layers, the attention context vectors,
the CNN layers, and the FC layers of the models to avoid overfitting. The objective
function was minimized through stochastic gradient descent over shuffled mini-batches
with Adam[
We evaluated the proposed models using train8, dev1, and test1. All models are trained using a single NVIDIA Titan X gpu. Table 4 presents the results of gender identifications. In the gender identifications, NN-FU performed better than NN-FT with one exception in Spanish. Table 5 shows the results of language variety identifications. The language variety identifications showed different characteristics where NN-FT performing better in all languages compared to NN-FU. 4.3
We chose the best performing models and s in the in-house experiment as models
and parameters for a submission run. In the cases of multiple best performing s, we
chose s that showed the best performances in test1. The submission run was done in
a TIRA virtual machine [
As future works of this study, we plan to analyze differences of internal states in NN-FT and NN-FU. The best performing models were different among profile traits and languages in the in-house experiment. We will like to unveil the causes of this differences to further improve our models. 16. Rangel Pardo, F., Rosso, P., Verhoeven, B., Daelemans, W., Potthast, M., Stein, B.: Overview of the 4th Author Profiling Task at PAN 2016: Cross-Genre Evaluations. In: Working Notes Papers of the CLEF 2016 Evaluation Labs (2016) 17. Zampieri, M., Malmasi, S., Ljubešic´, N., Nakov, P., Ali, A., Tiedemann, J., Scherrer, Y., Aepli, N.: Findings of the vardial evaluation campaign 2017. In: Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial). pp. 1–15 (2017)