Introduction

Researches to automatically extract author profile traits from social media have been done to empower activities such as advertisement, forensic, marketing, personalization, and security. PAN tasks have focused on traits like gender, age, and personality type in the past series. This year's author profiling task was to identify a gender and a language variety of a Twitter user [15]. In the gender identification, a task participant is required to determine whether a user is male or female from tweets. Similar gender identifications have been done in past PAN series with different native languages and domains. In the language variation identification, a task participant has to decide a language variety within a given native language from tweets. The study of language varieties has been done in VarDial shared tasks [17] targeting journalistic texts, but is new in PAN series targeting Twitter texts.

Statistical models using a machine learning method like support vector machine have shown effectiveness to identify profile traits in past PAN series. Various features were introduced to these models including word n-grams [6,12,3], character ngrams [6,12,3], part-of-speech tags [6,3], styles [6,12,3], and second order attributes [6]. We decided to tackle the identifications of gender and language variety using neural networks. Neural networks have shown effectiveness to capture complex representations combing simpler representations [9]. We aim to obtain complex representations that were expressed as independent features in the past studies using neural networks. Neural networks such as multilayer perceptron and restricted Boltzmann machine have been used in PAN 2016 [16] to obtain word embeddings [2] and as a classifier. Our models combine word information and character information with complex neural networks consisting of a recurrent neural network layer, a convolutional neural network layer, and an attention mechanism [1] layer to classify a profile trait.

In the following section of this paper, we first describe our neural network models in Section 2. Data used in the models are explained in Section 3 following Section 4 with the details of an experiment to confirm the performances of the models. Finally, Section 5 concludes the paper with some future directions.

Models

We propose two models that consist of multiple layers to classify a profile trait with neural networks. The architectures of the two models share most of their layers but differ in the fusion strategies of word information and character information. The first model NeuralNet-FusionTweet (NN-FT) combines the two kinds of information with a tweet-level fusion. The second model NeuralNet-FusionUser (NN-FU) performs a fusion process in user-level.

Model NN-FT

Figure 1 shows the architecture of NN-FT. For each user, the model accepts the words and the characters of user tweets. Note that the words and the characters are just dif- ferent representations of same tweet texts. The words and the characters are embedded with embedding layers and are processed with a recurrent neural network (RNN) layer, convolutional neural network (CNN) layers, attention mechanism [1] layers, a max-pooling layer, and fully-connected (FC) layers. As an implementation of RNN, we used Gated Recurrent Unit (GRU) [7] with a bi-directional setting.

x 1 x T … input h 1 bi-directional recurrent states … g 1 g 2 g T RNN features … x 2 u 1 context vectors + … α 1 g 1 α 2 g 2 α T g T Attention features m u 2 u T

Attention Layer

RNN Layerh 1 h 2 h 2 h T h T

word processes Figure 2 illustrates the overview of word processes by RNN W and Attention W . The input words are embedded to k w dimension word embeddings with embedding matrix E w to obtain x with x t ∈ R kw . x are then processed in RNN W with the following transition functions:

z t = σ (W z x t + U z h t−1 + b z ) (1) r t = σ (W r x t + U r h t−1 + b r ) (2) ht = tanh (W h x t + U h (r t ⊙ h t−1 ) + b h )(3)h t = (1 − z t ) ⊙ h t−1 + z t ⊙ ht(4)

where Attention W computes a tweet representation m as a weighted sum of g t with weight α t :

z t is an update gate, r t is a reset gate, ht is a candidate state, h t is a state, W z , W r , W h , U z , U r , U h are weight matrices, b z , b r , b h arem = ∑ t α t g t (5)α t = exp ( v T α u t ) ∑ t exp (v T α u t )(6)u t = tanh (W α g t + b α )(7)

where v α is a weight vector, W α is a weight matrix, and b α a bias vector. u t is an attention context vector calculated from g t with a single FC layer (Eq. 7). u t is normalized with softmax to obtain α t as a probability (Eq. 6).

character processes Figure 3 illustrates the overview of character processes by CNN C and MaxPooling C . The input characters are embedded to k c dimension character embeddings with character embedding matrix E c to obtain s with s i ∈ R kc . s is then passed to CNN C to obtain c with:

c i = f (W c s i:i+h−1 + b c )(8)

where f (•) is a non-linear function, W c is a weight matrix, h a convolution window size, and b c a bias vector. We used rectified linear unit for f (•). c is further processed with max-over time process [8]

Model NN-FU

Figure 4 shows the architecture of NN-FU. Many layers in NN-FU exist in NN-FT. Layers that are not apparent in NN-FT are Attention FUW , Attention FUC , FC FU1 , and FC FU2 . Attention FUW merges tweet representations obtained from word information. Similarly, Attention FUC merges tweet representations obtained from character information. The outputs of these attention processes are concatenated and is further processed with FC FU1 and FC FU2 . The attention processes in NN-FU are different from the attention processes in NN-FT, where word information and character information are concatenated prior to Attention FT . In NN-FU, word information and character information are concatenated after the attention processes with user-level representations. The other non-apparent layers FC FU1 and FC FU2 perform similarly as FC FT1 and FC FT2 in NN-FT to process a word+character user representation.

Data

The weights in the proposed models require some data to be trained. We used two datasets to train the proposed models with two different objectives.

PAN@CLEF 2017 Author Profiling Training Corpus

The first dataset we used to train the proposed models is the official PAN@CLEF 2017 Author Profiling Training Corpus. The dataset consists of 11, 400 Twitter users in four languages with the gold labels of gender and language variety. The languages, gender labels, and language variety included in this dataset is summarized in Table 1 This dataset is used to train the models to minimize an empirical loss between predictions and gold labels.

We divided this dataset into train 8 , dev 1 , and test 1 with a stratified sampling by ratio of 8:1:1. These subsets were made so that we can empirically decided some parameters of the models. We will describe the detail of parameter selection in Section 4.2.

Streaming Tweets

The second dataset we used to train the proposed models is tweets collected by Twitter Streaming APIs1 . We collected these tweets to pre-train the word embedding matrix E w of the models. Neural network models are known to perform better when word embeddings are pre-trained by a large-scale dataset [8]. The following steps describe the detail of the collection process:

1. Tweets with lang metadata of en, es, pt, and ar were collected via Twitter Streaming APIs during the period of March-May 2017. 2. Retweets are removed from the collected tweets.

3. Tweets posted by bots2 are deleted from the collected tweets.

Table 2 shows the number of resulting tweets. We will describe the detail of word embedding pre-training in Section 4.1.

Experiment

Model Configurations

Text Processor We applied a unicode normalization, a Twitter user name normalization, and a URL normalization for text pre-processing. Pre-processed texts were tokenized with the two kinds of tokenizers. Twokenizer [13] is used for English and NLTK [4] WordPunctTokenizer is used for other languages. Words are converted to lower case forms to ignore capitalization. Note that the lower case conversion is not performed for character inputs.

Initialization of Embeddings

We pre-trained word embeddings using streaming tweets of Section 3.2 by fastText [5] with the skip-gram algorithm. The pre-training parameters are dimension=100, learning rate=0.025, window size=5, negative sample size=5, and epoch=5. For character embeddings, we randomly initialized them with a uniform distribution.

Convolution Filter Sizes, Layer Unit Sizes, and Word Embedding Sizes Table 3 summarizes the sizes of various parameters included in the proposed models. In CNN C , two values are listed since we used the multiple filters approach [10]. Optimization Strategy We used cross-entropy loss as an objective function of the models. l 2 regularization was applied to the RNN layers, the attention context vectors, the CNN layers, and the FC layers of the models to avoid overfitting. The objective function was minimized through stochastic gradient descent over shuffled mini-batches with Adam [11]. For the initial learning rate of Adam, we set it to 1e −3 .

Language

Parameter Selection The models have regularization parameter α which is sensitive to a dataset. We selected optimal values for α:

α ∈ { 1e −3 , 5e −4 , 1e −4 , 5e −5 , 1e −5 , 5e −6 , 1e −6 , 5e −7 , 1e −7 }

in terms of accuracy with a grid search using dev 1 described in Section 3.1.

In-house Experiment

We evaluated the proposed models using train 8 , dev 1 , and test 1 . All models are trained using a single NVIDIA Titan X gpu. Table 4 presents the results of gender identifications. In the gender identifications, NN-FU performed better than NN-FT with one exception in Spanish. Table 5 shows the results of language variety identifications. The language variety identifications showed different characteristics where NN-FT performing better in all languages compared to NN-FU.

Submission Run

We chose the best performing models and αs in the in-house experiment as models and parameters for a submission run. In the cases of multiple best performing αs, we chose αs that showed the best performances in test 1 . The submission run was done in a TIRA virtual machine [14] with cpus. Table 6 summarizes the performances of the models in the submission run. The models showed a similar trend as in the in-house experiment. They ranked 3rd in gender ranking, 6th in language variety ranking, and 4th in the global ranking. Portuguese 99.89 + 1e -3 , 5e -4 , 1e -4, 5e -5 , 1e -5 , 5e -6 , 1e -6, 5e -7 , 1e -7 99.47 + 1e -3 , 5e -4 , 1e -4, 5e -5 , 1e -6, 5e -7 , 1e 6. The performances of the proposed models in the submission run.

Language

Conclusion

As described in this paper, we proposed two models, NN-FT and NN-FU, for author profiling. The two models differ in the fusion strategies of word information and character information. The models marked joint accuracies of 64-86% in the gender identification and the language variety identification of four languages. They performed better in gender identification compared to language variety identification. The average accuracies from the top systems were -1.26% for gender and -2.05% for language variety. This result is not so surprising since neural network models had shown difficulties adapting to language variety identification in past VarDial shared tasks [17].

As future works of this study, we plan to analyze differences of internal states in NN-FT and NN-FU. The best performing models were different among profile traits and languages in the in-house experiment. We will like to unveil the causes of this differences to further improve our models.