Author profiling aims at identifying the authors' traits on the basis of their sociolect aspect, that is, how language is shared by them. This work describes the system submitted by Symanto Research for the PAN 2017 Author Profiling Shared Task. The current edition is focused on language variety and gender identification on Twitter. We address these tasks by exploiting the morphology and semantics of the words. For that purpose, we generate embeddings of the authors' text based on subword character n-grams. These representations are classified using deep averaging networks. Experimental results show competitive performance in the evaluated author profiling tasks.
Author profiling aims at identifying the authors’ traits on the basis of their sociolect aspect, that is, how language is shared by them. It is used to determine language variety, gender, age, and personality type, among others. This task is specially attractive to industry representatives and particularly helpful for author opinion segmentation in social media. For instance, identifying the geographical distribution and gender of opinion authors may help to improve marketing campaigns. The task is also important for digital text forensics. Given a threat, knowing the possible author traits may help to its identification.
The Uncovering Plagiarism, Authorship, and Social Software Misuse1 (PAN) evaluation lab at the Conference and Labs of the Evaluation Forum2 (CLEF) promotes research and innovation in digital text forensics. Its Author Profiling Shared Task set the objective of classifying authors’ traits in several subtasks. These include the identification of age, cross-genre age, personality traits, and gender in social media. The current edition3 focuses on language variety and gender identification on Twitter.
Both morphological [
The rest of the work is structured as follows: in Section 2 we provide an overview of the state of the art in author profiling. In Section 3 we describe the system we employed for the PAN 2017 Author Profiling Shared Task. Next, in Section 4 we conduct our evaluation and discussion of the results. Finally, we draw our conclusions in Section 5. 2
Authorship attribution [
The Native Language Identification Shared Task [
This year, the PAN author profiling track is focused on the tasks of language variety
and gender identification. Regarding the latter, most of the recent work on gender
identification originated in the PAN evaluation lab. The system winner of the 2013-2015
editions is based on a representation for documents which captures discriminative and
subprofile-specific information [
The language variety identification task has attracted much interest in the last few
years. Character n-grams and other features have been employed to identify varieties
of Portuguese in news texts [
Unlike the majority of author profiling researchers, which employ stylistic and lexical features, our approach is based on character n-gram word embeddings, with exploit the morphology and semantics of words. This choice has also been driven by our motivation to experiment with a pipeline that could be replicated fairly simply by researchers who want to compare results and practitioners in need of a simple, yet accurate, pipeline to perform author profiling. 3
In this section we describe the system we designed for language variety and gender identification on Twitter. First, in Section 3.1 we describe our data preprocessing. Next, in Section 3.2 the embedding representations are described. Finally, in Section 3.3 we detail our classifier. 3.1
We preprocess each text with tokenization, word lowercase, and removing URLs. We use the Tweet NLP4 tokenizer, which is specific for English tweets. We slightly modified its regular expressions to consider Arabic, Portuguese, and Spanish punctuation, e.g. ’¿’ and ’¡’ were included for Spanish. 3.2
In recent years, word embeddings replaced the bag-of-words (BOW) representation as
the standard for text feature extraction.5 These representations are low d-dimensional
real-valued vectors which capture semantic and syntactic aspects of text. The
continuous skip-gram model [
We should note the importance of morphology in author profiling. For instance, the derivation of words is a discriminant feature in English language variety identification, e.g. regularized vs. regularised. As an additional example, the morphological refraction is indicative of gender in Latin languages, e.g. profesor vs. profesora in Spanish (male and female professor word translation, respectively).
In this work we use a recent variant of the continuous skip-gram model [
When it comes to learning these embeddings, the main difference of this subword model is in the scoring function used to estimate the probability of observing a context word wc given a target word wt. The original model used the scalar product of the word vectors as scoring: s(wt; wc) = uTwt vwc , where uwt and vwc are vectors in Rd. The subword model uses instead a scoring function which represents the target word as the sum of its character n-gram vectors: 4 http://www.cs.cmu.edu/~ark/TweetNLP/ 5 We note the increasing number of papers published at the ACL conference with "word embeddings" or "distributed representations" as part of the title: 0 (2013), 3 (2014), 15 (2015), and 29 (2016).
s(w; c) = X zgT vc; g2Gw (1) being Gwf1; :::; Gg the set of n-grams of the word w, and zg and vc vectors in Rd. Key of the model’s design is the use of a hashing function to map n-grams to integers that represent the vector index. This makes the model memory efficient and provides with an additional feature: it does not produce out-of-vocabulary words. The embedding of an unknown word is created by extracting its n-grams and doing the average of the vectors with the indexes returned by the hash function. For more details about the model please refer to its original work.
We generate a word embedding inventory for the training partition (see Section 4.1) of each language using the FastText library.6 We use 300-dimensional vectors, context windows of size 10, 20 negative words for each sample, 15 epochs, and 2M hashed character n-gram vectors. We extract n-grams with length in [3; :::; 6]. We post-process and enrich the embeddings with a proprietary model c Symanto Research. 3.3
A standard method to obtain vector representations of text consists on computing the
average of the word embeddings [
In this work we classify using Deep Averaging Networks (DAN) [
Our hidden layers have size equal to the embedding one and use the rectified linear
units (ReLU) [
In this section we evaluate our approach in the PAN 2017 Author Profiling Shared Task. 6 https://github.com/facebookresearch/fastText Dataset The objective of the PAN 2017 author profiling shared task is to identify the language variety and gender of Twitter users. Its corpus contains four languages and nineteen language varieties: – Arabic (Egypt, Gulf, Levantine, and Maghrebi). – English (Australia, Canada, Great Britain, Ireland, New Zealand, and United States). – Portuguese (Brazil and Portugal). – Spanish (Argentina, Chile, Colombia, Mexico, Peru, Spain, and Venezuela).
Next, we mention some key remarks about the dataset. The language of the user is known, so the dataset is composed by four partitions. In Table 1 we show the statistics. The labels are balanced at language variety and gender level. Finally, each Twitter user is represented by a set of approximately 100 tweets.7
In this work, we concatenate the user tweets to have an unique instance. We explored
other alternatives, as the independent classification of the tweets with a subsequent sum
of the class probabilities [
Methodology We compare our results with those obtained by the random baseline, a BOW model classified with random forest, a model based on continuous skip-gram embedding averages classified with logistic regression, and a model based on the subword 7 Each tweet is composed by up to 140 characters. Training users 2,400 3,600 Test users 800 1,200 Language varieties 4 6 embedding (see Section 3.2) averages classified with logistic regression. In the rest of the evaluation we refer to these models as Random, BOW, skip-gram emb., and subword emb., respectively. The prototype of our model (henceforth simply referred to as DAN) was designed using 10-fold cross-validation over the training sets. The parameter selection uses the same setting. The official measure of the competition is the accuracy. The ranking of the shared task participants is estimated as follows: i) for each language, the PAN organizers calculate individual accuracies for gender and variety identification; ii) they calculate the accuracy when both variety and gender are properly predicted together; and iii) the final ranking is obtained by averaging those accuracy values obtained per language. 4.2
We noticed during our experimentation phase that the performance of DAN is very sensitive to the number of hidden layers, which differ in function of the task and dataset. In Figure 2 we show the accuracy depending on the number of hidden layers, task, and language. As you can see, the two tasks benefit from adding layers after composition/averaging one. The best performance for language variety identification is achieved using two layers. In contrast, the optimal number of hidden layers for gender identification differs depending on the language. We use the best parameters determined in this section for the rest of the evaluation. 4.3
In this section we compare and discuss the results of our system. In Table 2 we show the development experiments and the comparison with the baseline models (see Section 4.1) using 10-fold cross-validation over the training set. As we can see, the three embedding-based models outperform BOW, the only purely lexical approach. The continuous skip-gram embedding averages classified with logistic regression obtain better results than the subword embedding averages in tasks such as language variety identification in Arabic or gender in Portuguese. However, the latter model offers in average higher results than the skip-gram one. Finally, DAN, using the same subword embeddings, obtains the highest results and proves that deep averaging networks are useful in author profiling to magnify the most discriminant values contained in an embedding average.
In Table 3 we show the results using the official test set of the shared task. This table also includes the joint accuracy, which is employed by organizers to determine the best 95 % 90 % system, i.e., when both variety and gender are properly predicted together. As we can see, DAN’s results are in line with those obtained using the 10-fold cross-validation setting. We also observe how the joint accuracy falls compared to the isolated language variety and gender results. This manifests the difficulty of this joint classification task, which continues being an open problem.
Our final comments are to analyse the difference in difficulty of this shared task depending on the task and language. Identifying gender is clearly more difficult than language variety. Despite the first task has only two possible labels, gender differences are generally more subtle and require more context and topic understanding. In contrast, the language variety peculiarities are both differentiable using lexical and semantic aspects of text. These lexical and semantic aspects are also the cause of the differences in function of the language. English and Arabic varieties are more similar at lexical level than Portuguese or Spanish ones. However, the low number of Portuguese varieties employed in this work affects too. Finally, considering the high number of Spanish varieties and its high results, we also consider that some languages have tweets with topics more indicative of the variety, e.g. topics about politics or events. 5
In this work we presented the system designed by Symanto Research for the PAN 2017 author profiling shared task. The pipeline we present in this paper is easily replicable and yields a good performance while promising to be robust and flexible in the presence of noisy data.
We described an approach based on subword character n-gram embeddings and deep averaging networks. We explained the rationale behind using these components in author profiling. We compared our approach with several well-known baseline models.
Random 25.0
variety Skip-gram emb. 73.0
Subword emb. 70.7
DAN 80.6 Gender
Random 50.0 BOW 66.4 Skip-gram emb. 71.2 Subword emb. 73.7 DAN 74.5
Experimental results in the tasks of native language and gender identification show the superiority of our approach and demonstrate that it is a competitive alternative.
Future work will investigate further how to employ semantic representations and deep learning techniques in the task of author profiling.