We describe our participation in the PAN 2019 shared task on author profiling, determine whether a tweet's author is a bot or a human, and in case of human, identify author's gender for English and Spanish datasets. In this paper, we investigate the complementarities of both stylometry methods and contentbased methods, putting forward various techniques for building flexible features. Acting as a complement to these methods, we investigate an ensemble learning method paves the way to improve the performance of AP tasks. Experimental results demonstrate that the ensemble method by the combination of the stylometry methods and content-based methods can more accurately capture the author profiles than traditional methods. Our proposed model obtained 0.9333 and 0.8352 of accuracy in the bot and gender identification tasks for English test dataset respectively.
The Author profiling (AP) deals with the classification of shared content in order to
predict general or demographic attributes of authors such as gender, age, personality,
native language, or political orientation, among others [
Broadly speaking, the approaches that tackle AP view the task as a multi-class or
single-label classification problem, when the set of the class label is known a priori [
In order to predict bot and gender, we propose the complementarities of both stylometry methods and content-based methods, putting forward various techniques for building flexible features (basic count features, psycholinguistic features, TF-IDF, Doc2vec). Acting as a complement to these features, we also investigate an ensemble learning method combining classification methods based on various features and BERT model paves the way to improve the performance of AP tasks. 2
Approaches for predicting an AP can be broadly categorized into two types of methods:
(1) stylometry methods which aim to capture an author’s writing style using different
statistical features (e.g., functional words, POS, punctuation marks, and emoticons), (2)
content-based methods that intend to identify an author’s profiles based on the content
of the text (e.g., bag of words, words n-gram, term vectors, TF-IDF n-grams, slang
words, emotional words), and topics discussed in the text (e.g., topic models such as
LDA, PLSA ) [
Every author’s writing style can be used to identify an author’s attributes. In
previous studies, style based features were used to predict the author’s attributes, age, and
gender [
Content-based methods employ specific words or special content which are used
more frequently in that domain than in other domains [
Recently, some works have used deep learning models and learning method of
representations for AP [
The preprocessing of the text data is an essential step as it makes the raw text ready for applying machine learning algorithms to it. The objective of this step is to clean noise those are less relevant to detect the AP on the texts.
We, at first, aggregate tweet posts published by an individual user into one document before training to alleviate the shortcomings of short texts. In order to utilize most of the information in text, we perform some transforming tasks of the short texts (i.e., XML parsing, contradictions unfolding, text tokenizing, stemming, lemmatization, and removing stopwords). Also, for utilization and word-level representation on most of the text information, we perform spell correction for informal words using SymSpell library1, word segmentation for splitting hashtags using WordSegment library2, and annotation (surround or replace with special tags such as <money>, <number>, <date>, <phone>, or <user>) as a text postprocessing task (see Figure 3). Previous works on AP tasks explore lexical, syntactic, and structural features. Lexical features are used to measure the habit of using characters and words in the text. The commonly used features in this kind consist of the number of characters, word, a frequency of each type of characters, etc. Syntactic features include the use of punctuations, part-of-speech (POS) tags, and functional words. Structural features represent how the author organizes their documents or other special structures such as greetings or signatures.
As shown Table 1, we construct a basic count feature set including punctuation characters (e.g., question marks, and exclamation marks) and other features (e.g., average syllable per word, functional word count, special character count, capital ratio, etc.).
The relationship between personality traits and the use of language has been widely
studied by the psycholinguist Pennebaker. [
LIWC has two types of categories; the first kind captures the writing style of the author like the POS frequency or the length of the used words (i.e., summary language variables, linguistic dimensions, other grammar). The second category (i.e., psychological processes) captures content information by counting the frequency of words related to some thematic categories such as affective processes, social processes, personal concerns, etc. Regarding the use of this tool, we focused on the content information. 3.4
TF-IDF We adapted the TF-IDF (Term Frequency-Inverse Document Frequency) method to judge the topics of each document by the words it contains. In the TF-IDF words are given weight – TF-IDF measures relevance, not frequency. Particularly, word counts are replaced with TF-IDF scores across the corpus. TF-IDF, at first, measures the number of times that words appear in a given document (i.e., term frequency). However, since words such as and or the appear frequently in all documents, those must be systematically discounted (i.e., inverse-document frequency). The more documents a word appears in, the less valuable that word is as a signal to differentiate any given document. That is intended to leave only the frequent and distinctive words as markers. TF-IDF relevance of each word is a normalized data format as the following formula:
where tfi;j is the number of occurrences of i in j ; dfi is the number of documenting containts i; N is the total number of documents.
We extract unigrams, bigrams, and trigrams derived from the bag of words representation of each Twitter posts. To account for occasional differences in content length between train dataset and test dataset, these features are encoded as TF-IDF values. 3.5
Doc2vec
We use the Doc2vec method [
In this work, different representations of the texts were used as input data types for
the Doc2vec method in order to evaluate the quality of different distributed
representation outputs. In particular we represented the texts in terms of word unigram, bigram
and trigram. For the implementation of the Doc2vec model, a freely available package
of the Doc2vec included in the Gensim module used. The Doc2vec method offers two
possible approaches (i.e., PV-DM, PV-DBOW) to build the model. As shown in
Table 2, our experimental results report better performance when both representations are
concatenated, thus our model’s final document vector is composed of the
concatenation of the representations obtained by the DM (Distributed Memory) and the DBOW
(Distributed Bag of Words) models.
BERT (Bidirectional Encoder Representations from Transformers) model’s key
innovation is applying the bidirectional training of Transformer, a popular attention model,
to language modelling [
When training language models, there is a challenge of defining a prediction goal. To overcome this challenge, BERT uses two training strategies: Masked LM (MLM) and the next sentence prediction (NSP). Their results show that BERT outperforms the state-of-the-art results in a wide variety of NLP tasks, including QA (SQuAD), Natural Language Inference (MNLI), and others. In this task, we used pre-trained BERT model (i.e., bert_uncased_L-24_H-1024_A-16) in TensorFlow Hub website 1. 4
The task of identifying bot and gender from the text is cast as a classification task. For
these tasks, we have performed binary classification task, i.e., the goal is to distinguish
between two classes: (1) bot and human (2) male and female in case of human class.
We have used 10-fold cross validation for experiments. We employed Gradient Boosted
Decision Tree (GBDT) classifier and BERT model to train and test our proposed
system as classifier models. Our results on PAN2019 subtasks are summarized in Table
3. Regarding the submissions to the task through the TIRA [
then, the trained models were deployed to TIRA to classify the unseen test dataset. We tuned our single models on the development datasets and submitted the final results using ensemble models. The ensemble models were built by averaging the outputs of two models, which are feature stacking model and deep learning model. Our submission results achieve an accuracy of 0.9333 for bot detection task and 0.8352 for gender identification task on English test dataset.
In the gender identification tasks, the content-based approaches show better results, especially the Doc2vec model shows significantly better results than other feature sets. The resulting accuracy scores by other features (i.e., Basic count features, psycholinguistic features, TF-IDF) indicate that these approaches address the problem to some extent but requires more distinctive features to further improve the accuracy. Interestingly, the concatenation of both stylometry and content-based approach proves highly effective, achieving the best results overall. However, Doc2vec method proves to be highly competitive as a single-handed feature set.
In terms of deep learning approach, even if BERT model managed to learn important features from basically any data structure without having to manually derive features, the performance in our experiment is not competitive. Our experimental results using the pre-trained model did not outperform other model’s accuracy and seem to require more fine-tuning. 5
This paper summarized our participation in PAN2019 shared task on AP, where we aimed to deal with the challenge of AP. We tried to approach the problem from the perspective of stylometry and content-based methods, as well as contextualized word embeddings from BERT model. In terms of training method, we adopted an ensemble approach and carried on experiments on PAN collections. Experimental results demonstrate that the ensemble method by the combination of the style-based and contentbased methods can more accurately capture the author profiles than traditional methods.
There is still room to improve our work in the future. In our system, we compared the BERT model showing state-of-the-art results in various NLP tasks without sophisticated fine-tunising. We will leave more fine-tuning work on our datasets as future work.