Paper describes approach leveraging deep learning principles in bots and gender profiling task at CLEF 2019 conference. Our approach is using hierarchical network for classification of tweets sequences. We achieved 90% accuracy of type profiling for English and 86.9% for Spanish language and 77.6% and 77.2% respectively accuracy in gender profiling. Task Description 1 https://pan.webis.de/clef19/pan19-web/author-profiling.html
This paper describes our approach to bots and gender profiling task for PAN at
CLEF 2019 [
Bots are commonly used in bank or insurance sector as chat bots. These chat bots act as first level support for customers. They are able to help people with simple problems. Another positive example are weather forecasting bots or stock exchange information bots. They are effective way for sending information to multiple users (via Twitter, Facebook and Instagram for example).
However, another type of bots is used to spread misleading information, fake news for example. And we need to filter out this kind of information, because people tend to believe in such information as this information is spread across the whole internet and looks like valid fact. 1.1
Aim of this task1 is to determine if given Twitter feed is written by human or bot. In case given Twitter feed is written by human, our next task is determine if it is written by male or female. Also, this task is multilingual, it consists of two sub datasets – English and Spanish. Despite of language separation, creators of dataset do not guarantee language consistency for all tweets in feed.
Our performance is evaluated by average accuracy of each subtask (human vs bot and male vs female) of each language. 2
In terms of stylometry, authorship attribution is application of linguistic style to
written language, but also to music [
In context of this paper we will focus on linguistic stylometry due to natural language character of Twitter feeds.
Problem of duplicate accounts on internet discussion forums was discussed in [
Other paper [
Our method is based on our previous work, which achieved state of the art results
in source code authorship attribution [
We have done experiments also with TF-IDF based approaches in combination
with different classifiers. This approach was superior to our method. However, we
used it as a good baseline for experiments [
Training dataset consists of 4120 English Twitter feeds and 3000 Spanish Twitter feeds. Each feed consists of exactly 100 tweets. Maximum tweet length is 140 or 280 characters2. Samples are provided in XML files, one sample per file.
As stated above, data for this task consists of unprocessed twitter feeds. Our brief data analysis shown that majority of tweets contain relatively large number of emojis, quiet large number of typos, double, triple chars, punctuation or mixed language.
Twitter users are standardly utilizing Unicode set of emojis3. Working with special Unicode characters is not convenient and is easier working with their word description. In theory this should not be necessary, because we are using word embeddings. But our training corpus is relatively small, so this step is helping us to better train these embeddings - we are de facto extending tweets and got more tokens in dataset. You can see example of such a transformation in Table I. 2 https://developer.twitter.com/en/docs/basics/counting-characters.html 3 http://www.unicode.org/emoji/charts/full-emoji-list.html
Next step of our preprocessing pipeline is lemmatization. Lemmatization is process of extracting word lemma (word root). We can think of this as dimensionality reduction which will potentially easier generalization of our model (Table 2). Usually lemmatization is not needed when word embeddings are used, however our as stated, our corpus is small, so we are using all available method to make embeddings more stable and more domain specific.
Next step is tokenization and token encoding. We used standard Keras framework function for these two steps. Tokens were split by space characters and we filter out special characters such as braces, hash key, punctuation characters, etc. with combination of converting tokens to their lowercase representation. Tokens were encoded to integers based on their index in corpus dictionary.
Last step is zero-padding of inputs to fixed length (our model doesn’t support variable length input sequence). We empirically chose sequence length of 60 tokens (words) based on histogram in Figure 1. 3.2
Our classifier is based on our previous work [
Convolutional neural networks are often used in image processing, where they achieved state of the art results in image recognition. They are gaining popularity also in natural language processing, because they act as feature extractors in texts too.
Embeddings are heavily used throughout variety of text processing problems. They are effectively encoding words into vector representation. Our embedding layer is randomly initialized (not pretrained) and trained on the fly - it should learn more specific domain specific embedding vectors this way.
Recurrent networks, especially Long-Short Term Memory units are used in state of the art models for text classification, emotions detection or speech recognition. They are good at sequence learning – and tweets are word sequences.
We were struggling with overfitting of the network, which we solve by adding dropouts. Dropout is randomly dropping connections between layers and therefore helping generalization of the model.
As stated above, tweets are word (character) sequences. We can say, that feeds are tweet sequences, therefore it could be beneficial to work with them as they are sequences. That’s why we propose hierarchical network on top of tweets and dealing with them as with sequences, with all 100 tweets from sample “at once”.
In text below are stated specific parameters of proposed and implemented neural network:
Embedding: vector length 30 Convolutional I: kernel size 2, number of filters 16, ReLU activation Convolutional II kernel size 2, number of filters 16, ReLU activation Max pooling: pooling size 2 Bidirectional LSTM: 16 units Dense: 24 units Dense (Output): 2 units, SoftMax activation Dropout rate 0.5
Batch size: 8 Epochs: 100 Early stopping: patience 5, validation loss monitoring Loss: categorical crossentropy Adam optimizer: learning rate 0.001, beta1 0.9, beta2 0.99 This chapter describes our testing results and task testing dataset results. Our testing results are average of multiple runs (10) of different dataset splits.
For our testing purposes we used 50/25/25 split for training, validation and testing fractions of data respectively. We are using accuracy metric, because classes are perfectly balanced, which means every class has exactly same number of samples.
Table 3 demonstrates our results on testing split (25% of training dataset). Our results were quite encouraging, although were significantly worse than results on organizers testing dataset (Table 4).
Also, our testing score (accuracy) was calculated differently than on task testing dataset. We were training exclusively on tweets posted by humans, so our gender testing accuracy is just from human samples. Gender task testing dataset was calculated from all samples (human and bot), this is main cause of big accuracy difference.
Our approach achieved roughly 90% accuracy in type recognition (human or bot) and 77.5% accuracy in gender recognition (male or female) for English Twitter feeds for task testing dataset. Spanish samples results were slightly worse – 86.9% accuracy for type recognition and 72.5% for gender recognition (Table 4).
It is evident that results on task testing dataset are significantly worse than results on our testing split. This is probably caused by insufficient generalization of our model, we suspect collected “vocabulary” is not large enough. Specifically speaking, vocabulary of our embedding layer is not large enough, which results in a lot out of vocabulary words and therefore next layers in our model don’t have enough information to make reliable decision. Testing dataset wasn’t published to the date of paper submission, so we are unable to make proper analysis in context of task testing dataset. Our final task ranking is 21 from total of 55 contestants. Unfortunately, even two baseline methods (word and character n-grams outperformed our solution). Despite of final ranking, we must say that this our first appearance in such a competition was not total disaster – we rank in better half of solutions.
Unfortunately, because of time and computational constrains we were not able to realize and test all our ideas. Task results give us also some ideas, what we could done better.
First of all, tokenization step could be improved, for example there is lot of URLs in tweets, and we could get links from them and use information from sites such as topic or language of site. Additionally, we could use for example hypernyms in preprocessing to normalize texts.
Discussed above, our vocabulary was probably very limited (due to small training dataset). We could overcome this problem using pretrained English and Spanish embedding vectors or enrich dataset using Twitter real time API.
We used simple word level embedding layer, however other papers show, that using more sophisticated methods such as ELMo or character-based embedding have better results in topic modeling for example. Therefore, we can deduct usage of these methods could improve our results.
This work was partially supported by Human Information Behavior in the Digital Space, the Slovak Research and Development Agency under the contract No. APVV15-0508, by the Slovak Research and Development Agency under the contract No. APVV-17-0267 - Automated Recognition of Antisocial Behaviour in Online Communities and by data space based on machine learning, the Scientific Grant Agency of the Slovak Republic, grant No. VG 1/0725/19.