In this paper, we present an approach for identifying Twitter bots based on their written tweets using a convolutional neural network. We experiment with various embedding methods (pretrained and trained on the training dataset) and convolutional neural network architectures and compare their performance. When evaluating our best performing approach on the actual test data set of the CLEF 2019 Bots Profiling Subtask (English language), we obtain an accuracy of 90.34%. We therefore see convolutional neural networks as a promising machine learning technique for Twitter bot detection.
With over 1 billion active users, Twitter is among the most frequently used social media
platforms. It has taken an important role as a medium disseminating information and
opinions. However, due to its potential to influence the reader’s opinion, bot nets are
considered to be a threat to society and democracy. For instance, it has been reported
that botnets have been identified in the context of public voting, such as the Brexit
voting [9] and the federal elections of the United States in 2016 [
In this paper, we consider the following task proposed as a shared task at CLEF
2019: “Given a Twitter feed, determine whether its author is a bot or a human.” [
This paper is structured as follows. In Section 2, we present related works on bot identification. In Section 3, we describe our approach for Twitter bot identification. In Section 4, we present the evaluation setup, the evaluation data set, and the evaluation results. We conclude in Section 5 with a summary and an outlook. 2
Given that millions of user accounts on social media platforms are bots, classifying user
accounts in social media has already been approached in various regards. In 2016,
Adewole et al. [
Non-content based approaches to bot identification. Non-content-based approaches use
patterns in the behaviour of the Twitter accounts. Chavosi et al. [
Content-based approaches to bot identification. Content-based approaches target the classification of Twitter accounts based on the account’s tweets. In [10], the authors use content-based features and pattern contrast features for bot detection. They obtain classification results over 0.90 of AUC with this approach.
Hybrid approaches to bot identification. In their paper [
Note that in this paper, we obtain an AUC score of 0.86 based solely on the content of the tweets. To the best of our knowledge, we are the first ones approaching such a task using a convolutional neural network model. 3
In this section, we present our approach for twitter bot identification. The source code of our implementation is available online at https://github.com/agon-qurdina/ author-profiling. Normally, the input for our model would be the set of tweets for a given user. But we decided to have the entire user’s feed of tweets as a single input to the model. Thus, we merged all of the author’s tweets together into a single block of text which we called “article.” Next, given a set of “articles” as input, we preprocessed them along the following steps:
Text Cleaning. New lines were replaced by spaces. We also expanded contractions and removed stop words, HTML tags, and special characters from the articles’ content.
Texts to Sequences. The articles’ content was tokenized and a word dictionary (size: 155,227 unique words) was generated.
Sequence Padding. We applied a post-padding with a fixed sequence length l. Due to the way we concatenated the tweets to a single “article,” we were left with a limited number of training samples. Thus, in order to keep as much information as possible, we set l = 11000. By choosing this high value for the sequence length, only a small percentage (284 from 2873) of the sequences were truncated. 3.2
Our basic architecture is shown in the Figure 1. We use a CNN architecture that is
based on Kim et al.’s approach for sentence classification [
Input layer. Considering the article’s words were embedded into d-dimensional vectors, the final matrix used as input to the model can be written as I = l d where l is the chosen sequence length (i.e., the length of the articles). Recall that l = 11000 in our setting.
First convolutional layer. The first transformation this embedded input goes through is a convolutional layer with f 1-dimensional filters of length k. Thus, the layer weights can be considered a matrix of shape W c 2 Rf k.
We used a filter size f of 64 and a filter length k of 4. In our context, having 1dimensional filters means that for each word in an article, its three adjacent words are considered as the context of the word. The output of the convolutional layer then is C = conv(I; W c) where conv is the convolutional operation applied to input I using the weights matrix W c. This operation includes applying the ReLu activation function to complete weights calculations. Also, a dropout function is used to prevent overfitting. We used a dropout rate of 0:5, which means 50% of the weights (randomly chosen) during each training epoch are set to 0.
First max pooling layer. The above output C is then considered the input to a 1dimensional Max Pooling layer. The purpose of the layer is to extract only the most important features of the convolution outputs. This is done by keeping only the max value from a pool size p. As we chose p = 4, the output of this layer can be written as M = max pool(C; p). This operation reduces the number of weights by four times.
pooling layer is the input of the second convolutional layer, and the whole process described above is applied to this input, to get a final output M2.
Fully connected layer. Given the two-dimensional matrix of weights from the last step, the next layer in the network is a fully connected one with a size of 256 hidden neurons. But in order for the convolutional output to serve as an input to this layer, its dimensionality needs to be reduced. This was done using the flatten method, which keeps all of the values but flatten them in a long vector. A ReLU activation function and a dropout layer with a rate of 0:5 were used here.
Output layer. The last layer is a fully connected layer with one neuron. The sigmoid activation function is used to provide a binary output. 3.3
1. The first variation uses 1-dimensional MaxPooling layers of size 2 after each of the Convolutional layers. That means the resulting number of features, after passing from the convolutional layer to the MaxPooling layer and then out of it, will be halved. Considering we start with a sequence length of 11,000, the fully-connected layer contains more than 11 million weights to train. We will call this variation Large Model in the remaining part of this paper. 2. The second variation uses 1-dimensional MaxPooling layers of size 4 after each of the Convolutional layers, which reduces the number of features generated by the Convolutional layers by a factor of 4. It does that by only keeping the max value from 4 adjacent features. If the same input size is used, this makes for a significant lower number of weights in the fully-connected layer (around 16,000) compared to the Large Model. 4
4.1
The actual CLEF 2019 Bots and Gender Profiling Task test data set is hidden and only used for official system submissions. Thus, we used the training and validation data set of the CLEF 2019 Bots and Gender Profiling Task data set for training and testing our models before the CLEF submissions. In Table 1, we outline noteworthy statistics about the used data set. Note that the data set provided by CLEF is balanced, i.e., the number of bot profiles and human profiles is the same. For the official system submissions, the performance of any system is evaluated based on the accuracy. We therefore also use this evaluation metric for our own experiments. We developed our models using Keras v2.1.2 with a Tensorflow v1.0.0 backend. Training the model was performed on a machine with 64GB memory and a GeForce GTX 1080 Ti GPU.
We implemented and evaluated our basic model using a word-based embedding method, where an embedding vector is generated for each unique word in the text corpus. The embedding formed the first layer of our model and was trained alongside the model itself. The texts are truncated to a length of 11,000 characters and we use a 300dimensional embedding. Thus, this layer on its own adds a further 46 million parameters to be trained in the model’s training process. By training the embedding as part of the model it allows the generated sequences of word indices to be the input to our model. The first layer of the network will be responsible for generating the word vectors for each of the words from the inputs.
We fine-tuned the hyperparameters of our basic model using the dedicated validation data set. In the end, we used the parameters as shown in Table 2. Note that these optimal hyperparameters performed best on both the architecture variations.
results for all the used embedding dimensions. The embedding dimensions gave similar results, with some slight differences in the model accuracy in favor of the model generating 300-dimensional vectors for each word. Thus, we decided to go with that embedding in the final model for the CLEF test runs.
same hyperparameters as our basic model and used word2vec as the method of generating the word embeddings, with a dimension size of 300 based on our results from Table 3. The evaluation results for the two architecture variations are shown in Table 4.
Although the training accuracies were similar, the validation accuracies were very different. Even though the Large Model is a deeper model (having more trainable weights and predicting power), the big difference between its training and validation accuracies, led us to believe that it was overfitting on the training data. One possible reason for this might be that the model is too complex for the limited number of training samples (articles) in our task. The table also shows that the Simple Model performs better in all the other metrics as well. Thus, the model chosen for the CLEF submission was the Simple Model.
Evaluating on CLEF’s Test Data Set Applying our basic model – using our simpler architecture variation, a word2vec embedding layer generating 300-dimensional vectors and the hyperparameters shown in Table 2 – on the official CLEF test data set via official approach submissions, we obtained an accuracy of 90.34%. Based on the accuracy alone, our model performed even better on the test data than in our experiments with the validation data. Comparing our approach to the approaches of other teams at CLEF 2019 with respect to bot identification on the English language – ignoring the other tasks proposed within the CLEF 2019 Bots and Gender Profiling Task –, our team ranked in the top half of all participating teams. 5
In this paper, we presented a convolutional neural network architecture for determining whether a Twitter feed is written by a bot or a human. In our experiments, we found that a convolutional neural network, using the flatten transition function and a 300dimensional word2vec embedding method performs best among our methods. In the official CLEF 2019 Bots and Gender Profiling Task test runs, we obtained a relatively high accuracy of 90.34%. In the future, besides evaluating a deeper convolutional neural network, we plan to develop further approaches based on LSTMs. 9. Llewellyn, C., Cram, L., Favero, A., Hill, R.L.: For Whom the Bell Trolls: Troll Behaviour in the Twitter Brexit Debate. CoRR abs/1801.08754 (2018) 10. Loyola-Gonza´lez, O., Monroy, R., Rodr´ıguez, J., Lo´pez-Cuevas, A., Mata-Sanchez, J.I.: Contrast pattern-based classification for bot detection on twitter. IEEE Access 7, 45800–45817 (2019) 11. Lundberg, J., Nordqvist, J., Laitinen, M.: Towards a language independent Twitter bot detector. In: Proceedings of the Digital Humanities in the Nordic Countries. pp. 308–319.
DHN’19 (2019) 12. Mazza, M., Cresci, S., Avvenuti, M., Quattrociocchi, W., Tesconi, M.: Rtbust: Exploiting temporal patterns for botnet detection on twitter. CoRR abs/1902.04506 (2019) 13. Rangel, F., Rosso, P.: Overview of the 7th Author Profiling Task at PAN 2019: Bots and Gender Profiling. In: Cappellato, L., Ferro, N., Losada, D., Mu¨ller, H. (eds.) CLEF 2019 Labs and Workshops, Notebook Papers. CEUR-WS.org (Sep 2019) 14. Varol, O., Ferrara, E., Davis, C.A., Menczer, F., Flammini, A.: Online Human-Bot Interactions: Detection, Estimation, and Characterization. In: Proceedings of the Eleventh International Conference on Web and Social Media. pp. 280–289. ICWSM’17 (2017)