Social spam produces a great amount of noise on social media services such as Twitter, which reduces the signal-tonoise ratio that both end users and data mining applications observe. Existing techniques on social spam detection have focused primarily on the identification of spam accounts by using extensive historical and network-based data. In this paper we focus on the detection of spam tweets, which optimises the amount of data that needs to be gathered by relying only on tweet-inherent features. This enables the application of the spam detection system to a large set of tweets in a timely fashion, potentially applicable in a realtime or near real-time setting. Using two large hand-labelled datasets of tweets containing spam, we study the suitability of five classification algorithms and four di↵erent feature sets to the social spam detection task. Our results show that, by using the limited set of features readily available in a tweet, we can achieve encouraging results which are competitive when compared against existing spammer detection systems that make use of additional, costly user features. Our study is the first that attempts at generalising conclusions on the optimal classifiers and sets of features for social spam detection over di↵erent datasets.
Social networking spam, or social spam, is increasingly
affecting social networking websites, such as Facebook,
Pinterest and Twitter. According to a study by the social media
security firm Nexgate [
Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00.
enced a 355% growth of social spam during the first half of
2013. Social spam can reach a surprisingly high visibility
even with a simple bot [
Social spam has di↵erent e↵ects and therefore its
definition varies across major social networking websites. One of
the most popular social networking services, Twitter, has
published their definition of spamming as part of their “The
Twitter Rules” 1 and provided several methods for users to
report spam such as tweeting “@spam @username” where
@username will be reported as a spammer. While as a
business, Twitter is also generous with mainline bot-level access
2 and allows some level of advertisements as long as they
do not violate “The Twitter Rules”. In recent years we have
seen Twitter being used as a prominent knowledge base for
discovering hidden insights and predicting trends from
finance to public sector, both in industry and academia. The
ability to sort out the signal (or the information) from
Twitter noise is crucial, and one of the biggest e↵ects of Twitter
spam is that it significantly reduces the signal-to-noise ratio.
Our work on social spam is motivated by the initial attempts
at harvesting a Twitter corpus around a specific topic with
a set of predefined keywords [
In previous works, the automatic detection of Twitter
spam has been addressed in two die↵rent ways. The first
way is to tackle the task as a user classification problem,
where a user can be deemed either a spammer or a
nonspammer. This approach, which has been used by the
majority of the works in the literature so far (see e.g., [
In this work, after discussing the definition of social spam and reviewing previous research in Twitter spam detection, we present a comparative study of Twitter spam detection systems. We investigate the use of di↵erent features inherent to a tweet so as to identify the sets of features that do best in categorising tweets as spam or not. Our study compares five die↵rent classification algorithms over two di↵erent datasets. The fact that we test our classifiers on two di↵erent datasets, collected in di↵erent ways, enables us to validate the results and claim repeatability. Our results suggest a competitive performance can be obtained using tree-based classifiers for spam detection even with only tweet-inherent features, as comparing to the existing spammer detection studies. Also the combination of di↵erent features generally lead to an improved performance, with User feature + Bi & Tri-gram (Tf) having the best results for both datasets. 2.
The detection of spam has now been studied for more
than a decade since email spam [
As we said before, most of the previous work in the area
has focused on the detection of users that produce spam
content (i.e., spammers), using historical or network features of
the user rather than information inherent to the tweet. Early
work by [
Lee et al. [
Few studies have addressed the problem of spam
detection. Santos et al. [
The few works that have dealt with spam detection are
mostly limited in terms of the sets of features that they
studied, and the experiments have been only conducted in
a single dataset (except in the case of [
In this section we describe the Twitter spam datasets we used, the text preprocessing techniques that we performed on the tweets, and the four di↵erent feature sets we used for training our spam vs non-spam classifier. 3.1
A labelled collection of tweets is crucial in a machine learning task such as spam detection. We found no spam dataset which is publicly available and specifically fulfils the requirements of our task. Instead, the datasets we obtained include Twitter users labelled as spammers or not. For our work, we used the latter, which we adapted to our purposes by taking out the features that would not be available in our scenario of spam detection from tweet-inherent features. We used two spammer datasets in this work, which have been created using die↵rent data collection techniques and therefore is suitable to our purposes of testing the spam classifier in die↵rent settings. To accomodate the datasets to our needs, we sample one tweet for each user in the dataset, so that we can only access one tweet per user and cannot aggregate several tweets from the same user or use social network features. In what follows we describe the two datasets we use.
manipulated (by posting random messages and engaging in
none of the activities of legitimate users) 60 social honeypot
accounts on Twitter to attract spammers. Their dataset
consists of 22,223 spammers and 19,276 legitimate users
along with their most recent tweets. They used
ExpectationMaximization (EM) clustering algorithm and then manually
grouped their harvested users into 4 categories: duplicate
spammers, duplicate @ spammers, malicious promoters and
friend infiltrators. 1KS-10KN Dataset: Yang et al. [
We used spammer vs. legitimate user datasets from [
Before we extract the features to be used by the classifier from each tweet, we apply a set of preprocessing techniques to the content of the tweets to normalise it and reduce the noise in the classification phase. The preprocessing techniques include decoding HTML entities, and expanding contractions with apostrophes to standard spellings (e.g. “I’m” -> “I am”). More advanced preprocessing techniques such as spell-checking and stemming were tested but later discarded given the minimal e↵ect we observed in the performance of the classifiers.
For the specific case of the extraction of sentiment-based features, we also remove hashtags, links, and user mentions from tweet contents. 3.3
As spammers and legitimate users have di↵erent goals in posting tweets or interacting with other users on Twitter, we can expect that the characteristics of spam tweets are quite die↵rent to the normal tweets. The features inherent to a tweet include, besides the tweet content itself, a set of metadata including information about the user who posted the tweet, which is also readily available in the stream of tweets we have access to in our scenario. We analyse a wide range of features that reflect user behaviour, which can be computed straightforwardly and do not require high computational cost, and also describe the linguistic properties that are shown in the tweet content. We considered four feature sets: (i) user features, (ii) content features, (iii) n-grams, and (iv) sentiment features.
User features include a list of 11 attributes about the
author of the tweet (as seen in Table 1) that is generated
from each tweet’s metadata, such as reputation of the user
[
Content features capture the linguistic properties from
the text of each tweet (Table 1) including a list of content
attributes and part-of-speech tags. Among the 17 content
attributes, number of spam words and number of spam words
per word are generated by matching a popular list of spam
words 3. Part-of-speech (or POS) tagging provides
syntactic (or grammatical) information of a sentence and has been
used in the natural language processing community for
measuring text informativeness (e.g. Tan et al. [
N-gram models have long been used in natural language processing for various tasks including text classification. Although it is often criticized for its lack of any explicit representation of long range or semantic dependency, it is surpris
blacklist/blob/master/blacklist.txt
Length of profile name Length of profile description Number of followings (FI) Number of followers (FE) Number of tweets posted Reputation of the user (FE/(FI + FE)) Following rate (FI/AU) Number of tweets posted per day Number of tweets posted per week Uni + bi-gram or bi + tri-gram
Automatically created sentiment lexicons Manually created sentiment lexicons Age of the user account, in hours (AU) Ratio of number of followings and followers (FE/FI) Mean word length Number of words Number of characters Number of white spaces Number of capitalization words Number of capitalization words per word Maximum word length Number of exclamation marks Number of question marks Number of URL links Number of URL links per word Number of hashtags Number of hashtags per word Number of mentions Number of mentions per word Number of spam words Number of spam words per word
Part of speech tags of every tweet ingly powerful for simple text classification with reasonable amount of training data. In order to give the best classification result while being computationally ecient we have tried uni + bi-gram or bi + tri-gram with binary (i.e. 1 for feature presence while 0 for absence), term-frequency (tf) and tf-idf (i.e. Term Frequency times Inverse Document Frequency) techniques.
Sentiment features: Ferrara et al. [
During the classification and evaluation stage, we tested 5 classification algorithms implemented using scikit-learn4: Bernoulli Naive Bayes, K-Nearest Neighbour (KNN), Support Vector Machines (SVM), Decision Tree, and Random Forests. These algorithms were chosen as being the most commonly used in the previous research on spammer detection. We evaluate using the standard information retrieval metrics of recall (R), precision (P) and F1-measure. Recall in this case refers to the ratio obtained from diving the number of correctly classified spam tweets (i.e. True Positives) by the number of tweets that are actually spam (i.e. True Positives + False Negatives). Precision is the ratio of the number of correctly classified spam tweets (i.e. True Positives) to the total number of tweets that are classified as spam (i.e. True Positives + False Positives). F1-measure can be interpreted as a harmonic mean of the precision and recall, where its score reaches its best value at 1 and worst at 0. It is defined as:
F 1 = 2 ⇤ (precision ⇤ recall)/(precision + recall)
In order to select the best classifier for our task, we have
used a subset of each dataset (20% for 1KS-10KN dataset
and 40% for Social Honeypot dataset, due to the die↵rent
sizes of the two datasets) to run a 10-fold cross validation for
optimising the hyperparameters of each classifier. By doing
so it minimises the risk of over-fitting in model selection and
hence subsequent selection bias in performance evaluation.
Such optimisation was conducted using all 4 feature sets
(each feature was normalised to fit the range of values [
As shown in Table 2, tree-based classifiers achieved very promising performances, among which Random Forests outperform all the others when we look at the F1-measure. This outperformance occurs especially due to the high precision values of 99.3% and 94.1% obtained by the Random Forest classifier. While Random Forests show a clear superiority in terms of precision, its performance in terms of recall varies for the two datasets; it achieves high recall for the Social Honeypot dataset, while it drops substantially for the 1KS-10KN dataset due to its approximate 1:10 spam/nonspam ratio. These results are consistent with the conclusion of most spammer detection studies; our results extend this conclusion to the spam detection task.
When we compare the performance values for the di↵erent
datasets, it is worth noting that with the Social Honeypot
dataset the best result is more than 10% higher than the
best result in 1KS-10KN dataset. This is caused by the
die↵rent spam/non-spam ratios in the two datasets, as the
Social Honeypot dataset has a roughly 50:50 ratio while in
1KS-10KN it is roughly 1:10 which is a more realistic ratio
to reflect the amount of spam tweets existing on Twitter
(In Twitter’s 2014 Q2 earnings report it says that less than
5% of its accounts are spam5, but independent researchers
believe the number is higher). In comparison to the original
papers, [
We trained our best classifier (i.e. Random Forests) with di↵erent feature sets, as well as combinations of the feature sets using the two datasets (i.e. the whole corpora), and under a 10-fold cross validation setting. We report our results in Table 3. As seen in 1KS-10KN dataset, the F1-measure for di↵erent feature sets ranges from 0.718 to 0.820 when using a single feature set. All feature set combinations except C + S (content + sentiment feature) perform higher than 0.810 in terms of F1-measure, reflecting that feature combinations have more discriminative power than a single feature set.
For the Social Honeypot dataset, we can clearly see User
features (U) having the most discriminative power as it has
a 0.940 F1-measure. Results without using User features
(U) have significantly worse performance, and feature
combinations with U give very little improvement with respect
to the original 0.940 (except for U + Uni & Bi-gram (Tf) +
S). This means U is dominating the discriminative power of
these feature combinations and other feature sets contribute
very little in comparison to U. This is potentially caused
by the data collection approach (i.e. by using social
honeypots) adopted by [
beneficial to each other irrespective of the di↵erent nature of datasets.
Another important aspect to take into account when choosing the features to be used is the computation time, especially when one wants to apply the spam classifier in realtime. Table 4 shows a eciency comparison for generating each feature from 1000 tweets, using a machine with 2.8 GHz Intel Core i7 processor and 16 GB memory. Some of the features, such as the User features, can be computed quickly and require minimal computational cost, as most of these features can be straightforwardly inferred from a tweet’s metadata. Other features, such as N-grams and part-of-speech counts (from Content features), can be affected by the size of the vocabulary in the training set. On the other hand, some of the features are computationally more expensive, and therefore worth studying their applicability. This is the case of Sentiment features, which require string matching between our training documents and a list of lexica we used. We keep the sentiment features since they have shown added value in the performance evaluation of feature set combinations. Similarly, Content features such as Number of spam words and Number of spam words per word also require string matching between our training documents and a dictionary containing 11,529 spam words. However, given that the latter did not provide significant improvements in terms of accuracy, most probably because the spam words were extracted from blogs, we conclude that Number of spam words and Number of spam words per word can be taken out from the representation for the sake of the classifier’s eciency. 5.
Our study looks at di↵erent classifiers and feature sets over two spam datasets to pick the settings that perform best. First, our study on spam classification buttresses previous findings for the task of spammer classification, where Random Forests were found to be the most accurate classifier. Second, our comparison of four feature sets reveals the features that, being readily available in each tweet, perform best in identifying spam tweets. While di↵erent features perform better for each of the datasets when using them alone, our comparison shows that the combination of different features leads to an improved performance in both datasets. We believe that the use of multiple feature sets increases the possibility to capture di↵erent spam types, and makes it more dicult for spammers to evade all feature sets used by the spam detection system. For example spammers might buy more followers to look more legitimate but it is still very likely that their spam tweet will be detected as its tweet content will give away its spam nature.
Due to practical limitations, we have generated our spam vs. non-spam data from two spammer vs. non-spammer datasets that were collected in 2011. For future work, we plan to generate a labelled spam/non-spam dataset which was crawled in 2014. This will not only give us a purposebuilt corpus of spam tweets to reduce the possible e↵ect of sampling bias of the two datasets that we used, but will also give us insights on how the nature of Twitter spam changes over time and how spammers have evolved since 2011 (as spammers do evolve and their spam content are manipulated to look more and more like normal tweet). Furthermore we will investigate the feasibility of cross-dataset spam classification using domain adaptation methods, and also whether
Bernoulli NB
KNN
SVM Decision Tree Random Forest unsupervised approaches work well enough in the domain of Twitter spam detection.
A caveat of the approach we relied on for the dataset generation is the fact that we have considered spam tweets posted by users who were deemed spammers. This was done based on the assumption that the majority of social spam tweets on Twitter are shared by spam accounts. However, the dataset could also be complemented with spam tweets which are occasionally posted by legitimate users, which our work did not deal with. An interesting study to complement our work would be to look at these spam tweets posted by legitimate users, both to quantify this type of tweets, as well as to analyse whether they present di↵erent features from those in our datasets, especially when it comes to the userbased features as users might have di↵erent characteristics. For future work, we plan to conduct further evaluation on how our features would function for spam tweets shared by legitimate users, in order to fully understand the e↵ects of bias of pursuing our approach of corpus construction. 6.
In this paper we focus on the detection of spam tweets, solely making use of the features inherent to each tweet. This die↵rs from most previous research works that classified Twitter users as spammers instead, and represents a real scenario where either a user is tracking an event on Twitter, or a tool is collecting tweets associated with an event. In these situations, the spam removal process cannot a↵ord to retrieve historical and network-based features for all the tweets involved with the event, due to high number of requests to the Twitter API that this represents. We have tested five die↵rent classifiers, and four di↵erent feature sets on two Twitter spam datasets with di↵erent characteristics, which allows us to validate our results and claim repeatability. While the task is more dicult and has access to fewer data than a spammer classification task, our results show competitive performances. Moreover, our system can be applied for detecting spam tweets in real time and does not require any feature not readily available in a tweet.
Here we have conducted the experiments on two di↵erent datasets which were originally collected in 2011. While this allows us to validate the results with two datasets collected in very di↵erent methods, our plan for future work includes the application of the spam detection system to more recent events, to assess the validity of the classifier with recent data as Twitter and spammers may have evolved.