The dataset was collected by Aragon et al. [1] using the Twitter API that we extended by a crawl of the user network. Our data set hence consists of two parts:
Tweet dataset: tweet text and user metadata on the
In case of a retweet, the Twitter API provides us with
the ID of the original tweet. By collecting retweets for a
given original tweet ID, we may obtain the set users who
have retweeted a given tweet with the corresponding retweet
timestamps. The Twitter API however does not tell us the
actual path of cascades if the original tweet was retweeted
1http://en.wikipedia.org/wiki/Occupy Wall Street
several times. The information from the Twitter API on
the tweet needs to be combined with the follower network
to reconstruct the possible information pathways for a given
tweet. However it can happen that for a given retweeter,
more than one friend has retweeted the corresponding tweet
before and hence we do not know the exact information
source of the retweeter. The retweet ambiguity problem is
well described in [
For a given tweet, the computed edges de ne us a retweet cascade. However our dataset contains only a sample of tweets on the given hashtags and hence may not be complete: it can happen that a few intermediate retweeters are missing from our data. As a result, sometimes the reconstructed cascade graphs are disconnected. As detailed in Fig. 2 (b) and (c), we handle this problem in two di erent ways. One possible solution is to only consider the rst connected component of the cascade (see Fig. 2 (b)). Another one is to connect each disconnected part to the root tweeter with one virtual cascade edge (see Fig. 2 (c)). In what follows, we work with cascades that contain virtual edges, therefore every retweeter is included in the cascade. 3.3
In Table 3, we give a few examples of highly retweeted messages with the actual urls and names replaced by [url] and [name]. 4.
To train our models, we generate features for each root tweet in the data and then we predict the future cascade size of the root tweet from these feature sets. For a given root tweet, we compute features about the author user and her follower network (network features) and the textual content of the tweet itself (content features).
The rst step of content processing is text normalization. We converted the text them into lower case form except those which are fully upper cased and replaced tokens by their stem given by the Porter stemming algorithm. We replaced user mentions (starting with '@') and numbers by placeholder strings and removed the punctuation marks.
The content features are extracted from the normalized
texts. The basic feature template in text analysis consists
the terms of the message. We used a simple whitespace
tokenizer rather than a more sophisticated linguistic tokenizer
as previous studies reported its empirical advantage [
Besides terms, we extracted the following features describing the orthography of the message:
Hashtags are used to mark speci c topics, they can be appended after the tweets or inline in the content, marked by #. From the counts of hashtags the user can tips the topic categories of tweet content but too many hashtag can be irritating to the readers as they just make confusion.
Telephone number: If the tweet contains telephone number it is more likely to be spam or ads.
Urls: The referred urls can navigate the reader to text, sound, and image information, like media elements and journals thus they can attract interested readers. We distinguish between full and truncated urls. The truncated urls are ended with three dot, its probably copied from other tweet content, so it was interested by somebody.
The like sign is an illustrator, encouragement to others to share the tweet.
The presence of a question mark indicates uncertainty.
In Twitter, questions are usually rhetorical|people do
not seek answers on Twitter [
The Exclamation mark highlights the part of the tweet, it expresses emotions and opinions.
If Numerical expressions are present the facts are quanti ed then it is more likely to have real information content. The actual value of numbers were ignored. Mentions: If a user mentioned (referred) in the tweet the content of the tweet is probably connected to the mentioned user. It can have informal or private content.
Emoticons are short character sequences representing emotions. We clustered the emoticons into positive, negative and neutral categories.
The last group of content features tries to capture the modality of the message:
Swear words in uence the style and attractiveness of the tweet. The reaction for swearing can be ignorance and also reattacking, which is not relevant in terms of retweet cascade size prediction. We extracted 458 swear words from http://www.youswear.com.
Weasel words and phrases2 aimed at creating an
impression that a speci c and/or meaningful statement
has been made when in fact only a vague or
ambiguous claim has been communicated. We used the weasel
word lexicon of [
We employed the linguistic inquiry categories (LIWC)
[
By using all the content features, we built n-grams as consecutive sequences in the tweet text that may include simply three terms (\posted a photo"), @-mentions, hashtags, url (\@OccupyPics Photo http://t.co/. . . " coded as [[user] Photo [url]]), numbers (\has [number] followers"), non-alphanumeric (\right now !") as well as markers for swear or weasel expressions (\[weasel word] people say"). We de ned the following classes of n-grams, for n 3: 2See http://en.wikipedia.org/wiki/Wikipedia: Embrace_weasel_words. Modality: The n-gram contains at least one swear or weasel word or expression (overall 208,368); Orthographic: No swear or weasel word but at least one orthographic term (overall 2,751,935); Terms: N-grams formed only of terms, no swear or weasel words and orthographic features (overall 771,196). For e ciency, we selected the most frequent 1,000 n-grams from each class. The entire feature set hence consists of 3,000 trigrams. 5.
Here we describe the way we generate training and test sets for our algorithms detailed in Section 6. First, for each root tweet we compute the corresponding network and content features. We create daily re-trained models: for a given day t, we train a model on all root tweets that have been generated before t but appeared later than t , where is the preset time frame. After training based on the data before a given day, we compute our predictions for all root tweets appeared in that day.
In order to keep the features up to date, we recompute all network properties online, on the y and use the new values to give predictions. By this method, we may immediately notice if a user starts gaining high attention or if a bursty event happens.
We take special attention to de ning the values used for training and evaluation. For evaluation, we used the information till the end of the three week data set collection period, i.e. we used all the known tweets that belong to the given cascade. However, for training, we are only allowed to use and count the tweets up to the end of the training period. Since the testing period is longer, we linearly approximated the values for the remaining part of the testing period.
Our goal is to predict cascade size at the time when the root tweet is generated. One method we use is regression, which directly predict the size of the retweet cascade. For regression, we only use the global error measures:
We also experiment with multiclass classi cation for ranges
of the cascade size. The cascade size follows a power law
distribution (see Fig. 3) and we de ned three buckets, one with
0. . . 10 (referred as \low"), one with 11. . . 100 (\medium")
and a largest one with more than 100 (\high") retweeters
participating in the cascade. We evaluate performance by
AUC [
By the probabilistic interpretation of AUC, we may realize that a classi er will perform well if it orders the users well with little consideration on their individual messages. Since our goal is to predict the messages in time and not the rather static user visibility and in uence, we de ne new averaging schemes for predicting the success of individual messages.
We consider the classi cation of the messages of a single user and de ne two aggregations of the individual AUC values. First, we simply average the AUC values of users for each day (user average)
N AU Cuser = 1 X AU Ci;
N i=1 Second, we are weighting the individual AUC values with the activity of the user (number of tweets by the user for the actual day)
AU Cwuser =
PN i=1 AU CiTi PiN Ti where Ti is the number of tweets by the i-th user.
We may also obtain regressors from the multiclass classi cation results. In order to make classi cation and regression comparable, we give a very simple transformation that replaces each class by a value that can be used as regressor. (1) (2)
We select and use the training set average value in each class as the ideal value for the prediction.
In this section, we train and evaluate rst the classi cation and then the regression models to predict the future cascade size of tweets. We predict day by day, for each day in the testing period. For classi cation, we also evaluate on the user level by using equations (1) and (2). For classi cation, we show the best performing features as well.
As mentioned in Section 5, we may train our model with di erent . In Figure 4 we show the average AUC value with di erent time frames. As Twitter trends change rapidly, we achieve the best average results if we train our algorithms on root tweets that were generated in the previous week (approximately seven days), both for global and for user level average evaluation.
First, we measure classi er performance by computing
the average AUC values of the nal results for the three
size ranges. We were interested in how di erent classi ers
perform and how di erent feature sets a ect classi er
performance. For this reason, we repeated our experiments
with di erent feature subsets. Figure 5 shows our results.
For each day, the network features give a strong baseline.
The combination of these features with the content result
in strong improvement in classi er performance. In Table 5
we summarize the average AUC values for di erent feature
subsets over all four datasets. Our results are consistent:
in all cases, the content related features improve the
performance. Finally, we give the performance of other classi ers
in Table 6 and conclude the superiority of the Random
Forest classi er [
We give regression results by the linear regression,
multilayer perceptron and the regression tree implementation
of Weka [
We selected the most important network features by
running a LogitBoost classi er [
We selected the most important content features by running logistic regression over the 3,000 trigrams described in Section 4.3. The features are complex expressions containing elements from the three major group of linguistic feature sets in the following order of absolute weight obtained by logistic regression: 1. Three words [marriage between democracy], in this order; 2. [at [hashtag occupywallstreet][url]]: the word \at", followed by the hashtag \#occupywallstreet", and a url; 3. [between democracy and]; 4. [capitalism is over]; 5. [[hashtag ows] pls]; 6. [[weasel word] marriage between]: the expression \marriage between" on the weasel word list, which counts as the third element of the trigram; 7. [[hashtag zizek] at [hashtag occupywallstreet]]; 8. [[hashtag occupywallstreet][url][hashtag auspol]]; 9. [over [hashtag zizek] at]; 10. [calientan la]: means \heating up".
Note that all these features have negative weight for the upper two classes and positive or close to 0 for the lower class. Hence the appearance of these trigrams decrease the value obtained by the network feature based model. We may conclude that the use of weasel words and uninformative phrases reduce the chance of getting retweeted, as opposed to the sample highly retweeted messages in Table 3. 6.6
To illustrate the importance of the temporal training and evaluation framework and the online update of the network features, we made an experiment where we replaced user features by static ones. The results are summarized in Table 9. Note that on the user level, all messages will have the same network features and hence classi cation will be random with AUC=0.5. In contrast, online updated network features are already capable of distinguishing between the messages of the same user, as seen in Tables 5 and 7. 7.
In this paper we investigated the possibility of predicting the future popularity of a recently appeared text message in Twitter's social networking system. Besides the typical user and network related features, we consider hashtag and linguistic analysis based ones as well. Our results do not only con rm the possibility of predicting the future popularity of a tweet, but also indicate that deep content analysis is important to improve the quality of the prediction.
In our experiments, we give high importance to the temporal aspects of the prediction: we predict immediately after the message is published, and we also evaluate on the user level. We consider user level evaluation key in temporal analysis, since the in uence and popularity of a given user is relative stable while the retweet count of her particular messages may greatly vary in time. On the user level, we observe the importance of linguistic elements of the content.
We thank Andreas Kaltenbrunner for providing us with the Twitter data set [1].
[1] P. Aragon, K. E. Kappler, A. Kaltenbrunner, D. Laniado, and Y. Volkovich. Communication dynamics in twitter during political campaigns: The case of the 2011 spanish national election. Policy & Internet, 5(2):183{206, 2013. [2] E. Bakshy, D. Eckles, R. Yan, and I. Rosenn. Social in uence in social advertising: evidence from eld experiments. In Proceedings of the 13th ACM Conference on Electronic Commerce, pages 146{161. ACM, 2012.