This paper describes the first place winning solution for the CIKM AnalytiCup 2020 COVID-19 retweet prediction challenge. The objective of the challenge is to predict the popularity of COVID-19 related tweets in terms of the number of retweets, and the submitted solutions of the challenge are ranked based on Mean Squared Logarithmic Error(MSLE) on the leaderboard. The proposed deep learning model to predict retweet counts uses minimal hand-engineered features and learns to predict retweet count based on a personalized attention mechanism. As a tweet keyword may have diferent informativeness for diferent users, the personalized attention mechanism helps the deep learning model to weigh the importance of tweet keywords based on a user's interest to retweet. Additional techniques such as adding external data sets to training and pseudolabeling are also experimented with to further improve the MSLE score. The final solution comprises of an ensemble of diferent personalized attention-based deep learning models, and the source code for the solution can be found at https://github.com/vinayakarajt/CIKM2020.
Understanding information difusion in social networks is
imperative as it helps to comprehend social interactions among users
in a better way. Information spread on a large scale in social
networks enables marketers, advertisers to design their campaigns
more efectively to target potential customers. In addition to that,
identifying influential users [
Modeling information difusion in social networks is a hot research topic that has garnered more interest in research communities of late. In CIKM AnalytiCup 2020, the competition objective is to model the information spreading mechanism during COVID-19 by predicting the retweet count of tweets on Twitter. Retweeting a tweet is one of the functions of Twitter that helps users to quickly share their tweets or tweets of other users to all of their followers. Retweets can be seen as one of the ways information spreads on
Twitter and are very crucial to understand the information difusion
mechanism on Twitter. Some of the practical applications of
information difusion using tweets are political audience design [
In this paper, all the techniques used to predict the retweet count of tweets are discussed in detail. The first section provides a summary of the dataset presented to solve the problem. Handengineering new features and their pre-processing techniques are also explained in this section. Model architecture and the personalized attention mechanism are described in the next section, and ifnally, in the last section, all the experiments carried out to improve the model score on the test leaderboard are explained in detail. 2
TweetsCOV19 [
In addition to the given metadata features, few more features are derived from the given tweet metadata information and used to predict the retweet count. A full list of features and their preprocessing techniques is provided in Table 1.
Both original tweet keywords and their respective annotated entities are extracted and considered for analysis. Numbers and special characters are removed from hashtags and mentions, and duplicate keywords are removed from entities, hashtags, and mentions. URLs are split into two parts. The hostname of the tweet URL is extracted as URL-1, and the path of the URL is considered as URL-2. Besides that, numbers and special characters are removed from URL-2. 3
Figure 1 shows the network architecture of the retweet prediction model and the attention mechanism. High cardinal feature such Feature week time year no_entities keyword_entities
no_hashtags keyword_hashtags no_mentions no_urls Sentiment #Favorites #Followers #Friends #Followers/#Friends #Friends/#Favorites #Favorites/#Followers username
Description Week info extracted from timestamp Time info extracted from timestamp Year info extracted from timestamp
No. of entities in a tweet No. of COVID related entities in a tweet
No. of hashtags in a tweet No. of COVID related hashtags in a tweet
No. of mentions in a tweet
No. of urls in a tweet Sentiment score from SentiStrength (-4 to 4)
Tweet favorites No. of followers of an user
No. of friends of an user No. of followers and friends ratio No. of friends and favorites ratio No. of favorites and followers ratio
Encrypted username
Log transformed and then standardized Log transformed and then standardized Log transformed and then standardized Log transformed and then standardized Log transformed and then standardized Log transformed and then standardized Log transformed and then standardized Log transformed and then standardized
One-hot-encoded Log transformed and then standardized Log transformed and then standardized Log transformed and then standardized Log transformed and then standardized Log transformed and then standardized Log transformed and then standardized Label encoded
Retweet Count Dense Layer 2 (150) Dense Layer 1 (500) Numeric and One Hot Encoded Features
Hashtag Representation Mentions Representation Url Host Representation Url Path Representation At ention Outputs User Dense
Vector User Dense Layer
LSTM/CNN Output Sequences
Entity Word Vectors LSTM/CNN Output Sequences
LSTM/CNN Output Sequences
Hashtag Word Vectors
Word Embedding Layer LSTM/CNN Output Sequences
Host Word Vectors Username Entities Hashtags Mentions Host Path URLs LSTM/CNN Output Sequences
Path Word Vectors
Personlaized At ention as username is embedded as a fixed-length vector using user embedding layer, which is then passed through a series of user dense layers to get the final representation of users.
The word embedding layer is shared by the preprocessed keywords of tweet entities, mentions, hashtags, and URLs and is initialized by any pre-trained word vectors. For a tweet, entity word vectors are a sequence of word vectors queried from the embedding
v1 a1 ai aj am vi vj vm
layer and the length of the sequence is equivalent to the number of entity keywords extracted from that tweet. The length of the sequence is fixed for the entire dataset and is a hyper-parameter in the model. Entity keywords smaller than the sequence length are padded with zeros on the left, and the larger ones are trimmed on the right. Word vectors of hashtags, mentions, and URLs are created the same way as the entity word vectors. These extracted word vectors are then inputted to an LSTM/CNN layer, which is then used to learn the representation of entities, mentions, hashtags, and URLs from their respective word vector sequence.
User vector representation and the LSTM/CNN output vectors
are used to create personalized attention mechanism [
( ) = Í
=1 ( ) where and are user projection parameters and is the sequence length. The final representation of entities/hashtags/mentions/URLs is given by: = Õ
∗ =1
The final representation vectors of entities, hashtags, mentions, and URLs are then concatenated together with the user vector and other features such as tweet metadata and time-based features. The ifnal concatenated feature vector is then passed through a series of dense layers to estimate the retweet count. 4 4.1
Dataset provided for the competition comprises of all COVID 19 related tweets from 2019-09-30 to 2020-05-31. Of which, the entire month of May 2020 is considered for testing and is split into two testing sets - testing set 1 & 2. Testing set 1 is used for validating the model on the leader board and testing set 2 is used to rank the final winners of the competition. The rest of the data set from 2019-09-30 to 2020-04-30 is used for training the model. The training data set is sorted in chronological order and the very recent 5% tweets of the training data set are filtered out and used as the validation set. Information about training, validation, and testing splits are provided in Table 2.
Mean Square Logarithmic Error (MSLE) is the evaluation metric used in this competition. MSLE is given by: 1 Õ =0 =
( (( + 1) − ( + 1))2 where and are the actual and predicted retweet counts respectively. MSLE penalizes under estimations more than over estimations.
The model described in Figure 1 is trained on a Tesla V100 GPU machine. The optimal hyper-parameter settings are selected based on the model with the best MSLE score on the validation set, and the tuned hyperparameters of the model setup are provided in Table 3. 4.2
The performance of the models on the final testing dataset is shown
in Table 4. A single personalized attention model with fast text
embedding and LSTM head for learning tweet representation provides
an MSLE score of 0.12860 on the test dataset. A large collection
of annotated tweets for the months of April 2020 and March 2020
from the dataset corpus TweetsKB is added to the training dataset
and a deep learning model is trained on the whole dataset. This
addition of an external dataset decreases the MSLE score by 3.76%.
TweetsCOV19 is the subset of TweetsKB and hence doesn’t include
all the tweets of users but their COVID related tweets. Including all
tweets of a user not only help the personalized mechanism to
understand the relation between users and their tweets but also help the
model to learn a rich representation of users and tweet keywords.
To further improve the score, techniques such as ensembling and
pseudo-labeling are also tried.
4.2.1 Ensembling. In addition to initialing the Twitter keywords
with fasttext embeddings [
A methodology to estimate retweet count for COVID related tweets is proposed in this paper. The personalized attention-based deep learning model described in this paper uses less hand-engineered features and learns a rich representation of users and tweet keywords to predict retweet count. To further improve the performance of the model, techniques such as adding external datasets, ensembling, and pseudo-labeling are also tried. The final solution to estimate retweet counts is created by an ensemble of deep learning models which placed the team first on the testing leaderboard.