In this paper, we present our solution for COVID-19 retweet prediction challenge. The proposed approach consists of feature engineering and modeling. For feature engineering, we leverage both hand-crafted and unsupervised learning features. As the provided data set is large, we implement auto-encoding algorithms to reduce feature dimension. To develop predictive models, we utilize ensemble learning and deep learning algorithms. We then combine these models to generate the final blended model. Moreover, to stabilize the predictions, we also apply bagging as well as down-sampling techniques to remove the tweets where number of retweets equals to zero. Our solution is ranked on the public test set and on the private test set.
Machine learning and Text mining have been proved to be a
powerful tools for processing unstructured text data and making
shortterm prediction, whether they are still helpful during the time of
crisis, e.g. the ongoing Coronavirus disease 2019 (COVID-19), is
still in question. In this paper, we take advantage of both text
mining and machine learning methods to build more accurate models
to predict number of retweets in Twitter. Our proposed approach
consists of two major parts: feature engineering and modelling.
For feature engineering, we leverage hand-crafted features and
auto feature learning based on neural networks models. We use
most of state-of-the-art algorithms in text mining and machine
learning. For example, we rely on word2vec [
The provided TweetsCOV19 dataset [
Our feature set consists of both hand-crafted features and unsupervised learning features. 3.1
We aggregate data by users to generate user-level features. For example, we group data by users and concatenate all tweets belonging to the users. We then generate features for them as follows. 3.1.1
Hand-crafted user feature . • Followers and friends ratio. Along with the number of followers and friends, we derive the ratio of followers/(friends+1). • Tweet count. Number of tweets for each user is calculated. • Statistical features. Since the followers, friends and favourites vary for a given user over time for each user, various statistics like sum, mean, median, min, max, standard deviation are used to aggregate these columns separately for a user. • Time based features. The number of tweets made by a user in a particular month and hour, number of diferent hours in which a user tweeted, the average time between two consecutive tweets were used as features. Also the total number of unique users who tweeted on a particular date and hour are also calculated. • User sentiments. For each user, the mean and standard deviation are calculated for both positive and negative sentiments. Also, the number of tweets for each value of positive and negative sentiments for a user are calculated. 3.1.2 Hashtags/Mentions. There are two kind of hashtags/mentions features in our models.
Hand-crafted Feature. We calculate count features from hashtags and mentions as follows.
• Number of unique hashtags and mentions used by a user are given as input for each tweet of that user.
• Number of tweets that hashtags or mentions occur in.
Word2Vec Features. There are only a few mentions in a tweet in
the given data set. Most of the time, users don’t mention anything
in their tweets. If we directly use mentions as features (e.g.
onehot encoding), the feature space will be very sparse. Therefore,
we propose to learn dense embedding features for mentions by
considering each mention as word and train a word2vec model of
64 dimension using Gensim [
• Pre-process entity triples and keep matched entities only. • Aggregate entities by concatenating them based on user • Train diferent embedding models with various parameters such as embedding size and vocabulary size.
• Extract embedding features for each user.
For embedding models, we try word2vec, doc2vec, and BERT models. In the following sections, we present how we train these model in detail.
Doc2Vec. In order to generate doc2vec features [
Bert. Similar to doc2vec model, we aggregate entities by users
and treat these as documents. We choose the top attention layer
of BERT model to extract features. As this layer has dimensions of
1024, we reduce its dimension using:
• Truncated SVD [
• Number of unique URLs used by a user
• Number of legit URLS in the tweet.
• URL field is fitted and transformed on a Count Vectorizer [
This is given as input to a SVD (Singular Value Decomposition) with 5 principal components to obtain 5 URL_SVD features. • Domain name and sufix(url part followed by the domain name) are extracted for each URL in a given tweet and following are created: (1) A flag that indicates the presence of ‘Twitter’ word in the domain name. (2) For each domain name, number of tweets with that domain name are recorded. If a tweet has multiple URLs, then statistics like min, max, average are computed on domain frequencies for each tweet.
• Same features are created for sufix also. 3.1.5
• For each tweet, the product of the number of followers and the sentiment values, both positive and negative were taken. For the negative sentiment values the absolute of the sentiment value was taken. • Similarly, the product of the number of friends and the sentiment values was also used as a feature. 3.2
For tweet-level features, we generate the features using label encoding for user ID, Hashtag, Mention and URL. We also use some features given in the raw data directly like favourites, friends, followers etc. Note that we only use label encoded features directly in tree-based/boosting models. For neural network models, we use embedding layer on top of these. 3.3
Text based attributes of tweets such as hashtags {ℎ }, mentions { }, and entities { } can have a time based trend, also for each tweet there can be multiple values of the above. Since these attributes can have a time based trend, we create features to capture those trends. For the explanation purpose, we will use { } for {ℎ }, { }, and { }. First we created features for each { } and since one tweet can have multiple { }, we aggregate these features to a tweet level.
• Score: Number of tweets which have used in that day • Age: Number of days since came into existence. • Life: Number of days has been active through the course of the data. • Trend: Change in the score of from the previous day. • Peak: Fraction of tweets of this day.
Below are the aggregate tweet level features: • Score based features: we take average and max of the score of all in a tweet. • Weighted score features: Instead of normal average and max of scores, we take weighted average where two type of weights are used: inverse of ‘Age’ and ‘Life’ of • Age and life based features: we take average, min and max of ‘Age’ of all in a tweet attribute. Similar features are created for life. • Trend: we take average trend, max trend of all in a tweet attribute. • Peak: we take average and max of peak values of all in a tweet. 3.4
In this section, we introduce our graph-based feature set. In order
to generate the features, we build a few kind of graphs such as
user-entity graph (UE), user-hashtag graph (UH), and combined
user-hashtag-URL-mention graph (CG).
3.4.1 UE. To generate graph feature, we build a user-entity graph
as follows: (i) each node represents a user or an entity and (ii) create
an edge between a user and an entity if user tweets about that entity.
We use pyTorch-biggraph [
Additional to embedding graph features, we use traditional graph
algorithms to extract information of users from the graphs such as
centrality, average neighbor degrees, etc. Moreover, we also train
node2vec [
We use hold-out testing technique to validate our models where 80%
of the data is for training and 20% is for testing. Our final solution
is an ensemble of 3 major popular machine learning algorithms:
Catboost [
We use Catboost (CAT) [
Neural Network Models 4.2.1 Convolutional Neural Networks (CNN). Figure 2 shows the architecture of the CNN and feed forward based model. Input to the model are all numerical features discussed above, embeddings for entity, hashtag, and mentions. We also graph based features derived from user-hashtag graph as described in Section 3.4.2. 1D convolutional layers are used to extract features from local input patches allowing for representation modularity and data eficiency. We use 64 filters for each variable followed by max pooling. We then concatenate the output of these CNN Layers. Simultaneously, we pass categorical features like hour, weekday, month, user id, hashtag, and entity to the embedding layer. We concatenate all of these layers together and pass them to a feed forward neural networks and obtain the final retweet count. 4.2.2 Embedding Neural Networks Model. To develop embedding neural networks model, we classify features into two categories: numerical and categorical features. For numerical features such as no. of friends, no. of followers, etc., we use log scale to transform the data. For categorical features such as mentions, hashtags, and entities, we use embedding layer to deal with them. 4.3
As discussed in Section 2, there are about 50% common users in train and test sets. This serves as the motivation to train a model specific to the common users only. Apart from the features discussed in Section 3, we create some additional features specific to common user model. We used catboost to train the model
K-Fold Target Encoding. Target encoding for each user is created to capture the popularity of a user, but a model using traditional target encoding (taking mean of the retweets for all the tweets of a user) tends to overfit the training data set and did not perform well. To tackle this problem, we use 5-Fold target encoding.
Moving Averages. To capture the trend of a particular user whether be the gain or loss in popularity over time, we create trend based features. For a user on a given day we find the moving averages of number of followers and friends for the last 3, 5, and 10 tweets. This is then used directly as a paramter in the model. 4.4
We notice that most of tweets have no retweet count. If we use the whole training data to train models, the prediction in the validation and test sets tend to be smaller than the actual number of retweets. Hence, we use k-fold down sampling these tweets as follows: • Split the training data into two sets A (retweets > 0) and B (retweets = 0). • Apply k-fold cross validation on set B to select 80% of the data to have a new data set C. Merge sets A and C to train predictive models and make prediction on validation and test sets. In this case, we have total 5 predictions , corresponding to each of the fold. • The final prediction of validation and test sets will be the averaged prediction of k folds.
NN test val 0.12763 0.12649
CNN test 0.13139 0.13018
In practice, combining multiple machine learning models will help improve the performance of the final model. In order to do that, the simplest way is to use weighted average the prediction of selected models. Our ensemble model prediction is a linear combination of Catboost, LightGBM, and neural networks models. We estimate the weights based on the performance of the models on the leaderboard where the weights of Catboost, LightGBM, and neural networks are 0.6, 0.3, and 0.1, respectively.
In this section, we discuss the experiment settings including hyperparameters selected for each model and the performance of each model in diferent settings based on our internal validation set.
We use two boosting based models: Catboost and LightGBM where we use larger learning rate and fewer trees for LightGBM (LGB Parameters - num_leaves: 35, max_depth: 8, min_child_sample: 100, subsample: 0.7, colsample_bytree: 0.7, n_estimators: 15k, learning rate: 0.2), and smaller learning rate for Catboost (CAT Parameters - iterations: 53k, border_count: 254, max_depth: 11, learning rate: 0.03).
As mentioned in Section 3.4, we generate two types of embeddings from the following types of graphs: user-hashtag graph (UH) and user entity graph (UE), where the embedding size is 64 in both the cases. Table 2 shows the performance of Catboost model on diferent feature sets. In this setting, we would like to study the impact of graph based features on the performance of our models. Without using any graph features, Catboost (CAT) has the error of 0.12494. By using normal graph features such as UH and UE, we can improve the model 0.0015. It means that adding the graph embeddings improves the performance of the models.
Table 1 shows the performance improvement using k-fold and common model strategy with the baseline models. It shows that all our base models improves by using these strategies. Further Table 3 shows the performance of various ensemble approaches, it shows that ensemble of all four models yields the best performance. We use the weighted average approach to combined our predictive models. Please note that we don’t have all results for every model. We only choose important models to submit and get the score.
Table 2: Graph Features on Internal Validation Model + Embedding
Validation Score
CAT 0.12494
CAT + UH 0.12464
CAT + UE 0.12343
As we only know the score after submitting our submission, this table only shows part of all results. Note that we don’t submit ∗∗ + ∗ ensemble in the validation phase so we don’t have its score. Based on the results, one can see that adding more models can improve the accuracy. For instance, ∗∗ + ∗ + ∗ ensemble works better than ∗ + ∗ and out final ensemble consists of 4 algorithms. ∗∗ + ∗ ∗∗ + ∗ ∗∗ + ∗ + ∗ ∗∗ + ∗ + ∗ + ∗
Note that ∗ indicates base model + k-fold and ∗∗: base + fold + common model.
We have presented our solution for COVID-19 retweet prediction which consists of feature engineering and bagging/down-sampling modelling techniques. The proposed approach is ranked and on the public and private test sets, respectively. With an exhausted feature set and powerful modeling techniques, we hope that our solution provides a solid baseline for retweet prediction research, especially in the crisis time like COVID-19 pandemic.