This paper evaluates the ability to predict COVID-19 caseloads in local areas using the text of geographically specific subreddits, in conjunction with other features. The problem is constructed as a binary classification task on whether the caseload change exceeds a threshold or not. We find that including Reddit features, alongside other informative resources, improves the models' performance in predicting COVID-19 cases. On top of this, we show that exclusive use of Reddit features can act as a strong alternative data source for predicting a short-term rise in caseload due to its strong performance and the fact that it is readily available and updates instantaneously.
A growing literature looks into the predictive
power of social media
To determine how well Reddit comments can predict future COVID-19 caseloads, this paper adopts the pipeline of Hofmann et al. (2020), a statistical NLP study using Reddit data in a very different application area (predicting the creation of new complex words). The pipeline uses a sliding window over the data stream, with each interval serving as the training data to predict the outcome in the subsequent window. Our model buckets every comment on the local subreddit into a set of daily documents F and selects a set W of important words using the inclusion criteria outlined in Section 2. The TF-IDF, T , of word w in the kth document of F is calculated:
Tw;F (k) = tfw;F (k) log( jF j ) dfw (1)
Where tfw;F (k) is the number of occurrences of w in F (k), jF j is the number of documents in F and dfw is the number of documents that contain w. This statistic provides a good method for comparing how over-represented a word is in each document. Once the TF-IDF is calculated for all of W , we take the 7-day moving average (7-MA) of TF-IDF. The 7-MA is used throughout this study because of fluctuations in language usage according to the day of the week and because the caseload reports have artefacts from the day of the week. This time-series data is then tabulated and combined with other relevant datasets in Section 2 to determine which features are important for our prediction task.
The data that we seek to predict is provided by the COVID-19 Tracking Project (CTP) 1. The current caseload (CCL) is also considered as a predictor for the subsequent change in caseload. Update frequency: 24 hours. Start date: 13/01/2020
The other predictors come from three sources. Each provides data about each day, but update speed differs. They were combined into a time-series dataset using data up to 17/01/2020. Days where data was incomplete were deleted.
Tracker (OxCGRT) - The OxCGRT
Reports (GCCMR) 2 - The GCCMR provided movement data within different areas such as parks, workplaces etc. The data has a high degree of geographic specificity. The movement statistic is relative to a benchmark taken between Jan. 3rd and Feb. 6th 2020. Update frequency: 2-3 days. Start date: 15/02/2020.
Pushshift API - The Pushshift API from Baumgartner et al. (2020) was used to the compile datasets of entire target subreddits. Update frequency: real-time. The post count P is considered as a predictor. For each subreddit, we also select Feature words by finding the most over-represented words, compared to a reference corpus R. To construct R, posts were randomly selected from S, the Unix time stamp of each was taken, and the following 100 posts from the whole of Reddit were downloaded. S and R were matched for the quantity of text at each time, as illustrated in Figure 1. The term frequency ratio between R and S was calculated, and the top 50 words were selected. To avoid over-reliance on rare words, the top 50 words with the highest term frequency in the top 1000 words in S were added for a total of 100 candidate word features. A chi-square test of independence was used to trim this candidate list to the 25 feature words with the
2https://www.google.com/covid19/mobility/ most significant relationship to the target classes, and these were used in the prediction models. Appendix A lists the word features that were selected for each state. The important features are divided amongst named entities (locations, organisations, and people), technical terms, and terms referring to aspects of everyday life. The problem is presented as a classification task. For each day the features were tagged with a binary value that indicated whether the 7-MA of the caseload went up by more than a certain threshold value within the given time frame. We defined this threshold in two different ways, relative and absolute: Relative change - the change is measured as a percentage of the current value.
r(t) = (t + ) (t) (t) Absolute change - the change is measured as the absolute change in the 7-MA.
a(t) = (t + ) (t) Where (t) is the 7-MA distribution of cases, and is the time delay that refers to the time horizon of the prediction. The relative threshold value ranges from 10% to 100%, and the absolute threshold ranges from 50 to 500. The predictive window iterates from 1 to 28 days.
Once the features were tagged with a binary value, the classes were balanced by identifying which class was larger and randomly deleting posts until they were equal in size. This was done to make the analysis more interpretable by making the accuracy comparable across the different classes. The minimum number of samples required was set at 40, 20 examples of both classes. Then the features were normalised, so they were all scaled between 0 and 1. Finally, the data was passed through the different classification models in Section 4. 4
Tree-based models were used to determine the
relative importance of the different features in
predicting the changes in caseload. The performances of
the models below were compared against a
Support Vector Machine (SVM) with a linear kernel
and a Logistic Classifier (LC) to see how the more
complex models compared to classifiers with linear
decision boundaries
Random Forest (RF) - The RF model
Regularised Greedy Forest (RGF) - The RGF
model
Each data source in Section 2 is used by itself to perform the classification task and then combined together to compare the performance across the different data sources. The default subreddit used below is Washington’s r/CoronavirusWA since it has the largest subreddit by comment number.
Models (average) 7 days
RF (.828) .805 SVM (.814) .766 LC (.814) .768
RGF (.807) .766 XGBOOST (.785) .753
Despite having more sparse feature sets, neither the RGF nor XGBOOST models outperform the RF model. The SVM and LC models perform very well; however, an RF model is used for the following analysis because of its high performance and because the Gini feature importances used in sklearn.ensemble.RandomForestClassifier are highly interpretable. The time delay that delivers the most precise results for each feature is tabulated in Table 2. In Table 2 the RF model is trained only using the data from each of the data sources in the Data Source column.
Table 2 shows that for the majority of data sources, a 14 days prediction horizon yields the best results. Below, a delay of 14 days is used to compare the difference in performance using different thresholds. It also shows that Tw performs very well as a single feature class but that there is an improvement when all data sources are included.
Table 3 breaks down the performance across different thresholds for the increase. As found in many other studies, more extreme events are easier to predict. The highest performance, apart from using all the data sources, is found when only word features were used. These aspects of the performance are consistent across both the relative and absolute thresholds.
Feature importance - The importance of feature type in the RF model is tabulated in Table 4.
The individual feature importances were added up by category. Clearly, the word features constitute the most important feature type in the prediction. This section compares the performance in multiple states to see that these results are not unique for r/CoronavirusWA and that the framework is applicable within multiple regions. For this analysis, state subreddits that were comparable in size, and were culturally different from one another were erally more significant for larger subreddits. The used. The list of subreddits is shown in Table 5.
State
Subreddit Washington r/CoronavirusWA California r/CoronavirusCA
Texas r/CoronaVirusTX Florida r/FloridaCoronavirus
Start date and CCL are combined and compared against the N R features generally outperform the Tw features; however, this is not surprising since the M and G feature sets provide high-quality complementary information that should result in a higher performance than using each feature set in isolation; the M features show whether the government restrictions outlined by the G features are being listened to. The fact that there is a performance improvement when using all of the features shows that the Reddit data provides further information that is not captured in the N R features. Reddit data provides an insight into what people are talking and thinking about, which could cause people to not listen to the government restrictions.
Using only Tw yields very good results at a
7day prediction horizon relative to other variable
sources. COVID-19 has an incubation period of
performance using all features, and the Reddit
fea5-6 days
Texas Florida Average
All .805 .765 .807 .850 .807
N R .783 .722 .781 .851 .784 7 days relative threshold values. N R - all data sources other than the subreddit data; Tw - as above; Diff. = All - N R the disease more rapidly than other data sources do.
That is, Reddit provides a strong live indicator of the experience and concerns in the population at any given time. In conjunction with real-time update frequency referenced in Section 2, this makes the use of subreddit data very convincing. 6
It is clear that the content of a local subreddit is a valuable data source for predicting the COVID19 caseload in specific regions. The Tw features provided the best single feature set in almost all experimental setups, as seen in Table 3. When combined with the comparison feature sets in Table 6, the Tw features provided complementary information that resulted in a performance improvement. The results in Washington were also reproduced in other states, highlighting the robustness of the method used. A further advantage is that subreddit data is readily available. As is shown in Section 2, many of the other data sources take hours/days to update, and some only exist because the world is in a pandemic, as is the case with the GCCMR data.
There is also scope for future development using other machine learning techniques. In particular, using contextualised word embeddings has the potential to exploit the semantic relationships between words that are not well captured by a Bagof-Words approach.
This work was supported in part by a grant from the Engineering and Physical Sciences Research Council (EP/T023333/1).
XgCoRR, A.1
’bothell’, ’kirkland’, ’omak’, ’oroville’, ’seatac’, ’skagit’, ’snohomish’, ’spokane’, ’thurston’ People ’bedford’, ’culp’, ’inslee’ Organisations ’esd’, ’peuc’ Technical ’7day’, ’coronavirus’, ’health’, ’sick’, ’virus’ Other ’adjudication’, ’business’, ’news’, ’open’, ’places’, ’social’ Locations ’alameda’, ’huntington’, ’merced’, ’modesto’, ’monterey’, ’norcal’, ’solano’, ’sonoma’, ’stanislaus’, ’stockton’ People ’garcetti’ Organisations ’ihme’ Technical ’cases’, ’comorbidities’, ’sick’ Other ’aerosols’, ’californian’, ’californians’, ’certain’, ’city’, ’defying’, ’school’, ’shelterinplace’, ’state’, ’states’ ’abilene’, ’brazoria’, ’christi’, ’galveston’, ’houston’, ’frisco’, ’nueces’