The COVID-19 pandemic is having a serious adverse impact on the lives of people across the world. COVID-19 has exacerbated community-wide depression, and has led to increased drug abuse brought about by isolation of individuals as a result of lockdown. Further, apart from providing informative content to the public, the incessant media coverage of COVID-19 crisis in terms of news broadcasts, published articles and sharing of information on social media have had the undesired snowballing efect on stress levels (further elevating depression and drug use) due to uncertain future. In this position paper, we propose a novel framework for assessing the spatio-temporal-thematic progression of depression, drug abuse, and informativeness of the underlying news content across the diferent states in the United States. Our framework employs an attention-based transfer learning technique to apply knowledge learned on a social media domain to a target domain of media exposure. To extract news articles that are related to COVID-19 communications from the streaming news content on the web, we use neural semantic parsing, and background knowledge bases in a sequence of steps called semantic filtering. We achieve promising preliminary results on three variations of Bidirectional Encoder Representations from Transformers (BERT) model. We compare our findings against a report from Mental Health America and the results show that our fine-tuned BERT models perform better than vanilla BERT. Our study can benefit epidemiologists by ofering actionable insights on COVID-19 and its regional impact. Further, our solution can be integrated into end-user applications to tailor news for users based on their emotional tone measured on the scale of depressiveness, drug abusiveness, and informativeness.
1
COVID-19 pandemic has changed our societal dynamics in diferent
ways due to the varying impact of the news articles and broadcasts
on a diverse population in the society. Thus, it is important to
place the news articles in their spatio-temporal-thematic
The impact of COVID-19 on mental health has been investigated
in recent studies
We propose an approach to measure depressiveness, drug
abusiveness, and informativeness as a result of media exposure for
various states in the US in the months from January 2020 to March
2020. Our study is focused on the first quarter of 2020 as this period
was critical in the spread of COVID-19 and its ominous impact;
this was a period when the public faced major changes to lifestyle
including lockdown, social distancing, closure of businesses,
unemployment, and broadly speaking, complete lack of control over the
unfolding situation precipitating in severe uncertainty about the
impending future. In consequence, this continued media exposure
progressively worsened the mental health of individuals across the
board. We analyze and score news content on three orthogonal
dimensions: spatial, temporal, and thematic. For spatial, we use
state boundaries. For temporal, we use monthly data analysis. For
thematic, we score news content on the category/dimension of
depression, drug abuse and informativeness (relevant to COVID-19
but not directly connected to either depression or drug-abuse).
Our study hinges on the use of domain-specific language
modeling and transfer learning to better understand how depressiveness,
drug abusiveness, and informativeness of news articles evolve in
response to media exposure by people. We conduct the transfer
of knowledge learned on a social media platform to the domain
of exposure to news using variations of the attention-based BERT
model
Having determined the scores for depressiveness, drug abusiveness, and informativeness of news articles for each state during the three months, we computed the aggregate score for each thematic category by summing up the scores for the news articles. We ifnally assigned the category with the highest score as a label for a state. For instance, if the aggregate score of depressiveness for the state of Iowa in the month of January 2020 is the highest of the three thematic categories, then the state of Iowa is assigned a label of depression for that month, which means the state of Iowa is most exposed to depressive news contents. Thus, identifying which states are consistently exposed to depressive or drug abusive news contents enables policy makers and epidemiologists to devise appropriate intervention strategies. 2
We collected 1.2 Million news articles from the Web and GDELT1 (a
resource that stores world news on significant events from diferent
countries) using semantic filtering
For this task, we explored 780 DBpedia categories that are
relevant to COVID-19 communications to create the most relevant
set of entities and news articles. Further, upon inspection of the
news articles, we discovered medical terms that were not available
in DBpedia. As a result, we used (“iii”) the MeSH terms hierarchy
in Unified Medical Language System (UMLS), the Diagnostic and
Statistical Manual for Mental Disorders (DSM-5) lexicon
We propose to use three variations of the BERT model for
representing news articles. In its basic form, we use vanilla BERT for encoding
news articles. For the remaining two variations, we fine-tune BERT
on a binary sequence classification task by independently training
on two corpora using masked language modeling (MLM) and next
sentence prediction (NSP) objectives. The two corpora used are: 1)
Subreddit Depression
In addition to using BERT for encoding news contents, we also use it for representing the entities in the background knowledge bases (i.e., PHQ-9, DAO, and DBpedia). Once we have encoded the news articles and the entities in the knowledge bases using vanilla BERT or fine-tuned BERT model, we generated depressiveness score, drug abusiveness score, and informativeness score corresponding to the entities in PHQ-9, DAO, and DBpedia respectively. The equation below gives the score of a news article for a category given one of the BERT models: 1
|EÕ | |E | =1 ( ) = (news, ) (1) where, m ∈ {vanilla-BERT, DPR-BERT, DA-BERT} c ∈ {informativeness, depressiveness, drug abuse} cossim (news, e): cosine similarity between a news content and an entity in KB
KB - a collection of entities present in PHQ-9, DBpedia, or DAO We used the base variant of the BERT model with 12 layers, 768 hidden units, and 12 attention heads. We use PyTorch 1.5.0+cu101 for fine-tuning our BERT models. All our programs were run on Google Colab’s NVIDIA Tesla P100 PCI-E GPU. 4
In this section, we report the state-wise labels (i.e., depressive, drug abusive, informative) for each month obtained after summing the scores of news articles as described. The category with the highest cumulative score is set as the label for a state.
Using vanilla-BERT (Figure 5), we can see that no state shows exposure to news content on drug abuse in January. Going from February to March, we see depressive news content move from inner-most states such as Missouri, Kansas, and Colorado to border states such as California, Montana, North Dakota, and Louisiana, making way for informative news content. Further, there are fewer states exposed to drug-related news content than those exposed to depressive or informative news content in February or March. Particularly, Arizona and Virginia show consistent exposure to drug-related news content in February and March.
Using depression-BERT, as shown in Figure 6, we see that states such as Texas, and Kansas are exposed to depressive news content for the month of January and February while states such as California, Montana, Alaska, and Michigan show higher consumption of depressive news content in February and March. With regard to informativeness, we see an overall even distribution of informative news content across the nation in February and March. Further, we see a few midwest states showing relatively higher instances of news content that are informative than depressive in February and March. It’s interesting to see a few southern states such as Oklahoma, Texas, and Arkansas transition from exposure to depressive news content in the month of February to drug use related news content in the month of March.
Using Drug Abuse-BERT model (Figure 7), states such as Texas, and Wisconsin shift from exposure of depressive news content in January to exposure of drug-related news content in February, while states such as California, and Oklahoma transition from exposure to depressive news content in February to drug-related news content in March. Further, we see the informativeness of news content sweeping from the east to the midwest, to parts of the south, and to some parts of the west from February to March.
Our results show that a fine-tuned BERT model cleanly separates the thematic categorical scores to a state. For instance, using DABERT for the month of March, the drug abuse score for the state of California is much higher than the score of depressiveness or informativeness for the same state. However, with the vanilla BERT model, the three scores computed for the various states and months are marginally diferent. Moreover, the results using DPR-BERT or DA-BERT capture the state-level ranking of mental disorders by Mental Health America 3 better than vanilla-BERT; for a few states, the fine-tuned BERT models identify more months to have media exposure to depression or drug abuse news content.
As indicated in Table 1, we report months showing predominant media exposure to either depressive or drug abuse news articles using the three variants of BERT model. We use 10 of the 13 states recognized as showing high prevalence of mental disorders according to a report by Mental Health America on overall mental disorder ranking. The 3 states not included in this table are Washington, Wyoming, and Idaho. We did not consider these 3 states as these states were not in our dataset cohort. For the Mental Health America (MHA) report, we make a practical assumption that each of the three months is either depressive or drug abusive for each state. Thus, our objective is to maximize the number of months with 3https://www.mhanational.org/issues/ranking-states exposure to depressive/drug abuse news content for each of the 10 states. We can see in Table 1 that fine-tuned BERT models help identify more months to having exposure to depressive or drug abuse news content than vanilla BERT does for the 10 states. For example, using DA-BERT, five states are identified to have at least two months showing exposure to depressive/drug abuse news content while DPR-BERT identifies six states to having been exposed to depressive/drug abuse news content for two months. On the other hand, vanilla-BERT identifies only two states with depressive/drug abuse news content for two months. To compare models with one another and against the report by Mental Health America (MHA), we compute a Jaccard Index between each pair of models and each model against the report from MHA. The equation below computes Jaccard similarity between the results of two models or a model’s results with an MHA report.
(1, 2) = Õ| | 1 ∩ 2 ∈ 1 ∪ 2 (2) where, 1, 2 ∈ {vanilla-BERT, DPR-BERT, DA-BERT, MHA} - Set of States in the US (Table 1) 1 , 2 : Number of depressive, drug abusive, or informative months for a state “i”
We report inter-model and model-to-MHA Jaccard similarity scores computed using equation (2) in Figure 8.
As shown in Figure 8, DA-BERT gives the best results against MHA report in Jaccard similarity (0.53), which means DA-BERT identifies over half of the state-to-month instances in MHA. On the other hand, vanilla-BERT has a Jaccard similarity of 0.37 with MHA, which can be interpreted as vanilla-BERT identifies a little over one-third of the state-to-month instances in MHA. The best Jaccard similarity is achieved between DPR-BERT and vanilla-BERT (0.7); thus, 70% of state-to-month mappings are shared between DPRBERT and vanilla-BERT based on Jaccard index. It’s interesting to see DA-BERT has the same Jaccard similarity with vanilla-BERT MHA States with high DPR and DA and DPR-BERT, subsuming the former and being subsumed by the latter in terms of depressive/drug abusive months. 5
In this paper, we model depressiveness, drug abusiveness, and
informativeness of news articles to assess the dominant category
characterizing each US state during each of the three months ( Jan
2020 to Mar 2020). We demonstrate the power of transfer learning
by fine-tuning an attention-based deep learning model on a
different domain and use the domain-tuned model for gleaning the
nature of media exposure. Specifically, we use background
knowledge bases for measuring depressiveness, drug abusiveness, and
informativeness of news articles. We found out DA-BERT identifies
the most number of state-to-month instances as being exposed
to depressive or drug abuse news content according to the report
from Mental Health America. In the future, we plan to incorporate
background knowledge bases in our attention-based transfer
learning framework to further investigate knowledge-infused learning