Emodynamics: Detecting and Characterizing Pandemic Sentiment Change Points on Danish Twitter

Abstract

In this paper, we present the results of an initial experiment using emotion classi昀椀cations as the basis for studying information dynamics in social media ('emodynamics'). To do this, we used Bert Emoti1o8n] [ to assign probability scores for eight di昀erent emotions to each text in a time series of 43 million Danish tweets from 2019-2022. We 昀椀nd that variance in the information signals novelty and resonance reliably identify seasonal shi昀琀s in posting behavior, particularly around the Christmas holiday season, whereas variance in the distribution of emotion scores corresponds to more local events such as major in昀氀ection points in the Covid-19 pandemic in Denmark. This work in progress suggests that emotion scores are a useful tool for diagnosing shi昀琀s in the baseline information state of social media platforms such as Twitter, and for understanding how social media systems respond to both predictable and unexpected external events.

Keywords

• eol>Change point detection
• Information theory
• Social media
• Covid-19

1. Introduction

The Covid-19 pandemic saw unprecedented activity on social media, as people’s social networks became limited to the virtual sphere. During this time, Twitter saw record user activity on its platform, including in Denmark where Tweet activity spiked during the 昀椀rst lockdown period (March-April 2020) and has remained high since (Figu1r)e. In contrast to other social media platforms, engagement on Twitter is primarily driven by informational needs and desire to engage with and react to news in real time8[ , 6, 17 ]. From the perspective of cultural dynamics, the COVID-19 pandemic provides a natural experiment that allows us to study the e昀ect of a global catastrophe on the informational and emotional dynamics of social media, at some level re昀氀ecting the a昀ective experience of a wide socio-cultural and political user spectrum. As such, social media content during the pandemic functions as a proxy for how cultural information systems respond to unexpected external events. We explore the e昀ect of Covid-19 on the dynamics of Danish Twitter by using methods derived from prior work on information dynamics which apply windowed relative entropy to unstructured texts in a time series. Speci昀椀cally, we extract information signals of novelty and resonance from 2019 to the present based on emotion classi昀椀cations of the content of Danish language tweets using BERT Emotion across eight categories [ 18 ], a method chosen to re昀氀ect the more a昀ective and emotion-driven nature of short-format social media texts23[ , 2 ].

1.1. Information dynamics

In line with developments in information theory, recent studies have used informationtheoretic models to track the states and dynamics of socio-cultural systems as re昀氀ected in lexical data 9[ , 1, 5, 16, 10 ]. Both Shannon entropy and relative entropy have been used to detect changes in prevalent mental states due to the socio-cultural context (e.g., state censorship, degree of recognition, religious observation16),[ 12 ]. One speci昀椀c information-theoretic approach applies windowed relative entropy to dense low-dimensional text representations to generate signals that capture informationnovelty as a reliable content di昀erence from the past and resonance as the degree to which future information conforms to said novel1ty,1[0]. Taking a more dynamic perspective on this approach, one recent study has shown that discussion boards on social media where the novelty signal displays both short-range correlations only and a particularly strong association with resonance are more likely to contain trending content [ 11 ]. Using the same approach, but combined with event detection, has also been shown to reliably predict major change points in historical da2t5a].[

Previously, information dynamics in newspapers during the 昀椀rst phase of COVID-19 have been used to examine national response strategies to the pandemic in Denmark and Sweden [ 14 ]. A peculiar behavior could be observed in news media when the 昀椀rst wave of COVID-19 virus spread across the world. In response to the pandemic event, the ordinary rate of change in news content was disrupted because nearly every story became associated with COVID-19. On the one hand, content novelty went down, because nearly every story became more similar to previous stories (i.e., news suddenly became ‘Corona news’), but on the other hand, the COVID-19 association became more prevalent, resulting in, at least initially, an increase in content persistence. A recent study, 1[ 3 ] argues that this behavior is an example of thneews information decoupling (NID) principle, according to which information dynamics of news media are (initially) decoupled by temporally extended catastrophes such that the content novelty decreases as media focus monotonically on the catastrophic event, but the resonant property of said content increases as its continued relevance propagate throughout the news information system. The same study further indicated that NID can be used to detect signi昀椀cant change in news media that originate in catastrophic events. We wish to explore whether the emotional novelty and resonance shows a similar dynamics on social media during the same period.

1.2. From infodynamics to emodynamics

Prior studies of information dynamics in media have used lexical co-occurrence as the basis for extracting information signals. This paper investigates whether the emotional character of social media texts can similarly capture event-related in昀氀ection points using windowed relative entropy. There are two motivating factors behind this choice. First, sentiment analysis and emotional classi昀椀cation are commonly used methods to characterize di昀erent states on social media 2[ 3 ], due to the more pronounced emotional valence found in social media texts compared to e.g. newspaper articles2[ 6, 2 ]. Second, LDA topic models—commonly used as the latent variables in measuring the information dynamics of texts—are di昀케cult to apply to ultrashort texts such as tweets, without aggregating individual tweets into larger chunks or threads [ 19 ]. Finally, representing texts as probability distributions over emotional categories allows us to measure the emotional and a昀ective impact of events in the social media sphere—that is, how users are reacting to and feeling about news and events in real time.

2. Methods

The dataset consists of 43,555,069 million Danish tweets (excluding retweets) from January 1 2019-August 16 2022 collected using the Twitter API V2 (academic track) (see Fig1.) . These tweets were queried using the most common Danish, Swedish and Norwegian words from the Opensubtitles word frequency lists containing 50.000 words per langua7g]e ([see Appendix A.1). The word lists were adjusted to remove words non-speci昀椀c to Scandinavian languages, and the 100 highest frequency unique words from each list were then combined and used as queries. The Danish subset of the collection was then extracted using Twitter’s native language classi昀椀er (which was found to be slightly more accurate than any of the language detection libraries for Python we compared against). Note that this sampling method, being based on Danish language queries, will not include data from multilingual Danes or non-Danish-speaking expats and immigrants, and therefore gives an extensive but not comprehensive representation of the daily discourse on the Danish Twittersphere.

2.1. Emotion classification

To obtain emotion classi昀椀cations of the tweets, we used the Danish BERT Emotion model1[ 8 ]. This model includes eight emotion categories (see Tab1l)eand outputs a predictive distribution over the emotion categories. Each tweet was then assigned a probability distribution across the eight emotion categories (Table1) using DaNLP’s pretrained BERT emotion models, 昀椀netuned on Danmarks Radio (DR) Facebook data using the Transformers library from HuggingFace, and based on pretrained Danish BERT representations by BotXO. The model classifying amongst eight emotions achieves an accuracy on 0.65 and a macro-f1 on 0.64 on the social media test set from DR’s Facebook dataset containing 999 examples. By running the Danish Bert Emotion model on our twitter dataset, we generate a predictive emotion distribution for each tweet which serves as the document representation for the extraction of information signals.

We summarize these probability distributions by averaging the probability of each emotion over one day, thus giving us a mean daily probability for each emotion. Fig2ursehows mean daily emotion distributions as time series signals.

2.2. Windowed relative entropy

The summarized daily probability distributions of emotion scores are then used to generate signals that capture informationovelty as a reliable content di昀erence from the past andresonance as the degree to which future information conforms to said novelt1y,, 1[0], with the latent variables being the emotion distribution.

We used Jensen-Shannon divergence (JSD) to quantify the amount of surprise between two probability distributions. The advantage of JSD over the closely related Kullback-Leibler divergence (KLD) is that it is symmetrical and smoothed, making it a distance metric20[]. JSD is calculated as Here,Ā(Ā) is the probability distribution at thje’th day (and similarly foĀr(ā)), = 1 (Ā(Ā)+ Ā(ā)), 2 and is KLD, which is de昀椀ned as corresponds to the number of labels in the probability distribuĀti(oĀ)n.

The emotion probability distributions of the BERT models were used as latent variables for the information dynamics measuresnovelty, transience, and resonance. These measures were calculated following previous de昀椀nitions1,[ 15 ]. Novelty of thej’th distribution was calculated as Here, w is the window size. Novelty of the probability distribution of a given day is thus the mean of the entropy between that distribution and thwe previous distributions.

Similarly, transience for thjeth distribution was calculated as ý (Ā) = 1 ý

∑ þ (Ā(Ā)|Ā(Ā− )) ý =1 ÿý (Ā) = 1 ý

∑ þ (Ā(Ā)|Ā(Ā+ )) ý =1 Transience of the probability distribution of a given day is thus the mean of entropy between that distribution and thew subsequent distributions. Finally, resonance was calculated as ýý (Ā) = ý (Ā) − ÿý (Ā)

2.3. Nonlinear Adaptive Filtering

Nonlinear adaptive 昀椀ltering is applied to the information signals because of the their inherent noisiness, [ 4 ]. First, the signal is partitioned into segments (or windows) of lengýth= 2 + 1 points, where neighboring segments overlap by+ 1. The time scale is + 1 points, which ensures symmetry. Then, for each segment, a polynomial of orderis 昀椀tted. Note that = 0 means a piece-wise constant, and = 1 a linear 昀椀t. The 昀椀tted polynomial for ÿāℎand (ÿ + 1)āℎ is denoted asÿ(ÿ)(Ă1), ÿ(ÿ+1)(Ă2), where Ă1, Ă2 = 1, 2, ..., 2 + 1. Note the length of the last segment may be shorter thaný . We use the following weights for the overlap of two segments. ÿ( )(Ă1) = ý1ÿ(ÿ)(Ă + ) + ý2ÿ(ÿ)(Ă), Ă = 1, 2, … , + 1 (1) where ý1 = (1 − Ă−1 ), ý2 = 1 − ý1 can be written as(1 − Ā ), Ā = 1, 2, where Ā denotes the distance between the point of overlapping segments and the center oÿf(ÿ), ÿ (ÿ+1). The weights decrease linearly with the distance between point and center of the segment. This ensures that the 昀椀lter is continuous everywhere, which ensures that non-boundary points are smooth.

A window of three days ý( = 3) was chosen for the analysis, meaning that resonance for each day was calculated relative to three previous and three following days. The chosen window can be thought of as deciding the granularity of the analysis. The longer the window, the less 昀椀ne-grained the analysis. In newspapers, a typical cycle is seven days1[ 5 ], but dynamics change much more quickly on social media21[]. By setting a window of three days, we can capture the main 昀氀uctuations in emotional dynamics related to external events.

2.4. Change Point Detection

The search method Pruned Exact Linear Time (PELT) was used to identify the change points. This method not only 昀椀nds the relevant change points but also determines the number of change points. By using a linear penalization on the number of change points, the PELT algorithm identi昀椀es the number of change points while aiming to minimize over昀椀tting [ 24 ]. PELT is an optimal search method, meaning that it is guaranteed to 昀椀nd the optimal segmentation of the signal given the cost function and penalization24[].

We used the radial basis function (rbf) as the cost function, which is a cost function based on a Gaussian kernel. The kerneāl for rbf is de昀椀ned as ā(þ, ÿ) = exp(− ‖þ − ÿ‖2) (2) Here ‖ ⋅ ‖ is the Euclidian norm and > 0 is the bandwidth parameter which is de昀椀ned as the inverse of the median of all pairwise distance2s4[]. When 昀椀tting the model, we used a smoothing parameter = 4 . A low would result in an increased segmentation of the signal while a higher value for would make the algorithm disregard more change points. Thus,

× ý slopes for each change point (CP) period together with the 95% confidence setting this parameter can be thought of as a trade-o昀 between complexity and goodness-of-昀椀t for the model. The model was 昀椀tted using theruptures python package [ 24 ]. 2.4.1. Resonance-novelty coupling To describe the changes in the signal between the di昀erent change point periods, we investigated the coupling between resonance and novelty followin1g5][. This was implemented as a linear regression model predicting resonance from novelty within each of the change point periods, ýÿ = 0 + 1 ÿ+ ÿ (3) where ýÿ and ÿ refer to resonance and novelty at thÿe’th day, respectively. 0 is the intercept, 1 the × ý slope, and ÿ the error term. To make the estimate of the × ý slope more interpretable, both resonance and novelty were z-scored before 昀椀tting the model.

3. Results

Resonance and novelty signals using windowý = 3 calculated from probability distributions of emotions from the Danish BERT Emotion model are visualizedFiingure 3. We observe clear and easily detectable tendencies in the signals: rather than decoupl1e4d],[major peaks in novelty and resonance appear to be strongly correlated and spaced at regular intervals corresponding to the Christmas/winter holiday period.

To better observe variance potentially related to Covid-19 pandemic evenFitgs,ure 4shows the unsmoothed resonance time series together with the change points periods and selected events related to COVID-19 from a timeline published by Statens Serum Institut (SS2I2)][and reproduced in Table3 in Appendix A.2. The changes in variance between the change point periods are visually apparent when inspecting the signal.

Table 2 below shows × ý slopes from the linear regression models predicting resonance from novelty, with a threshold 昀椀lter 0.01 applied to both novelty and resonance values. In change point period 1, the estimated coe昀케cient was 1 = 0.826 and was the smallest out of all of the four time periods, while the estimate of the× ý slope in the 昀椀琀h change point period was the largest with a coe昀케cient of 1 = 2.235.

The × ý slopes of the linear models are visualizedFiingure 5. Notice that the scale of both axes di昀er between change point periods. This is due to variations in the distribution of the data points, which makes visualizations using the same scales di昀케cult to interpret. The 昀椀gure shows a general positive coupling between resonance and novelty. Moreover, it can be seen that × ý slope in the holiday change point periods 3 and 5 are signi昀椀cantly steeper than all other periods.

Matrices showing the correlation between the individual emotion time series signals in each change point period are inFigure 6. The correlations between the emotions are generally strongest in change point periods 3 and 5, representing the 2020 and 2021 Christmas holidays, respectively. These periods contains very clear clusterhsa:ppiness and trust are positively correlated with each other while negatively correlated wsuitrhprise, anger, contempt, and fear. The four latter emotions are all positively correlated with each other. All of these observed correlations have a correlation coe昀케cientÿ > .5.

4. Discussion

Recall that novelty is a measure of the average amount of relative surprise between the probability distribution at a given time point and the probability distributions in a window with the ý previous time point (three days, in our case), while transience is comparing the probability distribution at that time point with thýe following probability distributions. Resonance is high if novelty is high while transience is low, meaning the documents are very surprising compared to previous documents but not the following. In other words, a stronger correlation between novelty and resonance means that as more information enters the system, more of it ”sticks” and remains relevant. Based on the slope of× ý for 2019 (change point period 1)—representing the pre-pandemic baseline—the normal state of Danish Twitter is one of high emotional entropy: new information is regularly entering the discourse, producing a novel distribution of emotional responses, but resonance is relatively weFaikgu(re 5). On an annual basis around the Christmas holiday, we see a marked shi昀琀 to a lower-entropy state where the emotion distribution is more predictable: new information 昀氀oods in, but resonance is high. This state persists only for a short time, until the holiday ends, people return to work, and the news media cycle returns to normal. Somewhat surprisingly, despite the occurrence of a major catastrophe in early 2020—the onset of the Covid-19 pandemic—our change point detection model does not distinguish this period as abnormal with respect to the baseline emotional dynamics of Twitter. This is because even as the onset of a major shock event 昀氀oods the system with new information with a high variety of di昀erent emotional reactions (novelty) the persistence of these patterns overtime is not signi昀椀cantly di昀erent from the normal baseline resonance rate; i.e. the high entropy conditions produced by even a major disaster are not so dissimilar from the normal state of a昀airs on Twitter, which is a high entropy system. Thus, change point detection based on emotions are only tuned to discern more systematic seasonal shi昀琀s from high-to-low emotional entropy.

To see the 昀椀ngerprint of Covid-19 events and other unpredictable local events in the time series, we must look to the dynamics of the individual emotion scorFeisg(ure 3). Variations in the dynamics of the di昀erent signals can be visually detected, e.g. the pandemic period in Denmark, starting in early 2020, has been marked by an overall shi昀琀 towardcosntempt being the predominant emotion on Twitter, and a gradual drop-o昀 oexfpectation, whilegrief appears the most stationary as well as the least likely emotion. The 昀椀gure also depicts sudden changes in values for some of the emotions, for example, the marked spikesfienar in late FebruaryMarch of 2020 and December 2021-February 2022, corresponding to the period of Denmark’s 椀昀rst lockdown and the surge of infection due to the omicron variant, respectively (cf. the timeline in Table3). The same period also shows a decrease itnrust, which has not recovered to baseline as of August 2022.

In our continuing work, we will experiment with di昀erent parameters and change point detection methods which might show higher sensitivity to micro-disruptions of the novelty and resonance signal triggered by external events. We will also experiment with coupling emotion distributions with other representations of document content in extracting information signals.

A. Appendix 1

A.1. Scraping keywords

Below follows the list of top words from Danish, Swedish and Norwegian used to scrape Twitter: a昀琀en, aldrig, alltid, altid, andet, arbejde, bedste, behöver, behøver, beklager, berätta, betyr, blev, blevet, blir, blitt, blive, bliver, bruge, burde, bättre, båe, bør, deim, deires, ditt, drar, drepe, dykk, dykkar, där, död, döda, død, døde, e昀琀er, elsker, endnu, faen, fandt, feil, 昀椀kk, 昀椀nner, 昀氀ere, forstår, fortelle, fortfarande, fortsatt, fortaelle, från, få, fået, får, fått, förlåt, första, försöker, før, først, første, gick, gikk, gillar, gjennom, gjerne, gjorde, gjort, gjør, gjøre, godt, gå, gång, går, göra, gør, gøre, hadde, hallå, havde, hedder, helt, helvete, hende, hendes, hennes, herregud, hjelp, hjelpe, hjem, hjälp, hjå, hjaelp, hjaelpe, honom, hossen, hvem, hvis, hvordan, hvorfor, händer, här, håll, håller, hør, høre, hører, igjen, ikkje, ingenting, inkje, inte, intet, jeres, jävla, kanske, kanskje, kender, kjenner, korleis, kvarhelst, kveld, kven, kvifor, känner, ledsen, lenger, lidt, livet, längre, låt, låter, laenge, meget, menar, mycket, mykje, må, måde, många, mår, måske, måste, måtte, navn, nogen, noget, nogle, noko, nokon, nokor, nokre, någon, något, några, nån, når, nåt, nødt, också, også, pengar, penger, pratar, prøver, på, redan, rundt, rätt, sagde, saker, samma, sammen, selv, selvfølgelig, sidan, sidste, siger, sikker, sikkert, själv, skete, skjedde, skjer, skulle, sluta, slutt, snakke, snakker, snill, snälla, somt, stadig, stanna, sted, står, synes, säger, sätt, så, sådan, såg, sånn, tager, tiden, tilbage, tilbake, tillbaka, titta, trenger, trodde, troede, tror, två, tycker, tänker, uden, undskyld, unnskyld, ursäkta, uten, varför, varit, varte, veldig, venner, verkligen, vidste, vilken, virkelig, visste, väg, väl, väldigt, vän, vår, våra, våre, vaek, vaer, vaere, vaeret, älskar, åh, år, åt, över.

Description The outbreak of the virus is declared a threat to global health by WHO.

The first Danish citizen is tested positive for COVID-19.

The Danish prime minister has her second press conference and she announces a two-week lockdown in Denmark. All schools, daycares and institutions are closing. Assembly ban for more than 100 people is introduced. Public employees with a non-critical functionality are sent home.

The Queen of Denmark speaks to the public about the COVID-19 crisis.

Partial reopening. Driving schools, hair dressers, research laboratories, and certain other liberal professions together with youngest grade levels and outdoor sport activities without body contact is allowed to reopen.

Danish health authority recommend wearing face masks in public transportations if there are many people.

The government decides to put down all mink on Danish mink farms due to an outbreak of a COVID-19 mutation.

The first Danish citizens are vaccinated using the Pfizer/BioNTech vaccine.

The lockdown in Denmark, which was introduced in December 2020, is prolonged until February 28, 2021.

The AstraZeneca vaccine is withdrawn completely from the Danish vaccination program.

The Danish coronapas app can now be downloaded in the App store.

The Delta variant now dominates in Denmark. Before this, the alpha variant dominated.

Covid-19 is no longer described as a socially critical disease in Denmark.

Covid-19 is again a disease critical to society in Denmark.

The restrictions are li昀琀ed and covid-19 is changed to no longer be a critical illness. Requirements for tests upon entry to Denmark are retained.

The rapid test centers close.

References

[1] A. T. J. Barron , J. Huang , R. L. Spang , and S. DeDeo. “ Individuals, institutions, and innovation in the debates of the French Revolution” . InP:roceedings of the National Academy of Sciences 115.18 ( 2018 ), pp. 4607 - 4612 . doi: 10 .1073/pnas.1717729115. url: http://ww w.pnas.org/lookup/doi/10.1073/pnas.171772911.5

[2] C. Duncombe . “ The Politics of Twitter: Emotions and the Power of Social Media” . In: International Political Sociology 13.4 ( 2019 ), pp. 409 - 429 . doi: 10 . 1093 / ips / olz013. url: https://doi.org/10.1093/ips/olz01.3

[3] J. Gao , J. Hu , and W.-w. Tung. “ Facilitating Joint Chaos and Fractal Analysis of Biosignals through Nonlinear Adaptive Filtering” . IPnl:os One 6.9 ( 2011 ), e24331 . doi: 10 .1371/jour nal. pone.0024331 . url: https://journals.plos.org/plosone/article?id= 10 .1371/journal.pon e. 0024331 .

[4] K. L. Gray . “ Comparison of Trend Detection Methods” . InG:raduate Student Theses, Dissertations, & Professional Papers 228 ( 2007 ), p. 98 .

[5] J. Guldi . “ The Measures of Modernity: The New Quantitative Metrics of Historical Change Over Time and Their Critical Interpretation” . IInn:ternational Journal for History, Culture and Modernity 7 .1 ( 2019 ), pp. 899 - 939 . doi: 10 .18352/hcm.589.

[6] P. R. Johnson and S. Yang . “ Uses and grati昀椀cations of Twitter: An examination of user motives and satisfaction of Twitter use”. InC:ommunication Technology Division of the annual convention of the Association for Education in Journalism and Mass Communication in Boston, MA. Vol. 54 . 2009 .

[7] P. Lison and J. Tiedemann . “Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles” . In: ( 2016 ).

[8] A. Masciantonio , D. Bourguignon , P. Bouchat , M. Balty , and B. Rimé . “ Don't put all social network sites in one basket: Facebook, Instagram, Twitter, TikTok, and their relations with well-being during the COVID-19 pandemic” . InP:loS one 16.3 ( 2021 ), e0248384 .

[9] J. Murdock , C. Allen , and S. DeDeo. “ Exploration and Exploitation of Victorian Science in Darwin's Reading Notebooks” . Ina:rXiv:1509.07175 ( 2015 ).

[10] D. Nguyen , M. Liakata , S. DeDeo, J. Eisenstein , D. Mimno , R. Tromble , and J. Winters .

“How we do things with words: Analyzing text as social and cultural data” . FIrno:ntiers in Arti昀椀cial Intelligence 3 ( 2020 ), p. 62 . doi: 10 .3389/frai. 2020 . 00062 .

[11] K. Nielbo , P. Vahlstrup , and A. Bechmann . “ Trend Reservoir Detection: Minimal Persistence and Resonant Behavior of Trends in Social Media” . IPnr:oceedings of Computational Humanities Research 1 ( 2021 ).

[12] K. L. Nielbo , K. F. Baunvig , B. Liu , and J. Gao . “ A curious case of entropic decay: Persistent complexity in textual cultural heritage” . IDni:gital Scholarship in the Humanities 34.3 ( 2019 ). doi: 10 .1093/llc/fqy05 4.

[13] K. L. Nielbo , R. B. Baglini , P. B. Vahlstrup , K. C. Enevoldsen , A. Bechmann , and A. Roepstor昀. “News Information Decoupling: An Information Signature of Catastrophes in Legacy News Media” . In: arXiv: 2101 .02956 [cs] ( 2021 ).

[14] K. L. Nielbo , K. Enevoldsen , R. Baglini , E. Fano, and J. Gao . “ Pandemic news information uncertainty - News dynamics mirror di昀erential response strategies to COVID-19” . In: ( 2021 ), p. 16 .

[15] K. L. Nielbo , F. Haestrup , K. C. Enevoldsen , P. B. Vahlstrup , R. B. Baglini , and A. Roepstor昀. “ When no news is bad news - Detection of negative events from news media content” . In: arXiv:2102 .06505 [cs] ( 2021 ). url: http://arxiv.org/abs/2102.06505.

[16] K. L. Nielbo , M. L. Perner , C. P. Larsen , J. Nielsen , and D. Laursen . “ Automated Compositional Change Detection in Saxo Grammaticus' Gesta Danorum” . InD:hn. 2019 , pp. 320 - 332 .

[17] C. S. Park . “ Does Twitter motivate involvement in politics? Tweeting, opinion leadership, and political engagement” . InC:omputers in human behavior 29.4 ( 2013 ), pp. 1641 - 1648 .

[18] A. B. Pauli , M. Barrett , O. Lacroix , and R. Hvingelby . “ DaNLP: An open-source toolkit for Danish Natural Language Processing” . InP:roceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa) . Reykjavik, Iceland (Online): Linköping University Electronic Press, Sweden, 2021 , pp. 460 - 466 . urlh:ttps://aclanthology.org/ 2021 .nodalida-main. 53 .

[19] J. Qiang , Z. Qian , Y. Li , Y. Yuan , and X. Wu . “ Short text topic modeling techniques, applications, and performance: a survey” . InIE:EE Transactions on Knowledge and Data Engineering 34 .3 ( 2020 ), pp. 1427 - 1445 .

[20] T. R. d . Reis. Measuring the statistical similarity between two samples using JensenShannon and Kullback-Leibler . 2020 . url: https://tinyurl.com/34jhn7xd.

[21] Z. Saeed , R. A. Abbasi , O. Maqbool , A. Sadaf , I. Razzak , A. Daud , N. R. Aljohani , and G. Xu. “ What's Happening Around the World? A Survey and Framework on Event Detection Techniques on Twitter” . In:Journal of Grid Computing 17.2 ( 2019 ), pp. 279 - 312 . doi: 10 .1007/s10723-019-09482-2. url: http://link.springer.com/10.1007/s10723-019- 09482 -. 2

[22] Statens Serum Institut . Tidslinje for covid-19 . 2022 . url: https://www.ssi.dk/-/media/ark iv/subsites/covid19/presse/tidslinje-over-covid-19/covid-19 - tidslinje-for-2020 -2022-la ng-version-- - version-1 --- april-2022.pdf?la=.da

[23] M. Thelwall , K. Buckley , and G. Paltoglou. “ Sentiment in Twitter events” . JIonu: rnal of the American Society for Information Science and Technology 62.2 ( 2011 ), pp. 406 - 418 . doi: 10 .1002/asi.21462. url: https://onlinelibrary.wiley.com/doi/abs/10.1002/asi.214.62

[24] C. Truong , L. Oudre , and N. Vayatis . “ Selective review of o昀툀ine change point detection methods” . In: Signal Processing 167 ( 2020 ), p. 107299 . doi: 10 .1016/j.sigpro. 2019 . 107299 . url: https://linkinghub.elsevier.com/retrieve/pii/S01651684193034.94

[25] E. Vrangbaek and K. Nielbo . “Composition and Change in De Civitate Dei: A Case Study of Computationally Assisted Methods” . InS:tudia Patristica ( 2021 ).

[26] K. Welbers and M. Opgenha昀en . “ Presenting News on Social Media” . In:Digital Journalism 7.1 ( 2019 ), pp. 45 - 62 . doi: 10 .1080/21670811. 2018 . 1493939 . url: https://doi.org/10.10 80/21670811. 2018 . 1493939 .