Diachronic Political Content Analysis: A Comparative Study of Topics and Sentiments in Echo Chambers and Beyond

Abstract

Over the past decade, social media platforms have emerged as significant arenas for political discourse and agenda-setting. Among these platforms, Reddit stands out as a prominent site where users actively engage in discussions on controversial topics, often becoming polarized through interactions with like-minded individuals. In this paper, we delve into the realm of political leanings, seeking to understand the predominant topics of interest within echo chambers and whether they diverge from those of unpolarized users. Our primary objective is to ascertain whether echo chambers are characterized by distinct themes discussed therein. Furthermore, we employ cross-sentiment analysis to investigate potential diferences in how these themes are perceived across diferent groups.

Keywords

• eol>natural language processing
• political analysis
• social network analysis
• echo chambers

1. Introduction

The rapid growth of social media platforms and online forums has fundamentally reshaped how individuals consume information, share opinions, and engage in political discourse. The proliferation of these online networks has not only transformed the landscape of political communication but has also amplified the formation and influence of echo chambers [ 1 ]. Echo chambers are defined as environments where individuals are predominantly exposed to information that reinforces their existing beliefs through repeated exposure to like-minded individuals. This redundancy of content, along with the shared perception of it among users, leads to users’ epistemological segregation [ 2, 3 ].

This phenomenon has garnered significant scholarly interest due to its potential impact on democratic processes and public opinion [ 4, 5 ]. In fact, echo chambers have been observed to contribute to increased polarization, confirmation bias [ 6 ], and homophily in online discussions, potentially leading to a distorted perception of reality and hindering constructive debate. The political implications of echo chambers are profound, as they can exacerbate partisan divides and diminish mutual understanding among opposing political groups [ 7, 8 ].

These efects are particularly relevant in light of the recent rise of right-wing populist parties. Echo chamber efects, in fact, have been identified as influential contributors to the rise of populist movements. While the roots of populism are multifaceted, scholars have noted the facilitative role of echo chambers in disseminating specialized populist messaging outside mainstream news and party establishments [ 9, 10, 11 ]. [ 12 ] suggests that individuals, feeling besieged as claimed by populist elites, tend to gravitate towards like-minded groups. Digital media platforms foster the formation and sustenance of homogeneous networks, appealing particularly to populists with their rhetoric of division between "us" and "them." [ 13 ] highlights the correlation between populism and the proliferation of "post-truth" politics, wherein factual accuracy is sidelined in favor of personal loyalty and ideological simplicity. Echo chambers, by insulating individuals from dissenting perspectives, can exacerbate this trend, distancing adherents from objective truths.

Yet, as emphasized by [ 14 ], empirical evidence supporting a distinct correlation between of right-wing populism leaning and echo chamber dynamics remains scant, with diferent studies demonstrating lfuctuating patterns across diferent periods and nations [ 15, 16, 17, 13 ].

To address this issue, the present study examines how populist political leanings and engagement within or outside echo chambers influence the content and modes of interaction among users on the social media platform Reddit. Indeed, in this social media, users engage with one another by posting and commenting in subreddits aligned with their interests. Using topic modeling, the study investigates distinctions in topic trends among Reddit users based on their political afiliations within and beyond echo chambers. Additionally, it tracks the methods through which discussions are conducted in these diferent environments. Additionally, the study adopts a diachronic perspective, aiming at providing valuable insights into the evolution of political discourse within echo chambers, identifying shifts in predominant topics and sentiments over time. This approach not only reveals temporal changes but also, in conjunction with our politically fine-grained method, allows for a nuanced examination of how diferent political afiliations influence the nature of discussions and sentiment expressions within these chambers. In accordance with the findings of [ 18 ], which investigated the topological stability of echo chambers, this study hypothesizes that echo chambers will exhibit greater stability in how topics are perceived compared to non-echo chamber structures.

The significance of this study lies in its potential to uncover patterns and trends between "closed online environment” and political communication that may contribute to polarization. By comparing the content and sentiment across politically diverse groups, the aim is to identify whether certain topics or sentiments are more prone to echo chamber efects and how these efects difer across the political spectrum. This study contributes to the broader field of political communication and the ongoing debate about the impact of social media on democratic engagement.

The paper is organized as follows: Section 2 proposes the main contributions and the previous related works constituting the basis of our application; Section 3 introduces the dataset used in this study; Section 4 illustrates the framework, constituted of two parts: Topic Modeling and Cross-Sentiment Analysis; Section 5 reports the data analysis of our case study, reporting the main findings. Finally, Section 6, concludes the paper and provides a look ahead on future research.

2. Related Works

Echo chambers are characterized by the reinforcement of ideas, beliefs, or opinions through repeated exposure within an enclosed system, such as online communities or social media networks. The following related works in the area of topic mining in echo chambers highlight the importance of understanding the structure and dynamics of echo chambers, as well as the topics that drive their formation.

Topological approach [ 7 ] used a network-based approach to identify echo chambers on Facebook, highlighting the role of confirmation bias and homophily in their formation. Similarly, [ 19 ] studied the partisan structure inside the retweeting mechanism of political tweets by two networks. They found that the users on the opposite political sides were weakly connected. On the same research line, [ 5 ] proposed a method for identifying echo chambers on Twitter by analyzing retweet networks and user ideology. Their findings revealed the existence of polarized echo chambers in political discussions.

Content approach [ 20 ] has investigated how diferent social media platforms influence information spread and the creation of echo chambers. By analyzing over 100 million pieces of content on controversial topics from Gab, Facebook, Reddit, and Twitter, two main dynamics were examined: homophily in interaction networks and biased information difusion. Their findings highlight that homophilic clustering is a dominant online behavior, with Facebook exhibiting higher segregation in news consumption compared to Reddit.

Instead, [ 21 ] performed a sociolinguistic analysis on tweets from users within echo chambers against those from users outside the echo chamber. Their investigation entails comparative scrutiny of tweet composition, lexical preferences, and thematic emphases, aiming to elucidate potential rationales underlying the observed disparities.

Mixed approach [ 22 ] focused on diverse subreddits concerning controversial topics and reconstructed the network interaction of users. [ 22 ] defined an approach to detect echo chambers on social networks. The framework comprises four steps: (i) the identification of a controversial issue; (ii) the inference of users’ ideology on the controversy; (iii) the construction of users’ debate network; and (iv) the detection of homogeneous meso-scale communities. By modeling the diachronical network’s cohesion and users’ political leaning and interactions, they detected diferent echo chambers. Authors of [ 18 ] proposed an analysis of topological stability and topic detection of the social clusters. By relying upon sentiment analysis and exploiting the textual information coming from sources like posts and comments, the authors investigated how people discussed and perceived a controversial topic. Despite the popularity of that methodology [ 23, 24 ], [ 25 ] outlined its limitations. Indeed, the viewpoints of diverse users are categorized based on the overall sentiment they convey regarding the topic, rather than their actual alignment on various aspects defining the analyzed subject.

Textual Forma Mentis Networks [ 26, 27, 28 ] applied a new approach: textual forma mentis networks (TFMNs), namely modeling textual concepts as graph neural networks to analyze both semantic and syntactic relationships. That methodology allowed us to simultaneously focus on sentimental, emotional, and rhetorical patterns entailed in online discourses.

Furthermore, [ 29 ] suggested applying two emotional lexicons to avoid leading to drastic misinterpretation and conclusions when performing emotion analysis on texts.

3. Data

In this study, three comprehensive datasets compiled, annotated, and preliminarly analyzed in [ 22, 18 ] were used. The statistics for the datasets, covering the period from 2017 to July 2019, are presented in Table 1.

In these works, by modeling users’ posts and comments on controversial topics, the authors were able to reveal distinct ideological leanings, categorizing users as pro-Trump, neutral, or anti-Trump.

Subsequently, they introduced a framework for identifying the formation of echo chambers by leveraging both user interaction networks and users’ ideological stances. The communities were delineated using three key metrics: modularity, to detect ideologically and topologically homogeneous nodes; purity, which measures the product of the frequencies of the most common labels among its nodes; and conductance, which calculates the fraction of total edge volume pointing outside the community. In [ 20 ], network structures were estimated based on the retention of specific labels within subsets of the network where users shared a common ideology on controversial topics.

[ 18 ] focuses on the diachronic evolution of echo chamber topologies. This analysis was enhanced by linking the temporal dimension to the topics discussed, providing insights into the stability of echo chambers over time and the propensity of their members to concentrate on single controversial topics.

This work focuses on a subset of the dataset: POLITICAL SPHERE, which comprises posts retrieved from the following subreddits as illustrated in Figure 1: r/esist, r/democrats, r/MarchAgainstTrump, r/Conservative, r/Libertarian, and r/Republican. This dataset includes users’ posts categorized by political leaning and echo chamber membership (see Table 2), focusing on discussions related to U.S. politics.

4. Methodology

4.1. Topic Modeling

In statistics and natural language processing, topic modeling is a commonly used text-mining tool for uncovering hidden semantic structures within a text corpus. In this work, we have applied the BERTopic [ 30 ] topic modeling technique to extract topics from texts. BERTopic leverages transformers and c-TF-IDF to create dense clusters, facilitating the generation of easily interpretable topics while retaining key words in the topic extractions. The output of BERTopic consists of generated topics and their probabilities.

Initially, BERTopic converts documents into numerical representations by embedding text in vector space, ensuring that similar texts are positioned closely together, which can be eficiently identified using cosine similarity. To reduce the dimensionality of these representations, we employed UMAP [ 31 ], which preserves both local and global information, allowing semantically similar documents to form clusters while reducing the dataset’s dimensionality. Using HDBSCAN [ 32 ], a density-based clustering technique, we detected clusters of various shapes and identified outliers. BERTopic’s outlier reduction method calculates the c-TF-IDF representation for each outlier document and finds the best matching c-TF-IDF topic representation using cosine similarity.

For word-level analysis within topics or clusters, a bag-of-words representation is needed. To highlight diferences between clusters, we applied a variant of class-based TF-IDF (c-TF-IDF). Essentially, BERTopic treats all documents within a single category as a single document and then applies TF-IDF. The more significant words within a cluster, the more representative they are of that topic. Consequently, each set of documents is reduced/converted into a single one.

The entire process described above was applied to distinct datasets, diferentiating between echo chamber and non-echo chamber contexts.

BERTopic parameters1 were selected considering the dimension of the echo chambers, aiming at extracting the best representation for our data. With this configuration the aim was looking for few and stable topics to capture the macro-diferences preserving both local and global structure in the data. Moreover, with BM25 weighting we stressed the importance of interpretability and diversity in topic representations, reducing the impact of common words while still capturing meaningful bi-grams. Thus, we ensure topics are significant in size and well-represented in the corpus.

The obtained topics were then used to compare online debates taking place within and outside the social clusters about political leaning.

4.2. Cross-Sentiment Analysis

To provide a focus on the sentiments and emotions elicited by user-generated contents, we applied two diferent lexicon-based sentiment analysis algorithms [ 33 ]. In details, we leveraged Valence Aware Dictionary and sEntiment Reasoner (VADER) [ 34 ] and NRC Emotion lexicon [ 35 ]. The former is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media. Its sentiment lexicon is composed of a list of lexical features labeled according to their semantic orientation as positive or negative and is attuned to microblog-like contexts. This way, VADER labels the text as positive, neutral, negative, and provides a compound. The NRC Emotion lexicon, on the other hand, assesses the emotional afect conveyed in a text, providing a score for each sentiment or emotion detected in it. Its afective dictionary encompasses approximately 27,000 words, derived from the National Research Council Canada (NRC) afect lexicon and the synonym sets from the WordNet library within the Natural Language Toolkit (NLTK). NRC Emotion Lexicon is constituted by a list of English words and their associations with eight basic emotions (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and two sentiments (negative and positive). In our case, we used the compound score, calculated by summing the valence scores of each word in the lexicon, adjusting them according to specific rules, and then normalizing the result to range from -1 (most extreme negative) to +1 (most extreme positive). Furthermore, we leveraged the temporal dimension to understand the evolution of the discussions.

5. Results

This section reports the experiments’ results on the topic modeling applied on the two networks and the sentiment analysis scores distinguishing the political leanings (anti-trump, pro-trump, neutral) in these clusters.

Topic Modeling Firstly, we aimed to identify content similarities and dissimilarities between Echo Chambers and Non-Echo Chambers. To extract and analyze the topics, we applied BERTopic. Table 3 present the top 20 most frequent topics in each network. Despite subtle diferences in the order and size 1UMAP(nneighbors:60, ncomponents:20, mindist:0.05, metric:cosine, randomstate=42); HDBSCAN(minclustersize=90, metric:euclidean, clusterselectionmethod=eom, predictiondata=True); CountVectorizer(stopwords=english, ngramrange=(1, 2)); ClassTfidfTransformer(bm25weighting=True, reducefrequentwords=True); MaximalMarginalRelevance(diversity=0.6), mintopicsize=300). of the two networks, we observed that common topics were discussed with similar frequency in both structures. These topics included Democrats, Conservatives, Libertarians, Gun Control, U.S.-Russia relations, and immigration narratives such as the wall proposed by Trump on the Mexican border. Additionally, in both networks, users debated the perception of popular media outlets like Fox News and CNN as sources of misinformation under the topic "Media, Fake News". Summarizing, that resulted in an homogenous coverage of the contents.

Sentiment Analysis To further explore potential discrepancies in the perception of these themes, we analyzed the average sentiment and emotion trends across political leanings in both networks. Firstly, using VADER’s compound score, we obtained a general understanding of the trends. As illustrated (a) POLITICAL SPHERE Positive

(b) POLITICAL SPHERE Negative (c) POLITICAL SPHERE Trust (d) POLITICAL SPHERE Fear in Figure 2, there is a general internal coherence in the patterns, with positive and negative peaks occurring during the same periods across diferent political leanings. The Non-Echo Chamber leanings showed a generally more neutral evolution than the Echo Chamber and a final peak towards positive perception from pro-trump users. Whereas, in Echo Chamber behavior is more fluctuating. Especially, in 2018, neutral and pro-trump users treated themes positively in contrast with anti-trump. We noted a common negative peak between August and September 2017 in both the networks in conjunction with the Afghanistan conflict being exacerbated and the unveiling of the RAISE Act, a bill introduced under Trump’s government to reduce levels of legal immigration to the United States by halving the number of green cards issued.

Then, to understand the internal coherence between the two networks, we opted to apply a more ifne-grained sentiment analysis, namely the NRC Emotion Lexicon, capable of considering ten diferent sentiments and emotions in a positive range. For the sake of space, we will present only the most relevant plots. By analyzing the users’ sentiments aggregated by learning, we aim to verify their distinct behaviors. Indeed, regardless of the political leaning, Echo Chambers tend to show a less sparse evolution over time. Specifically, antitrump users’ trend often followed neutral ones’ behavior. Non-echo chambers exhibit more volatility in both trust and fear sentiments, with wider confidence intervals and more pronounced peaks and troughs. Fear sentiment peaks are higher and more frequent in Non-Echo Chambers, suggesting more dynamic changes in sentiment outside of echo chambers. Trust levels are generally higher and more stable in Echo Chambers, while fear levels are more. However, both EC and Non-Echo Chambers show similar trends with peaks around the second quarter of 2018, but Non-Echo Chambers has greater variability. Thus, by looking at the data, we can confirm our initial assumption regarding that Echo Chamber’s sentiment is more linear and less fluctuating than in Non-Echo Chambers structures.

Lastly, to validate the coverage of the topics related to their public perception, we used Google Trends2 data. We tracked users’ searches using our extracted topics as keywords and manually matched these search trends with query trends. This allowed us to confirm that the activity identified in our sentiment analysis corresponded to the specific topics modeled with BERTopic.

6. Conclusion

Echo chambers generated in social networks like Reddit, promoting like-minded users interactions, can foster the formation of closed social clusters, where individuals reinforce their shared beliefs by consuming content that aligns with their ideologies. Such processes can then be alimented by political rhetoric gravitating around "in-group/out-group” divisions, such as the one of populist actors, with potential detrimental efects for democratic processes.

To address the lack of empirical research in this field (see Sec.1), in this work we introduced a methodology to assess and analyze the content inside communities reducing the bias towards a single lexicon-based approach.

We considered the first two years of Trump’s presidency. Interestingly, on a content level, our study revealed that there is almost no diference in the topics discussed by users in echo chambers and non-echo chambers.

To deepen our understanding of how these topics are discussed, we conducted a diachronic analysis of users’ sentiments. This analysis unveiled substantial diferences depending on whether users belonged to echo chambers or not, revealing that echo chambers are a more controlled environment, despite the high degree of polarization. This outcome could be explained by the fact that echo chambers are formed by users with the same interests and behaviors. As epistemologically closed clusters, echo chambers’ debate processes are more emotionally coherent and do not sufer from high volatility like those in non-echo chambers. Despite not triggering high values of sentiment, users in echo chambers often agree with the rest of the community, reinforcing the auto-exclusive mechanism that enhances the robustness of such networks. This process still promotes the solidification of users’ stances.

Additionally, we observed sentiment patterns depending on political leaning. Particularly, pro-trump users in non-echo chambers environments scored high values for each considered sentiment, proving that their vocabulary relies on the usage of more adjectives and more and more heated discussions.

Overall, these results underscore the importance of adopting a fine-grained approach to topic modeling that considers nuanced political orientations, enabling the identification of intricate behaviors at a microscopic level.

However, this study has certain limitations. Firstly, the political leanings of users are determined through a data-driven approach, which may not fully capture the complexity of their political orientations. Secondly, the population under consideration lacks specific social characteristics typically examined in social science studies. It is worth noting that the number of Non-Echo Chamber’s users is higher than Echo Chamber’s. This could result in biased sentiment analysis results. Additionally, we do not have the tools to collect sensitive variables (such as age, sex, and country of residence), which could significantly enhance the validity and depth of our research findings. Lastly, in 2019 no Echo Chamber in POLITICAL SPHERE was detected. Thus, our plots do not cover this period.

As future research, we plan to delve deeper into user-generated content peculiarities by performing stance detection and conducting rhetorical language analysis to better characterize linguistic diferences across users belonging (not belonging) to epistemic enclaves of diferent political orientations. Such a comprehensive approach will contribute to a deeper understanding of discussion dynamics and the nuances exhibited by dialogues occurring within/outside echo chambers in Reddit.

Acknowledgments

This work is supported by the EUHORIZON2021 European Union’s Horizon Europe research and innovation programme (https://cordis.europa.eu/project/id/101073351/es) the Marie Skłodowska-Curie Grant No.: 101073351. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Executive Agency (REA). Neither the European Union nor the granting authority can be held responsible for them. The authors have no relevant financial or non-financial interests to disclose.

References

[1] A. Mahmoudi , D. Jemielniak , L. Ciechanowski, Echo chambers in online social networks: A systematic literature review , IEEE Access ( 2024 ).

[2] T. Ulen , Democracy and the internet: Cass r . sunstein, republic.com. princeton, nj. princeton university press. pp. 224 . 2001 , SSRN Electronic Journal ( 2001 ). doi: 10 .2139/ssrn.286293.

[3] K. Jamieson , J. Cappella , Echo Chamber: Rush Limbaugh and the Conservative Media Establishment , 2008 .

[4] R. K. Garrett , Echo chambers online?: Politically motivated selective exposure among Internet news users1 , Journal of Computer-Mediated Communication 14 ( 2009 ) 265 - 285 . URL: https://doi.org/10.1111/j.1083- 6101 . 2009 . 01440 .x. doi: 10 .1111/j.1083- 6101 . 2009 . 01440 .x. arXiv:https://academic.oup.com/jcmc/article-pdf/14/2/265/21491614/jjcmcom0265.pdf.

[5] P. Barberá , J. T. Jost , J. Nagler , J. A. Tucker , R. Bonneau , Tweeting from left to right: Is online political communication more than an echo chamber? , Psychological science 26 ( 2015 ) 1531 - 1542 .

[6] A. Kim , P. L. Moravec , A. R. Dennis , Combating fake news on social media with source ratings: The efects of user and expert reputation ratings , Journal of Management Information Systems 36 ( 2019 ) 931 - 968 . URL: https://api.semanticscholar.org/CorpusID:149080476.

[7] M. Del Vicario , A. Bessi , F. Zollo , F. Petroni , A. Scala , G. Caldarelli, H. E. Stanley , W. Quattrociocchi, The spreading of misinformation online , Proceedings of the national academy of Sciences 113 ( 2016 ) 554 - 559 .

[8] S. Flaxman , S. Goel , J. M. Rao , Filter Bubbles, Echo Chambers, and Online News Consumption, Public Opinion Quarterly 80 ( 2016 ) 298 - 320 . URL: https://doi.org/10.1093/poq/nfw006. doi: 10 .1093/poq/nfw006. arXiv:https://academic.oup.com/poq/article-pdf/80/S1/298/17120810/nfw006.pdf.

[9] S. Engesser , N. Fawzi , A. O. Larsson , Populist online communication: Introduction to the special issue , 2017 .

[10] P. Gerbaudo , Social media and populism: an elective afinity? , Media, culture & society 40 ( 2018 ) 745 - 753 .

[11] C. Sandelind , European populism and winning the immigration debate , Fores , 2014 .

[12] P. Norris , R. Inglehart , Cultural backlash: Trump, Brexit, and authoritarian populism, Cambridge University Press, 2019 .

[13] S. Waisbord , The elective afinity between post-truth communication and populist politics , Communication Research and Practice 4 ( 2018 ) 17 - 34 .

[14] S. Boulianne , K. Koc-Michalska , B. Bimber , Right-wing populism, social media and echo chambers in western democracies , New media & society 22 ( 2020 ) 683 - 699 .

[15] F. Esser , A. Stępińska , D. N. Hopmann , Populism and the media: Cross-national findings and perspectives , in: Populist political communication in Europe, Routledge, 2016 , pp. 365 - 380 .

[16] A. Haller , K. Holt , Paradoxical populism: How pegida relates to mainstream and alternative media , Information, Communication & Society 22 ( 2019 ) 1665 - 1680 .

[17] K. Jacobs , N. Spierings , A populist paradise? examining populists' twitter adoption and use , Information, Communication & Society 22 ( 2019 ) 1681 - 1696 .

[18] E. Cau , V. Morini , G. Rossetti, Trends and topics: Characterizing echo chambers' topological stability and in-group attitudes, 2024 . URL: https://arxiv.org/abs/2307.15610. arXiv: 2307 . 15610 .

[19] M. Conover , J. Ratkiewicz , M. Francisco, B. Gonçalves , F. Menczer , A. Flammini , Political polarization on twitter , in: Proceedings of the international aaai conference on web and social media , volume 5 , 2011 , pp. 89 - 96 .

[20] M. Cinelli , G. De Francisci Morales , A. Galeazzi , W. Quattrociocchi , M. Starnini , The echo chamber efect on social media , Proceedings of the National Academy of Sciences 118 ( 2021 ) e2023301118 .

[21] N. Duseja , H. Jhamtani , A sociolinguistic study of online echo chambers on twitter , in: Proceedings of the third workshop on natural language processing and computational social science , 2019 , pp. 78 - 83 .

[22] V. Morini , L. Pollacci , G. Rossetti, Toward a standard approach for echo chamber detection: Reddit case study , Applied Sciences 11 ( 2021 ). URL: https://www.mdpi.com/2076-3417/11/12/5390. doi: 10 .3390/app11125390.

[23] M. Del Vicario , G. Vivaldo , A. Bessi , F. Zollo , A. Scala , G. Caldarelli, W. Quattrociocchi, Echo chambers: Emotional contagion and group polarization on facebook, Scientific reports 6 ( 2016 ) 37825 .

[24] D. Wollebaek , R. Karlsen , K. Steen-Johnsen , B. Enjolras , Anger, fear, and echo chambers: The emotional basis for online behavior , Social Media+ Society 5 ( 2019 ) 2056305119829859 .

[25] M. Amendola , D. Cavaliere , C. De Maio , G. Fenza , V. Loia , Towards echo chamber assessment by employing aspect-based sentiment analysis and gdm consensus metrics , Online Social Networks and Media 39 ( 2024 ) 100276 .

[26] K. Abramski , L. Ciringione , G. Rossetti, M. Stella , Voices of rape: Cognitive networks link passive voice usage to psychological distress in online narratives, Computers in Human Behavior ( 2024 ) 108266 .

[27] M. Stella , Cognitive network science for understanding online social cognitions: A brief review , Topics in Cognitive Science 14 ( 2022 ) 143 - 162 .

[28] M. Stella , Text-mining forma mentis networks reconstruct public perception of the stem gender gap in social media , PeerJ Computer Science 6 ( 2020 ) e295 .

[29] G. Czarnek , D. Stillwell , Two is better than one: Using a single emotion lexicon can lead to unreliable conclusions , PLoS ONE 17 ( 2022 ). URL: https://api.semanticscholar.org/CorpusID:252897016.

[30] M. Grootendorst , Bertopic: Neural topic modeling with a class-based tf-idf procedure , arXiv preprint arXiv:2203.05794 ( 2022 ).

[31] L. McInnes , J. Healy , J. Melville , Umap: Uniform manifold approximation and projection for dimension reduction , arXiv preprint arXiv: 1802 . 03426 ( 2018 ).

[32] C. Malzer , M. Baum , Hdbscan( ): An alternative cluster extraction method for HDBSCAN , CoRR abs/ 1911 .02282 ( 2019 ). URL: http://arxiv.org/abs/ 1911 .02282. arXiv: 1911 .02282.

[33] G. Czarnek , D. Stillwell , Two is better than one: Using a single emotion lexicon can lead to unreliable conclusions , Plos one 17 ( 2022 ) e0275910 .

[34] C. Hutto , E. Gilbert, Vader: A parsimonious rule-based model for sentiment analysis of social media text , in: Proceedings of the international AAAI conference on web and social media , volume 8 , 2014 , pp. 216 - 225 .

[35] S. M. Mohammad , P. D. Turney , Crowdsourcing a word-emotion association lexicon , Computational intelligence 29 ( 2013 ) 436 - 465 .