Refining Deliberative Standards for Online Political Communication: Introducing a Summative Approach to Designing Deliberative Recommender Systems

Abstract

Measuring deliberative debate quality is an emerging topic in computational work, since it allows applying deliberative democratic ideas in an online domain. When doing so, many studies follow Habermas [1, 2, 3] in defining norms of online deliberative debate quality. They then proceed to propose and test new ways to measure indicators like equality, diversity, interactiveness, rationality, and civility. Consequently, implementing them within recommender systems is the necessary next step to realize those values in online communication. Although this is important work, we argue that recent advances in political science suggest that constructing a system which produces such a deliberative debate is unlikely to, by itself, contribute in an optimal way to deliberative democracy at a societal scale. Instead, we propose a complementary, summative approach to designing deliberative recommender systems. It treats online platforms as complementary to other communication channels, and argues for optimizing how to best facilitate (summative) deliberation at a societal scale rather than perfecting (micro) discussions between citizens. We illustrate this with an example of how a news recommender system based on a summative approach would have to be designed vis-a-vis a more traditional, additive approach.

Keywords

• eol>deliberative democracy
• normative standards
• online debate quality
• computational text analysis
• moral recommender systems

1. Introduction

The scale and growing influence of online communication holds great promise to realize deliberative democratic ideals. Normative theory provides an alternative to revenue-based models to develop recommender systems that seek more positive efects for society. 1 But while many online platforms ofer the potential for users to interact with their content as well as with other users and thereby create the potential for political discussion, it is far from clear how recommender systems need to be designed to achieve positive contributions to political discourse at a societal scale. On the one hand, the development online deliberative debate indicators is a flourishing field [ 5 ] – which is very good, as after all, recommender systems need some metrics to optimize for or be evaluated on. On the other hand, from a normative point of view, these metrics also need to be grounded in democratic theory – and this is less straightforward than it my seem.

To begin with, scholars who wish to develop recommender systems that facilitate online discussions along normative democratic lines, have to choose which out of various alternative conceptions of democracy they want to rely on [ 6, 7 ]. For example, based on so-called liberal theories of democracy one may argue that topics that are prominent on the political agenda should also be prominent in a recommendation, while based on so-called critical theories, one would expect marginal topics to be recommended instead; so-called participative models would draw attention to topics citizens “should know”, and so-called deliberative models would focus more on the diversity of the items [ 6 ]. In this paper, we focus on deliberative models of democracy. Deliberative democracy is often considered the most demanding in terms of the required quality of (online) political discussion, and thereby provides a nice ideal point to strive for (cf. [ 8 ]). Its emphasis on discussion also fits well with the nature of online communication platforms.

Although there are many variants of deliberative theory, the work of Habermas [ 1, 2, 3 ] is considered a cornerstone in this literature (c.f.[ 9, 5 ]). Deliberative democracy is about more than making decisions based on aggregating the preferences of citizens, or even than preferences based on accurate information, but rather holds that (collective) preferences should be formed through inclusive, reasoned debate [ 10, 8 ]. In this view, online communication needs to provide relevant and accurate information, but also connect citizens and motivate them to share and debate their views in a deliberative way.

Scholars seeking to measure deliberation online have proposed and listed various indicators which can be used by recommender systems for this purpose. They measure concepts like equality, diversity, rationality, interactivity, civility and reference to the common good [ 7, 5, 11, 12 ]. Recommender systems are widely used on for example E-commerce platforms, social networks, video-sharing platforms, and news websites and apps. Each of these platforms ofer the potential for politically relevant discussions [ 13 ]. However, most computational work focuses on developing specific indicators relevant to deliberative democracy (e.g. [ 14 ], but recommender systems focusing on all indicators are needed to fully realize deliberative ideals [ 5 ]. In addition, less attention has been paid to whether and how these indicators can best be put to use to realize deliberative democratic ideals on a societal scale.

In political science, scholars have struggled with similar issues when studying the deliberative values of citizen participation initiatives. Although they are using non-computational methods, their insights can be of use to the computational community as well. Similar to computational scholars, deliberative political scientists started out evaluating whether and to what degree the elements of deliberative democratic debate could be found in the exchanges between citizens. After three decades of sustained, wide academic attention for this topic, they are now moving away from what is called an “additive” approach to deliberation and question whether deliberative quality is best treated as a single, unitary, concept consisting of all of its elements to an equal degree regardless of context [ 15, 16 ]. The additive approach holds that "Deliberation [...] is produced by specific methods or institutions which then add it—inject it, if you will—into the system more broadly" [15, p.7]. That is, they sought to construct an ideal forum to foster deliberation in all its facets and use this as a way to produce deliberative democracy at a societal scale. One example is forming a Citizen Assembly in which a stratified sample of the population is invited to deliberate on a policy proposal over several days and then share their arguments and propose their decision to the public at large and a legislative body in particular (e.g. see [ 17 ]). Deliberation in this view is often seen in a more-is-better fashion, so more diversity in a news feed is better for deliberative democracy than less diversity, more interactivity is better than less and so on [ 16 ]. Although intuitive, the additive approach has been criticized for failing to grasp the complexity of human behavior and for being ill-equipped to relate the outcomes of individual (online) interactions back to the over-arcing goals of deliberative democracy on a societal scale [ 18, 15 ]. Notably, even Habermas himself has recently questioned the approach of using his indicators as a yardstick for societal debate, stating that “I do not see deliberative politics as a farfetched ideal against which sordid reality must be measured” [19, p. 149].

In the next section, we outline the critique political scientists have levelled against the additive approach. Subsequently, we sketch diferent views on how (micro) online deliberations ift within the larger deliberative democratic ideal. Finally, we propose an alternative conception of debate quality for online platforms, based on the recently proposed systematic, summative approach to deliberative democracy [ 20, 15 ]. The summative approach does not seek to optimize deliberation at any single venue, but rather maximize its value at the aggregate, societal scale. This alternative treats online platforms as complementary rather than substitutive to traditional media, and seeks to realize deliberative goals in a summative rather than additive fashion.

2. More Is Better, Or Is It?

Up until recently, either implicitly or explicitly, proponents of computationally measuring deliberative quality often appear to advocate an additive, more-is-better approach. Some describe how a specific debate aspect, like diversity, is helpful for deliberative democracy and then propose how to measure it, others list indicators a good debate needs in order to be deliberative [ 5, 12, 7, 6 ]. Like early deliberative political scientists, computational scholars then evaluate whether a particular debate fulfills these criteria and to which degree (e.g. [ 21 ]). A recommender system based on this approach could either aggregate indicator scores into a total deliberative score to rank which items to recommend, or define minimum/maximum values needed to pass platform moderation (such as for civility). However, doing so would imply assuming that deliberation is a unitary concept, i.e. that all criteria need to be fulfilled together to a certain degree for the ascribed benefits to deliberative democracy to materialize. While intuitive, Mutz [ 18 ] argues that one should be careful, since this approach carries some assumptions about human behavior that are unwarranted. Also adding the scores of various criteria together to one overall deliberative score implies that deficiencies on the score for one criteria can be compensated for by higher scores on other criteria, while using indicators for minimum benchmark values implies that failing to meet the minimum level on one criteria disqualifies any achievements on other criteria.

A growing literature questions that deliberation is a unitary concept and all indicators necessarily need to go together (e.g. [ 18, 15 ]). In fact, there are many situations where various indicators of deliberation appear to be at odds with one another [ 16 ]. For example, diversity might conflict with inclusion: the more diverse the voices in a debate, the less likely it is that everyone participates, since debate requires conflicting arguments and most people feel uncomfortable being confronted by (too many) opposite opinions (cf [ 22, 23 ]). Likewise representation, accountability and openness might conflict with civility: publicity makes representatives more accountable to their supporters, but is also found to be less conducive to mutual respect and constructive politics than deliberation behind closed doors [ 15 ]. This directly highlights one of the main problems in deliberative democratic theory: that quality aspects of debates are frequently conflated with positive efects of deliberative debates in definitions of what counts as deliberative democracy [ 18, 24 ]. This lack of conceptual clarity in stipulating causes and efects masks the myriad in conceptualizations of what exactly constitutes deliberation as well as what its efects are supposed to be.

Like deliberative indicators, deliberative efects/outcomes are also more complex empirically than a unitary concept approach would appear to imply. A great many presumed efects can be found in the literature. Bächtiger and Parkinson [ 15 ] group them as: (1) epistemic outcomes (find best possible approach to handle a common problem); (2) ethical outcomes (follow the rational argument dominated deliberative process as a goal in itself); (3) providing legitimacy (of collective decisions formed through deliberation); (4) emancipation of minority groups (providing a space to make all citizens heard); (5) transformation and clarification of preferences (people learn from the debate and change their views or deepen their perspectives in return); with some also listing (6) consensus as a desired outcome. Although these goals overlap to a certain degree, diferent deliberative aspects contribute to them in a diferent degree. Scholars have listed many contradictions between diferent aspects of deliberation and deliberative outcomes. For example, if the process of rational argumentation is the goal, the resulting formal-tone of the debate might be of-putting to some citizens, and dominant groups might use their definition of what counts as a rational argument to suppress minority voices, also requiring each position to come with an elaborate set of supporting arguments might favor established and well documented positions over new voices that as of yet have not had the time and space to develop such arguments [ 25, 26 ].

Given these conflicts between indicators of deliberation both among themselves and in relation to the desired deliberative outcomes, Mutz [ 18 ] proposes to abandon the unitary concept of deliberation and instead investigate which deliberative aspects have which efects under which conditions. She argues that it is likely that diferent deliberative aspects might interact with each other to reach specific outcomes, and should therefore be studied separately as well as in diferent combinations. Applying this line of argumentation to recommender system design, we can say that if a system optimizes either for (1) all deliberative indicators, or (2) an indexed value of debate quality based on combining various indicators, or (3) some minimum/maximum values of deliberative indicators, then the contradictions between these indicators and outcomes as described above might lead this system to hinder rather than enable the deliberative process. In the next section, we will elaborate more on the argument that what is seemingly good for deliberation on a micro-level does not necessarily lead to good deliberative outcomes on the societal macro-level.

3. The Gap Between Micro and Macro

The second main critique of political scientists on using deliberative indicators as a yardstick to measure contributions to normative standards of deliberative democracy, is the problematic relation between specific debates (micro) and deliberative democracy on a societal scale (macro) [ 27 ]. Most accounts of deliberative democracy aim at the societal level rather than that of the individual debate. They require the deliberative process is in fact democratic: that it culminates in a collectively binding decision [ 16 ]. Habermas [ 19 ] stipulates how issues, information and arguments in debates between citizens are picked up by societal actors, like social movements, PR-organizations, political parties and the media which translate positions into coherent discourses relating distinct positions to relevant arguments, which then feed into the political arena to result in collective decisions. He appears to be unsure, though, how to fit online debates productively into his framework, and argues that such debates might actually be counterproductive warning that “[t]he platforms do not ofer their emancipated users any substitute for the professional selection and discursive examination of contents based on generally accepted cognitive standards” [19, p. 160], and “the increasing dissonance of a strident diversity of voices and the complexity of the challenging topics and positions is leading a growing minority of media consumers to use digital platforms to retreat into shielded echo chambers of the like-minded. For the digital platforms not only invite their users to spontaneously generate intersubjectively confirmed worlds of their own but seem to lend the stubborn internal logic of these islands of communication, in addition, the epistemic status of competing public spheres” [19, p. 162]2. On top of this, deliberation between all citizens in a modern society is practically unfeasible due to constraints in time and resources, and can therefore only be realized at the institutions of the state [ 19 ]. However, empirical research into the political arena, where the final deliberative debate between contrasting discourses should culminate into collective decisions, ifnds that parliamentary debate is oriented towards voting rather than aggregating information and participants rarely change their preferences in view of contrasting information [ 15 ]. It is therefore unclear whether and how online political communication contributes to deliberative democracy at all.

This focus on relating the micro to the macro is often labelled as the “systemic turn”, as it views deliberative democracy as a larger system rather than a debate at large [ 27, 29, 20 ]. Political scientists have proposed three main theoretical frameworks that specify how specific debates between citizens could translate into collectively binding decisions in a deliberative way. Bächtiger and Parkinson [ 15 ] group them into discursive, sequential and spatial models of deliberation. The discursive model focuses on how people understand and shape society through discourses that find their way into arguments, decisions and policy [ 30 ]. Various versions of the sequential model hold that feedback loops ensure deliberative outcomes over time, where societal debate influences political decisions, which are in turn part of societal debate to critique, alter, maintain or reject them at another round of political decisions. The spatial model specifies distinct deliberative functions of distinct institutions and the proper relations between these institutions to ensure deliberative outcomes. Each of these models has received its fair portion 2Note that Zuiderveen Borgesius et al. [ 28 ] found little empirical support for the relation between recommender systems and echo chambers or filter bubbles of critique, where the discursive model is unclear how deliberation contributes in what way to the forming of discourses and how these discourses feed into policy in a deliberative way, the sequential model is often empirically incorrect in policy preceding debate especially for non-salient issues, and the spatial model is found to be to static to encompass the creatively changing nature of political decision making with new forms or organisation continuously popping up (e.g. #MeToo) and actors reinterpreting their role and using their institution in new ways [ 15 ].

4. Where Do Online Platforms Fit in the Larger System of

Deliberative Democracy?

Regardless of whether one adopts a discursive, sequential or spatial perspective, for an online platform to contribute to deliberative democracy optimally it thus needs to produce some deliberative contribution and transfer this contribution in some way to the wider society and its political decision making bodies in particular. So what kind of contribution should online platforms, via the recommender systems employed by them, make in the normative ideal of deliberative democracy? Habermas [ 19 ] ascribes them a similar role to traditional media in his (sequential) approach to deliberative democracy, and then criticizes online platforms for failing to live up to those expectations: Platforms lack journalistic moderation and do not “qualitatively iflter opinions”, like the traditional media, where journalists scrutinize arguments and opinions for facts and counterarguments and professionally select what to present to their audience [ 19 ]. In addition, he is wary of personalization, since it may enable selective exposure and echo chambers. On top of this online platforms have not been very successful empirically in producing good deliberative debates. Where additive approaches to deliberative democracy seek to find sites that facilitate optimal deliberation and then to channel the results as best as possible to other parts of the deliberative system, most major online platforms are known to lack those deliberative qualities [ 31, 32, 33 ].

However, there might be a diferent contribution that online platforms can make to deliberative democracy, which is more feasible and better suited to their qualities. Habermas [ 19 ] notes that the contribution of political communication in the public sphere, where online platforms are located, is inherently limited, since only representative bodies make collective decisions. The normative requirements of achieving deliberation in all its facets need thus not be so strict for these platforms. As the discussion in Sections 2 and 3 has showed, it might not be optimal to strive for deliberation in all its facets on online platforms to begin with. Bächtiger and Parkinson [ 15 ] have recently proposed an alternative route to (macro) deliberative democracy which might better fit with the potential of online platforms: the summative approach. They explain that “the deliberative quality may emerge from the complex interactions of a variety of practices and institutions rather than an input generated by one or two of them” [15, p. 14]. Simply put: deliberative outcomes may be realized through non-perfect deliberative components, like online platforms.

We propose that by thinking of deliberative democracy as a summative quality, we arrive at other goals that recommender systems of online platforms should perform. They no longer need to facilitate deliberation between citizens as best as possible, but might focus on optimizing the larger goals of deliberative democracy: reaching collective decisions on a rational basis involving as many citizens on in the most equal way possible (cf. [ 16 ]). It needs some mechanism to (1) involve citizens including those acting on behalf of social groups, politicians, PR oficials etc.; to (2) make them share their views and information; to (3) facilitate them to interact with each other, existing discourses, and actions and words of political actors to develop and question their opinion; to (4) collect and scrutinize arguments and positions into coherent discourses concerning collective issues; to (5) communicate those discourses back to as many citizens as possible, but (6) also to their representatives in the political arena. We believe a normative design of recommender systems can play a role in each of these mechanisms.3

5. Moving Forward: A Complementary Role Within a

Summative Approach

We propose that in this summative understanding of deliberative democracy it is more helpful to think of online communication as complimentary to other forms of political communication, rather than as a substitute for traditional media. Our approach thus allocates a diferent function to online platforms within the spatial and sequential system of deliberative democracy than an additive approach would. For example, when facilitating debate between citizens, instead of aiming for civil conversation, it might be better for (macro) deliberative democracy, if in some cases people are allowed some incivility to make suppressed voices heard or to create a communicative environment where some might feel more at home, where they feel they don’t need to be eloquent and highly educated to be allowed to speak up. While at the same time, those who might be put of by such discourse could be shielded from exposure to uncivil content (cf. [ 34 ]). Where the one-to-many format of traditional media necessitates compromises in form and content to fit a larger audience at the cost of individual diferences, personalization enables online platforms to tune into the individual needs of each citizen [ 22 ]. Hereby, content can be presented to each citizen in a fashion tailored to encourage involvement, both in engaging with arguments and in building the eficacy needed to share one’s views and information (cf. [ 6 ]). Online platforms also provide opportunities to go beyond what can be achieved in deliberative terms by traditional media, by directly linking citizens to, for example, journalists and politicians [ 31 ]. Hereby they create a crucial link in facilitating deliberative sub-products, like suppressed voices and new positions and arguments, to reach the traditional media and institutional political arena.

Of course, such personalization is exactly what Habermas [ 19 ] criticises when he warns about the the potential of creating parallel public spheres. When each citizen receives her own tailor made version of online content, this might hinder a common understanding of the main issues, positions and arguments facing a society. However, from the viewpoint of the complementary role of online communication to other media and institutions, this can also be seen as an opportunity for recommender systems to provide that common information through sharing relevant content as provided by, for example, traditional media, politicians or activist groups, in a tailored way to the largest audience. In this way online platforms can 3Wessler [ 31 ] provides an alternative list of possible contributions of non-deliberative media to deliberative democracy. actually improve the deliberative value of intersubjective understanding by involving previously disafected groups.

Table 1 provides an example of how this summative approach to designing deliberative recommender systems difers from existing approaches in the parameters that need to be optimized. The left column (“additive deliberative recommender system”) shows how additive approaches seek to optimize all aspects of deliberation at once, and facilitate deliberative democracy through concrete instances of citizen deliberation on online platforms, while the right column shows that the summative approach instead optimizes deliberative outcomes at a societal level. Note how the goals listed in the right column match the mechanisms required for deliberative democracy outlined in Section 4. The right column focuses on optimizing exposure to foster the deliberative value of inclusion; on optimizing engagement to get citizens to interact with the debate; on optimizing the sharing of information to include as many insights from as many citizens as possible; on providing these insights to other users and political actors alike; and on including fact-check information to debunk misinformation and increase the factual quality of the arguments. The summative column thus seeks to explicitly and directly link citizens and political actors (e.g., politicians, but also including activist groups or PR-agencies), since connecting the diversity of arguments leveled by both groups to each other is a specific macro deliberative democratic value. While the additive approach (left column) thus seeks to fit the debate into a deliberative mold, the summative approach (right column) seeks to optimize societal deliberative outcomes.

The summative indicators proposed here are familiar ones in the field of recommender systems and partly overlap with both additive indicators and those used in commercial revenuebased applications. They are not meant to form a definitive list. They should rather be seen as an invitation to scholars to propose their own more efective set of indicators to make communication on online platforms contribute to the mechanisms required for deliberative democracy outlined in Section 4. The overlap with existing commercial recommender systems makes the summative approach more in line with existing practices on online platforms and potentially easier to realize (cf. [ 35 ]). It does not try to change what people like about online platforms, rather to guide them in a normative, societal deliberative direction. The familiarity of these indicators illustrates the feasibility of this alternative route to realizing deliberative values online.

6. Conclusion

Computational approaches to measuring deliberative indicators of online communication are a blooming field and much work has been done in constructing indicators for various aspects of deliberation, such as equality, rationality, interactivity, diversity and civility [ 5 ]. However, political scientists have levelled two main critiques against the common computational implementation of debate quality: deliberation is unlikely to be related to deliberative democracy in a unitary, more-is-better fashion and that (micro) online deliberation is unlikely to contribute in an additive way to (macro) deliberative democracy. So even if computational scholars could ifnd a way to overcome the current technical challenges and construct a perfect set of reliable and valid indicators of deliberative quality (see [ 5 ]), then still it would be questionable how • exposure • engagement (likes/comments) • sharing information • diversity of traditional news expo

sure • diversity of user and political actor

exposure • inclusion of fact-check info where

possible Weight of metrics determined on individual basis Metrics can also be optimized for sequentially; long-run outcome more important than simultaneously good scores on every metric Contribute to societal deliberation these indicators could be implemented in recommender systems to attain normative deliberative outcomes.

We propose that one way out of this dilemma could be to build on the summative approach, which seeks to optimize deliberative outcomes at a societal scale, rather than the additive approach which seeks to optimize deliberation at each site/venue. Instead of a straightforward design of recommender systems that either keep indicators of (micro) deliberation within acceptable bounds or that optimize for them, we have argued that the complexity of human behavior frustrates those eforts, and that it might lead to counterproductive results at a societal level. Instead, we propose a summative approach to designing deliberative recommender systems for online platforms. These systems take more account of the place of online platforms within the larger system of deliberative democracy, and respect the potential trade-ofs between diferent deliberative values. They select and optimize an alternative set of indicators directed at macro deliberative goals.

This approach aims to be both more fitting to the less than pure deliberative nature of online debate (cf. [ 31, 32 ]. In fact, designing a summative deliberative recommender system does not have to be at odds with commercial interests. For example, in the summative deliberative framework, one explicit goal is to increase exposure to and engagement with the “long tail” of content to make users aware of perspectives they may not be aware of, and allow them to contribute. But this can perfectly align with commercial interests: In many recommendations scenarios, it is an explicit goal to increase usage of long-tail items that the user would otherwise not find.

Through its better fit with both deliberative democracy at the societal level and the nature of online platforms, the summative approach proposed here can help recommender systems to increase the contribution that online communication can make to deliberative democracy and thereby also help reduce negative efects often associated to online communication, such as iflter bubbles, selective exposure and misinformation [ 36, 37, 34 ].

Acknowledgments

This research is fully funded by the European Union and part of the “TWON: Twins of Online Networks” EU HORIZON grant, project number 101095095. We would like to thank the anonymous reviewers whose detailed comments helped improve the paper. All opinions and remaining errors are our own.

References

[1] J. Habermas , Strukturwandel der Öfentlichkeit. Untersuchungen zu einer Kategorie der bürgerlichen Gesellschaft , Luchterhand, Neuwied and Berlin, 1962 .

[2] J. Habermas , The theory of communicative action , Beacon Press, Boston, 1984 .

[3] J. Habermas , Between Facts and Norms: Contributions to a Discourse Theory of Law and Democracy , The MIT Press, Cambridge, MA, 1996 .

[4] E. Knudsen , Modeling news recommender systems' conditional efects on selective exposure: evidence from two online experiments , Journal of Communication 73 ( 2023 ) 138 - 149 . doi: 10 .1093/joc/jqac047.

[5] A. Goddard , A. Gillespie , Textual Indicators of Deliberative Dialogue: A Systematic Review of Methods for Studying the Quality of Online Dialogues, Social Science Computer Review ( 2023 ) 089443932311566 . doi: 10 .1177/08944393231156629.

[6] N. Helberger , On the Democratic Role of News Recommenders, Digital Journalism 7 ( 2019 ) 993 - 1012 . doi: 10 .1080/21670811. 2019 . 1623700 .

[7] S. Vrijenhoek , M. Kaya , N. Metoui , J. Möller , D. Odijk , N. Helberger , Recommenders with a Mission: Assessing Diversity in News Recommendations , in: Proceedings of the 2021 Conference on Human Information Interaction and Retrieval , ACM , Canberra ACT Australia , 2021 , pp. 173 - 183 . doi: 10 .1145/3406522.3446019.

[8] J. Strömbäck , In Search of a Standard: four models of democracy and their normative implications for journalism , Journalism Studies 6 ( 2005 ) 331 - 345 . doi: 10 .1080/ 14616700500131950.

[9] D. Caluwaerts , K. Bernaerts , R. Kesberg , L. Smets , B. Spruyt , Deliberation and polarization: a multi-disciplinary review , Frontiers in Political Science 5 ( 2023 ) 1127372 . doi: 10 .3389/ fpos. 2023 . 1127372 .

[10] D. G. Freelon , Analyzing online political discussion using three models of democratic communication , New Media & Society 12 ( 2010 ) 1172 - 1190 . doi: 10 .1177/ 1461444809357927.

[11] M. Nelimarkka , J. P. Rancy , J. Grygiel , B. Semaan, (Re) Design to Mitigate Political Polarization: Reflecting Habermas' ideal communication space in the United States of America and Finland , Proceedings of the ACM on Human-Computer Interaction 3 ( 2019 ) 1 - 25 . doi: 10 .1145/3359243.

[12] L. Oswald , Automating the Analysis of Online Deliberation? A comparison of manual and computational measures applied to climate change discussions , SocArXiv ( 2022 ).

[13] M. E. Wojcieszak , D. C. Mutz , Online Groups and Political Discourse: Do Online Discussion Spaces Facilitate Exposure to Political Disagreement? , Journal of Communication 59 ( 2009 ) 40 - 56 . doi: 10 .1111/j.1460- 2466 . 2008 . 01403 .x.

[14] S. Vrijenhoek , G. Bénédict, M. G. Granada , D. Odijk , M. de Rijke, RADio - Rank-Aware Divergence Metrics to Measure Normative Diversity in News Recommendations, 2022 . URL: http://arxiv.org/abs/2209.13520, arXiv: 2209 .13520 [cs].

[15] A. Bächtiger , J. Parkinson , Mapping and Measuring Deliberation: Towards a New Deliberative Quality , 1 ed., Oxford University Press, 2019 . doi: 10 .1093/oso/9780199672196. 001.0001.

[16] D. F. Thompson , Deliberative Democratic Theory and Empirical Political Science, Annual Review of Political Science 11 ( 2008 ) 497 - 520 . doi: 10 .1146/annurev.polisci. 11 . 081306.070555.

[17] K. Már , J. Gastil , Do Voters Trust Deliberative Minipublics? Examining the Origins and Impact of Legitimacy Perceptions for the Citizens' Initiative Review , Political Behavior ( 2021 ). doi:10.1007/s11109-021-09742-6.

[18] D. C. Mutz , Is Deliberative Democracy a Falsifiable Theory? , Annual Review of Political Science 11 ( 2008 ) 521 - 538 . doi: 10 .1146/annurev.polisci. 11 .081306.070308.

[19] J. Habermas , Reflections and Hypotheses on a Further Structural Transformation of the Political Public Sphere , Theory, Culture & Society 39 ( 2022 ) 145 - 171 . doi: 10 .1177/ 02632764221112341.

[20] J. Mansbridge , J. Bohman , S. Chambers , T. Christiano , A. Fung , J. Parkinson , D. F. Thompson , M. E. Warren , A systemic approach to deliberative democracy , in: J. Parkinson , J. Mansbridge (Eds.), Deliberative Systems , 1 ed., Cambridge University Press, 2012 , pp. 1 - 26 . doi: 10 .1017/CBO9781139178914.002.

[21] N. Beauchamp , Modeling and Measuring Deliberation Online, in: B. Foucault Welles , S. González-Bailón (Eds.), The Oxford Handbook of Networked Communication , Oxford University Press, 2020 , pp. 320 - 349 . doi: 10 .1093/oxfordhb/9780190460518.013.23.

[22] M. Reuver , N. Mattis , M. Sax , S. Verberne , N. Tintarev , N. Helberger , J. Moeller , S. Vrijenhoek , A. Fokkens , W. Van Atteveldt , Are we human, or are we users? The role of natural language processing in human-centric news recommenders that nudge users to diverse content , in: Proceedings of the 1st Workshop on NLP for Positive Impact , Association for Computational Linguistics, Online, 2021 , pp. 47 - 59 . doi: 10 .18653/v1/ 2021 . nlp4posimpact- 1 .6.

[23] N. Mattis , P. Masur , J. Möller , W. Van Atteveldt , Nudging towards news diversity: A theoretical framework for facilitating diverse news consumption through recommender design , New Media & Society ( 2022 ) 146144482211044 . doi: 10 .1177/14614448221104413.

[24] D. Friess , C. Eilders , A Systematic Review of Online Deliberation Research, Policy & Internet 7 ( 2015 ) 319 - 339 . doi: 10 .1002/poi3. 95 .

[25] L. M. Sanders , Against Deliberation, Political Theory 25 ( 1997 ) 347 - 376 . doi: 10 .1177/ 0090591797025003002.

[26] Dahlberg , The Internet, deliberative democracy, and power: Radicalizing the public sphere , International Journal of Media & Cultural Politics 3 ( 2007 ). doi:10.1386/macp.3.1.47/ 1.

[27] S. Chambers , Rhetoric and the Public Sphere: Has Deliberative Democracy Abandoned Mass Democracy? , Political Theory 37 ( 2009 ) 323 - 350 . doi: 10 .1177/ 0090591709332336.

[28] F. J. Zuiderveen Borgesius , D. Trilling , J. Möller , B. Bodó , C. H. De Vreese , N. Helberger , Should we worry about filter bubbles? , Internet Policy Review 5 ( 2016 ). doi: 10 .14763/ 2016 .1.401.

[29] D. Owen , G. Smith , Survey Article: Deliberation, Democracy, and the Systemic Turn: Survey Article: Deliberation & the Systemic Turn , Journal of Political Philosophy 23 ( 2015 ) 213 - 234 . doi: 10 .1111/jopp.12054.

[30] J. Dryzek , Discursive democracy, Cambridge University Press, New York, 1990 .

[31] H. Wessler , Habermas and the Media, Polity Press, Cambridge, UK, 2018 .

[32] L. Dahlberg , The Internet and Democratic Discourse: Exploring The Prospects of Online Deliberative Forums Extending the Public Sphere , Information, Communication & Society 4 ( 2001 ) 615 - 633 . doi: 10 .1080/13691180110097030.

[33] K. Esau , D. Fleuß , S. Nienhaus , Diferent Arenas, Diferent Deliberative Quality? Using a Systemic Framework to Evaluate Online Deliberation on Immigration Policy in Germany , Policy & Internet 13 ( 2021 ) 86 - 112 . doi: 10 .1002/poi3. 232 .

[34] M. Fernandez , A. Bellogin , Recommender Systems and Misinformation: The Problem or the Solution? , OHARS Workshop. 14th ACM Conference on Recommender Systems , 22 -26 Sep 2020 , [Online] ( 2020 ).

[35] L. Heitz , J. A. Lischka , A. Birrer , B. Paudel , S. Tolmeijer , L. Laugwitz , A. Bernstein , Benefits of Diverse News Recommendations for Democracy: A User Study , Digital Journalism 10 ( 2022 ) 1710 - 1730 . doi: 10 .1080/21670811. 2021 . 2021804 .

[36] E. Pariser , The Filter Bubble: What the Internet is Hiding from You , Penguin, New York, 2011 .

[37] C. Sunstein , Republic.Com, Princeton University Press, Princeton, NJ, 2001 .