Recommender Systems and Misinformation: The Problem or the Solution? Miriam Fernandeza , Alejandro Bellogínb a Open University, United Kingdom b Universidad Autónoma de Madrid, Spain Abstract Recommender Systems have been pointed as one of the major culprits of misinformation spreading in the digital sphere. These systems have recently gone under heavy criticism for promoting the creation of filter bubbles, lowering the diversity of information users are exposed to and the social contacts they create. This influences the dynamics of social news sharing, and particularly the ways misinformation initiates and propagates. However, while Recommender Systems have been accused of fuelling the spread of misinformation, it is still unclear which particular types of recommender algorithms are more prone to recommend misinforming news, and if, and how, existing recommendation algorithms and evaluation metrics, can be modified or adapted to mitigate the misinformation spreading effect. In this position paper, we describe some of the key challenges behind assessing and measuring the effect of existing recommendation algorithms on the recommendation of misinforming articles and how such algorithms could be adapted, modified, and evaluated to counter this effect based on existing social science and psychology research. Keywords Misinformation, news, recommender systems 1. Introduction Misinformation has become a common part of our digital media environments, and it is compro- mising the ability of our societies to form informed opinions [1]. It generates misperceptions, which affect our decision making processes in many domains, including economy, health, environment, or elections. In 2016, post-truth was chosen by the Oxford Dictionary as the word of the year, after achieving a 2000% increase “in the context of the EU referendum in the United Kingdom and the presidential election in the United States”. Today, in the context of a global pandemic, misinformation has led to tragic results, including links to assaults, arson and deaths.1 Although misinformation is a common problem in all media, it is exacerbated in digital social media due to the speed and ease in which posts are spread. The social web enables people to spread information rapidly without confirmation of truth, and to paraphrase this information to fit their intentions and present beliefs [2]. OHARS’20: Workshop on Online Misinformation- and Harm-Aware Recommender Systems, September 25, 2020, Virtual Event email: miriam.fernandez@open.ac.uk (M. Fernandez); alejandro.bellogin@uam.es (A. Bellogín) orcid: 0000-0001-6368-2510 (A. Bellogín) © 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings CEUR Workshop Proceedings (CEUR-WS.org) http://ceur-ws.org ISSN 1613-0073 1 https://www.bbc.co.uk/news/stories-52731624 40 Multiple factors influence the spread of misinformation online including: (i) the ways in which information is constructed and presented [3, 4], (ii) the users’ personality, values and emotions [5, 6], (iii) the architectural characteristics of the digital platforms where such information is spread (i.e., the structure of the social networks, constraints on the type of messages or sharing permissions, etc.) [7] and (iv) the algorithms that power the recommendation of information within those platforms.2 While multiple works have concentrated on studying the effect of different types of information, users and digital platforms on the spread of misinformation, we argue in this paper that there is a need to further explore the effect of existing algorithms on the recommendation of false and misleading information. As mediators of online information consumption, recommendation algorithms have been strongly criticised for becoming unintended means for the amplification and distribution of misinformation [8].3 This problem is rooted in the core design and principles in which these algorithms are based. The assumption that users are interested in items that are similar to the ones for which they expressed a preference in the past, or in items that are liked by users that are similar to them, helps build up and boost the so-called “echo-chambers”. Moreover, in their attempt to deliver relevant suggestions, recommendation algorithms are prone to amplify biases, such as popularity and homogeneity biases [9, 10]. Echo-chambers and biases may limit the exposure of users to diverse points of view, potentially making them vulnerable to misinformation. Aiming to break these echo-chambers and to reduce the spread of misinformation, different online platforms are applying different strategies. Twitter, for example, started recommending popular tweets into the feeds of people who did not subscribe to the accounts that posted them. This approach, of providing popular opposing views, was however heavily criticised for amplifying inflammatory political rhetoric and misinformation.4 Additionally, research indicates that presenting people with corrective information is likely to fail in changing their salient beliefs and opinions, or may, even, reinforce them [11]. People often struggle to change their beliefs even after finding out that the information they already accepted is incorrect or misleading. Nevertheless, some strategies have been found to be effective in correcting misperceptions, such as exposing users to related but disconfirming stories [12], or revealing the demographic similarity of the opposing group [11]. In this paper we argue that, while recommendation algorithms have been heavily criticised for promoting the spread of misinformation, a more in-depth investigation is needed to better understand which of these algorithms are more prone or susceptible of spreading misinformation, under which circumstances, and how the internal functioning of such systems could be modified, or adapted, to counter their misinformation recommendation behaviour. The next section presents our proposed research vision, and the different research building blocks we envision are needed to target this problem. 2 https://www.wired.com/story/creating-ethical-recommendation-engines/, https://www.buzzfeednews.com/ article/craigsilverman/how-facebook-groups-are-being-exploited-to-spread 3 https://www.niemanlab.org/2020/01/youtubes-algorithm-is-pushing-climate-misinformation-videos-and-their- creators-are-profiting-from-it/ 4 https://edition.cnn.com/2019/03/22/tech/twitter-algorithm-political-rhetoric/index.html 41 Figure 1: Research Dimensions 2. Research Dimensions and Challenges Understanding which recommendation algorithms are more prone to spread misinformation, and under which circumstances is not a trivial problem. Misinformation spreading is a problem with a high number of dimensions that interrelate to one another, some of them also affecting what recommendation algorithms learn and therefore, how they will later on behave. Similarly, adapting such algorithms to counter their misinformation recommendation behaviour requires an in-depth understanding not only of the internal mechanisms of such algorithms, but also of the data they manipulate, the users they serve, and the platforms they operate in. In this section, we present our vision on how to address these problems based on four key building blocks or research dimensions, and we discuss the challenges associated to each of them. The proposed research dimensions are represented in Figure 1, together with how they interrelate with one another. The first one, Misinformation: Problem Dimensions, aims to understand what are the different dimensions of the misinformation problem, and within them, the aspects that may affect the behaviour of recommendation algorithms (e.g., the users, the type of information, etc.). The second one, Analysis of Recommendation Algorithms, refers to the need of conducting in-depth investigations of the different existing recommendation algorithms (content-based, collaborative filtering such as matrix factorisation, demographic- and knowledge-based techniques, and so on) and how their internal mechanisms may improve or worsen the spread of misinformation. The third one, Human-centred Evaluation, refers to the need of modifying existing evaluation methods and metrics to target not only the enhancement of content and/or user similarity, 42 or user satisfaction, but maybe promoting some degree of dissimilarity and cognitive disso- nance that could help users to break their filter bubbles. The fourth dimension, Adaptation, Modification and Vigilance, refers to the investigation of the different ways in which existing algorithms could be modified and adapted to counter their misinformation recommendation behaviour. Vigilance refers to the need of constant monitoring to ensure that: (i) the dynamics of misinformation are captured over time, and that algorithms are adapted accordingly and, (ii) that the proposed adaptations do not back-fire or introduce any additional ethical issues (e.g., algorithmic adaptations that may reduce the recommendation of misinformation, but that tend to promote misinformation of a more harmful nature should not be considered successful). As we can see in Figure 1, these four research dimensions closely interrelate with one another. The inner circle of the figure touches on the first three research dimensions. These dimensions encapsulate the first part of the investigation, understanding the effect of recommendation algorithms on the spread of misinformation. The outer circle also includes the fourth dimension, i.e, investigating how these algorithms could be modified to counter their harmful behaviour. Arrows in and out of these dimensions indicate how they influence one another. Understanding the different variables that affect the problem of misinformation (the types of content, the types of users, etc.) is key to assess which of those variables may also affect the development, training and learning of recommendation algorithms. Similarly, understanding the internal mechanisms of such algorithms, in conjunction with the problem of misinformation, is necessary to de- sign appropriate evaluation protocols that, while effectively satisfying the users’ information needs, also palliate the recommendation of misleading information. Research centred in all these four dimensions is needed to better comprehend if, and how, we could improve existing recommendation mechanisms. 2.1. Misinformation: Problem Dimensions As mentioned earlier, misinformation is a problem with multiple dimensions (human, sociologi- cal, technological). We will mention here some of the most prominent ones closely related to the development of recommendation algorithms. Note that this does not aim to be an exhaustive list, and more research is needed to identify the various dimensions that may intersect between recommendation and misinformation. Content is an important dimension of the misinformation problem, and also a key one to consider in the creation, assessment and adaptation of recommendation algorithms. In the case of our proposed vision, we consider that the items to be recommended are online news. News can be present in various forms (as news paper articles, blog posts, social media posts, etc.) and discuss a wide range of topics (health, elections, disasters, etc.). These items are not only textual, but sometimes include information in different formats, like images or videos. Note that combinations of these formats are frequently used to propagate misinformation (e.g., a news title linked with an image from a different place, or from a different time). The framing of misinforming articles also varies between false news, rumours, conspiracy theories, misleading content [4]. Other important elements to consider about content are their origin (news outlets, social contacts, public figures, etc.) as well as the time when they are posted. Note that recency is particularly relevant on the recommendation of news items. All these aspects of the content need to be considered in the design and adaptation of news recommendation algorithms. 43 Users are a key dimension of the misinformation problem, and a core one of the functioning of recommendation algorithms. Multiple works have therefore studied the effect of different motivations [13], personalities [14], values [15], and emotions [6] and their effect on misinforma- tion, as well as the susceptibility of users [16]. For example, extroverts and individuals with high cooperativeness and high reward dependence are founded more prone to share misinformation [13]. Psychology also shows that individuals with higher anxiety levels are more likely to spread misinformation [17]. These aspects have demonstrated to influence users in the spreading of misinformation, hence it would be important to consider and capture them, as much as possible, during the construction of user profiles for recommendation. Platform and Network features. Platforms that distribute online information are designed differently and therefore facilitate the spread of misinformation in different ways. Content limitations (e.g., Twitter and its 280 character limitation for posts), ability to share information and select the subsets of users with whom such information is shared (sharing permissions), the ability to vote (e.g., Reddit) or to express emotions for content (e.g. Facebook), are important aspects of platform design that may shape the content, the way information spreads, and the social network structure. The typology and topology of the network structure is also a key factor of misinformation dynamics [1]. Other dimensions of the misinformation problem that may affect the development of rec- ommendation algorithms include: the global events or the developments happening in the world in a particular moment in time, the ethics of information (tensions between privacy and security, censorship, cultural differences, etc.), the presence of malicious actors including bots or crowdturfing (where crowdworkers are hired to support and propagate arguments or claims, simulating grassroots social movement), or the presence of checked facts within the information space. 2.2. Analysis of Recommendation Algorithms A key aspect of our research is to understand which recommendation techniques are more prone to suggest misinformative items to users. For this, we need to select a representative pool of algorithms to test against appropriate datasets, among the well-known collaborative filtering (CF), content-based (CB), and hybrid techniques [18]. The recommender systems literature is mostly focused on CF, since these methods can be applied to any domain, only requiring user-item interactions, not needing additional item features or metadata. Studying these methods may help us to better understand whether the user-item interactions could help, by themselves, to spread or avoid misinformative items, since these models neglect any information about the items and their misinformative features. Our main hypothesis is that, since CF algorithms tend to reproduce the tastes of the majority [19], they will probably follow the trend (either spreading or avoiding misinformative items) for those topics where an opinion is already established by most of the community. Nonetheless, it should also be noted that, since the user’s previous activity is also considered, this effect may not be so clear, paving the way for dynamic approaches that combine personalised with global models to avoid propagating misinformation. While CB techniques are less common in the area, techniques based on tags or ’short texts’ (such as reviews, or other user generated content) [20, 21] have attracted attention due to their 44 ability to consider the content of the items and adapt in more detail to the domain at hand. In this particular context, where news items are the ones to be recommended, it is important to analyse their textual content. This requests to study recommendation algorithms dealing with natural language and its inherent subtleties (synonyms, negation, sarcasm, etc.). An in-depth algorithmic survey is therefore required to better understand the impact of these techniques in the recommendation of misinformation. This includes classical and hybrid collaborative algorithms [22, 23], and more recent methods aimed at understanding the natural language by, for instance, using Neural Networks [24, 25]. When analysing recommendation algorithms it is also particularly important to study how to define user and item profiles [20]. Multiple representations could be considered: using the full content of the news items, a summary, or even some tags or categories assigned to each item. Explicit modelling of whether an item contains misinformation or not, although helpful, might put these techniques in an unfair position with respect to the CF methods described before. This information should therefore not be included in the studied models, at least in the first stages, so that a fairer assessment of the recommendation algorithms, and their tendency to spread misinformative items, can be conducted. To the best of our knowledge, the current state-of-the-art on recommender systems does not answer important questions around whether, how, and which, recommender systems tend to spread misinformation. Some connections can be made with the popularity-diversity bias analysed in the context of CF [26], where algorithms optimised for accuracy tend to be biased towards popular items, while returning not novel and not diverse recommendations. Similarly, CB algorithms are well-known for their portfolio effect and content overspecialisation [27]. It is therefore expected that if a user tends to consume misinformative items these algorithms might reinforce such content. However, since this may occur in a user basis, its global effect in the community needs to be properly analysed. 2.3. Human-centred Evaluation Traditionally, ‘‘relevancy’’ has been considered as the primary dimension to determine the quality of recommendations. We hypothesise that ‘relevancy’, and metrics that target user satisfaction, may not be the most effective ones when aiming to reduce the impact and spread of misinformation. Algorithms promoting a certain degree of cognitive dissonance and metrics that focus on computing a balanced degree of user satisfaction and discomfort may be more suitable to assess and combat misperceptions. Moreover, in order to detect and contrast the spread of misinformation, we need datasets where some items have already been labelled or identified as such. Examples of these datasets include the NELA-GT-2018 dataset 5 , which contains 713 articles collected between 02/2018- 11/2018 directly from 194 news and media outlets including mainstream, hyper-partisan, and conspiracy sources, or the more recent COVID-19 dataset generated by fact-checkers in more than 70 countries.6 However, since we want to apply personalisation algorithms, we also need user profiles and ratings, which are not available in these data sources. Considering that the research on news recommendation is increasing in the last years, an alternative that could 5 https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/ULHLCB 6 https://www.poynter.org/ifcn-covid-19-misinformation/ 45 be considered is to combine public datasets containing user profiles (such as NewsReel [28]) with the previously mentioned labelled datasets. However, it is very likely that the coverage and overlap between those datasets is not large, it might be even minimal. As we can see, the creation of datasets for the evaluation of recommender systems in the context of misinformation spreading requires very careful consideration for the selection and construction of items, profiles and ratings. Besides the problem of acquiring ground truth data, in order to derive user-centric evaluation metrics that could capture, at the same time, the degree of user satisfaction and cognitive discomfort, we need to come up with a satisfactory answer to the following question: how can we measure when the task has been successfully addressed? This translates into assessing that the misinformation has decreased, but, at the same time, that relevant and hopefully novel, diverse, and out-of-the-bubble content is still presented to the user. We foresee a combination of metrics and evaluation dimensions should be analysed to somehow overcome this problem, in particular, from recent works focused on fairness and transparency [29], information dynamics [30], beyond-accuracy metrics [31], and other biases and characteristics that should be considered [32, 33]. 2.4. Adaptation, Modification and Vigilance Combating misinformation is a complex task, and there is consensus in psychology literature that simply presenting people with corrective information is likely to fail in changing their salient beliefs and opinions, or may, even, reinforce them [34]. People often struggle to change their beliefs even after finding out that the information they already accepted is incorrect [35]. Nevertheless, some strategies have been found to be effective in correcting misperceptions, such as providing an explanation rather than a simple refute, exposing users to related but disconfirming stories, or revealing similarities with the opposing group [1]. We believe that the adaptation of existing algorithms to counter the misinformation recom- mendation problem should build on existing social science and psychology theory, such as the one presented above. We therefore need to better profile users in order to capture their motivations and behaviours when spreading misinformation. We also need to adapt recommen- dation algorithms in a way that recommendations are not solely based on reinforcing similarity, but on introducing small degrees of opposing views, or content from opposing groups, so that similarities between those groups are also highlighted, and explanations can be provided. More specifically, we could propose recommendation techniques that, instead of being focused on the notion of similarity between users and content, they could be based on the notion of similarity (for users) and dissimilarity (for content), i.e., on the idea of providing divergent views from similar users. This is based on the hypothesis that sharing disconfirming stories from users that hold a degree of similarity could help correcting misperceptions [11, 12]. Furthermore, assuming we would observe a potential trade-off between personalisation (or at least, some bias coming from some families of techniques) and misinformation propagation, we could propose hybrid recommender systems that address these issues, probably at the expense of lower accuracy or strength of user preferences. 46 3. Discussion Breaking the cycle that occurs when misinforming news break and are spread through social and digital platforms is paramount. One of the key components of this problem are the recommen- dation algorithms that power these digital platforms, often accused of feeding and reinforcing such cycle – because of their important role in exposing users to filtered subsets of information –. We argue in this paper that, instead of being solely part of the problem, recommendation algorithms could become part of the solution. This requires a better understanding of how their existing internal mechanisms reinforce the problem of misinformation, and how such mech- anisms could be adapted to counter it. Our hypothesis is that this adaptation should be built on existing social science and psychology theory. Studies from these fields have investigated this problem for years, and the behaviours and motivations that are more commonly associated with the spread of misinformation, as well as the potential mechanisms to palliate this effect. With this goal in mind, our research vision revolves around the four building blocks presented before: understanding the dimensions of the misinformation problem, analysing the impact of different recommendation strategies on spreading misinformation, adapting and modifying evaluation methods and metrics to properly assess these effects on users, and investigating how the algorithms could be modified to counter their misinformation behaviour. While our proposal is related with the principles of Fairness, Accountability and Transparency (FAT), it provides a step forward existing works, which are mainly focused on reducing biases and discrimination. Our aim is to address the problem of misinformation, but not by means of fact-checked information (which is expensive to obtain), or by detecting and containing malicious accounts and messages (which keep propagating), but by translating successful misinformation management strategies from social science research [11, 12] into computational recommendation models, while allowing to break the filter bubbles that tend to appear when using recommender systems [36]. Acknowledgments This work has been co-funded by H2020 Co-Inform (ID:770302) and HERoS (ID:101003606) projects and the Ministerio de Ciencia e Innovación (project reference: PID2019-108965GB-I00). Both authors contributed equally to this research. References [1] M. Fernandez, H. Alani, Online misinformation: Challenges and future directions, in: Companion Proceedings of the The Web Conference 2018, 2018, pp. 595–602. [2] D. T. Nguyen, N. P. Nguyen, M. T. Thai, Sources of misinformation in online social networks: Who to suspect?, in: MILCOM 2012-2012 IEEE Military Communications Conference, IEEE, 2012, pp. 1–6. [3] M. Del Vicario, A. Bessi, F. Zollo, F. Petroni, A. Scala, G. Caldarelli, H. E. Stanley, W. Quat- 47 trociocchi, The spreading of misinformation online, Proceedings of the National Academy of Sciences 113 (2016) 554–559. [4] C. Wardle, H. Derakhshan, Information disorder: Toward an interdisciplinary framework for research and policy making, Council of Europe report 27 (2017). [5] N. A. Karlova, K. E. Fisher, A social diffusion model of misinformation and disinformation for understanding human information behaviour (2013). [6] S. Vosoughi, D. Roy, S. Aral, The spread of true and false news online, Science 359 (2018) 1146–1151. [7] H. Allcott, M. Gentzkow, C. Yu, Trends in the diffusion of misinformation on social media, Research & Politics 6 (2019). [8] E. Pariser, The filter bubble: What the Internet is hiding from you, Penguin UK, 2011. [9] A. Bellogín, P. Castells, I. Cantador, Statistical biases in information retrieval metrics for recommender systems, Information Retrieval Journal 20 (2017) 606–634. [10] D. Jannach, L. Lerche, I. Kamehkhosh, M. Jugovac, What recommenders recommend: an analysis of recommendation biases and possible countermeasures, User Modeling and User-Adapted Interaction 25 (2015) 427–491. [11] R. K. Garrett, E. C. Nisbet, E. K. Lynch, Undermining the corrective effects of media- based political fact checking? the role of contextual cues and naïve theory, Journal of Communication 63 (2013) 617–637. [12] L. Bode, E. K. Vraga, In related news, that was wrong: The correction of misinformation through related stories functionality in social media, Journal of Communication 65 (2015) 619–638. [13] X. Chen, S.-C. J. Sin, ‘misinformation? what of it?’motivations and individual differences in misinformation sharing on social media, Proceedings of the American Society for Information Science and Technology 50 (2013) 1–4. [14] B. Zhu, C. Chen, E. F. Loftus, C. Lin, Q. He, C. Chen, H. Li, G. Xue, Z. Lu, Q. Dong, Individual differences in false memory from misinformation: Cognitive factors, Memory 18 (2010) 543–555. [15] L. S. Piccolo, A. Puska, R. Pereira, T. Farrell, Pathway to a human-values based approach to tackle misinformation online, in: International Conference on Human-Computer Interaction, Springer, 2020, pp. 510–522. [16] C. Wagner, S. Mitter, C. Körner, M. Strohmaier, When social bots attack: Modeling susceptibility of users in online social networks., in: # MSM, 2012, pp. 41–48. [17] M. E. Jaeger, S. Anthony, R. L. Rosnow, Who hears what from whom and with what effect: A study of rumor, Personality and Social Psychology Bulletin 6 (1980) 473–478. [18] F. Ricci, L. Rokach, B. Shapira (Eds.), Recommender Systems Handbook, Springer, 2015. URL: https://doi.org/10.1007/978-1-4899-7637-6. [19] R. Cañamares, P. Castells, Should I follow the crowd?: A probabilistic analysis of the effectiveness of popularity in recommender systems, in: K. Collins-Thompson, Q. Mei, B. D. Davison, Y. Liu, E. Yilmaz (Eds.), The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, ACM, 2018, pp. 415–424. URL: https://doi.org/10.1145/3209978.3210014. [20] M. de Gemmis, P. Lops, C. Musto, F. Narducci, G. Semeraro, Semantics-aware content-based recommender systems, in: F. Ricci, L. Rokach, B. Shapira (Eds.), Recommender Systems 48 Handbook, Springer, 2015, pp. 119–159. URL: https://doi.org/10.1007/978-1-4899-7637-6_4. [21] A. Tommasel, D. Godoy, Short-text feature construction and selection in social media data: a survey, Artif. Intell. Rev. 49 (2018) 301–338. URL: https://doi.org/10.1007/s10462-016-9528-0. [22] F. Garcin, B. Faltings, O. Donatsch, A. Alazzawi, C. Bruttin, A. Huber, Offline and online evaluation of news recommender systems at swissinfo.ch, in: A. Kobsa, M. X. Zhou, M. Ester, Y. Koren (Eds.), Eighth ACM Conference on Recommender Systems, RecSys ’14, Foster City, Silicon Valley, CA, USA - October 06 - 10, 2014, ACM, 2014, pp. 169–176. URL: https://doi.org/10.1145/2645710.2645745. [23] M. Karimi, D. Jannach, M. Jugovac, News recommender systems - survey and roads ahead, Inf. Process. Manag. 54 (2018) 1203–1227. URL: https://doi.org/10.1016/j.ipm.2018.04.008. [24] G. de Souza Pereira Moreira, D. Jannach, A. M. da Cunha, On the importance of news con- tent representation in hybrid neural session-based recommender systems, in: Ö. Özgöbek, B. Kille, J. A. Gulla, A. Lommatzsch (Eds.), Proceedings of the 7th International Workshop on News Recommendation and Analytics in conjunction with 13th ACM Conference on Recommender Systems, INRA@RecSys 2019, Copenhagen, Denmark, September 20, 2019, volume 2554 of CEUR Workshop Proceedings, CEUR-WS.org, 2019, pp. 18–23. URL: http://ceur-ws.org/Vol-2554/paper_03.pdf. [25] N. Babanejad, A. Agrawal, H. Davoudi, A. An, M. Papagelis, Leveraging emotion features in news recommendations, in: Ö. Özgöbek, B. Kille, J. A. Gulla, A. Lommatzsch (Eds.), Proceedings of the 7th International Workshop on News Recommendation and Analytics in conjunction with 13th ACM Conference on Recommender Systems, INRA@RecSys 2019, Copenhagen, Denmark, September 20, 2019, volume 2554 of CEUR Workshop Proceedings, CEUR-WS.org, 2019, pp. 70–78. URL: http://ceur-ws.org/Vol-2554/paper_10.pdf. [26] T. Zhou, Z. Kuscsik, J.-G. Liu, M. Medo, J. R. Wakeling, Y.-C. Zhang, Solving the appar- ent diversity-accuracy dilemma of recommender systems, Proceedings of the National Academy of Sciences 107 (2010) 4511–4515. [27] R. D. Burke, Hybrid recommender systems: Survey and experiments, User Model. User Adapt. Interact. 12 (2002) 331–370. URL: https://doi.org/10.1023/A:1021240730564. [28] A. Lommatzsch, B. Kille, F. Hopfgartner, L. Ramming, Newsreel multimedia at mediaeval 2018: News recommendation with image and text content, in: M. A. Larson, P. Arora, C. Demarty, M. Riegler, B. Bischke, E. Dellandréa, M. Lux, A. Porter, G. J. F. Jones (Eds.), Working Notes Proceedings of the MediaEval 2018 Workshop, Sophia Antipolis, France, 29-31 October 2018, volume 2283 of CEUR Workshop Proceedings, CEUR-WS.org, 2018. URL: http://ceur-ws.org/Vol-2283/MediaEval_18_paper_5.pdf. [29] M. D. Ekstrand, R. Burke, F. Diaz, Fairness and discrimination in recommendation and retrieval, in: T. Bogers, A. Said, P. Brusilovsky, D. Tikk (Eds.), Proceedings of the 13th ACM Conference on Recommender Systems, RecSys 2019, Copenhagen, Denmark, September 16-20, 2019, ACM, 2019, pp. 576–577. URL: https://doi.org/10.1145/3298689.3346964. [30] J. Sanz-Cruzado, P. Castells, Enhancing structural diversity in social networks by recommending weak ties, in: S. Pera, M. D. Ekstrand, X. Amatriain, J. O’Dono- van (Eds.), Proceedings of the 12th ACM Conference on Recommender Systems, Rec- Sys 2018, Vancouver, BC, Canada, October 2-7, 2018, ACM, 2018, pp. 233–241. URL: https://doi.org/10.1145/3240323.3240371. [31] P. Castells, N. J. Hurley, S. Vargas, Novelty and diversity in recommender systems, in: 49 F. Ricci, L. Rokach, B. Shapira (Eds.), Recommender Systems Handbook, Springer, 2015, pp. 881–918. URL: https://doi.org/10.1007/978-1-4899-7637-6_26. [32] A. Olteanu, C. Castillo, F. Diaz, E. Kiciman, Social data: Biases, methodological pitfalls, and ethical boundaries, Frontiers Big Data 2 (2019) 13. URL: https://doi.org/10.3389/fdata. 2019.00013. [33] L. Boratto, M. Marras, Hands on data and algorithmic bias in recommender systems, in: T. Kuflik, I. Torre, R. Burke, C. Gena (Eds.), Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization, UMAP 2020, Genoa, Italy, July 12-18, 2020, ACM, 2020, pp. 388–389. URL: https://doi.org/10.1145/3340631.3398669. [34] B. Nyhan, J. Reifler, Which corrections work, Research results and practice recommenda- tion (2013). [35] E. Thorson, Belief echoes: The persistent effects of corrected misinformation, Political Communication 33 (2016) 460–480. [36] P. Resnick, R. K. Garrett, T. Kriplean, S. A. Munson, N. J. Stroud, Bursting your (filter) bubble: strategies for promoting diverse exposure, in: A. Bruckman, S. Counts, C. Lampe, L. G. Terveen (Eds.), Computer Supported Cooperative Work, CSCW 2013, San Antonio, TX, USA, February 23-27, 2013, Companion Volume, ACM, 2013, pp. 95–100. URL: https: //doi.org/10.1145/2441955.2441981. 50