Public Service Media, Diversity and Algorithmic Recommendation

Abstract

Public Service Media (PSM) websites are an interesting case for the implementation of recommender systems for media personalization, as the PSM organizations need to balance the optimization of exposure with traditional but ill-defined PSM policy goals such as fairness, viewpoint diversity and transparency. Furthermore, the mathematical logic of recommender system needs to be adapted to the legacy broadcasting scheduling and publishing strategies and procedures. Finally, as the PSM organizations step into new territories, domestication and adaption of the recommender system technologies must take place while PSM organizations try to embrace the new knowledge and new professions associated with recommender systems. Based on 25 in-depth interviews conducted from December 2016 to April 2019, this paper presents a cross-European analysis of the implementation of recommender systems in nine European public service media organizations from eight countries. The findings indicate that PSM organizations, although viewing personalisation as competitive necessity, approach recommendation systems with hesitation in order to maintain core PSM-values in the online environment. Furthermore, although the collaborative ifltering chosen by the PSM organizations indicate a usercentered approach, curation systems on top of recommender systems re-install a broadcaster-centric approach.

CCS CONCEPTS

General and reference Surveys and overviews; Empirical studies; Information systems Collaborative filtering; Personalization.

INTRODUCTION

Recommender systems have been build to serve individual interests by selecting and filtering content, while public service broadcasting traditionally have aimed to reach the whole population with content that editors think is beneficial for citizens. The intention is to mediate a public sphere. Bringing the two entities sounds as a contradiction in terms. Public service media have a special position in the media landscape, Copyright ' 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). as they in return for public funding have obligations to produce and promote content that e.g. strengthen local language and culture, help inclusion and societal cohesion, foster democratic processes, reflect diversity in opinions and worldviews. In the age of broadcasting, priority was given to this type of content by scheduling strategically before or after popular content. When content exposure is personalized via a recommender system, the question emerges: To which extend will the algorithm and the editors / data curators recommend content that reflects individual user interests if these do not align with the content obligations? The question is relevant for more than the user experience. The funding and political legitimacy of public service media is closely connected with the content obligations. If this content is not being exposed to users, the legitimacy of the institution may be endangered. Conversely, the programming and scheduling strategy of the PSB and PSM organizations have always been to ofer popular content, such as music, sports and entertainment to listeners, viewers and users. How will PSM organizations approach this balancing?

Based on 25 in-depth interviews with practitioners, like project leaders, product owners, programmers, data scientists and data curators the paper shows how they adopt PSM core values to algorithmic recommender systems. The paper is structured as follows: Below we introduce the special requirements to recommender systems that the PSM context poses. After that we provide an overview of PSM interviewees’ approaches to recommender system implementation. Finally, we point at a fundamental paradox related to the contextual rationality of recommender systems. 2

BACKGROUND

Public service media is often linked to a set of ideals—the PSM remit [ 27 ]. UNESCO [ 43 ] lists as principles for public service broadcasting: ∙ Universality ∙ Diversity ∙ Independence ∙ Distinctiveness

Public service media want to be agenda-setters with a mission of e.g. strengthening social cohesion, facilitating democratic discussions, supporting national culture and language, ensuring freedom of speech, supporting cultural diversity [ 9 ]. Diversity and inclusion are today important values, e.g. in the BBC [ 2 ]. Nissen [ 31 ] argues that “influencing the listener’s or viewer’s choices, and thus media consumption pattern is the very reason why public media were established and why their existence has been upheld even in times of abundant media supply.” (p. 69)

Idealized, the principle for selecting and recommending content to the audience is defined by a consensus about what citizens should learn and know to participate in a society, cf. the classic BBC motto “enlighten, educate and entertain” [ 35 ]. In reality, the production and exposure in the public service mass media has been and still is shaped by conflicting societal interests, opportunistic possibilities and media politicians’ expectations.

Traditional algorithmic recommendation, i.e. collaborative ifltering (CF) [ 18, 19, 23 ], on the other side addresses the individual user to optimize the exposure of content and, ideally, protecting him / her from information overflow. Typically, no communicative intention is associated with the recommendation. Instead, key performance indicators (KPIs) are being measured.

Will PSM organizations stick to the broadcaster-centric approach that prioritizes certain content or will they embrace the customer-choice logic of personalization? A key to this question is how diversity is created in the recommendations: Is it defined by an editor defined to reflects diferent viewpoints (source diversity) [ 29 ], or is calculated to reflect the global diversity of items available for recommendation, the diversity within the recommended items (intra-list diversity) [ 8 ], or does it take the point of departure in the users’ profile [ 46 ]? Due to the special content obligations, diversity is a sensitive topic in public service media.

In media politics, the diversity of media content providers, ownership and media outlets is a measure of a well-functioning democracy [ 17, 29 ]. In the area of news and media, a specific concern has grown that algorithmic recommender systems will lead to a loss of exposure diversity [ 29 ]. Subsequently, much discussion has revolved around filter bubbles [ 32 ] caused by collaborative filtering [ 5, 14, 30 ]. Suggestions for regulating exposure diversity have been proposed by [ 6, 7, 16, 17 ] but also criticized [ 39 ]. Recently, measurements in a news context have however showed that algorithms are as good at providing diversity as human editors [ 28 ] and that the problem of filter bubbles may be overestimated [ 47 ]. Fletcher [ 13 ] ifnds that news repertoires of users who find news via search engines have a more diverse and balanced news consumption, while [ 12 ] reaches a more mixed conclusion. However, diversity exposure concerns are still associated with algorithmic recommender systems [ 10, 15 ].

While diversity in itself is a problematic concept [ 22 ], achieving diversity in algorithmic recommendations is a problem with importance for user experience and cross-selling of products [ 8, 24 ]. Sørensen and Schmidt [ 39 ] indicates that there might exist a gap between how diversity is understood and treated as a mathematical concept in recommender systems [ 8 ], and how media policy and editors construct the concept [ 29 ]. Where the former seldom takes the semantic context in consideration, the core parameter in the latter is contextually and socially defined view-point diversity [ 1 ].

The case of recommender systems in the context of public service media has earlier been discussed at RecSys in a keynote by Berry from Canadian CBC [ 3 ], and by Fields, Jones and Cowlishaw from the BBC [ 11 ]. Furthermore, Van den Bulck and Moe [ 45 ] compare public service media organizations’ diferent strategies in respect to personalization, with the Belgium VRT and the Norwegian NRK as case studies. Po¨chhacker et al . [ 34 ] analyze the implementation of a recommender system in a German context, discussed in [ 36 ]. Sørensen [ 37 ] analyzes early attempts to personalize PSM online services. Bodo [ 4 ], Kunert and Thurman [ 25 ], Thurman and Schiferes [ 41 ] analyze private and public media use of personalization. 3

METHODOLOGY

Using convenience sampling, a set of case were selected representing both on-going projects and recommender systems in production. A sampling criteria is that the PSM organization is a member of European Broadcasting Union (EBU) and located in Europe. Among the 54 EBU member organizations located in Europe1, we selected nine cases of development or implementation of recommender systems. Developers, data scientists and project leaders from the nine PSM organizations were interviewed via semi-structured in-depth interviews.

Interviewees were selected based on their direct involvement in the development or implementation of a recommender system. The strategic management level has thus not been included. The interview data was collected between December 2016 and April 2019. Additionally, developers, data scientists and project leaders from five other European PSM organizations were interviewed: ARD (Germany), FranceTv (France), MDR (Germany), Radio France (France), RAI (Italy), YLE (Finland).

The interviews, 25 in total, were conducted as semi-structured expert interviews based on a question guide [ 26 ]. The question guide was adjusted to reflect both the development of the projects over time, as well as the insights gained during the research. The interviews were audio-recorded (in total: 820 minutes), transcribed and topic coded.

Most PSM organisations implement content-based filtering (C-B) and collaborative filtering (CF). A number of PSM organisation also report to use a diversity module for recommendations. Finally, some PSM organisations use the recommender system as part of a curation system, controlled by editors and scheduling staf via business rules. 4

ANALYSIS

Reflecting our inductive and praxis-oriented approach, the topics discussed by and with the interviewees can be grouped in the following categories.

∙ Organizational Opportunities and Challenges ∙ Diversity, Exposure and PSM Obligations ∙ Editorial control and curation vs. automation ∙ KPIs - evaluation of exposure 1https://www.ebu.ch/about/members accessed April 2, 2019 ∙ Trust and Transparency ∙ Build or buy?

In the following lines we will summarize the interviewees’ viewpoints on each of the topic areas, and discuss the perspectives. 4.1

Organizational Opportunities and

Challenges

A notion of strategic necessity was e.g. expressed December 2016 by the EBU President Jean-Paul Philippot at a Recommender Systems workshop for EBU members: “While PSM still detain the largest audience in the EU, they should take these trends [increasing use of on-demand services] into account if they want to keep an advantage over Netflix and other competitors. This migration implies new tools such as eficient recommender systems, new skills, and new mindsets.” 2 The objective here is the competitiveness of public service media.

Being able to ofer personalization is seen as an important value proposition. Users expect personalization features, e.g. My page or Recommended to you to see the PSM web service or app as a competitive alternative to other media providers. Furthermore, the technical departments in the PSM organizations want to build know-how on data-driven business, and management wants to explore demand-driven production of content and position the organization in the media market.

However, the data analysts, programmers and data curators complain that it is dificult to explain recommender systems to editors and journalists. E.g. the value of good meta data is not always clear to the editorial staf. Many editorial assistants may simply copy-paste generic meta data descriptions from one episode item to the next resulting in low-quality recommendations. 4.2

Diversity, Exposure and PSM

Obligations

The improvement of user loyalty and content exposure, particular of long-tail content and under-performing content, is a 2RTBF - EBU Workshop on Algorithms and ety 12-13/12 2016, accessed September 10, 2018 https://www.ebu.ch/contents/events/2016/12/big-data-initiativeworkshop-algorithms-and-society.html Sociat main motive for implementing recommender systems. The exposure of the less used content is part of the PSM obligation of promoting diversity. But interviewees are also concerned of being accused for producing filter bubbles. Recommender systems for PSM organizations must put a special emphasis on diversity. In some cases, software modules have been build to calculate diversity of the recommendations3, but the idea of regulating diversity via algorithms is generally received reluctantly. Most interviewees have an editorial understanding of diversity, arguing that the diversity is ensured by the global set of items (videos) available in the system. In one case, a business rule ensures that the user, after having seen two pieces of content of the same genre is presented to two other genres, e.g. two fiction videos are followed by documentary and news videos. The solution is by the interviewee described as “simplistic”, as the genre may not indicate diversity. The issue of viewpoint diversity [ 1 ] was expressed as a problem by one interviewee, discussing whether a system is needed to make sure that a users watches diferent political parties’ video presentations prior to elections.

Clear and oficial definitions of diversity are typically missing, leaving data curators on their own. Defining diversity has traditionally been an organizationally sensitive task, as diversity is understood diferently by media politicians, by PSM professionals and users. The absence of a formal definition has traditionally enabled PSM editors to navigate between conflicting ideas of diversity without direct confrontations. 4.3

Editorial control and Automation

During the two years of interviewing, we have observed a shift in PSM staf understanding of recommender systems: Initially skeptically approached as automated black boxes, recommender systems are now typically used as part of a curation system. Furthermore, editors have gained control over the recommendations, e.g. by setting business rules for certain slots on the page, by the tagging of content items or by giving priority to certain content in the keyword search system. Many systems also have a manual override functionality. 3the PEACH recommender system, produced in collaboration between BR (Germany), RTS (Switzerland) and EBU https://peach.ebu.io/ technical/tutorials/algorithms/diversified/ accessed October 1, 2018

A challenge for data curators is the manual configuration of business rules. Several interviewees report experiences with the content-based filtering that recommended out-dated sports- or news shows or inappropriate content: e.g. hard debating on Islam after a satirical video on the same topic. Maintaining the high level of editorial quality in the automated recommendation is a challenging task.

The business rules, the curation and the placement of the algorithmic recommended content typically at the bottom of the page or after video play-back imply that users only to a limited degree experience personalisation at the pages. There is only few opportunities to show users new, unknown content. As novelty of the content often is a criteria for recommendation the pool of relevant content is reduced. Furthermore, as content similar to the previous watched is given priority such as the next episode of a show / season the window for diversity exposure is even smaller, interviewees report. 4.4

KPIs - the evaluation of exposure

With personalized recommender systems, PSM organizations inherit a concept that has been invented for commercial purposes [ 33 ]. As PSM organization have other objectives than e.g. improving click-through-rates, other KPIs must be defined. However, defining PSM-specific KPIs, e.g. for the diversity of the recommendations, social cohesion or the public service value is a challenging and controversial task. Interviewees thus report that they mainly use KPIs internally in the development team to benchmark algorithm performance. The management levels of PSM organizations have so far not shown much interest in KPIs from the recommender systems; broadcast-related KPIs have a much bigger managerial attention. PSM managements typically see the algorithmic recommendation projects as experiments. In many organizations editors and journalists however have access to dashboards on online performance of the site and the content. 4.5

Trust and Transparency

Interviewees all stress that a PSM organization has a particular duty in explaining users how recommendations are being made, and how personal data is being treated. While the latter, at least from a legal perspective is covered by GDPR, the former is a bigger challenge. While content-based recommendations more easily can be explained to users, recommendations based on more complex data processing are hard to explain in details to users. The problem of explaining recommendations is discussed in general terms e.g. by [ 42 ], but what specifically are PSM users’ expectations concerning transparency? Interviewees agree that the PSM context is particular sensitive as public service media stands out as trusted institutions [ 38, 40 ]. Thus full transparency or at least openness on recommendations is needed, however competitive interests may prevent PSM organisations from sharing too many details about their recommender systems. However, the special position of PSM organisations in the media markets [ 21, 31, 44 ] may call for media regulation to monitor the level of transparency of the PSM recommender systems, to protect users’ rights of transparency [ 20 ] and the competitive balance between commercial media and publicly funded media. 4.6

Build or buy?

Most of the PSM organizations in our survey have either build their in-house systems with help of open-source tools, or commissioned external developers to develop a systems that later can be transferred to the PSM organization. Only one PSM organization (DR in Denmark) has chosen a proprietary solution ofered by an external provider. DR argues that the PSM organization neither is nor should be a technology organization; it should focus on content.

As reasons for building own systems, interviewees provide diferent reasons. For some, the in-house competence building that enables fine-tuned configuration, control and development is important. Other interviewees emphasize the complex organizational (data-) structures and the special requirements of PSM as motivation for custom-built systems. Some interviewees explained that no usable external solutions were available at project start, thus stimulating them to develop own systems. In the case of Estonia, the special grammar of the Estonian language forced developers to build their own stemming tools. Finally, ensuring the technical independence of PSM organization in a future with IP-based media consumption plays for many organizations a role for building their own systems [ 40 ].

The lack of systems addressing specific PSM requirements (e.g. diversity) prompted two software projects to build generic PSM recommender solutions: The PEACH project was initiated by Bayerische Rundfunk (BR) in Germany, and continued as a EBU project4. In parallel, a system was developed by the Belgian PSM organization RTBF. In both cases, the interest from other PSM organizations to implement has however been limited. 5

DISCUSSION AND CONCLUSION

PSM organizations are hesitant to personalize the content exposure. Organizational acceptance of algorithmic recommendation is one barrier, the broadcast work-flow, lack of good meta-data and technical skills are other barriers. With algorithmic recommendation legacy public service broadcasting organizations are stepping into unknown territory. On this journey, the potential of loosing the core value proposition, namely the agenda-setting, is felt as a danger. Movements in the wilderness are thus careful, as PSM organizations are vulnerable to political and commercial accusations of being too commercial, too personal, too little distinctive or for lack of diferent types of diversity could hurt the organizations severely.

However, a more profound reason for the hesitation could be suggested, based in a paradox: The public service mission is - as stated - to enlighten, inform and educate. The role of entertainment has either been the sugar-coating on 4https://peach.ebu.io accessed 2019-08-18 the pill of information, part of the provision of diversity or the justification of the license fee. Based on the apparent rationality of the enlightenment, information and education that PSM organizations are based on, PSM should warmly embrace recommender systems. These systems should be perfectly rational tools for eficient dissemination of knowledge. The hesitation we notice among the PSM organization could either be explained by the insensitivity of well-known recommender systems to the type of rationality desired, or it could point to the ambiguity and fragility of the enlightenment that PSM organizations claim to promote. The absence of clear definitions of diversity and of the relation between the personal and the societal could point to the latter. Not defining or operationalizing these concepts can however ironically be a strategically fortunate position for PSM organizations.

References

[1] Christian Baden and Nina Springer. 2017 . Conceptualizing viewpoint diversity in news discourse . Journalism: Theory, Practice & Criticism 18 , 2 (feb 2017 ), 176 - 194 . https://doi.org/10.1177/ 1464884915605028

[2] BBC. 2017 . Diversity and Inclusion Strategy 2016- 20 . ( 2017 ). http://downloads.bbc.co.uk/diversity/pdf/diversity-andinclusion-strategy- 2016 .pdf

[3] Christopher Berry . 2018 . Recommending social cohesion . In Proceedings of the 12th ACM Conference on Recommender Systems - RecSys '18 . ACM Press, New York, New York, USA, 3 - 3 . https://doi.org/10.1145/3240323.3267110

[4] Bal´azs Bodo. 2018 . Means, not an end (of the world) - the customization of news personalization by European news media . ( 2018 ). https://ssrn.com/abstract=3141810

[5] Engin Bozdag and Jeroen van den Hoven. 2015 . Breaking the filter bubble: democracy and design . Ethics and Information Technology 17 , 4 (dec 2015 ), 249 - 265 . https://doi.org/10.1007/ s10676-015-9380-y

[6] Mira Burri . 2016 . Nudging as a Tool of Media Policy . In Nudging - Possibilities, Limitations and Applications in European Law and Economics . Springer International Publishing, Cham, 315 - 341 . https://doi.org/10.1007/978-3- 319 -29562-6 16

[7] Mira Burri and Natali Helberger . 2015 . Public Service Media and Exposure Diversity: Introduction. International journal of communication 9 ( 2015 ), 1319 - 1323 .

[8] Pablo Castells , Neil J. Hurley , and Saul Vargas . 2015 . Novelty and Diversity in Recommender Systems . In Recommender Systems Handbook . Springer US, Boston, MA, 881 - 918 . https://doi.org/ 10.1007/978-1- 4899 -7637-6 26

[9] EBU. 2007 . EBU viewpoint: The UNESCO Convention on Cultural diversity . http://www.ebu.ch/CMSimages/en/ CulturalDiversity{ } 03 { }07{ }E{ }tcm6- 50247 .pdf

[10] Sarah Eskens , Natali Helberger, and Judith Moeller . 2017 . Challenged by news personalisation: five perspectives on the right to receive information . Journal of Media Law 9 , 2 (jul 2017 ), 259 - 284 . https://doi.org/10.1080/17577632. 2017 .1387353

[11] Ben Fields , Rhianne Jones , and Tim Cowlishaw . 2018 . The Case for Public Service Recommender Algorithms . ( 2018 ), 22 - 24 pages. https://piret.gitlab.io/fatrec2018/program/fatrec2018-fields.pdf

[12] Seth Flaxman , Sharad Goel, and Justin M. Rao . 2016 . Filter Bubbles, Echo Chambers, and Online News Consumption . Public Opinion Quarterly 80 , S1 ( 2016 ), 298 - 320 . https://doi.org/10. 1093/poq/nfw006

[13] Richard Fletcher and Rasmus Kleis Nielsen. 2018 . Automated Serendipity. Digital Journalism 6 , 8 (sep 2018 ), 976 - 989 . https: //doi.org/10.1080/21670811. 2018 .1502045

[14] Mario Haim , Andreas Graefe, and Hans-Bernd Brosius . 2017 . Burst of the Filter Bubble? Digital Journalism (jul 2017 ), 1 - 14 . https://doi.org/10.1080/21670811. 2017 .1338145

[15] B. Hallinan and T. Striphas . 2016 . Recommended for you: The Netflix Prize and the production of algorithmic culture . New Media & Society 18 , 1 (jan 2016 ), 117 - 137 . https://doi.org/10. 1177/1461444814538646

[16] Natali Helberger . 2011 . Diversity by design . Journal of Information Policy 1 ( 2011 ), 441 - 469 .

[17] Natali Helberger . 2015 . Public Service Media - Merely Facilitating or Actively Stimulating Diverse Media Choices? Public Service Media at the Crossroad . International Journal of Communication 9 ( 2015 ), 17 . http://ijoc.org/index.php/ijoc/article/view/2875

[18] Jonathan L Herlocker , Joseph A Konstan, Al Borchers, and John Riedl . 1999 . An algorithmic framework for performing collaborative filtering . In 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR 1999 . Association for Computing Machinery , Inc, 230 - 237 .

[19] Jonathan L. Herlocker , Joseph A. Konstan , and John Riedl . 2000 . Explaining collaborative filtering recommendations . InProceedings of the 2000 ACM conference on Computer supported cooperative work - CSCW '00 . ACM Press, New York, New York, USA, 241 - 250 . https://doi.org/10.1145/358916.358995

[20] High-Level Expert Group on Artificial Intelligence . 2019 . Ethics Guidelines for Trustworthy AI . Technical Report. European Commission , Brussels. https://ec.europa.eu/newsroom/dae/ document.cfm?doc{ }id=60419https://ec.europa.eu/digitalsingle-market/en/news/ethics -guidelines-trustworthy-ai

[21] Karol Jakubowicz . 2007 . Public Service Broadcasting in the 21st Century. What Chance for a New Beginning? In From Public Service Broadcasting to Public Service Media, Gregory Ferrell Lowe and Jo Bardoel (Eds.). Nordicom, G¨oteborg Universitet, Kunga¨lv, 29 - 49 .

[22] KENNETH JUNGE . 1994 . Diversity of ideas about diversity measurement . Scandinavian Journal of Psychology 35 , 1 (mar 1994 ), 16 - 26 . https://doi.org/10.1111/j.1467- 9450 . 1994 .tb00929. x

[23] Joseph A Konstan , Bradley N Miller , David Maltz, Jonathan L Herlocker, Lee R Gordon , and John Riedl . 1997 . GroupLens: applying collaborative filtering to Usenet news . Commun. ACM 40 , 3 (mar 1997 ), 77 - 87 . https://doi.org/10.1145/245108.245126

[24] Matevˇz Kunaver and Tomaˇz Poˇzrl . 2017 . Diversity in recommender systems: A survey . Knowledge-Based Systems 123 ( 2017 ), 154 - 162 . https://doi.org/10.1016/j.knosys. 2017 . 02 .009

[25] Jessica Kunert and Neil Thurman . 2019 . The Form of Content Personalisation at Mainstream , Transatlantic News Outlets: 20102016 . Journalism Practice ( 2019 ), 1 - 22 . https: //doi.org/10.1080/17512786. 2019 .1567271

[26] Steinar Kvale . 2007 . Doing Interviews . SAGE, Los Angeles, London, New Delhi, Singapore, Washington DC.

[27] Gregory Ferrell Lowe and Taisto Hujanen . 2003 . Broadcasting & Convergence: New Articulations of the Public Service Remit . RIPE@ 2003 . Nordicom, G¨oteborg. 335 pages.

[28] Judith Mo¨ller, Damian Trilling, Natali Helberger, and Bram van Es. 2018 . Do not blame it on the algorithm: an empirical assessment of multiple recommender systems and their impact on content diversity . Information, Communication & Society 21 , 7 (jul 2018 ), 959 - 977 . https://doi.org/10.1080/1369118X. 2018 .1444076

[29] Philip M. Napoli . 2011 . Exposure Diversity Reconsidered . Journal of Information Policy 1 ( 2011 ), 246 - 259 .

[30] Tien T. Nguyen , Pik-Mai Hui , F. Maxwell Harper , Loren Terveen, and Joseph A. Konstan . 2014 . Exploring the filter bubble . In Proceedings of the 23rd international conference on World wide web - WWW '14 . ACM Press, New York, New York, USA, 677 - 686 . https://doi.org/10.1145/2566486.2568012

[31] Christian S Nissen . 2006 . No public service without both Public and Service - Content provision between the Scylla of populism and the Charybdis of elitism. In Making a Difference: Public Service Broadcasting in the European Media Landscape , Christian S Nissen (Ed.). John Libbey Publishing, Eastleigh, Chapter 5 , 65 - 82 .

[32] Eli Pariser . 2011 . The filter bubble : what the Internet is hiding from you . Penguin Books.

[33] Don Peppers and Martha Rogers . 1993 . The One to One Future: Building Relationships One Customer at a Time . Currency Doubleday , New York.

[34] Nikolaus P ¨ochhacker, Marcus Burkhardt, Andrea Geipel, and Jan-Hendrik Passoth . 2017 . Interventionen in die Produktion algorithmischer O¨ ffentlichkeiten: Recommender Systeme als Herausforderung fu¨r ¨offentlich-rechtliche Sendeanstalten . kommunikation@ gesellschaft 18 ( 2017 ), 25 .

[35] John C W Reith . 1924 . Broadcast Over Britain . Hodder and Stoughton , London.

[36] Jan-Hinrik Schmidt , Jannick Kirk Sørensen, Stephan Dreyer, and Uwe Hasebrink . 2018 . Wie ko¨nnen Empfehlungssysteme zur Vielfalt von Medieninhalten beitragen? Perspektiven fu¨r o¨ffentlichrechtliche Rundfunkanstalten . Media Perspektiven 11 ( 2018 ), 522 - 531 .

[37] J. K. Sørensen . 2013 . PSB goes personal : The failure of personalised PSB web pages . MedieKultur 29 , 55 ( 2013 ), 43 - 71 . http://ojs.statsbiblioteket.dk/index.php/mediekultur/article/ view/7993

[38] Jannick Kirk Sørensen and Sokol Kosta . 2019 . Before and After GDPR: The Changes in Third Party Presence at Public and Private European Websites . In WWW '19 Companion Proceedings of the The Web Conference 2019 . ACM. https: //doi.org/10.1145/3308558.3313524

[39] Jannick Kirk Sørensen and Jan-Hinrik Schmidt . 2016 . An Algorithmic Diversity Diet? Questioning Assumptions behind a Diversity Recommendation System for PSM . In RIPE@ 2016 . Antwerp, Belgium. http://ripeat.org/library/2016

[40] Jannick Kirk Sørensen and Hilde Van den Bulck. 2018 . Public service media online, advertising and the third-party user data business . Convergence: The International Journal of Research into New Media Technologies (aug 2018 ). https://doi.org/10. 1177/1354856518790203

[41] Neil Thurman and Steve Schifferes . 2012 . The Future of Personalization at News Websites . Journalism Studies 13 , 5 - 6 (oct 2012 ), 775 - 790 . https://doi.org/10.1080/1461670X. 2012 .664341

[42] Nava Tintarev and Judith Masthoff . 2015 . Explaining Recommendations: Design and Evaluation . In Recommender Systems Handbook . Springer US, Boston, MA, 353 - 382 . https: //doi.org/10.1007/978-1- 4899 -7637-6 10

[43] UNESCO . 2001 . Public Broadcasting Why? How? Technical Report. UNESCO , Paris. 1-28 pages. http://unesdoc.unesco.org/ images/0012/001240/124058eo.pdf

[44] Hilde Van den Bulck. 2008 . Can PSB Stake its Claim in a Media World of Digital Convergence?: The Case of the Flemish PSB Management Contract Renewal from an International Perspective . Convergence 14 , 3 ( 2008 ), 335 - 349 . https://doi.org/10.1177/ 1354856508091085

[45] Hilde Van den Bulck and Hallvard Moe . 2017 . Public service media, universality and personalisation through algorithms: mapping strategies and exploring dilemmas . Media, Culture & Society (oct 2017 ), 016344371773440 . https://doi.org/10.1177/ 0163443717734407

[46] Michele Zanitti , Sokol Kosta, and Jannick Sørensen . 2018 . A UserCentric Diversity by Design Recommender System for the Movie Application Domain . In Companion of the The Web Conference 2018 on The Web Conference 2018 - WWW '18 . ACM Press, New York, New York, USA, 1381 - 1389 . https://doi.org/10.1145/ 3184558.3191580

[47] Frederik J Zuiderveen Borgesius , Damian Trilling, Judith Moeller, Bal´azs Bod´o, Claes H de Vreese , and Natali Helberger . 2016 . Should we worry about filter bubbles? Internet Policy Review 5 , 1 ( 2016 ).