Serendipitous encounters in news recommender systems ofer users both pleasant surprises and opportunities to engage with a wider range of news content. Existing research has focused predominantly on algorithmic strategies for promoting serendipity, yet little is known about how users perceive and respond to such elements. This study shifts the focus from algorithmic performance to user experience, introducing a “surprise box” feature as an interface afordance for serendipitous news exploration. We conducted an online experiment ( = 118 ) comparing user feedback between a random news recommender and a personalized, similarity-based recommender system. Quantitative results indicated that users exposed to the personalized recommender were more likely to interact with the “surprise box”, suggesting their willingness to opt out of monotonous news feeds and seek serendipitous content. Qualitative feedback further revealed user awareness of the lack of diversity in the personalized feed; both groups, however, appreciated such an interface afordance for serendipity encounters. Our findings highlight that news users are not passive recipients but active agents seeking diverse content and showcase the efectiveness of interface afordances for promoting serendipitous engagement in the context of news recommendations.
News consumption plays a crucial role in democratic participation by keeping citizens informed
from diverse perspectives [
To address these concerns, serendipity, broadly defined as unexpected yet valuable discoveries [
CEUR
ceur-ws.org
The common approach to fostering serendipity is content-based, with one line of research focusing on
algorithmic strategies, such as collaborative filtering [
In response to this call, we introduce the “surprise box”, an intuitive interface afordance designed
for serendipitous news encounters. Inspired by the blind box marketing strategy, where the content
remains unknown until opened [25], this feature aims to replicate the experience of building curiosity
that culminates in a moment of delight upon discovery [26]. Its implementation combines both content,
which is generated randomly, and presentation, which aligns with the afordance principles outlined by
Smets et al. [
This “surprise box” thus provides an opportunity to investigate how users respond to (interface) designs for serendipity within a personalized news environment. Although content-based approaches have made progress in promoting serendipity algorithmically, much less is known about user feedback when presented with an interface afordance for serendipity. This brings us back to the central theoretical question: Will users actively break free from feedback loops of increasingly-similar content recommendations when given the opportunity to do so? Personalized recommenders tend to reinforce previous preferences, but users are not necessarily passive recipients. When recommendations become increasingly homogeneous and boring, users may seek more diverse content, especially when an intuitive pathway to serendipity is provided, such as the “surprise box”. To examine this, we conducted a longitudinal user study using a live news app and tested the following hypothesis: News users who engage with a highly personalized recommender interact with the “surprise box” more than those using a random news recommender.
To complement objective behavioral data, we also collected subjective user experience data through
closed- and open-ended questions. This dual approach addresses a methodological gap in serendipity
research, where validated instruments for measuring user experiences remain scarce [
Following open science practice, we pre-registered this experiment1 and obtained ethical approval from the university (2020-PCJ-12660). After that, we conducted a longitudinal online user study on our customized news application (for a screenshot, see Figure 1), which ran online from November 24th in 2023 until January 10th in 2024. For reproducibility, code and data are available on GitHub.2
We adopted a between-subjects design with two groups difering in whether the news feed was generated by random recommendation or personalized recommendation. The app was built using Python’s Flask framework based on an existing news app originally developed by Loecherbach and Trilling [28]. It ofered a selection of regularly updated articles from 11 live sources that broadly related to news, including news websites, magazines, blogs, and a cooking website, covering a wide range of topics such as politics, sports, lifestyle, etc. During our experiment, we obtained in total 44,743 unique 1Note that a). the current paper is only part of the pre-registered study; b). compared to the pre-registration, we paraphrased the research question and hypothesis in the section above for better clarity, but the direction of our expectation remains the same. Materials can be found on OSF: https://osf.io/2rtb4/?view_only=b7d34b148824436cacbdb9aaba104a05. 2App: https://github.com/ccs-amsterdam/3bij3. Data and analysis: https://github.com/zzzilinlin/exp_surprise_box. articles from 9 sources through RSS feeds, APIs, and web scrapers to formed our news pool (RTL 43.10%, nu.nl 26.43%, NOS 17.30%, Tweakers 3.80%, Libelle 3.62%, Flair 3.25%, GeenStijl 2.27%, and StukRoodVlees 0.16%, OneMoreThing 0.08%). Among them, 13,013 items were selected by the algorithms and presented to the users. This version of the app also had other features, namely the “surprise box”, item ratings, reading/sharing nudges, and a profile page for tracking the progress of completion, contributing to a fun environment for news [29].
For the purpose of this experiment, the app was set up with two conditions: random and personalized. In the random group, participants encountered nine randomly selected news articles each time they refreshed the page. The central article was concealed within the “surprise box”. In contrast, in the personalized condition (see Figure 1 for the news page following a series of sports-related clicks), the remaining eight news articles displayed randomly on each page were personalized recommendations based on the participant’s previous clicks. The “surprise box” in the center still contained a randomly selected news piece. In both conditions, the content and presentation of the “surprise box” remained constant, while its surrounding context varied systematically (i.e., random versus personalized).
The personalized recommendations were generated by calculating the soft cosine similarity of the word embeddings between the articles previously clicked by the participant and the newly arriving articles, a common approach used in the industry. For this task, we used the Amsterdam Embedding Model (AEM), a domain-specific word embedding model trained on Dutch news content. 3 This design was intentionally kept simple, as our goal was not to optimize personalized algorithms, but to isolate the efect of an interface afordance within a realistic yet controlled environment.
Participants used the app throughout the entire experiment, including for news engagements as well as completing the pre- and post-surveys. First, after reading about the study and giving consent, participants were asked to answer a short survey before creating an account to use the app. They were then expected to actively engage with the app for at least five days in two weeks, 4 such as reading and sharing news. They were also asked to rate the news-ness and relevance of the items that they have clicked on. Participants earned one point for each login, share, and feedback given on news items, with a daily cap of 15 points. To complete the experiment and receive a 10 euro shopping voucher, participants needed to accumulate a minimum of 70 points. The final step was to fill out a post-usage questionnaire where they also answered a few open-ended questions about the app.
After a pilot test with a student sample, we worked with a panel company and aimed to achieve a representative sample of the Dutch population for the experiment. In total, = 257 participants signed up and registered an account. After excluding 3 outliers, 5 225 actually engaged with the app (i.e., clicked and rated at least one article). Of these, 104 participants used the app for more than seven days (as pre-registered), and 118 for more than five days. To balance adherence to our pre-registration with statistical power, we included the 118 users.
We conducted attrition analyses comparing included ( = 118) and excluded ( = 107) users using -tests and 2-tests. No significant diferences were found in experimental condition, age, gender, education, news interest, political interest, or political eficacy. The only significant diference was political orientation: excluded users leaned more right-wing (left = 21, neutral = 13, right = 73) compared to included users (left = 48, neutral = 18, right = 52).
Among the 118 included completes, 45.76% were female, with an average age of 48.97 ( = 15.16 ), and 47.17% held an academic degree beyond the bachelor’s level. They were in general rather interested in news ( = 2.20, = 0.96; on a 7-point scale from −3 not at all interested to 3 very much interested). Together, they clicked on 7,826 items and rated 5,173 items.
Participants were randomly assigned to the random condition ( = 61) or the personalized condition ( = 57). These two conditions difered in how the news items were recommended, except for the “surprise box” item that was randomly chosen for everyone in both conditions.
The “surprise box” was the central item on the news page. We measured user engagement with the “surprise box” as the ratio of “surprise box” clicks to total item clicks ( = 0.14, = 0.15, on a scale from 0, no engagement with the “surprise box” at all, to 1, only engage with the “surprise box”). 3https://github.com/annekroon/amsterdam-embedding-model 4We pre-registered “at least seven days” as the minimum duration. Due to the lower than expected number of participants that met this criterion, we reduced it to “at least five days” for a slightly larger sample size. In the end, we also kept our app online for more than a month for a few participants who took longer to complete the study. 5All three participants had cheated the panel company’s system.
On a scale from 0 to 5 that allows half-point increments (e.g., 3.5), participants were asked to rate the news-ness ( = 3.02, = 1.57) and the relevance ( = 2.66, = 1.57) of all the items that they have clicked on.6 News-ness was measured to capture news users’ diverging perceptions of what constitutes “news” [30], while relevance, a key concept in recommender systems research, was measured to account for the possibility that content may be perceived as news yet still lack contextual or personal relevance. We measured them by asking participants to rate the extent to which they think the item is “news” and relevant, both of which are widely accepted measures in communication science studies. Participants were also asked to rate perceived diversity ( = 1.19, = 1.73) in the news content they saw on the app in the post-usage questionnaire.
We used diferent regression models to perform randomization checks, manipulation checks, and to test if being in diferent conditions leads to diferent levels of user engagement with the “surprise box”. First, our randomization checks confirmed that there were no significant diferences between the two experimental groups in all individual characteristics measured, including age, gender, education, interest in news, political orientation, political interest, and political eficacy. Then, to check whether our manipulation worked, we used user ratings of perceived diversity in the news content they saw on the app. The users in the random condition gave diversity scores 1.36 points higher (on a six-point scale) than those in the personalized recommendation condition ( = 0.33, < 0.001), confirming that our manipulation was successful.
Next, we examined how users responded to a consistently homogeneous news feed based on their previous clicks. As shown in Figure 2, users exposed to such strong similarity-based personalization had, on average, a 11 percentage point higher rate of user engagement compared to those in the random condition ( = 0.02, < 0.001); in other words, they clicked more frequently on the “surprise box”. Exploratory analyses indicated that this efect remained robust when controlling for users’ individual characteristics, such as demographics, news interest, and political traits. These findings supported our hypothesis that interacting with a highly personalized news recommender increases users’ engagement with serendipitous interface design.
To further understand how users perceived the recommender systems and the serendipitous element, we analyzed the open-ended questions at the end of the study. When asked what they liked about the app, many users in the random recommender group highlighted the variety of news content in terms of topics, types, and sources. They also appreciated the “surprise box” feature, either simply the fact that there was this “surprising” button, the content hidden behind it (which they found surprising and fun7), or the surprising experience itself. For example, one user (ID186, a 24-year-old male with a bachelor’s degree) described that “it was exciting to see what content you might get”. In contrast, only a few users in the personalized recommender group commented on the diversity of the news. They did, however, also appreciate the “surprise box” and gave more detailed explanations. Some users, similar to the other group, noted their curiosity about what the “surprise” content might be (e.g., ID108, a 50-year-old man with a bachelor’s degree; ID285, a 49-year-old woman with a high school degree, etc.), while others complemented the content behind the “surprise box”: “still of good quality and depth” (ID156, a 43-year-old woman with a bachelor’s degree), indeed “unexpected” (ID84, a 34-year-old woman with 6These two scores showed a high correlation ( = 0.79, < 0.001). As two similar yet distinct concepts, they could be used for robustness checks in diferent models. 7Even though for this group all items, including those behind the “surprise box”, were randomly curated and presentation was the only diference. a master’s/above degree), or just liking “diferent surprising article every time” (ID373, a 30-year-old woman with a high school degree). Promisingly, one user stressed that it allowed her to be exposed to news she “would not normally read” (ID340, a 37-year-old woman with a bachelor’s degree).
Group diferences were more apparent when users discussed their dislikes and suggestions for future improvements. Those in the random condition were generally positive. A fair number of them indicated that there was nothing they disliked and nothing that needed to be changed. A few suggestions were made about the content curation: One user (ID162) preferred to be able to set preferences in advance, and another (ID331) suggested introducing “more personalization”. In contrast, almost all users in the personalized recommender group complained about the lack of diversity in their news feed, for instance, “lots of the same type of articles, day after day” (ID161), “a bit monotonous” (ID236), “too repetitive, too much of one topic” (ID260). Some users gave specific examples, such as “too many weather articles” (ID360) and “mainly focused on woke, LGBTQ+, and cooking”(ID349). Many users in this group later explicitly suggested “more diversity” as a desired improvement for the future.
We have established that the “surprise box”, a combination of both presentation and content, was an efective interface afordance for serendipity. We then further wondered if the presentation itself would influence one’s perception of content (i.e., the random news item), for example, the perceived news-ness and relevance of the content.
For this exploratory investigation, we only selected data from the random condition, where having an item in a “surprise box” meant merely concealing the content, instead of a diferent way of algorithmic curation. Multi-level models were used to explore what contributed to users’ news perception, focusing on whether an item was presented through the “surprise box”. We controlled for content/textual features, namely Formality and Factuality.8 To measure these two textual features, we used two fine-tuned BERTje models from Lin et al. [31] to classify each sentence within a news article as either formal or factual, then calculated the percentage of formal sentences and the percentage of factual sentences for each article. In the multi-level models, we also took into account individual variability by adding random efects for both the intercept and the slope associated with textual features for diferent users.
As shown in Table 1, news items hidden behind the “surprise box” were consistently rated lower news-ness scores and deemed less relevant, compared to items presented normally. This factor was statistically significant in all models ( < 0.001).
First, this study contributes to serendipity research by designing and empirically testing an interface
afordance, namely the “surprise box”, which reflects the feature repository proposed by Smets et al. [
Second, this study adds to the literature on news recommender systems by highlighting the active role of users. Drawing on both qualitative and quantitative insights, we conclude that news users have a strong preference for diversity, are sensitive to content homogeneity, and actively seek serendipitous content when intuitive afordances are available. Rather than reinforcing concerns about the risks of news recommendations [e.g., 34, 35], we demonstrated a promising way toward diversity from the user’s perspective. 8These two scores were significantly correlated and therefore added separately in diferent models for robustness checks.
Methodologically, our longitudinal online user study using a customized app with live-scraped news allowed us to observe real-life user behavior while maintaining experimental control over the platform. By combining objective observational data with subjective user experience data, we present a usercentric approach for studying serendipity in news recommender systems [27]. A key recommendation for longitudinal online user studies of this kind is to carefully balance internal validity with ecological validity.
This study also has several limitations. First, the retention rate was relatively low, and we had to exclude a considerable number of users from the analysis. Our analysis showed that the only significant diference between excluded and included users was political orientation, with excluded participants tending to lean more conservative. However, we do not believe this substantially biased our findings for three main reasons: a). political orientation was not a significant predictor in any model in the analysis; b). the groups were balanced on related variables such as political interest and eficacy; and c). the core finding aligned with our theoretical framework of content monotony and user engagement, not political attitudes. That said, we encourage future user studies to anticipate potential issues with low retention and to implement strategies to improve participation, such as ofering higher rewards or improving the app’s aesthetic appeal.
Second, we conceptually operationalized the “surprise box” as a combination of both presentation and content, which introduced some interpretive ambiguity. This overlap complicated our ability to disentangle the efects of each component and was the reason we relied only on data from the random condition in the exploratory analysis. Nevertheless, the current setup still treated the “surprise box” as a distinct interface afordance present in both conditions, allowing us to interpret users’ general preference for opting out of a feedback loop of increasingly similar content. To strengthen internal validity, a logical next step would be to adopt a factorial design that separates presentation from content. Future studies could also explore additional variations, such as changes in visual appearance or levels of content diversity. Alternatively, serendipitous afordances could also be tested in non-experimental settings.
Third, our personalization implementation was relatively simple, relying on soft cosine similarity of word embeddings. While this approach serves as a reproducible and interpretable baseline for studying user interaction with serendipitous interfaces, future research could incorporate more advanced personalization algorithms to improve generalizability and better reflect the real-world recommender systems. In practice, many news organizations use content-based personalization as just one part of their news oferings, often combining it with popularity-driven recommendations [ 36] or strong editorial control that allows manual selection or overriding of the top recommendations [37]. Taken together, this suggests two complementary directions for future work: one focusing on simplified designs to better establish causality and disentangle diferent mechanisms; and one that, in contrast, aiming for greater realism in the design to avoid overestimating real-world efects.
Despite its limitations, this study is an important starting point for exploring how interface afordances can promote serendipitous encounters in news recommender systems. It demonstrates that usercentered design can ofer a promising path toward more democratic and engaging news environments.
This work is part of a project that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No. 947695).
During the preparation of this work, the author(s) used Overleaf’s built-in AI tool for grammar and spelling check. After using these tool, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.
Journal of Documentation 68 (2012) 684–705. [22] J. Yang, Efects of popularity-based news recommendations (“most-viewed”) on users’ exposure to online news, Media Psychology 19 (2016) 243–271. [23] S. Nagulendra, J. Vassileva, Providing awareness, explanation and control of personalized filtering in a social networking site, Information Systems Frontiers 18 (2016) 145–158. [24] F. Garcin, B. Faltings, O. Donatsch, A. Alazzawi, C. Bruttin, A. Huber, Ofline and online evaluation of news recommender systems at swissinfo.ch, in: Proceedings of the 8th ACM Conference on Recommender Systems, 2014, pp. 169–176. [25] G. F. N. Mvondo, F. Jing, K. Hussain, What’s in the box? Investigating the benefits and risks of the blind box selling strategy, Journal of Retailing and Consumer Services 71 (2023) 103189. [26] M. Kim, C. A. Vogt, B. J. Knutson, Relationships among customer satisfaction, delight, and loyalty in the hospitality industry, Journal of Hospitality & Tourism Research 39 (2015) 170–197. [27] B. P. Knijnenburg, M. C. Willemsen, Z. Gantner, H. Soncu, C. Newell, Explaining the user experience of recommender systems, User Modeling and User-adapted Interaction 22 (2012) 441–504. [28] F. Loecherbach, D. Trilling, 3bij3–developing a framework for researching recommender systems and their efects, Computational Communication Research 2 (2020) 53–79. [29] S. M. Schöbel, A. Janson, M. Söllner, Capturing the complexity of gamification elements: A holistic approach for analysing existing and deriving novel gamification designs, European Journal of Information Systems 29 (2020) 641–668. [30] Z. Lin, News in perspective: What is news to news users in the digital age?, Ph.D. thesis, University of Amsterdam, Amsterdam, The Netherlands, 2025. [31] Z. Lin, K. Welbers, S. Vermeer, D. Trilling, Beyond discrete genres: Mapping news items onto a multidimensional framework of genre cues, in: Proceedings of the International AAAI Conference on Web and Social Media, volume 17, 2023, pp. 542–553. [32] A. Smets, J. Vannieuwenhuyze, P. Ballon, Serendipity in the city: User evaluations of urban recommender systems, Journal of the Association for Information Science and Technology 73 (2022) 19–30. [33] E. G. Toms, Serendipitous information retrieval, in: Proceedings of the 1st DELOS Network of Excellence Workshop on Information Seeking, Searching and Querying in Digital Libraries, 2000, pp. 11–12. [34] E. Pariser, The filter bubble: How the new personalized web is changing what we read and how we think, Penguin, 2011. [35] N. J. Stroud, Polarization and partisan selective exposure, Journal of Communication 60 (2010) 556–576. [36] D. Trilling, R. Dubèl, R. Kiddle, A. C. Kroon, Z. Lin, M. Simon, S. Vermeer, K. Welbers, M. Boukes, What is popular gets more popular? Exploring over-time dynamics in article readership using log data, Journalism Studies 25 (2024) 2051–2071. [37] M. Borchgrevink-Braekhus, «Det er ikke plass til alt på internett»: Algoritmestyrte forsider og redaksjonelle vurderinger, Norsk Medietidsskrift 29 (2022) 1–17.