The pervasiveness of manipulative designs - i.e. dark patterns - in everyday applications and their impact on users is raising concerns among policymakers and scholars. These designs use techniques to nudge users to make decisions that they would not make if fully informed, causing them various types of harm. Yet, the ubiquitous mechanisms used are intertwined with other platform afordances, which makes it hard for users to perceive the manipulation. While understanding the efects of manipulative designs on users is key to the design of protective countermeasures, the academic community faces several methodological challenges. How can we investigate the efects of what is often not perceived by the users? In this paper, we reflect on the challenges inherent to the study of manipulative designs through three case studies. We describe key challenges and discuss methodological insights relevant to the empirical study of manipulative designs.
CEUR ceur-ws.org
Manipulative designs — i.e., dark patterns1 — are design interface elements that try to steer,
coerce, or manipulate users into decisions that, if well informed, they would not make [
Hence, we argue for the necessity to assess the methodological implications underlying the study of manipulative designs to advance the field and support practitioners and policymakers with a solid body of knowledge to inform countermeasures. In this paper, we articulate the methodological challenges inherent to the study of online manipulation, through a brief review of the limitations of prior work. We illustrate these challenges through three case studies, reflecting on how each method we used embodied these challenges. We discuss these challenges with regards to the methodological choices we make. This paper contributes to methodological reflexivity in the context of manipulative design research, in particular when studying the efects of manipulative designs on users. We open a conversation in the community about how to creatively overcome methodological challenges inherent to manipulative design research.
In this section, we gather the main methodological limitations and challenges discussed by
scholars in research papers on manipulative designs. Various methods from diverse perspectives
— legal [16, 17], design [18, 19, 20, 21], or computer sciences [
One of the biggest challenges in the study of manipulative designs is the search for ecological validity, to account for the situatedness of human action and contingency on contextual factors. Indeed, the specificity of every setting and context limits the transferability of insights into the efects of manipulative design. Bielova et al. [9] explained how their findings on cookie consent banners are hard to translate to other banners. The authors acknowledged their setting might be diferent from the usual settings in which users interact with consent banners, including the attentional limitations that users experience. The fact that fake experimental settings do not embody the same consequences that the actual interaction on the user experience is a known limitation [30]. With manipulative designs, Tuncer et al. [24] warn readers to take their results cautiously because the simulated e-commerce website used to test scarcity cues did not incorporate real financial incentives. Similar limitations are reported by van Nimwegen and de Wit [23]. To overcome this, Utz et al. [13] partnered with a company to study manipulative designs in a real context and Nouwens et al. [14] used a browser extension to test the manipulative designs in participants’ everyday interactions. Moser et al. [31] envision industrial partnerships as a necessary step for future research on manipulative designs, given the complexity of the topic. We share some of their vision that underlies why ecological validity is key on this topic; while companies have the resources and real settings to test manipulative strategies on users as “attackers”, researchers’ resources as “defenders” are much more limited.
An additional problem arising from the dependency on context is the dissociation between the efects of manipulative designs and the rest of the platform’s afordances. Bongard-Blanchy et al. [8] explained that when asked about felt manipulation, participants mentioned other intertwined afordances instead. Therefore, several studies acknowledge their results to be valid only in the specific contexts and with the specific manipulative designs studied [ 9, 32, 23, 31]
Another reported limitation relates to the population and sample strategies used. Many studies on manipulative designs use crowdsourcing platforms to collect data on users. While these systems have some merits in some contexts, their use in the context of online manipulation might decrease the study’s validity and lead to inconclusive results. Berens et al. [10] explained that by trusting the study, crowdsourced participants would increase the acceptance of cookies. Many studies report sampling limitations. While trying to sample a broad range of participants’ profiles, Bongard-Blanchy et al. [8] and van Nimwegen and de Wit [23] fell short in including older adults. Avolicino et al. [27] even presented their oldest cohort as ”above 35 years old”. Several studies acknowledge limitations related to convenience samplings — via crowdsourcing platfoms or recruitment within universities – as leading to the inclusion of an overall younger and more tech-savvy population, even when the researchers aimed for generalisation [10, 8, 31]. Noteworthily, with the few aforementioned exceptions that included some diversity in age ranges, the set of literature that includes the analysis of other variables as mediators of the efects of manipulative designs is scarce Luguri and Strahilevitz [28], Bongard-Blanchy et al. [8], van Nimwegen and de Wit [23], Avolicino et al. [27]; and none of these included any population that embody vulnerabilities, which is often labelled as ”hard-to-reach” Shaghaghi et al. [ 33].
Many methodological challenges in our subfield seem to arise from the quest for generalizability,
as a major part of the community adheres to quantitative research approaches. Yet it is worth
reflecting on whether that path is an adequate starting point and which research objectives
are meaningful. The literature review by Gray et al. [15] shows a wide use of qualitative
methods from a designerly perspective. Examples of qualitative methods to understand users
include Maier and Harr [7]’s focus group study, providing first insights into the perceptions
of manipulative designs on users. Gray et al. [
We reflect on the challenges inherent to the study of manipulative designs, by reporting on three methodological approaches we have used. We describe key challenges and methodological insights leveraging their strengths and acknowledging their limitations given the specific nature of manipulative designs. These cases are not exhaustive and their selection relates to our research experiences. Our analysis emerges from a willingness to examine the choices we knowingly or unknowingly make when studying manipulative designs. This analysis can be a useful resource for design scholars toward more reflexive methodological choices.
Looking back at our work, we started our empirical explorations on the experience of manipulative designs by using traditional research methods like interviews. These felt like a natural ifrst step in exploring the topic from the users’ perspective, with the opportunity to collect rich accounts of experiences and understand the contextual aspects involved. However, we quickly faced dilemmas around how to approach the topic with users, due to the sensitivity of the matter and their potential vulnerability, but also the exploration of an issue they might not be fully aware of. How to discuss the efects of something interviewees do not perceive? Is it even possible without prompting awareness, or is there an adequate amount of prompting? What are the implications - on the data and the participants - of bringing online manipulation to awareness through our studies? These questions led us to experiment with specific interview techniques, which we reflect on below. After these studies, we turned to other disciplines beyond social sciences, seeking diferent methodological lenses to approach online manipulation with users. Design methods and their underlying epistemologies appeared worthwhile in this regard, as reflected in our third case study on the use of magic machines workshops.
Critical incidents is an interview technique that explores moments in which the user has experienced a significant “problem” or an “incident” to understand the context of a particular phenomenon [35]. This technique is helpful for interviewees to be situated in a specific context.
We conducted interviews using the critical incidents technique to understand felt manipulation in the presence of manipulative designs online. We asked young adults at social exclusion risk (N=22) to describe situations in which they did something they did not want to do online; they felt deceived or, ultimately, manipulated. By using the laddering technique to investigate these incidents, we explored real scenarios in which manipulative designs impacted our participants. We employed a similar protocol in a study with teenagers (N=6).
Regarding manipulative designs, critical incidents are useful to understand felt manipulation
as a proxy of the efects of manipulative designs, given their relationality [
Some limitations of the critical incidents technique are common to interview methods in general. They rely on interviewee’s memory and tend to highlight the most prominent or recent experiences. The interview guide is likely to prompt users on a topic. Factors of social desirability might also play a role, with either a desire to provide the interviewer with the information they seem to expect or to silence some experiences to avoid the embarrassment of reporting having been manipulated online. For instance, a teenager explained he experienced nagging techniques with pop-ups having a hidden close button (X) while watching movies online. He sometimes “failed” to close the pop-up and has to “endure” the forced advertisement because he failed. He said he does act to cope with it and should “learn to pay more attention the next time”. Feelings of embarrassment, self-blame, and social desirability are frequently triggered by online manipulation. Reporting that one has been manipulated is sensitive, even more so when invited to report the most ’critical instances’ and when the research focuses on non-normative collectives and populations in situations of vulnerability.
The use of probes is a common method to gain insights into a particular phenomenon by showing artefacts to participants to react and reflect on. For manipulative designs, probes have been proven useful given the subtlety of the mechanisms that make users unaware of the influence. Maier and Harr [7] explored students’ perceptions by showing them manipulative designs during a focus group. Bongard-Blanchy et al. [8] used fake interfaces in a survey to understand users’ perceptions. Similar to Bhoot et al. [26], Di Geronimo et al. [36] exposed participants to manipulative designs to analyse their “blindness” towards these elements.
As a follow-up of our study described in 3.1, we used interface screenshots including manipulative designs as probes to gain insights into the relationship between participants and manipulative designs. We selected the interfaces by contexts according to the ontology of manipulative designs by Gray et al. [37] and to represent a variety of mainstream platforms and manipulative elements. They were shown simultaneously on a digital board: ”I provide you with the following images. Please have a look and tell me what you think”. As participants reacted to the probes, the interviewer followed up with multiple ”why?” questions to elicit participants’ awareness (or lack of) of manipulation. The interviewer then asked ”What do you do when you encounter such an interface?” (to understand the behavior and potential vulnerability to the manipulative element) and ”What happens after?” (to understand the potential harms).
The use of probes allows participants to reflect on the efects of those interfaces, by commenting on the specific context they find them in and in which specific ways they interact with these elements, sometimes in other contexts. This has been useful to document new contexts of online manipulation afecting specific groups. For instance, we showed teenagers e-commerce interfaces with nagging — continuously asking the user to make decisions they already took [37]— and changing the hierarchy of choice architecture [37]— but they recognised the strategy as very common in other contexts closer to their routines — e.g., ofline fast-food kiosks and pirated-content websites. Without showing the probes and relying on critical incidents, we might not have elicited these contexts that add new knowledge to the community. Probes are also helpful with populations in which psychological harm may arise easily (e.g., low self-esteem). We observed how low-educated young adults experienced nervousness when asked about online manipulation using the critical incident technique. The subtlety of these mechanisms made participants feel embarrassed: a researcher was asking them about things they had not considered until that moment. Using probes of manipulative designs changed that feeling and created relief, as they recognised almost all the interfaces presented.
Only relying on probes of manipulative designs has some important limitations. Directly showing manipulative designs to participants can have a priming efect, possibly leading to participants not reporting their actual encounters or perceptions about these patterns. Thus, the researcher’s ability to avoid priming into the aspects of the interface that can be manipulative is key if the intention is to elicit perceptions of manipulative designs. We thus suggested combining it with another method to triangulate data.
Magic machines workshops [38] are speculative co-creation methods, which place the participants in a “magical context” so they do not face the hurdles of talking about technology, while expressing their needs freely. This method can be used to understand users’ experiences with manipulative designs. We conducted three “magic machines” workshops with older adults (N=31) to understand their experiences with online manipulation. As a warm-up activity, the participants introduced themselves briefly by sharing a superpower they would like to have. In the setting stage, we invited participants to recall moments in which they were manipulated online. We mentioned diferent contexts and specific examples of felt manipulation: “When was the last time you were on internet/phone and did something you did not want to, did not expect or perhaps even regretted? (...) Maybe you gave your privacy unintentionally? Subscribing for an unexpected service or newsletter (...) you could spend more money than you expected, paying unexpected extra fees or buying something in a rush you didn’t really need. (...)”
We used sentence completion [39] to facilitate a spontaneous elicitation of an experience of manipulation: “The last time I did something I did not want to or did not initially intend to do on the internet was... because...” Participants were then asked to create a magic machine to protect themselves from the manipulative experience they reported, using craft material. The instructions emphasised that participants are in control and there are no limits to their imagination. Participants are invited to explain their machines and answer questions.
In the context of manipulative designs, the low-tech and seemingly not-so-serious aspects of the method helped avoid the hurdles of interacting with a platform and, more broadly, talking about technology, which could be problematic for populations with less self-eficacy levels. As the method aims to empower users, participants are more at ease and less prone to social desirability biases. We aimed to shift the power dynamics and bring the magical atmosphere as early as the warm-up, asking them to choose a superpower. With older adults, we observed how the session’s dynamic helped them talk about their experiences. Participants created a magic machine that protects them on their own terms. Through this group activity, participants recognised themselves in other stories and provide their perspectives.
This technique, applied as a user research method, tend to reveal major issues encountered by participants when interacting with manipulative designs, rather than specific issues at the interaction level. On some occasions, participants talk about usability problems rather than the efects of manipulation. For this reason, the onboarding of participants and prompt into the problem is key. While introducing manipulation as part of the workshop instructions might look like a limitation, we aimed to situate participants into moments of felt manipulation (and avoid having them build machines to overcome of-topic usability-related issues).
The evolution in the discourse on manipulative designs around 2018 [40] suggests the existence
of a first wave of methodological approaches in our field. Researchers initially focused on
understanding what manipulative designs were, how they were used, and where they were
found; after all, studying the efect of a phenomenon requires first scoping and defining it.
Therefore, the first wave of papers set the ground with analyses of artefacts [ 41, 42], taxonomies
[
Adopting a systemic view by using ecologically valid methods studying manipulative design in-situ, in a variety of contexts. The efects and harms of manipulative designs are situated and contextual and call for in-situ approaches and the documentation of a wider range of contexts of online manipulation, beyond the widely studied cookie banners, e-commerce, and social media contexts. In our case studies, we gave participants the freedom to choose the context of manipulation they wished to comment on, either by leaving it entirely open (critical incidents and magic machines) or by ofering a variety of choices (probes). This made sense when considering manipulative designs from a user- or harm-centered perspective. We want users to talk about their lived experiences, so inquiring about a context/platform they do not use is irrelevant. Sometimes, researchers will prefer to narrow down a study to a specific context and population, which is an equally valuable endeavor. This aspect also calls for a systemic view to understand people’s ecologies. Beyond mentioning widely known field methods drawing from ethnographic or sociological approaches, our case studies brought reflections on the engagement of the researcher in the topic. For instance, the prior knowledge required to conduct and analyse the data of studies on manipulative designs. To prepare for our interviews, we got acquainted with the platforms that embedded manipulative designs. It felt essential to engage in inquiry and to know how to disentangle elements in the subsequent analysis. Long-term evaluation paradigms, to understand the antecedents, correlates, and consequences incurred by manipulative designs. The understanding of the antecedents, correlates, and consequences of manipulative designs calls for a temporal perspective in our approaches, beyond the main paradigm focusing on momentary interactions. While not all methods in our cases aforded to talk about antecedents or long-term efects, the critical incidents or magic machines used a retrospective approach for participants to vividly recall and discuss these aspects. Using probes or fake experimental material is less likely to inform temporal aspects of the relation between users and manipulative designs.
Conscious methodological choices around the “unawareness” of online manipulation, which is often pervasive and subtle. Online manipulation is a phenomenon that (some) users are likely to be unaware of. This potential unawareness is a key challenge, which calls for a reflective and creative approach to research. Related fields face similar challenges, in particular usable security and privacy [46, 47]. On the one hand, we can choose to prompt the participants about the specific context of manipulation. This approach has been dominant in the qualitative inquiries proposed by our community so far, and we do not deny its merits in investigating specific research questions. In our case studies, we initially tried hard to resist prompting participants to not influence the types of experiences they would share. In our instructions, we used the harms as an entry point and proxy for detecting and framing online manipulation, e.g., When is the last time you did something you did not want to or did not initially intend to do on the internet?. While successful to some extent, our pilot tests have shown that the ambiguity of this prompt created confusion or led to of-topics conversations (e.g., on usability-related issues), which would unethically waste the time of the participants and the researchers alike. Eventually, we added examples to support framing the topic of the interviews or workshop sessions. In the probe study, we did not label the screenshots of interfaces as manipulative, and used the laddering technique to unveil the awareness of manipulation.
At times, prompting the participants on the topic of manipulative designs triggered negative feelings of embarrassment, shame, or guilt. Imagine how uncomfortable it can be to realise during an interview that you have been manipulated regularly without being aware? How can we acknowledge the sensitivity of this topic and ”prompt” with care? On the other hand, attempts to study manipulative designs without bringing them to the awareness of the participants were mostly made in experimental settings [9, 10, 24], and often reflected various uses of deception as a research mechanism. Here again, various ethical implications are at stake, and we will discuss in section 4.1.4 why they are more salient in the context of manipulative designs. In other domains, the ”invisible” or ”intimate” aspects that are hardly accessible to the researcher can be researched through first-person methodologies like auto-ethnographies [48, 49]. While it can be used to address some research questions to manipulative designs, these approaches do not support overcoming the ”unawareness” challenge nor is the researcher usually representative of populations likely to be more vulnerable to online manipulation. Caring about the populations most impacted by online manipulation. Due to their nature, specific populations might be more impacted by manipulative designs and likely more vulnerable and harder to reach when researching technology. Our case studies all engaged with populations at risk of vulnerability, which led to invaluable insights into the experience of manipulation that prior work did not address. We argue for a focus on care to conduct research that goes beyond observing generalisable causes. Some methods can help create a nuanced knowledge of the impacts rather than solely proving a change in behaviour caused by manipulative designs. To generate this knowledge, the study of manipulative designs not only requires knowledge of the contexts but also interest and sensitivity towards the populations that are more vulnerable: they are more likely to be impacted and with fewer resources to recover from those impacts. The choice of methods should account for the more vulnerable populations because they are more impacted. As an illustration, while deception studies are worthwhile in many domains related to usable security (e.g., phishing interventions [50]), deception also entails the risk of harming research participants - a consideration that cannot be avoided when working with vulnerable populations and ”practicing what we preach”. Adopting the posture of a sensitive researcher [51], making eforts to include the voices of hard-to-reach populations, and seeking methodologies that balance the power between researchers and participants (e.g., magic machines workshops) should be key elements informing our methodological approaches.
Naturally, some of these considerations are not novel and resonate with prior work in the user experience domain [30], in particular, the situatedness of experiences [52] or the need for longterm evaluation paradigms [53]. Others are particularly salient in research on manipulative designs, like unawareness or vulnerability. We hope these considerations can inspire and support designers in making instrumental judgments [54] about the methods used and the value of the insights they bring forth. In future work, we encourage the community to leverage the interdisciplinary strengths of HCI, combining forces and drawing inspiration from social sciences, design, computer sciences, and legal sciences – to only name a few relevant disciplines. We also encourage the community to rethink how companies take part in our research endeavors and to make them active actors in investigating the efects of manipulative designs over time, in situated contexts, and with special care to foster the empowerment of vulnerable populations. Naturally, there is no one-size-fits-all approach to overcome all the challenges we shed light on. Reflecting on these considerations and making them transparent is an important step forward. While limited, the reflections we shared illustrate how to think of research trade-ofs as well as our positionality as researchers in this field. By caring and tackling the uneasy, by making eforts to include hard-to-reach populations and acknowledging the vulnerability aspects involved, we can create a more robust body of knowledge on the efects of manipulative designs on users.
This research is funded by the Luxembourg National Research Fund (FNR, grant no. IS/14717072 Deceptive Patterns Online (Decepticon)). We thank the anonymous reviewers for their detailed constructive feedback. [7] M. Maier, R. Harr, Dark design patterns: An end-user perspective, Human Technology 16 (2020) 170–199. doi:10.17011/ht/urn.202008245641. [8] K. Bongard-Blanchy, A. Rossi, S. Rivas, S. Doublet, V. Koenig, G. Lenzini, ”i am definitely manipulated, even when i am aware of it. it’s ridiculous!” - dark patterns from the end-user perspective, in: Designing Interactive Systems Conference 2021, ACM, Virtual Event USA, 2021, p. 763–776. URL: https://dl.acm.org/doi/10.1145/3461778.3462086. doi:10.1145/ 3461778.3462086. [9] N. Bielova, L. Litvine, A. Nguyen, M. Chammat, V. Toubiana, E. Hary, The efect of design patterns on (present and future) cookie consent decisions (????). [10] B. M. Berens, H. Dietmann, C. Krisam, O. Kulyk, M. Volkamer, Cookie disclaimers: Impact of design and users’ attitude, in: Proceedings of the 17th International Conference on Availability, Reliability and Security, ACM, Vienna Austria, 2022, p. 1–20. URL: https: //dl.acm.org/doi/10.1145/3538969.3539008. doi:10.1145/3538969.3539008. [11] A. Chaudhary, J. Saroha, K. Monteiro, A. G. Forbes, A. Parnami, “are you still watching?”: Exploring unintended user behaviors and dark patterns on video streaming platforms, in: Designing Interactive Systems Conference, ACM, Virtual Event Australia, 2022, p. 776–791.
URL: https://dl.acm.org/doi/10.1145/3532106.3533562. doi:10.1145/3532106.3533562. [12] A. Monge Rofarello, K. Lukof, L. De Russis, Defining and identifying attention capture deceptive designs in digital interfaces, in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, ACM, Hamburg Germany, 2023, p. 1–19. URL: https://dl.acm.org/doi/10.1145/3544548.3580729. doi:10.1145/3544548.3580729. [13] C. Utz, M. Degeling, S. Fahl, F. Schaub, T. Holz, (un)informed consent: Studying gdpr consent notices in the field, in: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, ACM, London United Kingdom, 2019, p. 973–990.
URL: https://dl.acm.org/doi/10.1145/3319535.3354212. doi:10.1145/3319535.3354212. [14] M. Nouwens, I. Liccardi, M. Veale, D. Karger, L. Kagal, Dark patterns after the gdpr: Scraping consent pop-ups and demonstrating their influence, in: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 2020, p. 1–13. URL: http://arxiv.org/abs/2001.02479. doi:10.1145/3313831.3376321, arXiv:2001.02479 [cs]. [15] C. M. Gray, L. Sanchez Chamorro, I. Obi, J.-N. Duane, Mapping the landscape of dark patterns scholarship: A systematic literature review, in: Designing Interactive Systems Conference, ACM, Pittsburgh PA USA, 2023, p. 188–193. URL: https://dl.acm.org/doi/10. 1145/3563703.3596635. doi:10.1145/3563703.3596635. [16] J. Gunawan, C. Santos, I. Kamara, Redress for dark patterns privacy harms? a case study on consent interactions, in: Proceedings of the 2022 Symposium on Computer Science and Law, ACM, Washington DC USA, 2022, p. 181–194. URL: https://dl.acm.org/doi/10. 1145/3511265.3550448. doi:10.1145/3511265.3550448. [17] C. Matte, N. Bielova, C. Santos, Do cookie banners respect my choice? measuring legal compliance of banners from iab europe’s transparency and consent framework, arXiv:1911.09964 [cs] (2020). URL: http://arxiv.org/abs/1911.09964, arXiv: 1911.09964. [18] L. Sánchez Chamorro, K. Bongard-Blanchy, V. Koenig, Ethical tensions in ux design practice: Exploring the fine line between persuasion and manipulation in online interfaces, Pittsburgh PA USA, 2023. [19] L. Nelissen, M. Funk, Rationalizing dark patterns: Examining the process of designing privacy ux through speculative enactments (2022). URL: http://www.ijdesign.org/index. php/IJDesign/article/view/4117. doi:10.57698/V16I1.05. [20] S. S. Chivukula, J. Brier, C. M. Gray, Dark intentions or persuasion?: Ux designers’ activation of stakeholder and user values, in: Proceedings of the 2018 ACM Conference Companion Publication on Designing Interactive Systems, ACM, Hong Kong China, 2018, p. 87–91.
URL: https://dl.acm.org/doi/10.1145/3197391.3205417. doi:10.1145/3197391.3205417. [21] S. S. Chivukula, C. Watkins, L. McKay, C. M. Gray, “nothing comes before profit”: Asshole design in the wild, in: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, ACM, Glasgow Scotland Uk, 2019, p. 1–6. URL: https: //dl.acm.org/doi/10.1145/3290607.3312863. doi:10.1145/3290607.3312863. [22] A. Curley, D. O’Sullivan, D. Gordon, B. Tierney, I. Stavrakakis, The design of a framework for the detection of web-based dark patterns, in: ICDS 2021: The 15th International Conference on Digital Society, Nice, France (Online), 2021, p. 8. [23] C. van Nimwegen, J. de Wit, Shopping in the Dark: Efects of Platform Choice on Dark Pattern Recognition, volume 13304 of Lecture Notes in Computer Science, Springer International Publishing, Cham, 2022, p. 462–475. URL: https://link.springer.com/10.1007/ 978-3-031-05412-9_32. doi:10.1007/978- 3- 031- 05412- 9_32. [24] R. Tuncer, A. Sergeeva, K. Bongard-Blanchy, V. Distler, S. Doublet, V. Koenig, Running out of time(rs): efects of scarcity cues on perceived task load, perceived benevolence and user experience on e-commerce sites, Behaviour Information Technology (2023) 1–19. doi:10.1080/0144929X.2023.2242966. [25] J. Tiemessen, H. Schrafenberger, G. Acar, The time is ticking: The efect of limited time discounts on consumers’ buying behavior and experience, in: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, ACM, Hamburg Germany, 2023, p. 1–11. URL: https://dl.acm.org/doi/10.1145/3544549.3585735. doi:10.1145/3544549. 3585735. [26] A. Bhoot, M. A. Shinde, W. P. Mishra, Towards the identification of dark patterns: An analysis based on end-user reactions, in: IndiaHCI ’20: Proceedings of the 11th Indian Conference on Human-Computer Interaction, ACM, Online India, 2020, p. 24–33. URL: https://dl.acm.org/doi/10.1145/3429290.3429293. doi:10.1145/3429290.3429293. [27] S. Avolicino, M. Di Gregorio, F. Palomba, M. Romano, M. Sebillo, G. Vitiello, AI-Based Emotion Recognition to Study Users’ Perception of Dark Patterns, volume 13516 of Lecture Notes in Computer Science, Springer International Publishing, Cham, 2022, p. 185–203. URL: https: //link.springer.com/10.1007/978-3-031-17615-9_13. doi:10.1007/978- 3- 031- 17615- 9_ 13. [28] J. Luguri, L. J. Strahilevitz, Shining a light on dark patterns, Journal of Legal Analysis 13 (2021) 43–109. doi:10.1093/jla/laaa006. [29] J. Aagaard, M. E. C. Knudsen, P. Baekgaard, K. Doherty, A game of dark patterns: Designing healthy, highly-engaging mobile games, in: CHI Conference on Human Factors in Computing Systems Extended Abstracts, ACM, New Orleans LA USA, 2022, p. 1–8. URL: https://dl.acm.org/doi/10.1145/3491101.3519837. doi:10.1145/3491101.3519837. [30] C. Lallemand, V. Koenig, Lab testing beyond usability: Challenges and recommendations for assessing user experiences - jux, 2017. URL: https://uxpajournal.org/ lab-testing-beyond-usability/. [31] C. Moser, S. Y. Schoenebeck, P. Resnick, Impulse buying: Design practices and consumer needs, in: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, ACM, Glasgow Scotland Uk, 2019, p. 1–15. URL: https://dl.acm.org/doi/10.1145/ 3290605.3300472. doi:10.1145/3290605.3300472. [32] C. Voigt, S. Schlögl, A. Groth, Dark Patterns in Online Shopping: of Sneaky Tricks, Perceived Annoyance and Respective Brand Trust, volume 12783 of Lecture Notes in Computer Science, Springer International Publishing, Cham, 2021, p. 143–155. URL: https://link. springer.com/10.1007/978-3-030-77750-0_10. doi:10.1007/978- 3- 030- 77750- 0_10. [33] A. Shaghaghi, R. S. Bhopal, A. Sheikh, Approaches to recruiting ‘hard-to-reach’ populations into re- search: A review of the literature, Health Promotion Perspectives 1 (????). [34] S. L. Star, The ethnography of infrastructure, American Behavioral Scientist 43 (1999) 377–391. [35] I. Roos, Methods of investigating critical incidents: A comparative review, Journal of
Service Research 4 (2002) 193–204. doi:10.1177/1094670502004003003. [36] L. Di Geronimo, L. Braz, E. Fregnan, F. Palomba, A. Bacchelli, Ui dark patterns and where to find them: A study on mobile applications and user perception, in: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ACM, Honolulu HI USA, 2020, p. 1–14. URL: https://dl.acm.org/doi/10.1145/3313831.3376600. doi:10.1145/ 3313831.3376600. [37] C. M. Gray, C. Santos, N. Bielova, T. Mildner, An ontology of dark patterns knowledge: Foundations, definitions, and a pathway for shared knowledge-building (2023). URL: http://arxiv.org/abs/2309.09640, arXiv:2309.09640 [cs]. [38] K. Andersen, R. Wakkary, The magic machine workshops: Making personal design knowledge, in: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, ACM, Glasgow Scotland Uk, 2019, p. 1–13. URL: https://dl.acm.org/doi/10.1145/ 3290605.3300342. doi:10.1145/3290605.3300342. [39] C. Lallemand, E. Mercier, Optimizing the use of the sentence completion survey technique in user research: A case study on the experience of e-reading, in: CHI Conference on Human Factors in Computing Systems, ACM, New Orleans LA USA, 2022, p. 1–18. URL: https://dl.acm.org/doi/10.1145/3491102.3517718. doi:10.1145/3491102.3517718. [40] I. Obi, C. M. Gray, S. S. Chivukula, J.-N. Duane, J. Johns, M. Will, Z. Li, T. Carlock, Let’s talk about socio-technical angst: Tracing the history and evolution of dark patterns on twitter from 2010–2021 (????) 31. [41] S. Greenberg, S. Boring, J. Vermeulen, J. Dostal, Dark patterns in proxemic interactions: A critical perspective (????) 10. [42] J. Gunawan, A. Pradeep, D. Chofnes, W. Hartzog, C. Wilson, A comparative study of dark patterns across web and mobile modalities, Proceedings of the ACM on Human-Computer Interaction 5 (2021) 1–29. doi:10.1145/3479521. [43] G. Conti, E. Sobiesk, Malicious interface design: exploiting the user, in: Proceedings of the 19th international conference on World wide web - WWW ’10, ACM Press, Raleigh, North Carolina, USA, 2010, p. 271. URL: http://portal.acm.org/citation.cfm?doid=1772690.1772719. doi:10.1145/1772690.1772719. [44] J. P. Zagal, S. Björk, C. Lewis, Dark patterns in the design of games, Foundations of Digital
Games 2013 (2013) 8. [45] A. Mathur, J. Mayer, M. Kshirsagar, What makes a dark pattern... dark? design attributes, normative considerations, and measurement methods, arXiv:2101.04843 [cs] (2021). URL: http://arxiv.org/abs/2101.04843. doi:10.1145/3411764.3445610, arXiv: 2101.04843. [46] V. Distler, M. Fassl, H. Habib, K. Krombholz, G. Lenzini, C. Lallemand, L. F. Cranor, V. Koenig, A systematic literature review of empirical methods and risk representation in usable privacy and security research, ACM Transactions on Computer-Human Interaction 28 (2021) 1–50. doi:10.1145/3469845. [47] V. Distler, M. Fassl, H. Habib, K. Krombholz, G. Lenzini, C. Lallemand, V. Koenig, L. F. Cranor, Empirical Research Methods in Usable Privacy and Security, Springer International Publishing, Cham, 2023, p. 29–53. URL: https://link.springer.com/10.1007/978-3-031-28643-8_3. doi:10.1007/978-3-031-28643-8_3. [48] C. Neustaedter, P. Sengers, Autobiographical design in hci research: Designing and learning through use-it-yourself (????). [49] H. Xue, P. M. Desmet, Researcher introspection for experience-driven design research,
Design Studies 63 (2019) 37–64. doi:10.1016/j.destud.2019.03.001. [50] V. Distler, The influence of context on response to spear-phishing attacks: an in-situ deception study, in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, ACM, Hamburg Germany, 2023, p. 1–18. URL: https://dl.acm.org/doi/ 10.1145/3544548.3581170. doi:10.1145/3544548.3581170. [51] P. Liamputtong, Researching the vulnerable: A guide to sensitive research methods (2006) 1–256. [52] C. Lallemand, V. Koenig, Measuring the contextual dimension of user experience: Development of the user experience context scale (uxcs), in: Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society, ACM, Tallinn Estonia, 2020, p. 1–13. URL: https://dl.acm.org/doi/10.1145/3419249.3420156. doi:10.1145/3419249.3420156. [53] E. Karapanos, J. Zimmerman, J. Forlizzi, J.-B. Martens, User experience over time: an initial framework, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, Boston MA USA, 2009, p. 729–738. URL: https://dl.acm.org/doi/10.1145/ 1518701.1518814. doi:10.1145/1518701.1518814. [54] C. M. Gray, P. C. Parsons, Building student capacity to engage with design methods, in: Proceedings of the 5th Annual Symposium on HCI Education, ACM, Hamburg Germany, 2023, p. 19–22. URL: https://dl.acm.org/doi/10.1145/3587399.3587415. doi:10.1145/3587399.3587415.