RepliPRI: Challenges in Replicating Studies of Online Privacy Sameer Patil Helsinki Institute for Abstract Information Technology HIIT Replication of prior results has recently attracted Aalto University attention and interest from the CHI community. This Aalto 00076, FInland paper focuses on the challenges and issues faced in sameer.patil@hiit.fi carrying out meaningful and valid replications of HCI studies. I attribute these challenges to two main underlying factors: (i) a domain of inquiry that simultaneously covers people, social systems, and technology; and (ii) deficiencies in result reporting and data archiving. Using examples from investigations of online privacy, I outline how these challenges manifest themselves in HCI studies. Longitudinal approaches, international collaboration, and sharing of study instruments could help address these challenges. Author Keywords Replication, Privacy, Cultural differences ACM Classification Keywords H.1.2 [User/Machine Systems]: Human factors. General Terms Human Factors, Security Presented at RepliCHI2013. Copyright c 2013 for the indi- Introduction vidual papers by the papers authors. Copying permitted only Replication of prior results has recently attracted for private and academic purposes. This volume is published attention and interest from the CHI community. The and copyrighted by its editors. resulting discussions tackle replication from two is the publication describing the results of the important perspectives: higher level epistemological study. Unfortunately, due to page limits and other debate on the place and merits of replication in the editorial reasons, publications often do not scientific (publishing) enterprise and the lower-level include all information — about methods and/or practical considerations for replicating previous studies data — necessary for carrying out the study the from the literature. Growing interest in RepliCHI way it was originally conducted. For instance, suggests increasing recognition for the value of instead of including the entire questionnaire replicating prior studies. I hope and anticipate that this instrument, the publication may include only trend will foster continued community discussion on those questionnaire items that led to statistically how to justify, appreciate, and reward replication as a significant results. Similarly, results may be valuable scientific pursuit. Therefore, in this paper I presented in the aggregate or as percentages, focus on the latter aspect, viz., challenges and issues making it difficult to replicate analyses that faced in carrying out meaningful and valid replications require details of individual data points. of HCI studies. I attribute these challenges to two main factors: In the following section, I outline how I have found these challenges to manifest themselves in investigation of user preferences and practices 1. Domain of inquiry: A large proportion of HCI regarding online privacy. I conclude with some studies tackle research problems where results thoughts on addressing the challenges. typically exhibit simultaneous and interacting influence of individuals, social systems, and Replicating Studies of Online Privacy technology. Each of these three factors changes When thinking about and carrying out replications of at drastically different rates and magnitudes. For research related to privacy, I have encountered several instance, technology used in a study may practical challenges: become obsolete within months or a couple of years, while physical and cognitive capabilities of Privacy is a nuanced and complex issue affected by adults change at much slower rates (and the individual characteristics, context of operation, and the magnitude of the change is often comparatively technology under consideration. For instance, small and predictable). These differences in the individuals have been classified into different groups evolution trajectories of humans, cultures, and based on their inherent level of privacy concern [7], technology make it difficult to replicate studies at and privacy concerns have been shown to exhibit a later time and to determine and attribute cultural variation [3]. People’s mental models and causes behind differences in results, if any. understanding of the underlying technology also affects their preferences and practices regarding privacy [4]. 2. Insufficient and/or incomplete reporting: Typically This implies that even when considering the same the only resource available for replicating a study technology, replication conducted at a later time ought to take into account the impact of learning effects on a simple case of re-running the study with subjects privacy issues. Replications may also encounter the drawn from a different culture, with translation of selection-maturation threat to validity owing to major instruments and study materials, if necessary. In external events that occur after the original study, such practice, however, cultural differences pose several as news coverage of privacy breaches. Such events hurdles. For instance, the same word or term may be affect the population’s overall understanding and interpreted differently leading to the same question awareness of privacy issues, thereby potentially being answered differently. For example, we found that affecting the results of replications of studies that were the term “cubicle” was understood differently in the US originally conducted prior to these event(s). and India owing to differences in office layouts and density. This difference was one of the factors crucial The majority of attention in replication has been for understanding the differences in results between devoted to replication at a different (later) time. In the the US and India [5]. In other studies, I discovered that case of privacy, however, it is equally important to the demographic question about ethnicity, which is consider replication across different cultures. For commonly asked in the US (and even mandated for example, we administered a questionnaire NSF-sponsored studies), was considered potentially simultaneously in the US and India, enabling us to offensive and confusing in Europe. Differences in draw interesting and surprising observations from lifestyle and beliefs can also affect whether questions comparison across cultures [5]. Our results confirmed and tasks from one study can yield valid results, or earlier findings regarding low levels of consumer even make sense, when replicated in a different privacy concerns in India. Surprisingly, by examining cultural context. For instance, some privacy studies interpersonal privacy separately from consumer have asked Western respondents about premarital sex, privacy, we found that interpersonal privacy concerns sexual practices, extramarital affairs, and number of in India were not only higher than consumer privacy sexual partners (e.g., [1]). Such questions are unlikely concerns but also higher than interpersonal privacy to produce meaningful results in cultures where such concerns in the US. Our study considered culture at practices are uncommon and/or forbidden. Resolving the broad level of national cultures. However, it should this issue can be complicated when such be noted that for replication purposes “culture” could culturally-specific questions comprise parts of standard be construed to connote any large groups with shared scales; using the scale without modifications will not characteristics and/or values, such as students, yield meaningful results and dropping and/or modifying engineers, mothers, liberals, etc. Moreover, if items in the scale risks affecting the validity of replication across cultures is conducted at a time later comparison across studies. Finally, it is also necessary than the original study, then learning effects and to consider whether results across cultures are maturation threats need to be taken into account (as affected by differences in sampling techniques and discussed above). sample characteristics. For instance, although our comparison of the US and India was limited to software In theory, replication with a different cultural sample is professionals, the mean and median ages of the Indian participants were lower than those of the US simultaneously covers individuals, social systems, and participants. technology; and (ii) result reporting and data archiving. We found that understanding privacy-related cultural The second of these, in particular, could be easily nuance often requires insights derived from qualitative addressed by requiring inclusion of full instruments and methods (such as interviews, focus groups, field visits, study protocols as appendices1 . Similarly, authors of etc.) and/or insider knowledge of the culture and its accepted papers could be asked, or even required, to practices [6]. Currently the CHI community is focused upload the raw data after taking steps necessary to mostly on replication of studies that employ quantitative protect participant anonymity. In this regard, ACM, methods, such as experiments, questionnaires, or IEEE, NSF, and other prominent HCI funding and usability evaluations. Complementing quantitative sponsoring organizations can follow the lead of the replications with qualitative insights has potential to NIH, which mandates raw data availability. In a similar broaden the scope of these replication endeavors. vein, an open source inspired approach could Toward this end, it may also be fruitful to tackle whether encourage authors to release the source code of and how qualitative studies could be effectively systems and scripts used for conducting studies and replicated. carrying out analyses. An open question regarding data and code sharing is how to deal with Discussion and Conclusion commercialization and intellectual property issues The previous section utilized examples from (especially when corporate entities are involved in investigations of online privacy attitudes and behaviors conducting the study)2 . to illustrate some of the challenges and issues in replicating HCI studies. Online privacy cuts across the One approach for addressing the issue of intersection individual, the social, and the technical, in much the of people and technology is to encourage longitudinal same way as many studies in HCI do. Therefore, I investigations carried out at regular intervals over believe that many, if not all, of these concerns are also several years. Depending on the details and logistics of likely to arise in HCI investigations of other topics. the study, a longitudinal investigation could utilize the same participants or different participants with the The RepliCHI workshop is an important milestone same sampling method and sample characteristics. toward developing a comprehensive compilation and The former approach can help examine the impact of understanding of various challenges involved in the changes in individual characteristics, evolution in replication of HCI studies. Moving forward, it is lifestyles, and effects of learning. The latter approach necessary to apply this knowledge and insight for can help illuminate the impact of changes in constructing best practices to follow and pitfalls to 1 This also provides the additional benefit of addressing one of the avoid. Toward this end, I offer suggestions that address most common comments raised in peer reviews — lack of method- the two important considerations outlined in the ological detail. Introduction, viz., (i) domain of inquiry that 2 Data used by studies conducted by corporations was a hotly de- bated topic at the WWW 2012 conference [2]. technology. For replications across cultures, however, it researchers. is perhaps best to target simultaneous study http://www.nytimes.com/2012/05/22/science/big- deployment. Fostering international collaborations data-troves-stay-forbidden-to-social-scientists.html, and/or leveraging international students to gain cultural May 2012. knowledge and access could help in this regard. [3] Milberg, S., Burke, S., Smith, H., and Kallman, E. Values, personal information privacy, and Requiring a replication component in Bachelor’s and regulatory approaches. Communications of the Master’s theses could provide a starting point for ACM 38, 12 (1995), 65–74. repeating studies from the literature, simultaneously [4] Patil, S., and Kobsa, A. Uncovering privacy serving a valuable pedagogical purpose by training the attitudes and practices in Instant Messaging. In next generation. Further, conferences and journals Proceedings of the 2005 International ACM could explicitly solicit replications of specific studies. SIGGROUP Conference on Supporting Group Special conference sessions or journal sections could Work, GROUP ‘05, ACM (New York, NY, USA, be devoted solely to replication studies. Discussions 2005), 109–112. and follow-up activities from the RepliCHI workshop [5] Patil, S., Kobsa, A., John, A., and Seligmann, D. could lead the way toward legitimizing and promoting Comparing privacy attitudes of knowledge workers replication as a valuable scientific pursuit within HCI. in the U.S. and India. In Proceedings of the 3rd International Conference on Intercultural Acknowledgments Collaboration, ICIC ‘10, ACM (New York, NY, USA, I thank Mihir Mahajan and John McCurley for editorial 2010), 141–150. comments. [6] Patil, S., Kobsa, A., John, A., and Seligmann, D. Methodological reflections on a field study of a References globally distributed software project. Information [1] Grossklags, J., and Acquisti, A. When 25 cents is and Software Technology 53, 9 (2011), 969–980. too much: An experiment on willingness-to-sell and [7] Taylor, H. Most people are “privacy pragmatists” willingness-to-protect personal information. In who, while concerned about privacy, will sometimes Workshop on the Economics of Information trade it off for other benefits. The Harris Poll 17 Security (WEIS) (2007). (2003), 19. [2] Markoff, J. Troves of personal data, forbidden to