Evaluating ambiguity of privacy indicators in a secure email app

Abstract

Informing laymen of security situations is a notoriously hard problem. Users are usually not cognoscenti of all the various secure and insecure situations that may arise, and this can be further worsened by certain visual indicators that instead of helping users, fail to convey clear and unambiguous messages. Even in well established and studied applications, like email clients providing end-to-end encryption, the problem seems far from being solved. Motivated to verify this claim, we studied the communication qualities of four privacy icons (in the form of coloured shapes) in conveying speci c security messages, relevant for a particular secure emailing system called p p. We questioned 42 users in three di erent sessions, where we showed them 10 privacy ratings, along with their explanations, and asked them to match the rating and explanation with the four privacy icons. We compared the participants' associations to those made by the p p developers. The results, still preliminary, are not encouraging. Except for the two most extreme cases, Secure and trusted and Under attack, users almost entirely missed to get the indicators' intended messages. In particular, they did not grasp certain concepts such as Unsecure email and Secure email, which in turn were fundamental for the engineers. Our work has certain limitations and further investigation is required, but already at this stage our research calls for a closer collaboration between app engineers and icon designers. In the context of p p, our work has triggered a deeper discussion on the icon design choices and a potential revamp is on the way.

Introduction

This work reports on preliminary research where we questioned how users understand security indicators|either used independently, or alongside text labels, or with explanations|that are shown in an email application that o ers end-to-end encryption1.

Security indicators are graphical clues or icons that, in secure email apps, are reserved to tell users about two speci c concepts: con dentiality of a message, meaning that an email is or has been encrypted, and authenticity and integrity, meaning that the message is coming from a trusted party, that is, from a party whose public key we hold and trust. Di erent choices exist to express those concepts either in their positive and in their negative variants (e.g. violation of con dentiality, untruthfulness of a party, and lack of any knowledge on the matter), but there is no uniformity in how di erent apps should graphically convey those messages.

We investigate the question in the context of \Pretty Easy Privacy"2(p p), a relatively new secure email app that attempts to deploy a tra c-light semantic as a clear and easily understandable presentation of the di erent privacy states that messages and communication peers can have[ 16 ]. The design choices regarding the indicators within p p diverge from those of other applications, and how this re ects on users has not been studied before. Furthermore, other secure email apps also di er from one another. Overlooking that Lausch et al. [ 15 ] discuss that an \envelope" in various conditions (e.g. broken, closed, open) is a better metaphor than a \padlock" in its various forms (e.g. open or closed, and red or green-coloured), applications opt for their own security icons and metaphors|for instance, the popular open-source Enigmail3 for Thunderbird uses padlocks for con dentiality and sealed envelopes for authentication and integrity (see Figure 1). The reasoning underpinning choices is often not explicitly stated. 1.1

The hard quest for good security indicators

The lack of standards is surely not helping secure email designers to converge in choosing the same security indicators. Part of the problem is the considerable amount of di erent situations that arise when talking about email con dentiality, authenticity, and integrity: distinguishing and representing all of them with icons, or with a combination of icons, does not have any obvious solutions.

Even the choice of what is the right metaphor is unclear and, at least in the end-to-end encryption case, it seems that metaphors do not help users understand the real functionality of an application [ 3 ]. Thus, designers nd themselves in the di cult position to either simplify the message, at risk of paternalizing users (= not letting them be in control), or deliver a fully edged description, at risk of confusing them.

Recently the situation may have gotten worse. The General Data Protection Regulation (GDPR) suggests the use of icons in order to give a meaningful overview of the intended processing in an \easily visible, intelligible and clearly legible manner" (Art. 12.7) [ 8 ]. The GDPR has renewed a general interest in security indicators. While most of them are variants of padlocks and shields, many others are new, and more are expected to be proposed in response to the GDPR call.

Without a common agreement on what security icons to use and in which circumstances, having a large pool of icons to choose from may actually confuse app designers and users alike. For example, research on security indicators in the context of web browsers, which has been an active research area for almost two decades [ 4, 12 ], cautions that people don't always understand security indicators. In contrast to the pictograms of bio or chemical hazards, which are standardized internationally by the Globally Harmonized System [ 17 ], at the moment there is no equivalent agreement that can characterize what security `hazards' are and how they can be represented.

And, unlike written texts for which tests of understandability exist, icons do not have an accepted intelligibility test. Thus, secure email engineers have to nd their ways without clear guidelines and instruments for the design of security indicators.

Even in the case of security applications that have been available for decades, like secure email apps, there is still room for improvement. Our investigation brings up some data that can revive a discussion about the pros and cons of certain design choices in this application domain. By shedding light on the use of the tra c-light semantic within a new system that aspires to bring crypto to the masses, our work contributes to the discourse on the usability and e ectiveness of security and privacy indicators, which in the secure email context has received relatively little attention. 2

Background and Research Questions

p p is an opportunistic peer-to-peer end-to-end encryption software which tries to unburden users from managing their encryption keys. p p automatically generates user keys, appends the public key to each outgoing message, and extracts and stores keys from incoming messages. Messages are automatically encrypted and decrypted. Peers can be veri ed to be authentic by a second-channel out-of-bound communication, e.g. a phone call where the peers verify a humanfriendly version of their ngerprint; this version is a sequence of easily readable words, called trustwords, taken from a dictionary according to an index that depends on the combination of the two peers' ngerprints.

Depending on several factors, each communication channel to di erent peers may have a di erent privacy status. For example, the system can independently and automatically categorise a particular message as reliable whenever it can be encrypted or decrypted with su cient cryptographic parameters. However, the system cannot independently categorise the message as trusted unless the user carries out a p p handshake and con rms to trust the sender in the p p interface.

Internally, p p distinguishes among 13 situations which are surjectively mapped into colour codes

(chosen according to a tra c light semantics), privacy rating labels, as well as corresponding privacy rating explanations [ 16 ]. Table 1 provides an overview of the di erent internal ratings, codes and labels. Table 2 provides an overview of how these ratings are currently displayed in the user interface of p p for Thunderbird.

For instance, the p p rating \mistrust" is assigned the colour code \red", the user interface label \Mistrusted", and explanation \This message has a communication partner that has previously been marked as mistrusted".

In addition, p p uses privacy indicators which are icons from an icon set made up of four coloured shapes (e.g. see Figure 2 and Figure 3) corresponding to the colour code assigned to each situation. \Under Attack", \Broken", and \Mistrusted" are the situations represented by a red square, for instance. The rating codes from 0 to 5 are not assigned any colour label, however, in the user interface, these codes are represented by a gray circle. Reliable communication (i.e. Rating Code -3 -2 -1 0 1 2 3 4 5 6 7 8 9

Rating Label under attack broken mistrust unde ned cannot decrypt have no key unencrypted unencrypted for some unreliable reliable trusted trusted and anonymized fully anonymous rating code 6 ) is represented using a yellow shape, and trusted communication (i.e. colour code 7 ) is represented with a green shape.

The design choice of p p's privacy icons is justi ed by arguments, but not by evidence. While discussing with the p p developers, we were told that the shapes are meant to be easily understood by colour-blind persons, and were suggested after consultation with experts. The colour choices are meant to re ect the universally-deployed tra c light semantic.

There are many interesting questions that could be investigated, such as, how to draw user attention to these indicators; where to display those icons in the user interface; what situations do such shapes suggest to users; are shapes better than conventional icons (e.g. envelopes), etc.

Nevertheless, here we conduct an enquiry into the most basic question of \how would prospective or rst-time p p users understand the p p privacy indicators". Formally stated, we intend to shed light on the following research questions: 1. Which of the 4 visual icons do users associate with the di erent p p privacy ratings? 2. Which of the 4 visual icons do users associate with the di erent p p privacy rating explanations? 3

Methodology

We conducted three online studies to assess how people would interpret the various privacy rating indicators o ered by the p p email encryption system. Table 3 provides an overview of the user test sessions, the number of participants per session as well as the focus of investigation.

Session 1 Session 2

Session 3 Total evaluations

Privacy

Ratings

Privacy Rating Explanations

42

26 The rst part of the study contained a block of 10 questions which asked participants to choose an icon which according to them best corresponds to a given privacy statement i.e. rating (see Figure 3 upper part for an example). The second part of the study similarly asked the participants to match an icon to a privacy rating explanation (see Figure 3 lower part). The last part of the study asked about demographics. The 10 privacy rating statements and explanations were drawn from the p p for Thunderbird distribution (Enigmail/p p version 2.0.12 (20190707-1417). To minimize order bias, the sequence of all questions per block was randomized for each test participant. The order of all answer choices (icons) per question was also randomized. The studies were administered via Qualtrics4. 3.2

Participants

All participants are relatively tech-savvy, with at least a Bachelor's degree. The participants of Session 1 and 2 are based in Luxembourg and were recruited at the University of Luxembourg via an email invitation to participate in a pilot study (Session 1) and during a lecture on Security Engineering (Session 2). No incentive was o ered to the participants of Sessions 1 and 2. The participants of Session 3 are based in di erent European countries and were recruited in Portugal during a workshop on User Experience in security and privacy-critical systems. As a compensation for their participation in the study, all participants of Session 3 were o ered a commercial license of p p for continued use of their paid apps i.e. the Outlook and Android distributions. 3.3

Analysis

We performed a comparative analysis to understand if our participants associated icons to the various privacy ratings and explanations in the same way as implemented by p p. If the icon chosen by the majority of participants is the same as the one chosen by p p, the alignment test for that rating or explanation equals \MATCH", and otherwise \NO MATCH".

The Match strength refers to how many participants selected the same icon as p p. Hence, the higher the match, the narrower the gap between what the developers wanted to communicate via the system and what the users understood. Similarly, the lower the match, the higher the ambiguity of the intended privacy indicator.

Results

Understanding why there is a dichotomy between what the developers wanted to convey with the di erent privacy indicators in p p and how prospective users would interpret them, or which privacy indicators could be better in narrowing this chasm, is not in the scope of this investigation. Nevertheless, we hypothesize that the following elements potentially play a role: the shapes of the indicators the colours of the indicators the tra c light metaphor used for the indicators the choice of words in the statements and explanations the perception of risk associated with the shapes, colours, metaphor and wordings of the indicators the clustering of risks the understandability of the indicators the awareness and concern about di erent scenarios and privacy ratings (For each item, the match strength refers to the percentage of participants that associated an icon to a statement or explanation in the same fashion as it is currently implemented by p p.)

It is often the case that visual input tends to dominate other modalities when it comes to our perceptual and memorial judgements [ 18 ]. Colour is one of the characteristics of human visual perception that can carry important meaning and can have an important impact on peoples a ect, cognition, and behaviour [ 6 ]. According to Elliot & Maier's colour-in-context theory [ 5 ], some colour meanings and e ects are biologically-based, while others are posited to stem from the repeated pairing of colour and particular concepts, messages, and experiences. They state that observing colour-meaning associations over time and cultures can contribute to reinforcing and extending the applicability of those links to objects in the broader environment, such as signs and signals. We did not have the opportunity to perform this investigation with existing p p users. We would be interested in comparing such results with the current ndings and looking at the role of experience with the system on the interpretation of the privacy ratings and indicators.

Disregarding some regional variations, tra c signs and tra c lights are now found all over the world, and their meaning is internationally recognizable. The corresponding tra c light rating system (red, amber, green) is something we have repurposed in many di erent domains, from nutrition labels for pre-packed products [ 22 ] to energy consumption labeling [ 9 ] and project management status reporting [ 7 ], to name a few. In that respect, the provision of the tra c light colour codes can serve to communicate more accurate, relevant, and comparable information to users, as well as to transmit certain levels of risk or allow for a quick recognition of potential hazards.

Nevertheless, while signs and pictograms have been standardized in speci c areas, in many di erent contexts harmonized communication or a shared understanding of the risk communicated by signs, symbols, or colours cannot be taken for granted [ 21 ]. The reasons can range from cross-cultural di erences [ 14 ] to varying levels of technical expertise within a speci c domain. Research on human aspects in the context of end-to-end email encryption suggests that non-expert users have incomplete threat models and a general absence of understanding of the email architecture [ 19 ]. As expressed in Section 1, the number of di erent situations that arise in secure email is not so small. Deciding which ones and how many to represent graphically, as well as, which metaphor to use, is not an easy choice. There is probably not a straightforward answer and there is de nitely not a unique one. There are di ering views about how transparent should systems for end-to-end email encryption be [ 1 ], [ 20 ]. In the case of Item 9, it is evident that p p attempts to nd the balance between these two approaches: on the one hand they would like to instill a sense of security provided by the automatic end-to-end encryption akin to other secure messaging and emailing systems, but on the other hand, they would still like to warn the user of potential threats such as a man-in-the-middle attack that they could be susceptible to if they do not verify the corresponding party via a second secure channel (e.g. by comparing the trustwords in person or over the phone).

While over time, we are likely going to recognize more consistency in the symbols used by security and privacy-critical systems, widely-available UI kits or even standardized icon sets, in the immediate term, developers of such systems should devote an equal amount of attention and resources to understanding their (target) users and the di erent dimensions and requirements of their socio-technical proposition. As a starting point, developers, and especially teams without user research/UX pro les, should inform themselves of the general paradigms and design principles that align security and usability [ 23, 13 ], followed by more recent lessons learnt in this highly-challenging domain that brings together the usable security, HCI and UX disciplines. Fine-grained inspiration could perhaps be drawn from the vast body of work on browser security indicators and warnings (e.g. [ 11 ]), in particular, the incremental user-centred approach where proposed designs and changes were validated with thousands of users. Caplin's book [ 2 ] could potentially be a useful reference to some developers in the speci c context of icons in computer interface design, however, as pointed out by Felt et al. "Millions of Internet users have recently come online via smartphones without learning 'standard' iconography from desktop browsers" [ 11 ], thus it is important to acknowledge that the expectations of users in terms of interfaces are not necessarily associated with desktop computing and, in many cases, obsolete metaphors. 6

Conclusion and Future Work

We reported on a 42-participant study of users' perceptions of email privacy ratings in the context of pretty Easy privacy. Although our preliminary study has an evident limitation mainly due to the limited sample size, the outcome suggests that prospective or rst-time p p users would have a di culty understanding the privacy information communicated by p p.

The ndings call for a broader and deeper investigation that would seek to assert which design choices in terms of the privacy rating statement, explanation and visual icon (shape, colour, metaphor etc.) would need to be reconsidered if p p would like to accurately communicate the degree of protection that it o ers to its users as they send and receive email through its system bearing in mind their experience, awareness and concerns. Our work has triggered a deeper discussion at p p on the existing icon design choices and a potential overhaul of the privacy indicators is being deliberated.

From a broader perspective we believe that further and more holistic analyses of the di erent secure communication systems would be needed in order to identify the di erent types and degrees of security information communicated by those systems, the e ectiveness of the deployed security and privacy indicators as well as their suitability for target audiences with di erent characteristics and levels of expertise.

We are of the opinion that despite looking trivial, this interaction experience should not be in the way of users adopting systems for end-to-end email encryption, let alone a source of confusion or frustration that could result in unsecure behavior or unwanted leakage of con dential information. This is particularly relevant within an organizational setting, where policy and culture may also contribute towards the way users go about employing end-to-end email encryption. 7

Acknowledgments

Authors are supported by the Luxembourg National Research Fund through grant PRIDE15/ 10621687/SPsquared, and the project pEp Security SA / SnT Protocols for Privacy Security Analysis. We would like to thank the p p team for their collaboration, input and feedback in this project, as well as, the participants of the Human-centered cybersecurity workshop at CHItaly 2019 for the fruitful discussions. Last but not least, we would like to thank the anonymous reviewers at ITASEC 2020 for their comments and suggestions for improvement.

References

[1] Erinn Atwater , Cecylia Bocovich, Urs Hengartner, Ed Lank, and Ian Goldberg . Leading Johnny to Water: Designing for Usability and Trust . pages 69 { 88 . USENIX Association, 2015 .

[2] Steve Caplin . ICON Design: Graphic Icons in Computer Interface Design . Watson-Guptill Publications , Inc., USA, 2001 .

[3] Albese Demjaha , Jonathan Spring, Ingolf Becker, Simon Parkin, and M Angela Sasse . Metaphors considered harmful ? An exploratory study of the e ectiveness of functional metaphors for endto-end encryption . (February): 1 { 12 , 2018 .

[4] Rachna Dhamija , J. D. Tygar , and Marti Hearst . Why phishing works . In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '06 , pages 581 { 590 , New York, NY, USA, 2006 . ACM.

[5] Andrew J Elliot and Markus A Maier. Chapter two - Color-in-Context Theory . volume 45 of Advances in Experimental Social Psychology, pages 61 { 125 . Academic Press, 2012 .

[6] Andrew J Elliot and Markus A Maier. Color Psychology : E ects of Perceiving Color on Psychological Functioning in Humans . Annual Review of Psychology , 65 ( 1 ): 95 {120, jan 2014 .

[7] C. N. Enoch and L. Labuschagne . Project portfolio management: using fuzzy logic to determine the contribution of portfolio components to organizational objectives . Paper presented at PMI Research and Education Conference , Limerick, Munster, Ireland. Project Management Institute, Newtown Square, PA, 2012 .

[8] EU. Regulation (EU ) 2016 / 679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data . O cial Journal of the European Union, L119:1{88 , 2016 .

[9] EU. Regulation (EU ) 2017 / 1369 of the European Parliament and of the Council of 4 July 2017 setting a framework for energy labelling and repealing Directive 2010 /30/EU ( Text with EEA relevance . ). O cial Journal of the European Union, L198:1{23 , 2017 .

[10] Michael W Eysenck. Cognitive psychology : a student's handbook , 2015 .

[11] Adrienne Porter Felt , Robert W Reeder, Alex Ainslie, Helen Harris, Max Walker, Christopher Thompson, Mustafa Emre Acer, Elisabeth Morant, Sunny Consolvo, and U C Berkeley. Rethinking Connection Security Indicators . the Symposium On Usable Privacy and Security (SOUPS) , (Soups): 1 { 14 , 2016 .

[12] Batya Friedman , David Hurley, Daniel C. Howe, Edward Felten , and Helen Nissenbaum . Users' conceptions of web security: A comparative study . In CHI '02 Extended Abstracts on Human Factors in Computing Systems, CHI EA '02 , pages 746 { 747 , New York, NY, USA, 2002 . ACM.

[13] Simson L. Gar nkel. Design Principles and Patterns for Computer Systems That Are Simultaneously Secure and Usable . PhD thesis , Massachusetts Institute of Technology, 2005 .

[14] Ralph B Hupka , Zbigniew Zaleski, Jurgen Otto, Lucy Reidl, and Nadia V Tarabrina. The Colors of Anger, Envy, Fear, and Jealousy: A Cross-Cultural Study . Journal of Cross-Cultural Psychology , 28 ( 2 ): 156 { 171 , 1997 .

[15] Joscha Lausch , Oliver Wiese, and Volker Roth . What is a Secure Email? EuroUSEC 2017 , 2017 .

[16] Hern ^ani Marques and Bernie Hoeneisen. pretty Easy privacy (pEp): Mapping of Privacy Rating , 2019 . IETF Internet-Draft, https://tools.ietf.org/html/draft-marques - pep-rating- 01 , Accessed: 30 June 2019 .

[17] United Nations . Globally harmonized system of classi cation and labelling of chemicals (GHS) - sixth revised edition . 2015. Accessed: 30 June 2019 .

[18] Michael I. Posner , Mary J. Nissen , and Raymond M. Klein . Visual dominance: An informationprocessing account of its origins and signi cance . Psychological Review , 83 ( 2 ): 157 { 171 , 1976 .

[19] Karen Renaud , Melanie Volkamer, and Arne Renkema-Padmos. Why Doesn't Jane Protect Her Privacy ? In Emiliano De Cristofaro and Steven J Murdoch, editors, Privacy Enhancing Technologies , pages 244 { 262 , Cham , 2014 . Springer International Publishing.

[20] Scott Ruoti , Je Andersen , Scott Heidbrink, Mark O'Neill , Elham Vaziripour, Justin Wu, Daniel Zappala, and Kent Seamons . "We'Re on the Same Page": A Usability Study of Secure Email Using Pairs of Novice Users . CHI '16 , pages 4298 { 4308 . ACM, 2016 .

[21] Tonya L Smith-Jackson and Michael S Wogalter. Users' hazard perceptions of warning components: An examination of colors and symbols . Proceedings of the Human Factors and Ergonomics Society Annual Meeting , 44 ( 32 ): 6 { 55 {6{ 58 , 2000 .

[22] UK Department of Health and Social Care . Guide to creating a front of pack (FoP) nutrition label for pre-packed products sold through retail outlets , 2013 .

[23] Ka-Ping Yee . Aligning security and usability . IEEE Security & Privacy , 2 ( 5 ): 48 { 55 , 2004 .