=Paper= {{Paper |id=Vol-2582/paper1 |storemode=property |title=Explaining Data Driven Personas to End Users |pdfUrl=https://ceur-ws.org/Vol-2582/paper1.pdf |volume=Vol-2582 |authors=Soon-gyo Jung,Joni Salminen,Bernard J. Jansen |dblpUrl=https://dblp.org/rec/conf/iui/JungSJ20a }} ==Explaining Data Driven Personas to End Users== https://ceur-ws.org/Vol-2582/paper1.pdf
                      Explaining Data-Driven Personas to End Users
                                        Soon-gyo Jung*, Joni Salminen*§, Bernard J. Jansen*
                                                *Qatar Computing Research Institute, Doha, Qatar
                                                               §University of Turku

                                                    {sjung, jsalminen, bjansen}@hbku.edu.qa


ABSTRACT                                                                       layouts [28], not much is known about the digital persona UIs and
                                                                               their user experience (UX). Apart from exploratory studies [37, 38,
Enabled by digital user data and algorithms, persona user
                                                                               41], usability problems and interaction patterns in DDP context
interfaces (UI) are moving to digital formats. However, algorithms
                                                                               remain unchartered. More particularly, there has been little
and user data, if left unexplained to end users, might leave data-
                                                                               research on how to make the digital persona UIs transparent [31].
driven personas (DDPs) difficult to understand and trust. This is
                                                                               Transparency refers to providing explanations on how opaque
because the data and the way it is processed are complex and not
                                                                               algorithms produce information for end users [13].
self-evident, requiring explanations of the DDP information and
                                                                                  This research aims to shed some light into these unexplored
UIs. In this research, we provide a proof of concept for adding
                                                                               areas, with a specific focus on the design goal of making DDPs
transparency to DDP using a real system UI. Furthermore, we
                                                                               understandable and trustworthy from the perspective of their
demonstrate ways to add breakdown information that can help
                                                                               users (e.g., journalists, marketers, online content creators, medical
alleviate user stereotyping associated with the use of personas.
                                                                               professionals, corporate decision makers, and so on).
                                                                                  As a contribution, we demonstrate ways for adding
KEYWORDS                                                                       transparency in DDPs by two means: (1) adding explanations of
Personas; digital interfaces; transparency, information design.                persona information and how it was produced and (b) adding
                                                                               breakdown information of the representative persona characteristics,
ACM Reference format:
                                                                               towards the goal of mitigating stereotypical thinking.
Soon-Gyo Jung, Joni Salminen and Bernard J. Jansen. 2020. Explaining
                                                                                  Therefore, our goals with this research are to demonstrate
Data-Driven Personas to End Users. In Proceedings of the IUI workshop
on Explainable Smart Systems and Algorithmic Transparency in Emerging          means to add transparency to DDPs to increase persona users’
Technologies (ExSS-ATEC'20). Cagliari, Italy, 7 pages.                         understanding and trust towards the personas (both being risks
                                                                               noted in previous research [8, 23]); and to add information
CSS CONCEPTS                                                                   breakdowns that show the persona is a composite representation
                                                                               of a group of users, thereby providing means to alleviate user
• Human-centered computing ~ Human computer interaction
                                                                               stereotyping, a risk stressed in the persona research [15, 22, 46].
(HCI)

                                                                               2 Related Literature
1 Introduction                                                                 Personas were introduced as a HCI technique [29] in software
A persona represents the goals, behaviors and characteristics of a             development [9, 17]. There are a variety of benefits attributed to
user segment [4, 29]. While personas are typically created                     personas [1], such as focusing on user outcomes, consensus
qualitatively from user interviews [14, 17], qualitative approaches            building among designers and developers, user-centricity, and
are newly complemented by data-driven personas (DDPs) that use                 more granular product targeting [33, 36]. Personas provide
quantitative methods, algorithms, and online user data [11, 22, 35].           communication benefits within teams [7] and organizations [28].
This transition from traditional personas to DDPs is associated                Personas can enable designers to identify with backgrounds
with the digitalization of persona user interfaces (UIs). While                different from their own and realize that the user preferences may
personas are traditionally presented in one or two page paper                  deviate from their personal preferences [16, 17, 45].
profiles [6, 26], DDPs are presented in a digital UIs that the                     Yet, to achieve the said benefits, it is critical that personas are
persona users can interact with (see example in Figure 1).                     perceived as credible and trustworthy by their end users [32]. To
    Data-driven persona generation is becoming increasingly                    achieve trust credibility, one proposed technique is explaining
popular in the industry [27, 34, 35, 43]. The challenge relating to            how the personas were created, what design choices were made
this shift is that while research has been done on traditional paper           and why. Such transparency has been found especially important
                                                                               in algorithmic systems that, due to their complexity, may appear
                                                                               suspicious to end users [10, 13].
Copyright © 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
Figure 1: A DDP from a real production system1. There are several ways for end users to interact with the system, including, e.g.,
selecting the persona, changing the number of personas generated, and filtering the comments of the persona.
    Persona transparency is an unexplored area in the HCI                 The advice from the previous research is that persona creators
literature. Among the rare studies focused on persona                  should seek to experiment with novel designs of transparency for
transparency, Salminen et al. [42] analyze the impact of               DDPs [42]. To this end, we present some explanations and data
explanations added in the persona UI on user perceptions. They         breakdowns we have implemented in a persona system. Note that
find that higher transparency (in the form of explanations)            these findings represent only the added explanations and
increased the perceived completeness and clarity of the shown          breakdowns and do not include an empirical user study on their
personas. However, there was an undesirable effect of the              implementation on the persona UX. Such a study is a planned next
explanations decreasing the credibility of the persona. The            step in the research agenda.
researchers interpreted this as an indication of a transparency
trade-off, according to which the technical explanations disrupt
the façade of personas being perceived as real people.
    It is also worth to note that Salminen et al. [42] implemented
the explanations “forcefully”, meaning that they were shown as
open pop-up type of boxes to the users (see Figure 2). In contrast,
we implement the explanations as interactive tooltip definitions
that the users can reveal by hovering over the tooltip icon.
    Overall, computational techniques are becoming more
common in persona development, with several researchers
presenting their versions of algorithmically generated personas
[2, 3, 24, 47]. The users are given access to the generated personas
through system UIs where they can interact with the personas,
including selecting each persona and viewing their information
The users of these DDPs may question the information in persona
profiles because they are unsure of how it was produced. This is a
special concern for DDPs because their creation relies on opaque       Figure 2: Explanations implemented in Salminen et al. [42].
algorithmic processes that are often difficult to communicate in       These were shown “forcefully” without giving the users ability
layman terms [5]. This difficulty can be seen from the findings of     to enable or disable the explanations.
user studies that report issues of confusion and information
design of algorithmically generated DDPs [36–38, 41].




1
    https://persona.qcri.org
                     Table 1: Explanations implemented in the DDP system. Adapted from Salminen et al. [42].
 Section           Explanation provided                                                                                  Implemented
                                                                                                                         (✗ = No, ✓ = Yes)
 Name              Persona’s name is chosen by retrieving common names from a popular online social network of                  ✗
                   people with a given age, gender, and country. Tools we use: Python, Pandas, Database
 Picture           Persona’s picture is chosen from pictures downloaded from online photobanks, tagged for age,                 ✗
                   gender, country, and ethnicity. Tools we use: Python, Online photobanks, Database
 Demographics      Persona’s demographic information (age, gender, country) is retrieved from aggregated YouTube                ✗
                   viewer statistics of this channel’s videos. Tools we use: Python, YouTube API
 Job               Job is shown based on Facebook audience sizes. The system collects Facebook audience sizes                   ✓
                   based on persona’s demographic, interests, and language.
 Education         Education Level is shown based on Facebook audience sizes. The system collects Facebook                      ✓
 Level             audience sizes based on persona’s demographic, interests, and language.
 Relationship      Relationship Status is shown based on Facebook audience sizes. The system collects Facebook                  ✓
 Status            audience sizes based on persona's demographic, interests, and language.
 Topics of         Topics of interest are retrieved by classifying the content to descriptive categories and choosing           ✓
 Interest          the most corresponding ones for this persona. Tools we use: Python, Pandas, Scikit-learn (Latent
                   Dirichlet Allocation), supervised machine learning, Database
 Most Viewed       Most viewed contents are retrieved from the aggregated view counts of YouTube videos and are                 ✓
 Contents          chosen to describe the taste of this persona. Tools we use: Python, Database, YouTube API
 Viewed            Persona’s quotes are retrieved from the comments of most viewed videos of this persona. Tools                ✓
 conversations     we use: Python, Database, YouTube API
 Audience Size     Audience size is calculated by searching the number of people on Facebook with similar attributes            ✓
                   to this persona, including age, gender, country, language, and topics of interest. Tools we use:
                   Python, Facebook Marketing API, Database

                                                                        multiple online analytics and social media platforms, including
                                                                        Facebook Insights, YouTube Analytics, and Google Analytics.
3 Implementing DDP Transparency                                            Thus, we implement the explanations of the previous user
We adopted a simple design principle for transparency: explain to       study [42] in APG. The following sections demonstrate the
the user what the information is and where it comes from. The           implementation through practical examples from the UI. We first
explanations were then crafter by one of the researchers for all        demonstrate the explanations and then the data breakdowns. Note
the information elements in the persona UI, as defined in Table 1.      that all explanations require the user to hover either the tooltip
After this, the other researchers gave feedback on the wording and      icon (the small question mark in Figure 4) or the element itself to
content of the explanations. Finally, after being reviewed by           show. The breakdowns require the user to click on the breakdown
everyone in the research team, the explanations were                    icon (the small magnifying glass in Figure 3).
implemented in the persona system.
   Note that the explanations are the same as the ones used in a        3.2 Explanations
previous user study [42]. That study, however, tested only              Figure 3 demonstrates the explanation for the stability indicator.
persona mockups, not a live system. Here, we implement the              The stability function informs the user of how frequently this
explanations in a live system for real client organizations1.           persona appears in different persona sets over time. If the persona
                                                                        appears often, he or she is labeled as a “Loyal” persona. Otherwise,
3.1 Persona System                                                      the persona is labeled as a “Occasional” persona.
The persona system is called Automatic Persona Generation                  Figure 4 shows the sentiment explanation. Sentiment score is
(APG) and it has been widely reported in previous research [2, 3,       calculated as an aggregate score from the comments associated
18, 19, 39]. APG is both a system and methodology for generating        with the persona and describes the persona’s overall attitude.
personas from online analytics and social media user data. The             Figure 5 shows the explanation for topics of interest. Topics
system uses application programming interfaces (APIs) to                are reflective of the content consumption preferences of online
automatically collect online user data with channel owners’             audience personas [39]. Similarly, most viewed contents describe
permission. It then carries out algorithmic data analyses and           the content that the group corresponding to the persona has most
outputs a set of DDPs that the end users can interact with using        viewed (see Figure 6). The comments shown in the persona profile
the system UI. APG uses a robust Web framework for Python               are inferred from this content (see Figure 7). Each persona has
(Flask) and a stable back-end database (PostgreSQL). It supports
                                                                        Figure 4: Explanation for the sentiment of the persona.
      Figure 3: Explanation for the stability of the persona.




            Figure 5: Explanation for topics of interest.
                                                                             Figure 6: Explanation for the viewed contents.




                                                                              Figure 8: Explanation for the audience size.



               Figure 7: Explanation for the quotes.


demographic traits (age, gender, location) and topics of interest.    this persona. In other words, there is diversity within the persona.
Based on these, audience size is calculated. This corresponds to      In a similar vein, Figure 10 illustrates the distribution of topics of
the number of people with the said characteristics (see Figure 8).    interest. It shows how much, in quantitative terms, the persona
                                                                      prefers, or does not prefer, a given topic. Again, as these measures
3.3 Data Breakdowns                                                   are calculated using machine learning models (see [3]), we can
To reduce stereotyping and to facilitate the understanding of the     obtain and present a probabilistic score for the persona users.
data, APG provides breakdowns of information. Figure 9 shows
the demographic groups that have the highest quantitative             3.4 Explaining Algorithmic Process of DDPs
association with the content engagement pattern that the persona      One challenging – perhaps even the most challenging – aspect of
is based on. The point is to show to the users that even though the   explanation in DDPs is the functioning of the core algorithm. This
persona has a representative demographic group (in this case,         has previously been done using equations [2, 3, 39, 40] and figures
Male 25-34 India), there are also other demographic group that fit,   (see Figure 12).
with different degrees of association, to the behavioral pattern of
 Explaining Data-Driven Personas                                                                                       S.-G. Jung et al.

                                                                      (see Figure 11) and complex mathematical denotations (see Figure
                                                                      12) explanations. Essentially, communicating algorithmic
                                                                      processes is a hard problem to solve as the process has many steps
                                                                      and a high degree of technical complexity. Explaining these
                                                                      processes in a simple graph, text, or table seem not very user-
                                                                      friendly. As a future course of action, we are planning to produce
                                                                      an explainer video – the advantages of a video are many: we can
                                                                      use different screens, views, and animations to simplify the
                                                                      algorithm; we can break down the process into a logical narrative;
                                                                      we can give examples that make the storyline more concrete; and
                                                                      we can support the conveying of the message with visual, textual
                                                                      and auditory information (i.e., voiceover). This facilitates learning
                                                                      and understanding by different user types.




Figure 9: Breakdown of demographic groups, intended to
decrease stereotyping by showing that not only one
demographic group corresponds to the shown persona. For
example, figure shows that although the dominant
demographic group is male, also females (e.g., Female 18-24           Figure 11: Stepwise explanation of the APG algorithm [3].
India) correspond to the behavioral pattern of the persona.




                                                                      Figure 12: Symbolic explanation of the APG algorithm [2].

                                                                      4 Discussion
                                                                      4.1 Contribution
                                                                      Personas are said to be cognitively compelling [4] and empathetic
                                                                      [21, 25], as they put a human face on otherwise obscure user data.
                                                                      Pruitt and Grudin [30] outline that psychological theory explains
                                                                      why personas should be engaging, pointing out that personas
                                                                      provide a conduit for conveying a broad range of user attributes.
   Figure 10: Distribution of the persona’s topical interests.
                                                                          Yet, personas have been repeatedly challenged in the literature
    To generate the personas, APG uses the underlying data to         for their “imaginary” nature [8], abstraction and lack of credibility
obtain a grouped interaction matrix V (𝑉 = 𝑔 ∗ 𝑐), where the          [23, 32]. DDPs provide features and functionalities that can
columns of the said matrix, for our task, are video content (c) and   provide partial or complete solutions to long-standing persona
the rows represents demographic user groups (g). The element in       weaknesses, such as being slow to create and rapidly expiring [2]
the matrix are the view-counts of the videos for each demographic     and being subjective instead of fact-based [3, 8]. In addition,
group. The system then applied non-negative matrix factorization      personas have been criticized for lack a real value to enhance user
(NMF) [20] to V to discern p latent video viewing behaviors, using    insights, especially in the modern environment with many other
the resultant weights from the NMF. These groups, p, are then         analytics tools are available for probing into online audiences [1].
enriched by adding attributes, including a name, profile picture,     Using digital persona UIs could potentially provide ailments to
their topic of interest among others.                                 enhance the value decision makers get from personas.
    Various ways to explain this algorithmic approach have been
attempted in previous literature, including a simple stepwise list
 ExSS-ATEC'20, March 2020, Cagliari, Italy                                                                                         S.-G. Jung et al.

    Nonetheless, due to prevalence of paper as the default choice       workshops. For persona adoption in the target organization,
of UI for personas, there is currently a lack of empirical studies      explanations can be useful but not necessarily enough.
focused on investigating the UX of digital persona UIs, as most
persona interfaces have not been available in Web systems.              REFERENCES
According to previous literature [37, 38], the concerns relate to       [1]    Alhadreti, O. and Mayhew, P. 2017. To Intervene or Not to Intervene: An
navigation, understanding, and credibility. Users also have                    Investigation of Three Think-aloud Protocols in Usability Testing. Journal
questions about the algorithm behind the personas.                             of Usability Studies,. 12, 3 (2017), 111–132.
                                                                        [2]    An, J. et al. 2018. Customer segmentation using online platforms: isolating
    To this end, this research provides design suggestions and                 behavioral and demographic segments for persona creation via aggregated
ideas for information breakdowns that aim to explain the                       user data. Social Network Analysis and Mining. 8, 1 (2018).
                                                                               DOI:https://doi.org/10.1007/s13278-018-0531-0.
information content in the DDPs and challenge the assumption            [3]    An, J. et al. 2018. Imaginary People Representing Real Numbers: Generating
that the persona is just one person, instead of being representative           Personas from Online Social Media Data. ACM Transactions on the Web
                                                                               (TWEB). 12, 4 (2018), Article No. 27. DOI:https://doi.org/10.1145/3265986.
of a whole group.                                                       [4]    Bødker, S. et al. 2012. Personas, people and participation: challenges from
                                                                               the trenches of local government. Proceedings of the 12th Participatory
4.2 Future Work                                                                Design Conference: Research Papers - Volume 1 (Roskilde, Denmark, 2012),
                                                                               91–100.
First, empirical user studies to test the design ideas outlined here    [5]    Brickey, J. et al. 2012. Comparing Semi-Automated Clustering Methods for
                                                                               Persona Development. IEEE Transactions on Software Engineering. 38, 3 (May
are highly called for. The real impact of explanations on UX needs             2012), 537–546. DOI:https://doi.org/10.1109/TSE.2011.60.
to be corroborated. Particularly, it is essential to address two core   [6]    Chang, Y. et al. 2008. Personas: From Theory to Practices. Proceedings of the
design questions:                                                              5th Nordic Conference on Human-computer Interaction: Building Bridges (New
                                                                               York, NY, USA, 2008), 439–442.
    (a) do the explanations really increase users’ understanding        [7]    Chapman, C.N. et al. 2008. Quantitative Evaluation of Personas as
         about the persona information?                                        Information. Proceedings of the Human Factors and Ergonomics Society
                                                                               Annual Meeting (Sep. 2008), 1107–1111.
    (b) Do breakdowns really reduce stereotypical thinking about        [8]    Chapman, C.N. and Milham, R.P. 2006. The Personas’ New Clothes:
         the persona?                                                          Methodological and Practical Arguments against a Popular Method.
                                                                               Proceedings of the Human Factors and Ergonomics Society Annual Meeting
    Second, user-specific differences towards explanations can
                                                                               (Oct. 2006), 634–636.
affect their implementation [12]. One aspect that makes adding          [9]    Cooper, A. 2004. The Inmates Are Running the Asylum: Why High Tech
explanations challenging is the technical savviness among users.               Products Drive Us Crazy and How to Restore the Sanity (2nd Edition). Pearson
                                                                               Higher Education.
For some users, more technical information can be irrelevant and        [10]   Diakopoulos, N. and Koliska, M. 2017. Algorithmic transparency in the news
even alienate them for the “easily approachable” personas, while               media. Digital Journalism. 5, 7 (2017), 809–828.
                                                                        [11]   Drego, V.L. et al. 2010. The ROI Of Personas. Forrester Research.
others crave for such information. Thus, a design challenge is to       [12]   Eslami, M. et al. 2018. Communicating Algorithmic Process in Online
provide technical explanations for those who need them without                 Behavioral Advertising. Proceedings of the 2018 CHI Conference on Human
interfering with the self-explanatory nature of the persona UI.                Factors in Computing Systems (Montréal, Canada, 2018), 432.
                                                                        [13]   Garfinkel, S. et al. 2017. Toward Algorithmic Transparency and
    Third, experimenting with new explanation types (e.g.,                     Accountability. Commun. ACM. 60, 9 (Aug. 2017), 5–5.
innovative use of video, graphics, product walkthroughs…) is                   DOI:https://doi.org/10.1145/3125780.
                                                                        [14]   Goodman, J. et al. 2007. Formats for User Data in Inclusive Design. Universal
needed. This research focused on a very specific implementation                Acess in Human Computer Interaction. Coping with Diversity (Jul. 2007), 117–
of tooltip explanations, whereas software systems enable other                 126.
complementary approaches that should be tested.                         [15]   Hill, C.G. et al. 2017. Gender-Inclusiveness Personas vs. Stereotyping: Can
                                                                               We Have it Both Ways? Proceedings of the 2017 CHI Conference (Denver,
    Fourth, the described explanation approaches were generic                  Colorado, USA, 2017), 6658–6671.
across all personas. They are tied to shared components such as         [16]   Hilton, K. and Henderson, K. 2008. Developing criminal personas for
                                                                               designers. Papers from the British Criminology Conference. 8, (2008), 175–186.
sections, datasets, and algorithms, but not specific to the current     [17]   Judge, T. et al. 2012. Comparing Collaboration and Individual Personas for
persona being shown. For example, users may wonder why the                     the Design and Evaluation of Collaboration Software. Proceedings of the
                                                                               SIGCHI Conference on Human Factors in Computing Systems (New York, NY,
current persona is of specific age and has specific interests. The             USA, 2012), 1997–2000.
generic explanations are indirect and may require users to figure       [18]   Jung, S. et al. 2018. Automatically Conceptualizing Social Media Analytics
out the exact answers independently. For this reason, persona-                 Data via Personas. (San Francisco, California, USA, Jun. 2018).
                                                                        [19]   Jung, S. et al. 2017. Persona Generation from Aggregated Social Media Data.
specific explanations may be needed.                                           Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors
    Overall, even though APG contains tooltip definitions of each              in Computing Systems (Denver, Colorado, USA, 2017), 1748–1755.
                                                                        [20]   Lee, D.D. and Seung, S.H. 1999. Learning the Parts of Objects by Non-
information section, its algorithmic transparency [44] may not be              negative Matrix Factorization. Nature. 401, 6755 (1999), 788–791.
adequate but additional explanations may be needed.                     [21]   Long, F. 2009. Real or imaginary: The effectiveness of using personas in
                                                                               product design. Proceedings of the Irish Ergonomics Society Annual
                                                                               Conference (2009).
4.3 Practical Implications                                              [22]   Marsden, N. and Haag, M. 2016. Stereotypes and Politics: Reflections on
                                                                               Personas. Proceedings of the 2016 CHI Conference on Human Factors in
Finally, the frequency of understanding issues reported in related             Computing Systems (San Jose, USA, 2016), 4017–4031.
research [36–38] implies there is a need for system-specific            [23]   Matthews, T. et al. 2012. How Do Designers and User Experience
training. For optimal usability, self-explanatory features would be            Professionals Actually Perceive and Use Personas? Proceedings of the SIGCHI
                                                                               Conference on Human Factors in Computing Systems (Austin, Texas, USA,
the ideal design goal. However, interactive persona UIs and DDPs               2012), 1219–1228.
can become complex, prompting the use of various educational            [24]   McGinn, J.J. and Kotamraju, N. 2008. Data-driven persona development.
                                                                               Proceedings of the SIGCHI Conference on Human Factors in Computing
means, such as explainer videos, video tutorials, and collaborative            Systems (Florence, Italy, 2008), 1521–1524.
 Explaining Data-Driven Personas                                                        S.-G. Jung et al.

[25]   Miaskiewicz, T. et al. 2009. A Preliminary Examination of Using Personas to
       Enhance User-Centered Design. AMCIS 2009 Proceedings (2009), Article 697
       http://aisel.aisnet.org/amcis2009/697.
[26]   Miaskiewicz, T. and Kozar, K.A. 2011. Personas and user-centered design:
       How can personas benefit product design processes? Design Studies. 32, 5
       (2011), 417–430.
[27]   Mijač, T. et al. 2018. The potential and issues in data-driven development of
       web personas. 2018 41st International Convention on Information and
       Communication Technology, Electronics and Microelectronics (MIPRO) (May
       2018), 1237–1242.
[28]   Nielsen, L. et al. 2015. A Template for Design Personas: Analysis of 47
       Persona Descriptions from Danish Industries and Organizations. Int. J.
       Sociotechnology Knowl. Dev. 7, 1 (2015), 45–61.
[29]   Nielsen, L. 2019. Personas - User Focused Design. Springer.
[30]   Pruitt, J. and Grudin, J. 2003. Personas: Practice and Theory. Proceedings of
       the 2003 Conference on Designing for User Experiences (San Francisco,
       California, USA, 2003), 1–15.
[31]   Rönkkö, K. 2005. An Empirical Study Demonstrating How Different Design
       Constraints, Project Organization and Contexts Limited the Utility of
       Personas. Proceedings of the Proceedings of the 38th Annual Hawaii
       International Conference on System Sciences - Volume 08 (Washington, DC,
       USA, 2005).
[32]   Rönkkö, K. et al. 2004. Personas is Not Applicable: Local Remedies
       Interpreted in a Wider Context. Proceedings of the Eighth Conference on
       Participatory Design: Artful Integration: Interweaving Media, Materials and
       Practices - Volume 1 (Toronto, Ontario, Canada, 2004), 112–120.
[33]   Russo, J.E. et al. 1989. The validity of verbal protocols. Memory & Cognition.
       17, 6 (1989), 759–769.
[34]   Salminen, J. et al. 2020. A Template for Data-Driven Personas: Analyzing 31
       Quantitatively Oriented Persona Profiles. In the Proceedings of the 22nd
       International Conference on Human-Computer Interaction (HCII’20)
       (Copenhagen, Denmark, Jul. 2020).
[35]   Salminen, J. et al. 2018. Are personas done? Evaluating their usefulness in
       the age of digital analytics. Persona Studies. 4, 2 (Nov. 2018), 47–65.
       DOI:https://doi.org/10.21153/psj2018vol4no2art737.
[36]   Salminen, J. et al. 2019. Confusion and information triggered by photos in
       persona profiles. International Journal of Human-Computer Studies. 129,
       (Sep. 2019), 1–14. DOI:https://doi.org/10.1016/j.ijhcs.2019.03.005.
[37]   Salminen, J. et al. 2019. Design Issues in Automatically Generated Persona
       Profiles: A Qualitative Analysis from 38 Think-Aloud Transcripts.
       Proceedings of the ACM SIGIR Conference on Human Information Interaction
       and Retrieval (CHIIR) (Glasgow, UK, Mar. 2019), 225–229.
[38]   Salminen, J. et al. 2018. Findings of a User Study of Automatically Generated
       Personas. Extended Abstracts of the 2018 CHI Conference on Human Factors
       in Computing Systems (Montréal, Canada, 2018), LBW097:1–LBW097:6.
[39]   Salminen, J. et al. 2018. From 2,772 segments to five personas: Summarizing
       a diverse online audience by generating culturally adapted personas. First
       Monday. 23, 6 (Jun. 2018). DOI:https://doi.org/10.5210/fm.v23i6.8415.
[40]   Salminen, J. et al. 2017. Generating Cultural Personas from Social Data: A
       Perspective of Middle Eastern Users. Proceedings of The Fourth International
       Symposium on Social Networks Analysis, Management and Security (SNAMS-
       2017) (Prague, Czech Republic, 2017).
[41]   Salminen, J. et al. 2018. “Is More Better?”: Impact of Multiple Photos on
       Perception of Persona Profiles. Proceedings of ACM CHI Conference on
       Human Factors in Computing Systems (CHI2018) (Montréal, Canada, Apr.
       2018).
[42]   Salminen, J. et al. 2019. Persona Transparency: Analyzing the Impact of
       Explanations on Perceptions of Data-Driven Personas. International Journal
       of Human–Computer Interaction. 0, 0 (Nov. 2019), 1–13.
       DOI:https://doi.org/10.1080/10447318.2019.1688946.
[43]   Salminen, J. et al. 2019. The future of data-driven personas: A marriage of
       online analytics numbers and human attributes. ICEIS 2019 - Proceedings of
       the 21st International Conference on Enterprise Information Systems
       (Heraklion, Greece, Jan. 2019), 596–603.
[44]   Springer, A. and Whittaker, S. 2019. Progressive disclosure: empirically
       motivated approaches to designing effective transparency. Proceedings of the
       24th International Conference on Intelligent User Interfaces (Marina del Ray,
       California, 2019), 107–120.
[45]   Sproull, L. et al. 1996. When the Interface Is a Face. Human–Computer
       Interaction.          11,         2         (Jun.         1996),      97–124.
       DOI:https://doi.org/10.1207/s15327051hci1102_1.
[46]   Turner, P. and Turner, S. 2011. Is stereotyping inevitable when designing
       with personas? Design studies. 32, 1 (2011), 30–44.
[47]   Zhang, X. et al. 2016. Data-driven Personas: Constructing Archetypal Users
       with Clickstreams and User Telemetry. Proceedings of the 2016 CHI
       Conference on Human Factors in Computing Systems (San Jose, California,
       USA, 2016), 5350–5359.