Toward General Design Principles for Generative AI Applications

Abstract

Generative AI technologies are growing in power, utility, and use. As generative technologies are being incorporated into mainstream applications, there is a need for guidance on how to design those applications to foster productive and safe use. Based on recent research on human-AI co-creation within the HCI and AI communities, we present a set of seven principles for the design of generative AI applications. These principles are grounded in an environment of generative variability. Six principles are focused on designing for characteristics of generative AI: multiple outcomes & imperfection; exploration & control; and mental models & explanations. In addition, we urge designers to design against potential harms that may be caused by a generative model's hazardous output, misuse, or potential for human displacement. We anticipate these principles to usefully inform design decisions made in the creation of novel human-AI applications, and we invite the community to apply, revise, and extend these principles to their own work.

Keywords

• eol>generative AI
• design principles
• human-centered AI
• foundation models

1. Introduction

As generative AI technologies continue to grow in power

and utility, their use is becoming more mainstream. Generative models, including LLM-based foundation models [ 1 ], are being used for applications such as general Q&A (e.g. ChatGPT1), software engineering assistance (e.g. Copilot2), task automation (e.g. Adept3), copywriting (e.g. Jasper.ai4), and the creation of high-fidelity artwork (e.g. DALL-E 2 [ 2 ], Stable Difusion [ 3 ], Midjourney5). Given the explosion in popularity of these new kinds of generative applications, there is a need for guidance on how to design those applications to foster productive and safe use, in line with human-centered AI values [ 4 ].

Fostering productive use is a challenge, as revealed in a recent literature survey by Campero et al. [ 5 ]. They found that many human-AI collaborative systems failed to achieve positive synergy – the notion that a humanAI team is able to accomplish superior outcomes above either party working alone. In fact, some studies have found the opposite efect, that human-AI teams produced inferior results to either a human or AI working alone [ 6, 7, 8, 9 ].

Fostering safe use is a challenge because of the potential risks and harms that stem from generative AI, either because of how the model was trained (e.g. [ 10 ]) or because of how it is applied (e.g. [ 11, 12 ]). tize designers to the idea that generative AI applications

In order to address these issues, we propose a set of may cause a variety of harms (likely inadvertently, but design principles to aid the designers of generative AI sys- possibly intentionally). We hope these principles provide tems. These principles are grounded in an environment the human-AI co-creation community with a reasoned of generative variability, indicating the two proper- way to think through the design of novel generative AI ties of generative AI systems inherently diferent from applications. traditional discriminative6 AI systems: generative, because the aim of generative AI applications is to produce artifacts as outputs, rather than determine decision bound- 2. Design Principles for aries as discriminative AI systems do, and variability, Generative AI Applications indicating the fact that, for a given input, a generative system may produce a variety of possible outputs, many We developed seven design principles for generative AI of which may be valid; in the discriminative case, it is applications based on recent research in the HCI and AI expected that the output of a model does not vary for a communities, specifically around human-AI co-creative given input. processes. We conducted a literature review of research

We note that our principles are meant to generally studies, guidelines, and analytic frameworks from these apply to generative AI applications. Other sets of de- communities [17, 18, 19, 22, 23, 20, 21, 24, 25, 26, 27, sign principles exist for specific kinds of generative AI 28, 29, 30], which included experiments in human-AI applications, including Liu and Chilton’s guidelines for co-creation [ 31, 32, 33, 34, 35, 36, 37 ], examinations of engineering prompts for text-to-image models [ 13 ], and representative generative applications [38, 39, 40, 34, 41, advice about one-shot prompts for generation of texts of 2, 3, 42], and a review of publications in recent workdiferent kinds [ 14, 15, 16 ]. There are also more general shops [ 43, 44, 45, 46 ].

AI-related design guidelines [ 17, 18, 19, 20, 21 ].

Six of our principles are presented as “design for...” 2.1. The Environment: Generative statements, indicating the characteristics that designers Variability should keep in mind when making important design decisions. One is presented as a “design against...” statement, Generative AI technologies present unique challenges directing designers to design against potential harms that for designers of AI systems compared to discriminative may arise from hazardous model outputs, misuse, poten- AI systems. First, generative AI is generative in nature, tial for human displacement, or other harms we have not which means their purpose is to produce artifacts as outyet considered. The principles interact with each other put, rather than decisions, labels, classifications, and/or in complex ways, schematically represented via overlap- decision boundaries. These artifacts may be comprised of ping circles in Figure 1. For example, the characteristic diferent types of media, such as text, images, audio, anidenoted in one principle (e.g. multiple outputs) can some- mations or videos. Second, the outputs of a generative AI times be leveraged as a strategy for addressing another model are variable in nature. Whereas discriminitive AI principle (e.g. exploration). Principles are also connected aims for deterministic outcomes, generative AI systems by a user’s aims, such as producing a singular artifact, may not produce the same output for a given input each seeking inspiration or creative ideas, or learning about a time. In fact, by design, they can produce multiple and domain. They are also connected by design features or divergent outputs for a given input, some or all of which attributes of a generative AI application, such as the sup- may be satisfactory to the user. Thus, it may be dificult port for versioning, curation, or sandbox environments. for users to achieve replicable results when working with

Our aim for these principles is threefold: (1) to pro- a generative AI application. vide the designers of generative AI applications with the Although the very nature of generative applications language to discuss issues unique to generative AI; (2) to violates the common HCI principle that a system should provide strategies and guidance to help designers make respond consistently to a user’s input (for critiques of this important design decisions around how end users will position, see [ 47, 48, 49, 50, 51, 12 ]), we take the position interact with a generative AI application; and (3) to sensi- that this environment in which generative applications 6Our use of the term discriminative is to indicate that the task operate – generative variability – is a core strength. Genconducted by the AI algorithm is one of determining to which erative applications enable users to explore or populate a class or group a data instance belongs; classification and clus- “space” of possible outcomes to their query. Sometimes, tering algorithms are examples of discriminative AI. Although this exploration is explicit, as in the case of systems that our use of the term discriminative may evoke imagery of hu- enable latent space manipulations of an artifact. Other gmeannetdici,scorrimoitnhaetriolnin(ees.)g,. ouvriaursaecifaoll,lorwelsigtihouess,cgieenntdificercoidnevnetnit-y, times, exploration of a space occurs when a generative tion established in the machine learning community (see, e.g., model produces multiple candidate outputs for a given https://en.wikipedia.org/wiki/Discriminative_model) input, such as multiple distinct images for a given prompt [ 2, 3 ] or multiple implementations of a source code pro- generate such earlier outputs? One strategy is to keep gram [ 36, 37 ]. Recent studies also show how users may track of all of these outputs, as well as the parameters that improve their knowledge of a domain by working with a produced them, by versioning them. Such versioning can generative model and its variable outputs [ 36, 42 ]. happen manually (e.g. the user clicks a button to “save”

This concept of generative variability is crucially im- their current working state) or automatically. portant for designers of generative AI applications to communicate to users. Users who approach a generative 2.2.2. Curation AI system without understanding its probabilistic nature and its capacity to produce varied outputs will struggle to interact with it in productive ways. The design principles we outline in the following sections – designing for multiple outcomes & imperfection, for exploration & human control, and for mental models & explanations – are all rooted in the notion that generative AI systems are distinct and unique because they operate in an environment of generative variability.

When a generative model is capable of producing multi

ple outputs, users may need tools to curate those outputs. Curation may include collecting, filtering, sorting, selecting, or organizing outputs (possibly from the versioned queue) into meaningful subsets or groups, or creating prioritized lists or hierarchies of outputs according to some subjective or objective criteria. For example, CogMol7 generates novel molecular compounds, which can be sorted by various properties, such as their molecular weight, toxicity, or water solubility [ 56, 57 ]. In addition, the confidence of the model in each output it produced may be a useful way to sort or rank outputs, although in some cases, model confidence scores may not be indicative of the quality of the model’s output [ 32 ].

2.2. Design for Multiple Outputs

7http://covid19-mol.mybluemix.net

Generative AI technologies such as encoder-decoder models [ 52, 53 ], generative adversarial networks [ 54 ], and transformer models [ 55 ] are probabilistic in nature and thus are capable of producing multiple, distinct outputs for a user’s input. Designers therefore need to under- 2.2.3. Annotation stand the extent to which these multiple outputs should When a generative model has produced a large number be visible to users. Do users need the ability to anno- of outputs, users may desire to add marks, decorators, tate or curate? Do they need the ability to compare or or annotations to outputs of interest. These annotations contrast? How many outputs does a user need? may be applied to the output itself (e.g. “I like this”) or it

Understanding the user’s task can help answer these may be applied to a portion or subset of the output (e.g. questions. If the user’s task is one of production, in which lfagging lines of source code that look problematic and the ultimate goal is to produce a single, satisfying arti- need review). fact, then designs that help the user filter and visualize diferences may be preferable. For example, a software engineer’s goal is often to implement a method that per- 2.2.4. Visualizing Diferences forms a specific behavior. Tools such as Copilot take a In some cases, a generative model may produce a diverse user’s input, such as a method signature or documen- set of distinct outputs, such as images of cats that look tation, and provide a singular output. Contrarily, if the strikingly diferent from each other. In other cases, a user’s task is one of exploration, then designs that help generative model may produce a set of outputs for which the user curate, annotate, and mutate may be preferable. it is dificult to discern diferences, such as a source code For example, a software engineer may wish to explore a translation from one language to another. In this case, space of possible test cases for a code module. Or, an artist tools that aid users in visualizing the similarities and may wish to explore diferent compositions or styles to diferences between multiple outputs can be useful. Desee a broad range of possibilities. Below we discuss a set pending on the users’ goals, they may seek to find the of strategies for helping design for multiple outputs. invariant aspects across outcomes, such as identifying which parts of a source code translation were the same 2.2.1. Versioning across multiple translations, indicating a confidence in its correctness. Or, users may prioritize the variant aspects for greater creativity and inspiration. For example, Sentient Sketchbook [ 58 ] is a video game co-creation system that displays a number of diferent metrics of the maps it generates, enabling users to compare newly-generated maps with their current map to understand how they difer.

Because of the randomness involved in the generative process, as well as other user-configurable parameters (e.g. a random seed, a temperature, or other types of user controls), it may be dificult for a user to produce exactly the same outcome twice. As a user interacts with a generative AI application and creates a set of outputs, they may find that they prefer earlier outputs to later ones.

How can they recover or reset the state of the system to

2.3. Design for Imperfection

may contain imperfections (and thus require human review, discussed further in Weisz et al. [ 36 ]) can be an efective way for handling imperfection.

It is highly important for users to understand that the quality of a generative model’s outputs will vary. Users who expect a generative AI application to produce exactly the artifact they desire will experience frustration when 2.3.3. Co-Creation they work with the system and find that it often pro- User experiences that allow for co-creation, in which duces imperfect artifacts. By “imperfect,” we mean that both the user and the AI can edit a candidate artifact, will the artifact itself may have imperfections, such as visual be more efective than user experiences that assume or misrepresentations in an image, bugs or errors in source aim for the generative model to produce a perfect output. code, missing desired elements (e.g. “an illustration of a Allowing users to edit a model’s outputs provides them bunny with a carrot” fails to include a carrot), violations with the opportunity to find and fix imperfections, and of constraints specified in the input prompt (e.g. “write ultimately achieve a satisfactory artifact. One example a 10 word sentence” produces a much longer or shorter of this idea is Github Copilot [ 62 ], which is embedded sentence), or even untruthful or misleading answers (e.g. in the VSCode IDE. In the case when Copilot produces a summary of a scientific topic that includes non-existent an imperfect block of source code, developers are able to references [ 59 ]). But, “imperfect” can also mean “doesn’t edit it right in context without any friction. By contrast, satisfy the user’s desire,” such as when the user prompts tools like Midjourney or Stable Difusion only produce a model and doesn’t get back any satisfying outputs (e.g. a gallery of images to chose from; editing those images the user didn’t like any of the illustrations of a bunny requires the user to shift to a diferent environment (e.g. with a carrot). Below we discuss a set of strategies for Photoshop). helping design for imperfection. 2.3.4. Sandbox / Playground Environment 2.3.1. Multiple Outputs

A sandbox or playground environment ensures that when

Our previous design principle is also a strategy for han- a user interacts with a generated artifact, their interacdling imperfect outputs. If a generative model is allowed tions (such as edits, manipulations, or annotations) do to produce multiple outputs, the likelihood that one of not impact the larger context or environment in which those outputs is satisfying to the user is increased. One they are working. Returning to the example of Github example of this efect is in how code translation models Copilot, since it is situated inside a developer’s IDE, code are evaluated, via a metric called @ [ 60, 61 ]. The it produces is directly inserted into the working code idea is that the model is allowed to produce code trans- file. Although this design choice enables co-creation, it lations for a given input, and if any of them pass a set of also poses a risk that imperfect code is injected into a unit tests, then the model is said to have produced a cor- production code base. A sandbox environment that rerect translation. In this way, generating multiple outputs quires users to explicitly copy and paste code in order to serves to mitigate the fact that the model’s most-likely commit it to the current working file may guard against output may be imperfect. However, it is left up to the the accidental inclusion of imperfect outputs in a larger user to review the set of outputs and identify the one that environment or product. is satisfactory; with multiple outputs that are very similar to each other, this task may be dificult [ 37 ], implying 2.4. Design for Human Control the need for a way to easily visualize diferences. 2.3.2. Evaluation & Identification Given that generative models may not produce perfect (or perfectly satisfying) outputs, they may still be able to provide users with a signal about the quality of its output, or indicate parts that require human review. As previously discussed, a model’s per-output confidence scores may be used (with care) to indicate the quality of a model’s output. Or, domain-specific metrics (e.g. molecular toxicity, compiler errors) may be useful indicators to evaluate whether an artifact achieved a desirable level of quality. Thus, evaluating the quality of generated artifacts and identifying which portions of those artifacts Keeping humans in control of AI systems is a core tenet of human-centered AI [ 63, 64, 4 ]. Providing users with controls in generative applications can improve their experience by increasing their eficiency, comprehension, and ownership of generated outcomes [ 34 ]. But, in cocreative contexts, there are multiple ways to interpret what kinds of “control” people need. We identify three kinds of controls applicable to generative AI applications. 2.4.1. Generic Controls

One aspect of control relates to the exploration of a de

sign space or range of possible outcomes (as discussed in Section 2.5). Users need appropriate controls in order to drive their explorations, such as control over the num- satisfy the constraints). In either case, the control itself is ber of outputs produced from an input or the amount dependent on the fact that the model is producing a speof variability present in the outputs. We refer to these cific kind of artifact, such as a molecule, and would not kinds of controls as generic controls, as they are applica- logically make sense for other kinds of artifacts in other ble to any particular generative technology or domain. domains (e.g. how would you control the water solubility As an example, some generative projects may involve a for a text-to-image model?). Thus, we refer to these types “lifecycle” pattern in which users benefit from seeing a of controls, independent of how they are implemented, as great diversity of outputs in early stages of the process domain specific . Other examples of domain-specific conin order to search for ideas, inspirations, or directions. trols include the reading level of a text, the color palette Later stages of the project may focus on a smaller number or artistic style of an image, or the run time or memory (or singular) output, requiring controls that specifically eficiency of source code. operate on that output. Many generative algorithms include a user-controllable parameter called temperature. 2.5. Design for Exploration A low temperature setting produces outcomes that are very similar to each other; conversely, a high tempera- Because users are working in an environment of generture setting produces outcomes that are very dissimilar ative variability, they will need some way to “explore” to each other. In the “lifecycle” model, users may first or “navigate” the space of potential outputs in order to set a high temperature for increased diversity, and then identify one (or more) that satisfies their needs. Below reduce it when they wish to focus on a particular area of we discuss a set of strategies for helping design for exinterest in the output space. This efect was observed in ploration. a study of a music co-creation tool, in which novice users dragged temperature control sliders to the extreme ends 2.5.1. Multiple Outputs to explore the limits of what the AI could generate [ 34 ].

The ability for a generative model to produce multiple

outputs (Section 2.2) is an enabler of exploration. Return2.4.2. Technology-specific Controls ing to the bunny and carrot example, an artist may wish Other types of controls will depend on the particular gen- to explore diferent illustrative styles and prompt (and reerative technology being employed. Encoder-decoder prompt) the model for additional candidates of “a bunny models, for example, often allow users to perform latent with a carrot” in various kinds of styles or configurations. space manipulations of an artifact in order to control Or, a developer can explore diferent ways to implement semantically-meaningful attributes. For example, Liu an algorithm by prompting (and re-prompting) a model to and Chilton [ 65 ] demonstrate how semantic sliders can produce implementations that possess diferent attributes be used to control attributes of 3D models of animals, (e.g. “implement this using recursion,” “implement this such as the animal’s torso length, neck length, and neck using iteration,” or “implement this using memoization”). rotation. Transformer models use a temperature parame- In this way, a user can get a sense of the diferent possiter to control the amount of randomness in the genera- bilities the model is capable of producing. tion process [ 66 ]. Natural language prompting, and the emerging discipline of prompt engineering [ 13 ], provide 2.5.2. Control additional ways to tune or tweak the outputs of large language models. We refer to these kinds of controls as technology-specific controls , as the controls exposed to a user in a user interface will depend upon the particular generative AI technology used in the application.

Depending on the specific technical architecture used by the generative application, there may be diferent ways for users to control it (Section 2.4). No matter the specific mechanisms of control, providing controls to a user provides them with the ability to interactively work with the model to explore the space of possible outputs for their given input. 2.4.3. Domain-specific Controls Some types of user controls will be domain-specific, dependent on the type of artifact being produced. For exam- 2.5.3. Sandbox / Playground Environment ple, generative models that produce molecules as output might be controlled by having the user specify desired properties such as molecular weight or water solubility; these types of constraints might be propagated to the model itself (e.g. expressed as a constraint in the encoder phase), or they may simply act as a filter on the model’s output (e.g. hide anything from the user that doesn’t A sandbox or playground environment can enable exploration by providing a separate place in which new candidates can be explored, without interfering with a user’s main working environment. For example, in a project using Copilot, Cheng et al. [ 67 ] suggest providing, “a sandbox mechanism to allow users to play with the prompt in the context of their own project.” 2.5.4. Visualization partner? does it act proactively or does it just respond to the user? does it make changes to an artifact directly or does it simply make recommendations for the user?

2.7. Design for Explanations

One way to help users understand the space in which

they are exploring is to explicitly visualize it for them.

Kreminski et al. [ 33 ] introduce the idea of expressive range coverage analysis (ERCA) in which a user is shown a visualization of the “range” of possible generated artifacts across a variety of metrics. Then, as users interact with the system and produce specific artifact instances, those instances are included in the visualization to show how much of the “range” or “space” was explored by the user.

Generative AI applications will be unfamiliar and possibly unusual to many users. They will want to know what the application can (and cannot) do, how well it works, and how to work with it efectively. Some users may even wish to understand the technical details of how the underlying generative AI algorithms work, although these details may not be necessary to work efectively 2.6. Design for Mental Models with the model (as discussed in [ 36 ]). In recent years, the explainable AI (XAI) community Users form mental models when they work with techno- has made tremendous progress at developing techniques logical systems [ 68, 69, 70 ]. These models represent the for explaining how AI systems work [ 76, 77, 21, 78, 79 ]. user’s understanding of how the system works and how Much of the work in XAI has focused on discriminative to work with it efectively to produce the outcomes they algorithms: how they generally make decisions (e.g. via desire. Due to the environment of generative variabil- interpretable models [80, Chapter 5] or feature impority, generative AI applications will pose new challenges tance [80, Section 8.5], and why they make a decision in a to users because these applications may violate existing specific instance (e.g. via counterfactual explanations [ 80, mental models of how computing systems behave (i.e. Section 9.3]. in a deterministic fashion). Therefore, we recommend Recent work in human-centered XAI (HCXAI) has designing to support users in creating accurate mental emphasized designing explanations that cater to human models of generative AI applications in the following knowledge and human needs [ 77 ]. This work grew out of ways. a general shift toward human-centered data science [ 47 ], in which the import of explanations is not for a technical 2.6.1. Orientation to Generative Variability user (data scientist), but for an end user who might be Users may need a general introduction to the concept impacted by a machine learning model. of generative AI. They should understand that the sys- In the case of generative AI, recent work has begun tem may produce multiple outputs for their query (Sec- to explore the needs for explainability. Sun et al. [ 35 ] extion 2.2), that those outputs may contain flaws or im- plored explainability needs of software engineers workperfections (Section 2.3), and that their efort may be ing with a generative AI model for various types of use required to collaborate with the system in order to pro- cases, such as code translation and autocompletion. They duce desired artifacts via various kinds of controls (Sec- identified a number of types of questions that software tion 2.4). engineers had about the generative AI, its capabilities, and its limitations, indicating that explainability is an im2.6.2. Role of the AI portant feature for generative AI applications. They also identified several gaps in existing explainability frameResearch in human-AI interaction suggests that users works stemming from the generative nature of the AI may view an AI application as filling a role such as an system, indicating that existing XAI techniques may not assistant, coach, or teammate [ 29 ]. In a study of video be suficient for generative AI applications. Thus, we game co-creation, Guzdial et al. [ 71 ] found participants make the following recommendations for how to design to ascribe roles of friend, collaborator, student, and man- for explanations. ager to the AI system. Recent work by Ross et al. [ 42 ] examined software engineers’ role orientations toward 2.7.1. Calibrate Trust by Communicating a programming assistant and found that people viewed Capabilities and Limitations the assistant with a tool orientation, but interacted with it as if it were a social agent. Clearly establishing the role Because of the inherent imperfection of generative AI of a generative AI application in a user’s workflow, as outputs, users would be well-served if they understood well as its level of autonomy (e.g. [ 72, 73, 74, 75 ]), will the limitations of these systems [ 81, 82 ], allowing them help users better understand how to interact efectively to calibrate their trust in terms of what the application with it. Designers can reason about the role of their ap- can and cannot do [ 83 ]. When these kinds of imperfecplication by answering questions such as, is it a tool or tions (Section 2.3) are not signaled, users of co-creative tools may mistakenly blame themselves for shortcomings mans, and cultures. Even with our focus on the design of generated artifacts in co-creative applications [ 34 ], of generative applications, an analysis of harms that is and users in Q & A use cases can be shown deceptive limited to design concepts may blur into technosolutionmisconceptions and harmful falsehoods as objective an- ism [ 89, 90, 91 ]. swers [ 84 ]. One way to communicate the capabilities of We do posit that human-centered approaches to genera generative AI application is to show examples of what ative AI design are a useful first step, but must be part it can do. For example, Midjourney provides a public dis- of a larger strategy to understand who are the direct and cussion space to orient new users and show them what indirect stakeholders of a generative application [ 92, 93 ], other users have produced with the model. This space and to work directly with those stakeholders to identify not only shows the outputs of the model (e.g. images), harms, understand what are their difering priorities and but the textual prompts that produced the images. In this value tensions [94], and negotiate issues of culture, policy, way, users can more quickly come to understand how and (yes) technology to meet these diverse challenges diferent prompts influence the application’s output. To (e.g., [95, 96, 97]). communicate limitations, systems like ChatGPT contain modal screens to inform users of the system’s limitations. 2.8.1. Hazardous Model Outputs

Generative AI applications may produce artifacts that

2.7.2. Use Explanations to Create and Reinforce cause harm. In an integrative survey paper, Weidinger

Accurate Mental Models et al. [ 10 ] list six types of potential harms of large lanWeisz et al. [ 36 ] explored how a generative model’s con- guage models, three of which regard the harms that may ifdence could be surfaced in a user interface. Working be caused by the model’s output: with a transformer model on a code translation task, • Discrimination, Exclusion, and Toxicity. Generthey developed a prototype UI that highlighted tokens ative models may produce outputs that promote disin the translation that the model was not confident in. crimination against certain groups, exclude certain In their user study, they found that those highlights also groups from representation, or produce toxic content. served as explanations for how the model worked: users Examples include text-to-image models that fail to procame to understand that each source code token was cho- duce ethnically diverse outputs for a given input (e.g. sen probabilistically, and that the model had considered a request for images of doctors produces images of other alternatives. This design transformed an algorith- male, white doctors [98] or language models that promic weakness (imperfect output) into a resource for users duce inappropriate language such as swear words, hate to understand how the algorithm worked, and ultimately, speech, or ofensive content [ 17, 20 ]. to control its output (by showing users where they might need to make changes).

2.8. Design Against Harms

The use of AI systems – including generative AI applications – may unfortunately lead to diverse forms of harms, especially for people in vulnerable situations. Much work in AI ethics communities has identified how discriminative AI systems may perpetuate harms such as the denial of personhood or identity [ 49, 85, 86 ]; the deprivation of liberty or children [ 87, 88 ], and the erasure of persons, cultures, or nations through data silences [ 81 ]. We identify four types of potential harms, some of which are unique to the generative domain, and others which represent existing risks of AI applications that may manifest in new ways.

Our aim in this section is to sensitize designers to the potential risks and harms that generative AI systems may pose. We do not prescribe solutions to address these risks, in part because it is an active area of research to understand how these kinds of risks could be mitigated.

Risk identification, assessment, and mitigation is a sociotechnical problem involving computing resources, hu• Information Hazards. Generative models may inadvertently leak private or sensitive information from their training data. For example, Carlini et al. [99] found that strategically prompting GPT-2 revealed an individual’s full name, work address, phone number, email, and fax number. Additionally, larger models may be more vulnerable to these types of attacks [99, 100]. • Misinformation Harms. Generative models may produce inaccurate misinformation in response to a user’s query. Lin et al. [ 84 ] found that GPT-3 can provide false answers that mimic human falsehoods and misconceptions, such as “coughing can help stop a heart attack” or “[cold weather] tells us that global warming is a hoax”.

Singhal et al. [101] caution against the tendency of LLMs to hallucinate references, especially if consulted for medical decisions. Albrecht et al. [102] claim that LLMs have few defenses against adversarial attacks while advising about ethical questions. The Galactica model was found to hallucinate non-existent scientific references [103], and Stack Overflow has banned responses sourced from ChatGPT due to their high rate of incorrect, yet plausible, responses [104].

In addition to those harms, a generative model’s out- or multiple people may be required to review or approve puts may be hazardous in other ways as well. a model’s outputs before they can be used. 2.8.3. Human Displacement • Deceit, Impersonation, and Manipulation. Generative algorithms can be used to create false records or “deep fakes” (e.g., [ 11, 105 ]), to impersonate others (e.g. [106]), or to distort information into politically-altered content [107]. In addition, they may manipulate users who believe that they are chatting with another human rather than with an algorithm, as in the case of an unreviewed ChatGPT “experiment” in which at least 4,000 people seeking mental health support were connected to a chatbot rather than a human counselor [108].

One consequence of the large-scale deployment of generative AI technologies is that they may come to replace, rather than augment human workers. Such concerns have been raised in related areas, such as the use of automated AI technologies in data science Wang et al. [114, 115]. Weidinger et al. [ 10 ] specifically discuss the potential economic harms and inequalities that may arise as a consequence of widespread adoption of generative AI. If a generative model is capable of producing high• Copyright, Licenses, and Intellectual Property. fidelity outputs that rival (or even surpass) what can Generative models may have been trained on data pro- be created by human efort, are the humans necessary tected by regulations such as the GDPR, which pro- anymore? Contemporary fears of human displacement hibits the re-use of data beyond the purposes for which by generative technologies are beginning to manifest in it was collected. In addition, large language models mainstream media, such as in the case of illustrators’ have been referred to as “stochastic parrots” due to concerns that text-to-image models such as Stable Diftheir ability to reproduce data that was used during fusion and Midjourney will put them out of a job [116]. their training [109]. One consequence of this efect We urge designers to find ways to design generative AI is that the model may produce outputs that incorpo- applications that enhance or augment human abilities, rate or remix materials that are subject to copyright or rather than applications that aim to replace human workintellectual property protections [110, 111, 112]. For ex- ers. Copilot serves as one example of a tool that clearly ample, the Codex model, which produces source code enhances the abilities of a software engineer: it operates as output, may (re-)produce source code that is copy- on the low-level details of a source code implementation, righted or subject to a software license, or that was freeing up software engineers to focus more of their atopenly shared under a creative commons license that tention on higher-level architectural and system design prohibits commercial re-use (e.g., in a pay-to-access issues.

LLM). Thus, the use of a model’s outputs in a project may cause that project to violate copyright protections, or subject that project to a restrictive license (e.g. GPL). 3. Discussion As of this writing, there is a lawsuit against GitHub, Microsoft, and OpenAI on alleged copyright violations 3.1. Designing for User Aims in the training of Codex [113].

Users of generative AI applications may have varied aims

or goals in using those systems. Some users may be in 2.8.2. Misuse pursuit of perfecting a singular artifact, such as a method Weidinger et al. [ 10 ] describe how generative AI appli- implementation in a software program. Other users may cations may be misused in ways unanticipated by the be in pursuit of inspiration or creative ideas, such as when creators of those systems. Examples include making dis- exploring a visual design space. As a consequence of information cheaper and more efective, facilitating fraud working with a generative AI application, users may also and scams, assisting code generation for cyberattacks, or enhance their own learning or understanding of the domain conducting illegitimate surveillance and censorship. In in which they are operating, such as when a software addition to these misuses, Houde et al. [ 11 ] also identify engineer learns something new about a programming business misuses of generative AI applications such as language from the model’s output. Each of these aims facilitating insurance fraud and fabricating evidence of can be supported by our design principles, as well as a crime. Although designers may not be able to prevent help designers determine the appropriate strategy for users from intentionally misusing their generative AI addressing the challenges posed by each principle. applications, there may be preventative measures that To support artifact production, designers ought to caremake sense for a given application domain. For exam- fully consider how to manage a model’s multiple, imperple, output images may be watermarked to indicate they fect outputs. Interfaces ought to support users in curating, were generated by a particular model, blocklists may be annotating, and mutating artifacts to help users refine a used to disallow undesirable words in a textual prompt, singular artifact. The ability to version artifacts, or show a history of artifact edits, may also be useful to enable a generative AI application, and explicitly enumerating their values, designers can make more reasoned judgments about how those stakeholders might be impacted by hazardous model outputs, model misuse, and issues of human displacement.

4. Limitations and Future Work

Generative AI applications are still in their infancy, and new kinds of co-creative user experiences are emerging at a rapid pace. Thus, we consider these principles to be in their infancy as well, and it is possible that other important design principles, strategies, and/or user aims have been overlooked. In addition, although these principles can provide helpful guidance to designers in making specific design decisions, they need to be validated in real-world settings to ensure their clarity and utility. 5. Conclusion users to revisit discarded options or undo undesirable modifications. For cases in which users seek to produce one “ideal” artifact that satisfies some criteria, controls that enable them to co-create with the generative tool can help them achieve their goal more eficiently, and explanations that signal or identify imperfections can tell them how close or far they are from the mark.

To support inspiration and creativity, designers also ought to provide adequate controls that enable users to explore a design space of possibilities [ 33, 117 ]. Visualizations that represent the design space can also be helpful as they can show which parts the user has vs. has not explored, enabling them to explore the novel parts of that space. Tools that help users manage, curate, and filter the diferent outputs created during their explorations can be extremely helpful, such as a digital mood board for capturing inspiring model outputs.

Finally, to support learning how to efectively interact with a generative AI application, designers ought to help users create accurate mental models [118] through explanations [ 76, 77, 21, 78, 79 ]. Explanations can help answer general questions such as what a generative AI application is capable or not capable of generating, how the model’s controls impact its output, and how the model was trained and the provenance of its training data. They can also answer questions about a specific model output, such as how confident the model was in that output, which portions of that output might need human review or revision, how to adjust or modify the input or prompt to adjust properties of the output, or what other options or alternatives exist for that output.

We present a set of seven design principles for generative

AI applications. These principles are grounded in an environment of generative variability, the key characteristics of which are that a generative AI application will generate artifacts as outputs, and those outputs may be varied in nature (e.g. of varied quality or character). The principles focus on designing for multiple outputs and the imperfection of those outputs, designing for exploration of a space or range of possible outputs and maintaining human control over that exploration, and designing to establish accurate mental models of the generative AI 3.2. The Importance of Value-Sensitive application via explanations. We also urge designers to design against the potential harms that may be caused

Design in Mitigating Potential Harms by hazardous model output (e.g. the production of inDesigners need to be sensitive to the potential harms that appropriate language or imagery, the reinforcement of may be caused by the rapid maturation and widespread existing stereotypes, or a failure to inclusively represent adoption of generative AI technologies. Although so- diferent groups), by misuse of the model (e.g. by creating ciotechnical means for mitigating these harms have yet to disinformation or fabricating evidence), or by displacing be developed, we recommend that designers use a Value human workers (e.g. by designing for the replacement Sensitive Design approach [ 92, 93 ] when reasoning about rather than the augmentation of human workers). We enhow to design generative AI applications. By clearly iden- vision these principles to help designers make reasoned tifying the diferent stakeholders and impacted parties of choices as they create novel generative AI applications.

References

[1] R. Bommasani , D. A. Hudson , E. Adeli, R. Altman , S. Arora , S. von Arx, M. S. Bernstein , J. Bohg , A. Bosselut , E. Brunskill , et al., On the opportunities and risks of foundation models , arXiv preprint arXiv:2108.07258 ( 2021 ).

[2] A. Ramesh , P. Dhariwal , A. Nichol , C. Chu , M. Chen , Hierarchical text-conditional image generation with clip latents , arXiv preprint arXiv:2204.06125 ( 2022 ).

[3] R. Rombach , A. Blattmann , D. Lorenz , P. Esser , B. Ommer , High-resolution image synthesis with latent difusion models , in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022 , pp. 10684 - 10695 .

[4] B. Shneiderman , Human-Centered AI , Oxford University Press, 2022 .

[5] A. Campero , M. Vaccaro , J. Song , H. Wen , A. Almaatouq , T. W. Malone , A test for evaluating performance in human-computer systems , arXiv preprint arXiv:2206.12390 ( 2022 ).

[6] E. Clark , A. S. Ross , C. Tan , Y. Ji , N. A. Smith , Creative writing with a machine in the loop: Case studies on slogans and stories , in: 23rd International Conference on Intelligent User Interfaces , 2018 , pp. 329 - 340 .

[7] Z. Buçinca , M. B. Malaya , K. Z. Gajos , To trust or to think: cognitive forcing functions can reduce overreliance on ai in ai-assisted decision-making , Proceedings of the ACM on Human-Computer Interaction 5 ( 2021 ) 1 - 21 .

[8] M. Jacobs , M. F. Pradier , T. H. McCoy , R. H. Perlis , F. Doshi-Velez , K. Z. Gajos, How machine-learning recommendations influence clinician treatment selections: the example of antidepressant selection , Translational psychiatry 11 ( 2021 ) 1 - 9 .

[9] B. Kleinberg , B. Verschuere , How humans impair automated deception detection performance , Acta Psychologica 213 ( 2021 ) 103250 .

[10] L. Weidinger , J. Mellor , M. Rauh , C. Grifin , J. Uesato , P.-S. Huang, M. Cheng, M. Glaese, B. Balle , A. Kasirzadeh , et al., Ethical and social risks of harm from language models , arXiv preprint arXiv:2112.04359 ( 2021 ).

[11] S. Houde , V. Liao , J. Martino , M. Muller , D. Piorkowski , J. Richards , J. D. Weisz , Y. Zhang, Business (mis)use cases of generative ai, in: Joint Proceedings of the Workshops on Human-AI CoCreation with Generative Models and User-Aware Conversational Agents co-located with 25th International Conference on Intelligent User Interfaces (IUI 2020 ), 2020 .

[12] M. Muller , S. I. Ross , S. Houde , M. Agarwal , F. Martinez , J. T. Richards , K. Talamadupula , J. D. Weisz , Drinking chai with your (AI) programming partner: A design fiction about generative AI for software engineering 107-122 , in: A. Smith-Renner , O. Amir (Eds.), Joint Proceedings of the IUI 2022 Workshops: APEx-UI, HAI-GEN, HEALTHI, HUMANIZE, TExSS, SOCIALIZE co-located with the ACM International Conference on Intelligent User Interfaces (IUI 2022 ), Virtual Event, Helsinki, Finland, March 21 -22, 2022 , volume 3124 of CEUR Workshop Proceedings, CEUR-WS.org , 2022 , pp. 107 - 122 .

[13] V. Liu , L. B. Chilton , Design guidelines for prompt engineering text-to-image generative models , in: CHI Conference on Human Factors in Computing Systems , 2022 , pp. 1 - 23 .

[14] C. Greyling , Prompt engineering, text generation and large language models , 2022 . URL: https://cobusgreyling.medium. com/promptengineering-text-generation-large-languagemodels-3d90c527c6d5.

[15] L. Reynolds , K. McDonell , Prompt programming for large language models: Beyond the few-shot paradigm , in: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems , 2021 , pp. 1 - 7 .

[16] P. Denny , V. Kumar , N. Giacaman , Conversing with copilot: Exploring prompt engineering for solving cs1 problems using natural language , 2022 . URL: https://arxiv.org/abs/2210.15157.

[17] ACM , Words matter: Alternatives for charged terminology in the computing profession , 2023 . URL: https://www.acm.org/diversity-inclusion/ words-matter.

[18] S. Amershi , D. Weld , M. Vorvoreanu , A. Fourney , B. Nushi , P. Collisson , J. Suh , S. Iqbal , P. N. Bennett , K. Inkpen , et al., Guidelines for human-ai interaction , in: Proceedings of the 2019 chi conference on human factors in computing systems , 2019 , pp. 1 - 13 .

[19] A. Computer , Human interface guidelines, 2022 . URL: https://developer.apple.com/design/humaninterface-guidelines/guidelines/overview.

[20] IBM , Racial equity in design, 2023 . URL: https:// www.ibm.com/design/racial-equity-in-design/.

[21] Q. V. Liao , D. Gruen , S. Miller , Questioning the ai: informing design practices for explainable ai user experiences , in: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , 2020 , pp. 1 - 15 .

[22] S. Deterding , J. Hook , R. Fiebrink , M. Gillies , J. Gow , M. Akten , G. Smith , A. Liapis , K. Compton , Mixed-initiative creative interfaces , in: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems , 2017 , pp. 628 - 635 .

[23] I. Grabe , M. González-Duque , S. Risi , J. Zhu , Towards a framework for human-ai interaction patterns in co-creative gan applications , Joint Proceedings of the ACM IUI Workshops 2022 , March 2022 , Helsinki, Finland ( 2022 ).

[24] M. L. Maher , Computational and collective creativity: Who's being creative? , in: ICCC, Citeseer , 2012 , pp. 67 - 71 .

[25] M. L. Maher , B. Magerko , D. Venura , D. Fisher, R. Cardona-rivera, N. Fulda, J. Gooth , M. Lee , D. Wilson , J. Kaufman, et al., A research plan for integrating generative and cognitive ai for human centered, explainable co-creative ai , in: ACM CHI Conference on Human Factors in Computing Systems , 2022 .

[26] M. Muller , J. D. Weisz , W. Geyer, Mixed initiative generative ai interfaces: An analytic framework for generative ai applications , ICCC 2020 Workshop, The Future of Co-Creative Systems , 2020 . URL: https://computationalcreativity.net/ workshops/cocreative-iccc20/papers/Future_of_ co-creative _systems_185.pdf .

[27] M. Muller , J. Weisz , Extending a human-ai collaboration framework with dynamism and sociality , in: 2022 Symposium on Human-Computer Interaction for Work , 2022 , pp. 1 - 12 .

[28] T. Lubart , How can computers be partners in the creative process: classification and commentary on the special issue , International Journal of Human-Computer Studies 63 ( 2005 ) 365 - 369 .

[29] I. Seeber , E. Bittner, R. O. Briggs , T. De Vreede, G. -J. De Vreede , A. Elkins , R. Maier , A. B. Merz , S. OesteReiß, N. Randrup , et al., Machines as teammates: A research agenda on ai in team collaboration , Information & management 57 ( 2020 ) 103174 .

[30] A. Spoto , N. Oleynik , Library of mixedinitiative creative interfaces , 2017 . URL: http://mici.codingconduct.cc/.

[31] M. Agarwal , J. J. Barroso , T. Chakraborti , E. M. Dow , K. Fadnis , B. Godoy , M. Pallan , K. Talamadupula , Project clai: Instrumenting the command line as a new environment for ai agents , arXiv preprint arXiv: 2002 . 00762 ( 2020 ).

[32] M. Agarwal , K. Talamadupula , S. Houde , F. Martinez , M. Muller , J. Richards , S. Ross , J. D. Weisz , Quality estimation & interpretability for code translation , in: Proceedings of the NeurIPS 2020 Workshop on Computer-Assisted Programming (NeurIPS 2020 ), 2020 .

[33] M. Kreminski , I. Karth, M. Mateas , N. WardripFruin, Evaluating mixed-initiative creative interfaces via expressive range coverage analysis ., in: IUI Workshops , 2022 , pp. 34 - 45 .

[34] R. Louie , A. Coenen , C. Z. Huang , M. Terry , C. J. Cai , Novice-ai music co-creation via ai-steering tools for deep generative models , in: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , 2020 , pp. 1 - 13 .

[35] J. Sun , Q. V. Liao , M. Muller , M. Agarwal , S. Houde , K. Talamadupula , J. D. Weisz , Investigating explainability of generative ai for code through scenario-based design , in: 27th International Conference on Intelligent User Interfaces , 2022 , pp. 212 - 228 .

[36] J. D. Weisz , M. Muller , S. Houde , J. Richards , S. I. Ross , F. Martinez , M. Agarwal , K. Talamadupula , Perfection not required? human-ai partnerships in code translation , in: 26th International Conference on Intelligent User Interfaces , 2021 , pp. 402 - 412 .

[37] J. D. Weisz , M. Muller , S. I. Ross , F. Martinez , S. Houde , M. Agarwal , K. Talamadupula , J. T. Richards , Better together? an evaluation of aisupported code translation , in: 27th International Conference on Intelligent User Interfaces , 2022 , pp. 369 - 391 .

[38] T. Brown , B. Mann , N. Ryder , M. Subbiah , J. D. Kaplan , P. Dhariwal , A. Neelakantan , P. Shyam , G. Sastry , A. Askell , S. Agarwal , A. Herbert-Voss , G. Krueger, T. Henighan , R. Child , A. Ramesh , D. Ziegler , J. Wu , C. Winter , C. Hesse , M. Chen , E. Sigler, M. Litwin , S. Gray , B. Chess , J. Clark , C. Berner , S. McCandlish , A. Radford , I. Sutskever , D. Amodei , Language models are few-shot learners , in: H. Larochelle , M. Ranzato , R. Hadsell , M. Balcan , H. Lin (Eds.), Advances in Neural Information Processing Systems , volume 33 , Curran Associates , Inc., 2020 , pp. 1877 - 1901 . URL: https://proceedings.neurips.cc/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf .

[39] T. E. Johnson , Y. Lee , M. Lee , D. L. O'Connor , M. K. Khalil , X. Huang , Measuring sharedness of team-related knowledge: Design and validation of a shared mental model instrument , Human Resource Development International 10 ( 2007 ) 437 - 454 .

[40] B. Kaiser , A. Csiszar , A. Verl , Generative models for direct generation of cnc toolpaths , in: 2018 25th International Conference on Mechatronics and Machine Vision in Practice (M2VIP) , IEEE, 2018 , pp. 1 - 6 .

[41] C. Metz , Meet gpt-3. it has learned to code (and blog and argue ). (published 2020 ), 2022 . URL: https://www.nytimes.com/ 2020 /11/24/science/ artificial-intelligence- ai-gpt3 .html.

[42] S. I. Ross , F. Martinez , S. Houde , M. Muller , J. D. Weisz , The programmer's assistant: Conversational interaction with a large language model for software development , in: 28th International Conference on Intelligent User Interfaces , 2023 .

[43] W. Geyer , L. B. Chilton , J. D. Weisz , M. L. Maher , Hai-gen 2021 : 2nd workshop on human-ai co-creation with generative models , in: 26th International Conference on Intelligent User InterfacesCompanion , 2021 , pp. 15 - 17 .

[44] M. Muller , L. B. Chilton , A. Kantosalo , C. P. Martin , G. Walsh, Genaichi: Generative ai and hci , in: CHI Conference on Human Factors in Computing Systems Extended Abstracts , 2022 , pp. 1 - 7 .

[45] M. Muller , P. Agelov , H. Daume , Q. V. Liao , N. Oliver , D. Piorkowski , et al., Hcai@neurips 2022 , human centered ai , in: Annual Conference on Neural Information Processing Systems , 2022 .

[46] J. D. Weisz , M. L. Maher , H. Strobelt , L. B. Chilton , D. Bau , W. Geyer, Hai-gen 2022 : 3rd workshop on human-ai co-creation with generative models , in: 27th International Conference on Intelligent User Interfaces , 2022 , pp. 4 - 6 .

[47] C. Aragon , S. Guha , M. Kogan , M. Muller , G. Nef, Human-Centered Data Science: An Introduction , MIT Press, 2022 .

[48] D. Boyd , K. Crawford , Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon , Information, communication & society 15 ( 2012 ) 662 - 679 .

[49] S. Costanza-Chock , Design justice: Communityled practices to build the worlds we need , The MIT Press, 2020 .

[50] C. D'ignazio , L. F. Klein , Data feminism , MIT press, 2020 .

[51] L. Gitelman , Raw Data is an Oxymoron , MIT Press, 2013 .

[52] I. Sutskever , O. Vinyals , Q. V. Le , Sequence to sequence learning with neural networks , Advances in neural information processing systems 27 ( 2014 ).

[53] K. Cho , B. Van Merriënboer , C. Gulcehre , D. Bahdanau , F. Bougares , H. Schwenk , Y. Bengio , Learning phrase representations using rnn encoderdecoder for statistical machine translation , arXiv preprint arXiv:1406.1078 ( 2014 ).

[54] I. Goodfellow , J. Pouget-Abadie , M. Mirza , B. Xu , D. Warde-Farley , S. Ozair , A. Courville , Y. Bengio , Generative adversarial networks , Communications of the ACM 63 ( 2020 ) 139 - 144 .

[55] A. Vaswani , N. Shazeer , N. Parmar , J. Uszkoreit , L. Jones , A. N. Gomez , Ł. Kaiser, I. Polosukhin , Attention is all you need , Advances in neural information processing systems 30 ( 2017 ).

[56] V. Chenthamarakshan , P. Das , S. C. Hofman , H. Strobelt , I. Padhi, K. W. Lim , B. Hoover , M. Manica , J. Born , T. Laino , et al., Cogmol: targetspecific and selective drug design for covid-19 using deep generative models , arXiv preprint arXiv: 2004 . 01215 ( 2020 ).

[57] V. Chenthamarakshan , P. Das , I. Padhi , H. Strobelt , K. W. Lim , B. Hoover , S. C. Hofman , A. Mojsilovic , Target-specific and selective drug design for covid-19 using deep generative models , 2020 . arXiv: 2004 .01215.

[58] A. Liapis , G. N. Yannakakis , J. Togelius , et al., Sentient sketchbook: Computer-aided game level authoring ., in: FDG , 2013 , pp. 213 - 220 .

[59] J. Rose , Facebook pulls its new 'ai for science' because it's broken and terrible , Vice ( 2022 ). URL: https://www.vice.com/en/article/3adyw9/ facebook-pulls -its-new-ai-for-science-becauseits-broken-and-terrible.

[60] B. Roziere , M. - A. Lachaux , L. Chanussot , G. Lample, Unsupervised translation of programming languages ., in: NeurIPS, 2020 .

[61] S. Kulal , P. Pasupat , K. Chandra , M. Lee , O. Padon , A. Aiken , P. S. Liang , Spoc: Search-based pseudocode to code , Advances in Neural Information Processing Systems 32 ( 2019 ).

[62] Github , Copilot, 2021 . URL: https: //copilot.github.com.

[63] B. Shneiderman , Human-centered artificial intelligence: Reliable, safe & trustworthy, International Journal of Human-Computer Interaction 36 ( 2020 ) 495 - 504 .

[64] B. Shneiderman , Human-centered ai , Issues in Science and Technology 37 ( 2021 ) 56 - 61 .

[65] V. Liu , L. B. Chilton , Neurosymbolic generation of 3d animal shapes through semantic controls ., in: IUI Workshops , 2021 .

[66] P. von Platen , How to generate text: using different decoding methods for language generation with transformers , Hugging Face Blog ( 2020 ). URL: https://huggingface.co/blog/how-to-generate.

[67] R. Cheng, R. Wang, T. Zimmermann , D. Ford , "it would work for me too": How online communities shape software developers' trust in aipowered code generation tools , arXiv preprint arXiv:2212.03491 ( 2022 ).

[68] M. Scheutz , S. A. DeLoach , J. A. Adams , A framework for developing and using shared mental models in human-agent teams , Journal of Cognitive Engineering and Decision Making 11 ( 2017 ) 203 - 224 .

[69] S. M. Fiore , E. Salas , J. A. Cannon-Bowers , Group dynamics and shared mental model development, How people evaluate others in organizations 234 ( 2001 ).

[70] J. E. Mathieu , T. S. Hefner , G. F. Goodwin , E. Salas , J. A. Cannon-Bowers , The influence of shared mental models on team process and performance ., Journal of applied psychology 85 ( 2000 ) 273 .

[71] M. Guzdial , N. Liao , J. Chen , S.-Y. Chen, S. Shah , V. Shah , J. Reno , G. Smith , M. O. Riedl , Friend, collaborator, student, manager : How design of an ai-driven game level editor afects creators , in: Proceedings of the 2019 CHI conference on human factors in computing systems , 2019 , pp. 1 - 13 .

[72] P. M. Fitts , M. Viteles , N. Barr , D. Brimhall , G. Finch , E. Gardner , W. Grether , W. Kellum , S. Stevens , Human engineering for an efective air-navigation and trafic-control system, and appendixes 1 thru 3 , Technical Report , Ohio State Univ Research Foundation Columbus, 1951 .

[73] T. B. Sheridan , W. L. Verplank , Human and computer control of undersea teleoperators , Technical Report, Massachusetts Inst of Tech Cambridge Man-Machine Systems Lab , 1978 .

[74] R. Parasuraman , T. B. Sheridan , C. D. Wickens , A model for types and levels of human interaction with automation , IEEE Transactions on systems, man, and cybernetics-Part A: Systems and Humans 30 ( 2000 ) 286 - 297 .

[75] E. Horvitz , Principles of mixed-initiative user interfaces , in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '99 , Association for Computing Machinery, New York, NY, USA, 1999 , p. 159 - 166 . URL: https://doi.org/10.1145/302979.303030. doi: 10 .1145/302979.303030.

[76] V. Arya , R. K. Bellamy , P.-Y. Chen, A. Dhurandhar , M. Hind , S. C. Hofman , S. Houde , Q. V. Liao , R. Luss , A. Mojsilovic , et al., Ai explainability 360 : An extensible toolkit for understanding data and machine learning models ., J. Mach. Learn. Res . 21 ( 2020 ) 1 - 6 .

[77] U. Ehsan , P. Wintersberger , Q. V. Liao , E. A. Watkins , C. Manger , H. Daumé III , A. Riener , M. O. Riedl , Human-centered explainable ai (hcxai): beyond opening the black-box of ai , in: CHI Conference on Human Factors in Computing Systems Extended Abstracts , 2022 , pp. 1 - 7 .

[78] Q. V. Liao , M. Singh , Y. Zhang , R. Bellamy, Introduction to explainable ai, in: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems , 2021 , pp. 1 - 3 .

[79] A. Simkute , A. Surana , E. Luger, M. Evans , R. Jones , Xai for learning: Narrowing down the digital divide between “new” and “old” experts , in: Adjunct Proceedings of the 2022 Nordic Human-Computer Interaction Conference , 2022 , pp. 1 - 6 .

[80] C. Molnar, Interpretable machine learning , Lulu. com , 2020 .

[81] M. Muller , A. Stroymayer , Forgetting practices in the data sciences , in: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , 2022 . In press.

[82] C. Pinhanez , Expose uncertainty, instill distrust, avoid explanations: Towards ethical guidelines for ai , in hcai@neurips 2021 workshop , 2021 . URL: https://www.google. com/url?q=https%3A% 2F%2Farxiv.org%2Fabs%2F2112 .01281& sa =D.

[83] C. Pinhanez , Breakdowns, language use, and weird errors: Past, present, and future of research on conversational agents at brl , in ibm research cambridge lab guess speaker series, 2022 .

[84] S. Lin , J. Hilton , O. Evans , Truthfulqa: Measuring how models mimic human falsehoods , arXiv preprint arXiv:2109.07958 ( 2021 ).

[85] S. Kantayya , Coded bias, 2020 . URL: https://www.pbs.org/independentlens/ documentaries/coded-bias/.

[86] K. Spiel , ” why are they all obsessed with gender?”-(non) binary navigations through technological infrastructures , in: Designing Interactive Systems Conference 2021 , 2021 , pp. 478 - 494 .

[87] A. Lyn , Risky business: Artificial intelligence and risk assessments in sentencing and bail procedures in the united states , Available at SSRN 3831441 ( 2020 ).

[88] D. Saxena , K. Badillo-Urquiola , P. J. Wisniewski , S. Guha , A framework of high-stakes algorithmic decision-making for the public sector developed through a case study of child-welfare , Proceedings of the ACM on Human-Computer Interaction 5 ( 2021 ) 1 - 41 .

[89] S. Lindtner , S. Bardzell , J. Bardzell , Reconstituting the utopian vision of making: Hci after technosolutionism , in: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems , 2016 , pp. 1390 - 1402 .

[90] M. A. Madaio , L. Stark , J. Wortman Vaughan , H. Wallach , Co-designing checklists to understand organizational challenges and opportunities around fairness in ai , in: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , 2020 , pp. 1 - 14 .

[91] A. Resseguier , R. Rodrigues , Ethics as attention to context: recommendations for the ethics of artificial intelligence , Open Research Europe 1 ( 2021 ) 27 .

[92] B. Friedman , D. G. Hendry , Value sensitive design: Shaping technology with moral imagination , Mit Press, 2019 .

[93] D. G. Hendry , B. Friedman , S. Ballard , Value sensitive design as a formative framework , Ethics and Information Technology 23 ( 2021 ) 39 - 44 .