<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <article-id pub-id-type="doi">10.23919/TST.2017.8195348</article-id>
      <title-group>
        <article-title>Towards a Framework for Human-AI Interaction Patterns in Co-Creative GAN Applications</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Imke Grabe</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Miguel González-Duque</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Sebastian Risi</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Jichen Zhu</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>IT University of Copenhagen (ITU)</institution>
          ,
          <addr-line>Rued Langgaards Vej 7, 2300 Copenhagen</addr-line>
          ,
          <country country="DK">Denmark</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2018</year>
      </pub-date>
      <volume>3313831</volume>
      <fpage>4401</fpage>
      <lpage>4410</lpage>
      <abstract>
        <p>With the rise of Generative Adversarial Networks (GANs), AI has increasingly become a partner to human designers in co-creating cultural artifacts. While generative models have been applied in various creative tasks across disciplines, a theoretical foundation for understanding human-GAN collaboration is yet to be developed. Drawing from the mixed-initiative co-creation community, we propose a preliminary framework to analyze co-creative GAN applications. We identify four primary interaction patterns: Curating, Exploring, Evolving, and Conditioning. The suggested framework enables us to discuss the afordances and limitations of the diferent kind of interactions underlying co-creative GAN applications.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;generative adversarial networks</kwd>
        <kwd>mixed-initiative co-creation</kwd>
        <kwd>human-AI collaboration</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        co-creation with AI agents [8]. For example, Spoto and
Oleynik [13] propose a framework to map the primary
With the recent development in human-AI collaborative interactions taken in the co-creative process between
applications, deep generative models have become poten- humans and computational agents [13]. Muller et al. [14]
tial co-designers to humans in creative tasks. Especially further expand the above-mentioned framework by
inGenerative Adversarial Networks (GANs) have gained at- cluding actions specific to generative AI interfaces. While
tention from designers and non-programmers across cre- Muller et al.’s expanded framework provides a tool for
ative disciplines such as architecture, fashion, computer analyzing generative AI interfaces, its extensive action
games, and art [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. Over the recent years, GANs have set is not aligned with how the functioning of latent
been of particular interest to the human-computer inter- variable models, like GANs’ generators, integrates into
action (HCI) community for their ability to create high- co-creation. We argue that a smaller but more specific set
resolution output [
        <xref ref-type="bibr" rid="ref2 ref3 ref4">2, 3, 4</xref>
        ] . of technically-grounded actions is suficient to describe
      </p>
      <p>
        While technological advancements underlying GANs GANs’ interactive capabilities accurately.
progress rapidly [
        <xref ref-type="bibr" rid="ref5 ref6 ref7">5, 6, 7</xref>
        ], how to develop GANs that We hence suggest tailoring the framework to
cocan efectively co-create with human designers is still creative GAN applications by 1) reducing the action space
an open problem. Recent HCI research has identified to a minimal set and 2) incorporating new actions better
various challenges for designers to work with machine aligned with the algorithmic properties of GANs. With
learning (ML) models [8, 9]. More specifically, Buschek this vocabulary, we aim to describe co-creative GAN
et al. [10] outline the challenges in designing interactions applications while providing insight into GANs’ inner
with generative models. A key dificulty when designing workings. We developed and tested our framework by
anwith GANs is the knowledge gap regarding the technical alyzing related literature in art, computer games, fashion,
possibilities and limitations of the models [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. Due to and object design.
its complexity, designers who apply GANs in creative This paper aims to answer the question: How do
coprocesses cannot oversee their technical functioning. At creative GAN applications support co-creativity? To do
the same time, ML engineers who develop the algorithmic so, we adapt an existing MI-CC framework previously
aspects of the models are not acquainted with the design applied to analyze a broad range of co-creative
interrequirements of a particular creative task. faces [13], and further developed to describe generative
      </p>
      <p>Research in mixed-initiative co-creation (MI-CC) [11, models [14]. Based on the MI-CC frameworks, we
sug12] ofers a promising starting point to bridge this knowl- gest a preliminary framework for in-depth analysis of
edge gap. The research area of MI-CC focuses on how hu- co-creative GAN applications. We show that the
sugmans and machines can co-create together, including the gested taxonomy lets us analyze emerging patterns in
the interaction with GANs. We limit our study to GANs
because the majority of examples in the field of
generative design use them, but we believe that our framework
applies to other deep generative models. More
specifically, the proposed set of actions supports the mapping
Joint Proceedings of the ACM IUI Workshops 2022, March 2022,
Helsinki, Finland
nEvelop-O imgr@itu.dk (I. Grabe); migd@itu.dk (M. González-Duque);
sebr@itu.dk (S. Risi); jicz@itu.dk (J. Zhu)</p>
      <p>© 2022 Copyright © 2022 for this paper by its authors. Use permitted under
CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCreEatUiveRCoWmmoornksLsichenospeAPttrribouctioene4d.0iInntgersna(tiConEal U(CCRB-YW4.0S)..org)
of interactions with generators via latent codes. Hence, proposing new GAN applications [21, 22, 23, 24] or
sugthese actions are applicable to other generative models, gesting algorithmic methods to control GANs without
such as variational autoencoders (VAEs), consisting of a pre-defined use cases [25, 26, 27]. As part of a study on
generator network with outputs that are sampled based AI-augmented typeface design, Zeng et al. [28] frame a
on alterable latent codes. GAN as a source of inspiration in a creativity model. To</p>
      <p>
        By synthesizing existing GAN applications and MI- date, the only (systematic) review in the area is Hughes
CC theory, our work aims to shed light on how GANs et al.’s [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ] survey on collaborative applications of GANs
participate in co-creative design processes. With the in design tasks. Their analysis focuses on the design
framework’s GAN-specific grounding, we hope to make domains, inspecting applications of GANs by discipline.
the technical and thereby interactive capabilities better Across all approaches, they identify beautification and
understandable when mapping interactions. By increas- variation as modes of operation. While this helps us
uning awareness of the system’s functioning, this insight derstand what the GAN adds to the creative process, it
might allow us, and non-experts in particular, to design does not reflect on how it is involved in co-creation. We
better interactive applications. With the identification aim to dig deeper into the mechanisms at play and the
of emerging patterns, we aim to enable a discussion on creative role of GANs in collaborative tasks.
the current trends in co-creative GAN applications and
illuminate possible future research in this field. 2.2. Mapping mixed-initiative co-creation
      </p>
      <p>In summary, this paper presents our preliminary
framework for mapping the interaction in co-creative GAN The research area MI-CC investigates interactive
proapplications and discusses the co-creative afordance of cesses in which both human and machine contribute
the emerging interaction patterns. The rest of the paper proactively to producing artifacts [12]. MI-CC
appliis structured as follows. In the next section, we sum- cations can be placed on a continuum, describing the
marize related work. Then, we present the framework extent to which human and computational agent take
supported by exemplary co-creative GAN applications initiative in the creative process [29, 30]. To analyze the
drawing on existing literature. We discuss how the iden- initiatives taken in such collaboration, several theoretical
tified interaction patterns support co-creativity, followed approaches have been suggested. Spoto and Oleynik [13]
by a conclusion and outlook on future work. proposed to decompose the MI-CC process into seven
primary actions: ideate, constrain, produce, suggest,
select, assess, adapt. The actions can be used to map the
2. Related Work “creative flow” [13] between human and computer as a
graph. By mapping the series of actions taken by the two
In this section, we summarize theoretical work on under- agents, the authors outline the creative (iterative) process
standing co-creative GAN applications and approaches for more than 70 works ranging from game level creation
to map MI-CC interactions. to manufacturing design systems. When it comes to
including AI agents in co-creation with human designers,
2.1. Understanding human-AI new interaction possibilities emerge, requiring
comprecollaboration with GANs hensive analysis frameworks. Muller et al. [14] adapt
the notation of Spoto and Oleynik’s framework to
processes including generative AI agents. They add four
actions that are specific to generative models, such as
learn to describe the computer’s training process. Their
extended framework can be used to visualize “human-AI
interaction patterns in the generative space” [14, p.1].</p>
      <p>Our work aims to tailor these MI-CC methods to analyze
co-creative GAN applications. We build on existing
mapping methods suggested by MI-CC to outline common
interaction patterns in co-creation between human and
GAN.</p>
      <p>
        At the intersection of HCI and AI research, designing
human-AI interaction remains an open issue due to the
complexity of AI models and their lack of
interpretability [9, 15]. This includes designing co-creative GAN
applications. GANs learn to generate content from latent
features, but this mapping is not explainable and
blackbox, making it dificult to control the desired output after
training [
        <xref ref-type="bibr" rid="ref5 ref7">5, 7</xref>
        ]. As a generative model, GANs are a novel
design tool that brings new challenges when it comes
to interaction [10]. Though non-programmers and
designers have applied the tool, a theoretical framework
for analyzing GANs’ co-creative potential is yet to be
developed. While surveys have been conducted on the
technical aspects [16, 17, 18, 19] as well as its
application in creative domains [20], there has not been much
work on understanding GANs’ role in co-creative
applications. Research in the area mainly consists of studies
      </p>
    </sec>
    <sec id="sec-2">
      <title>3. Mapping interaction patterns with GANs (GAN-MIP)</title>
      <sec id="sec-2-1">
        <title>We suggest a framework that applies mixed-initiative</title>
        <p>methods to understand how GANs function in
cocreation. As a starting point of this study, we reviewed Initialize. To initialize refers to setting up the GAN,
GAN applications in salient design domains. Reviewed such as choosing a dataset and a model architecture.
studies ranged from prompting GANs to invent new art Learn. To learn describes the process of internalizing
styles [31], over breeding game levels [21], to adjusting information. In line with Muller et al. [14], this refers
fashion items as part of a person’s outfit [25]. To make to the training GANs on a chosen data distribution. For
sense of the surveyed GAN studies, we leveraged exist- the human counterpart, the action would correspond to
ing MI-CC frameworks to map the actions in co-creative learning new skills or adapting to a new domain.
tasks. They allowed us to analyze how GANs get involved Constrain. To constrain refers to the process of
speciin creative processes. In an iterative manner, we analyzed fying desired characteristics of the target artifacts, hence
studies and revised the frameworks to derive one that is restricting the conceptual space.
applicable across co-creative GAN applications. During Create. To create describes the action of generating
this process, we focused on how the frameworks’ actions new (candidates of) artifacts. The number of artifacts to
reflect the GANs’ technical functioning. Along the way, be created can vary from one to many.
predominant flows in how humans and GANs interact Select. To select refers to the action of choosing one or
in co-creation stood out among the studies, which we more artifacts and excluding others from the further
prorefer to as interaction patterns. This section presents the cess. The action encompasses diferent selection
mechasuggested framework and demonstrates the four primary nisms including the structuring of subsets such as
rankinteraction patterns we identified with it. ing.</p>
        <p>Adapt. To adapt describes the act of making changes
3.1. Framework to existing artifacts. The agents can edit artifacts directly
or adjust their representation, namely the latent code,
Our framework builds on the framework suggested by that is used to create a new artifact.
Spoto and Oleynik [13], further extended to map co- Combine. To combine describes the process of
concreation with generative models by Muller et al. [14]. structing a new artifact that inherits parts of existing</p>
        <p>We adapt their proposed actions to a minimal set re- artifacts. As with adapting, this is done via the artifacts’
quired to describe co-creative GAN applications. For latent code.
example, Muller et al. distinguish between producing one The actions described above present a minimal set
sufiartifact and suggesting a set of artifacts to choose from cient to describe the core interactions of co-creative GAN
as separate actions. We combine them into one category applications. For example, we left out the action ideate,
called create, as the artifact sampling from the GAN de- which describes creating high-level concepts in existing
picts the same action in either case. In addition, we add frameworks [13, 14]. As we find that conceptual ideation
new actions, such as initialize, which we find can present is often expressed by conditions imposed on a GAN, the
a crucial design choice taken by human agents [24]. The action constrain covers it in our framework, aligned with
ifnal set of actions illustrates the co-creation as a graph the GANs’ algorithmic function. The derived action set
mapping along the two axes of agents and actions. This aims to align with the algorithmic properties of GANs
section defines the agents making up the horizontal axis as well as the interactive abilities of the human designer.
and explains what the actions on the vertical axis stand Mapping interactions with the framework may help
nonfor. technical users better understand a co-creative partner’s
algorithmic actions and show how the human designer’s
3.1.1. Agents actions embed into it. The set of core actions allows us to
map the interactions of the co-creative GAN applications
we surveyed.</p>
        <sec id="sec-2-1-1">
          <title>3.2. Primary Interaction Patterns</title>
          <p>Using the above framework of agents and actions, we
identified four interaction patterns which we present
along with an exemplary selection of co-creative GAN
applications in which they appear. We reviewed studies
that propose models for co-creation with GANs tested
in user studies, such as a GAN that allows game
designers to generate Mario levels [21]. Besides that, we also
considered approaches that are not explicitly embedded
into use cases yet but suggest applicable models that
The agents conducting the actions are (1) a human
designer and (2) a computational system including a GAN
as the main technology, which might be supported by
other algorithms facilitating the interaction with it.
3.1.2. Actions
The actions are to be understood as activities that
influence the artifact or the other agent directly. How actions
can be executed depends on several factors, such as the
design of the interface, the implementation details of a
model, or the abilities of a human agent. Seven actions
are used to map the interaction between the two agents,
as described below.
could support co-creative GAN applications. For exam- They replace the production step of design drawing in a
ple, Yildirim et al. [26] trained a GAN to generate dress fashion design workflow with GAN creations. Then, the
designs based on given features such as color. Among all output design is given to pattern makers.
studies, we found four underlying interaction patterns Besides selecting the training data, the model
architecfor which we show the graphical notations in Figure 1. ture can also pose a design factor. Elgammal et al. [31]
Interaction patterns are not mutually exclusive and can demonstrate that with art-generating Creative
Adversarbe combined. Table 1 provides an overview of how the ial Networks (CANs). They apply a style ambiguity loss
surveyed examples apply the patterns. when training the networks on an art dataset assigned
to diferent art styles as categories. As a result, the CAN
3.2.1. Curating learns to output artifacts that deviate from existing art
styles. While there are no limitations to the
complexThe most straight-forward interaction pattern is catego- ity of designing the initial GAN set-up, Curating is the
rized as Curating. After being initialized by a human simplest form for GANs to partake in the subsequent
designer, a GAN learns a to generate artifacts by being creative process.
trained through gradient descent. Then, a set of outputs
is created by sampling from the GAN, from which the 3.2.2. Exploring
human agent selects a set of final artifacts. Note that the
selection can also include all generated artifacts. After The interaction pattern Exploring describes an
interacthe training, the GAN is the only one influencing the tion where the human iteratively adapts artifacts that
creation process. It does not receive any input from the are created by sampling from the GAN. This is typically
human side. The main creative human input is given done by introducing alterations via the latent
representaduring the initialization. Hence, choosing a dataset and a tions of artifacts. Altering the latent code of an artifact
model plays a crucial role in the human agent’s initiative. can be understood as providing a recipe to create a new</p>
          <p>Curating has been applied to produce new artifacts by artifact. By steering the following creation process, the
adding variation to an existing dataset in several domains. designer guides the exploration path through the space
For instance, fashion designers might purposely choose a of possible designs. The latent space can be explored by
collection of clothing designs as training distribution.1 In traversing along diferent directions. These directions
that way, they determine the overall style of newly gen- cause diferent changes to the output features. Users can
erated artifacts to which they aim to add novelty through control the change in direction by e.g. moving a slider.
the GAN’s variation. Kato et al. [32] train a GAN to pro- We present three common directions to adapt artifacts
duce outputs of similar quality to human-made designs. below. Figure 2 provides a visual example per direction.</p>
          <p>
            Interpolation. One direction to move an artifact
through latent space is towards other artifacts in the
1https://www.vogue.com/fashion-shows/fall-2020menswear/acne-studios
space. In doing so, one explores intermediate alterna- gled during the training process. In practice, diferent
tives between existing artifacts. Among other methods, attributes link to diferent layers and sliders let users
Schrum et al. [21] apply interpolation to explore the de- manipulate target features. On top of that, Artbreeder
sign space of GANs trained to create tile-based rooms allows users to introduce semantic directions based on
for the game The Legend of Zelda. By traversing be- sample images containing target features.
tween two diferent level designs, the game designer can The features along which users explore the conceptual
prompt the GAN to create new designs lying in between. space can also be combined. For example, CREA.blender
CREA.blender [33], a tool that tests users’ creativity in SDG [
            <xref ref-type="bibr" rid="ref4">4</xref>
            ], a tool to develop utopian and dystopian
landcreating GAN outputs, allows the exploration of the inter- scape images, lets users create interpolations between
mediate space of up to five pre-selected images. Sliders images as in CREA.blender. With the generator from
let the user control the strength of an image’s represen- a StyleGAN, the computational agent learned to
distintation in the generated result. guish between content and style of the environment. In
          </p>
          <p>Latent directions. Instead of altering GAN-generated the interaction, two sliders below each image let the
huartifacts in relation to one another, they can also be man agent control how much of its content and style feed
adapted by directly increasing or decreasing their latent into the resulting design. Hence, the simple interpolation
variables. As a result, the artifact moves along the GAN’s of artifacts expands to consider semantic properties.
latent directions. In another part of Schrum et al.’s study, Bau et al. [38] also bridge the gap between the
interusers adjust a game level design by moving a slider per nal representation of GANs and interpretable concepts.
latent variable. Because the latent variables do not cor- By analyzing how the latter encode in the former, they
respond to visual output features, changes to the output propose GANDissect, a method to identify latent units
design are not necessarily explainable, which users might in the generator’s layers that activate certain aspects in
experience as frustrating when having a target change the output image. Taken to practice in the application
in mind [21]. GANPaint [39], the authors propose an interface that</p>
          <p>Semantic features. The adaption along directions in lets users add, delete, or alter aspects in images. Using a
latent space that represent semantic attributes is better brush to indicate target locations, or pixels, in the image,
understandable. After finding such an attribute vector the human agent can adapt GAN-generated artifacts by
in the latent space, an artifact’s latent code can traverse indirectly de-/activating the corresponding latent units.
along it. That influences the appearance of the attribute By representing actual photographs as a GAN’s
generain the artifact. Research in the domain of semantic face tion, the changes can be applied to existing images. This
editing [34, 35] has widely explored this technique to allows a human agent “to manipulate a photograph not
alter facial features. For example, the editing function with physical colors, but with abstract concepts such as
in the interface of the Composite Generating GAN (CG- object types and visual attributes” [39, p.2].
GAN) [23] supports interaction in that manner. Human
agents adapt generated faces by adding or subtracting 3.2.3. Evolving
certain facial traits. They may choose to lock other facial
features during that process to avoid correlated changes. In the third identified pattern, Evolving, the human
se</p>
          <p>
            ArtBreeder [36], a website where users can make GAN- lects artifacts for the computational agent to adapt and
generated art, applies another approach to disentangle combine in the following step. Within the field of
evolusemantic features, resulting in a similar control to CG- tionary computation, these actions are also referred to as
GAN for a faces generator. By applying the BigGAN [37] mutation and recombination, respectively. This
interacor StyleGAN [
            <xref ref-type="bibr" rid="ref7">7</xref>
            ] architecture, categories present in the tion utilizes interactive evolutionary computation [40],
training data, such as facial attributes, can be disentan- more specifically an interactive genetic algorithm (IGA).
Here, the latent codes represent the genotype of artifacts,
which are in turn considered as the phenotype. For a Hence, a requirement is that the GAN architecture can
number of iterations, users make a selection from a pop- take conditions into account when sampling artifacts.
ulation of artifacts to be adjusted for the next generation. The constraint by the human can be expressed in
sevIn addition, users might be given the option to frame this eral modalities, of which three examples are depicted in
process by specifying the exact action for an artifact or Figure 3 and described in the following. In the domain
the operation parameters. While Exploring implies that of fashion design, one can think of several parameters
the human agent directly controls the features that arti- that might guide the creation of a clothing item. Yildirim
facts are adapted by, Evolving suggests they only select et al. [42] for instance propose a GAN model that
disenartifacts for the computational agent to develop further. tangles color, texture, and shape through diferent loss
          </p>
          <p>Bontrager et al. [41] let human agents evolve faces, functions. During training, parts of the latent code are
shoes, and chairs in that manner. Users select artifacts assigned to the three features. With this model, a user
from the first creation step. Their latent codes are then could pre-define the shape of a dress in a hand drawing.
recombined though crossover and mutated by adding The drawing is then captured as a 512-dimensional mask
Gaussian noise. With the adapted latent codes, new items and included into the input given to the GAN. Similarly,
are sampled and the process repeats. With the power Xin and Arakawa [43] apply a GAN model that takes
to guide the process through selecting artifacts to sur- a contour image as input as part of their object design
vive, the human agent optimizes the resulting artifacts system. In that, the human designer can determine an
according to their preferences. It is the computational outline before the GAN generates the artifact, such as the
agent that adjusts the artifacts primarily. However, the contour of a shoe. Zhao and Ma [44] note that providing
human agent can influence that, too, by specifying the a conditional GAN with a hand-drawn constraint can
rules for this traversal search [12]. In practice, this would cause dificulties when the drawing style difers from the
imply setting algorithmic parameters such as mutation distribution of training drawings [44].
or crossover rate. Schrum et al. demonstrated that in the Setting landmarks of a drawing may help further define
evolving functionality of their study [21]. Here, users a target design. Therefore, the authors apply a second
select generated game levels to be recombined and mu- generative network in the constraint step, that transforms
tated for the next generation, also referred to as selective those landmarks into a compensation image. Together
breeding. By moving a slider, the user sets the rate for with the user’s drawing, it acts as a reference to guide
the applied polynomial mutation. In the evolving process the generation by the main GAN. The example shows
underlying CG-GAN, the user can choose along which that constraints can consist of multiple inputs, as is also
semantic features the algorithms should mutate artifacts. the case in the following case for clothing design
sysSimilar directions for traversing the latent space as pre- tem suggested by Zhu et al. [25]. The authors let human
sented in section 3.2.2 could also be specified for the designers guide the fashion design flow by giving an
computational agent. The human agent might even se- instruction in text form. First, a segmentation mask is
lect them during the design process. However, as the derived from the photo of the person to be dressed.
Tohuman agent does not directly adapt artifacts in this pat- gether with the text encoding of an outfit description,
tern, the directions do not concern the interaction and the mask is given to their GAN model as input. This
are not stated here. allows them to dress a person according to a human’s
idea. The GAN contributes creative initiative through
3.2.4. Conditioning unpredictable variation.</p>
        </sec>
      </sec>
      <sec id="sec-2-2">
        <title>Conditioning describes an interaction in which the hu</title>
        <p>man constrains the desired artifact characteristics before
outputs are created through sampling from the GAN.</p>
        <sec id="sec-2-2-1">
          <title>3.3. Combination of interaction patterns</title>
        </sec>
      </sec>
      <sec id="sec-2-3">
        <title>The primary interaction patterns presented above are</title>
        <p>to be understood as an elementary set of interactions
between human and GAN. As some of the reviewed
applications show, they can be combined to create more
advanced co-creative processes.</p>
        <p>For instance, Xin and Arakawa [43] combine Evolving
and Conditioning in their suggestion of an object design
system. Figure 4 shows the interaction pattern for the
system trained on datasets of shoes, handbags, and other
fashion items. In the first step of their user study, users
evolved shoe designs in an IGA without the option to
constrain the design space, following simply the Evolving
pattern. Secondly, they extend the generation process
so that users could prompt the GAN’s creation with a
contour image of the desired shoe (see Figure 3). By
adding Conditioning to the Evolving interaction pattern,
users can better derive designs that resemble the target
sneaker, as the example in Figure 4 shows.</p>
        <p>Instead of allowing the user to pre-limit the conceptual
space of a GAN, Zaltron et al. [23] give users the chance to
adapt generated images. The interaction underlying their
proposed CG-GAN combines the Evolving and Exploring
pattern, as displayed in Figure 5. First, the model lets
users evolve faces similar to Xin and Arakawa’s IGA for
object design. But during the process, users can edit
artifacts by stepping out of the evolutionary loop. In
doing so, users fine-tune a generated design by traversing
along semantic facial features with the help of sliders.
The evolved artifact is then added back into the loop.</p>
        <p>Oppositely, we can observe the Exploring
pattern being expanded with Evolving functionalities in
CREA.blender [33]. In its simplest form, CREA.blender
asks users to develop animal-like shapes based on
preselected basis images. Then, in the open-play mode, users
can themselves select basis images by replacing the
current ones. With the selected images as a starting point,
they explore new images. In comparison to the “more
goal-oriented creative tasks” [33, p.4], letting the user
take part in the selection supports their “open-ended
creativity” [33, p.4].</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>4. Discussion</title>
      <p>isolated process. However, in the case of
Conditioning, a user might map the constraint space and learn
In this section we discuss the co-creative afordance of its constraint language by iterating the pattern
repeatthe interaction patterns and reflect on the potential usage edly. While our preliminary framework does not
curof the framework. rently capture this step, this points to the asymmetry</p>
      <p>The interaction patterns we identified based on the in our framework, focusing on the GAN-specific
techframework allow us to analyze how co-creative GAN ap- nical grounding of actions rather than human-specific
plications support co-creativity to diferent degrees. For aspects.
example, where Muller et al. [14] identify one interaction Opposed to the one-shot patterns, Exploring and
Evolvpattern, we distinguish between Curating and Condition- ing imply iterative interaction between human and GAN.
ing. More specifically, they categorize Elgammal et al.’s While human initiative appears only after the first
creCAN and Zhao and Ma’s design method into the same in- ation stage in both patterns, they support the co-creative
teraction pattern characterized by the population of a so- exchange diferently. Through selection in an Evolving
lution space with a generative model. However, the two lfow, the human agent narrows the conceptual space of
examples are fundamentally diferent in how they sup- the GAN. Hence, they restrict its creative potential to
deport co-creativity, and our proposed adaption allows us veloping a chosen population of designs. In comparison,
to make a fine-grained distinction between the two ways Exploring leaves the conceptual space as is but lets the
of interacting. Curating, as in the case of CAN, lets the human designer traverse through it. However, the
direchuman pre-determine the GAN’s design space through tions taken restrict the exploration. Freely altering the
its initialization as an active design choice. Hence, the latent variables allows for a broader exploration than the
human agent’s primary input limits the GAN’s creative interpolation between two artifacts, whereas traversing
potential before training. Conditioning allows the hu- along semantic features limits the search to given paths.
man to narrow the existing design space of the GAN after Overall, Exploring afords co-creativity by simply
allowtraining. We categorize Zhao and Ma’s example as such ing the human agent to move in the conceptual space.
because the human agent steers the GANs by providing In contrast, Evolving supports a more targeted search
constraints, in this case, a drawing and landmarks, after strategy for the user but restricts the GAN’s creative
the initial training. While Curating leaves the human potential.
agent out of the actual (co-)creative process and gives the As the identified patterns support diferent purposes,
GAN complete creative control, Conditioning restricts their mapping allows for a reflection about their
applithe GAN’s creativity by assigning the human agent more cation and combination. When designing a process that
creative authority along the way, hence allowing for more starts from pre-defined characteristics, modeling the
inco-creativity. By committing to a human-set frame, the teraction according to the Conditioning pattern would
GAN loses parts of its creative potential, leading to a let humans set the scope right from the beginning of a
trade-of between novelty and typicality with regards to creation process. But when designing for a scenario that
humans’ expectations. requires targeted search for distinguished properties in</p>
      <p>We regard Curating and Conditioning as one-shot ap- a design space, Evolving might be the pattern of choice.
proaches since the computational agent does not recall Exploring, on the other hand, could support a more
inspipreviously created artifacts, making each iteration an rational excursion through artifact possibilities. When</p>
    </sec>
    <sec id="sec-4">
      <title>5. Conclusion and Future Work</title>
      <p>Our preliminary framework suggests mapping
interactions as a basis for understanding co-creative GAN
applications. We identified four primary interaction patterns,
which we demonstrated along with examples of existing
approaches. The patterns let us understand how
diferent GAN applications support co-creativity. This insight
can inform how we design co-creative GAN applications.</p>
      <p>The preliminary framework contributes to bridging the
knowledge gap between machine learning engineers and
[26] G. Yildirim, N. Jetchev, R. Vollgraf, U. Bergmann, //ieeexplore.ieee.org/document/9157070/. doi:10.</p>
      <p>Generating High-Resolution Fashion Model Images 1109/CVPR42600.2020.00926.</p>
      <p>Wearing Custom Outfits, 2019. URL: http://arxiv. [36] J. Simon, Artbreeder, 2022. URL: https://www.
org/abs/1908.08847. artbreeder.com/browse.
[27] B. Roziere, F. Teytaud, V. Hosu, H. Lin, J. Rapin, [37] A. Brock, J. Donahue, K. Simonyan, Large Scale
M. Zameshina, O. Teytaud, EvolGAN: Evolutionary GAN Training for High Fidelity Natural Image
Generative Adversarial Networks, 2020. URL: http: Synthesis, arXiv:1809.11096 [cs, stat] (2019). URL:
//arxiv.org/abs/2009.13311. http://arxiv.org/abs/1809.11096, arXiv: 1809.11096.
[28] Z. Zeng, X. Sun, X. Liao, Artificial Intelligence [38] D. Bau, J.-Y. Zhu, H. Strobelt, B. Zhou, J. B.
TenenAugments Design Creativity: A Typeface Fam- baum, W. T. Freeman, A. Torralba, GAN
Dissecily Design Experiment | SpringerLink, in: De- tion: Visualizing and Understanding Generative
sign, User Experience, and Usability. User Expe- Adversarial Networks, in: Proceedings of the
Interrience in Advanced Technological Environments, national Conference on Learning Representations
2019. URL: https://link.springer.com/chapter/10. (ICLR), 2019, p. 19.</p>
      <p>1007/978-3-030-23541-3_29. [39] D. Bau, H. Strobelt, W. Peebles, J. Wulf, B. Zhou,
[29] S. Deterding, J. Hook, R. Fiebrink, M. Gillies, J. Gow, J.-Y. Zhu, A. Torralba, Semantic photo
manipulaM. Akten, G. Smith, A. Liapis, K. Compton, Mixed- tion with a generative image prior, ACM
TransInitiative Creative Interfaces, in: Proceedings of actions on Graphics 38 (2019) 1–11. URL: https:
the 2017 CHI Conference Extended Abstracts on //dl.acm.org/doi/10.1145/3306346.3323023. doi:10.
Human Factors in Computing Systems, ACM, Den- 1145/3306346.3323023.
ver Colorado USA, 2017, pp. 628–635. URL: https: [40] H. Takagi, Interactive evolutionary computation:
//dl.acm.org/doi/10.1145/3027063.3027072. doi:10. fusion of the capabilities of EC optimization and
1145/3027063.3027072. human evaluation, Proceedings of the IEEE 89
[30] A. Liapis, G. N. Yannakakis, Boosting Computa- (2001) 1275–1296. URL: http://ieeexplore.ieee.org/
tional Creativity with Human Interaction in Mixed- document/949485/. doi:10.1109/5.949485.
Initiative Co-Creation Tasks, in: Proceedings of the [41] P. Bontrager, W. Lin, J. Togelius, S. Risi, Deep
InterICCC workshop on Computational Creativity and active Evolution, in: International Conference on
Games, Paris, 2016. Computational Intelligence in Music, Sound, Art
[31] A. Elgammal, B. Liu, M. Elhoseiny, M. Mazzone, and Design : EvoMUSART 2018, 2018.</p>
      <p>CAN: Creative Adversarial Networks, Generating [42] G. Yildirim, C. Seward, U. Bergmann, Disentangling
”Art” by Learning About Styles and Deviating from Multiple Conditional Inputs in GANs, in: KDD 2018
Style Norms (2017). URL: http://arxiv.org/abs/1706. Conference AI for Fashion Workshop, 2018. URL:
07068. http://arxiv.org/abs/1806.07819, arXiv: 1806.07819.
[32] N. Kato, H. Osone, K. Oomori, C. W. Ooi, Y. Ochiai, [43] C. Xin, K. Arakawa, Object Design System
GANs-based Clothes Design: Pattern Maker Is All by Interactive Evolutionary Computation Using
You Need to Design Clothing, in: Proceedings of the GAN with Contour Images, in: Human
Cen10th Augmented Human International Conference tred Intelligent Systems - Proceedings of
KES2019, ACM, Reims France, 2019, pp. 1–7. URL: https: HCIS 2021 Conference, volume 244, Springer
Singa//dl.acm.org/doi/10.1145/3311823.3311863. doi:10. pore, Singapore, 2021, pp. 66–75. URL: https://link.
1145/3311823.3311863. springer.com/10.1007/978-981-16-3264-8_7. doi:10.
[33] M. Gajdacz, J. Rafner, S. Langsford, A. Hjorth, 1007/978- 981- 16- 3264- 8_7, series Title: Smart
C. Bergenholtz, M. M. Biskjaer, L. Noy, S. Risi, Innovation, Systems and Technologies.
J. Sherson, CREA.blender: a GAN based casual [44] Z. Zhao, X. Ma, A Compensation Method of
Twocreator for creativity assessment, in: Proceedings Stage Image Generation for Human-AI
Collaboof the International Conference on Computational rated In-Situ Fashion Design in Augmented
ReCreativity (ICCC ’21), 2021, p. 5. ality Environment, in: 2018 IEEE International
[34] E. Denton, B. Hutchinson, M. Mitchell, T. Gebru, Conference on Artificial Intelligence and Virtual
A. Zaldivar, Image Counterfactual Sensitivity Anal- Reality (AIVR), IEEE, Taichung, Taiwan, 2018, pp.
ysis for Detecting Unintended Bias (2019). URL: 76–83. URL: https://ieeexplore.ieee.org/document/
http://arxiv.org/abs/1906.06439. 8613637/. doi:10.1109/AIVR.2018.00018.
[35] Y. Shen, J. Gu, X. Tang, B. Zhou, Interpreting the [45] D. Bau, S. Liu, T. Wang, J.-Y. Zhu, A. Torralba,
Latent Space of GANs for Semantic Face Editing, Rewriting a Deep Generative Model, in:
Proceedin: 2020 IEEE/CVF Conference on Computer Vi- ings of the European Conference on Computer
Vision and Pattern Recognition (CVPR), IEEE, Seat- sion (ECCV), 2020.
tle, WA, USA, 2020, pp. 9240–9249. URL: https:</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>R. T.</given-names>
            <surname>Hughes</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Zhu</surname>
          </string-name>
          , T. Bednarz,
          <article-title>Generative Adversarial Networks-Enabled Human-Artificial Intelligence Collaborative Applications for Creative and Design Industries: A Systematic Review of Current Approaches</article-title>
          and Trends,
          <source>Frontiers in Artificial Intelligence</source>
          <volume>4</volume>
          (
          <year>2021</year>
          ). doi:
          <volume>10</volume>
          .3389/frai.
          <year>2021</year>
          .
          <volume>604234</volume>
          , publisher: Frontiers Media
          <string-name>
            <surname>SA</surname>
          </string-name>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>T.</given-names>
            <surname>Park</surname>
          </string-name>
          , M.-Y. Liu, T.-
          <string-name>
            <surname>C. Wang</surname>
          </string-name>
          , J.-Y. Zhu,
          <article-title>Semantic Image Synthesis with Spatially-Adaptive Normalization</article-title>
          ,
          <source>in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)</source>
          ,
          <year>2019</year>
          , pp.
          <fpage>2337</fpage>
          -
          <lpage>2346</lpage>
          . URL: https://openaccess.thecvf.com/content_CVPR_ 2019/html/Park_Semantic_Image_
          <article-title>Synthesis_ With_Spatially-Adaptive_Normalization_CVPR_ 2019_paper</article-title>
          .html.
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>A.</given-names>
            <surname>Noyman</surname>
          </string-name>
          , K. Larson,
          <article-title>DeepScope: HCI Platform for Generative Cityscape Visualization, in: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems</article-title>
          , CHI EA '
          <volume>20</volume>
          ,
          <string-name>
            <surname>Association</surname>
          </string-name>
          for Computing Machinery, New York, NY, USA,
          <year>2020</year>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>9</lpage>
          . URL: https://doi.org/10.1145/ 3334480.3382809. doi:
          <volume>10</volume>
          .1145/3334480.3382809.
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>J.</given-names>
            <surname>Rafner</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Langsford</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Hjorth</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Gajdacz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Philipsen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Risi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Simon</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Sherson</surname>
          </string-name>
          , Utopian or Dystopian?
          <article-title>: using a ML-assisted image generation game to empower the general public to envision the future</article-title>
          ,
          <source>in: Creativity and Cognition</source>
          ,
          <string-name>
            <surname>ACM</surname>
          </string-name>
          , Virtual Event Italy,
          <year>2021</year>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>5</lpage>
          . URL: https: //dl.acm.org/doi/10.1145/3450741.3466815. doi:
          <volume>10</volume>
          . 1145/3450741.3466815.
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>I. J.</given-names>
            <surname>Goodfellow</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Pouget-Abadie</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Mirza</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Xu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Warde-Farley</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Ozair</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Courville</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Bengio</surname>
          </string-name>
          , Generative Adversarial Nets,
          <source>in: NIPS'14: Proceedings of the 27th International Conference on Neural Information Processing Systems</source>
          , volume
          <volume>2</volume>
          ,
          <year>2014</year>
          , pp.
          <fpage>2672</fpage>
          -
          <lpage>2680</lpage>
          . URL: http://www.github.com/goodfeli/ adversarial.
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>A.</given-names>
            <surname>Radford</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Metz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Chintala</surname>
          </string-name>
          ,
          <article-title>Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks (</article-title>
          <year>2015</year>
          ). URL: http: //arxiv.org/abs/1511.06434.
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>T.</given-names>
            <surname>Karras</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Laine</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Aila</surname>
          </string-name>
          ,
          <article-title>A Style-Based Generator Architecture for Generative Adversarial Networks</article-title>
          ,
          <source>in: Proceedings of the IEEE/CVF conference on</source>
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>