Motivated by the mixed initiative generative AI interfaces (MIGAI), we propose bridging the gap between StyleGAN3 and human-AI co-creative patterns by augmenting the latent variable model with the ability of image-conditional generation. We modify the existing generator architecture in StyleGAN3, enabling it to use high-level visual ideas to guide the human-AI co-creation. The resulting model, StyleGAN-Canvas, can solve various image-to-image translation tasks while maintaining the internal behaviour of StyleGAN3. We deploy our models to a real-time graphic interface and conduct qualitative human opinion studies. We use the MIGAI framework to frame our findings and present a preliminary evaluation of our models' usability in a generic co-creative context.
erative neural networks, StyleGAN models have been
found to be widely used as creativity support tools,
creating unconventional visual aesthetics [
Muller et al. [13] adapt notations introduced by Spoto
FCiagnuvraes 1m: oAdeplrotrtaontyslpaetiinngteprafapceer ecnarcdaplasuyolauttinigntao SfatycleesG.TAhNe- and Oleynik [14], presenting mixed initiative generative
user adjusts layouts while the model provides synchronous AI interfaces (MIGAI), an analytical framework with 11
generation based on visual similarity. A screen recording is vocabularies of actions to describe a human-AI
interacavailable at: https://youtu.be/9AsfsT8uXGY tion process. These actions are analysed into sequences
to form generic human-AI co-creative patterns.
However, Grabe et al. [15] identify a gap between the MIGAI
1. Introduction framework and latent variable models such as GANs.
This is partially due to latent variable models’ deficiency
Generative adversarial networks (GANs) [
Therefore, the primary aim of this research is to aug- eventually produces the output image = (0; ).
ment StyleGAN3 with image-conditional generation abil- Its high-level style control is achieved by adaptive
inity for a co-creative context. To achieve this, we adapt the stance normalisation (AdaIN) [21], an approach to
ampliexisting model architecture in StyleGAN3, which takes fying specific channels of feature maps on a per-sample
a latent vector and a class label as the model’s input [
To evaluate our model in a generic co-creative context, make the network equivariant, which means any operawe build a graphic interface to facilitate the real-time tion in the network is equivariant to a spatial transforinteraction between users and the model. We conduct mation ( ∘ = ∘ ). This eliminates positional referqualitative human opinion studies, identify potential co- ences; therefore, the model can be trained on unaligned creative patterns using the MIGAI analytical framework, data, and the "texture sticking" behaviour in standard and present an exploratory human subject study on our GAN models is removed. model. By aligning StyleGAN-Canvas with the actions Moreover, the input of the synthesis network of Styleset in MIGAI, we hope to bring its capability into the GAN3 uses a spatial map 0 defined by Fourier features discussion of co-creative design processes, and provide [24] to precisely model the translation and rotation. The a preliminary insight into the unexplored interaction spatial map 0 is sampled from uniformly distributed frepossibilities enabled by StyleGAN. quencies, fixed after initialisation, and spatially infinite
The rest of the paper is structured as follows. We sum- [
Next, we evaluate the model in a co-creative context in 2.2. Image-Conditional Generation Section 5. Section 6 highlights limitations and future studies.
(Alias-Free GAN). This section summarises the background of StyleGAN and reviews the continuous convolution approach introduced by StyleGAN3 that enables the translation and rotation equivariant feature.
StyleGAN [20] is a style-based generator with a regularised latent space ofering high-level style control over image generation. The StyleGAN generator comprises a mapping network that transforms the initial latent code to intermediate latent code ∼ , and a synthesis network with a sequence of synthesis blocks, Methods for image-conditional generation aim to generate a corresponding image given an image from the source domain, depicting objects or scenes in diferent styles or conditions. Current solutions to this are usually categorised under two approaches. The first approach uses a linear encoder-decoder architecture [26], in which the input image is encoded to a vector to match the target domain, and then decoded into an output image. This method was later extended to a generic framework for various tasks, such as sketches and layout to image, facial frontalisation, inpainting, and super-resolution [27].
The second direction uses conditional GANs [17] with U-net architectures [28]. It uses a similar encoderdecoder setting, but instead of encoding the input image into a vector, it uses the residual feature maps from the encoder as spatial information and propagates them to the decoder through skip connections [29]. The propagated features are concatenated with the outputs from
use images as conditions to guide the generation. This section will discuss our modification to the current StyleGAN3 architecture to address this objective.
As mentioned in Section 2.1, current StyleGAN3 models learn a mapping from a random noise vector to an output image = (). The modification aims to extend the input from a vector to a conditional image combined with , and generate that is close to the corresponding ground truth . To do this, we first need a feature extraction encoder that extracts features from , then adapts the generator to produce outputs based on the extracted features and the input vector , i.e. = ((), ). Besides, the training objective should push the generation closer to the corresponding image . corresponding decoder layers. This method aims to provide the generator with a mechanism to circumvent the bottleneck layer and allow spatial information to be shuttled from the encoder to the decoder [28]. Therefore, it introduces a strong locality bias [26] into the generation, which means each pixel in the output has a positional reference to the input, and the general image structure is preserved during translation. This method has been extended for various tasks such as line-conditioned generation [19], layout-to-image synthesis [30], and semantic region-adaptive synthesis [31].
describe modes of interaction in which both human and generative AI is engaged in a creative process. It aims to frame subprocesses in a creative flow by an action set with 11 vocabularies. A generic co-creative pattern is identified using the MIGAI framework, in which the AI system first learns a target domain, then the human ideates a design concept to guide the artefact; subse- 3.1. Feature Extraction Encoder quently, the human and AI system take turns to eval- 3.1.1. Adapted Residual Network uate and adjust, eventually produce the outcome [15].
Our study focuses on ideating, a process of conceptual- The feature extraction encoder employs an adapted ising design solutions to high-level abstractions. Later ResNet [29] architecture as the encoder backbone, which exploratory study emphasising the potential of enhanc- has been previously used for feature maps extraction in ing the creative process through human-AI collaborative image-to-image translation works [27]. As shown in Figideation [16]. Methods for this have been advanced in di- ure 2 (left), the encoder network downsamples feature verse fields of application: co-creative game content [ 32], maps to (/26, /26), where and denote the width collaborative text editing [33], and text-guided image and height of the input image. In addition, as StyleGAN generation [34]. uses mixed-precision to speed up training and inferences
Previous works in StyleGAN encoder [27] have highlighted that replacing select layers of the extended latent space + by computed latent codes can facilitate multimodal synthesis. And the extended latent space + can be roughly divided into coarse, medium, and fine layers, corresponding to diferent levels of detail and editability [36]. This motivates us to add a conditional mapping network as the epilogue layer of our residual feature encoder, which uses the same mapping network architecture in StyleGAN, but takes a flattened 512 vector sampled from the encoder’s bottleneck and produces a replacement latent space + ′. And + ′ is then concatenated with a portion of the latent space + produced from the original mapping network. This aims to facilitate multi-modal generation. [24], we utilise similar techniques and reduce the pre- U-Net and its variants has demonstrated that a simplified cision to FP16 for the rfist five residual blocks in the structure with fewer feature fusions can achieve reasonencoder. Consequently, it requires pre-normalisation and able results [37, 38]. Therefore, we reduce the number of an extra clamping layer that clamps the output of the feature fusions and limit connections to only the first convolutional layers to ± 29 [35]. layers. In practice, skip connections in these five models each connect layer − of the encoder to layer , where 3.1.2. Conditional Mapping Network is the total number of layers in the encoder, and is limited to ∈ (0, 5]. The experiment in Section ?? will show that = 5 is the best configuration that leads to more stable training and eficient generation. This reduction in the network maintains unification of the network’s internal behaviour, while taking advantage of the eficiency of U-shaped structural models.
GAN loss function (i.e., logistic loss) and regularisation terms (i.e., 1). We incorporate the training objectives in StyleGAN with pixel-wise distance and perceptual loss that have been used in conditional GANs.
The model is trained using diferent combinations of
objectives at two diferent training phases. The first phase
starts from zero to the first 300k images, and this is also
3.2. Adapting Generator the phase where the training images are blurred with
a Gaussian filter to prevent early collapses [
We start by modifying the input layer in its synthesis network. The original network utilises a fixed-size spatial map 0 defined by Fourier features [ 24] as its input to (, , ) = E[()] (2) model translation and rotation parameters. However, the +E,[(1 − (((), )))] entire input layer is left out, and we use feature maps from the last layer of the encoder directly as the synthesis network’s input. This lets the translation and rotation (, ) = E,[‖()− ¯‖2 +‖ ()− ¯‖2] (3) parameters be inherited from the spatial feature maps.
Next, connections between the feature encoder aim where and denote the generator and the
discriminato provide precise structural information about input tor, denotes the feature encoder, and denotes the
images. A U-net [28] architecture is well-suited for prop- mapping network in the generator. Then, the training
agating high-level details from the encoder to the decoder loss ℎ1 is defined as:
[26]. However, as mentioned in Section 2.1, experiments ℎ1(, , ) = 12(, )
on StyleGAN3 have demonstrated that high-level feature (4)
maps in the synthesis network encode information in + (, , ) + (, )
continuous domains instead of discrete domains [
Then, the phase 2 loss is then calculated as follows: ℎ2(, , ) = 12(, ) + 2 (, ) + (, , ) + (, )
(6) where denotes the pre-trained VGG19 feature extractor. 1 and 2 are constant numbers used to weigh the loss parameters, which vary across diferent training data and configurations.
Results. Figure 3 (left) shows the results of the ablation test for the = 5 to = 2 models trained on 1680k samples. The rest of the two models ( = 1 and = 0) are unable to converge to sensible results after 800k samples and were therefore aborted. The results show notable improvements when increasing the number of skip connections, especially in preserving details such as background, hands, unique make-up and even the details in hairs. Therefore, = 5 is decided as the final setting.
In the following Section 4.1, we analyse the efectiveness of the U-net proposed in Section 3.2 by a set of ablation studies. Next, in Section 4.2, we demonstrate the training process of our model for several image-to-image translation tasks with diferent datasets and showcase their results. We also experiment with scaling the model to larger canvases in Section 4.3. Finally, we build a graphic interface that implements our models with a set of transformation filters in Section 4.4.
In addition, the canny edges model provides an alternative approach to local editing. Modifying edges in the condition image allows the model to alter semantical ele- absolute positional references [44], we hypothesised that ments in the generation. Illustrated in Figure 4 (bottom), our modified architecture induced an extendable generathe modified conditions can be obtained by combining tion canvas that can be enlarged after trained on a fixed and adapting existing conditions from other images. For resolution, without additional training required. Thereexample, in Figure 8, we superimposed edges processed fore, we enlarge the input resolution to test the model’s from other images to the original edges to add hair fringe, ability on a larger canvas. glasses and smile; we painted on the original edges to The model for landscape photo generation was trained modify eyes and add sunglasses. on the dataset with 512 × 512 resolution, taking inputs with 256× 256 resolution. To enlarge the generation can4.3. Large Canvas vas, we doubled and tripled the width of inputs, expanding their resolution to 1024 × 256 and 768 × 256. Then, As mentioned in Section 3.2, the padding layers in the the expanded inputs were taken directly into the genersynthesis network are removed, and the entire generator ator and convolved by each convolutional layer. Theredoes not have unintentional positional references with fore, the expected output resolutions are 2048 × 512 and 1536 × 512. Additional training is not required during from 10 to 32 ( ⃗ ∈ 32) to confront larger numbers of the experiment. channels in StyleGAN3-R.
Figure 9 shows the resulting generations. While the outputs are expanded up to four times larger than Interface design. Figure 10 shows a screenshot of the the original canvas, the generation quality remains un- deployed system. The interface allows inputs from a webchanged. This ensures that the input canvas can be flex- cam or locally selected image files (top left). The interface ibly expanded when the models are implemented into allows users to pause, resume generation, and switch the user interfaces. input between the webcam and local files. Users can insert transformation filters into certain groups in certain 4.4. User Interface layers. The list on the bottom left side presents layers available for network bending operations. Once a specific Implementation. The deblurring models were de- layer is selected, the system shows current clusters of ployed to a graphic user interface. The models were feature maps in the layer. Users can regenerate clusters running on a cloud server. We used Flask and Socke- according to feature maps from the current frame. Then, tIO for bi-directional communications between the web users activate the ‘route’ button to apply transformation client and the server. The generation runs in real-time at filters to the cluster. The system provides basic transforroughly ten frames per second on an NVIDIA RTX A4000 mation filters, including erosion and dilation, multiply, GPU. The code for our implementation is available at translations, rotation, and scale. https://github.com/jasper-zheng/realtime-flask-model.
Combining network bending. In addition to the baseline model, the generation system is implemented with network bending [45], an approach to alter a trained model’s computational graph by inserting transformation iflters between diferent convolutional layers, allowing the model to generate novel samples that are diverse from the training data [46]. We used the clustering algorithm presented by Broad et al. [45] to group spatially similar feature maps in selected layers and allow the transformation filters to be inserted into specific groups. We trained softmax feature extraction CNNs for each layer and clustered each flatten layer by the k-means algorithm.
However, diferent from the implementation in Broad et al.’s work, we increase the length of the flatten vector
our models in a human-AI co-creation context. The study uses a thematic analysis approach to identify potential co-creative patterns defined by the MIGAI framework [13] that underlies the interaction.
In the user study, we asked six participants to use the Figure 11: Examples of works created by participants generation interface described in Section 4.4. For the webcam inputs, coloured paper cards and scissors were available to the participants. Participants were asked to arrange and layout paper cards in front of a webcam We used the MIGAI analytic framework [13] to frame the pointing to a paper canvas, aiming to control the gen- human-AI co-creative patterns. eration until it reached something they liked. Besides, the transformation filters were also available to fine-tune 5.2.1. Learn the creation further. Models used in Section 4.2 are available to the participants, including the deblurring model The action learn in the MIGAI framework describes the and the edge-to-face model trained on Flickr-Faces-HQ AI model using training data to construct its knowledge, (FFHQ) [20], the deblurring model trained on Landscapes usually involving the choice of datasets, model archiHigh-Quality (LHQ) [43]. tectures, and training configurations. Most participants
The experiment followed the qualitative research pro- used the model trained on the landscape dataset in the cedure described by Adams et al. Adams et al. [47]. Six end. When questioned on the reason for this decision, participants were divided into two groups of three. We they suggested the surprising, unexpected results proconducted the study on the first group and ran an analy- duced by the landscape model are more likely to be acsis to emphasise issues raised by the participants based cepted by their visual aesthetic. Although both models on their frequency and fundamentality, leading to tenta- can create unrealistic imagery, distortion and oddities on tive findings. Then, the interview questions are revised human faces may easily lead to uncanny feelings and, for the second group to probe and grow these findings. more importantly, negative ethical issues such as bias The study was divided into two parts, each lasting 15 and ofences. Therefore, the choice of training data is minutes. The first part asked participants to familiarise critical for co-creation. the interface and explore the system components. The second part asked the participant to create a work they 5.2.2. Ideate liked. Observation is conducted while participants interact with the framework, and a 5-minute semi-structured interview with open-ended questions follows each part of the experiment.
Participants were questioned on their attitudes regarding this form of interaction, the creation process, their generation outcomes, and the diferences in their perceptions of diferent models. We loosely followed the interview template during the interviews while ensuring these four topics were covered.
Figure 11 shows some examples of works created by the participants.
The action ideate describes the process of using highlevel concepts to guide the generation. We collected comments aggregated around this process. Some positive comments indicate that creating shapes using paper cards is an intuitive generation process. Whereas some negative comments suggest that the lack of controls in details (e.g., textures in the landscapes, details of the facial elements) leads to confusion and complaint. Most participants pointed out that they would use this form of generation as inspiration, giving them a high-level sketch developed from their ideated concept. Alternatively, they may treat it as a playful experience or simply in pursuit of an abstract visual efect. However, if the generation aims to create a serious piece of work, they would eventually switch to more stable methods with lower-level control to refine the work, such as Photoshop or CAD software.
This is because, in this case, they want to be manually in charge of finer details such as lighting and textures.
5.2.3. Adapt 5.2.4. IterativeLoop The action adapt describes adjusting existing artefacts. The MIGAI framework uses IterativeLoop to describe huDuring the study, we observed that participants primar- mans reflectively learning from the co-creative process. ily focused on exploring the outcome from the compu- We observed that some participants might memorise usetational agent by arbitrarily arranging the paper cards. ful patterns of shapes and colours that may trigger satisWhen they reach a layout that triggers interesting results, factory results, and later use the paper card to reproduce they iteratively adjust the arrangement until achieving these patterns. Some participants describe this process a satisfying result. Some participants tried to tackle the as a way to learn the preferences of the AI model to steer lack of control by utilising other pre-processing or post- the co-creation through reflection. processing processes. Figure 12 illustrates an example of an action that attempts to fix the oddity by slightly 5.3. Discussion warping the intermediate image in other image editing software. Figure 13 illustrates a sequence of editing per- The motivation of this paper was to augment style-based formed on intermediate condition images to intentionally GAN models into a co-creative context. By extending the create unrealistic and novel outcomes. The edges were StyleGAN model to image-conditional generation, these edited and assembled before the generation using Photo- models allow human agents to ideate a high-level conshop, and then uploaded to the interface as inputs. The cept of the artefacts, like blurred arrangement or edges. user evaluated the current outcome, and then iteratively The flexibility of input and the semantically meaningful ifne-tuned the edges according to their preferences. control are critical features for efective co-creative experiences. Meanwhile, to create a sense of a "cooperated partner", the computational agent needs to maintain a certain level of unpredictability, and AI’s contribution needs to partly influence the human agents’ decision [ 49].
Therefore, the co-creative process balances machine autonomy and human creativity. Our current results are good indications that human agents act as a director who steers the model by organising and reusing learned knowledge in the computational agent. While further studies on the model architecture may improve the generation quality, the current results in this research show that bridging StyleGAN3 to the co-creative context is possible. Furthermore, it could be employed for novel and unique co-creative experiences. For example, Figure 14 shows an interactive installation with webcam inputs that produce 704 × 1280 images in real-time.
of image-conditional generation, enabling it to turn highlevel visual ideas into generation. This augmentation aligned latent variable models with the co-creative patterns mentioned in the MIGAI framework and brought StyleGAN3 into a co-creative context. We adapted the existing model architecture and proposed an encoder network to extract information from the conditional image. We demonstrated the modified architecture, StyleGANCanvas, on various image-to-image translation tasks with diferent datasets. In addition, we deployed our models to a graphic interface to facilitate the real-time interaction between users and the model. To evaluate our models in a co-creative context, we conducted qualitative human opinion studies and identified potential co-creative patterns using the MIGAI analytic framework.
An overall criticism was the need for more interpretable control in the system. While the input image acts as a blueprint for the generation, the user also needs precise control over the details when using the model as a creative tool. This leads us to rethink the design of the intermediate representation. Our framework currently only implements deblurring models for the experiment, however, it might be more useful to use intermediate representations that encode more detailed information (e.g., boundary maps or edges) like the interactive demos in pix2pixHD.
Besides, the proposed architecture can also be improved in technical aspects. Our approach proposes an alternative for extending StyleGAN models to imageconditional generation. Although it has demonstrated its potential in solving several image-to-image translation tasks, the detailed architecture still needs further investigation and refinement to improve the generation quality. Our model architecture utilises the equivariant generator in StyleGAN3. However, our feature extractor still needs to be rotation equivariant. Therefore, the generation may sufer when the rotation is not encoder. Figure 15 show an example of failure where the encoder does not preserve the rotation. It would be beneficial to make the feature encoder equivariant in future work.
by GAN [50] were considered throughout the project. The models trained on the FFHQ dataset are for purely academic purposes, and its interactive prototype will not be publicly distributed by any means. Model trainings used approximately 300 hours of computation on A100 SXM4 80GB (TDP of 400W). Total emissions are estimated to be 25.92kg 2, as calculated by MachineLearning Impact calculator [51]. Paper cards in the experiments were limitedly allocated to participants, reused during and after the experiments.
[8] T. Park, M.-Y. Liu, T.-C. Wang, J.-Y. Zhu, Seman- 48550/ARXIV.1711.11585. tic image synthesis with spatially-adaptive normal- [19] Y. Li, X. Chen, F. Wu, Z.-J. Zha, Linestofacephoto: ization, in: Proceedings of the IEEE/CVF Con- Face photo generation from lines with condiference on Computer Vision and Pattern Recogni- tional self-attention generative adversarial network, tion (CVPR), 2019, pp. 2337–2346. doi:10.48550/ 2019. URL: https://arxiv.org/abs/1910.08914. doi:10.
ARXIV.1903.07291. 48550/ARXIV.1910.08914. [9] J. Rafner, S. Langsford, A. Hjorth, M. Gajdacz, [20] T. Karras, S. Laine, T. Aila, A style-based generator L. Philipsen, S. Risi, J. Simon, J. Sherson, Utopian or architecture for generative adversarial networks, dystopian?: Using a ml-assisted image generation 2018. URL: https://arxiv.org/abs/1812.04948. doi:10. game to empower the general public to envision the 48550/ARXIV.1812.04948. future, in: Creativity and Cognition, Association for [21] X. Huang, S. Belongie, Arbitrary style transfer in Computing Machinery, New York, NY, USA, 2021, real-time with adaptive instance normalization, in: p. 5. URL: https://doi.org/10.1145/3450741.3466815. ICCV, 2017.
doi:10.1145/3450741.3466815. [22] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehti[10] Y. Wang, W. Zhou, J. Bao, W. Wang, L. Li, H. Li, nen, T. Aila, Analyzing and improving the image Clip2gan: Towards bridging text with the latent quality of stylegan, 2019. URL: https://arxiv.org/abs/ space of gans, 2022. URL: https://arxiv.org/abs/2211. 1912.04958. doi:10.48550/ARXIV.1912.04958. 15045. doi:10.48550/ARXIV.2211.15045. [23] N. Dey, A. Chen, S. Ghafurian, Group equiv[11] J. Simon, Artbreeder, 2018. URL: https://www. ariant generative adversarial networks, in: Inartbreeder.com/about. ternational Conference on Learning Representa[12] S. Shahriar, Gan computers generate arts? a tions, 2021. URL: https://openreview.net/forum?id= survey on visual arts, music, and literary text rgFNuJHHXv. generation using generative adversarial network, [24] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich2021. URL: https://arxiv.org/abs/2108.03857. doi:10. Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, 48550/ARXIV.2108.03857. J. Barron, R. Ng, Fourier features let networks [13] M. Muller, J. D. Weisz, W. Geyer, Mixed initiative learn high frequency functions in low dimensional generative ai interfaces: An analytic framework domains, in: H. Larochelle, M. Ranzato, R. Hadsell, for generative ai applications, in: Proceedings of M. Balcan, H. Lin (Eds.), Advances in Neural the Workshop The Future of Co-Creative Systems- Information Processing Systems, volume 33, A Workshop on Human-Computer Co-Creativity Curran Associates, Inc., 2020, pp. 7537–7547. URL: of the 11th International Conference on Computa- https://proceedings.neurips.cc/paper/2020/file/ tional Creativity (ICCC 2020), 2020. 55053683268957697aa39fba6f231c68-Paper.pdf. [14] A. Spoto, N. Oleynik, Library of mixed- [25] Y. Alaluf, O. Patashnik, Z. Wu, A. Zamir, E. Shechtinitiative creative interfaces, 2017. URL: man, D. Lischinski, D. Cohen-Or, Third time’s http://mici.codingconduct.cc/. the charm? image and video editing with style[15] I. Grabe, M. G. Duque, S. Risi, J. Zhu, Towards gan3, 2022. URL: https://arxiv.org/abs/2201.13433. a framework for human-ai interaction patterns in doi:10.48550/ARXIV.2201.13433. co-creative gan applications, in: Joint Proceed- [26] E. Richardson, Y. Weiss, The surprising efectiveings of the ACM IUI Workshops 2022, March 2022, ness of linear unsupervised image-to-image transHelsinki, Finland, 2022. URL: https://ceur-ws.org/ lation, 2020. URL: https://arxiv.org/abs/2007.12568.
Vol-3124/paper9.pdf. doi:10.48550/ARXIV.2007.12568. [16] J. Kim., M. L. Maher., S. Siddiqui., Collabora- [27] E. Richardson, Y. Alaluf, O. Patashnik, Y. Nitzan, tive ideation partner: Design ideation in human- Y. Azar, S. Shapiro, D. Cohen-Or, Encoding in style: ai co-creativity, in: Proceedings of the 5th Inter- a stylegan encoder for image-to-image translation, national Conference on Computer-Human Inter- 2020. URL: https://arxiv.org/abs/2008.00951. doi:10. action Research and Applications (CHIRA 2021), 48550/ARXIV.2008.00951.
INSTICC, SciTePress, 2021, pp. 123–130. doi:10. [28] O. Ronneberger, P. Fischer, T. Brox, U-net: Convo5220/0010640800003060. lutional networks for biomedical image segmenta[17] P. Isola, J.-Y. Zhu, T. Zhou, A. A. Efros, Image- tion, 2015. URL: https://arxiv.org/abs/1505.04597. to-image translation with conditional adversarial doi:10.48550/ARXIV.1505.04597. networks, CVPR (2017). [29] K. He, X. Zhang, S. Ren, J. Sun, Deep residual [18] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, learning for image recognition, 2015. URL: https: B. Catanzaro, High-resolution image synthesis //arxiv.org/abs/1512.03385. doi:10.48550/ARXIV. and semantic manipulation with conditional gans, 1512.03385. 2017. URL: https://arxiv.org/abs/1711.11585. doi:10. [30] X. Liu, G. Yin, J. Shao, X. Wang, H. Li, Learning to predict layout-to-image conditional convolutions 48550/ARXIV.1901.08644. for semantic image synthesis, 2019. URL: https: [41] J.-Y. Zhu, P. Krähenbühl, E. Shechtman, A. A. Efros, //arxiv.org/abs/1910.06809. doi:10.48550/ARXIV. Generative visual manipulation on the natural im1910.06809. age manifold, in: B. Leibe, J. Matas, N. Sebe, [31] P. Zhu, R. Abdal, Y. Qin, P. Wonka, Sean: Image M. Welling (Eds.), Computer Vision – ECCV 2016, synthesis with semantic region-adaptive normaliza- Springer International Publishing, Cham, 2016, pp. tion, in: IEEE/CVF Conference on Computer Vision 597–613.
and Pattern Recognition (CVPR), 2020. [42] J. Canny, A computational approach to edge de[32] A. Liapis, G. Smith, N. Shaker, Mixed-initiative tection, IEEE Transactions on Pattern Analysis content creation, Springer International Publish- and Machine Intelligence PAMI-8 (1986) 679–698. ing, Cham, 2016, pp. 195–214. URL: https://doi. doi:10.1109/TPAMI.1986.4767851. org/10.1007/978-3-319-42716-4_11. doi:10.1007/ [43] I. Skorokhodov, G. Sotnikov, M. Elhoseiny, Align978-3-319-42716-4_11. ing latent and image spaces to connect the uncon[33] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. De- nectable, arXiv preprint arXiv:2104.06954 (2021). langue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Fun- [44] R. Xu, X. Wang, K. Chen, B. Zhou, C. C. Loy, Potowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, sitional encoding as spatial inductive bias in gans, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gug- in: arxiv, 2020. ger, M. Drame, Q. Lhoest, A. Rush, Transform- [45] T. Broad, F. F. Leymarie, M. Grierson, Network benders: State-of-the-art natural language process- ing: Expressive manipulation of deep generative ing, in: Proceedings of the 2020 Conference on models, 2020. URL: https://arxiv.org/abs/2005.12420. Empirical Methods in Natural Language Process- doi:10.48550/ARXIV.2005.12420. ing: System Demonstrations, Association for Com- [46] T. Broad, S. Berns, S. Colton, M. Grierson, Active diputational Linguistics, Online, 2020, pp. 38–45. vergence with generative deep learning – a survey URL: https://aclanthology.org/2020.emnlp-demos.6. and taxonomy, 2021. arXiv:arXiv:2107.05599. doi:10.18653/v1/2020.emnlp-demos.6. [47] A. Adams, P. Lunt, P. Cairns, A qualititative ap[34] R. Gal, O. Patashnik, H. Maron, G. Chechik, proach to hci research, in: P. Cairns, A. Cox D. Cohen-Or, Stylegan-nada: Clip-guided do- (Eds.), Research Methods for Human-Computer Inmain adaptation of image generators, 2021. teraction, Cambridge University Press, Cambridge, arXiv:2108.00946. UK, 2008, pp. 138–157. URL: http://oro.open.ac.uk/ [35] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehti- 11911/.
nen, T. Aila, Training generative adversarial net- [48] V. Braun, V. Clarke, Thematic analysis., APA handworks with limited data, in: Proc. NeurIPS, 2020. book of research methods in psychology, Vol 2: Re[36] X. Mao, L. Cao, A. T. Gnanha, Z. Yang, Q. Li, R. Ji, search designs: Quantitative, qualitative, neuropsyCycle encoding of a stylegan encoder for improved chological, and biological. (2012) 57–71. doi:10. reconstruction and editability, 2022. URL: https: 1037/13620-004. //arxiv.org/abs/2207.09367. doi:10.48550/ARXIV. [49] M. T. Llano, M. d’Inverno, M. Yee-King, J. Mc2207.09367. Cormack, A. Ilsar, A. Pease, S. Colton, Explain[37] H. Huang, L. Lin, R. Tong, H. Hu, Q. Zhang, able computational creativity (2022). URL: https: Y. Iwamoto, X. Han, Y.-W. Chen, J. Wu, Unet 3+: A //arxiv.org/abs/2205.05682. doi:10.48550/ARXIV. full-scale connected unet for medical image segmen- 2205.05682. tation, 2020. URL: https://arxiv.org/abs/2004.08790. [50] V. U. Prabhu, D. A. Yap, A. Wang, J. Whaley, Covdoi:10.48550/ARXIV.2004.08790. ering up bias in celeba-like datasets with markov [38] H. Lu, Y. She, J. Tie, S. Xu, Half-unet: A blankets: A post-hoc cure for attribute prior avoidsimplified u-net architecture for medical im- ance, 2019. URL: https://arxiv.org/abs/1907.12917. age segmentation, Frontiers in Neuroinformat- doi:10.48550/ARXIV.1907.12917. ics 16 (2022). URL: https://www.frontiersin.org/ [51] A. Lacoste, A. Luccioni, V. Schmidt, T. Dandres, articles/10.3389/fninf.2022.911679. doi:10.3389/ Quantifying the carbon emissions of machine learnfninf.2022.911679. ing, 2019. URL: https://arxiv.org/abs/1910.09700. [39] K. Simonyan, A. Zisserman, Very deep convolu- doi:10.48550/ARXIV.1910.09700. tional networks for large-scale image recognition, 2014. URL: https://arxiv.org/abs/1409.1556. doi:10.
48550/ARXIV.1409.1556. [40] R. Meyes, M. Lu, C. W. de Puiseau, T. Meisen,
Ablation studies in artificial neural networks, 2019. URL: https://arxiv.org/abs/1901.08644. doi:10.