Generative models such as generative adversarial networks are now being studied extensively. Eventually, however, many of them are intended for non-programmers to work with, e.g. designers, artists, or other content creators. What happens when such individuals are confronted with using GANs? We present a case study - a new course intended for non-programmer MA students in human-computer interaction, aimed at training them in authoring content using generative models. As their final assignment, the students were asked to train generative adversarial networks in order to generate images from a predefined category of their choice. The students either used a graphical user interface (GUI)-based software or modified preexisting python code using simplified Google Colab notebooks. We present several lessons learned from this course. First, we analyze the joint human-AI creation process and recognize points where students could intervene, with anecdotal examples of how they creatively explored these opportunities. Interestingly, while the majority of algorithmic research is focused on how to make models more controllable (e.g., via conditioning or latent space disentanglement), the students found ways to obtain their creative needs by mostly exploring the dataset level (as opposed to the model architecture). Additionally, we present the results of a short survey, comparing the two modes of work (GUI vs code).
We are witnessing a rapid advance of “artificial
intelligence” (AI) and machine learning (ML)
technologies, and these techniques are penetrating into an
increasing range and diversity of aspects of
everyday life. It seems important that the responsibility
for these systems would not only lie on the
shoulders of programmers, but that additional professions
would be involved in intelligent system design,
development, and evaluation. As the current zeitgeist is
that of data-driven methods with “deep” neural
networks, explainability has become a major concern [
In generative AI, rather than just explainbility (or
instead?), we often strive for ‘controllability’, or the
degree to which humans can control and shape the
results generated by the system. Indeed, there is
ample work on on reversible generative models [
Our goal is to explore how such individuals can work with novel generative models such as generative adversarial networks (GANs). The opportunity came up in the form of a course for MA students in human computer interaction (HCI), most of them non programmers. As their final assignment in the course they were guided in developing a project in which they train a GAN to generate a specific set of images, in the spirit of “This X does not exist”1. Following the success of StyleGAN to generate highly photo-realistic human faces, a website showing such images called “thispersondoesnotexist” went viral. This was followed by attempts in training GANs specializing in generating cats, rental apartments, snack, and the list of project seems to be still growing. The task is similar to the Little Prince’s request: “draw me a sheep”. Unlike the majority of human-AI co-creation tasks, in which the human is expected to be creative and the machine is expected to assist, in this task the machine is expected to be creative and generate interesting samples of X, and the human is only expected to assist. The students were given the choice whether to use programming or a GUI-based software, and following the course, they were asked to answer some questions.
This paper’s contribution is from lessons learned
from teaching generative models and synthetic media
to non-programmers, anecdotal lessons learned from
their projects, an analysis of the human intervention
points that they ‘discovered’, and results from a
sur1http://thisxdoesnotexist.com
vey in which they provided some feedback2. Interest- that there are also important diferences. For example,
ingly, while the overwhelming majority of algorithmic in those artist groups the importance of the
individuresearch is focused on how to make models more con- als comprising the group was much reduced; however,
trollable (e.g., via conditioning or latent space disen- in our case we are still interested in distinguishing the
tanglement), the students found ways to obtain their role and contribution of the machine and the human.
creative needs by mostly exploring the dataset level (as Bidgoli et al. actually suggest that the machines (at
opposed to the model architecture). least in this point in time) do not have an identity
to contribute; rather, it is the identity of the people
who designed it. “From this point of view, co-creation
2. Background is a special case of collaboration where the tool acts
as a “machinic surrogate” to represent the identity of
The “draw me a sheep” scenario raises several ques- its “toolmakers”.” We suggest that this, again,
undertions. The first is: can machines be creative and if so mines the important diferences between humans and
what does it mean? Generative algorithms are able machines.
to generate a huge number of outcomes; however, it Interestingly, the current legal perspective seems to
could be that most or even all of the results are uninter- agree that the results of human-AI collaboration may
esting. Boden [
Thus, we suggest that machine creativity is not the a larger role in an increasing number of domains, this
most appropriate framework for our case study, but gap will need to be bridged, and ideally it should be
rather the appropriate framework is human-AI collab- addressed by the research community, not only by
inoration. Here we can distinguish between two ap- dustry teams working on specific products. In the
doproaches. The first approach is more human-centric, main of music generation, Huang et al. [
This leads us to the second, more recent approach, very diferent from other research projects in the field
where there is growing interest in human-AI collabo- of human-AI co-creation, and we suggest that the
asration as a new entity, where the whole is larger than sumptions and range of questions that addressed by
the sum of the parts. Bidgoli et al. [
For example, cultural diversity was highly encouraged For example, Karimi et al. [
Co-creativity has also been framed as mixed
initiative interaction [
Instead, we suggest that an appropriate framework provided a brief introduction to ‘’classic ML’ and deep is to view the human-AI collaboration as a single pro- neural networks, and the second half of the course was cess. The analysis we perform (in Section Results) is dedicated to a more in depth discussion of GANs and aimed at identifying the sub tasks in this creative pro- some additional topics (“deep fake”, sequences, lancess, and who is in charge of each sub task. In our case guage models). We have introduced some mathematthe machine would ideally be autonomous, but we see ical notation, for example, discussing loss functions, that there are quite a few points at which human in- but the course was intended for non-mathematicians tervention is desired or even required. and most of the discussion was at the level of popular science. Due to the Covid-19 pandemic most of the lessons were hybrid, with half of the students in class 3. Method and the rest at home over Zoom. In their assignments, the students explored both an 3.1. The Students easy-to-use software – RunwayML5, and a simple Colab notebook with python code. RunwayML is a comThe course was an elective as part of an MA degree mercial product intended for artists and creators to apon human-technology relationship. Thirty two stu- ply a wide range of “deep learning” models. It prodents enrolled in the course, most of them with a back- vides a relatively easy to use graphical user interface ground in design or social science, and only a minor- (GUI) that serves as a front end to existing implemenity with programming experience or computer science tations of deep neural networks. It allows you to run background. All students learned basic programming the trained models on your input, to train models on (with p5js3) in the first semester, and the course was new datasets, and to concatenate models; i.e., you can 3http://p5js.org concatenate networks A and B if the type of output of network A is consistent with the type of input of network B. The payment is mostly per cloud GPU runtime. Such services raise interesting and important questions around copyright and ownership of intellectual property, which are out of the scope of this paper.
As a first generative assignment the students were
asked to explore “deep dream” [
For the final assignment the “brief” was to come up sults matched their expectation in quality, and iii) the
with images in the spirit of thisXdoesnotexist6. The results matched their expectation in intent.
students were provided with a Colab python
notebook for scraping images online. Next, they were 4. Results
told to choose between two options: a notebook with
documented deep convolutional GAN (DC-GAN) [
The programmers were assigned to separate teams, captured by the model, when comparing the two platand they were not allowed to use RunwayML. They forms (t=1.0, df = 8, p=0.347). Nevertheless, the correwere told to explore a range of techniques and hyper- lation between perceived quality and captured intent parameters and provide documentation of training was high and significant for both RunwayML (r=0.76, runs, such as plots of generator and discriminator loss. df=12, p=0.003) and Colab (r=0.66, df=13, p=0.01). Most of them started with DC-GAN but moved on to StyleGAN-2 in order to improve their results. 3.3. Questionnaire A week after the submission of the final project, the students were asked to fill in a questionnaire, and 18 students responded. We asked about programming experience and what option they used for their final project. Next we asked them to rate, for either python code or RunwayML, the extent to which: i) they liked using it, ii) they found it dificult, iii) the reWe analyze the projects in terms of the points where the human users could intervene, with focus on the non-programmers (including students with some very basic programming experience). The first human intervention point was in determining the goal – allegedly, this was completely determined by the humans. However, it could be argued that the machine ‘took part’ in this stage as well, since about half of the teams modified their project goal after some attempts. Typically, after realizing that their initial expectations were not realistic, the students converged on more specific outcomes, since they realized that the datasets have to be homogeneous for obtaining good results. Eventually, the non-programmer teams decided to generate: human head statues (protomes), holocaust victims, Gaudi-style architecture, animalelectronic device hybrids, city maps, Simpson characters, politicians, butterflies, protected flowers, Disney characters, smartphone application icons, and bicycles. The programmer teams decided to generate: Marvel superheros, Yayoi Kusama art, best of art paintings (unspecified), and McDonald’s toys (mixed team).
The students quickly learned that the main way for them to afect the results is by modifying the training set. Although this was discussed in class, many were surprised that the datasets need to be very homogeneous and that “the AI” could not deal with simple invariants such as location of the main object within the frame. Some surprising artifacts were discovered; for example, the Gaudi team had to clean cranes from pictures of the Sagrada Familia, which has been under construction for a long time. Sometimes such biases were considered undesired, but in other cases the students were excited to see these biases emerge from the model’s results. For example, it was considered a success to witness that the GAN had incorporated lfags into many images of generated politicians, and often placed them in a ‘speech stance’, thus capturing stereotypical features of politician pictures that distinguish them from other pictures of humans (see Figure 3).
The next choices were technical: what algorithm to use (the students were pointed to DC-GAN, StyleGAN, and StyleGAN-2), and automatic image preprocessing, which still left decisions regarding image size and crop type.
For Style-GAN based projects the networks were always pre-trained, and an important decision was what pre-trained model to use. Sometimes the decision was obvious, but in some cases students explored what happened when they override the most reasonable decision. For example, the Disney team wanted to generate whole body images and avoid photo-realistic faces, so they opted for a model pre-trained on animals rather than realistic human faces (Figure 4). As another example, the Gaudi architecture team realized that using a model pre-trained on skyscrapers resulted in no green (trees and plants), so they preferred the results obtained with a model pre-trained on objects (Figure 5). The butterfly team realized that it is relatively easy to generate beautiful butterflies, but using a model pre-trained on faces resulted in more symmetric butterflies (Figure 6a) than when using a model pretrained on objects (Figure 6b).
One team deliberately trained a model pre-trained
on one category (cats or horses) with a very diferent
dataset (toasters or kitchen aid devices), with the aim
of creating a hybrid animal and electronic device (e.g.,
toaster cats in Figure 7). They realized that training
over many steps resulted in interpolating the model
from one category to another, and they have
specifically tried to find the point in time (number of
training steps) where the balance is of interest,
systematically testing when the horses disappear into kitchen
aid (they were happy with the results obtained after on the nature of the creative process, making it
essen680 steps; Figure 8). While there has been computa- tially ‘non-interactive’ and limiting the scope of the
tional eforts to train GANs on such mixed datasets, work to a small number of iterations.
the goal in those cases was to teach the GANs to learn The next choice was the number of training steps or
separate modes [
Dataset level user manipulations are not popular in to a flat curve (i.e., the model keeps generating similar the field; one of the main reasons is most likely that images regardless of additional training), sometimes re-training is resource (time) consuming. Each ‘exper- earlier results are preferred. The only information iment’, even if it only involves fine tuning, typically they could use in RunwayML, other than manually intakes at least several hours. Clearly, this has an efect specting the resulting images, are the Frechet Inception distance (FID) scores (only programmers learned to plot and analyze additional information, such as learning curves).
One team realized that they can keep fine tuning the network more than once, over multiple steps. First, they generated butterflies form models pre-trained on faces or objects. Next, they wanted to further shape the results, with the aim at generating butterflies with fractal patterns (failed) or animal patterns. They continued training their fine-tuned butterfly model for a smaller number of steps with pictures of animal skin. Moreover, they realized that they could easily cut those pattern pictures into a rough shape of a butterlfy, so that the results will not lose the butterfly shape (Figure 9).
After the training process, the students learned the controversial art of ‘cherry picking’. While in most academic contexts this practice should be avoided, we programmers or by slightly modifying pre-exisiting simple code. Questionnaire results indicate that using the GUI with StyleGAN-2 was easier than code and resulted in better perceived quality. Interestingly, there was no significant diference in the perceived degree to which the results matched the original students’ intent. This is not because intent was very low, because the mean reported intent is higher than the average, and perceived quality and intent are highly correlated.
Nevertheless, our assessment in this respect is limited, because we are not only comparing two tools, but in many cases we are also comparing two models – most students who opted to use code used DC-GAN, whose results are inferior as compared to StyleGAN. We suggest that our course is not only useful for academic institutions but may also be useful in industry, for training non-programmers to co-create with AI.
We have analyzed the human intervention points in the creative process. The students were required to intervene in several sub-tasks that were not implemented in software. Additionally, some students intervened in order to refine the results. While we perceived the task in the context of almost autonomous AI, at least two teams interpreted the task in terms of AI assisting human creativity: one project aimed at augmenting the design of electronic devices with Figure 10: Automatically generated applications icons: the inspiration from animals, and the other project used results were low quality (left) so they were recreated man- the GAN to come up with preliminary sketches for apually (right). Results obtained with DC-GAN in Colab, by plication icons, which were then finalized by the huMaor Bluman and Bat Primo. mans. Allowing non-programmers more control and intervention points is clearly desired. However, while it is relatively straightforward to expose a few more suggest that in the context of AI-aided design or art hyper-parameters into software such as RunwayML or this is quite legitimate. It is important to understand ‘friendly’ code wrappers, the challenge is in providing the implications of the systems being generative, i.e., non-experts with intuitions about the expected way to once developed they can generate a practically infinite deal with these hyper-parameters. number of results. If one of the results is what you Finally, a very active area of current computational are looking for, and what you are looking for is very research is how to make generative models such as special and dificult to obtain, “cherry picking” is rea- GANs more ‘controllable’, using latent space algebra, sonable. Finally, one team of non-programmers who latent space disentanglement and more (e.g., [23, 24, opted to use DC-GAN were rightly disappointed from 25, 26]). We suggest that it is both interesting and the results, which were noisy. Nevertheless, they went important to see what happens when such tools are on to manually clean the results, suggesting that even “unleashed” to the hands of non programmers. As we if the AI ‘artist’ or ‘designer’ is not as competent as show here, they may discover new ways to improve the human, it can nevertheless be used as a source of results, which were not planned for by the algorithm inspiration (Figure 10). designers. Notably, and contrary to the majority of algorithmic efort, students tried to obtain their goal by 5. Discussion modifying the training data set – either by selecting a counter-intuitive pre-trained option, or by modifying their own datasets that were used for fine tuning the models.
[26] T. Park, M.-Y. Liu, T.-C. Wang, J.-Y. Zhu, Gaugan: semantic image synthesis with spatially adaptive normalization, in: ACM SIGGRAPH 2019 RealTime Live!, 2019, pp. 1–1.