Non-representational art-such as works by Wassily Kandinsky, Joan Mitchell, Willem de Kooning, etc.-showcases diverse artistic expressions and challenges viewers with its interpretive open-endedness and lack of a clear mapping to our everyday reality. Human cognition and perception nonetheless aid us in making sense of, reasoning about, and discussing the perceptual features prevalent in such non-representational art. While there have been various Computational Creative systems capable of generating representational artwork, only a few existing Computational (Co)Creative systems for visual arts can produce nonrepresentational art. How would a co-creative AI that incorporates elements of the human visual perception theory be able to collaborate with a human in co-creating a nonrepresentational art? This paper explores this challenge in detail, describes potential machine learning and non-machine learning approaches for designing an AI agent and introduces a new web-based, multi-agent AI drawing application, called Drawcto, capable of co-creating non-representational artwork with human collaborators.
Playing video games can be a highly creative activity,
requiring individuals to engage in creative behaviors like
content creation, collaborative building, problem solving, etc.
Computational Creativity (CC) in the visual arts has
gained attention since the early days of AARON
People often use abstract art and non-representational art
interchangeably to refer to the same painting style, yet there
are crucial differences between the two terms
Non-representational art represents the spiritual, mystic,
non-materialistic, experiential, or creative painting/thought
process of the artist
Generating visually sensible content in such a dynamic scenario is the main challenge for developing an AI agent for co-creating non-representational art. We cannot simply train the agent to use object detection or classification to make sense of and generate new strokes as usually there are no recognizable objects. At the same time, we can’t generate random strokes as they would not be visually sensible. Therefore, developing an AI that can create various strokes based on its perceptual ability to understand and reason with the quality of strokes made by the human collaborator is the challenge we address in this research.
We utilize the perceptual organization theory (or Gestalt
theory) for the agent(s) to make sense of and generate new
strokes while co-creating a non-representational artwork.
Gestalt theory describes a finite set of rules that guide and
aid the reasoning of our visual system. Some of the gestalt
grouping principles are proximity, balance, continuity,
similarity, etc.
We have organized the paper as follows. We examine potential learning and non-learning approaches for developing an AI agent in Related Work. The System Design section describes the current version of Drawcto and explains each component in detail. In the Discussion section, we reflect on the present drawbacks of the three drawing agents. Finally, we share potential future avenues of research we have identified for Drawcto in Future Work.
In recent years, research on developing image/sketch generation AI has gained a lot of interest. As a result, many research projects and AI architectures have explored image generation from various perspectives and for multiple reasons like co-creating, sketch-based image retrieval, image completion, design ideation, image stylizing, etc. This literature review focuses on diverse learning and nonlearning approaches for stroke generation for abstract or non-representational art.
The Sketch-RNN model
Similar to Sketch-RNN, another influential model is GAN
(Goodfellow et al. 2020). In GAN, two neural networks–
generator and discriminator–compete with each other to
make predictions. The role of the generator is to produce
output that highly resembles the actual data and, the role of
the discriminator is to identify the artificially created data.
Some of the research projects based on GAN are
SketchGAN
A transformer is a sequence-to-sequence model that uses an
attention-mechanism to identify important context, helping
it provide better results than an RNN
Many research projects deal with learning stroke generation
using reinforcement learning (RL). Researchers typically
train an agent by letting it interact with a simulated
painting environment. The painting environment can be
continuous (e.g., SPIRAL
There are many non-learning approaches to generate strokes
for abstract/non-representational art. For example, AARON
The above-related research highlights the following gaps in existing systems for (co)creating abstract artistic images.
All the learning approaches are black-box approaches. The AI agent can’t justify why a certain stroke in a specific location and particular style (color, width, length, weight, etc.) makes sense to the entire composition. Even with rulebased or shape grammar approaches, the agent can’t convey its perception of the composition as a whole. In other words, existing agents can not reason about the visual design or justify their actions while creating the artwork.
Barring Drawing Apprentice, none of the CC systems are capable of co-creating a non-representative art. But, a limitation of Drawing Apprentice is that it does not have any perceptual knowledge bootstraps, so Drawing Apprentice takes a black box learning approach to train the agent which results in it not being able to discuss its intention with a human collaborator. Even with generative systems, only a few research projects focus on creating non-representational art, and almost none discuss the intent behind the composition. It is also interesting to note that the existing systems are either single-agent or multi-feature systems.
The shortcomings mentioned above informed us to develop Drawcto- a multi-agent system for co-creating nonrepresentational artwork, which can explain its actions based on the current state of the canvas. In the following section, we discuss the system design and various agents’ logic for the current prototype of Drawcto.
We developed Drawcto as an easily accessible web-based application with the graphical interaction happening through a P5.js canvas. The canvas (frontend) communicates to the python server (backend) using HTTP Methods. The backend is responsible for the different AI agents’ logic. We developed the python server using the micro web framework Flask. Currently, we are hosting the application on Heroku. As shown in Figure 2, the user draws strokes and selects the agent on the interface; in response, the AI responds with its strokes and a textual description of its stroke intent. We have developed the backend in a modular manner allowing us to add or remove an agent based on its performance. In the following subsections, we describe the user interface and the AI agents in detail.
We designed all of the UI features to foster a human-like collaboration between an AI and a human. To anthropomorphize the AI, we named our AI avatar “Dr. Drawctopus” and created a vector image of a cyborg octopus to represent the AI. We chose an octopus to represent our AI with the idea that each tentacle will correspond to a different agent, symbolically conveying that it is part of the same system.
Prior research
Creating non-representational art requires time for
reflection-in-design, and turn-taking interaction can
facilitate this process. But, turn-taking can be a dynamic process;
the most straightforward approach seen in the literature we
adopted for Drawcto is simple turn-alternation
We wanted to present the AI stroke intention to the human collaborator coherently without disturbing the creative collaboration. Hence we decided to show the stroke logic in a dialog box, seeming as if Drawcto is communicating. Further, the AI dialogs were written in a way that reflects the friendly persona of the AI.
Along with the UI features mentioned above, the human collaborator can also toggle between the three agents via a clickable arrow. Figure 3 highlights the different parts of the user interface.
When the rule-based agent is selected, it reacts to the user’s
stroke(s) based on a set of hand-authored rules. We derived
these rules from perceptual grouping theory, such as
balance, symmetry, continuity, closure, etc.
The agent makes sense of the strokes strictly based on the observable, salient features on the canvas. We use the OpenCV library to make sense of and extract features from the strokes. The feature set includes - number of contours (for whole canvas or current stroke), the center of mass, white space, and four-way symmetry. We make use of this feature set to traverse a decision tree to find an applicable rule. Some examples of rules include - closing a stroke if the user drew an open stroke; connecting strokes if the user drew more than one separate stroke; enclosing a stroke if the no. of contours is above a threshold; creating similar strokes if the canvas has an empty area, etc.
To better understand how a rule-based agent works consider the following scenario. Assume the human collaborator draws an open shape like a ‘U’ shape or a polygon with one side missing on the canvas and finishes their turn. Then, a snapshot of the canvas is sent to the backend. With the help of functions in the OpenCV library like findContour or isClosed, we develop a feature set that indicates that the shape is open. Following this, the agent traverses a predefined decision tree and comes up with two possible moves– to close the open-shape or, to draw a similar but new distorted (scaled up or down, sheared, etc.) open shape. The agent randomly chooses between these moves, produces the relevant stroke on the canvas, and presents a textual description of the rule and why it was triggered to the human collaborator.
We were curious if we could build a data-driven agent in
Drawcto that relied on latent information learned from
existing artworks or sketch datasets instead of following
predefined rules. We developed two separate agents to explore
this “authorless” approach in Drawcto - the Quick Draw and
Artist agents. Both the agents are based on Google’s
interactive SketchRNN
producing a new stroke in exact continuation to the human
collaborator’s last drawn stroke. The agent utilizes the ml5.js
library’s SketchRNN model
SketchRNN Artist Agent This agent–to produce strokes– utilizes the SketchRNN model trained on our custom dataset of around 2000 images of non-representational artworks from the web. The goal for the Artist agent was to see if the model would automatically learn visual concepts such as symmetry, shape-completion, balance, etc. However, getting the correct training data was the biggest challenge. To overcome this, we obtained famous non-representational artworks and extracted edges from them and converted them into simple sequential vector drawings. Figure 5 shows examples of the edge extraction we did to collect data. We can see that these images are composed of distinct shapes - like circles, rectangles, etc., have a sense of composition - like positive & negative space, symmetries, etc., and textures are depicted through different densities of the shorter lines. We trained the Artist agent on this data, and Figure 4b shows the interaction with the artist agent.
In the related work section, we identified various learning and non-learning approaches that we could take to tackle the challenge of AI generating different semantically accurate strokes while co-creating non-representational art. From that, we chose one non-learning approach - a rule-based Agent, and one learning approach - RNN Agent(s) to incorporate in the current prototype of Drawcto. In this section, we reflect on the current limitations of the three co-creative agents.
The rule-based agent currently echoes and acknowledges the user’s strokes but rarely produces a novel stroke to the composition. In other words, though the rule-based agent can generate and justify new strokes, it lacks stroke variability and can become predictable after using it a few times.
The RNN agents, on the other hand, especially the quick draw agent, produce a whole variety of strokes due to the diversity and volume of the Quick! Draw data. The dialogues for both the RNN agents give a clue about the training data but fail to reason about a particular stroke. We will have to develop a separate gestalt module to analyze the produced stroke and provide suitable explanations. We notice that a lot of the responses from the artist agent are two lines at right angles. We believe this is because each training image had a boundary. Hence the model learned it as an essential component to any drawing. However, we noticed that the artist agent could respond to the user’s stroke, complementing the essence of their drawing style.
Adding color, texture, line variation, etc., in a particular artist’s style can enhance the co-creative experience. Figure 4e - Figure 4h shows our initial experiment with style transfer; The images in Figure 4 show art in Kandinsky’s style. In the future, we hope to develop an agent which will let the user choose to add a stroke in a particular artist’s style.
Drawing is an embodied activity, and studies show that
maintaining embodied interaction can improve the
cocreative drawing experience
Lastly, For building a learning agent capable of
producing stokes based on gestalt theory, Reinforcement learning
approaches appear to be a very promising avenue, especially
with research like PQA
We thank Arpit Mathur, Bhavika Devnani, Laney Light, Luowen Qiao, and Tianbai Jia for collaborating to develop this project. We thank Erin Truesdell for providing helpful insights and feedback. you think you can sketch? ACM Transactions on Graphics 39(6):1–15.