=Paper=
{{Paper
|id=Vol-2903/IUI21WS-HAIGEN-7
|storemode=property
|title=Studying the Impact of AI-based Inspiration on Human Ideation in a Co-Creative Design System
|pdfUrl=https://ceur-ws.org/Vol-2903/IUI21WS-HAIGEN-7.pdf
|volume=Vol-2903
|authors=Jingoog Kim,Mary Lou Maher,Safat Siddiqui
|dblpUrl=https://dblp.org/rec/conf/iui/KimMS21
}}
==Studying the Impact of AI-based Inspiration on Human Ideation in a Co-Creative Design System==
https://ceur-ws.org/Vol-2903/IUI21WS-HAIGEN-7.pdf
Studying the Impact of AI-based Inspiration on Human Ideation
in a Co-Creative Design System
Jingoog Kima, Mary Lou Mahera and Safat Siddiquia
a
University of North Carolina at Charlotte, 9201 University City Blvd, Charlotte, NC 28223, USA
Abstract
Co-creative systems in design enable users to collaborate with an AI agent on open-ended
creative tasks in the design process. This paper describes a co-creative system that supports
design creativity by encouraging the exploration of design solutions in the initial idea
generation process. The Collaborative Ideation Partner (CIP) is a co-creative design system that
provides inspirational sketches based on the visual and conceptual similarity to sketches drawn
by a designer. To evaluate the effect of CIP on design ideation, we conducted an exploratory
study that measures ideation in a co-creative system. To measure the ideation, we developed a
way of measuring ideation in a co-creative system including an outcome and a process
approach. From the exploratory study, we learned that the image quality in the dataset is
important in AI-based creativity and inspirations based on conceptual similarity to the target
design have more impact on ideation than inspirations based on visual similarity to sketches
drawn by a designer. We present the architecture of the CIP system and a study design based
on what we learned from the exploratory study.
Keywords 1
Co-Creativity, Co-Creative System, Ideation, Collaboration
1. Introduction We present a co-creative sketching AI
partner, the Collaborative Ideation Partner
(CIP), that provides inspirational sketches
Computational co-creative systems are a
based on the visual and conceptual similarity to
growing research area in computational
sketches drawn by a designer. To select an
creativity. While some research on
inspiring sketch, the AI model of CIP computes
computational creativity has a focus on
the visual similarity of images in a data set
generative creativity [1]–[9], co-creative
based on the vector representations of visual
systems focus on how systems that implement
features of the sketches and the conceptual
generative creativity can work with humans on
similarity based on the category names of the
a creative task [10]–[17]. Co-creative systems
sketches using two pre-trained word2vec
have enormous potential to enhance human
models. The turn-taking interaction between
creativity since they can be applied to a variety
the user and the AI partner is designed to
of domains associated with creativity and
facilitate communication for design ideation.
encourage designers’ creative thinking.
The CIP was developed to support an
Understanding the effect of co-creative systems
exploratory study that evaluates the effect of an
in the ideation process can aid in the design of
AI model for visual and conceptual similarity
the generative AI models in co-creative systems
on design ideation in a co-creative design tool.
and the evaluation of the impact of co-creative
systems on human creativity.
J
oint Proceedings of the ACM IUI 2021 Workshops, April 13-17,
2021, College Station, USA
EMAIL: jkim168@uncc.edu (A. 1); M.Maher@uncc.edu (A. 2);
ssiddiq6@uncc.edu (A. 3)
ORCID: 0000-0003-3597-6153 (A. 1)
Copyright ©2021 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
Table 1
Comparison of original design and current design of CIP
Original Design of CIP Current Design of CIP
Interface
Stimuli Visual and Conceptual Similarity Conceptual Similarity
Inspiring Low fidelity sketches of a general object high fidelity images of a creative design
images
Modes of 4 modes: random, similar, conceptually 2 modes: random inspiration,
inspiration similar and visually different, visually conceptually similar
similar and conceptually different
Dataset 3450 Sketches (QuickDraw [18]) 100 images
To evaluate the impact of co-creative CIP system and study design based on what we
systems in design, we measure design ideation learned from the exploratory study. Table 1
in a co-creative system. Ideation, an idea shows the comparison between the CIP system
generation process for conceptualizing a design used for the exploratory study and the current
solution, is a key step that can lead a designer CIP system based on what we learned from the
to an innovative design solution in the design exploratory study. The current CIP system
process. Idea generation is a process that allows focuses on conceptually similar inspirations to
designers to explore many different areas of the the target design and provides high fidelity
design solution space [19]–[24]. Ideation has images of creative designs. This study aims to
been studied in human design tasks and identify the effect of AI inspiration on design
collaborative tasks in which all participants are ideation through a way of measuring ideation in
human. Collaborative ideation can help people a co-creative system.
generate more creative ideas by exposing them
to ideas different from their own [25]. This 2. Computational co-creative
paper has a focus on evaluating how a co-
creative agent influences the ideation process in systems
a human-AI collaboration.
In this paper, we describe an exploratory Computational co-creative systems are one
study measuring ideation when co-creating of the growing fields in computational
with the CIP and what we learned from the creativity that involves human users
exploratory study. To measure ideation in a co- collaborating with an AI agent to make creative
creative system, we employ two approaches: an artifacts. The distinction of co-creativity from
outcome-based approach that focuses on the computational creativity is that co-creativity is
end product of the design, and a process-based a collaboration in which multiple parties
approach that focuses on thought processes contribute to the creative process in a blended
during the design. From the exploratory study, manner [26]. Co-creative systems have been
we learned that the quality of the images in the applied in different creative domains such as
dataset is important in AI-based creativity for art, music [14], dance [15,27], drawing [28],
the impact on designer’s creativity and and game design [39,56].
inspirations based on conceptual similarity to Evaluating co-creative systems is still an
the target design has more impact on ideation open research question and there is no standard
than inspirations based on visual similarity to metric for measuring computational co-
sketches drawn by a designer. We updated the creativity [31].
Figure 1: Evaluating the creative ideation partner
The research on co-creative systems shows to explore different areas of the design solution
various approaches to evaluate co-creative space [23,32]. A design process is an evolution
systems and computational co-creativity. Some of different kinds of representations [33]. In a
researches focus on evaluating the interactive design process, designers externalize and
experience [29,14,30,31] and others focus on visualize their design intentions and
the effectiveness of the system to produce or communicate with visualizations to interact
generate a creative outcome [39,56]. Karimi et with their internal mental images [34]. During
al. [31] presented a framework for evaluating ideation, designers commonly use freehand
creativity in computational co-creative systems. sketches and rough physical models as a tool for
This framework responds to four questions that constructing external representations as
serve to characterize the many and varied cognitive artifacts of design [35]. Making
approaches to evaluating computational co- sketches and physical models is an interaction,
creativity: who is evaluating the creativity, a conversation [36]. In the ideation stage,
what is being evaluated, when does evaluation designers frame problems producing new
occur, and how the evaluation is performed. discoveries through the conversation. The
The framework enables comparisons of graphical and physical representations as
evaluation focus and methods across existing cognitive artifacts are essential in the ideation
co-creative systems. Using this framework, we process.
have shown that the evaluations of the existing Many ideation methods have been
co-creative systems described in this section developed to support designers in generating
respond to “what is being evaluated” with a innovative design solutions. Ideation methods
focus on evaluating the interactive experience provide a normative procedure on how to
and the final product. In this paper, we respond overcome certain blocks to creativity [37].
to “what is being evaluated” and “how is the Analogy is an ideation method and we focus on
evaluation performed” by evaluating the analogy to develop a co-creative design tool.
ideation process using FBS and Problem- Analogical reasoning is an inference method in
Solution index, and the metrics for evaluating design cognition to develop a design leading to
the novelty, variety, quality, and quantity of unexpected discoveries [38]. Design-by-
ideas in the creative outcome, as shown in Analogy (DbA) is a design tool that provides
Figure 1. Section 3 describes how we define inspiration for innovative design solutions.
and measure ideation in more detail. Inspirations in Design-by-Analogy (DbA) are
achieved by transferring a design problem
3. Defining and measuring design (source) to a solution (target) in another domain
[39]. The association between a source design
ideation and a target design can be based on semantic
(conceptual) characteristics or visual
Ideation is a creative process where (structural) representations. The semantic and
designers generate, develop, and communicate visual stimuli thus can be a basis for developing
new ideas. Ideation in design can lead to computational tools that support design
innovative design solutions through generating ideation. The Collaborative Ideation Partner
diverse concepts [19]–[22], [24]. The goal of (CIP), a co-creative design system we present
design is to develop useful and innovative in this paper, uses visual and conceptual
solutions and design ideation allows designers
similarity metrics as key factors for classes are augmented by requirements (R) that
collaborative ideation using design by analogy. come from outside the designer and description
Evaluation of ideation can be classified into (D) that is the document of any aspect of
two groups: outcome-based approaches and designing. In this ontological view, the goal of
process-based approaches [40]. Outcome-based designing is to transform a set of requirements
approaches focus on evaluating the ideation and functions into a set of design descriptions.
process based on the designs, or outcomes, and The transformation of one design issue into
the characteristics of ideas generated. Process- another is defined as a design process [44].
based approaches focus on evaluating idea The design process can be viewed as
generation processes based on the cognitive interactions between two notional design
processes inherent to creative thought. Process- spaces: problem space and solution space
based approaches collect data via a protocol [45,46]. The Problem-Solution (P-S) index
study and analysis using ideation cognitive [47,48] is a measurement capturing the meta-
models. Outcome-based approaches have level structures of design cognition in terms of
become more prevalent than process-based problem-focused and solution-focused design
approaches due to the inherent complexity and issues. This measurement uses an integration of
difficulties in using process-based approaches the FBS ontologically-based coding scheme
[41]. There have been several metrics used to with a Problem-Solution (P-S) division [47,48].
evaluate the performance of idea generation In the P-S division, design issues of R, F, and
techniques such as fluency and novelty that Be are mapped to problem space and design
cognitive psychologists consider as the primary issues of Bs, and S are mapped to solution space
measures of idea generation. Shah et al. [41] [47]. A design session with a P-S index larger
introduced four types of outcome-based metrics than 1 as one with a problem-focused designing
for measuring ideation effectiveness that are style, and a session with a P-S index value less
commonly used for evaluating idea generation than or equal to 1 as one with a solution-focused
in design: novelty, variety, quality, and quantity style. The P-S index can be used to compare
of designs. Novelty is a measure of how design cognition while using different
unusual or unexpected an idea is as compared creativity techniques for concept generation in
to other ideas. Variety is a measure of the collaborative design settings.
explored solution space during the idea
generation process. The generation of similar 4. The collaborative ideation
ideas indicates low variety and hence, less
probability of finding better ideas in other areas partner (CIP)
of the solution space. Quality is a subjective
measure of the feasibility of an idea and how We developed the Collaborative Ideation
close it comes to meet the design specifications. Partner (CIP) as a co-creative design system
Quantity is the total number of ideas generated, which builds on previous works [49,50] that
generating more ideas increases the possibility interprets sketches drawn by a user and
of better ideas. These metrics enable a provides inspirational sketches based on visual
comparison of a designer’s exploration of a similarity and conceptual similarity. We
design space using different ideation methods. developed the CIP to explore evidence for the
Process-based approaches evaluate idea hypothesis that: AI models for contributions to
generation based on the cognitive processes via a creative product based on a measure of visual
a protocol analysis and cognitive models. The and conceptual similarity produce different
Function-Behavior-Structure (FBS) ontology ideation processes and outcomes than the
[42,43] is a design ontology that describes random condition.
designed things, or artifacts, irrespective of the The user interface of CIP is shown in Figure
specific discipline of designing. The function 2. There are two main spaces in the CIP
(F) of a designed object is defined as its interface: the drawing space (pink area) and the
teleology; the behavior (B) of that object is inspiring sketch space (purple area). The
either derived (Bs) or expected (Be) from the drawing space consists of a design task
structure, where structure (S) represents the statement, undo button, clear button, and user’s
components of an object and their canvas.
compositional relationships. These ontological
Figure 2: User interface of collaborative ideation partner
The design task statement in the drawing during an online game where players were
space includes the object to be designed as well asked to draw a particular object within 20
as a context to further specify the objects’ use seconds. The dataset includes 345 categories
and environment. The user can draw a sketch in with more than 50 million labeled sketches,
the drawing space and edit the sketch using the where sketches are the array of the x and y
undo and clear button. The inspiring sketch coordinates of the strokes. The system uses the
space includes an “inspire me” button, the name simplified drawing json files that use Ramer–
of the inspiring object, and a space for Douglas–Peucker algorithm [51,52] to simplify
presenting the AI partner’s sketch. When the the strokes, and position and scale the sketches
user clicks the “inspire me” button after into a 256 X 256 region. The stroke data
sketching their design concept, the AI partner associated with these sketches are used to
provides an inspiring sketch based on visual calculate the visual similarity and the
and conceptual similarity. An ideation process corresponding category names are used to
using CIP involves turn-taking measure the conceptual similarity.
communications between the user and the AI
partner. Another part of the CIP interface in 4.2. AI models for visual and
addition to the two main spaces is the top area
(grey area) including a hamburger menu and an conceptual similarity
introductory statement. The hamburger menu
on the top-left corner of the interface includes The CIP has 2 distinct components for
four design tasks (i.e. sink, bed, table, chair) measuring similarity between the user’s sketch
and allows the experiment facilitator to select and the sketches in the dataset: one component
one of the design tasks. Each design task is for calculating visual similarity and another
associated with different categories of ideation component for calculating conceptual
stimuli. similarity. Figure 3 shows how the CIP system
identifies an inspiring sketch: the visual
4.1. Dataset similarity is based on the vector representations
of visual features of the sketches and the
conceptual similarity based on the category
For the source of inspiring sketches, the names of the sketches using two pre-trained
original CIP uses a public benchmark dataset word2vec models.
called QuickDraw! [18], which was created
Figure 3: AI-Based Co-Creativity in the CIP
For the visual similarity component, we component of CIP that considers the conceptual
followed the precedent for using neural similarities of the sketches.
network models in computational creativity For the conceptual similarity component, we
described in [50,53] and trained a model with 3 considered sketch category names in the
convolutional layers, 2 LSTM layers, and a QuickDraw dataset as the concepts of the
softmax output layer on the QuickDraw dataset. sketches that contain 345 unique categories. We
This model provides a latent space used two pre-trained word2vec models, Google
representation for measuring the distance, or News [54] and Wikipedia [55], and calculated
similarity, between 2 sketches. For all the cosine similarities for measuring the conceptual
sketches in the dataset, we collected the last similarities between the object categories of the
LSTM layer of the trained model and used that design tasks and the categories of inspiring
as the vector representations of visual features sketches from the dataset. For each category of
of the sketches. We used the K-means the design tasks, we generated two sorted lists
algorithm to identify 10 clusters of sketches and of conceptually similar category names, one for
randomly selected one sketch of each cluster as each word2vec model, and then used human
a typical sketch for that cluster of sketches. judgement to compare the sorted lists and select
Thus, we converted the QuickDraw dataset of the top 15 common conceptually similar
50 million sketches into 3450 sketches (345 category names that appear in both lists. This
categories, each has 10 sketches). To prepare final step of using human judgement improved
the user’s sketch for comparison with the the alignment between the conceptual
sketches in the dataset, the CIP collects the user similarities of AI models and human
sketches as an array of x and y coordinates of perception. The conceptual similarity
strokes and simplifies the strokes using Ramer– component of CIP uses the common list of
Douglas–Peucker algorithm [51,52]. It also category names for sorting the sketches based
positions and scales the user’s sketch into the on the conceptual similarities.
256 X 256 region to match the sketch format
with the input dataset of the trained model. The 4.3. AI-based inspiration in CIP
CIP takes the last LSTM layer of the trained
model as the vector representation of visual
features of the user’s sketch, and calculates the To support an exploratory study that
Euclidean distance to measure visual similarity measures ideation when co-creating with CIP,
between the user’s sketch and 3450 sketches of the interaction with CIP has four distinct modes
of inspiration that vary the visual and
Quickdraw dataset. The visual similarity
component of the CIP prepares a sorted list of conceptual similarity. Each of the four modes
visually similar sketches to generate the final appears as a design task (i.e. sink, bed, table,
sequence of sketches in the conceptual chair) in the CIP interface.
• Inspire with a random sketch (sink): • Condition B (treatment condition):
The CIP selects a sketch randomly from the visually and conceptually similar (bed)
sketch dataset to be displayed on the AI • Condition C (treatment condition):
partner’s canvas. conceptually similar and visually different
• Inspire with a visually and (table)
conceptually similar sketch (bed): The CIP • Condition D (treatment condition):
selects a sketch from a set of sketches where visually similar and conceptually different
each one is similar visually and conceptually (chair)
to the user’s sketch (e.g. user sketch - a bed,
AI sketch - a similar shape of bed to the During the study, for each participant and
user’s sketch). for each condition we collected video protocol
• Inspire with a conceptually similar and data during the design session and a
visually different sketch (table): The CIP retrospective protocol after the design session.
selects a sketch from a set of sketches where The protocol including the informed consent
each one is conceptually similar but visually document has been reviewed and approved by
different to the user’s sketch (e.g. user our IRB and we obtained informed consent
sketch - a square table, AI sketch - a round from all participants to conduct the experiment.
table). We recruited 12 students from human-centered
• Inspire with a visually similar and design courses for the participants: each
conceptually different sketch (chair): The participant engaged in 2 conditions: a control
CIP selects a sketch from a set of sketches condition and one of the treatment conditions,
where each one is visually similar but with 4 participants for each of the 3 groups of
conceptually different to the user’s sketch within-subject design (i.e. A&B, A&C, A&D).
(e.g. user sketch - a circular chair back, AI The experiment is a mixed design with N=4 and
sketch - a face). a total of 12 participants.
The task is an open-end design task in which
5. Exploratory study: measuring participants were asked to design an object in a
given context through sketching. To reduce the
ideation when co-creating with learning effect, different objects for the design
the CIP task were used for each condition: a sink for a
accessible bathroom (condition A), a bed for a
The goal of the exploratory study is to senior living facility (condition B), a table for a
evaluate the effect of AI inspiration on ideation tinkering studio, a collaborative space for
through an analysis of the correlation between designing, making, building, crafting, etc.
conceptual and visual similarity with (condition C), a chair for a gaming computer
characteristics of ideation. To measure ideation desk (condition D). The participants used a
when co-creating with the collaborative laptop and interacted with the CIP interface
ideation partner, we applied both evaluation using a mouse to draw a sketch while
methods of ideation: an outcome-based performing the design task.
approach (i.e. novelty, variety, quality, The procedure consists of a training session,
quantity) and a process-based approach (i.e. P- two design task sessions, and two retrospective
S index). protocol sessions. In the training session, the
participants are given an introduction to the
features of the CIP interface and how they work
5.1. Study design to enable the AI partner to provide inspiration
during their design task. After the training
The type of study is a mixed design of session, the participants perform two design
between-subject and within-subject design with tasks in a control condition and a treatment
four conditions. There are 3 groups of within- condition. The study used a counterbalanced
subject design (i.e. A&B, A&C, A&D) in this order for the two design tasks. The participants
study and each group has a control condition have no time limits to complete the design task.
(i.e. condition A) and one of 3 treatment The participants were given as much time as
conditions (i.e. condition B, C, D). needed to perform the design task until they
• Condition A (control condition): were satisfied with their design. The
randomly (sink)
participants are free to click the “inspire me” don’t have any inspiration with the pictures.”
button as many times as they would like to get This case shows an example that participants do
inspiration from the system. However, the not have many ideas from random inspirations.
participants were told to have at least 3 As shown in Figure 4b, P4 drew a basic bed
inspirational sketches (i.e. clicking the “inspire and a pillow before getting the first inspiration
me” button at least 3 times during a design then requested inspiration from the AI partner.
session), a minimum number of inspirations, The first inspiring sketch was a chair and P4
from the system. The facilitator is present added a stool, table, and a stair next to the bed.
during the design task but does not interfere in After that, P4 had two more inspirations, bed
the design process. Once the participants finish and couch, and added bed guard around the bed.
the two design task sessions, the participants P4 described that the bed guard idea came from
are asked to explain what they were thinking the armrest of the couch. P4 then had 2 more
based on watching their design session inspiring sketches, couch and sleeping bag, and
recording as time goes on, and how the AI's added a curtain. P4 mentioned that the curtain
sketches inspired their design in the idea came from the enclosing feature of the
retrospective protocol session. sleeping bag and couch. The next inspiring
sketch is a table and P4 edited the foldable table
5.2. Observations of ideation with on the bed. After that, P4 had three more
inspiring sketches and added a slide that helps
CIP getting out of the bed easily. P4 described that
the slide idea came from the shape of the tent.
We observed the video stream data to see As shown in Figure 4c, P2 drew a rectangle
how participants develop their design ideas for a table before getting the first inspiration
communicating with the inspirations and the then tried to get an inspiration from the AI
participants' responses to inspirations show partner. The first inspiring sketch was a golf
different patterns of users on the use of CIP in club and P2 added table legs mimicking the
an ideation process. Figure 4 shows typical shape of a golf club. P2 then had a fireplace
examples of the process for the evolution of the sketch and added a large grid paper on the table.
participant’s sketch using CIP in each P2 described that the grid paper idea came from
condition. In an evolution of the participant’s the way the lines are drawn in the fireplace.
sketch, participants in each condition start with After that, P2 had matches and added a table
a basic shape of the target design then develop lamp. P2 then had two pool sketches and added
the design with inspiration from the AI partner. a pencil cup. The last inspiring sketch is a wine
Participants explored many inspiring sketches glass but P2 did not change the design with the
in condition A but did not have many design wine glass.
changes; while participants in conditions B, C, As shown in Figure 4d, P3 drew a basic
and D developed their design in response to chair without any special function for the
fewer inspiring sketches. context of gaming before getting the first
As shown in Figure 4a, P1 drew a basic sink inspiration then requested inspiration from the
with a handrail before getting the first AI partner. The first inspiring sketch was a
inspiration then tried to get an inspiration from raccoon and P3 added an ear shape decoration
the AI partner. P1 had 7 inspiring sketches but on the top of the chair and an eye shape headrest
did not change anything for the design. P1 then getting an inspiration from the shape of the
cleaned all the canvas then drew a new sketch raccoon sketch (i.e. ear, and eye). P3 described
which is a sink with a motion sensor. P1 had 4 that “So, I saw the raccoon and I kind of liked
inspiring sketches and did not change anything how its ears were. Cause I have seen things,
again for the design. P1 cleaned the canvas and where people have really interesting chairs,
drew a new sketch again applying the motion and I think people that game may usually want
sensor idea again then had 2 inspiring sketches. more interesting chairs. So, I thought it'd be
However, P1 finally finished the design without cool to have little ears at the top, and then make
any changes. During the retrospective session, the mask kind of like, a pillow.” After that, P3
P1 mentioned he did not get ideas from the had the second inspiring sketch which is a
inspiring sketches several times, for example “I power outlet.
(a) The evolution of P1 design in condition A (randomly)
(b) The evolution of P4 design in condition B (visually and conceptually similar)
(c) The evolution of P2 design in condition C (conceptually similar and visually different)
(d) The evolution of P3 design in condition D (visually similar and conceptually different)
Figure 4: AI-Based Co-Creativity in the CIP
P3 added a speaker on the ear decoration, tasks and verbalizing the ideation process
buttons on the armrest to control sound during the retrospective protocol. We recorded
volume/massage/lights, and power cord. P3 the entire design task sessions and retrospective
described that “the power outlet really gave me sessions for each participant. The sketch data
a lot of the inspiration. I thought... instead of collected from the recordings of design task
just random ears, it could be like a speaker. sessions shows the progress of design and the
Then, I thought all the little dots on the armrest final design visually for each design task
could be buttons, to do things. If it's different session. The verbal data collected from the
things, like sound volume, or could do a recordings of retrospective sessions records
massage. The little line coming out of it, would how the participants came up with ideas
be to plug it into the wall, so all the buttons collaborating with the AI partner and applied
could work.” In this case, the idea came from the ideas to their design.
the inspiring sketch was transferred to new
functions of the chair while the idea came from 5.4. Data segmentation and
the raccoon was transferred to the shape of the
chair. P3 then had 5 more inspiring sketches coding
(i.e. rain, hurricane, zigzag, and camouflage).
P3 mentioned that they were inspired from the To analyze the verbal data collected from
irregular lines of the sketches and added sound the retrospective sessions, the verbal data of all
projecting lines next to the ear shape speaker retrospective protocol sessions (i.e. 12 sessions
and a pillow on the seat. After that, P3 had nine of condition A, 4 sessions of condition B, 4
more inspiring sketches, but did not change sessions of condition C, and 4 sessions of
anything for the chair design. P3 talked about a condition D) was transcribed. The transcripts
decoration idea from the star shape of an were segmented based on the inspiring sketches
aircraft carrier and snowflake but did not the participant clicked. A segment starts with an
change the chair design. inspiring sketch and ends when the inspiration
is clicked for the next sketch. To identify each
idea in an inspiring sketch segment, we
5.3. Data collected
segmented the inspiring segments again based
on FBS ontology [42,43] as an idea segment,
Two types of data were collected for since an inspiring sketch segment includes
analyzing the study results: a set of sketches multiple ideas. An inspiring segment thus
that participants produced during the design includes multiple idea segments. The idea
segments were coded based on FBS ontology condition as low/medium/high quality, in two
[42,43] as requirement (R), function (F), evaluation rounds. In the first-round of
expected behavior (Be), behavior from evaluation, each judge evaluated the final
structure (Bs), and structure (S). A R segment designs identifying some criteria for evaluating
is an utterance that talks about the given the quality of ideas. Once the judges finished
requirement in the statement of design task (e.g. the first-round of evaluation, they shared the
accessible bathroom) or a new requirement the criteria they identified/used, not sharing the
participant came up with for the design (e.g. if results of the evaluation, then made a consensus
someone is not able to reach the height); a F for the criteria that will be used for the second-
segment is an utterance that talks about a round evaluation. The criteria that the judges
purpose or a function of the design object (e.g. agreed for evaluating the quality of ideas in this
more accessible); a Be segment is an utterance study are the number of features, how
that talks about an expected behaviors from the responsive the features are to the specific task,
structure (e.g. water could automatically come how creative the design is. In the second-round
out), a Bs segment is an utterance that talks evaluation, each judge evaluated the final
about a behavior derived from the structure design again using the agreed criteria. Quantity
(e.g. pressing on), a S segment is an utterance is the total number of ideas generated. For
that talks about a component of the design measuring quantity in this study, the number of
object (e.g. button). Two coders coded the idea ideas both new ideas and repeated ideas coded
segments individually based on the coding as R/F/B/S is counted in a design.
scheme above then came to consensus for the
different coding results. 5.6. Measuring ideation: process-
based approach
5.5. Measuring ideation:
outcome-based approach For the process-based approach, we used the
P-S index [48] to examine the design cognition
For the outcome-based approach, we from a meta-level view (i.e., a single-value
developed four metrics based on [41]: novelty, measurement). For the meta-level view, the P-
variety, quality, and quantity of design. Novelty S index is calculated by computing the number
is a measure of how unusual or unexpected an of the total occurrences of the design issues
idea is as compared to other ideas. In this study, concerned with the problem space (i.e. R, F, Be)
a novel idea is defined as a unique idea across and related to the solution space (i.e. Bs, S). A
all design sessions in a condition. For design session with a P-S index larger than 1 as
measuring novelty, we counted how many one with a problem-focused style, and a session
novel ideas in the entire collection of ideas in a with a P-S index value less than or equal to 1 as
design session (personal level of novelty) and a one with a solution-focused style. In addition to
condition (condition level of novelty). We calculating the P-S index of each design
removed the same ideas across all design session, we looked at the number of problems
sessions in a condition then counted the number and solutions to identify a distinct difference
of ideas. Variety is a measure of the explored between the conditions.
solution space during the idea generation
process. Each idea segment was coded whether 5.7. What we learned from
it is a new idea or a repeated idea in a design
session. For measuring variety in this study, exploratory study
only the number of new ideas coded as R/F/B/S
is counted in a design session while the metric With the data collected in the exploratory
of quantity includes both new ideas and study, we compared outcome-based features
repeated ideas. Quality is a subjective measure (i.e. novelty, variety, quality, quantity) and
of the design. In this study, quality is measured process-based features (i.e. P-S index). Our
using the Consensual Assessment Technique findings show that the AI-based stimuli
(CAT) [56], a method in which a panel of expert produce different ideation outcomes and
judges is asked to rate the creativity of projects. processes when compared to random stimuli.
Two judges, researchers involved in this study,
individually evaluated the final design in each
Figure 5: Current CIP user interface
Novel ideation, evidenced by an increase in participant’s sketch was more accurate at the
the variety and quantity of ideas, is associated beginning of the design session, but was less
with AI-based conceptually similar stimuli. The accurate as the participant’s sketch became
findings from analysis of the P-S index show more complicated. Third, the CIP in condition
that AI-based visual and conceptual similarity D (visually similar and conceptually different)
is associated with a problem-focused designing often provides sketches that are not visually
style that produces more solutions than we similar to the participant’s sketch since the
found in the condition with random inspiring sketches are first selected to be
inspirations. We found that participants in conceptually different, and that reduces the
condition C (conceptually similar and visually potential for identifying sketches that are
different stimuli) produced more functions than visually similar.
in condition A (random).
In our observations of the exploratory study, 6. Current CIP and study design
we identified some issues on the sketch data set
and AI-based visually similar stimuli. First, the
quality of the sketch dataset is very important From the exploratory study we learned that
to inspire participants to come up with new inspiration based on conceptual similarity has
more impact on the novelty and variety of ideas
ideas. The sketches in this dataset are not the
result of a design process. The sketches in the than visual similarity and that the quality of the
QuickDraw dataset are generated to represent dataset is important for the design ideation. For
the basic shape of a given well known object. our current CIP and study design, we developed
Based on the retrospective protocol data, a more comprehensive model for conceptual
participant’s ideas mostly came up from similarity based on multiple features of the
purposes, functions, features, and structures of design rather than only a categorical word, and
the inspiring sketches, and the simple collected a dataset of designs as the basis for
representation of objects in the QuickDraw inspiration rather than a dataset of sketches on
dataset were not providing very rich inspiration. well known objects. Figure 5 shows the current
Second, the complexity of participants’ CIP user interface providing an inspiring image
sketches increased during the design session, of a creative design instead of a simple sketch.
affecting the accuracy of the visual similarity The target design is a bed for a senior living
facility and the inspiring image is a smart
measure used to select an inspiring sketch. The
AI model for visually similarity to the patient room. The smart patient room is the
most conceptually similar design to the target statement) and each word in set 2 (i.e. words in
design. The design of the smart patient room the design feature). A Wikipedia pre-trained
includes many functions and objects associated word2vec model is used to calculate the
with the context of a senior living facility such similarity between the two words using a pair-
as reclining bed, bed table, magazine holder, wise comparison: a word from the design task
trash can, digital screen for health care, and statement and a word from the design features
wheels for mobility. of an image. We calculate the cosine similarity
score for each pair of a design task statement
6.1. Dataset word and a design feature word and create a set
of cosine similarity scores including all pairs of
design task statement words and design task
For the source of inspiring designs, we feature words for each image in the image
collected a dataset of high fidelity images of dataset. As a conceptual similarity score
creative designs. To create the new dataset, we between the target design and the image, we use
selected 20 common categories from the the average score of cosine similarity scores for
categories of QuickDraw dataset that are each image. For example, a design statement
conceptually similar to the target designs of the
includes 4 words (i.e. bed, senior, living,
exploratory study (i.e. sink, bed, table, chair).
facility) and an image includes 4 words of
We then searched for images of 5 creative
design features (e.g. comfort, massage,
designs online for each category using combinational, chair). For measuring the
keywords “creative”, “novel”, “unusual”, conceptual similarity between the target design
“design” (e.g. creative sink, unusual bed). The and the image, we calculate each cosine
dataset thus contains 20 categories with 100 similarity score for 16 pairs of words (4 words
labeled images. Each image has three fields: id, x 4 words) then calculate the average cosine
object name, and design feature. Id is the unique similarity. We create the conceptual similarity
identifier that is assigned to each image. Object
ranking based on the similarity score of each
name is the name of the design that is image. The system selects from the most
represented in the image (e.g. electric massage conceptually similar image in order when the
bed, robotic advisor, smart sofa). Design user clicks the inspire button.
feature is keywords that represent the design
features and unique functionalities of the design
(e.g. multi-functional, entertainment, massage, 6.3. Study design
combinational, digital, tv).
In our study design we focus on the impact
6.2. AI model for conceptual of the AI model for conceptual similarity on
design ideation. The experimental conditions
similarity include a control condition and one treatment
condition. The experiment focuses on
The AI model for conceptual similarity identifying distinct patterns of the participant's
computes the degree of similarity between a set ideation in a human-AI collaboration where the
of words in the design task and a set of words AI partner contributes content based on
for each image in the image dataset. While the conceptual similarity. The experiment is a
previous CIP system used the object category within-subject design that compares
of the design task (e.g. bed) to measure the participants’ ideation outcome and process
conceptual similarity, the updated CIP used a while engaged in a design task with different
set of words in the design task statement (i.e. ideation stimuli: a control condition with
bed, senior, living, facility) to include the random inspirations (condition A), a treatment
context of the design object for measuring the condition with conceptually similar
conceptual similarity. For measuring inspirations.
conceptual similarity, we thus use the words in • Condition A (control condition):
the design task statement (i.e. bed, senior, randomly (sink)
living, facility) and the words in the design • Condition B (treatment condition):
features of each image in the image dataset. We conceptually similar (bed)
generate a pair-wise similarity score for each We recruited 50 university students (N=50)
word in set 1 (i.e. words in the design task for the participants: each participant engaged in
2 conditions: a control condition (condition A) updated the CIP system and study design based
and a treatment condition (condition B). We use on what we learned from the exploratory study.
two design tasks (i.e. condition A: design a sink The current CIP system focuses on
for an accessible bathroom, condition B: design conceptually similar inspirations to the target
a bed for a senior living facility) that was used design and provides high fidelity images of
for the exploratory study. The data collection creative designs.
includes the video of the design sessions and
video of the retrospective protocol sessions, as 8. References
in the exploratory study. The data from this
study is still being collected.
[1] Y.-G. Cheong and R. M. Young,
Narrative generation for suspense:
7. Conclusion Modeling and evaluation, in Joint
International Conference on Interactive
This paper presents a co-creative design tool Digital Storytelling, 2008, pp. 144–155.
called Collaborative Ideation Partner (CIP) that [2] S. Colton, J. Goodwin, and T. Veale, Full-
supports the idea generation of new designs FACE Poetry Generation., in ICCC,
with stimuli that vary in similarity to the user’s 2012, pp. 95–102.
design in two dimensions: conceptual and [3] M. Cook and S. Colton, Ludus Ex
visual similarity. The AI models for measuring Machina: Building A 3D Game Designer
similarity in the CIP use deep learning models That Competes Alongside Humans., in
and cosine similarity to the user’s sketch and ICCC, 2014, pp. 54–62.
design task. The interactive experience allows [4] L. A. Gatys, A. S. Ecker, and M. Bethge,
the user to seek inspiration as needed. To study A neural algorithm of artistic style, arXiv
the impact of varying levels of visual and preprint arXiv:1508.06576, 2015.
conceptual similar stimuli, we performed an [5] F. Rashel and R. Manurung, Pemuisi: a
exploratory study with four conditions for the constraint satisfaction-based generator of
AI inspiration: random, high visual and topical Indonesian poetry., in ICCC,
conceptual similarity, high conceptual 2014, pp. 82–90.
similarity with low visual similarity, and high [6] T. Veale, Coming good and breaking bad:
visual similarity with low conceptual similarity. Generating transformative character arcs
We developed an approach for measuring for use in compelling stories, 2014.
ideation that has two components: an outcome- [7] T. Veale and Y. Hao, A fluid knowledge
based approach and a process-based approach. representation for understanding and
The outcome-based approach adapts existing generating creative metaphors, in
quantitative metrics for ideation: novelty, Proceedings of the 22nd International
variety, quality, and quantity of ideas expressed Conference on Computational Linguistics
in the outcome. The process-based approach (Coling 2008), 2008, pp. 945–952.
uses existing cognitive models of design, [8] G. A. Wiggins, Searching for
including the FBS ontology and the P-S index, computational creativity, New
to code and analyze the verbal protocol of the Generation Computing, vol. 24, no. 3, pp.
designers. These measures can be used in 209–222, 2006.
evaluating the impact of AI contributions in [9] G. A. Wiggins, A preliminary framework
other co-creative systems that support design for description, analysis and comparison
creativity. We applied these measures in the of creative systems, Knowledge-Based
four conditions in the CIP to demonstrate how Systems, vol. 19, no. 7, pp. 449–458,
to operationalize our approach for measuring 2006.
ideation in a co-creative system. [10] J. A. Biles, Interactive GenJam:
From the exploratory study, we learned that Integrating real-time performance with a
the quality of dataset is important in AI-based genetic algorithm, 1998.
creativity for the impact on designer’s [11] K. Compton and M. Mateas, Casual
creativity and inspirations based on conceptual Creators., in ICCC, 2015, pp. 228–235.
similarity to the target design leads to more [12] N. Davis, E. Y.-L. Do, P. Gupta, and S.
novel ideation than inspirations based on visual Gupta, Computing harmony with
similarity to sketches drawn by a designer. We PerLogicArt: perceptual logic inspired
collaborative art, in Proceedings of the generation, Design studies, vol. 24, no. 4,
8th ACM conference on Creativity and pp. 341–355, 2003.
cognition, 2011, pp. 185–194. [25] J. Chan et al., Semantically far
[13] N. M. Davis, Human-computer co- inspirations considered harmful?
creativity: Blending human and accounting for cognitive states in
computational creativity, 2013. collaborative ideation, in Proceedings of
[14] G. Hoffman and G. Weinberg, Gesture- the 2017 ACM SIGCHI Conference on
based human-robot jazz improvisation, in Creativity and Cognition, 2017, pp. 93–
2010 IEEE International Conference on 105.
Robotics and Automation, 2010, pp. 582– [26] L. Mamykina, L. Candy, and E.
587. Edmonds, Collaborative creativity,
[15] M. Jacob, A. Zook, and B. Magerko, Communications of the ACM, vol. 45, no.
Viewpoints AI: Procedurally 10, pp. 96–99, 2002.
Representing and Reasoning about [27] M. Jacob, G. Coisne, A. Gupta, I. Sysoev,
Gestures., 2013. G. G. Verma, and B. Magerko,
[16] T. Lubart, How can computers be partners Viewpoints ai, 2013.
in the creative process: classification and [28] N. Davis, C.-Pi. Hsiao, K. Y. Singh, L. Li,
commentary on the special issue, S. Moningi, and B. Magerko, Drawing
International Journal of Human- apprentice: An enactive co-creative agent
Computer Studies, vol. 63, no. 4–5, pp. for artistic collaboration, in Proceedings
365–369, 2005. of the 2015 ACM SIGCHI Conference on
[17] B. Magerko, C. Fiesler, A. Baumer, and Creativity and Cognition, 2015, pp. 185–
D. Fuller, Bottoms up: improvisational 186.
micro-agents, in Proceedings of the [29] P. Lucas and C. Martinho, Stay Awhile
Intelligent Narrative Technologies III and Listen to 3Buddy, a Co-creative
Workshop, 2010, pp. 1–8. Level Design Support Tool., in ICCC,
[18] J. Jongejan, H. Rowley, T. Kawashima, J. 2017, pp. 205–212.
Kim, and N. Fox-Gieg, The quick, draw!- [30] G. N. Yannakakis, A. Liapis, and C.
ai experiment, Mount View, CA, Alexopoulos, Mixed-initiative co-
accessed Feb, vol. 17, p. 2018, 2016. creativity, 2014.
[19] Ö. Akin, Necessary conditions for design [31] P. Karimi, K. Grace, M. L. Maher, and N.
expertise and creativity, Design Studies, Davis, Evaluating creativity in
vol. 11, no. 2, pp. 107–113, 1990. computational co-creative systems, arXiv
[20] C. J. Atman, J. R. Chimka, K. M. Bursic, preprint arXiv:1807.09886, 2018.
and H. L. Nachtmann, A comparison of [32] A. Newell and H. A. Simon, Human
freshman and senior engineering design problem solving, vol. 104. Prentice-Hall
processes, Design studies, vol. 20, no. 2, Englewood Cliffs, NJ, 1972.
pp. 131–152, 1999. [33] V. Goel and P. Pirolli, The structure of
[21] D. R. Brophy, Comparing the attributes, design problem spaces, Cognitive
activities, and performance of divergent, science, vol. 16, no. 3, pp. 395–429, 1992.
convergent, and combination thinkers, [34] T. Dorta, Design flow and ideation,
Creativity research journal, vol. 13, no. 3– International Journal of Architectural
4, pp. 439–455, 2001. Computing, vol. 6, no. 3, pp. 299–316,
[22] N. Cross, Design cognition: Results from 2008.
protocol and other empirical studies of [35] W. Visser, The cognitive artifacts of
design activity, in Design knowing and designing. CRC Press, 2006.
learning: Cognition in design education, [36] T. Dorta, E. Perez, and A. Lesage, The
Elsevier, 2001, pp. 79–103. ideation gap:: hybrid tools, design flow
[23] S. R. Daly, S. Yilmaz, J. L. Christian, C. and practice, Design studies, vol. 29, no.
M. Seifert, and R. Gonzalez, Design 2, pp. 121–141, 2008.
heuristics in engineering concept [37] N. V. Hernandez, J. J. Shah, and S. M.
generation, 2012. Smith, Understanding design ideation
[24] Y.-C. Liu, A. Chakrabarti, and T. Bligh, mechanisms through multilevel aligned
Towards an ‘ideal’approach for concept empirical studies, Design Studies, vol. 31,
no. 4, pp. 382–410, 2010.
[38] J. S. Gero and M. L. Maher, Mutation and 25th International Conference on
analogy to support creativity in computer- Intelligent User Interfaces, 2020, pp.
aided design, 1991. 221–230.
[39] D. P. Moreno et al., Fundamental studies [51] D. H. Douglas and T. K. Peucker,
in Design-by-Analogy: A focus on Algorithms for the reduction of the
domain-knowledge experts and number of points required to represent a
applications to transactional design digitized line or its caricature,
problems, Design Studies, vol. 35, no. 3, Cartographica: the international journal
pp. 232–272, 2014. for geographic information and
[40] B. A. Nelson, J. O. Wilson, D. Rosen, and geovisualization, vol. 10, no. 2, pp. 112–
J. Yen, Refined metrics for measuring 122, 1973.
ideation effectiveness, Design Studies, [52] U. Ramer, An iterative procedure for the
vol. 30, no. 6, pp. 737–743, 2009. polygonal approximation of plane curves,
[41] J. J. Shah, S. M. Smith, and N. Vargas- Computer graphics and image processing,
Hernandez, Metrics for measuring vol. 1, no. 3, pp. 244–256, 1972.
ideation effectiveness, Design studies, [53] P. Karimi, M. L. Maher, N. Davis, and K.
vol. 24, no. 2, pp. 111–134, 2003. Grace, Deep Learning in a Computational
[42] J. S. Gero, Design prototypes: a Model for Conceptual Shifts in a Co-
knowledge representation schema for Creative Design System, arXiv preprint
design, AI magazine, vol. 11, no. 4, pp. arXiv:1906.10188, 2019.
26–26, 1990. [54] T. Mikolov, K. Chen, G. Corrado, and J.
[43] J. S. Gero and U. Kannengiesser, The Dean, Efficient estimation of word
situated function–behaviour–structure representations in vector space, arXiv
framework, Design studies, vol. 25, no. 4, preprint arXiv:1301.3781, 2013.
pp. 373–391, 2004. [55] R. Rehurek and P. Sojka, Software
[44] J. S. Gero, Generalizing design cognition framework for topic modelling with large
research, DTRS, vol. 8, pp. 187–198, corpora, 2010.
2010. [56] T. M. Amabile, Social psychology of
[45] K. Dorst and N. Cross, Creativity in the creativity: A consensual assessment
design process: co-evolution of problem– technique., Journal of personality and
solution, Design studies, vol. 22, no. 5, social psychology, vol. 43, no. 5, p. 997,
pp. 425–437, 2001. 1982.
[46] M. Maher and H.-H. Tang, Co-evolution
as a computational and cognitive model
of design, Research in Engineering
design, vol. 14, no. 1, pp. 47–64, 2003.
[47] J. S. Gero, H. Jiang, and C. B. Williams,
Design cognition differences when using
unstructured, partially structured, and
structured concept generation creativity
techniques, International Journal of
Design Creativity and Innovation, vol. 1,
no. 4, pp. 196–214, 2013.
[48] H. Jiang, J. S. Gero, and C. Yen,
Exploring designing styles using a
problem–solution index, 2014.
[49] N. Davis, S. Siddiqui, P. Karimi, M. L.
Maher, and K. Grace, Creative Sketching
Partner: A Co-Creative Sketching Tool to
Inspire Design Creativity., in ICCC,
2019, pp. 358–359.
[50] P. Karimi, J. Rezwana, S. Siddiqui, M. L.
Maher, and N. Dehbozorgi, Creative
sketching partner: an analysis of human-
AI co-creativity, in Proceedings of the