Studying the Impact of AI-based Inspiration on Human Ideation in a Co-Creative Design System Jingoog Kima, Mary Lou Mahera and Safat Siddiquia a University of North Carolina at Charlotte, 9201 University City Blvd, Charlotte, NC 28223, USA Abstract Co-creative systems in design enable users to collaborate with an AI agent on open-ended creative tasks in the design process. This paper describes a co-creative system that supports design creativity by encouraging the exploration of design solutions in the initial idea generation process. The Collaborative Ideation Partner (CIP) is a co-creative design system that provides inspirational sketches based on the visual and conceptual similarity to sketches drawn by a designer. To evaluate the effect of CIP on design ideation, we conducted an exploratory study that measures ideation in a co-creative system. To measure the ideation, we developed a way of measuring ideation in a co-creative system including an outcome and a process approach. From the exploratory study, we learned that the image quality in the dataset is important in AI-based creativity and inspirations based on conceptual similarity to the target design have more impact on ideation than inspirations based on visual similarity to sketches drawn by a designer. We present the architecture of the CIP system and a study design based on what we learned from the exploratory study. Keywords 1 Co-Creativity, Co-Creative System, Ideation, Collaboration 1. Introduction We present a co-creative sketching AI partner, the Collaborative Ideation Partner (CIP), that provides inspirational sketches Computational co-creative systems are a based on the visual and conceptual similarity to growing research area in computational sketches drawn by a designer. To select an creativity. While some research on inspiring sketch, the AI model of CIP computes computational creativity has a focus on the visual similarity of images in a data set generative creativity [1]–[9], co-creative based on the vector representations of visual systems focus on how systems that implement features of the sketches and the conceptual generative creativity can work with humans on similarity based on the category names of the a creative task [10]–[17]. Co-creative systems sketches using two pre-trained word2vec have enormous potential to enhance human models. The turn-taking interaction between creativity since they can be applied to a variety the user and the AI partner is designed to of domains associated with creativity and facilitate communication for design ideation. encourage designers’ creative thinking. The CIP was developed to support an Understanding the effect of co-creative systems exploratory study that evaluates the effect of an in the ideation process can aid in the design of AI model for visual and conceptual similarity the generative AI models in co-creative systems on design ideation in a co-creative design tool. and the evaluation of the impact of co-creative systems on human creativity. J oint Proceedings of the ACM IUI 2021 Workshops, April 13-17, 2021, College Station, USA EMAIL: jkim168@uncc.edu (A. 1); M.Maher@uncc.edu (A. 2); ssiddiq6@uncc.edu (A. 3) ORCID: 0000-0003-3597-6153 (A. 1) Copyright ©2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) Table 1 Comparison of original design and current design of CIP Original Design of CIP Current Design of CIP Interface Stimuli Visual and Conceptual Similarity Conceptual Similarity Inspiring Low fidelity sketches of a general object high fidelity images of a creative design images Modes of 4 modes: random, similar, conceptually 2 modes: random inspiration, inspiration similar and visually different, visually conceptually similar similar and conceptually different Dataset 3450 Sketches (QuickDraw [18]) 100 images To evaluate the impact of co-creative CIP system and study design based on what we systems in design, we measure design ideation learned from the exploratory study. Table 1 in a co-creative system. Ideation, an idea shows the comparison between the CIP system generation process for conceptualizing a design used for the exploratory study and the current solution, is a key step that can lead a designer CIP system based on what we learned from the to an innovative design solution in the design exploratory study. The current CIP system process. Idea generation is a process that allows focuses on conceptually similar inspirations to designers to explore many different areas of the the target design and provides high fidelity design solution space [19]–[24]. Ideation has images of creative designs. This study aims to been studied in human design tasks and identify the effect of AI inspiration on design collaborative tasks in which all participants are ideation through a way of measuring ideation in human. Collaborative ideation can help people a co-creative system. generate more creative ideas by exposing them to ideas different from their own [25]. This 2. Computational co-creative paper has a focus on evaluating how a co- creative agent influences the ideation process in systems a human-AI collaboration. In this paper, we describe an exploratory Computational co-creative systems are one study measuring ideation when co-creating of the growing fields in computational with the CIP and what we learned from the creativity that involves human users exploratory study. To measure ideation in a co- collaborating with an AI agent to make creative creative system, we employ two approaches: an artifacts. The distinction of co-creativity from outcome-based approach that focuses on the computational creativity is that co-creativity is end product of the design, and a process-based a collaboration in which multiple parties approach that focuses on thought processes contribute to the creative process in a blended during the design. From the exploratory study, manner [26]. Co-creative systems have been we learned that the quality of the images in the applied in different creative domains such as dataset is important in AI-based creativity for art, music [14], dance [15,27], drawing [28], the impact on designer’s creativity and and game design [39,56]. inspirations based on conceptual similarity to Evaluating co-creative systems is still an the target design has more impact on ideation open research question and there is no standard than inspirations based on visual similarity to metric for measuring computational co- sketches drawn by a designer. We updated the creativity [31]. Figure 1: Evaluating the creative ideation partner The research on co-creative systems shows to explore different areas of the design solution various approaches to evaluate co-creative space [23,32]. A design process is an evolution systems and computational co-creativity. Some of different kinds of representations [33]. In a researches focus on evaluating the interactive design process, designers externalize and experience [29,14,30,31] and others focus on visualize their design intentions and the effectiveness of the system to produce or communicate with visualizations to interact generate a creative outcome [39,56]. Karimi et with their internal mental images [34]. During al. [31] presented a framework for evaluating ideation, designers commonly use freehand creativity in computational co-creative systems. sketches and rough physical models as a tool for This framework responds to four questions that constructing external representations as serve to characterize the many and varied cognitive artifacts of design [35]. Making approaches to evaluating computational co- sketches and physical models is an interaction, creativity: who is evaluating the creativity, a conversation [36]. In the ideation stage, what is being evaluated, when does evaluation designers frame problems producing new occur, and how the evaluation is performed. discoveries through the conversation. The The framework enables comparisons of graphical and physical representations as evaluation focus and methods across existing cognitive artifacts are essential in the ideation co-creative systems. Using this framework, we process. have shown that the evaluations of the existing Many ideation methods have been co-creative systems described in this section developed to support designers in generating respond to “what is being evaluated” with a innovative design solutions. Ideation methods focus on evaluating the interactive experience provide a normative procedure on how to and the final product. In this paper, we respond overcome certain blocks to creativity [37]. to “what is being evaluated” and “how is the Analogy is an ideation method and we focus on evaluation performed” by evaluating the analogy to develop a co-creative design tool. ideation process using FBS and Problem- Analogical reasoning is an inference method in Solution index, and the metrics for evaluating design cognition to develop a design leading to the novelty, variety, quality, and quantity of unexpected discoveries [38]. Design-by- ideas in the creative outcome, as shown in Analogy (DbA) is a design tool that provides Figure 1. Section 3 describes how we define inspiration for innovative design solutions. and measure ideation in more detail. Inspirations in Design-by-Analogy (DbA) are achieved by transferring a design problem 3. Defining and measuring design (source) to a solution (target) in another domain [39]. The association between a source design ideation and a target design can be based on semantic (conceptual) characteristics or visual Ideation is a creative process where (structural) representations. The semantic and designers generate, develop, and communicate visual stimuli thus can be a basis for developing new ideas. Ideation in design can lead to computational tools that support design innovative design solutions through generating ideation. The Collaborative Ideation Partner diverse concepts [19]–[22], [24]. The goal of (CIP), a co-creative design system we present design is to develop useful and innovative in this paper, uses visual and conceptual solutions and design ideation allows designers similarity metrics as key factors for classes are augmented by requirements (R) that collaborative ideation using design by analogy. come from outside the designer and description Evaluation of ideation can be classified into (D) that is the document of any aspect of two groups: outcome-based approaches and designing. In this ontological view, the goal of process-based approaches [40]. Outcome-based designing is to transform a set of requirements approaches focus on evaluating the ideation and functions into a set of design descriptions. process based on the designs, or outcomes, and The transformation of one design issue into the characteristics of ideas generated. Process- another is defined as a design process [44]. based approaches focus on evaluating idea The design process can be viewed as generation processes based on the cognitive interactions between two notional design processes inherent to creative thought. Process- spaces: problem space and solution space based approaches collect data via a protocol [45,46]. The Problem-Solution (P-S) index study and analysis using ideation cognitive [47,48] is a measurement capturing the meta- models. Outcome-based approaches have level structures of design cognition in terms of become more prevalent than process-based problem-focused and solution-focused design approaches due to the inherent complexity and issues. This measurement uses an integration of difficulties in using process-based approaches the FBS ontologically-based coding scheme [41]. There have been several metrics used to with a Problem-Solution (P-S) division [47,48]. evaluate the performance of idea generation In the P-S division, design issues of R, F, and techniques such as fluency and novelty that Be are mapped to problem space and design cognitive psychologists consider as the primary issues of Bs, and S are mapped to solution space measures of idea generation. Shah et al. [41] [47]. A design session with a P-S index larger introduced four types of outcome-based metrics than 1 as one with a problem-focused designing for measuring ideation effectiveness that are style, and a session with a P-S index value less commonly used for evaluating idea generation than or equal to 1 as one with a solution-focused in design: novelty, variety, quality, and quantity style. The P-S index can be used to compare of designs. Novelty is a measure of how design cognition while using different unusual or unexpected an idea is as compared creativity techniques for concept generation in to other ideas. Variety is a measure of the collaborative design settings. explored solution space during the idea generation process. The generation of similar 4. The collaborative ideation ideas indicates low variety and hence, less probability of finding better ideas in other areas partner (CIP) of the solution space. Quality is a subjective measure of the feasibility of an idea and how We developed the Collaborative Ideation close it comes to meet the design specifications. Partner (CIP) as a co-creative design system Quantity is the total number of ideas generated, which builds on previous works [49,50] that generating more ideas increases the possibility interprets sketches drawn by a user and of better ideas. These metrics enable a provides inspirational sketches based on visual comparison of a designer’s exploration of a similarity and conceptual similarity. We design space using different ideation methods. developed the CIP to explore evidence for the Process-based approaches evaluate idea hypothesis that: AI models for contributions to generation based on the cognitive processes via a creative product based on a measure of visual a protocol analysis and cognitive models. The and conceptual similarity produce different Function-Behavior-Structure (FBS) ontology ideation processes and outcomes than the [42,43] is a design ontology that describes random condition. designed things, or artifacts, irrespective of the The user interface of CIP is shown in Figure specific discipline of designing. The function 2. There are two main spaces in the CIP (F) of a designed object is defined as its interface: the drawing space (pink area) and the teleology; the behavior (B) of that object is inspiring sketch space (purple area). The either derived (Bs) or expected (Be) from the drawing space consists of a design task structure, where structure (S) represents the statement, undo button, clear button, and user’s components of an object and their canvas. compositional relationships. These ontological Figure 2: User interface of collaborative ideation partner The design task statement in the drawing during an online game where players were space includes the object to be designed as well asked to draw a particular object within 20 as a context to further specify the objects’ use seconds. The dataset includes 345 categories and environment. The user can draw a sketch in with more than 50 million labeled sketches, the drawing space and edit the sketch using the where sketches are the array of the x and y undo and clear button. The inspiring sketch coordinates of the strokes. The system uses the space includes an “inspire me” button, the name simplified drawing json files that use Ramer– of the inspiring object, and a space for Douglas–Peucker algorithm [51,52] to simplify presenting the AI partner’s sketch. When the the strokes, and position and scale the sketches user clicks the “inspire me” button after into a 256 X 256 region. The stroke data sketching their design concept, the AI partner associated with these sketches are used to provides an inspiring sketch based on visual calculate the visual similarity and the and conceptual similarity. An ideation process corresponding category names are used to using CIP involves turn-taking measure the conceptual similarity. communications between the user and the AI partner. Another part of the CIP interface in 4.2. AI models for visual and addition to the two main spaces is the top area (grey area) including a hamburger menu and an conceptual similarity introductory statement. The hamburger menu on the top-left corner of the interface includes The CIP has 2 distinct components for four design tasks (i.e. sink, bed, table, chair) measuring similarity between the user’s sketch and allows the experiment facilitator to select and the sketches in the dataset: one component one of the design tasks. Each design task is for calculating visual similarity and another associated with different categories of ideation component for calculating conceptual stimuli. similarity. Figure 3 shows how the CIP system identifies an inspiring sketch: the visual 4.1. Dataset similarity is based on the vector representations of visual features of the sketches and the conceptual similarity based on the category For the source of inspiring sketches, the names of the sketches using two pre-trained original CIP uses a public benchmark dataset word2vec models. called QuickDraw! [18], which was created Figure 3: AI-Based Co-Creativity in the CIP For the visual similarity component, we component of CIP that considers the conceptual followed the precedent for using neural similarities of the sketches. network models in computational creativity For the conceptual similarity component, we described in [50,53] and trained a model with 3 considered sketch category names in the convolutional layers, 2 LSTM layers, and a QuickDraw dataset as the concepts of the softmax output layer on the QuickDraw dataset. sketches that contain 345 unique categories. We This model provides a latent space used two pre-trained word2vec models, Google representation for measuring the distance, or News [54] and Wikipedia [55], and calculated similarity, between 2 sketches. For all the cosine similarities for measuring the conceptual sketches in the dataset, we collected the last similarities between the object categories of the LSTM layer of the trained model and used that design tasks and the categories of inspiring as the vector representations of visual features sketches from the dataset. For each category of of the sketches. We used the K-means the design tasks, we generated two sorted lists algorithm to identify 10 clusters of sketches and of conceptually similar category names, one for randomly selected one sketch of each cluster as each word2vec model, and then used human a typical sketch for that cluster of sketches. judgement to compare the sorted lists and select Thus, we converted the QuickDraw dataset of the top 15 common conceptually similar 50 million sketches into 3450 sketches (345 category names that appear in both lists. This categories, each has 10 sketches). To prepare final step of using human judgement improved the user’s sketch for comparison with the the alignment between the conceptual sketches in the dataset, the CIP collects the user similarities of AI models and human sketches as an array of x and y coordinates of perception. The conceptual similarity strokes and simplifies the strokes using Ramer– component of CIP uses the common list of Douglas–Peucker algorithm [51,52]. It also category names for sorting the sketches based positions and scales the user’s sketch into the on the conceptual similarities. 256 X 256 region to match the sketch format with the input dataset of the trained model. The 4.3. AI-based inspiration in CIP CIP takes the last LSTM layer of the trained model as the vector representation of visual features of the user’s sketch, and calculates the To support an exploratory study that Euclidean distance to measure visual similarity measures ideation when co-creating with CIP, between the user’s sketch and 3450 sketches of the interaction with CIP has four distinct modes of inspiration that vary the visual and Quickdraw dataset. The visual similarity component of the CIP prepares a sorted list of conceptual similarity. Each of the four modes visually similar sketches to generate the final appears as a design task (i.e. sink, bed, table, sequence of sketches in the conceptual chair) in the CIP interface. • Inspire with a random sketch (sink): • Condition B (treatment condition): The CIP selects a sketch randomly from the visually and conceptually similar (bed) sketch dataset to be displayed on the AI • Condition C (treatment condition): partner’s canvas. conceptually similar and visually different • Inspire with a visually and (table) conceptually similar sketch (bed): The CIP • Condition D (treatment condition): selects a sketch from a set of sketches where visually similar and conceptually different each one is similar visually and conceptually (chair) to the user’s sketch (e.g. user sketch - a bed, AI sketch - a similar shape of bed to the During the study, for each participant and user’s sketch). for each condition we collected video protocol • Inspire with a conceptually similar and data during the design session and a visually different sketch (table): The CIP retrospective protocol after the design session. selects a sketch from a set of sketches where The protocol including the informed consent each one is conceptually similar but visually document has been reviewed and approved by different to the user’s sketch (e.g. user our IRB and we obtained informed consent sketch - a square table, AI sketch - a round from all participants to conduct the experiment. table). We recruited 12 students from human-centered • Inspire with a visually similar and design courses for the participants: each conceptually different sketch (chair): The participant engaged in 2 conditions: a control CIP selects a sketch from a set of sketches condition and one of the treatment conditions, where each one is visually similar but with 4 participants for each of the 3 groups of conceptually different to the user’s sketch within-subject design (i.e. A&B, A&C, A&D). (e.g. user sketch - a circular chair back, AI The experiment is a mixed design with N=4 and sketch - a face). a total of 12 participants. The task is an open-end design task in which 5. Exploratory study: measuring participants were asked to design an object in a given context through sketching. To reduce the ideation when co-creating with learning effect, different objects for the design the CIP task were used for each condition: a sink for a accessible bathroom (condition A), a bed for a The goal of the exploratory study is to senior living facility (condition B), a table for a evaluate the effect of AI inspiration on ideation tinkering studio, a collaborative space for through an analysis of the correlation between designing, making, building, crafting, etc. conceptual and visual similarity with (condition C), a chair for a gaming computer characteristics of ideation. To measure ideation desk (condition D). The participants used a when co-creating with the collaborative laptop and interacted with the CIP interface ideation partner, we applied both evaluation using a mouse to draw a sketch while methods of ideation: an outcome-based performing the design task. approach (i.e. novelty, variety, quality, The procedure consists of a training session, quantity) and a process-based approach (i.e. P- two design task sessions, and two retrospective S index). protocol sessions. In the training session, the participants are given an introduction to the features of the CIP interface and how they work 5.1. Study design to enable the AI partner to provide inspiration during their design task. After the training The type of study is a mixed design of session, the participants perform two design between-subject and within-subject design with tasks in a control condition and a treatment four conditions. There are 3 groups of within- condition. The study used a counterbalanced subject design (i.e. A&B, A&C, A&D) in this order for the two design tasks. The participants study and each group has a control condition have no time limits to complete the design task. (i.e. condition A) and one of 3 treatment The participants were given as much time as conditions (i.e. condition B, C, D). needed to perform the design task until they • Condition A (control condition): were satisfied with their design. The randomly (sink) participants are free to click the “inspire me” don’t have any inspiration with the pictures.” button as many times as they would like to get This case shows an example that participants do inspiration from the system. However, the not have many ideas from random inspirations. participants were told to have at least 3 As shown in Figure 4b, P4 drew a basic bed inspirational sketches (i.e. clicking the “inspire and a pillow before getting the first inspiration me” button at least 3 times during a design then requested inspiration from the AI partner. session), a minimum number of inspirations, The first inspiring sketch was a chair and P4 from the system. The facilitator is present added a stool, table, and a stair next to the bed. during the design task but does not interfere in After that, P4 had two more inspirations, bed the design process. Once the participants finish and couch, and added bed guard around the bed. the two design task sessions, the participants P4 described that the bed guard idea came from are asked to explain what they were thinking the armrest of the couch. P4 then had 2 more based on watching their design session inspiring sketches, couch and sleeping bag, and recording as time goes on, and how the AI's added a curtain. P4 mentioned that the curtain sketches inspired their design in the idea came from the enclosing feature of the retrospective protocol session. sleeping bag and couch. The next inspiring sketch is a table and P4 edited the foldable table 5.2. Observations of ideation with on the bed. After that, P4 had three more inspiring sketches and added a slide that helps CIP getting out of the bed easily. P4 described that the slide idea came from the shape of the tent. We observed the video stream data to see As shown in Figure 4c, P2 drew a rectangle how participants develop their design ideas for a table before getting the first inspiration communicating with the inspirations and the then tried to get an inspiration from the AI participants' responses to inspirations show partner. The first inspiring sketch was a golf different patterns of users on the use of CIP in club and P2 added table legs mimicking the an ideation process. Figure 4 shows typical shape of a golf club. P2 then had a fireplace examples of the process for the evolution of the sketch and added a large grid paper on the table. participant’s sketch using CIP in each P2 described that the grid paper idea came from condition. In an evolution of the participant’s the way the lines are drawn in the fireplace. sketch, participants in each condition start with After that, P2 had matches and added a table a basic shape of the target design then develop lamp. P2 then had two pool sketches and added the design with inspiration from the AI partner. a pencil cup. The last inspiring sketch is a wine Participants explored many inspiring sketches glass but P2 did not change the design with the in condition A but did not have many design wine glass. changes; while participants in conditions B, C, As shown in Figure 4d, P3 drew a basic and D developed their design in response to chair without any special function for the fewer inspiring sketches. context of gaming before getting the first As shown in Figure 4a, P1 drew a basic sink inspiration then requested inspiration from the with a handrail before getting the first AI partner. The first inspiring sketch was a inspiration then tried to get an inspiration from raccoon and P3 added an ear shape decoration the AI partner. P1 had 7 inspiring sketches but on the top of the chair and an eye shape headrest did not change anything for the design. P1 then getting an inspiration from the shape of the cleaned all the canvas then drew a new sketch raccoon sketch (i.e. ear, and eye). P3 described which is a sink with a motion sensor. P1 had 4 that “So, I saw the raccoon and I kind of liked inspiring sketches and did not change anything how its ears were. Cause I have seen things, again for the design. P1 cleaned the canvas and where people have really interesting chairs, drew a new sketch again applying the motion and I think people that game may usually want sensor idea again then had 2 inspiring sketches. more interesting chairs. So, I thought it'd be However, P1 finally finished the design without cool to have little ears at the top, and then make any changes. During the retrospective session, the mask kind of like, a pillow.” After that, P3 P1 mentioned he did not get ideas from the had the second inspiring sketch which is a inspiring sketches several times, for example “I power outlet. (a) The evolution of P1 design in condition A (randomly) (b) The evolution of P4 design in condition B (visually and conceptually similar) (c) The evolution of P2 design in condition C (conceptually similar and visually different) (d) The evolution of P3 design in condition D (visually similar and conceptually different) Figure 4: AI-Based Co-Creativity in the CIP P3 added a speaker on the ear decoration, tasks and verbalizing the ideation process buttons on the armrest to control sound during the retrospective protocol. We recorded volume/massage/lights, and power cord. P3 the entire design task sessions and retrospective described that “the power outlet really gave me sessions for each participant. The sketch data a lot of the inspiration. I thought... instead of collected from the recordings of design task just random ears, it could be like a speaker. sessions shows the progress of design and the Then, I thought all the little dots on the armrest final design visually for each design task could be buttons, to do things. If it's different session. The verbal data collected from the things, like sound volume, or could do a recordings of retrospective sessions records massage. The little line coming out of it, would how the participants came up with ideas be to plug it into the wall, so all the buttons collaborating with the AI partner and applied could work.” In this case, the idea came from the ideas to their design. the inspiring sketch was transferred to new functions of the chair while the idea came from 5.4. Data segmentation and the raccoon was transferred to the shape of the chair. P3 then had 5 more inspiring sketches coding (i.e. rain, hurricane, zigzag, and camouflage). P3 mentioned that they were inspired from the To analyze the verbal data collected from irregular lines of the sketches and added sound the retrospective sessions, the verbal data of all projecting lines next to the ear shape speaker retrospective protocol sessions (i.e. 12 sessions and a pillow on the seat. After that, P3 had nine of condition A, 4 sessions of condition B, 4 more inspiring sketches, but did not change sessions of condition C, and 4 sessions of anything for the chair design. P3 talked about a condition D) was transcribed. The transcripts decoration idea from the star shape of an were segmented based on the inspiring sketches aircraft carrier and snowflake but did not the participant clicked. A segment starts with an change the chair design. inspiring sketch and ends when the inspiration is clicked for the next sketch. To identify each idea in an inspiring sketch segment, we 5.3. Data collected segmented the inspiring segments again based on FBS ontology [42,43] as an idea segment, Two types of data were collected for since an inspiring sketch segment includes analyzing the study results: a set of sketches multiple ideas. An inspiring segment thus that participants produced during the design includes multiple idea segments. The idea segments were coded based on FBS ontology condition as low/medium/high quality, in two [42,43] as requirement (R), function (F), evaluation rounds. In the first-round of expected behavior (Be), behavior from evaluation, each judge evaluated the final structure (Bs), and structure (S). A R segment designs identifying some criteria for evaluating is an utterance that talks about the given the quality of ideas. Once the judges finished requirement in the statement of design task (e.g. the first-round of evaluation, they shared the accessible bathroom) or a new requirement the criteria they identified/used, not sharing the participant came up with for the design (e.g. if results of the evaluation, then made a consensus someone is not able to reach the height); a F for the criteria that will be used for the second- segment is an utterance that talks about a round evaluation. The criteria that the judges purpose or a function of the design object (e.g. agreed for evaluating the quality of ideas in this more accessible); a Be segment is an utterance study are the number of features, how that talks about an expected behaviors from the responsive the features are to the specific task, structure (e.g. water could automatically come how creative the design is. In the second-round out), a Bs segment is an utterance that talks evaluation, each judge evaluated the final about a behavior derived from the structure design again using the agreed criteria. Quantity (e.g. pressing on), a S segment is an utterance is the total number of ideas generated. For that talks about a component of the design measuring quantity in this study, the number of object (e.g. button). Two coders coded the idea ideas both new ideas and repeated ideas coded segments individually based on the coding as R/F/B/S is counted in a design. scheme above then came to consensus for the different coding results. 5.6. Measuring ideation: process- based approach 5.5. Measuring ideation: outcome-based approach For the process-based approach, we used the P-S index [48] to examine the design cognition For the outcome-based approach, we from a meta-level view (i.e., a single-value developed four metrics based on [41]: novelty, measurement). For the meta-level view, the P- variety, quality, and quantity of design. Novelty S index is calculated by computing the number is a measure of how unusual or unexpected an of the total occurrences of the design issues idea is as compared to other ideas. In this study, concerned with the problem space (i.e. R, F, Be) a novel idea is defined as a unique idea across and related to the solution space (i.e. Bs, S). A all design sessions in a condition. For design session with a P-S index larger than 1 as measuring novelty, we counted how many one with a problem-focused style, and a session novel ideas in the entire collection of ideas in a with a P-S index value less than or equal to 1 as design session (personal level of novelty) and a one with a solution-focused style. In addition to condition (condition level of novelty). We calculating the P-S index of each design removed the same ideas across all design session, we looked at the number of problems sessions in a condition then counted the number and solutions to identify a distinct difference of ideas. Variety is a measure of the explored between the conditions. solution space during the idea generation process. Each idea segment was coded whether 5.7. What we learned from it is a new idea or a repeated idea in a design session. For measuring variety in this study, exploratory study only the number of new ideas coded as R/F/B/S is counted in a design session while the metric With the data collected in the exploratory of quantity includes both new ideas and study, we compared outcome-based features repeated ideas. Quality is a subjective measure (i.e. novelty, variety, quality, quantity) and of the design. In this study, quality is measured process-based features (i.e. P-S index). Our using the Consensual Assessment Technique findings show that the AI-based stimuli (CAT) [56], a method in which a panel of expert produce different ideation outcomes and judges is asked to rate the creativity of projects. processes when compared to random stimuli. Two judges, researchers involved in this study, individually evaluated the final design in each Figure 5: Current CIP user interface Novel ideation, evidenced by an increase in participant’s sketch was more accurate at the the variety and quantity of ideas, is associated beginning of the design session, but was less with AI-based conceptually similar stimuli. The accurate as the participant’s sketch became findings from analysis of the P-S index show more complicated. Third, the CIP in condition that AI-based visual and conceptual similarity D (visually similar and conceptually different) is associated with a problem-focused designing often provides sketches that are not visually style that produces more solutions than we similar to the participant’s sketch since the found in the condition with random inspiring sketches are first selected to be inspirations. We found that participants in conceptually different, and that reduces the condition C (conceptually similar and visually potential for identifying sketches that are different stimuli) produced more functions than visually similar. in condition A (random). In our observations of the exploratory study, 6. Current CIP and study design we identified some issues on the sketch data set and AI-based visually similar stimuli. First, the quality of the sketch dataset is very important From the exploratory study we learned that to inspire participants to come up with new inspiration based on conceptual similarity has more impact on the novelty and variety of ideas ideas. The sketches in this dataset are not the result of a design process. The sketches in the than visual similarity and that the quality of the QuickDraw dataset are generated to represent dataset is important for the design ideation. For the basic shape of a given well known object. our current CIP and study design, we developed Based on the retrospective protocol data, a more comprehensive model for conceptual participant’s ideas mostly came up from similarity based on multiple features of the purposes, functions, features, and structures of design rather than only a categorical word, and the inspiring sketches, and the simple collected a dataset of designs as the basis for representation of objects in the QuickDraw inspiration rather than a dataset of sketches on dataset were not providing very rich inspiration. well known objects. Figure 5 shows the current Second, the complexity of participants’ CIP user interface providing an inspiring image sketches increased during the design session, of a creative design instead of a simple sketch. affecting the accuracy of the visual similarity The target design is a bed for a senior living facility and the inspiring image is a smart measure used to select an inspiring sketch. The AI model for visually similarity to the patient room. The smart patient room is the most conceptually similar design to the target statement) and each word in set 2 (i.e. words in design. The design of the smart patient room the design feature). A Wikipedia pre-trained includes many functions and objects associated word2vec model is used to calculate the with the context of a senior living facility such similarity between the two words using a pair- as reclining bed, bed table, magazine holder, wise comparison: a word from the design task trash can, digital screen for health care, and statement and a word from the design features wheels for mobility. of an image. We calculate the cosine similarity score for each pair of a design task statement 6.1. Dataset word and a design feature word and create a set of cosine similarity scores including all pairs of design task statement words and design task For the source of inspiring designs, we feature words for each image in the image collected a dataset of high fidelity images of dataset. As a conceptual similarity score creative designs. To create the new dataset, we between the target design and the image, we use selected 20 common categories from the the average score of cosine similarity scores for categories of QuickDraw dataset that are each image. For example, a design statement conceptually similar to the target designs of the includes 4 words (i.e. bed, senior, living, exploratory study (i.e. sink, bed, table, chair). facility) and an image includes 4 words of We then searched for images of 5 creative design features (e.g. comfort, massage, designs online for each category using combinational, chair). For measuring the keywords “creative”, “novel”, “unusual”, conceptual similarity between the target design “design” (e.g. creative sink, unusual bed). The and the image, we calculate each cosine dataset thus contains 20 categories with 100 similarity score for 16 pairs of words (4 words labeled images. Each image has three fields: id, x 4 words) then calculate the average cosine object name, and design feature. Id is the unique similarity. We create the conceptual similarity identifier that is assigned to each image. Object ranking based on the similarity score of each name is the name of the design that is image. The system selects from the most represented in the image (e.g. electric massage conceptually similar image in order when the bed, robotic advisor, smart sofa). Design user clicks the inspire button. feature is keywords that represent the design features and unique functionalities of the design (e.g. multi-functional, entertainment, massage, 6.3. Study design combinational, digital, tv). In our study design we focus on the impact 6.2. AI model for conceptual of the AI model for conceptual similarity on design ideation. The experimental conditions similarity include a control condition and one treatment condition. The experiment focuses on The AI model for conceptual similarity identifying distinct patterns of the participant's computes the degree of similarity between a set ideation in a human-AI collaboration where the of words in the design task and a set of words AI partner contributes content based on for each image in the image dataset. While the conceptual similarity. The experiment is a previous CIP system used the object category within-subject design that compares of the design task (e.g. bed) to measure the participants’ ideation outcome and process conceptual similarity, the updated CIP used a while engaged in a design task with different set of words in the design task statement (i.e. ideation stimuli: a control condition with bed, senior, living, facility) to include the random inspirations (condition A), a treatment context of the design object for measuring the condition with conceptually similar conceptual similarity. For measuring inspirations. conceptual similarity, we thus use the words in • Condition A (control condition): the design task statement (i.e. bed, senior, randomly (sink) living, facility) and the words in the design • Condition B (treatment condition): features of each image in the image dataset. We conceptually similar (bed) generate a pair-wise similarity score for each We recruited 50 university students (N=50) word in set 1 (i.e. words in the design task for the participants: each participant engaged in 2 conditions: a control condition (condition A) updated the CIP system and study design based and a treatment condition (condition B). We use on what we learned from the exploratory study. two design tasks (i.e. condition A: design a sink The current CIP system focuses on for an accessible bathroom, condition B: design conceptually similar inspirations to the target a bed for a senior living facility) that was used design and provides high fidelity images of for the exploratory study. The data collection creative designs. includes the video of the design sessions and video of the retrospective protocol sessions, as 8. References in the exploratory study. The data from this study is still being collected. [1] Y.-G. Cheong and R. M. Young, Narrative generation for suspense: 7. Conclusion Modeling and evaluation, in Joint International Conference on Interactive This paper presents a co-creative design tool Digital Storytelling, 2008, pp. 144–155. called Collaborative Ideation Partner (CIP) that [2] S. Colton, J. Goodwin, and T. Veale, Full- supports the idea generation of new designs FACE Poetry Generation., in ICCC, with stimuli that vary in similarity to the user’s 2012, pp. 95–102. design in two dimensions: conceptual and [3] M. Cook and S. Colton, Ludus Ex visual similarity. The AI models for measuring Machina: Building A 3D Game Designer similarity in the CIP use deep learning models That Competes Alongside Humans., in and cosine similarity to the user’s sketch and ICCC, 2014, pp. 54–62. design task. The interactive experience allows [4] L. A. Gatys, A. S. Ecker, and M. Bethge, the user to seek inspiration as needed. To study A neural algorithm of artistic style, arXiv the impact of varying levels of visual and preprint arXiv:1508.06576, 2015. conceptual similar stimuli, we performed an [5] F. Rashel and R. Manurung, Pemuisi: a exploratory study with four conditions for the constraint satisfaction-based generator of AI inspiration: random, high visual and topical Indonesian poetry., in ICCC, conceptual similarity, high conceptual 2014, pp. 82–90. similarity with low visual similarity, and high [6] T. Veale, Coming good and breaking bad: visual similarity with low conceptual similarity. Generating transformative character arcs We developed an approach for measuring for use in compelling stories, 2014. ideation that has two components: an outcome- [7] T. Veale and Y. Hao, A fluid knowledge based approach and a process-based approach. representation for understanding and The outcome-based approach adapts existing generating creative metaphors, in quantitative metrics for ideation: novelty, Proceedings of the 22nd International variety, quality, and quantity of ideas expressed Conference on Computational Linguistics in the outcome. The process-based approach (Coling 2008), 2008, pp. 945–952. uses existing cognitive models of design, [8] G. A. Wiggins, Searching for including the FBS ontology and the P-S index, computational creativity, New to code and analyze the verbal protocol of the Generation Computing, vol. 24, no. 3, pp. designers. These measures can be used in 209–222, 2006. evaluating the impact of AI contributions in [9] G. A. Wiggins, A preliminary framework other co-creative systems that support design for description, analysis and comparison creativity. We applied these measures in the of creative systems, Knowledge-Based four conditions in the CIP to demonstrate how Systems, vol. 19, no. 7, pp. 449–458, to operationalize our approach for measuring 2006. ideation in a co-creative system. [10] J. A. Biles, Interactive GenJam: From the exploratory study, we learned that Integrating real-time performance with a the quality of dataset is important in AI-based genetic algorithm, 1998. creativity for the impact on designer’s [11] K. Compton and M. Mateas, Casual creativity and inspirations based on conceptual Creators., in ICCC, 2015, pp. 228–235. similarity to the target design leads to more [12] N. Davis, E. Y.-L. Do, P. Gupta, and S. novel ideation than inspirations based on visual Gupta, Computing harmony with similarity to sketches drawn by a designer. We PerLogicArt: perceptual logic inspired collaborative art, in Proceedings of the generation, Design studies, vol. 24, no. 4, 8th ACM conference on Creativity and pp. 341–355, 2003. cognition, 2011, pp. 185–194. [25] J. Chan et al., Semantically far [13] N. M. Davis, Human-computer co- inspirations considered harmful? creativity: Blending human and accounting for cognitive states in computational creativity, 2013. collaborative ideation, in Proceedings of [14] G. Hoffman and G. Weinberg, Gesture- the 2017 ACM SIGCHI Conference on based human-robot jazz improvisation, in Creativity and Cognition, 2017, pp. 93– 2010 IEEE International Conference on 105. Robotics and Automation, 2010, pp. 582– [26] L. Mamykina, L. Candy, and E. 587. Edmonds, Collaborative creativity, [15] M. Jacob, A. Zook, and B. Magerko, Communications of the ACM, vol. 45, no. Viewpoints AI: Procedurally 10, pp. 96–99, 2002. Representing and Reasoning about [27] M. Jacob, G. Coisne, A. Gupta, I. Sysoev, Gestures., 2013. G. G. Verma, and B. Magerko, [16] T. Lubart, How can computers be partners Viewpoints ai, 2013. in the creative process: classification and [28] N. Davis, C.-Pi. Hsiao, K. Y. Singh, L. Li, commentary on the special issue, S. Moningi, and B. Magerko, Drawing International Journal of Human- apprentice: An enactive co-creative agent Computer Studies, vol. 63, no. 4–5, pp. for artistic collaboration, in Proceedings 365–369, 2005. of the 2015 ACM SIGCHI Conference on [17] B. Magerko, C. Fiesler, A. Baumer, and Creativity and Cognition, 2015, pp. 185– D. Fuller, Bottoms up: improvisational 186. micro-agents, in Proceedings of the [29] P. Lucas and C. Martinho, Stay Awhile Intelligent Narrative Technologies III and Listen to 3Buddy, a Co-creative Workshop, 2010, pp. 1–8. Level Design Support Tool., in ICCC, [18] J. Jongejan, H. Rowley, T. Kawashima, J. 2017, pp. 205–212. Kim, and N. Fox-Gieg, The quick, draw!- [30] G. N. Yannakakis, A. Liapis, and C. ai experiment, Mount View, CA, Alexopoulos, Mixed-initiative co- accessed Feb, vol. 17, p. 2018, 2016. creativity, 2014. [19] Ö. Akin, Necessary conditions for design [31] P. Karimi, K. Grace, M. L. Maher, and N. expertise and creativity, Design Studies, Davis, Evaluating creativity in vol. 11, no. 2, pp. 107–113, 1990. computational co-creative systems, arXiv [20] C. J. Atman, J. R. Chimka, K. M. Bursic, preprint arXiv:1807.09886, 2018. and H. L. Nachtmann, A comparison of [32] A. Newell and H. A. Simon, Human freshman and senior engineering design problem solving, vol. 104. Prentice-Hall processes, Design studies, vol. 20, no. 2, Englewood Cliffs, NJ, 1972. pp. 131–152, 1999. [33] V. Goel and P. Pirolli, The structure of [21] D. R. Brophy, Comparing the attributes, design problem spaces, Cognitive activities, and performance of divergent, science, vol. 16, no. 3, pp. 395–429, 1992. convergent, and combination thinkers, [34] T. Dorta, Design flow and ideation, Creativity research journal, vol. 13, no. 3– International Journal of Architectural 4, pp. 439–455, 2001. Computing, vol. 6, no. 3, pp. 299–316, [22] N. Cross, Design cognition: Results from 2008. protocol and other empirical studies of [35] W. Visser, The cognitive artifacts of design activity, in Design knowing and designing. CRC Press, 2006. learning: Cognition in design education, [36] T. Dorta, E. Perez, and A. Lesage, The Elsevier, 2001, pp. 79–103. ideation gap:: hybrid tools, design flow [23] S. R. Daly, S. Yilmaz, J. L. Christian, C. and practice, Design studies, vol. 29, no. M. Seifert, and R. Gonzalez, Design 2, pp. 121–141, 2008. heuristics in engineering concept [37] N. V. Hernandez, J. J. Shah, and S. M. generation, 2012. Smith, Understanding design ideation [24] Y.-C. Liu, A. Chakrabarti, and T. Bligh, mechanisms through multilevel aligned Towards an ‘ideal’approach for concept empirical studies, Design Studies, vol. 31, no. 4, pp. 382–410, 2010. [38] J. S. Gero and M. L. Maher, Mutation and 25th International Conference on analogy to support creativity in computer- Intelligent User Interfaces, 2020, pp. aided design, 1991. 221–230. [39] D. P. Moreno et al., Fundamental studies [51] D. H. Douglas and T. K. Peucker, in Design-by-Analogy: A focus on Algorithms for the reduction of the domain-knowledge experts and number of points required to represent a applications to transactional design digitized line or its caricature, problems, Design Studies, vol. 35, no. 3, Cartographica: the international journal pp. 232–272, 2014. for geographic information and [40] B. A. Nelson, J. O. Wilson, D. Rosen, and geovisualization, vol. 10, no. 2, pp. 112– J. Yen, Refined metrics for measuring 122, 1973. ideation effectiveness, Design Studies, [52] U. Ramer, An iterative procedure for the vol. 30, no. 6, pp. 737–743, 2009. polygonal approximation of plane curves, [41] J. J. Shah, S. M. Smith, and N. Vargas- Computer graphics and image processing, Hernandez, Metrics for measuring vol. 1, no. 3, pp. 244–256, 1972. ideation effectiveness, Design studies, [53] P. Karimi, M. L. Maher, N. Davis, and K. vol. 24, no. 2, pp. 111–134, 2003. Grace, Deep Learning in a Computational [42] J. S. Gero, Design prototypes: a Model for Conceptual Shifts in a Co- knowledge representation schema for Creative Design System, arXiv preprint design, AI magazine, vol. 11, no. 4, pp. arXiv:1906.10188, 2019. 26–26, 1990. [54] T. Mikolov, K. Chen, G. Corrado, and J. [43] J. S. Gero and U. Kannengiesser, The Dean, Efficient estimation of word situated function–behaviour–structure representations in vector space, arXiv framework, Design studies, vol. 25, no. 4, preprint arXiv:1301.3781, 2013. pp. 373–391, 2004. [55] R. Rehurek and P. Sojka, Software [44] J. S. Gero, Generalizing design cognition framework for topic modelling with large research, DTRS, vol. 8, pp. 187–198, corpora, 2010. 2010. [56] T. M. Amabile, Social psychology of [45] K. Dorst and N. Cross, Creativity in the creativity: A consensual assessment design process: co-evolution of problem– technique., Journal of personality and solution, Design studies, vol. 22, no. 5, social psychology, vol. 43, no. 5, p. 997, pp. 425–437, 2001. 1982. [46] M. Maher and H.-H. Tang, Co-evolution as a computational and cognitive model of design, Research in Engineering design, vol. 14, no. 1, pp. 47–64, 2003. [47] J. S. Gero, H. Jiang, and C. B. Williams, Design cognition differences when using unstructured, partially structured, and structured concept generation creativity techniques, International Journal of Design Creativity and Innovation, vol. 1, no. 4, pp. 196–214, 2013. [48] H. Jiang, J. S. Gero, and C. Yen, Exploring designing styles using a problem–solution index, 2014. [49] N. Davis, S. Siddiqui, P. Karimi, M. L. Maher, and K. Grace, Creative Sketching Partner: A Co-Creative Sketching Tool to Inspire Design Creativity., in ICCC, 2019, pp. 358–359. [50] P. Karimi, J. Rezwana, S. Siddiqui, M. L. Maher, and N. Dehbozorgi, Creative sketching partner: an analysis of human- AI co-creativity, in Proceedings of the