=Paper=
{{Paper
|id=Vol-2903/IUI21WS-HAIGEN-12
|storemode=property
|title=Creative PenPal: A Virtual Embodied Conversational AI Agent to Improve User Engagement and Collaborative Experience in Human-AI Co-Creative Design Ideation
|pdfUrl=https://ceur-ws.org/Vol-2903/IUI21WS-HAIGEN-12.pdf
|volume=Vol-2903
|authors=Jeba Rezwana,Mary Lou Maher,Nicholas Davis
|dblpUrl=https://dblp.org/rec/conf/iui/RezwanaMD21
}}
==Creative PenPal: A Virtual Embodied Conversational AI Agent to Improve User Engagement and Collaborative Experience in Human-AI Co-Creative Design Ideation==
https://ceur-ws.org/Vol-2903/IUI21WS-HAIGEN-12.pdf
Creative PenPal: A Virtual Embodied Conversational AI
Agent to Improve User Engagement and Collaborative
Experience in Human-AI Co-Creative Design Ideation
Jeba Rezwanaa , Mary Lou Mahera and Nicholas Davisa
a
University of North Carolina at Charlotte, 9201 University City Blvd, Charlotte, NC 28223, US
Abstract
In recent years, researchers have designed many co-creative systems that are very promising with a powerful AI, yet some
fail to engage the users due to the unimpressive quality of the collaboration and interaction. Most of the existing co-creative
systems use instructing interaction where users only communicate with the AI by providing instructions for contribution.
In this paper, we demonstrate the prototype of a co-creative system for design ideation, Creative PenPal that utilizes an
interaction model that includes human-AI conversing interaction using text and a virtual embodiment of the AI character. We
hypothesize that this interaction model will improve user engagement, user perception about the AI, and the collaborative
experience. We describe the study design to investigate the impact of this particular interaction model on user engagement
and the overall collaborative experience. By the time of the workshop, we will have the data and insights from the study.
Keywords
Human AI Co-creation, User Engagement, Virtual Embodied AI, Conversational AI
1. Introduction With the intention of investigating the trends in the
interaction design of the existing co-creative systems, we
AI agents are becoming a part of our everyday life, thanks utilized the archival website, ”Library of Mixed-Initiative
to artificial intelligence technologies. Human-AI co cre- Creative Interfaces” (LMICI), which archives 74 exist-
ativity involves a human and an AI collaborating on cre- ing co-creative systems [5]. Angie Spoto and Natalia
ative tasks as partners [1]. Rather than being perceived as Oleynik created this archive after a workshop on mixed-
a support tool, AI agents in co-creative systems should be initiative interfaces led by Deterding et al. in 2017 [6, 5].
regarded as a co-equal partner. This field has the poten- The archive provides the corresponding literature and
tial to transform how people perceive and interact with other relevant information for each of the systems. We
AI. A study showed that AI ability alone does not ensure analyzed the interaction designs of the co-creative sys-
a positive collaborative experience of users with the AI tems present in the archive. Apparently, most of the co-
[2]. In recent years, researchers have designed many creative systems use instructing interaction [7], where
co-creative systems with powerful AI ability, yet some- users provide instructions to the system using buttons,
times users fail to maintain their interest and engagement sliders, or text to communicate directly with the AI (other
while collaborating with the AI due to the quality of the than communicating through the creative product). How-
collaboration and interaction. The literature asserts that ever, using buttons and sliders, users can communicate
user engagement is associated with the way users interact with the AI in a very constrained and minimal way in
with a system [3]. Interaction design is often an untended most of the systems. Very few systems use text, voice,
topic in the co-creativity literature despite being a funda- or embodied communication for user to AI direct com-
mental property of co-creative systems. Bown asserted munication during a co-creation to provide information
that the success of a creative system’s collaborative role to the AI, give feedback to the AI, etc. For example, Im-
should be further investigated through interaction design age to Image [8] is a co-creative system that converts a
as interaction plays a key role in the creative process of line drawing of a particular object from the user into a
co-creative systems [4]. Therefore, as a young field, there photo-realistic image. The user interface has only one
are potential areas of interaction design to be explored button that users click to tell the AI to convert the draw-
for designing effective co-creative systems that engage ing. Other than the button, there is no way of commu-
users and provide a better collaborative experience. nicating with the AI to provide information, suggestion
or feedback. In a human collaboration, collaborators
Joint Proceedings of the ACM IUI 2021 Workshops, College Station, communicate to provide feedback and convey important
USA information to each other and is a major component of
Envelope-Open jrezwana@uncc.edu (J. Rezwana); m.maher@uncc.edu the mechanics of co-creation [9]. The literature about
(M. L. Maher); ndavis64@uncc.edu (N. Davis) human-AI co-creation says that embodied communica-
© 2021 Copyright for this paper by its authors. Use permitted under Creative
CEUR
Workshop
http://ceur-ws.org
ISSN 1613-0073
Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
tion improves coordination between the human and the
Proceedings
AI [10]. Additionally, literature asserts that a commu- buttons or using function keys, etc. In contrast, the con-
nication channel for conversation between co-creators versing interaction type is where users have a dialogue
other than communicating through the shared creative with a system. Users can speak via an interface or type in
product improves user engagement in a human creative questions or answers to which the system replies via text
collaboration [11]. These literatures led us to investi- or speech output [13]. Conversational agents have tran-
gate the impact of embodied communication from the AI sitioned into multiple industries with increased ability
and a conversation between the human and AI on user for user engagement in intelligent conversation.
engagement and collaborative experience in human-AI The literature asserts that embodied communication
co-creativity. Our research questions emerged from the aids synchronization and coordination in improvisational
issue that most existing co-creative systems use instruct- human-computer co-creativity [10]. Being able to con-
ing interaction type, which uses one-way communica- verse with each other shows an increased engagement
tion, human to AI. For this work, we will investigate level in a human creative collaboration [11]. A user’s
the impact of conversing interaction and AI embodiment confidence in an AI agent’s ability to perform tasks is
on user engagement, user perception about the AI and improved when imbuing the agent with embodiment and
the overall collaborative experience. The two research social behaviors compared to the agent solely depending
questions we have are- on conversation [14]. Bente et al. reported that em-
bodied telepresent communication improved both social
• How does AI embodiment and conversing inter- presence and interpersonal trust in remote collaboration
action influence user engagement? settings with a high level of nonverbal activity [15].
• How does AI embodiment and conversing interac- User engagement with virtual embodied conversational
tion influence user perception about the AI agent agents can be measured via user self-reports; by mon-
as the collaborative partner and the overall col- itoring the user’s responses, tracking the user’s body
laborative experience? postures, head movements and facial expressions dur-
ing the interaction, or by manually logging behavioral
For investigating the research questions, we have de- responses of user experience [16]. Carrol and Latulipe
veloped a prototype of a co-creative system named Cre- proposed a quantitative and psychometric survey, called
ative PenPal where the user and the AI collaborate on a Creativity Support Index (CSI), to assess a tool’s creativ-
design ideation task. Users can generate ideas for design- ity support by measuring six dimensions of creativity
ing a particular object by sketching on a canvas, and the via self reports: Exploration, Expressiveness, Immersion,
AI will also contribute to the design ideation by showing Enjoyment, Results Worth Effort and Collaboration [17].
different inspirational sketches. Creative PenPal utilizes
a conversing interaction for the communication between
the human and the AI. Additionally, a virtual embodied
character for the AI agent is utilized. For investigating 3. Interface
the research questions, we describe the study design in
Creative PenPal is an interactive prototype, created with
the paper. By the time of the workshop, we will have the
Javascript, which has all the interaction components ex-
data and insights from the study.
cept the back-end AI model. We have selected a collection
of sketches as the database for creating a seamless expe-
2. Related Work rience that mimics an actual implementation of the AI
model. The sketch generation is automated where the
Louie et al. identified that AI ability alone does not ensure
system selects sketches from the collection. We have two
a positive collaborative experience of users with the AI
versions of the Creative PenPal prototype to investigate
[2]. Bown asserted that the success of a creative system’s
and compare the user engagement and collaborative ex-
collaborative role should be further investigated in terms
perience between the two versions. The original version
of interaction design as interaction plays a key role in
uses a conversing interaction and a virtual embodied AI
the creative process of co-creative systems [4]. Later Yee-
(see Figure 1). The virtual embodied AI character, a pen-
King and d’Inverno argued for a stronger focus on the
cil, is shown in section A of Figure 1. We will address
user experience, suggesting a need for further integration
the AI character as PenPal in the rest of the paper. Sec-
of interaction design practice into human-AI co-creativity
tion B is where the conversation happens between the
research [12].
PenPal and the user via text and buttons. We can see
Interaction types are ways a user interacts with a prod-
the design task displayed in section C. Both the user and
uct or application [13]. Instructing interaction is where
the AI collaborate in a design ideation task where both
users issue instructions to a system. This can be done
collaborators generate ideas for the design of an object as
in many ways, including typing in commands, selecting
sketches. Users will design the specified object in the task
options from menus or on a multitouch screen, pressing
A B
C
D E
F G
Figure 1: Original Creative PenPal Interface with AI Embodiment and Conversing Interaction
by sketching on the canvas shown in section F. Users can its contribution, it will be seen as happy and when the
undo a stroke using the ”Undo Previous Sketch” button user does not like the contribution from the AI, it will
and start the design ideation over by using the ”Clear the be sad. The conversation is divided into five different
canvas” button. When users hit the ”Inspire me” button situational phases demonstrated in Figure 2. Each phase
shown in section B, the virtual AI character will show includes the embodied state of the AI and conversational
an inspirational sketch of a conceptually similar object, interaction between the user and the PenPal. The text
an object that have similar working mechanism or usage without a comment bubble represents the embodied state
as the design task object, on its canvas shown in sec- of the AI in Figure 2. The texts with comment bubbles
tion G. Previous work on co-creative design ideation [18] represent dialogues of the user and the AI, and the icon
showed that users were more inspired by conceptually indicates which dialogue belongs to whom. Different
similar objects than visually similar objects that share responses from the user initiate another phase, which is
structural similarity as the design task object. Users can shown using arrows in Figure 2. If the user can respond
also ask for visually similar objects or sketches of the with different options, “/” sign is used in the Figure.
design task object to get inspiration by saying they didn’t
like the conceptually similar object (described in the next 4.1. PenPal Introduction
section). Section E shows the name of the object located This phase will start when the user starts the design
in the PenPal generated sketch. The other version uses task. PenPal will introduce itself and ask the users if
an instructing interaction where users can instruct the they want to see an inspirational sketch from the AI by
AI using buttons without AI embodiment (Figure 2). We saying ”Hi! I am your Creative PenPal. Do you want
will use both of these two versions to compare the impact me to inspire you?”. Users can respond immediately by
of two different interaction designs on user engagement pressing the button ”Inspire me” or they can keep ideating
and collaborative experience with an AI. by sketching and respond later.
4. Interaction Model 4.2. PenPal Generating Sketch and
Collecting User Preferences
For the interaction model, we choose a conversing in-
teraction. The conversation with the virtual embodied When the user hits the button ”Inspire me” indicating the
AI is simple so that the user will be able to go deeper desire to see an inspirational sketch, the PenPal will move
into the ideation process without any interruption in the to the canvas and generate a sketch. The PenPal will ask
design flow. The embodied virtual agent will show some the user whether they liked the sketch or not. The user
affective characteristics, for example, when the user likes
Figure 2: Creative PenPal Interface with Instructing Interaction and without AI Embodiment
Figure 3: Conversation Model between PanPal and the User
can reply with the ”Yes” button or the ”No” button. This 4.3. User Liked PenPals Sketch
phase is for collecting user preferences.
When users select the ”Yes” button in response to Pen-
Pal’s question to determine whether the user liked the
sketch or not, it means the sketch inspired the user in
Figure 4: PenPal Collecting User Preference
their design ideation. The PenPal will arrive with a happy including 25 males and 25 females as participants. This
face and say, ”I am glad that you liked the sketch! Let me study will use a between-subject study where one group
know if you want to see another inspirational sketch as of participants will test the version with instructing in-
an idea”. If The user wants to see an inspiration again, teraction and without any embodied AI character. The
they will select the ”Inspire me” button. other group will test the version with the conversing
interaction and a virtual embodied AI agent. The study
4.4. User Did not Like PenPals Sketch will start with a short pre-study survey to collect some
When users click the ”No” button, indicating that Pen- demographic information about the participants, for ex-
Pal’s generated sketch did not inspire them, PenPal ar- ample, gender, age-range, drawing/ sketching skills, etc.
rives with a sad face and says, ”Sorry that I could not Then, the participant will carry out the design task using
inspire you!” (left side of Figure 4, the gree arrow indi- either one version of the Creative PenPal. The task for
cates transition). Then it suggests the user ask for specific this study is- “Ideate the design of a shopping cart for
types of objects as inspiration by saying, ”Let’s try to be the elderly within 20 minutes. You must include three
more specific about what you want me to inspire with” design inspirations from the AI in the design”. The whole
(Right side of Figure 4). The user can respond witree task will be screen recorded. After the task, the partici-
options, ”Design Task Objects” (as our design task object pants will fill out Creativity Support Index (CSI), which
is shopping cart, the button says ”Shopping Carts”), ”vi- is a well known psycho metric survey, for measuring six
sually similar objects”, or ”conceptually similar objects”. dimensions of creativity: Exploration, Expressiveness,
Visually similar objects have visual structural similarity Immersion, Enjoyment, Results Worth Effort and Collab-
as the design task object and conceptually similar objects oration [17] to evaluate user engagement, collaboration
have similar working mechanism or usage as the design and immersion. After that, a retrospective think-aloud
task object. When the user clicks any of these three will be conducted as the participants watch the screen-
buttons, the PenPal will generate a sketch accordingly. recording video of the task to understand the rationale
behind the user interaction process and user experience.
4.5. User Finished Sketching The study will end with a follow-up semi-structured in-
terview to determine in depth qualitative data about the
When the user finishes the design ideation sketching, user engagement and overall experience with the AI.
they let the virtual agent know by clicking the ”Finish
ideation” button. The virtual agent arrives and greets the
user for completing the design ideation task by saying, 6. Discussion
”Well done! You did a great job! ”.
In the young yet fast-growing field of human-AI co-
creativity, attention is needed to design human-centered
5. Study Protocol co-creative systems where users are engaged in a success-
ful collaborative experience. Interaction design where
The user experiment will take place virtually. We will users can communicate with the AI for providing user
use Google Meet to connect with the study participants. preference improves the collaborative experience and
The target sample size for the study is 50 participants user attitude towards the AI [2]. Conversing virtual
Figure 5: User Did not Like PenPal’s Sketch
agents have transitioned into services such as ecommerce, References
leading to an increased ability for user engagement. In a
conversing interaction, users can provide their feedback [1] N. Davis, Human-computer co-creativity: Blending
on the AI’s contribution, which provides more informa- human and computational creativity, in: Proceed-
tion to the AI about user preferences. Conversing inter- ings of the AAAI Conference on Artificial Intelli-
action also helps users perceive the AI as a partner rather gence and Interactive Digital Entertainment, vol-
than a tool. A user’s confidence in an AI agent’s ability to ume 9, 2013.
perform tasks is improved when imbuing the agent with [2] R. Louie, A. Coenen, C. Z. Huang, M. Terry, C. J.
embodiment and social behaviors compared to the agent Cai, Novice-ai music co-creation via ai-steering
solely depending on conversation [14]. The embodiment tools for deep generative models, in: Proceedings
improves the user perception of an AI agent in terms of of the 2020 CHI Conference on Human Factors in
a collaborative partner, an entity. Users also tend to trust Computing Systems, 2020, pp. 1–13.
an AI in terms of their ability when they can see their [3] A. Sutcliffe, Designing for user engagement: Aes-
presence. The embodiment also helps design affective AI thetic and attractive user interfaces, Synthesis lec-
where an AI’s feelings are visible in its expression or ges- tures on human-centered informatics 2 (2009) 1–55.
ture. As a young field and new research area, interaction [4] O. Bown, Player responses to a live algorithm: Con-
design is rarely discussed in the existing literature despite ceptualising computational creativity without re-
being a fundamental property of an adequate co-creative course to human comparisons?, in: ICCC, 2015, pp.
system. An adequate interaction model dramatically im- 126–133.
proves the quality of the collaboration and engages users. [5] Library of mixed-initiative creative interfaces,
Investigating the impact of conversing interaction and AI http://mici.codingconduct.cc/, ???? (Accessed on
embodiment for designing effective co-creative systems 05/31/2020).
that engage users is essential. [6] S. Deterding, J. Hook, R. Fiebrink, M. Gillies, J. Gow,
We develop the prototype of Creative PenPal as an M. Akten, G. Smith, A. Liapis, K. Compton, Mixed-
effort to explore the impact of a conversing embodied initiative creative interfaces, in: Proceedings of
co-creative AI agent on user engagement, user percep- the 2017 CHI Conference Extended Abstracts on
tion and overall collaborative experience. We describe Human Factors in Computing Systems, 2017, pp.
the study design that will provide insights for designing 628–635.
effective co-creative systems that engage users and im- [7] Y. Rogers, H. Sharp, J. Preece, Interaction design:
prove their collaborative experience with the AI agent. beyond human-computer interaction, John Wiley
With the insights and results from the study, we will & Sons, 2011.
improve the interaction design of Creative PenPal and [8] P. Isola, J.-Y. Zhu, T. Zhou, A. A. Efros, Image-
implement the AI model in the improved prototype. At to-image translation with conditional adversarial
the time of the workshop, we will have the data and in- networks, in: Proceedings of the IEEE conference
sights in our hands from the study, revealing the impact on computer vision and pattern recognition, 2017,
of the interaction model we used. During the workshop, pp. 1125–1134.
we will be able to demonstrate the results and insights. [9] C. Gutwin, S. Greenberg, M. Roseman, Workspace
awareness in real-time distributed groupware:
Framework, widgets, and evaluation, in: People
and Computers XI, Springer, 1996, pp. 281–298.
[10] G. Hoffman, G. Weinberg, Interactive improvisation
with a robotic marimba player, Autonomous Robots
31 (2011) 133–153.
[11] N. Bryan-Kinns, F. Hamilton, Identifying mutual
engagement, Behaviour & Information Technology
31 (2012) 101–125.
[12] M. Yee-King, M. d’Inverno, Experience driven de-
sign of creative systems (2016).
[13] J. Preece, H. Sharp, Y. Rogers, Interaction design:
beyond human-computer interaction, John Wiley
& Sons, 2015.
[14] K. Kim, L. Boelling, S. Haesler, J. Bailenson,
G. Bruder, G. F. Welch, Does a digital assistant
need a body? the influence of visual embodiment
and social behavior on the perception of intelligent
virtual agents in ar, in: 2018 IEEE International
Symposium on Mixed and Augmented Reality (IS-
MAR), IEEE, 2018, pp. 105–114.
[15] G. Bente, S. Rüggenberg, N. C. Krämer, Social pres-
ence and interpersonal trust in avatar-based, col-
laborative net-communications, in: Proceedings
of the Seventh Annual International Workshop on
Presence, 2004, pp. 54–61.
[16] C. Clavel, A. Cafaro, S. Campano, C. Pelachaud,
Fostering user engagement in face-to-face human-
agent interactions: a survey, in: Toward Robotic
Socially Believable Behaving Systems-Volume II,
Springer, 2016, pp. 93–120.
[17] E. Cherry, C. Latulipe, Quantifying the creativ-
ity support of digital tools through the creativity
support index, ACM Transactions on Computer-
Human Interaction (TOCHI) 21 (2014) 1–25.
[18] P. Karimi, J. Rezwana, S. Siddiqui, M. L. Maher,
N. Dehbozorgi, Creative sketching partner: an
analysis of human-ai co-creativity, in: Proceedings
of the 25th International Conference on Intelligent
User Interfaces, 2020, pp. 221–230.