Creative PenPal: A Virtual Embodied Conversational AI Agent to Improve User Engagement and Collaborative Experience in Human-AI Co-Creative Design Ideation Jeba Rezwanaa , Mary Lou Mahera and Nicholas Davisa a University of North Carolina at Charlotte, 9201 University City Blvd, Charlotte, NC 28223, US Abstract In recent years, researchers have designed many co-creative systems that are very promising with a powerful AI, yet some fail to engage the users due to the unimpressive quality of the collaboration and interaction. Most of the existing co-creative systems use instructing interaction where users only communicate with the AI by providing instructions for contribution. In this paper, we demonstrate the prototype of a co-creative system for design ideation, Creative PenPal that utilizes an interaction model that includes human-AI conversing interaction using text and a virtual embodiment of the AI character. We hypothesize that this interaction model will improve user engagement, user perception about the AI, and the collaborative experience. We describe the study design to investigate the impact of this particular interaction model on user engagement and the overall collaborative experience. By the time of the workshop, we will have the data and insights from the study. Keywords Human AI Co-creation, User Engagement, Virtual Embodied AI, Conversational AI 1. Introduction With the intention of investigating the trends in the interaction design of the existing co-creative systems, we AI agents are becoming a part of our everyday life, thanks utilized the archival website, ”Library of Mixed-Initiative to artificial intelligence technologies. Human-AI co cre- Creative Interfaces” (LMICI), which archives 74 exist- ativity involves a human and an AI collaborating on cre- ing co-creative systems [5]. Angie Spoto and Natalia ative tasks as partners [1]. Rather than being perceived as Oleynik created this archive after a workshop on mixed- a support tool, AI agents in co-creative systems should be initiative interfaces led by Deterding et al. in 2017 [6, 5]. regarded as a co-equal partner. This field has the poten- The archive provides the corresponding literature and tial to transform how people perceive and interact with other relevant information for each of the systems. We AI. A study showed that AI ability alone does not ensure analyzed the interaction designs of the co-creative sys- a positive collaborative experience of users with the AI tems present in the archive. Apparently, most of the co- [2]. In recent years, researchers have designed many creative systems use instructing interaction [7], where co-creative systems with powerful AI ability, yet some- users provide instructions to the system using buttons, times users fail to maintain their interest and engagement sliders, or text to communicate directly with the AI (other while collaborating with the AI due to the quality of the than communicating through the creative product). How- collaboration and interaction. The literature asserts that ever, using buttons and sliders, users can communicate user engagement is associated with the way users interact with the AI in a very constrained and minimal way in with a system [3]. Interaction design is often an untended most of the systems. Very few systems use text, voice, topic in the co-creativity literature despite being a funda- or embodied communication for user to AI direct com- mental property of co-creative systems. Bown asserted munication during a co-creation to provide information that the success of a creative system’s collaborative role to the AI, give feedback to the AI, etc. For example, Im- should be further investigated through interaction design age to Image [8] is a co-creative system that converts a as interaction plays a key role in the creative process of line drawing of a particular object from the user into a co-creative systems [4]. Therefore, as a young field, there photo-realistic image. The user interface has only one are potential areas of interaction design to be explored button that users click to tell the AI to convert the draw- for designing effective co-creative systems that engage ing. Other than the button, there is no way of commu- users and provide a better collaborative experience. nicating with the AI to provide information, suggestion or feedback. In a human collaboration, collaborators Joint Proceedings of the ACM IUI 2021 Workshops, College Station, communicate to provide feedback and convey important USA information to each other and is a major component of Envelope-Open jrezwana@uncc.edu (J. Rezwana); m.maher@uncc.edu the mechanics of co-creation [9]. The literature about (M. L. Maher); ndavis64@uncc.edu (N. Davis) human-AI co-creation says that embodied communica- © 2021 Copyright for this paper by its authors. Use permitted under Creative CEUR Workshop http://ceur-ws.org ISSN 1613-0073 Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) tion improves coordination between the human and the Proceedings AI [10]. Additionally, literature asserts that a commu- buttons or using function keys, etc. In contrast, the con- nication channel for conversation between co-creators versing interaction type is where users have a dialogue other than communicating through the shared creative with a system. Users can speak via an interface or type in product improves user engagement in a human creative questions or answers to which the system replies via text collaboration [11]. These literatures led us to investi- or speech output [13]. Conversational agents have tran- gate the impact of embodied communication from the AI sitioned into multiple industries with increased ability and a conversation between the human and AI on user for user engagement in intelligent conversation. engagement and collaborative experience in human-AI The literature asserts that embodied communication co-creativity. Our research questions emerged from the aids synchronization and coordination in improvisational issue that most existing co-creative systems use instruct- human-computer co-creativity [10]. Being able to con- ing interaction type, which uses one-way communica- verse with each other shows an increased engagement tion, human to AI. For this work, we will investigate level in a human creative collaboration [11]. A user’s the impact of conversing interaction and AI embodiment confidence in an AI agent’s ability to perform tasks is on user engagement, user perception about the AI and improved when imbuing the agent with embodiment and the overall collaborative experience. The two research social behaviors compared to the agent solely depending questions we have are- on conversation [14]. Bente et al. reported that em- bodied telepresent communication improved both social • How does AI embodiment and conversing inter- presence and interpersonal trust in remote collaboration action influence user engagement? settings with a high level of nonverbal activity [15]. • How does AI embodiment and conversing interac- User engagement with virtual embodied conversational tion influence user perception about the AI agent agents can be measured via user self-reports; by mon- as the collaborative partner and the overall col- itoring the user’s responses, tracking the user’s body laborative experience? postures, head movements and facial expressions dur- ing the interaction, or by manually logging behavioral For investigating the research questions, we have de- responses of user experience [16]. Carrol and Latulipe veloped a prototype of a co-creative system named Cre- proposed a quantitative and psychometric survey, called ative PenPal where the user and the AI collaborate on a Creativity Support Index (CSI), to assess a tool’s creativ- design ideation task. Users can generate ideas for design- ity support by measuring six dimensions of creativity ing a particular object by sketching on a canvas, and the via self reports: Exploration, Expressiveness, Immersion, AI will also contribute to the design ideation by showing Enjoyment, Results Worth Effort and Collaboration [17]. different inspirational sketches. Creative PenPal utilizes a conversing interaction for the communication between the human and the AI. Additionally, a virtual embodied character for the AI agent is utilized. For investigating 3. Interface the research questions, we describe the study design in Creative PenPal is an interactive prototype, created with the paper. By the time of the workshop, we will have the Javascript, which has all the interaction components ex- data and insights from the study. cept the back-end AI model. We have selected a collection of sketches as the database for creating a seamless expe- 2. Related Work rience that mimics an actual implementation of the AI model. The sketch generation is automated where the Louie et al. identified that AI ability alone does not ensure system selects sketches from the collection. We have two a positive collaborative experience of users with the AI versions of the Creative PenPal prototype to investigate [2]. Bown asserted that the success of a creative system’s and compare the user engagement and collaborative ex- collaborative role should be further investigated in terms perience between the two versions. The original version of interaction design as interaction plays a key role in uses a conversing interaction and a virtual embodied AI the creative process of co-creative systems [4]. Later Yee- (see Figure 1). The virtual embodied AI character, a pen- King and d’Inverno argued for a stronger focus on the cil, is shown in section A of Figure 1. We will address user experience, suggesting a need for further integration the AI character as PenPal in the rest of the paper. Sec- of interaction design practice into human-AI co-creativity tion B is where the conversation happens between the research [12]. PenPal and the user via text and buttons. We can see Interaction types are ways a user interacts with a prod- the design task displayed in section C. Both the user and uct or application [13]. Instructing interaction is where the AI collaborate in a design ideation task where both users issue instructions to a system. This can be done collaborators generate ideas for the design of an object as in many ways, including typing in commands, selecting sketches. Users will design the specified object in the task options from menus or on a multitouch screen, pressing A B C D E F G Figure 1: Original Creative PenPal Interface with AI Embodiment and Conversing Interaction by sketching on the canvas shown in section F. Users can its contribution, it will be seen as happy and when the undo a stroke using the ”Undo Previous Sketch” button user does not like the contribution from the AI, it will and start the design ideation over by using the ”Clear the be sad. The conversation is divided into five different canvas” button. When users hit the ”Inspire me” button situational phases demonstrated in Figure 2. Each phase shown in section B, the virtual AI character will show includes the embodied state of the AI and conversational an inspirational sketch of a conceptually similar object, interaction between the user and the PenPal. The text an object that have similar working mechanism or usage without a comment bubble represents the embodied state as the design task object, on its canvas shown in sec- of the AI in Figure 2. The texts with comment bubbles tion G. Previous work on co-creative design ideation [18] represent dialogues of the user and the AI, and the icon showed that users were more inspired by conceptually indicates which dialogue belongs to whom. Different similar objects than visually similar objects that share responses from the user initiate another phase, which is structural similarity as the design task object. Users can shown using arrows in Figure 2. If the user can respond also ask for visually similar objects or sketches of the with different options, “/” sign is used in the Figure. design task object to get inspiration by saying they didn’t like the conceptually similar object (described in the next 4.1. PenPal Introduction section). Section E shows the name of the object located This phase will start when the user starts the design in the PenPal generated sketch. The other version uses task. PenPal will introduce itself and ask the users if an instructing interaction where users can instruct the they want to see an inspirational sketch from the AI by AI using buttons without AI embodiment (Figure 2). We saying ”Hi! I am your Creative PenPal. Do you want will use both of these two versions to compare the impact me to inspire you?”. Users can respond immediately by of two different interaction designs on user engagement pressing the button ”Inspire me” or they can keep ideating and collaborative experience with an AI. by sketching and respond later. 4. Interaction Model 4.2. PenPal Generating Sketch and Collecting User Preferences For the interaction model, we choose a conversing in- teraction. The conversation with the virtual embodied When the user hits the button ”Inspire me” indicating the AI is simple so that the user will be able to go deeper desire to see an inspirational sketch, the PenPal will move into the ideation process without any interruption in the to the canvas and generate a sketch. The PenPal will ask design flow. The embodied virtual agent will show some the user whether they liked the sketch or not. The user affective characteristics, for example, when the user likes Figure 2: Creative PenPal Interface with Instructing Interaction and without AI Embodiment Figure 3: Conversation Model between PanPal and the User can reply with the ”Yes” button or the ”No” button. This 4.3. User Liked PenPals Sketch phase is for collecting user preferences. When users select the ”Yes” button in response to Pen- Pal’s question to determine whether the user liked the sketch or not, it means the sketch inspired the user in Figure 4: PenPal Collecting User Preference their design ideation. The PenPal will arrive with a happy including 25 males and 25 females as participants. This face and say, ”I am glad that you liked the sketch! Let me study will use a between-subject study where one group know if you want to see another inspirational sketch as of participants will test the version with instructing in- an idea”. If The user wants to see an inspiration again, teraction and without any embodied AI character. The they will select the ”Inspire me” button. other group will test the version with the conversing interaction and a virtual embodied AI agent. The study 4.4. User Did not Like PenPals Sketch will start with a short pre-study survey to collect some When users click the ”No” button, indicating that Pen- demographic information about the participants, for ex- Pal’s generated sketch did not inspire them, PenPal ar- ample, gender, age-range, drawing/ sketching skills, etc. rives with a sad face and says, ”Sorry that I could not Then, the participant will carry out the design task using inspire you!” (left side of Figure 4, the gree arrow indi- either one version of the Creative PenPal. The task for cates transition). Then it suggests the user ask for specific this study is- “Ideate the design of a shopping cart for types of objects as inspiration by saying, ”Let’s try to be the elderly within 20 minutes. You must include three more specific about what you want me to inspire with” design inspirations from the AI in the design”. The whole (Right side of Figure 4). The user can respond witree task will be screen recorded. After the task, the partici- options, ”Design Task Objects” (as our design task object pants will fill out Creativity Support Index (CSI), which is shopping cart, the button says ”Shopping Carts”), ”vi- is a well known psycho metric survey, for measuring six sually similar objects”, or ”conceptually similar objects”. dimensions of creativity: Exploration, Expressiveness, Visually similar objects have visual structural similarity Immersion, Enjoyment, Results Worth Effort and Collab- as the design task object and conceptually similar objects oration [17] to evaluate user engagement, collaboration have similar working mechanism or usage as the design and immersion. After that, a retrospective think-aloud task object. When the user clicks any of these three will be conducted as the participants watch the screen- buttons, the PenPal will generate a sketch accordingly. recording video of the task to understand the rationale behind the user interaction process and user experience. 4.5. User Finished Sketching The study will end with a follow-up semi-structured in- terview to determine in depth qualitative data about the When the user finishes the design ideation sketching, user engagement and overall experience with the AI. they let the virtual agent know by clicking the ”Finish ideation” button. The virtual agent arrives and greets the user for completing the design ideation task by saying, 6. Discussion ”Well done! You did a great job! ”. In the young yet fast-growing field of human-AI co- creativity, attention is needed to design human-centered 5. Study Protocol co-creative systems where users are engaged in a success- ful collaborative experience. Interaction design where The user experiment will take place virtually. We will users can communicate with the AI for providing user use Google Meet to connect with the study participants. preference improves the collaborative experience and The target sample size for the study is 50 participants user attitude towards the AI [2]. Conversing virtual Figure 5: User Did not Like PenPal’s Sketch agents have transitioned into services such as ecommerce, References leading to an increased ability for user engagement. In a conversing interaction, users can provide their feedback [1] N. Davis, Human-computer co-creativity: Blending on the AI’s contribution, which provides more informa- human and computational creativity, in: Proceed- tion to the AI about user preferences. Conversing inter- ings of the AAAI Conference on Artificial Intelli- action also helps users perceive the AI as a partner rather gence and Interactive Digital Entertainment, vol- than a tool. A user’s confidence in an AI agent’s ability to ume 9, 2013. perform tasks is improved when imbuing the agent with [2] R. Louie, A. Coenen, C. Z. Huang, M. Terry, C. J. embodiment and social behaviors compared to the agent Cai, Novice-ai music co-creation via ai-steering solely depending on conversation [14]. The embodiment tools for deep generative models, in: Proceedings improves the user perception of an AI agent in terms of of the 2020 CHI Conference on Human Factors in a collaborative partner, an entity. Users also tend to trust Computing Systems, 2020, pp. 1–13. an AI in terms of their ability when they can see their [3] A. Sutcliffe, Designing for user engagement: Aes- presence. The embodiment also helps design affective AI thetic and attractive user interfaces, Synthesis lec- where an AI’s feelings are visible in its expression or ges- tures on human-centered informatics 2 (2009) 1–55. ture. As a young field and new research area, interaction [4] O. Bown, Player responses to a live algorithm: Con- design is rarely discussed in the existing literature despite ceptualising computational creativity without re- being a fundamental property of an adequate co-creative course to human comparisons?, in: ICCC, 2015, pp. system. An adequate interaction model dramatically im- 126–133. proves the quality of the collaboration and engages users. [5] Library of mixed-initiative creative interfaces, Investigating the impact of conversing interaction and AI http://mici.codingconduct.cc/, ???? (Accessed on embodiment for designing effective co-creative systems 05/31/2020). that engage users is essential. [6] S. Deterding, J. Hook, R. Fiebrink, M. Gillies, J. Gow, We develop the prototype of Creative PenPal as an M. Akten, G. Smith, A. Liapis, K. Compton, Mixed- effort to explore the impact of a conversing embodied initiative creative interfaces, in: Proceedings of co-creative AI agent on user engagement, user percep- the 2017 CHI Conference Extended Abstracts on tion and overall collaborative experience. We describe Human Factors in Computing Systems, 2017, pp. the study design that will provide insights for designing 628–635. effective co-creative systems that engage users and im- [7] Y. Rogers, H. Sharp, J. Preece, Interaction design: prove their collaborative experience with the AI agent. beyond human-computer interaction, John Wiley With the insights and results from the study, we will & Sons, 2011. improve the interaction design of Creative PenPal and [8] P. Isola, J.-Y. Zhu, T. Zhou, A. A. Efros, Image- implement the AI model in the improved prototype. At to-image translation with conditional adversarial the time of the workshop, we will have the data and in- networks, in: Proceedings of the IEEE conference sights in our hands from the study, revealing the impact on computer vision and pattern recognition, 2017, of the interaction model we used. During the workshop, pp. 1125–1134. we will be able to demonstrate the results and insights. [9] C. Gutwin, S. Greenberg, M. Roseman, Workspace awareness in real-time distributed groupware: Framework, widgets, and evaluation, in: People and Computers XI, Springer, 1996, pp. 281–298. [10] G. Hoffman, G. Weinberg, Interactive improvisation with a robotic marimba player, Autonomous Robots 31 (2011) 133–153. [11] N. Bryan-Kinns, F. Hamilton, Identifying mutual engagement, Behaviour & Information Technology 31 (2012) 101–125. [12] M. Yee-King, M. d’Inverno, Experience driven de- sign of creative systems (2016). [13] J. Preece, H. Sharp, Y. Rogers, Interaction design: beyond human-computer interaction, John Wiley & Sons, 2015. [14] K. Kim, L. Boelling, S. Haesler, J. Bailenson, G. Bruder, G. F. Welch, Does a digital assistant need a body? the influence of visual embodiment and social behavior on the perception of intelligent virtual agents in ar, in: 2018 IEEE International Symposium on Mixed and Augmented Reality (IS- MAR), IEEE, 2018, pp. 105–114. [15] G. Bente, S. Rüggenberg, N. C. Krämer, Social pres- ence and interpersonal trust in avatar-based, col- laborative net-communications, in: Proceedings of the Seventh Annual International Workshop on Presence, 2004, pp. 54–61. [16] C. Clavel, A. Cafaro, S. Campano, C. Pelachaud, Fostering user engagement in face-to-face human- agent interactions: a survey, in: Toward Robotic Socially Believable Behaving Systems-Volume II, Springer, 2016, pp. 93–120. [17] E. Cherry, C. Latulipe, Quantifying the creativ- ity support of digital tools through the creativity support index, ACM Transactions on Computer- Human Interaction (TOCHI) 21 (2014) 1–25. [18] P. Karimi, J. Rezwana, S. Siddiqui, M. L. Maher, N. Dehbozorgi, Creative sketching partner: an analysis of human-ai co-creativity, in: Proceedings of the 25th International Conference on Intelligent User Interfaces, 2020, pp. 221–230.