Generative Artificial Intelligence (AI) encounters limitations in eficiency and fairness within the realm of Procedural Content Generation (PCG) when human creators solely drive and bear responsibility for the generative process. Alternative setups, such as Mixed-Initiative Co-Creative (MI-CC) systems, exhibited their promise. Still, the potential of an active mixed initiative, where AI takes a role beyond following, is understudied. This work investigates the influence of the adaptive ability of an active and learning AI agent on creators' expectancy of creative responsibilities in an MI-CC setting. We built and studied a system that employs reinforcement learning (RL) methods to learn the creative responsibility preferences of a human user during online interactions. Situated in story co-creation, we develop a Multi-armed-bandit agent that learns from the human creator, updates its collaborative decision-making belief, and switches between its capabilities during an MI-CC experience. With 39 participants joining a human subject study, Our developed system's learning capabilities are well recognized compared to the non-learning ablation, corresponding to a significant increase in overall satisfaction with the MI-CC experience. These findings indicate a robust association between efective MI-CC collaborative interactions, particularly the implementation of proactive AI initiatives, and deepened understanding among all participants.
Recent advancements in Machine Learning (ML)–
powered Artificial Intelligence (AI), such as large
language models (LMs) [
The paradigm of human creators working with
generative AI via prompting is just one of many
theoretical ways for a human creator and an AI system
to interact [
At the heart of MI-CC systems is the question of
whether and how the AI creative agent knows and
understands (a) the intentions and goals of the human
creator and (b) how the user wants to work with the
AI system. These questions pose significant challenges,
especially within domains critical to game designers
utilizing AI, such as Computational Creativity and
PCG. In other domains, the goal may be provided to AI
in advance, making it easier to identify opportunities
to take the initiative with respect to contributing to a
solution—the extreme of which is the AI system
knowing the goal and solving the goal completely on its own.
When it comes to creating games, however, the human
creators’ intent is harder to articulate completely[
In this paper, we examine Co-Creative systems in a mixed-initiative setting and study the dynamics of managing creative responsibility between human and
AI initiatives. We ask: What influence does an AI agent’s ability to actively adapt to creators’ expectancy of creative responsibility in an MI-CC system have on creator experience and perception?
In particular, we make the assumption that the AI agent is capable of working in the creative domain if given explicit prompts but is unaware of the human creator’s preferences for distributing creative responsibility between humans and the AI. We explore the usage of Reinforcement Learning (RL) methods in this setting and demonstrate that the creative responsibility learning challenge in MI-CC systems can be addressed by a multi-armed bandit (MAB) algorithm that observes feedback from users iteratively, updates its beliefs, and carries out its capabilities to facilitate the MI-CC collaboration. The learning is done online in real-time during the MI-CC process, and the human creator is not expected to have previous knowledge of the AI agent or time to pre-train it with regard to their collaboration style.
Working in the domain of structured story
cocreation, we invite 39 participants to a human subject
study. We quantitatively measure the human creator’s
perceived learning performance of the agent and the
overall level of satisfaction with the collaboration. We
use the Creative Support Index (CSI) [
The procedure of an MI-CC system learning its
creative responsibilities can be described as a
decisionmaking process, where the agent communicates with
the human creator, gathers information, and chooses
among its capabilities. This is not as straightforward
as asking human creators to prompt AI agents because:
• Just like the Cold Start problem experienced by
AI agents lacking prior preferential knowledge
from their creators [
For these reasons, relying solely on human creators for direct collaborative prompting, regardless of the capability of the AI models, has its limitations, leading to eficiency, cognitive load, fairness, and equity issues.
Alternatively, a model can be built on human
feedback without users directly communicating their goals.
Researchers demonstrated their potential in
transferring human knowledge to AI [
To focus on the active probing challenge, we
formalize it as a Multi-Armed Bandit (MAB) problem [
In this section, we present the study we designed to examine the AI agent we created that adapts to creators’ expectancy of creative responsibility. We seek to determine how this changes the perception of the creators toward the AI and the creative experience the system supplies to the human creators. The Delegation Setup. For the experiments, we spotlight a specific but generalizable collaborative setup: Learning a delegation. In this setup, both parties take a subset (or entirety, if preferred) of responsibilities in an MI-CC activity towards the common goal. The human creator concentrates on specific parts of the creative task while not losing control of the other parts; the AI agent needs to strategically shift its focus towards the parts that the human creator is not focusing on and actively determine how to make improvements. Furthermore, as these interactions are not without cost, such as creators’ cognitive load, it is also important to minimize such costs towards learning these responsibilities. We denote the expected and delegated responsibility that the AI agent needs to learn during the interaction preferred work style for a particular human creator.
Domain: Storytelling. Given the mounting interest
in co-creative storytelling [
For our experimental system, We use
Llama2-13bchat [
We now describe the AI system we built for the purpose
of the study. The experimental system is based on
the Creative Wand framework [
The Creative Context is the abstraction of generative models for this system.
In this paper, we study stories containing four components inspired by the Narrative Arc theory: the beginning, development or rising action, climax, and conclusion. We design an AI framework that writes each component of the story using language models and prompt engineering (See Appendix C for more details). Both the human participant and the AI are instructed to write about 20 to 30 words per component, and the target length of the whole story is around 100 words.
Once we set up the model, it will take requests from Communications.
Communications describes the interactions between the human creators and the AI; they also double as the capabilities the AI agent possess. To focus on how the agents would choose their creative responsibilities, we implement a minimalistic yet complete set of capabilities for the creative experience. This allows us to focus on research questions about the creative experience while minimizing the cognitive load of the participants. Our agent possesses the following capabilities, implemented as prompts to the LM describing the responsibilities (See Appendix C for details): • (Re)write the beginning and development; • (Re)write the climax and conclusion; • Write a review of the story, one sentence positive, one negative, and one suggestion for improvements.
These two modules manage the interactive experience and workflow.
We implement a Finite State Machine to manage the experience. Figure 2 shows the states with the overall flow of interaction each participant experiences in one experiment session. One session of the MI-CC experience is separated into multiple “turns”, where both parties iteratively improve the story, sharing the same text fields in the editing process. The participants are not directly notified of the internal states of the system.
Human Initiative. During this phase, human creators contribute to the story by making edits in any of the four text fields. This phase ends when the agent decides to take the initiative. We implement a pointbased heuristic based on pilot studies: the agent would assign points for changes it observes, and will take initiative whenever enough points are accumulated, signifying substantial edits from the human creators, in the following criteria: • Each new character would add 5 points; • Each time the human creator switches between ifelds after any changes, 100 points are added; • Whenever the human creator leaves a text field with 200 points accumulated (roughly one full sentence or two minor changes), the agent will take the initiative by locking the editing interface and resetting the counter.
This heuristic provides two advantages compared to other ways this decision can be made: First, this heuristic is computationally fast and enables responsive interactions; Second, it additionally provides visualization for the users. As shown in Figure 1, we present this right above the text boxes for the stories, with a text hint and a progress bar representing the ideation process of the agent. We additionally provide a “skip” function that forces agent initiative. Agent Initiative. In this phase, the agent decides which capability best fosters the collaborative experience and carries out the corresponding Communication. We build a Multi-Armed Bandit-based agent in our system that is responsible for choosing which Communication to invoke, with Thompson Sampling as the chosen algorithm for the experimental system within the AI agent. Formally, an agent interacts with a set of arms 1 · · · , each of which is associated with Communication and underlying capabilities and an unknown reward distribution. Whenever an arm is pulled, the agent seeks feedback from the human creator on the initiative, which is treated as a reward signal. (See next paragraph.) The goal of the agent is to maximize the total reward obtained by repeatedly pulling arms during the session. See subsection A for more details on the design choices of the MAB agent. Once an arm is pulled, the agent executes a Communication, interacts with the user, and updates the story as needed.
Learning from human. The system will ask about (Action Feedback) the way they just worked and (Content Feedback) the updates and content changes. The participants choose between “Good” (Reward of 1) and “Bad” (Reward of 0). “Bad” feedback on generated text leads to a reversion to the original content, though it is not used to improve the LM in any way.
A weighted mean is employed to integrate both types of feedback into a singular reward signal. For the study, a weight hyperparameter of 80% is applied to the Action Feedback and 20% to the Content Feedback. This prioritizes learning action-level responsibilities rather than the preference for LM-generated text, in which the full system and the baseline share implementation. This reward signal is then used to train the agent.
For this experiment, an MAB agent with Thompson Sampling is used in the experimental system. See Appendix A for a discussion and experiments related to this choice.
Once the learning process is complete, “human initiative” starts again. To maintain user engagement, text responses are morphed each time to avoid repetitiveness, while contextual hints are also strategically provided throughout the experience. Figure 1 shows the user interface.
To study the perception of human creators towards MICC systems equipped with these learning capabilities, we conduct a study summarized in figure 3 on the AI system.
We compare our system, the “Full” system, with an ablation named “baseline”. The “baseline” ablation does not learn. It chooses each of the 3 Communications with a 1/3 probability at all times and provides only a reverting option when “asking for feedback”. These systems are codenamed “Echo Wand” and “Harmony Wand” respectively, not to reveal the details of the systems to the participants during the study.
We recruited 39 United States participants 2 on Prolific 3 with adequate English proficiency. Each experiment session lasted for approximately 40 minutes, and we paid the participants $15 per hour for perfect completion of the study.
Pre-study. Before the experience, participants answer four 5-point Likert-scale questions on (Q1) Expertise in Computer-Assisted Designing (CAD), (Q2) Expertise in writing stories, (Q3) Frequency using AI, and (Q4) Understanding of AI. 4
We then present instructions to familiarize the participants with our systems by providing annotated screenshots of the interface, which is a copy of Figure 1, but with additional numeric overlays, descriptions of components, and a brief introduction to the workflow of co-creating a story.
They are then assigned the delegation task to focus on writing the beginning and the development of the story while leaving the other parts of the story to AI as much as possible. They are also made aware that the AI does not know this setup in advance. Experience. Participants are assigned to interact with the full system and the baseline ablation, presented in random ordering, counter-balanced. They are given 10 turns per each of the 2 sessions. 2Only counting participants who finished the whole study with valid sessions and responses. 3prolific.co 4See Appendix B for the full question text.
Post-study. After participants finished two sessions
using our system, they were asked about the process
they had just experienced. inspired by Creative
Support Index (CSI) [
Specifically, we ask which system(s), are (Q5, Learning, Collaboration) learning to collaborate, (Q6, Enjoyment, Immersion) more capable and easy to work with, (Q7, Expressiveness, Exploration, Results worth efort) enabling better stories; For Q5 through Q7, participants can choose either system, both systems, or neither to be chosen, leading to a potential total exceeding 100%. We ask one final question (Q8) on which system will they recommend more, framed in a win-draw-lose format.
Although these questions are presented in the same order for all participants, the order of the options is randomized to reduce bias towards any system. All questions are followed by an open-text question prepared to collect justifications from the participants.
Table 1 shows a summary of the creative backgrounds of the participants. Although a median of 4 on all questions implies that participants are familiar with the recent advancement of AI, when specifically asking whether they can build one, only 1 participant answered “yes” (5 in Q4), meaning that most of the participants do not have a technical background.
However, comparing to 26% reported in [
We commence by presenting the quantitative results of the study through the choices made by the participants in the multiple-choice questions.
When asked which system(s) learned to collaborate with them under the delegation arrangement (Q5), the “Full” system is chosen 69% ( = 39) of the times, compared to 51% for the baseline ( < 0.018, under a Q (See Appendix B for full questions)
Q1: CAD skills
Q2: Writing skills
Q3: Frequency of using AI Q4: Understanding of AI Tech. binomial test where 0 := no observable diference in distribution; The same for all p-values in this section). We clearly see the “Full” system with learning capabilities enabled being perceived significantly better at learning the delegation than the baseline, demonstrating the efectiveness of the MAB-based model From the human creator perspective learning from their feedback.
When asked which system to recommend, this trend also persists: Our system is preferred (wins) 43.6% of the time, versus 20.5% (loses) for the baseline ( < 0.001); 35.9% of the participants do not have a preference (draw). The “Full” system is only diferent from the baseline system with the learning capabilities and corresponding frontend elements, yet we see a statistically significant improvement in preference towards our “Full” system, illustrating the potential of our method in enhancing MI-CC experience and making such system better for human creators.
When it comes to which system(s) gave a good story (Q7), 72% of the participants agree that the “Full” system made a good story, while 69% selected the baseline system ( > 0.05). We were unable to statistically determine whether an agent learning the delegation would produce a better story; This is expected, We focused on studying the sharing of responsibilities and enforced a delegation setting. In an actual MI-CC experience, without such a prior, A human creator would utilize the agent’s learning capability to promote their strengths and discourage their weaknesses, and an improvement in perceived performance is more likely to be observed in that setting.
Finally, when queried about the collaboration itself
(Q6), 62% of the participants think the “Full” system
is capable and made the collaboration easy, while 56%
voted for the baseline system (p>0.05). We also were
unable to statistically determine whether the “Full”
system is more enjoyable and immersive. Although the
diference between the “Full” system and the baseline
is substantial enough both implementation-wise and
towards the perception of learning, from the angle of
the user interface, the only diference is 10 additional
questions from the “Full” system per session.
Previously, Larsson et al. [
We now show the results from the open-ended
questions following each multiple-choice question.
Openended justifications participants provided for each of
the four questions are evaluated with thematic
analysis [
A MI-CC system that understands the intents of the human creators and follows them by learning is overall favored and collaborates well with the creators. Participants demonstrated their observation of the learning capabilities of the “full” system, identifying them as “better about learning that I specifically wanted help with” (P34) and “listened to my feedback.”(P39). In comparison, the baseline system is identified as “did less of the work ... did not necessarily learn what its role was expected to be” (P19). this resulted in a preference for the Full system for P32, as the Full system is quoted as a “more useful helper". This aligns with the quantitative observations. Good content suggestions may give people the feeling that the system is learning how to collaborate with them, regardless of how AI is actually doing so. Despite specifically asking participants to discuss whether the agent has “learned to collaborate with you under that arrangement” (Q5), Participants are also rating the system based on the generated content: (P25, emphasis asked) This one learned from me because it was able to build of of my original foundation of my story that
I typed.
P18, who rated their familiarity with AI as Familiar (4 out of 5) and AI usage as “Always / as much as possible” (5 out of 5), wrote that the “Full” system is learning from them: I could see Echo Wand adding more detail and building out more creatively than with Harmony Wand.
This participant is familiar with recent generative AI and mentions “adding details” and “building,” which are traits that these AI are optimized for. As both the “Full” and the baseline use the same underlying generative AI capabilities, P18 could not distinguish between the “improvements” on generated contents and the performance of the MAB-based agent. The apparent improvements of generated stories may result from a wide range of reasons, such as participants providing diferent input and LM sampled diferently, unrelated to both the underlying LM and the learner, creating noises in the perception of participants. Diversity is also important, it may not be the best strategy for a learning agent to pick the “best options”, and sometimes the agent may want to intentionally surprise their teammates. P23 was impressed by the range of capabilities both agents possess, seeing “They were both impressive, being able to take my story and to word it better, or even add things to change it to make it better”. When asked about the generated story, P39 mentioned that “ Both of them gave bad stories.” and “I need much more control and options”. Curiously, this is the same participant that enjoyed the agent that “listened to my feedback.”. P36 preferred the baseline system that executes random actions:
I did all of the work with Echo, despite my best eforts to get it to collaborate with me. Harmony had much more interesting suggestions and rightfully pointed out when a section became too dense.
It balanced the second two sections to match my intro and build up, unlike Echo who almost refused to work on them.
For this study, we assigned delegation tasks to the
participants. This is only a subset of possible
responsibilities that the AI agent can take and the human
creators may expect. Lin et al. [
Creator control is important, and creators may want their ideas to be included even when AI can provide better candidates. Beyond the need for control mentioned by P39, P28 mentioned that they were impressed by the capabilities of both systems in “finish the story that I started with.” (Emphasis added). P27 mentioned further on their justification: ... I was in control of the final text to accept changes or not, or to make my own.
In a system involving a creator who wishes to create content to their liking, it is expected that the creator wishes to solicit as much control as possible. However, if the AI agent does not have any final say on the contents, should we expect it to take any creative responsibilities? Although we acknowledge that this is more of a philosophical question, way out of the scope of our work, what if the agent would understand what their counterpart is actually seeking and use this information to determine what contribution they should stick to by understanding what human creators are thinking?
Distilling from these findings ranging from the
perception of collaboration, good writing skills, diversity
in capabilities, and creators’ need for control, a
common implication surfaces: Getting the mental model
of the creators right, the system will succeed; Getting
it wrong, failure cases would surface. A mental model
is described by Kieras et al. [
We start to see a consistent narrative: creators are
interpreting the capabilities of our AI agent learning
as an attempt the AI agent made to learn a mental
model of themselves; Because our agent determines
which Communication to use and the efect of it on the
contents being collaborated on, We observe the
participants treating proper learning of Communication
choices (expected) and the content generated
(emerging) as both evidence that the agent is learning from
them and traits leading to their preferences towards
these systems. This also, to some extent, explains
the placebo efect we observe on the baseline system:
around half of the participants believe that the
baseline system is learning from them, significantly more
than 0, despite the baseline system only making
decisions randomly. In this controlled comparative study,
to avoid a bias towards either of the systems, we
intentionally did not disclose any diference between the
“full” system and the baseline. This perception may
have arisen from the capability of our agent to
generate part of stories that follow the context that the
participants provided. Although we acknowledge that
these factors are hard to decouple, this finding also
hints at the potential of our methods in understanding
the human creator holisticly. Upol et al. [
In this paper, we showcased how an MI-CC system is capable of listening to human feedback and improving itself towards a better understanding of how it should collaborate with human creators in a storytelling domain. Inviting 39 participants and comparing two such systems with and without these learning capabilities, we found that this capability was well recognized by the participants and led to better satisfaction overall. To this end, we further encourage the designers of MI-CC systems to pay attention to both the human creators and the AI agent, study how each party should, or is already, adapting to and creating mental models of their counterpart, based on their creative roles taken, their previous experience, and capabilities, and most importantly, the wishes of the human creators.
In this section, we provide more information on the
design choice of the MAB agent. Following results
from Vinogradov et al. [
-greedy [
UCB1, or Upper Confidence Bound 1 [
Finally, Thompson Sampling is a robust Bayesian
approach first introduced by Thompson [
We carried out an Oracle-based experiment to determine the MAB algorithm of choice for the study. Using an oracle, which simulates a human creator interacting with the system, gives us total control of their behaviour. We measure the performance of the agents at various levels of human feedback accuracy, to seek an agent that generally performs well on all accuracy levels so that it serves a wider variety of human creators well.
We study four diferent agents and baselines: greedy, UCB1, Thompson Sampling, and Random Baseline, where a universally random arm is chosen each time. We give the agents 3 arms to pull, where one is “liked” and two others are “unliked”. Each arm would give either a reward of 1 if liked or 0 otherwise when pulled, by the oracle; We define human feedback accuracy as the probability of the oracle giving a reward of 1 on pulling the “liked” arm and a 0 on pulling the “not liked” arm. As this value gets lower, closer to 50%, the simulated oracle becomes less clear on which arm it liked and becomes a less eficient feedback provider. We simulated 5 levels of this accuracy, from 60% to 100% with equal intervals.
-greedy is highly sensitive to its parameter chosen, and we report with the best performing -greedy agent in the with = 0.2. We report the “normalized rewards”, which is the agent’s reward relative to the theoretical maximum of always choosing the “liked” arm. We repeat each experiment condition 100 times and report the mean normalized rewards after 10 steps to simulate a scenario where the MI-CC agent has to quickly learn from their human counterparts, similar to our actual study.
Figure 4 summarizes the results from the Oracle experiments. As we only gave these agents 10 steps to learn the arms, the agent may not have yet converged. This is expected in a quick-learning scenario. -greedy performed poorly, even worse than the random baseline, likely due to its inability to quickly change focus between exploration and exploitation; UCB1 and Thompson perform at similar levels, demonstrating their capabilities to calculate an upper-bound reward and use it in their decision-making process.
Although UCB1 and Thompson performed similarly, Thompson Sampling is preferred because of its sampling behavior. UCB1 schedules its exploration over a very long session in a deterministic way (exploring once after exploiting times). As we aim for quick learning and adaptation, without sampling, UCB1 risks showing “stubbornness” to a suboptimal arm without any probability to unstuck itself, a behavior that is less preferred from an MI-CC perspective. Thompson Sampling, on the other side, exhibits its capability to dynamically change its exploration aggressiveness based on previous observations, while using a Bayesian prior instead of greedy sampling, both benefiting its application in our experiment MI-CC setup. This results in both an efectively dynamic “epsilon” compared to epsilon-greedy and some randomness instead of being fully greedy per each step, compared to UCB1.
We chose Thompson Sampling as the MAB algorithm used in the experimental system.
Pre-study. Four 5-point Likert scale questions are asked: • Q1: Do you agree that you are familiar with the process of creating content, such as writing articles, drawing pictures or creating a video game stage, using a computer? (Strongly Disagree → Strongly Agree) • Q2: Do you agree that you are good at writing or telling a story, either real or fictional? (Strongly Disagree / Never attempted in the past 5 years → Strongly Agree) • Q3: How frequently do you use or interface with artificial intelligence? For example, using map services to find a route to your destination, playing a game with a computer-controlled character, or using a chatbot. (Never used → Always / For as many things as possible) • Q4: How much understanding do you have of the recent developments in Artificial Intelligence technologies? (Very unfamiliar → Very familiar / I can build one) Post-study. Four questions are asked regarding the systems they used during the study.
• Q5-(Learning, Collaboration) You were assigned a specific way to collaborate with the assistant Wands, and the assistant is not informed of this arrangement in advance. Which assistant wand learned to collaborate with you under that arrangement? If you have chosen at least one of the assistant wands, how did you know they learned from you? • Q6-(Enjoyment, Immersion) Which assistant wand is more capable and made the collaboration easy for you? If you have chosen at least one of the assistant wands, how did the assistant(s) impress you with their capabilities? • Q7-(Expressiveness, Exploration, Results worth efort) With these assistant wands, which collaborative experience ended up in a good story? If you have chosen at least one of the assistant wands, What do you think helped? If you chose neither, what went wrong? • Q8-Lastly, which assistant wand would you recommend more to a friend or a colleague story writer? Please let us know if you have any other message or comment to share.
For Q5 to Q7, Participants may select one, both, or neither system; For Q8, as it is a comparative question, the option of "neither" is not available. All questions are followed by an open-text question prepared to collect justifications from the participants.
Prompts for Communications start with “You are an AI writing assistant, collaborating with a human on the task of writing a story.You are very concise, and answer only what is absolutely necessary, without any explanations or introductions.You make sure that all your answers are surrounded by an underscore, such as _My answer_ .” and are followed by a few examples of the tasks, along with the constraints, formed in a question-answering format; The final question does not come with an answer, and the continuation is treated as the response.