We discuss the first phase of a multi-year efort to construct a dataset based on an interactive narrative exercise between pairs of humans. One human took the role of the player in a hypertext-based game, while the other took the role of a game master controlling the non-player characters. Alongside gameplay logs, we collected users' explanations for their own choices and for each other's choices, and their ratings for the quality of story structure and personal agency. Through repeated iterations of this process, we hope to support the design and testing of AI experience managers that maximize structure and agency by capturing a human game master's capability to reason about and influence the player's desires and expectations.
A classic challenge in computer interactive narrative
design [
In complex, procedural interactive narratives where it is
not practical to anticipate all possible paths the player may
take, an intelligent experience manager agent [
How can an experience manager promote structure and
agency? Because there can be conflict between these two
criteria, one option is to prioritize one or the other. For
instance, a well-structured story may be crafted through
meticulous planning, but a player making unexpected choices
could cause those plans to fail; to trade away agency for
a stronger guarantee of structure, an experience manager
could model its interaction with the player adversarially [
But in in human-led interactive narrative practices, such
as improvisational theater [
There have been calls to model computer interactive
narrative in the same way, where the system and player work
together as storycrafting co-creators [
This paper discusses our first step in that investigation: collecting data from an interactive narrative hypertext game where the roles of both player and experience manager were iflled by humans. Besides the logs from the gameplay itself, we elicited participants’ explanations of why they made the choices they did, as well as why they believed their partner made choices. Through refinement and repetition of this process, we plan to amass a dataset where these explanations can be used to analyze how users reason about each other’s intentions and expectations, and to evaluate how well a given experience management model can replicate that reasoning.
So far, we have conducted one iteration of our data collection efort, between students in a classroom exercise. We describe the initial design of the interactive narrative exercise and give our reflections on the design of future versions. We also present an observational study using the gameplay logs and user explanations combined with users’ reported perceptions of the structure and agency aforded to them in the game. Although our data collection will need some adjustments to be as efective as possible for our original goal of helping to model implicit communication between participants, our data so far did provide the promising result of a positive correlation between a user’s structure ratings and their agency ratings.
For an overview of experience management in general, see
Riedl and Bulitko [
Although players in interactive narratives can find
meaningful ways to defy game expectations that are still done
with a prosocial mentality [
Our in-progress dataset is unique in combining player
intentions with structure and agency ratings in a symbolic
planning-oriented interactive narrative environment, but
there are existing datasets that have some of these features.
A dataset by Zhu et al. [
We collected data from a classroom exercise with
University of Kentucky students in an undergraduate game
development class. The exercise consisted of pairs of students
interacting with each other over the Web in a text
adventure game with a Twine-like [
Before the exercise, the class was given a brief presentation on the exercise and a basic description of the story characters. Then, to familiarize them with the game interface, users participated in a tutorial session where they were instructed to take specific actions. Users then played sessions in a freeform manner until the end of the class period, paired automatically by the server so that no user had the same role (GM or player) or partner twice in a row.
The game setting and mechanics were based on the Save
Gramma adventure game [
One user was able to act at a time within a session, with control going to the GM by default until they opted to let the player take the next action. Furthermore, we modified some actions to allow for success or failure at the GM’s discretion; for instance, when either user had one character attack another, the GM was prompted to choose whether the attack landed or missed.
At any time during gameplay, users were able to submit star ratings on a 5-point scale of their current evaluation of agency (“I can have meaningfully diferent experiences depending on my own choices”) and structure (“These events feel like a story, rather than a random sequence”). After every three actions in the game besides walking, the interface gave users a visual reminder to update their ratings; however, at only one point in the game—at the end of the session—did the interface force users to rate.
At the end of each session, the interface gave each user a post-game interview (Figure 2): The user was incrementally shown a history of all the actions in the game session. After each action besides walking, the user was prompted to give text input to the question “Why did you choose this action?” for their own actions or “Why do you think your partner chose this action?” for the other user’s actions.
Afterward, we asked students to voluntarily sign a consent form for the release of their data, and deleted the logs from each session that included a participant who did not sign a consent form.1 The remaining data consisted of logs for 22 sessions among 18 participants.
The game interface and the use of the Save Gramma story domain in a multiplayer text game were successful overall: Users quickly grasped most of the game elements after the tutorial, and a wide variety of stories emerged even with the simplicity of the domain. A point for improvement in later iterations is the set of goals available to the player: With the player having a single predefined game objective, there is not much room for investigating how the GM would reason about the player’s long-term goals, so we are planning the next game to let the player choose from several objectives.
Along with the game itself, there are design decisions to iterate on about the form and timing of eliciting other information from the users: their evaluations of the story experience, and their explanations for their own and each other’s actions.
The questions about a user’s intentions behind an action— “Why did you choose this action?” or “Why do you think your partner chose this action?”—had advantages and drawbacks for any choice of timing: Querying a user immediately after they took an action would capture the user’s true expectations at the time better than querying at the end, which would be afected by the user’s knowledge about how the story actually unfolded after the action. But immediate queries would also interrupt the flow of gameplay and risk afecting the user’s decision-making process for later actions. Ultimately, our choice was informed by our intention to query about almost every action, in order to obtain enough data from a small number of playthroughs: Because so many mid-game queries would add up to a large distraction for the users, we chose to do all questioning in a post-game interview.
The responses to these questions took a wide variety of forms; we provide an informal catalog in Table 1. Although each kind provides valuable information in its own right, the diversity would also make it challenging to represent all of the explanations in a unified computational model. A collection of explanations like rows 1 through 3, for instance, could be formulated as goals for a narrative planning problem and be used to validate planning-based models of gameplay; meanwhile, rows 7 or 8 would be valuable for developing models of shared authorship but would require
a notion of story-encompassing goals that does not fit as easily into a state-centric planning formalism.
When eliciting the users’ perceptions of structure and agency, we chose a star rating system because, unlike the text-entry form for action explanations, it could easily be integrated into the game interface in a minimalist manner that allowed quick and (ideally) frequent feedback. However, some drawbacks became apparent later on: First, the ratings in our data turned out sparse, with many users providing only the mandatory final ratings and not the optional midsession ratings. Second, the connotations of star ratings and the fact that structure-agency pairs were very often identical raises the possibility that users may have sometimes submitted ratings inattentive to the meaning—e.g., clicking a single rating in both star fields to represent the overall quality of the story.
We are exploring possible changes to the elicitation for
future iterations. For the action explanations, we are
considering how the way we ask questions to users could be
changed to more precisely target styles of explanation that
lend themselves to evaluating computational models. For
the structure and agency evaluations, we are investigating
ways to survey users that ofer more psychometric validity
than the star rating system. For instance, the IRIS project
has constructed a set of scales to measure aspects of user
experience in interactive narrative [
Issues discussed in Section 4, such as low rating frequency and nonuniformity of how users interpreted the post-game interview questions, meant that the initial data did not enable the experiments we originally planned such as using it to validate experience management techniques. In order to explore the data nonetheless, we conducted a series of post hoc analyses, positing and testing several hypotheses about factors that might correlate with structure and agency. Each hypothesis is described in a subsection later in this section.
For these analyses, we used Spearman’s rank correlation
coeficient [
Because of the large number of hypotheses tested, we
used false discovery rate control. This adjusts the -values
so the significance threshold (we use a standard < 0.05)
bounds the number of false positives relative to the total
number of significant results. In particular, we chose the
Benjamini-Yekutieli procedure [
The numeric results are shown in Table 2: the Spearman correlation for each pairing, the original -values (), and the adjusted -values ( ). The rest of this section Example The player took the potion. The player explained their choice as “to save grandma”.
The player stole a weapon from the Guard, who had attacked them earlier. The player explained their choice as “self defense”.
The player looted a coin from the dead Merchant. The GM explained the player’s choice as “Because capitalism (who doesn’t want more money)”.
The Guard attacked the lawbreaking player. The player explained the GM’s choice as “I committed thievery right in front of him”.
The Bandit robbed the player. The player explained the GM’s choice as “an action a bandit would do”.
The Guard tried to rob the player. The GM explained their choice as “he decided on a career change, he’s now also a bandit”.
The Bandit took the potion after the Guard died holding it. The GM explained their choice as “To keep the story going, it’d be too easy if the player got the potion immediately”. The Merchant died ofscreen at the beginning of the game, leaving their wares free to plunder. The GM explained their choice as “I thought it would be interesting for the player to not need money”.
The Merchant ofered to trade the player a sword in exchange for a jewel, which the player did not yet have. The GM explained their choice as “To try to get the player to go to the camp and steal the jewel”.
The Guard looted the Bandit’s corpse. The GM explained their choice as “I think it’d be funny for the guard, after seeing the player rummage the corpse of the bandit, [to] also start rummaging through the belongings”.
The Bandit picked up the jewel. The GM explained their choice as “I didn’t know what this would do and thought it would be interesting”. elaborates on each hypothesis and summarizes the results. Throughout the section, we use the terms “GM structure”, “GM agency”, “player structure”, and “player agency” as shorthand to refer to those participants’ star ratings of those qualities. Furthermore, we consider only the mandatory ratings from the end of the session due to the low frequency of optional mid-session ratings.
There were statistically significant correlations between a user’s final structure ratings and their final agency ratings; this applied for both the GM and the player. These correlations were positive, supporting our claim that structure and agency are not necessarily in tension with each other.
On the other hand, although the correlation coeficients for all combinations of a GM rating and a player rating were positive, they were too small to be statistically significant.
We hypothesized that higher agreement between a user’s stated intention for their action and their partner’s believed intention for their action would correlate with higher structure and agency ratings for the story. To test this, we labeled each pair of user explanations for the same action on a four-point ordinal scale: A score of 1 indicated that the explanations directly contradicted each other (e.g., one user said the Guard’s attack on the Bandit was unprovoked; the other said the attack was provoked by the Bandit’s attack on the player), 2 indicated that the explanations were unrelated to each other but not contradictory (e.g., one user said the Merchant ofered the potion for trade to avoid being killed by the player; the other said the Merchant made the ofer because it was a profitable trade), 3 indicated that the explanations were plausibly related such as one being a subgoal of the other (e.g., one user said the player stole the potion to get the potion; the other said the player stole the potion to win the game), and 4 indicated that the explanations expressed the same idea (e.g., both users attributed the Merchant looting a corpse to the Merchant being greedy).
There was no statistical significance for the correlations (positive for GM, negative for player) of either user’s ratings with the agreement scores. In later iterations, we plan to collect data that supports an analysis of whether users are fulfilling each other’s intent, beyond simple awareness of that intent, and the relationship of between this fulfillment and structure or agency.
In the game, the GM was able to deny certain player choices;
e.g., if the player character tried to rob an NPC, the GM could
ensure the NPC escaped, or if an NPC tried to rob the player
character and the player character tried to escape, the GM
could ensure the escape attempt would fail. This is akin to
what Riedl et al. [
Because interventions prevent a player from doing something they want to, but protect authorial intent, it is intuitive to expect them to improve structure at the cost of player agency. Based on this intuition, we investigated the relationship between interventions and the actual player ratings. We defined a metric of intervention frequency as the number of times in a session that the GM caused a player’s action attempt to fail, divided by number of times in the session where that was possible. However, the (positive-coeficient) correlations between this metric and player structure or agency were not statistically significant.
Recall that the GM chose when the player was allowed to act. We computed the ratio of player actions to total actions in each session. We posited that this ratio would have a relationship to player agency ratings, but the (positive) correlation coeficient was not statistically significant. A potential factor in this result is the lack of variation in this ratio; most sessions fell within a small range of player-to-GM action proportions.
Narratologists have proposed causality as a critical tool for
story coherence [
Among other factors, causal connection of an event to
the ending of a text story has been shown to predict readers’
perceived importance of the event [
We define a measure of causal connectedness in stories as
follows. The planning domain counterpart to our game [
We computed our causal connectedness measure for each
story by using Sabre [
A limitation is that we defined causality only in terms of
action preconditions and efects, e.g., the Bandit must
possess the potion for the player to steal it from the Bandit, so
the Bandit picking up the potion enables the player’s theft.
Causality defined more broadly would incorporate how
action choices afect each other via motivational changes or
information flow [
This paper presented the beginnings of a dataset we will construct over multiple years. The data from each iteration, as well as more detailed descriptions of the data collection process and related publications, will be made available online.2 A distinct feature of our data collection is the supplementing of game logs with users’ explanations of both their own and their partners’ choices. We hope to use this data to develop better models of how users reason about each other in human interactive narratives and how this can translate to computer interactive narratives where an experience manager agent reasons about the implicit communications between itself and the player through gameplay choices.
Our analysis of the initial data is exploratory in nature rather than drawing definitive conclusions, and it is an observational study rather than a formal controlled experiment that establishes causation. However, the analysis provides preliminary support for one of the premises of our broader project: the possibility for user’s senses of cohesive story structure and personal agency to exist interdependently rather than in conflict.
Beyond data collection, our project will eventually grow to allow for testing community-designed experience managers with a larger and more diverse pool of human players in a variety of story domains, using the same platform that we are developing for human-to-human gameplay. We are interested in feedback from the community about how to make both the platform and our eventual dataset as useful as possible for other experience management researchers.
We thank the other researchers from our lab who helped us test and improve the interactive narrative game used in the data collection: Alex Barrera, Rachelyn Farrell, Mira Fisher, and Lasantha Senanayake. We also thank the reviewers, including those who provided very detailed feedback on a previous submission.
This material is based upon work supported by the National Science Foundation under Grant No. IIS-2145153. Any
opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.