=Paper=
{{Paper
|id=Vol-1914/NHT17-5
|storemode=property
|title= What’s the Story? A Proposed Approach for the Evaluation of Experimental Interactive Narrative
|pdfUrl=https://ceur-ws.org/Vol-1914/NHT17-5.pdf
|volume=Vol-1914
|authors=Charlie Hargood,Ben Artis,Corey Stevens
|dblpUrl=https://dblp.org/rec/conf/ht/HargoodAS17
}}
== What’s the Story? A Proposed Approach for the Evaluation of Experimental Interactive Narrative==
https://ceur-ws.org/Vol-1914/NHT17-5.pdf
What’s the Story? A Proposed Approach for the
Evaluation of Experimental Interactive Narrative.
Charlie Hargood Ben Artis Corey Stevens
Creative Technology Creative Technology Creative Technology
Bournemouth University Bournemouth University Bournemouth University
chargood@bournemouth.ac.uk i7673319@bournemouth.ac.uk i7673602@bournemouth.ac.uk
ABSTRACT 1 INTRODUCTION
Evaluation of experimental digital narrative often focuses Digital narrative is, by its nature, experimental. How authors
on the overall user experience. While this is important, we tell stories through technology has been changing rapidly
recognise the need for more granular forms of evaluation to following continued exploration of the possibilities and the
measure the efficacy of individual digital narrative delivery poetics of a range of digital mediums. Interactive fiction,
techniques. These continue to grow in variety as authors ex- hypertext narrative, RPGs, and computer games have all ex-
plore different approaches to telling stories using interactive perimented with different delivery mechanisms for stories
media. In this paper we propose a multi-layered evaluation that play with both the interactive nature of digital nar-
methodology based on the principle of deconstructing an rative as well as its dynamic qualities to create nonlinear
interactive narratives internal fabula and story. This is fol- narratives. These delivery techniques include hypertextual
lowed by three separate stages of interview for collecting patterns of narrative structure [3], hypertextual presentation
evidence of the efficacy of different techniques used within paradigms such as spatial hypertext [4], and game narrative
story payloads to deliver content. This proposed methodol- delivery techniques such as environmental storytelling [10]
ogy shows early promise, and potentially provides a means and mechanics as metaphor [12]. For the purpose of this work
to identify the individual efficacy of techniques within a we might define narrative ’delivery techniques’ as a story
wider digital narrative. structure, medium, or other delivery format design decision
within the narrative made for the presentation of a part of
CCS CONCEPTS the story, whether a section of plot content or something
• Human-centered computing → Hypertext / hyper- more subtextual such as a theme.
media; While there has been a range of work exploring these
different possibilities there is somewhat less work in the
KEYWORDS area of bespoke evaluation methodologies for the narrative
efficacy of these different techniques. Existing evaluations
Digital Narrative, Evaluation, Game Narrative of these approaches normally follow the path of traditional
user experience studies - often seeking to deconstruct the
Reference format:
usage of a system and its impact on the user [8]. However
Charlie Hargood, Ben Artis, and Corey Stevens. 2017. What’s the
Story? A Proposed Approach for the Evaluation of Experimental
while these tell us much the ability of the system as a whole
Interactive Narrative. In Proceedings of Narrative and Hypertext to deliver on ’immersion’ and ’flow’ [8] they tell us little
Workshop @ ACM Hypertext 2017, Prague, Czech Republic, July 2017 about how much of the narrative was successfully delivered
(NARRATIVE AND HYPERTEXT’17), 5 pages, CEUR-WS.org. and understood by the audience or the efficacy of individual
techniques within the technology.
In this paper we propose a multi-layered approach to the
evaluation of the efficacy of digital narrative delivery tech-
niques to deliver story. We present two games we have cre-
ated as a group, and how we have deconstructed their nar-
rative design in order to form an evaluation methodology
Copyright held by the author(s).
that goes beyond user experience and seeks to explore the
efficacy of individual techniques to deliver story elements.
NARRATIVE AND HYPERTEXT’17, July 2017, Prague, Czech Republic Charlie Hargood, Ben Artis, and Corey Stevens
2 BACKGROUND its story through a sequence of micro narratives embedded
This work draws on both techniques for digital narrative into the world itself [5]. This includes the idea of environ-
delivery (in surveying the different forms that might be eval- mental storytelling - that the story is not only delivered
uated) and interactive entertainment evaluation (existing through the core plot surrounding its protagonist but also
methodologies that might be applied). through the world and environment. In his work on envi-
ronmental storytelling Jenkins [10] explores four different
Hypertext Narrative techniques through which it has been delivered: evocative
Hypertext fiction such as Joyce’s ’afternoon, a story’ and spaces (such as richly defined level environments that con-
Moulthrop’s ’Victory garden’, has explored a range of nar- tribute thematically and depict the world), enacted stories
rative delivery methods, often through different hypertext (micro narratives encountered as part of the idea of story as
structures such as the calligraphic patterns originally ex- journey [1]), embedded stories (fragments of narrative de-
plored by Bernstein [3]. This form of structural storytelling livered through discoverable game artifacts), and emerging
may be used to deliver its own poetic effect whether it be narratives (actions and events in front of the player that lead
the emphasis or change of context from rereading given to them to infer their own narratives).
us by a cycle or the feeling of disorientation from a tangle.
Interactive Entertainment Evaluation
Similarly, sculptural and location-aware hypertext presents
its own patterns [9] where an author may choose to deliver Existing game evaluation, as with most application evalua-
narrative through simultaneously available parallel threads tion, is user experienced focussed. In her summary Denisova
or an array of concurrent nodes. The variety of hypertext et al. [8] explore a range of game evaluation methodolo-
forms and structures extends further still, and includes the gies exploring the player experience including the IEQ [11],
juxtaposed montages of spatial hypertext [4]. These design GEQ [6], and PENS [16] methods. These approaches use ques-
choices are important to the poetic value of the story and tionnaires to explore Flow, Presence, Control, and a range of
they may radically change how a story is interpreted. They other variables connected to the UX of game play. While this
are vehicles for delivery of narrative content to the reader is undoubtedly valuable it only provides limited insight into
which are themselves part of the message. Where one struc- the narrative impact of a technique used. These approaches
ture might be effective for a given story and might be less might tell us something about the players general enjoyment
effective or distort the delivery in ways both desirable and or the suitability of the modes of interaction but not whether
undesirable. a particular technique communicates the story we want it
to.
Game Narrative
3 DEVELOPMENT CONTEXT
While some modern game narratives leverage the techniques
common to more traditional mediums, such as film or text, A pair of narrative centric games under recent development
there are experiments in new delivery mechanisms to deliver (which employ some of the delivery techniques discussed) as
narrative in less traditional ways that leverage the interactive part of an exploration into the efficacy of these techniques
nature of the form. The idea of ’Mechanics as Metaphor’ [12] was the primary motivator for this work. To give context we
builds upon the more theory of ’the medium is the mes- offer the following descriptions of these games Paramnesia
sage’ [15] from more traditional media and is used to describe and Blitz (as shown in figure 1’s screenshots).
games who use their mechanics (the actions and interactions
the player makes) as a means for conveying narrative itself. Paramnesia
This may be rules of the game that reflect aspects of the The premise of Paramnesia is to use subconscious psycho-
characters, changes to the dynamics to communicate affect logical techniques to control player actions as well as convey
or emphasis in a scene, or metaphorical interactions within a strong narrative; using these subconscious techniques the
gameplay. This is strongly connected to the idea of ’Diegetic’ theory is to better convey narrative and object points to
choice within games [14] where choices made within the players, resulting in a closer cohesion of scripted narrative
narrative of the game are made within the game’s own me- and the player’s interpretation of story. These subliminal
chanics and dynamics, rather than abstracting the agency of techniques are similar in premise to conventional forms of
the game away from the game play itself (so called ’Extra environmental storytelling in that the narrative emerges
Diegetic’ choice). through the delivery of small sections often through the
Other game narrative designers prefer to avoid more tra- space the player explores - though they are more heavily
ditional grand narratives and instead use different narrative couched in related psychological theory. Paramnesia is a
delivery techniques that focus on the game world delivering first person atmospheric game set in an apartment block and
What’s the Story? NARRATIVE AND HYPERTEXT’17, July 2017, Prague, Czech Republic
told the numbers. Suggestion [13] allows control over the
player’s intentions through merely introducing the idea in a
specific way. For example, throughout the game ’the voice’
uses the phrase ’We’d greatly appreciate it if you could...’.
This is purposely used, and can be broken down to: ’We’d’ -
Which suggests more than one person is watching and added
pressure, ’greatly appreciate it’ - a nicety, which creates a
bond between communicator and receiver quickly, ’if you
could’ - which creates pressure just to accept while the re-
ceiver is still thinking about the two previous steps. This
culminates in a stronger bond and higher levels of compli-
ance between ’the voice’ and the player.
Blitz
Blitz is a first person interactive story set in 1940s London.
You play as a young boy who has been abandoned by his par-
ents and has woken up in a version of his childhood house
that has been separated from society. Exploring the house,
the player discovers what happened to their parents. Blitz
makes use of both Environmental Storytelling, and the no-
Figure 1: Paramnesia (top) and Blitz (bottom) tion of ’Mechanics as Metaphor’. Environmental storytelling
being the technique that is more heavily used.
Throughout the first area there are clues and hints to the
subway. The player is convinced to find, set and detonate a location, time and events that relate to the house you are in.
bomb in an empty subway, under the intention of it ending a Graffiti on the walls as well as a range of propaganda posters
communist uprising currently taking place in America. After reveal somewhat happened to the father and gives an idea of
the detonation of the bomb, it becomes apparent that the the date that the game is set. There are other subtler hints,
player was manipulated to perform this terrorist attack un- such as the photographs on the wall; in one of the images,
der false pretenses. There were 4 psychological techniques a young boy is wearing glasses, and in the next area, there
employed within the level: Conditioning, Compliance, Mere are a pair of glasses on a crate, that indicate you are the boy
Exposure and Suggestion. in that photograph.The use of ’Mechanics as Metaphor’ is
Conditioning [7] allows control over player direction, con- somewhat lighter. Player movement speed is slowed, and the
tinuation of player motivation and player satisfaction. For camera height of the player lowered and increased in FOV so
example, the use of red light was common throughout the that the player felt surrounded by everything within vision
level which was paired with the idea of communism, this to highlight the overwhelming notion of events beyond the
therefore meant a red light could be used to trigger the idea of protagonist.
communism for the player allowing to keep the motivation
4 PROPOSED EVALUATION METHODOLOGY
for their actions going. Compliance [17] allows the designer
to control the player’s reaction to requests. This was used In order to explore the efficacy of the techniques within
for control over the player’s direction as well as the player’s the games, we required a methodology that was granular
interpretation of the narrative. For example, the voice (which enough to focus in on the impact of the delivery of individual
is used to convey some objective and narrative) is created as pieces of story. We require a method that can consider the
an authoritative figure to create compliance from the player coherency of delivery (did the technique successful deliver
towards the instructions given. Mere Exposure [18] allows content in a way the audience understands), and to what
for numbers, patterns and objects to be in the mind of the degree was its efficacy different to other techniques.
player, purely because it has been show to them multiple
times. For example, the code for the security door is never Multi-Layered Deconstruction
explicitly told to the players, however, the player sees the 4 To begin to understand the answer to these questions we
digit code up to 10 times on the way down to the subway in need to first identify what techniques are used where and
graffiti, posters, price tags etc. This results in the numbers to deliver what. This process is effectively identifying what
being in the player’s mind without them specifically being Narratologists such as Bal [2] would call the ’Fabula’. Here
NARRATIVE AND HYPERTEXT’17, July 2017, Prague, Czech Republic Charlie Hargood, Ben Artis, and Corey Stevens
the ’Fabula’ describes the collection of elements that com- if the payload successfully delivered its content and how
prise the story’s content: its characters, events, places, and this was received by the user. This way we can evaluate the
facts. We can then connect these fabula elements to their techniques based on how successful they were at content
delivery in ’story payloads’ that are presented to the player delivery in the context of the payloads in which they were
through particular techniques in a given scene. Each story used. It is not, however, suitable to simply ask a player ’did
payload recognises the combination of 1 to n fabula elements you notice element ?’ as the question itself informs the
and 1 to n delivery techniques as depicted in figure 2. player before they give their answer. As such we propose a
three layer evaluation methodology for subsequent player
interviews:
(1) User-driven discussion - The user is encouraged to
lead the discussion by discussing the story and its
setting general terms without guidance to particular
parts. This collects evidence for all story payloads
without the player being lead to the significance of
particular story elements through questioning. The
player may mention particular elements and conse-
quently demonstrate some effect for the payloads
Figure 2: Each story payload is made up of 1-n fabula ele- connected to those elements.
ments and 1-n delivery techniques (2) Interviewer-driven discussion - Fabula - The inter-
viewer now delivers pre-constructed questions for
As an example the code to the security door in Paramne- each fabula element to explore whether this content
sia is a fabula element, it is delivered to the player in ten was successfully delivered to the player and how
different payloads through out the game. These payloads all it was interpreted. This stage may lead the player
make use of the delivery technique of ’Mere Exposure’ as to elements but not their delivery techniques, and
part of environmental storytelling. It is sometimes packaged ensures what content was or wasn’t received by the
into a payload with other fabula elements, for example the player.
posters also communicate the anti-communist themes of the (3) Interviewer-driven discussion - Story - The interviewer
piece. Consequently we now have a mapping from ’fabula’ now delivers pre-constructed questions for each story
elements, to delivery payloads as part of the story which payload. This is the final layer of questioning and
uses a technique. This example is depicted in figure 3. directs the players attention to specific parts of the
story delivery. This may heavily lead the answer, but
won’t affect previous sections and captures specific
efficacy of each delivery technique.
For each stage of this method the interviewer does not
need to ask specific questions that have already been covered
by evidence in a previous stage. For example, if evidence is
gathered to support the player understood a given fabula
element in stage 1, then the question for that element does
not need to be asked in stage 2. As the interview progresses
the evidence for player engagement becomes weaker due
to their being potentially lead by the lines of inquiry, but it
also becomes more targeted. In analysis, a researcher might
conclude that a technique that was noted as effective for de-
Figure 3: Example of two story payloads within Paramnesia
livering part of the story unprompted in stage 1 has stronger
evidence than one that has been discussed in stage 3 af-
ter prompting. As a methodology we would also propose
Multi-Layered Interview Methodology. Having identified that alongside the suggested interview approach an obser-
exactly what is delivered and how within our narratives, we vational study is conducted where by the researcher studies
can now propose a methodology for detecting the efficacy the player playing the game noting interaction with and
of these techniques. The premise here is to interview users comment on different story payloads. This will then provide
with questions connected to each story payload to detect a three layered data set for analysis from the interview along
What’s the Story? NARRATIVE AND HYPERTEXT’17, July 2017, Prague, Czech Republic
Figure 4: After deconstruction, three stages of interview capture evidence from the player on the efficacy of payloads. Starting
with general discussion that will not lead the player (stage 1) and then progressing to fabula specific (stage 2) and story specific
(stage 3) questions.
with context from the observational study. This method is [5] Jim Bizzocchi. 2007. Games and narrative: An analytical framework.
described diagrammatically in figure 4. Loading-The Journal of the Canadian Games Studies Association 1, 1
(2007), 5–10.
[6] Jeanne H Brockmyer, Christine M Fox, Kathleen A Curtiss, Evan McB-
5 CONCLUSIONS AND FUTURE WORK room, Kimberly M Burkhart, and Jacquelyn N Pidruzny. 2009. The
In this paper we have presented a multi-layered method- development of the Game Engagement Questionnaire: A measure of
engagement in video game-playing. Journal of Experimental Social
ology for the evaluation of interactive digital storytelling Psychology 45, 4 (2009), 624–634.
technique efficacy. We have given context for our work in [7] Dennis Coon and John O Mitterer. 2012. Introduction to psychology:
the form of the games Paramnesia and Blitz, and detailed a Gateways to mind and behavior with concept maps and reviews. Cengage
multi-layered approach to deconstruction of an interactive Learning.
narrative and then an evaluation methodology that this de- [8] Alena Denisova, A Imran Nordin, and Paul Cairns. 2016. The Conver-
gence of Player Experience Questionnaires. In Proceedings of the 2016
construction enables. This methodology potentially enables Annual Symposium on Computer-Human Interaction in Play. ACM,
game and interactive fiction analysis to go beyond criticism 33–37.
and user experience and explore the efficacy of different tech- [9] Charlie Hargood, Verity Hunt, Mark Weal, and David E. Millard. 2016.
niques which are increasingly varied in this fundamentally Patterns of Sculptural Hypertext in Location Based Narratives. In
experimental medium. Proceedings of the 27th ACM Conference on Hypertext and Social Media.
ACM, New York, NY, USA.
While this position paper provides a proposition for the [10] Henry Jenkins. 2004. Game design as narrative. Computer 44 (2004),
technique, future work needs to validate its efficacy as a 53.
form of evaluation. Initial studies on the two games detailed [11] Charlene Jennett, Anna L Cox, Paul Cairns, Samira Dhoparee, Andrew
are underway and initial researcher feedback suggests that Epps, Tim Tijs, and Alison Walton. 2008. Measuring and defining the
this methodology is effective. One researcher stated ’the experience of immersion in games. International journal of human-
computer studies 66, 9 (2008), 641–661.
multi-layered approach allows me to get a more in-depth [12] Matthew Wilhelm Kapell. 2015. The Play Versus Story Divide in Game
understanding of what technique was more influential, what Studies: Critical Essays. McFarland.
design implementation communicated the technique etc’. [13] Irving Kirsch and Wayne Braffman. 2001. Imaginative suggestibility
However the subtle nature of some of these techniques con- and hypnotizability. Current directions in psychological science 10, 2
tinues to make them particularly challenging to evaluate. (2001), 57–61.
[14] Stacey Mason. 2013. On Games and Links: Extending the Vocabulary
of Agency and Immersion in Interactive Narratives. Springer.
REFERENCES [15] Marshall McLuhan. 1994. Understanding media: The extensions of man.
[1] Ernest Adams. 2014. Fundamentals of Game Design (3rd ed.). New MIT press.
Riders Publishing, Thousand Oaks, CA, USA. [16] Richard M Ryan, C Scott Rigby, and Andrew Przybylski. 2006. The mo-
[2] Mieke Bal. 2009. Narratology: Introduction to the theory of narrative. tivational pull of video games: A self-determination theory approach.
University of Toronto Press. Motivation and emotion 30, 4 (2006), 344–360.
[3] Mark Bernstein. 1998. Patterns of Hypertext. In Proceedings of the [17] Chun Siong Soon, Marcel Brass, Hans-Jochen Heinze, and John-Dylan
Ninth ACM Conference on Hypertext and Hypermedia. ACM, New York, Haynes. 2008. Unconscious determinants of free decisions in the
NY, USA, 21–29. human brain. Nature neuroscience 11, 5 (2008), 543–545.
[4] Mark Bernstein. 2011. Can we talk about spatial hypertext. In Proceed- [18] Robert B Zajonc. 2001. Mere exposure: A gateway to the subliminal.
ings of the 22nd ACM conference on Hypertext and hypermedia. ACM, Current directions in psychological science 10, 6 (2001), 224–228.
103–112.