Evaluation of experimental digital narrative often focuses on the overall user experience. While this is important, we recognise the need for more granular forms of evaluation to measure the e cacy of individual digital narrative delivery techniques. These continue to grow in variety as authors explore di erent approaches to telling stories using interactive media. In this paper we propose a multi-layered evaluation methodology based on the principle of deconstructing an interactive narratives internal fabula and story. This is followed by three separate stages of interview for collecting evidence of the e cacy of di erent techniques used within story payloads to deliver content. This proposed methodology shows early promise, and potentially provides a means to identify the individual e cacy of techniques within a wider digital narrative.
• Human-centered computing → Hypertext / hypermedia; Copyright held by the author(s).
Digital narrative is, by its nature, experimental. How authors
tell stories through technology has been changing rapidly
following continued exploration of the possibilities and the
poetics of a range of digital mediums. Interactive ction,
hypertext narrative, RPGs, and computer games have all
experimented with di erent delivery mechanisms for stories
that play with both the interactive nature of digital
narrative as well as its dynamic qualities to create nonlinear
narratives. These delivery techniques include hypertextual
patterns of narrative structure [
While there has been a range of work exploring these
di erent possibilities there is somewhat less work in the
area of bespoke evaluation methodologies for the narrative
e cacy of these di erent techniques. Existing evaluations
of these approaches normally follow the path of traditional
user experience studies - often seeking to deconstruct the
usage of a system and its impact on the user [
In this paper we propose a multi-layered approach to the evaluation of the e cacy of digital narrative delivery techniques to deliver story. We present two games we have created as a group, and how we have deconstructed their narrative design in order to form an evaluation methodology that goes beyond user experience and seeks to explore the e cacy of individual techniques to deliver story elements. 2
This work draws on both techniques for digital narrative delivery (in surveying the di erent forms that might be evaluated) and interactive entertainment evaluation (existing methodologies that might be applied).
Hypertext ction such as Joyce’s ’afternoon, a story’ and
Moulthrop’s ’Victory garden’, has explored a range of
narrative delivery methods, often through di erent hypertext
structures such as the calligraphic patterns originally
explored by Bernstein [
While some modern game narratives leverage the techniques
common to more traditional mediums, such as lm or text,
there are experiments in new delivery mechanisms to deliver
narrative in less traditional ways that leverage the interactive
nature of the form. The idea of ’Mechanics as Metaphor’ [
Other game narrative designers prefer to avoid more
traditional grand narratives and instead use di erent narrative
delivery techniques that focus on the game world delivering
its story through a sequence of micro narratives embedded
into the world itself [
Existing game evaluation, as with most application
evaluation, is user experienced focussed. In her summary Denisova
et al. [
A pair of narrative centric games under recent development (which employ some of the delivery techniques discussed) as part of an exploration into the e cacy of these techniques was the primary motivator for this work. To give context we o er the following descriptions of these games Paramnesia and Blitz (as shown in gure 1’s screenshots).
The premise of Paramnesia is to use subconscious psychological techniques to control player actions as well as convey a strong narrative; using these subconscious techniques the theory is to better convey narrative and object points to players, resulting in a closer cohesion of scripted narrative and the player’s interpretation of story. These subliminal techniques are similar in premise to conventional forms of environmental storytelling in that the narrative emerges through the delivery of small sections often through the space the player explores - though they are more heavily couched in related psychological theory. Paramnesia is a rst person atmospheric game set in an apartment block and subway. The player is convinced to nd, set and detonate a bomb in an empty subway, under the intention of it ending a communist uprising currently taking place in America. After the detonation of the bomb, it becomes apparent that the player was manipulated to perform this terrorist attack under false pretenses. There were 4 psychological techniques employed within the level: Conditioning, Compliance, Mere Exposure and Suggestion.
Conditioning [
Blitz is a rst person interactive story set in 1940s London. You play as a young boy who has been abandoned by his parents and has woken up in a version of his childhood house that has been separated from society. Exploring the house, the player discovers what happened to their parents. Blitz makes use of both Environmental Storytelling, and the notion of ’Mechanics as Metaphor’. Environmental storytelling being the technique that is more heavily used.
Throughout the rst area there are clues and hints to the location, time and events that relate to the house you are in. Gra ti on the walls as well as a range of propaganda posters reveal somewhat happened to the father and gives an idea of the date that the game is set. There are other subtler hints, such as the photographs on the wall; in one of the images, a young boy is wearing glasses, and in the next area, there are a pair of glasses on a crate, that indicate you are the boy in that photograph.The use of ’Mechanics as Metaphor’ is somewhat lighter. Player movement speed is slowed, and the camera height of the player lowered and increased in FOV so that the player felt surrounded by everything within vision to highlight the overwhelming notion of events beyond the protagonist. 4
In order to explore the e cacy of the techniques within the games, we required a methodology that was granular enough to focus in on the impact of the delivery of individual pieces of story. We require a method that can consider the coherency of delivery (did the technique successful deliver content in a way the audience understands), and to what degree was its e cacy di erent to other techniques.
To begin to understand the answer to these questions we
need to rst identify what techniques are used where and
to deliver what. This process is e ectively identifying what
Narratologists such as Bal [
As an example the code to the security door in Paramnesia is a fabula element, it is delivered to the player in ten di erent payloads through out the game. These payloads all make use of the delivery technique of ’Mere Exposure’ as part of environmental storytelling. It is sometimes packaged into a payload with other fabula elements, for example the posters also communicate the anti-communist themes of the piece. Consequently we now have a mapping from ’fabula’ elements, to delivery payloads as part of the story which uses a technique. This example is depicted in gure 3.
Multi-Layered Interview Methodology. Having identi ed exactly what is delivered and how within our narratives, we can now propose a methodology for detecting the e cacy of these techniques. The premise here is to interview users with questions connected to each story payload to detect if the payload successfully delivered its content and how this was received by the user. This way we can evaluate the techniques based on how successful they were at content delivery in the context of the payloads in which they were used. It is not, however, suitable to simply ask a player ’did you notice element <x>?’ as the question itself informs the player before they give their answer. As such we propose a three layer evaluation methodology for subsequent player interviews: (1) User-driven discussion - The user is encouraged to lead the discussion by discussing the story and its setting general terms without guidance to particular parts. This collects evidence for all story payloads without the player being lead to the signi cance of particular story elements through questioning. The player may mention particular elements and consequently demonstrate some e ect for the payloads connected to those elements. (2) Interviewer-driven discussion - Fabula - The interviewer now delivers pre-constructed questions for each fabula element to explore whether this content was successfully delivered to the player and how it was interpreted. This stage may lead the player to elements but not their delivery techniques, and ensures what content was or wasn’t received by the player. (3) Interviewer-driven discussion - Story - The interviewer now delivers pre-constructed questions for each story payload. This is the nal layer of questioning and directs the players attention to speci c parts of the story delivery. This may heavily lead the answer, but won’t a ect previous sections and captures speci c e cacy of each delivery technique.
For each stage of this method the interviewer does not need to ask speci c questions that have already been covered by evidence in a previous stage. For example, if evidence is gathered to support the player understood a given fabula element in stage 1, then the question for that element does not need to be asked in stage 2. As the interview progresses the evidence for player engagement becomes weaker due to their being potentially lead by the lines of inquiry, but it also becomes more targeted. In analysis, a researcher might conclude that a technique that was noted as e ective for delivering part of the story unprompted in stage 1 has stronger evidence than one that has been discussed in stage 3 after prompting. As a methodology we would also propose that alongside the suggested interview approach an observational study is conducted where by the researcher studies the player playing the game noting interaction with and comment on di erent story payloads. This will then provide a three layered data set for analysis from the interview along with context from the observational study. This method is described diagrammatically in gure 4. 5
In this paper we have presented a multi-layered methodology for the evaluation of interactive digital storytelling technique e cacy. We have given context for our work in the form of the games Paramnesia and Blitz, and detailed a multi-layered approach to deconstruction of an interactive narrative and then an evaluation methodology that this deconstruction enables. This methodology potentially enables game and interactive ction analysis to go beyond criticism and user experience and explore the e cacy of di erent techniques which are increasingly varied in this fundamentally experimental medium.
While this position paper provides a proposition for the technique, future work needs to validate its e cacy as a form of evaluation. Initial studies on the two games detailed are underway and initial researcher feedback suggests that this methodology is e ective. One researcher stated ’the multi-layered approach allows me to get a more in-depth understanding of what technique was more in uential, what design implementation communicated the technique etc’. However the subtle nature of some of these techniques continues to make them particularly challenging to evaluate.