=Paper=
{{Paper
|id=Vol-2618/paper6
|storemode=property
|title=Wearable Theatre - Immersive Storytelling and Theatrical VR
|pdfUrl=https://ceur-ws.org/Vol-2618/paper6.pdf
|volume=Vol-2618
|authors=Markus Wintersberger,Weiss Marcus Josef,Thomas Wagensommerer,Georg Vogt,Colleen Rae Holmes,Ulrich Kühn,Christian Munk,Julia Püringer
}}
==Wearable Theatre - Immersive Storytelling and Theatrical VR==
https://ceur-ws.org/Vol-2618/paper6.pdf
Wearable Theatre - Immersive Storytelling and Theatrical VR
Markus Wintersberger*, Thomas Wagensommerer, Georg Vogt, Julia Püringer, Christian Munk, Ulrich Kühn
St. Pölten University of Applied Sciences
Marcus Josef Weiss, Colleen Rae Holmes
Open Acting Academy - Conservatory for Acting
ABSTRACT During the course of the research additional experiments were
The following paper presents the main findings of the three-year conducted, and the findings of those experiments introduced in a
art-based research project Wearable Theatre - The Art of broader theatrical setting. Live Performances were recorded,
Immersive Storytelling. It introduces the basic art-based streamed and reworked. The following paper gives insight into,
research premise of the project, the technological framework and and an overview of the project and addresses specific questions
its artistic concept. It also presents the prototype of a project encountered by the research team. The project was featured
museum or interactive art-repository that offers insights into the prominently (amongst other events) in the Volkstheater Vienna as
project’s outcome. part of the Digital Natives Festival 2018 and invited as an
international guest to the New Forms of Theatre Festival in
Keywords: VR, immersive media, storytelling, XR, experimental Warsaw in 2019. Chapter 1 gives an overview of the project and
media, art. introduces the idea of artistic research being collective aesthetic
work, inherent to the project itself. Chapter 2 addresses the
evolving state of VR technology during the 3-year research
Index Terms: Virtual reality, digital archive, art-based research
period, the artistic concept, and the fundamental groundwork of
adapting literary material for theatrical VR (illustrated explicitly
1 INTRODUCTION in the stage production ÆON), and the very different forms of
The art-based research project, Wearable Theatre - The Art of spatial perception inherent in this form of art. The created
Immersive Storytelling, explored, during its 3-year span, the artefacts were subject to an open dissemination strategy
potential of storytelling in VR. The project team consisted of (described in chapter 2.3). Chapter 3 reviews the findings,
artists and researchers from the St. Pölten University of Applied suggests further research efforts towards an XR Theatre and
Sciences and the Open Acting Academy in Vienna. [1] This presents the first steps towards a digital museum or repository for
research project identified and structured aesthetic variables and digital artefacts - as well as aesthetic phenomena. This can also be
applied them to twelve experimental settings, each focusing on a seen as part of an open-source aesthetic toolbox that might be
specific aspect of VR sensory experience. The goal was to artistically appropriated. Finally, the consolidation of the
understand and master the visual, acoustic and atmospheric project’s findings is discussed, including how these, along with
possibilities that 360° VR offers. The aim was to merge the the initial mission to provide useful artefacts and artistic practices,
aesthetic variables with a narrative and produce a unique has led to new research questions. Because the research-team
immersive VR art experience. Three specific literary works were chose initially to concentrate specifically on the questions posed
chosen to provide this narrative foundation for the experimental by content and artistic forms of expression of the AV technology,
steps. (See chapter 2.1) particularly regarding immersion, perspective, orientation, focus,
and attention, it was preferable in the first prototypical
experiments, to experiment with low-budget technology and its
limitations. The video studio made available to the research team
by the St. Pölten University of Applied Sciences, with its
elaborate studio technology, provided an ideal spatial reference
point throughout, and, particularly at the beginning of the
research, functioned as a prototypical laboratory.
2 WEARABLES AND THE THEATRICAL USE OF VR
* markus.wintersberger@fhstp.ac.at
2.1 State of technology and the research settings
The complex technological challenges relating to 360° AV
production, in particular concerning the image quality, resolution
and low light capture abilities were continuously shifting
parameters during the project. Technology saw vast improvements
during the project durations, enabling additional scenarios that
were not possible in the initial outline throughout the course of the
project. In 2016, at the time the project proposal was originally
formulated, 360° AV technology was only available in a very
early and immature version. An experimental “uncertainty” was,
thereby, calculated into the proposal itself and into the artistic-
Copyright © 2020 for this paper by its authors. Use permitted under
technological vision of the research, and, with foresight of the
Creative Commons License Attribution 4.0 International (CC BY 4.0). coming developmental steps as regards 360° AV, VR and that of
XR in the years between 2017 and 2020, was imagined on an On the formal level a foundation of texts was selected, all of
evolutionary trajectory. which were written in the literary form of reported speech (a first-
In 2014 / 2015 the subject of VR appeared on the horizon, person narrative), thereby providing a subjectively filtered
hyped by Facebook, Oculus Rift and HTC Vive as showing great narrative perspective. The variability and fluctuations in the
technological promise. [2] [3] The 360° AV technology as an personal-narrative reporting (simultaneously a part of the literary
“intermediary” was supposed to fulfill a hybrid function: To fascination) is directly analogue to the experiential spectra of
capture the presence of the live experience and merge it with the applied XR technology.
virtual realm and its potential channels of communication such as On the contentual level, all foundational texts stem from the
streaming and presence in social networks. same dramatic premise. Both “The Fall” (Albert Camus) [5] and
The research team has tried throughout the entire time-frame of “Homo Faber” (Max Frisch) [6] as well as “Demons – The
3 years (2017-2020) to process the specific question as to the Confession of Stavrogin” (Fjodor Dostoyevsky) [7] describe with
connection between the “real” and the virtual, and is currently extreme psychological precision, the protagonists’ search for
working on a subsequent research proposal for 2021 to 2024: „XR redemption and lead the sentient subject into the state of
Theatre. The Art of Extended Storytelling“, with particular consciousness – Metanoia – a remorse of the deepest and most
emphasis on new technology such as photogrammetry and Real- immediate kind when confronted with mortality.
time 3D. The unifying literary premise of these first-person narratives
“With its predefined experiments, yet open outlook the project could be summarised as: Independent of the intensity of guilt the
told itself a tale of unending possibilities as the logic of the pursuit of personal purification by both being and consciousness
experiment. Experimentation, theatre, finding meaning and is inescapable.
inventing meaning determine a Wearable Theatre; determine also The first experiments led to the recognition that in the
the subsequent extension of the ideas into a digital sphere, the application and integration of the “Subjective Goggles” into a
indomitable and yet so fragile digitality. The uncanny is hidden in theatrical form, the aesthetic and contentual Theories of
the niches between these poles; hidden in the search for the Expressionism in the 20th century provided (in this context) a
connections between reality and reality.” Initially, the practice- valuable reference point.
led research aspects were explored using the available technology These first critical elucidative insights were condensed into the
in a studio setting. [4] fragment “Nachtgerüche” (Night Rumours), which was premiered
The spatial “neutrality” of the studio, which, as a result of the at the Festival ‘Die Kunst der Nachbarschaft’ (Neighbourhood
insights gained during the research process, developed into an Art) in cooperation with the Vienna Volkstheater. The festival
“immersive media laboratory”, is an almost mandatory itself was consequently awarded the Dorothea-Neff Special Prize
prerequisite for the examination of the technological, artistic and for boundary-breaking theatre work.
scientific questions posed by the research project “Wearable Building on this, the focus was laid on the planning,
Theatre. The Art of Immersive Storytelling.” dramatization and application of the postulated experimental
The simple “Black Box” situation of the studio, whose almost narrative patterns and forms of staging.
permanent changeability through real-virtual scenographic The resulting experiment was a completely dramatized and
designs and almost unlimited changes in lighting, form a good staged theatrical prototype entitled ‘ÆON’, which was premiered
basis for content-driven research perspectives on “Space-time- on the 31st of May 2019 on the main stage of the
theatre”. These features were deliberately included in the planning Volkstheater,Vienna.
before the structuring of the application was begun, their actual It is important to explain the processes involved in the
potential, however, could only be proven and developed development of ‘ÆON’, from both a dramatic and a directing
exploratorily through the “Wearable Theatre” project itself. perspective.
The amalgam, comprising acting, action, interaction, immersion
and light-projection, soon formed a crucial reference point, also as 2.2.1 ÆON – the transcendence of the “I” (Self) – The
regards the hypotheses set out in the research proposal. dramaturgical experiment from the perspective of
To what extent is it possible to conceive, implement, record and the author.
perform theatrically in a stage-setting based on real artefacts and In terms of the dramaturgical exposition, the point of departure
persons in a 360° AV dimension? was the first-person narrator (the I-narrator), or, to be more
Which specific 360° AV technological challenges emerge as a precise: the taking apart of the self-aware and reflecting textual
result of these assumptions and circumstances? “Self” and dividing it into its opposing forces and effective
To this precise end, the team, by careful use of the literary prose mechanisms.
examples specified in the proposal and the conscious decision to A context was created for the inner psychological processes of
focus on short action sequences extrapolated from this literary suppression, dawning (or forced) awareness up to and including
source-material, was able to define a clear artistic-scientific the search for forgiveness, on which basis a dramatic-stage
“language” in accordance with the project proposal. progression was developed.
The splitting apart and character-specific personification of the
2.2 The investigation of 360° VR in the actualisation of aspects of awareness within the consciousness of a protagonist
literary material and its conception as a seems not only to be a justifiable narrative strategy, but one of the
theatrical genre demands and challenges inherent in combining the different levels
The initial artistic hypothesis of the research project was to of narrative potential within a single entity in a coherent totality.
illustrate effectively, on the basis of a specific literary format, the It is also a logical analogy to the ambivalence regarding
rationale in employing the 360° VR-Medium in theatre, as well as immersive VR-Media as to their uses and disadvantages. The
its contentual and aesthetic applications. These experiments depth of the layers of psychological experience and meaning
would, according to the hypothesis, have the potential to expand made possible by the application of this technology is magnified
the experiential narrative patterns and staging forms of theatre by and demonstrated specifically through ÆON.
a further dimension. ÆON is thereby, based on the hypothesis of the PEEK research
There were two determining parameters for this: one formal and project “Wearable Theatre. The Art of Immersive Storytelling”, a
the other contentual.
drama about the potential conflict of an individual (protagonist)
told from the perspective of his/her inner “Self”.
2.2.2 ÆON – the transcendence of the “I” (Self) – The
theatrical experiment from the perspective of the
director.
When, as in the case of ÆON, the complete awareness of the
protagonist – the inner “I” (Self) – is dissolved, it is necessary in a
theatrical implementation to establish a clear connection between
the dissolution and disintegration process and the universe of the
physically real theatre in order to allow the impact of the drama to
unfold.
Through this, reflections on the technical aspect of staging Figure 2: VR-montage generated from the performance at the
arose, stemming from two affiliated questions: “New-Forms-of-Theatre” festival.
What if the entire theatre in the totality of its (physical)
functions and spatial structures were an analogy for human
consciousness? 2.3 The Triad of perception
What if the VR-Perspective was an elementary catalyser for the The radical changes inherent in the application of new technology
interfacing of these physical structures and the scenic-staged to the theatrical medium has forced a re-evaluation and
interaction? reinvention of the hitherto, defining components. The roles of
The directorial onus was to disassemble the totality of the performer and audience, performative space, text, light and
theatrical space as an in-itself-interconnected organism, to scenographic elements are in the process of being deconstructed
reorganise and codify the individual parts, and, by means of the and redefined according to the nature and possibilities of new
staged arc of the narrative, dissolve this organisation and connect media, specifically: immersive Extended Virtual Reality
it anew, bringing the levels together in a single multi-layered technology.
space at the high point of the drama. Immersive XR theatre implies the re-conjoining of the world of
For these alternating interactions of the individual “Selves” the drama and that of the spectator. The melding of the entities of
analogies were created in the form of individual theatrical spaces spectator and performer, auditorium and stage, enables an
which were allocated and interconnected. alternative option allowing the spectator to be essentially a part of
The following analogy was finally used as the foundational the dramatic narrative, navigating through the fabric of the XR
spatial structure for the staging of ÆON: drama and by his/her choices experiencing the event individually
The area below the stage and the apparatus and construction of and uniquely.
the revolving stage mechanism represented analogously the The theatre as we see it traditionally is a place of multiple
human subconscious. orientation: the spaces we see and those of which we are aware
The main stage and the entire lighting plot from the central and do not see. XR technology affords ways of presenting and
perspective was representative of actual consciousness. extending time and space which transcend and escape the
The transformation and refining of actual consciousness into conventions of the physical stage, the confines of a defined
Dream-consciousness was staged by means of extending the theatrical location and the limitations of linear narrative. The
action into the auditorium and the further refining and elevation potentially infinite dimensions of cyberspace can be accessed in
into the aware / visionary “I” (Self) illustrated by the churning relation to, and in opposition to the defined physical space. Spaces
firmament of the theatre ceiling. within the space, unfolding around and within each other can host
scenery, performers and all the elements and aesthetic options
comprising XR Scenography. Expanded space establishes
different perspectives for the participants by manipulating their
points of view or sense of situatedness.
The perception of space has been defined as a triadic model:
the dimensional (the actual spatial reality of the individual), the
representational (that which is described or represented in models
or drawings) and the experiential (which is entirely personal to the
individual). Whereby the action of drama takes place in
representational space, in XR the artistic elements combine and
interact in experiential space, enabling ongoing processes of
worlding to operate through discrete acts or sensory and narrative
inference which include haptic and acoustic proxemics and pre-
existing orders of knowledge.
The crafting, configuration and navigation of space, place
Figure 1: 360° technology integrated into the set on stage. orientation and levels of sensory input in relation to the
performance event incorporates not only the environment of the
protagonist but also that of the visitor. Both participants become
interactive, vital aspects of the scenographic concept, inhabiting
the spatial imaginary.[8].
The XR Drama with its malleable and unpredictable suspension
of situatedness and atypical interrelationships transcending
spatial, positional and experiential boundaries, is an entirely
personal, individual and unique experience; the foundation for a
new level of theatrical artistry “The imaginary which fills the leads the user in a first-person perspective through an
empty spaces of thought.” [9]. environment of new information, insight and increasing
knowledge, but also adds an overlap of environment and
2.4 Open dissemination information to intensify and validate the overall experience, both
The aesthetic phenomena created during the project are not on a perceptual and rational level.
limited to the VR sequences available as 360° Video. All the steps
3.1.2 A toolbox (outlook)
in the research process have been thoroughly documented,
providing a vast repository of aesthetic representations, artefacts As with every museum which has a basement, depot, rear
and interventions. Next to the open access live-streams, which chamber, etc., this application also contains special settings, that
exist as recordings on the project’s YouTube channel, an immense hold concrete, practical data which the user may download and
number of photos, videos, 3D-scans, photogrammetric images and keep. It is in these areas that most of the actual archiving is done
other digital media artefacts has been created, and uploaded to and this archive should be kept fully accessible without
public repositories and social networks such as instagram, restrictions. Data such as point clouds, models, videos, images,
facebook, YouTube and sketchfab. One of the last steps in the papers, evaluations, figures, can be stored in the archive and
project has been the creation of a digital project-archive, which, accessed via application. This application prompts the user to find
true to the nature of the project, invites users to navigate the and use these areas in an open source and creative way. The user
findings and experimental material gathered by the may implement the data in her/his own body of work and thereby
researchers.[10] create an expanding universe of virtual, but theatrical
experimentation, that reflects on itself.
3 OUTCOMES - WHAT REMAINS TO BE SEEN? To ensure broad accessibility, this application is produced as a
WebGL Project, to be hosted via HTML5 in common browsers
The Wearable Theatre project has managed to explore the
via the web. All the accessible data is stored in common formats,
qualities of VR in the abovementioned settings. All the parameters
such as *.obj, *.fbx, *.ply, *.txt, *.jpg, *.wav, etc.
and variables defined in the proposal have been researched.
Beside the concepts and stage works outlined above, the project
produced a vast number of artefacts. By rearranging and sharing
them it has become very apparent that these are an essential part
of the project’s outcome, not only as documentation of the process
or mediatised artefacts of performative acts, but as actual aesthetic
knowledge derived from the project itself. In addition to the use of
established platforms the team has also created the first prototype
of a virtual archive in which all artefacts are arranged and can be
navigated.
3.1 Living archive and toolkit
To document the artefacts, that were produced in the course of the
project and to preserve their live and real-time capability, an
online application that provides background information and
invites the user to select her/his own way in terms of the use of
Figure 3: Approaching the museum in virtual space.
space and progression has been created to facilitate the possibility
to interact with real-time settings. The user can rest in a setting,
while it keeps evolving around her/him; additionally, the user can
navigate in every direction - front [W], back [S], left [A], right
[D], up [E], down [Q]) and rotate his/her virtual head ([MOUSE
CLICK + DRAG]. Special areas of context can be found in each
setting. These areas hold objects, digital fragments, texts, sounds,
videos, images focused on certain common themes, events,
periods of research, installations, performances etc. The areas are
arranged to overlap, so that they interact with each other,
overriding and multiplying, but always based upon the perspective
chosen by the user. The settings also contain elements of real
surroundings evolving in real-time, that change quite radically
over time. Thus, if the user decides to let the application run for
some time, the artefacts and areas will grow like a virtual garden.
3.1.1 An interactive museum
This application can be considered a virtual, interactive museum.
The user strolls through the settings and inspects the artefacts,
their background information and the evolving space itself. The
user may decide to change the settings when arriving at certain
trigger points, enabling the transportation to another, new setting
(and context). These settings can be considered as levels (when
choosing a game-based approach) or floors (when choosing a Figure 4: Inside the archive: Recording of the Red Bar /
museum reference). This virtual museum applies the approach of Volkstheater Vienna.
a so-called Walking Simulator (genre of video games) which
3.1.3 Technical workflow
To create the individual settings or levels/floors, 3D objects are
generated throughout the project in a photogrammetric workflow
within 3DF Zephyr Lite, that allows up to 500 pictures (in this
case mostly frames from a video) for the reconstruction of the
objects. Dense point clouds and high-definition textures are also
achieved in the process.
These objects are then imported into the game-engine Unity for
further development. This application is hosted in the Universal
Render Pipeline (URP) of Unity 2019.3 and upwards. The URP is
a prebuilt Scriptable Render Pipeline that is optimized for a wide
range of target platforms. From Windows and UWP, Mac and
iOS, Android, XBox One, PlayStation4, Nintendo Switch,
WebGL to all current VR platforms. While this ensures high
flexibility in deploying, the main focus remains to target WebGL
and for special cases SteamVR.
Figure 7: Approaching a photogrammetry in the archive.
4 CONCLUSION AND OUTLOOK
Not only the VR technology on which the project's practicalities
hugely depended, but also the discussion on art-based research
and its benefits has intensified in the past years.
Venturing to different venues with the artefacts already created in
previous steps proved extremely beneficial and informed what
Gesa Ziemer has called creative collectivity (kreative
Kollektivität.). [7] For Ziemer the quality of such an “unlikely
assembly” (unwahrscheinliche Versammlung) (ibid.) lies exactly
Figure 5: 3D scan of the “Fuehrerbunker” installation in the archive. in the cooperation of individuals that would not likely have
cooperated in other settings.
In Unity the objects are arranged and procedural transformation Wearable Theatre, due to its constant iterative production of
(position and rotation) is applied to generate an ongoing change of performances and media artefacts was able to not only incorporate
the spatial setting. Furthermore, the objects are edited with the use people, but also its aesthetic artefacts into this assembly. Things
of custom-built shaders made in Unity’s shader graph, that allows such as concepts, recordings, fragments of code and interfaces
a node-based creation of physically based shaders. This process that had already been created in the project formed the basis of the
enables the possibility to change the appearance and behavior of next steps at a different venue, confronting not only the human
the object quite radically over time and based on the perspective protagonists of the project with a new setting, but also the things
of the user. they brought along.
This application is built for WebGL 2.0 which consists of In their Manifesto of artistic research Henke et al. [8] reaffirm the
control code written in JavaScript and shader code written in idea of a specific knowledge that can be derived from art-based
OpenGL ES3, ensuring the provision of complex, evolving research that is located beyond the dualism of theory and practise.
settings even on low standard devices. In contrast to the discursive modes, a language of things or
aesthetic artefacts operates quite differently. “They think
multimodally, compositionally, and in many media
simultaneously” [8] p. 39. Besides exploring and incorporating the
artistic variables of a wearable theatre, the project also created a
new outlook on theatre itself, as a mode of producing an aesthetic
kind of knowledge.
”Aesthetic practises map out non-scientific epistemologies by
drawing their form of knowledge not from syntheses but rather
from the sensuous relations of non-predicative conjunctions in
which their insights merge and coincide.” (ibid.) Theatre, as the
historical paradigm of production of such knowledge, seems to be
the ideal site to explore the further possibilities in aspects of XR /
Extended Reality.
REFERENCES
[1] Wearable Theatre. Retrieved April 10, 2020 from
https://wearabletheatre.fhstp.ac.at/en
[2] Deeming, Skot. The new VR. Broken Pencil, no. 63, Spring 2014, p.
9
[3] Dylan Bushell-Embling, Dylan. "VR content market to swell to
Figure 6: Navigating the artefacts in the archive.
$3.3b by 2020." Telecom Asia, December 2016, 6. Gale General
OneFile. Retrieved from: https://link-gale-
com.uaccess.univie.ac.at/apps/doc/A502420884/ITOF?u=43wien&si
d=ITOF&xid=f3847da3.
[4] Experiment. Retrieved April 10, 2020 from:
https://wearabletheatre.fhstp.ac.at/en/experiment/
[5] Albert Camus. Der Fall, Rowohlt, ger. 1968
[6] Max Frisch. Homo Faber. Suhrkamp 1968
[7] Fjodor Michailowitsch Dostojewskij. Die Dämonen. Deutscher
Taschenbuch Verlag GmbH & Co.KG, GER 1977
[8] Edward W. Soya. Thirdspace, Blackwell, USA 1996, p.56
[9] Rachel Hann. Beyond Scenography, Routledge, 2019, p.24
[10] Archive Retrieved from https://wearabletheatre.fhstp.ac.at/en/archive
[11] Gesa Ziemer. Kollektives Arbeiten, In: Jens Badura, Selma Dubach,
Anke Haarmann, Dieter Mersch, Anton Rey, Christoph Schenker,
German Toro Pérez: Künstlerische Forschung – Ein Handbuch,
Diaphanes, Ger 2015
[12] Silvia Henke, Dieter Mersch, Nocolaj van der Meulen, Thomas
Sträsle, Jörg Wiesel: Manifesto of artistic research. Diaphanes, CH
2020.