The following paper presents the main findings of the three-year art-based research project Wearable Theatre - The Art of Immersive Storytelling. It introduces the basic art-based research premise of the project, the technological framework and its artistic concept. It also presents the prototype of a project museum or interactive art-repository that offers insights into the project's outcome.
The art-based research project, Wearable Theatre - The Art of
Immersive Storytelling, explored, during its 3-year span, the
potential of storytelling in VR. The project team consisted of
artists and researchers from the St. Pölten University of Applied
Sciences and the Open Acting Academy in Vienna. [
During the course of the research additional experiments were conducted, and the findings of those experiments introduced in a broader theatrical setting. Live Performances were recorded, streamed and reworked. The following paper gives insight into, and an overview of the project and addresses specific questions encountered by the research team. The project was featured prominently (amongst other events) in the Volkstheater Vienna as part of the Digital Natives Festival 2018 and invited as an international guest to the New Forms of Theatre Festival in Warsaw in 2019. Chapter 1 gives an overview of the project and introduces the idea of artistic research being collective aesthetic work, inherent to the project itself. Chapter 2 addresses the evolving state of VR technology during the 3-year research period, the artistic concept, and the fundamental groundwork of adapting literary material for theatrical VR (illustrated explicitly in the stage production ÆON), and the very different forms of spatial perception inherent in this form of art. The created artefacts were subject to an open dissemination strategy (described in chapter 2.3). Chapter 3 reviews the findings, suggests further research efforts towards an XR Theatre and presents the first steps towards a digital museum or repository for digital artefacts - as well as aesthetic phenomena. This can also be seen as part of an open-source aesthetic toolbox that might be artistically appropriated. Finally, the consolidation of the project’s findings is discussed, including how these, along with the initial mission to provide useful artefacts and artistic practices, has led to new research questions. Because the research-team chose initially to concentrate specifically on the questions posed by content and artistic forms of expression of the AV technology, particularly regarding immersion, perspective, orientation, focus, and attention, it was preferable in the first prototypical experiments, to experiment with low-budget technology and its limitations. The video studio made available to the research team by the St. Pölten University of Applied Sciences, with its elaborate studio technology, provided an ideal spatial reference point throughout, and, particularly at the beginning of the research, functioned as a prototypical laboratory.
The complex technological challenges relating to 360° AV production, in particular concerning the image quality, resolution and low light capture abilities were continuously shifting parameters during the project. Technology saw vast improvements during the project durations, enabling additional scenarios that were not possible in the initial outline throughout the course of the project. In 2016, at the time the project proposal was originally formulated, 360° AV technology was only available in a very early and immature version. An experimental “uncertainty” was, thereby, calculated into the proposal itself and into the artistictechnological vision of the research, and, with foresight of the coming developmental steps as regards 360° AV, VR and that of XR in the years between 2017 and 2020, was imagined on an evolutionary trajectory.
In 2014 / 2015 the subject of VR appeared on the horizon,
hyped by Facebook, Oculus Rift and HTC Vive as showing great
technological promise. [
The research team has tried throughout the entire time-frame of 3 years (2017-2020) to process the specific question as to the connection between the “real” and the virtual, and is currently working on a subsequent research proposal for 2021 to 2024: „XR Theatre. The Art of Extended Storytelling“, with particular emphasis on new technology such as photogrammetry and Realtime 3D.
“With its predefined experiments, yet open outlook the project told itself a tale of unending possibilities as the logic of the experiment. Experimentation, theatre, finding meaning and inventing meaning determine a Wearable Theatre; determine also the subsequent extension of the ideas into a digital sphere, the indomitable and yet so fragile digitality. The uncanny is hidden in the niches between these poles; hidden in the search for the connections between reality and reality.” Initially, the practiceled research aspects were explored using the available technology in a studio setting. [4]
The spatial “neutrality” of the studio, which, as a result of the insights gained during the research process, developed into an “immersive media laboratory”, is an almost mandatory prerequisite for the examination of the technological, artistic and scientific questions posed by the research project “Wearable Theatre. The Art of Immersive Storytelling.”
The simple “Black Box” situation of the studio, whose almost permanent changeability through real-virtual scenographic designs and almost unlimited changes in lighting, form a good basis for content-driven research perspectives on “Space-timetheatre”. These features were deliberately included in the planning before the structuring of the application was begun, their actual potential, however, could only be proven and developed exploratorily through the “Wearable Theatre” project itself.
The amalgam, comprising acting, action, interaction, immersion and light-projection, soon formed a crucial reference point, also as regards the hypotheses set out in the research proposal.
To what extent is it possible to conceive, implement, record and perform theatrically in a stage-setting based on real artefacts and persons in a 360° AV dimension?
Which specific 360° AV technological challenges emerge as a result of these assumptions and circumstances?
To this precise end, the team, by careful use of the literary prose examples specified in the proposal and the conscious decision to focus on short action sequences extrapolated from this literary source-material, was able to define a clear artistic-scientific “language” in accordance with the project proposal. 2.2
The initial artistic hypothesis of the research project was to illustrate effectively, on the basis of a specific literary format, the rationale in employing the 360° VR-Medium in theatre, as well as its contentual and aesthetic applications. These experiments would, according to the hypothesis, have the potential to expand the experiential narrative patterns and staging forms of theatre by a further dimension.
There were two determining parameters for this: one formal and the other contentual.
On the formal level a foundation of texts was selected, all of which were written in the literary form of reported speech (a firstperson narrative), thereby providing a subjectively filtered narrative perspective. The variability and fluctuations in the personal-narrative reporting (simultaneously a part of the literary fascination) is directly analogue to the experiential spectra of applied XR technology.
On the contentual level, all foundational texts stem from the same dramatic premise. Both “The Fall” (Albert Camus) [5] and “Homo Faber” (Max Frisch) [6] as well as “Demons – The Confession of Stavrogin” (Fjodor Dostoyevsky) [7] describe with extreme psychological precision, the protagonists’ search for redemption and lead the sentient subject into the state of consciousness – Metanoia – a remorse of the deepest and most immediate kind when confronted with mortality.
The unifying literary premise of these first-person narratives could be summarised as: Independent of the intensity of guilt the pursuit of personal purification by both being and consciousness is inescapable.
The first experiments led to the recognition that in the application and integration of the “Subjective Goggles” into a theatrical form, the aesthetic and contentual Theories of Expressionism in the 20th century provided (in this context) a valuable reference point.
These first critical elucidative insights were condensed into the fragment “Nachtgerüche” (Night Rumours), which was premiered at the Festival ‘Die Kunst der Nachbarschaft’ (Neighbourhood Art) in cooperation with the Vienna Volkstheater. The festival itself was consequently awarded the Dorothea-Neff Special Prize for boundary-breaking theatre work.
Building on this, the focus was laid on the planning, dramatization and application of the postulated experimental narrative patterns and forms of staging.
The resulting experiment was a completely dramatized and staged theatrical prototype entitled ‘ÆON’, which was premiered on the 31st of May 2019 on the main stage of the Volkstheater,Vienna.
It is important to explain the processes involved in the development of ‘ÆON’, from both a dramatic and a directing perspective. 2.2.1 ÆON – the transcendence of the “I” (Self) – The dramaturgical experiment from the perspective of the author.
In terms of the dramaturgical exposition, the point of departure was the first-person narrator (the I-narrator), or, to be more precise: the taking apart of the self-aware and reflecting textual “Self” and dividing it into its opposing forces and effective mechanisms.
A context was created for the inner psychological processes of suppression, dawning (or forced) awareness up to and including the search for forgiveness, on which basis a dramatic-stage progression was developed.
The splitting apart and character-specific personification of the aspects of awareness within the consciousness of a protagonist seems not only to be a justifiable narrative strategy, but one of the demands and challenges inherent in combining the different levels of narrative potential within a single entity in a coherent totality.
It is also a logical analogy to the ambivalence regarding immersive VR-Media as to their uses and disadvantages. The depth of the layers of psychological experience and meaning made possible by the application of this technology is magnified and demonstrated specifically through ÆON.
ÆON is thereby, based on the hypothesis of the PEEK research project “Wearable Theatre. The Art of Immersive Storytelling”, a drama about the potential conflict of an individual (protagonist) told from the perspective of his/her inner “Self”. 2.2.2 ÆON – the transcendence of the “I” (Self) – The theatrical experiment from the perspective of the director.
When, as in the case of ÆON, the complete awareness of the protagonist – the inner “I” (Self) – is dissolved, it is necessary in a theatrical implementation to establish a clear connection between the dissolution and disintegration process and the universe of the physically real theatre in order to allow the impact of the drama to unfold.
Through this, reflections on the technical aspect of staging arose, stemming from two affiliated questions:
What if the entire theatre in the totality of its (physical) functions and spatial structures were an analogy for human consciousness?
What if the VR-Perspective was an elementary catalyser for the interfacing of these physical structures and the scenic-staged interaction?
The directorial onus was to disassemble the totality of the theatrical space as an in-itself-interconnected organism, to reorganise and codify the individual parts, and, by means of the staged arc of the narrative, dissolve this organisation and connect it anew, bringing the levels together in a single multi-layered space at the high point of the drama.
For these alternating interactions of the individual “Selves” analogies were created in the form of individual theatrical spaces which were allocated and interconnected.
The following analogy was finally used as the foundational spatial structure for the staging of ÆON:
The area below the stage and the apparatus and construction of the revolving stage mechanism represented analogously the human subconscious.
The main stage and the entire lighting plot from the central perspective was representative of actual consciousness.
The transformation and refining of actual consciousness into Dream-consciousness was staged by means of extending the action into the auditorium and the further refining and elevation into the aware / visionary “I” (Self) illustrated by the churning firmament of the theatre ceiling. The radical changes inherent in the application of new technology to the theatrical medium has forced a re-evaluation and reinvention of the hitherto, defining components. The roles of performer and audience, performative space, text, light and scenographic elements are in the process of being deconstructed and redefined according to the nature and possibilities of new media, specifically: immersive Extended Virtual Reality technology.
Immersive XR theatre implies the re-conjoining of the world of the drama and that of the spectator. The melding of the entities of spectator and performer, auditorium and stage, enables an alternative option allowing the spectator to be essentially a part of the dramatic narrative, navigating through the fabric of the XR drama and by his/her choices experiencing the event individually and uniquely.
The theatre as we see it traditionally is a place of multiple orientation: the spaces we see and those of which we are aware and do not see. XR technology affords ways of presenting and extending time and space which transcend and escape the conventions of the physical stage, the confines of a defined theatrical location and the limitations of linear narrative. The potentially infinite dimensions of cyberspace can be accessed in relation to, and in opposition to the defined physical space. Spaces within the space, unfolding around and within each other can host scenery, performers and all the elements and aesthetic options comprising XR Scenography. Expanded space establishes different perspectives for the participants by manipulating their points of view or sense of situatedness.
The perception of space has been defined as a triadic model: the dimensional (the actual spatial reality of the individual), the representational (that which is described or represented in models or drawings) and the experiential (which is entirely personal to the individual). Whereby the action of drama takes place in representational space, in XR the artistic elements combine and interact in experiential space, enabling ongoing processes of worlding to operate through discrete acts or sensory and narrative inference which include haptic and acoustic proxemics and preexisting orders of knowledge.
The crafting, configuration and navigation of space, place orientation and levels of sensory input in relation to the performance event incorporates not only the environment of the protagonist but also that of the visitor. Both participants become interactive, vital aspects of the scenographic concept, inhabiting the spatial imaginary.[8].
The XR Drama with its malleable and unpredictable suspension of situatedness and atypical interrelationships transcending spatial, positional and experiential boundaries, is an entirely personal, individual and unique experience; the foundation for a new level of theatrical artistry “The imaginary which fills the empty spaces of thought.” [9]. 2.4
The aesthetic phenomena created during the project are not limited to the VR sequences available as 360° Video. All the steps in the research process have been thoroughly documented, providing a vast repository of aesthetic representations, artefacts and interventions. Next to the open access live-streams, which exist as recordings on the project’s YouTube channel, an immense number of photos, videos, 3D-scans, photogrammetric images and other digital media artefacts has been created, and uploaded to public repositories and social networks such as instagram, facebook, YouTube and sketchfab. One of the last steps in the project has been the creation of a digital project-archive, which, true to the nature of the project, invites users to navigate the findings and experimental material gathered by the researchers.[10] 3
The Wearable Theatre project has managed to explore the qualities of VR in the abovementioned settings. All the parameters and variables defined in the proposal have been researched. Beside the concepts and stage works outlined above, the project produced a vast number of artefacts. By rearranging and sharing them it has become very apparent that these are an essential part of the project’s outcome, not only as documentation of the process or mediatised artefacts of performative acts, but as actual aesthetic knowledge derived from the project itself. In addition to the use of established platforms the team has also created the first prototype of a virtual archive in which all artefacts are arranged and can be navigated. 3.1
To document the artefacts, that were produced in the course of the project and to preserve their live and real-time capability, an online application that provides background information and invites the user to select her/his own way in terms of the use of space and progression has been created to facilitate the possibility to interact with real-time settings. The user can rest in a setting, while it keeps evolving around her/him; additionally, the user can navigate in every direction - front [W], back [S], left [A], right [D], up [E], down [Q]) and rotate his/her virtual head ([MOUSE CLICK + DRAG]. Special areas of context can be found in each setting. These areas hold objects, digital fragments, texts, sounds, videos, images focused on certain common themes, events, periods of research, installations, performances etc. The areas are arranged to overlap, so that they interact with each other, overriding and multiplying, but always based upon the perspective chosen by the user. The settings also contain elements of real surroundings evolving in real-time, that change quite radically over time. Thus, if the user decides to let the application run for some time, the artefacts and areas will grow like a virtual garden. 3.1.1
This application can be considered a virtual, interactive museum. The user strolls through the settings and inspects the artefacts, their background information and the evolving space itself. The user may decide to change the settings when arriving at certain trigger points, enabling the transportation to another, new setting (and context). These settings can be considered as levels (when choosing a game-based approach) or floors (when choosing a museum reference). This virtual museum applies the approach of a so-called Walking Simulator (genre of video games) which leads the user in a first-person perspective through an environment of new information, insight and increasing knowledge, but also adds an overlap of environment and information to intensify and validate the overall experience, both on a perceptual and rational level. 3.1.2
As with every museum which has a basement, depot, rear chamber, etc., this application also contains special settings, that hold concrete, practical data which the user may download and keep. It is in these areas that most of the actual archiving is done and this archive should be kept fully accessible without restrictions. Data such as point clouds, models, videos, images, papers, evaluations, figures, can be stored in the archive and accessed via application. This application prompts the user to find and use these areas in an open source and creative way. The user may implement the data in her/his own body of work and thereby create an expanding universe of virtual, but theatrical experimentation, that reflects on itself.
To ensure broad accessibility, this application is produced as a WebGL Project, to be hosted via HTML5 in common browsers via the web. All the accessible data is stored in common formats, such as *.obj, *.fbx, *.ply, *.txt, *.jpg, *.wav, etc. To create the individual settings or levels/floors, 3D objects are generated throughout the project in a photogrammetric workflow within 3DF Zephyr Lite, that allows up to 500 pictures (in this case mostly frames from a video) for the reconstruction of the objects. Dense point clouds and high-definition textures are also achieved in the process.
These objects are then imported into the game-engine Unity for further development. This application is hosted in the Universal Render Pipeline (URP) of Unity 2019.3 and upwards. The URP is a prebuilt Scriptable Render Pipeline that is optimized for a wide range of target platforms. From Windows and UWP, Mac and iOS, Android, XBox One, PlayStation4, Nintendo Switch, WebGL to all current VR platforms. While this ensures high flexibility in deploying, the main focus remains to target WebGL and for special cases SteamVR. In Unity the objects are arranged and procedural transformation (position and rotation) is applied to generate an ongoing change of the spatial setting. Furthermore, the objects are edited with the use of custom-built shaders made in Unity’s shader graph, that allows a node-based creation of physically based shaders. This process enables the possibility to change the appearance and behavior of the object quite radically over time and based on the perspective of the user.
This application is built for WebGL 2.0 which consists of control code written in JavaScript and shader code written in OpenGL ES3, ensuring the provision of complex, evolving settings even on low standard devices. Not only the VR technology on which the project's practicalities hugely depended, but also the discussion on art-based research and its benefits has intensified in the past years.
Venturing to different venues with the artefacts already created in previous steps proved extremely beneficial and informed what Gesa Ziemer has called creative collectivity (kreative Kollektivität.). [7] For Ziemer the quality of such an “unlikely assembly” (unwahrscheinliche Versammlung) (ibid.) lies exactly in the cooperation of individuals that would not likely have cooperated in other settings.
Wearable Theatre, due to its constant iterative production of performances and media artefacts was able to not only incorporate people, but also its aesthetic artefacts into this assembly. Things such as concepts, recordings, fragments of code and interfaces that had already been created in the project formed the basis of the next steps at a different venue, confronting not only the human protagonists of the project with a new setting, but also the things they brought along.
In their Manifesto of artistic research Henke et al. [8] reaffirm the idea of a specific knowledge that can be derived from art-based research that is located beyond the dualism of theory and practise. In contrast to the discursive modes, a language of things or aesthetic artefacts operates quite differently. “They think multimodally, compositionally, and in many media simultaneously” [8] p. 39. Besides exploring and incorporating the artistic variables of a wearable theatre, the project also created a new outlook on theatre itself, as a mode of producing an aesthetic kind of knowledge. ”Aesthetic practises map out non-scientific epistemologies by drawing their form of knowledge not from syntheses but rather from the sensuous relations of non-predicative conjunctions in which their insights merge and coincide.” (ibid.) Theatre, as the historical paradigm of production of such knowledge, seems to be the ideal site to explore the further possibilities in aspects of XR / Extended Reality. OneFile. Retrieved from: https://link-galecom.uaccess.univie.ac.at/apps/doc/A502420884/ITOF?u=43wien&si d=ITOF&xid=f3847da3. [4] Experiment. Retrieved April 10, 2020 from: https://wearabletheatre.fhstp.ac.at/en/experiment/ [5] Albert Camus. Der Fall, Rowohlt, ger. 1968 [6] Max Frisch. Homo Faber. Suhrkamp 1968 [7] Fjodor Michailowitsch Dostojewskij. Die Dämonen. Deutscher
Taschenbuch Verlag GmbH & Co.KG, GER 1977 [8] Edward W. Soya. Thirdspace, Blackwell, USA 1996, p.56 [9] Rachel Hann. Beyond Scenography, Routledge, 2019, p.24 [10] Archive Retrieved from https://wearabletheatre.fhstp.ac.at/en/archive [11] Gesa Ziemer. Kollektives Arbeiten, In: Jens Badura, Selma Dubach, Anke Haarmann, Dieter Mersch, Anton Rey, Christoph Schenker, German Toro Pérez: Künstlerische Forschung – Ein Handbuch, Diaphanes, Ger 2015 [12] Silvia Henke, Dieter Mersch, Nocolaj van der Meulen, Thomas Sträsle, Jörg Wiesel: Manifesto of artistic research. Diaphanes, CH 2020.