Wearable Theatre - Immersive Storytelling and Theatrical VR Markus Wintersberger*, Thomas Wagensommerer, Georg Vogt, Julia Püringer, Christian Munk, Ulrich Kühn St. Pölten University of Applied Sciences Marcus Josef Weiss, Colleen Rae Holmes Open Acting Academy - Conservatory for Acting ABSTRACT During the course of the research additional experiments were The following paper presents the main findings of the three-year conducted, and the findings of those experiments introduced in a art-based research project Wearable Theatre - The Art of broader theatrical setting. Live Performances were recorded, Immersive Storytelling. It introduces the basic art-based streamed and reworked. The following paper gives insight into, research premise of the project, the technological framework and and an overview of the project and addresses specific questions its artistic concept. It also presents the prototype of a project encountered by the research team. The project was featured museum or interactive art-repository that offers insights into the prominently (amongst other events) in the Volkstheater Vienna as project’s outcome. part of the Digital Natives Festival 2018 and invited as an international guest to the New Forms of Theatre Festival in Keywords: VR, immersive media, storytelling, XR, experimental Warsaw in 2019. Chapter 1 gives an overview of the project and media, art. introduces the idea of artistic research being collective aesthetic work, inherent to the project itself. Chapter 2 addresses the evolving state of VR technology during the 3-year research Index Terms: Virtual reality, digital archive, art-based research period, the artistic concept, and the fundamental groundwork of adapting literary material for theatrical VR (illustrated explicitly 1 INTRODUCTION in the stage production ÆON), and the very different forms of The art-based research project, Wearable Theatre - The Art of spatial perception inherent in this form of art. The created Immersive Storytelling, explored, during its 3-year span, the artefacts were subject to an open dissemination strategy potential of storytelling in VR. The project team consisted of (described in chapter 2.3). Chapter 3 reviews the findings, artists and researchers from the St. Pölten University of Applied suggests further research efforts towards an XR Theatre and Sciences and the Open Acting Academy in Vienna. [1] This presents the first steps towards a digital museum or repository for research project identified and structured aesthetic variables and digital artefacts - as well as aesthetic phenomena. This can also be applied them to twelve experimental settings, each focusing on a seen as part of an open-source aesthetic toolbox that might be specific aspect of VR sensory experience. The goal was to artistically appropriated. Finally, the consolidation of the understand and master the visual, acoustic and atmospheric project’s findings is discussed, including how these, along with possibilities that 360° VR offers. The aim was to merge the the initial mission to provide useful artefacts and artistic practices, aesthetic variables with a narrative and produce a unique has led to new research questions. Because the research-team immersive VR art experience. Three specific literary works were chose initially to concentrate specifically on the questions posed chosen to provide this narrative foundation for the experimental by content and artistic forms of expression of the AV technology, steps. (See chapter 2.1) particularly regarding immersion, perspective, orientation, focus, and attention, it was preferable in the first prototypical experiments, to experiment with low-budget technology and its limitations. The video studio made available to the research team by the St. Pölten University of Applied Sciences, with its elaborate studio technology, provided an ideal spatial reference point throughout, and, particularly at the beginning of the research, functioned as a prototypical laboratory. 2 WEARABLES AND THE THEATRICAL USE OF VR * markus.wintersberger@fhstp.ac.at 2.1 State of technology and the research settings The complex technological challenges relating to 360° AV production, in particular concerning the image quality, resolution and low light capture abilities were continuously shifting parameters during the project. Technology saw vast improvements during the project durations, enabling additional scenarios that were not possible in the initial outline throughout the course of the project. In 2016, at the time the project proposal was originally formulated, 360° AV technology was only available in a very early and immature version. An experimental “uncertainty” was, thereby, calculated into the proposal itself and into the artistic- Copyright © 2020 for this paper by its authors. Use permitted under technological vision of the research, and, with foresight of the Creative Commons License Attribution 4.0 International (CC BY 4.0). coming developmental steps as regards 360° AV, VR and that of XR in the years between 2017 and 2020, was imagined on an On the formal level a foundation of texts was selected, all of evolutionary trajectory. which were written in the literary form of reported speech (a first- In 2014 / 2015 the subject of VR appeared on the horizon, person narrative), thereby providing a subjectively filtered hyped by Facebook, Oculus Rift and HTC Vive as showing great narrative perspective. The variability and fluctuations in the technological promise. [2] [3] The 360° AV technology as an personal-narrative reporting (simultaneously a part of the literary “intermediary” was supposed to fulfill a hybrid function: To fascination) is directly analogue to the experiential spectra of capture the presence of the live experience and merge it with the applied XR technology. virtual realm and its potential channels of communication such as On the contentual level, all foundational texts stem from the streaming and presence in social networks. same dramatic premise. Both “The Fall” (Albert Camus) [5] and The research team has tried throughout the entire time-frame of “Homo Faber” (Max Frisch) [6] as well as “Demons – The 3 years (2017-2020) to process the specific question as to the Confession of Stavrogin” (Fjodor Dostoyevsky) [7] describe with connection between the “real” and the virtual, and is currently extreme psychological precision, the protagonists’ search for working on a subsequent research proposal for 2021 to 2024: „XR redemption and lead the sentient subject into the state of Theatre. The Art of Extended Storytelling“, with particular consciousness – Metanoia – a remorse of the deepest and most emphasis on new technology such as photogrammetry and Real- immediate kind when confronted with mortality. time 3D. The unifying literary premise of these first-person narratives “With its predefined experiments, yet open outlook the project could be summarised as: Independent of the intensity of guilt the told itself a tale of unending possibilities as the logic of the pursuit of personal purification by both being and consciousness experiment. Experimentation, theatre, finding meaning and is inescapable. inventing meaning determine a Wearable Theatre; determine also The first experiments led to the recognition that in the the subsequent extension of the ideas into a digital sphere, the application and integration of the “Subjective Goggles” into a indomitable and yet so fragile digitality. The uncanny is hidden in theatrical form, the aesthetic and contentual Theories of the niches between these poles; hidden in the search for the Expressionism in the 20th century provided (in this context) a connections between reality and reality.” Initially, the practice- valuable reference point. led research aspects were explored using the available technology These first critical elucidative insights were condensed into the in a studio setting. [4] fragment “Nachtgerüche” (Night Rumours), which was premiered The spatial “neutrality” of the studio, which, as a result of the at the Festival ‘Die Kunst der Nachbarschaft’ (Neighbourhood insights gained during the research process, developed into an Art) in cooperation with the Vienna Volkstheater. The festival “immersive media laboratory”, is an almost mandatory itself was consequently awarded the Dorothea-Neff Special Prize prerequisite for the examination of the technological, artistic and for boundary-breaking theatre work. scientific questions posed by the research project “Wearable Building on this, the focus was laid on the planning, Theatre. The Art of Immersive Storytelling.” dramatization and application of the postulated experimental The simple “Black Box” situation of the studio, whose almost narrative patterns and forms of staging. permanent changeability through real-virtual scenographic The resulting experiment was a completely dramatized and designs and almost unlimited changes in lighting, form a good staged theatrical prototype entitled ‘ÆON’, which was premiered basis for content-driven research perspectives on “Space-time- on the 31st of May 2019 on the main stage of the theatre”. These features were deliberately included in the planning Volkstheater,Vienna. before the structuring of the application was begun, their actual It is important to explain the processes involved in the potential, however, could only be proven and developed development of ‘ÆON’, from both a dramatic and a directing exploratorily through the “Wearable Theatre” project itself. perspective. The amalgam, comprising acting, action, interaction, immersion and light-projection, soon formed a crucial reference point, also as 2.2.1 ÆON – the transcendence of the “I” (Self) – The regards the hypotheses set out in the research proposal. dramaturgical experiment from the perspective of To what extent is it possible to conceive, implement, record and the author. perform theatrically in a stage-setting based on real artefacts and In terms of the dramaturgical exposition, the point of departure persons in a 360° AV dimension? was the first-person narrator (the I-narrator), or, to be more Which specific 360° AV technological challenges emerge as a precise: the taking apart of the self-aware and reflecting textual result of these assumptions and circumstances? “Self” and dividing it into its opposing forces and effective To this precise end, the team, by careful use of the literary prose mechanisms. examples specified in the proposal and the conscious decision to A context was created for the inner psychological processes of focus on short action sequences extrapolated from this literary suppression, dawning (or forced) awareness up to and including source-material, was able to define a clear artistic-scientific the search for forgiveness, on which basis a dramatic-stage “language” in accordance with the project proposal. progression was developed. The splitting apart and character-specific personification of the 2.2 The investigation of 360° VR in the actualisation of aspects of awareness within the consciousness of a protagonist literary material and its conception as a seems not only to be a justifiable narrative strategy, but one of the theatrical genre demands and challenges inherent in combining the different levels The initial artistic hypothesis of the research project was to of narrative potential within a single entity in a coherent totality. illustrate effectively, on the basis of a specific literary format, the It is also a logical analogy to the ambivalence regarding rationale in employing the 360° VR-Medium in theatre, as well as immersive VR-Media as to their uses and disadvantages. The its contentual and aesthetic applications. These experiments depth of the layers of psychological experience and meaning would, according to the hypothesis, have the potential to expand made possible by the application of this technology is magnified the experiential narrative patterns and staging forms of theatre by and demonstrated specifically through ÆON. a further dimension. ÆON is thereby, based on the hypothesis of the PEEK research There were two determining parameters for this: one formal and project “Wearable Theatre. The Art of Immersive Storytelling”, a the other contentual. drama about the potential conflict of an individual (protagonist) told from the perspective of his/her inner “Self”. 2.2.2 ÆON – the transcendence of the “I” (Self) – The theatrical experiment from the perspective of the director. When, as in the case of ÆON, the complete awareness of the protagonist – the inner “I” (Self) – is dissolved, it is necessary in a theatrical implementation to establish a clear connection between the dissolution and disintegration process and the universe of the physically real theatre in order to allow the impact of the drama to unfold. Through this, reflections on the technical aspect of staging Figure 2: VR-montage generated from the performance at the arose, stemming from two affiliated questions: “New-Forms-of-Theatre” festival. What if the entire theatre in the totality of its (physical) functions and spatial structures were an analogy for human consciousness? 2.3 The Triad of perception What if the VR-Perspective was an elementary catalyser for the The radical changes inherent in the application of new technology interfacing of these physical structures and the scenic-staged to the theatrical medium has forced a re-evaluation and interaction? reinvention of the hitherto, defining components. The roles of The directorial onus was to disassemble the totality of the performer and audience, performative space, text, light and theatrical space as an in-itself-interconnected organism, to scenographic elements are in the process of being deconstructed reorganise and codify the individual parts, and, by means of the and redefined according to the nature and possibilities of new staged arc of the narrative, dissolve this organisation and connect media, specifically: immersive Extended Virtual Reality it anew, bringing the levels together in a single multi-layered technology. space at the high point of the drama. Immersive XR theatre implies the re-conjoining of the world of For these alternating interactions of the individual “Selves” the drama and that of the spectator. The melding of the entities of analogies were created in the form of individual theatrical spaces spectator and performer, auditorium and stage, enables an which were allocated and interconnected. alternative option allowing the spectator to be essentially a part of The following analogy was finally used as the foundational the dramatic narrative, navigating through the fabric of the XR spatial structure for the staging of ÆON: drama and by his/her choices experiencing the event individually The area below the stage and the apparatus and construction of and uniquely. the revolving stage mechanism represented analogously the The theatre as we see it traditionally is a place of multiple human subconscious. orientation: the spaces we see and those of which we are aware The main stage and the entire lighting plot from the central and do not see. XR technology affords ways of presenting and perspective was representative of actual consciousness. extending time and space which transcend and escape the The transformation and refining of actual consciousness into conventions of the physical stage, the confines of a defined Dream-consciousness was staged by means of extending the theatrical location and the limitations of linear narrative. The action into the auditorium and the further refining and elevation potentially infinite dimensions of cyberspace can be accessed in into the aware / visionary “I” (Self) illustrated by the churning relation to, and in opposition to the defined physical space. Spaces firmament of the theatre ceiling. within the space, unfolding around and within each other can host scenery, performers and all the elements and aesthetic options comprising XR Scenography. Expanded space establishes different perspectives for the participants by manipulating their points of view or sense of situatedness. The perception of space has been defined as a triadic model: the dimensional (the actual spatial reality of the individual), the representational (that which is described or represented in models or drawings) and the experiential (which is entirely personal to the individual). Whereby the action of drama takes place in representational space, in XR the artistic elements combine and interact in experiential space, enabling ongoing processes of worlding to operate through discrete acts or sensory and narrative inference which include haptic and acoustic proxemics and pre- existing orders of knowledge. The crafting, configuration and navigation of space, place Figure 1: 360° technology integrated into the set on stage. orientation and levels of sensory input in relation to the performance event incorporates not only the environment of the protagonist but also that of the visitor. Both participants become interactive, vital aspects of the scenographic concept, inhabiting the spatial imaginary.[8]. The XR Drama with its malleable and unpredictable suspension of situatedness and atypical interrelationships transcending spatial, positional and experiential boundaries, is an entirely personal, individual and unique experience; the foundation for a new level of theatrical artistry “The imaginary which fills the leads the user in a first-person perspective through an empty spaces of thought.” [9]. environment of new information, insight and increasing knowledge, but also adds an overlap of environment and 2.4 Open dissemination information to intensify and validate the overall experience, both The aesthetic phenomena created during the project are not on a perceptual and rational level. limited to the VR sequences available as 360° Video. All the steps 3.1.2 A toolbox (outlook) in the research process have been thoroughly documented, providing a vast repository of aesthetic representations, artefacts As with every museum which has a basement, depot, rear and interventions. Next to the open access live-streams, which chamber, etc., this application also contains special settings, that exist as recordings on the project’s YouTube channel, an immense hold concrete, practical data which the user may download and number of photos, videos, 3D-scans, photogrammetric images and keep. It is in these areas that most of the actual archiving is done other digital media artefacts has been created, and uploaded to and this archive should be kept fully accessible without public repositories and social networks such as instagram, restrictions. Data such as point clouds, models, videos, images, facebook, YouTube and sketchfab. One of the last steps in the papers, evaluations, figures, can be stored in the archive and project has been the creation of a digital project-archive, which, accessed via application. This application prompts the user to find true to the nature of the project, invites users to navigate the and use these areas in an open source and creative way. The user findings and experimental material gathered by the may implement the data in her/his own body of work and thereby researchers.[10] create an expanding universe of virtual, but theatrical experimentation, that reflects on itself. 3 OUTCOMES - WHAT REMAINS TO BE SEEN? To ensure broad accessibility, this application is produced as a WebGL Project, to be hosted via HTML5 in common browsers The Wearable Theatre project has managed to explore the via the web. All the accessible data is stored in common formats, qualities of VR in the abovementioned settings. All the parameters such as *.obj, *.fbx, *.ply, *.txt, *.jpg, *.wav, etc. and variables defined in the proposal have been researched. Beside the concepts and stage works outlined above, the project produced a vast number of artefacts. By rearranging and sharing them it has become very apparent that these are an essential part of the project’s outcome, not only as documentation of the process or mediatised artefacts of performative acts, but as actual aesthetic knowledge derived from the project itself. In addition to the use of established platforms the team has also created the first prototype of a virtual archive in which all artefacts are arranged and can be navigated. 3.1 Living archive and toolkit To document the artefacts, that were produced in the course of the project and to preserve their live and real-time capability, an online application that provides background information and invites the user to select her/his own way in terms of the use of Figure 3: Approaching the museum in virtual space. space and progression has been created to facilitate the possibility to interact with real-time settings. The user can rest in a setting, while it keeps evolving around her/him; additionally, the user can navigate in every direction - front [W], back [S], left [A], right [D], up [E], down [Q]) and rotate his/her virtual head ([MOUSE CLICK + DRAG]. Special areas of context can be found in each setting. These areas hold objects, digital fragments, texts, sounds, videos, images focused on certain common themes, events, periods of research, installations, performances etc. The areas are arranged to overlap, so that they interact with each other, overriding and multiplying, but always based upon the perspective chosen by the user. The settings also contain elements of real surroundings evolving in real-time, that change quite radically over time. Thus, if the user decides to let the application run for some time, the artefacts and areas will grow like a virtual garden. 3.1.1 An interactive museum This application can be considered a virtual, interactive museum. The user strolls through the settings and inspects the artefacts, their background information and the evolving space itself. The user may decide to change the settings when arriving at certain trigger points, enabling the transportation to another, new setting (and context). These settings can be considered as levels (when choosing a game-based approach) or floors (when choosing a Figure 4: Inside the archive: Recording of the Red Bar / museum reference). This virtual museum applies the approach of Volkstheater Vienna. a so-called Walking Simulator (genre of video games) which 3.1.3 Technical workflow To create the individual settings or levels/floors, 3D objects are generated throughout the project in a photogrammetric workflow within 3DF Zephyr Lite, that allows up to 500 pictures (in this case mostly frames from a video) for the reconstruction of the objects. Dense point clouds and high-definition textures are also achieved in the process. These objects are then imported into the game-engine Unity for further development. This application is hosted in the Universal Render Pipeline (URP) of Unity 2019.3 and upwards. The URP is a prebuilt Scriptable Render Pipeline that is optimized for a wide range of target platforms. From Windows and UWP, Mac and iOS, Android, XBox One, PlayStation4, Nintendo Switch, WebGL to all current VR platforms. While this ensures high flexibility in deploying, the main focus remains to target WebGL and for special cases SteamVR. Figure 7: Approaching a photogrammetry in the archive. 4 CONCLUSION AND OUTLOOK Not only the VR technology on which the project's practicalities hugely depended, but also the discussion on art-based research and its benefits has intensified in the past years. Venturing to different venues with the artefacts already created in previous steps proved extremely beneficial and informed what Gesa Ziemer has called creative collectivity (kreative Kollektivität.). [7] For Ziemer the quality of such an “unlikely assembly” (unwahrscheinliche Versammlung) (ibid.) lies exactly Figure 5: 3D scan of the “Fuehrerbunker” installation in the archive. in the cooperation of individuals that would not likely have cooperated in other settings. In Unity the objects are arranged and procedural transformation Wearable Theatre, due to its constant iterative production of (position and rotation) is applied to generate an ongoing change of performances and media artefacts was able to not only incorporate the spatial setting. Furthermore, the objects are edited with the use people, but also its aesthetic artefacts into this assembly. Things of custom-built shaders made in Unity’s shader graph, that allows such as concepts, recordings, fragments of code and interfaces a node-based creation of physically based shaders. This process that had already been created in the project formed the basis of the enables the possibility to change the appearance and behavior of next steps at a different venue, confronting not only the human the object quite radically over time and based on the perspective protagonists of the project with a new setting, but also the things of the user. they brought along. This application is built for WebGL 2.0 which consists of In their Manifesto of artistic research Henke et al. [8] reaffirm the control code written in JavaScript and shader code written in idea of a specific knowledge that can be derived from art-based OpenGL ES3, ensuring the provision of complex, evolving research that is located beyond the dualism of theory and practise. settings even on low standard devices. In contrast to the discursive modes, a language of things or aesthetic artefacts operates quite differently. “They think multimodally, compositionally, and in many media simultaneously” [8] p. 39. Besides exploring and incorporating the artistic variables of a wearable theatre, the project also created a new outlook on theatre itself, as a mode of producing an aesthetic kind of knowledge. ”Aesthetic practises map out non-scientific epistemologies by drawing their form of knowledge not from syntheses but rather from the sensuous relations of non-predicative conjunctions in which their insights merge and coincide.” (ibid.) Theatre, as the historical paradigm of production of such knowledge, seems to be the ideal site to explore the further possibilities in aspects of XR / Extended Reality. REFERENCES [1] Wearable Theatre. Retrieved April 10, 2020 from https://wearabletheatre.fhstp.ac.at/en [2] Deeming, Skot. The new VR. Broken Pencil, no. 63, Spring 2014, p. 9 [3] Dylan Bushell-Embling, Dylan. "VR content market to swell to Figure 6: Navigating the artefacts in the archive. $3.3b by 2020." Telecom Asia, December 2016, 6. Gale General OneFile. Retrieved from: https://link-gale- com.uaccess.univie.ac.at/apps/doc/A502420884/ITOF?u=43wien&si d=ITOF&xid=f3847da3. [4] Experiment. Retrieved April 10, 2020 from: https://wearabletheatre.fhstp.ac.at/en/experiment/ [5] Albert Camus. Der Fall, Rowohlt, ger. 1968 [6] Max Frisch. Homo Faber. Suhrkamp 1968 [7] Fjodor Michailowitsch Dostojewskij. Die Dämonen. Deutscher Taschenbuch Verlag GmbH & Co.KG, GER 1977 [8] Edward W. Soya. Thirdspace, Blackwell, USA 1996, p.56 [9] Rachel Hann. Beyond Scenography, Routledge, 2019, p.24 [10] Archive Retrieved from https://wearabletheatre.fhstp.ac.at/en/archive [11] Gesa Ziemer. Kollektives Arbeiten, In: Jens Badura, Selma Dubach, Anke Haarmann, Dieter Mersch, Anton Rey, Christoph Schenker, German Toro Pérez: Künstlerische Forschung – Ein Handbuch, Diaphanes, Ger 2015 [12] Silvia Henke, Dieter Mersch, Nocolaj van der Meulen, Thomas Sträsle, Jörg Wiesel: Manifesto of artistic research. Diaphanes, CH 2020.