In the last years, several artists have begun to employ tools and software from the field of Artificial Intelligence for producing artworks. The most known collaborations on this topic concern the fields of visual art, drawing, plot writing, and music composition, but there are also several experimental uses of AI in theatrical performances. Here we propose a theoretical framework for contemporary theatrical pieces where AI becomes an integral part of staged actions. We study the relevance of AI as a non-deterministic element that fosters extemporaneous outputs, where the staging implies self-standing algorithms that affect dramaturgy and define peculiar artistic approaches. A cross-section of 13 works is considered to analyse the most recent applications and provide a comprehensive categorisation. Specifically, the framework entails two main phases of the artistic practice: 1) the preliminary setting of algorithms; 2) their function and representation on stage. The former regards the dataset definition and training process and highlights the author's perspective in structuring the software for further staged performance; descriptions of the architectures of the algorithm are provided to delve into some implementations. The latter is related to the scenic interpretation of AI within the dramaturgical concept; examples of the mise-en-scène are considered to describe the role of the software in relation to human agents. The analysis proposes a rather broad and versatile preliminary model useful for both artistic and academic purposes that can be extended to future employments of AI.
The employment of Artificial Intelligence (AI from now on) in Western theatre is firstly related to the
growing importance of digital performance that occurred over the last decades [
Within this context, AI might occur as a technical element that enhances the expressive
possibilities of the pieces but, differently from other digital media, fostering real-time interactivity,
processing of huge datasets, and autonomous learning. The most recent academic literature focused on
various topics of AI regarding visual art and creative issues [50]; interactive and installation art [43],
interoperability and storytelling [41], descriptive or interpretative models [
The present contribution refers to these studies insofar as the performance is a process rather than a datum: it is significantly affected by the algorithms and involves different media that foster the author’s interpretation of AI. Hence, we will not address AI as a mere topic within the show, because as such it might not imply the computational influence on the dramaturgy. AI is here considered as a leading factor also during the enactment, as processing software that holds a dynamic relationship with the performer and eventually with the audience. As such, the plays imply the scenic representation of human-computer interaction. We will address these topics firstly in relation to the implementation of algorithms, analysing in detail the dataset and training of some cases through the source codes made available by the artists or their technical collaborators. A broader overview of other pieces will be provided to give evidence of the efficacy of the model. Then, we will delve into the analysis of the AI staging as oriented towards specific dramaturgical aims. These perspectives will be finally compared to propose a comprehensive framework of current AI applications, showing its social, dialogical, or technical prominence within the theatrical milieu.
The algorithm component is a core factor of the pieces here addressed as both influencing the stage setting and concerning a creative process related to the specific instances of each model. In this section, we will address the author’s preliminary setting depending on whether the database is built by the author or not. Consequently, we infer the four categories shown in Figure 1: 1) ‘scenic data’ – where data are gathered from staged elements (clearly, except for AI outputs themselves), implying a direct relationship between the software and the performance; 2) ‘autonomous data’ – as inputs refer to a dataset without any previous relationship with the stage, thus expressly built for scenic purposes; 3) ‘external data’ – as other pre-existing datasets are employed for feeding the algorithm; 4) ‘subsidiary data’ – as the database is entailed in the operative functions employed within the show. The first two cases might be conceived as akin, as implying both the creation of the database and the AI processing implemented by the author. They have still been separated because the former regards a mutual relationship with the performance since the algorithm implementation, whereas the latter concerns an intrinsic dichotomy between the previous training and the actual stage processing. Furthermore, the second and third categories are similar due to the mediation with data other than those regarding the stage. However, the third case retains its dramaturgical autonomy as the algorithm is structured to process information belonging to a pre-existent database, whereas the dataset of the second category is expressly built. Finally, the fourth category regards the database as an implicit element, where the dramaturgical focus is mainly on the employed function. In summary, the author, in relation to each category, deals with 1) quantifying scenic elements and integrating them into the plot; 2) generating autonomous data and making them interact with performers and/or audience; 3) mediating between the pre-existing dataset and the scene; 4) using pre-trained functions. We assume that the dramaturgical intent is strictly related to which of these cases is involved, where also structured according to the specific setting and the aesthetic intent. For each category, the analysis of four pieces and the related algorithms will be provided: the first one will be discussed concerning the dance play Discrete Figures; the second about the performance Δnfang (pronounced Anfang); the third in relation to the performative installation The Great Outdoors; the fourth regarding the performance DoPPioGioco. The other cases mentioned in Figure 1 will also be described for supporting the hypothesis and better classify borders and nuances of the theoretical framework.2
Description Examples Manually defined Pre-existent
scenography ‘Scenic data’ discretisation of subjects/objects involved on stage Discrete Figures
Convergence
Corpus Nil Ultrachunk 2) not regarding the scenography ‘Autonomous data’ generation of data further related to the scenic action Δnfang Asterism
external sources ‘External data’ dramaturgical relationship with other media The Great Outdoors Frankenstein AI
Improbotics
trained functions ‘Subsidiary data’ employment of pre-trained functions DoPPioGioco Sight Machine Creation + Mediation Mediation
The first category regards the employment of algorithms articulated on datasets about subjects or objects on stage. Discrete Figures (2018) by Elevenplay, Rhizomatiks Research, and Kyle McDonald stands as a significant example of this approach [42]. The dance play enacts performers interacting with pre-recorded digital figures shown on a projection canvas or semi-transparent wireless frames. These characters are generated through preordained data and rendered as pre-recorded 2D line shapes or 3D human-like entities. Their motion and facet are shaped through recording sessions of the performers’ play, previously made and stored through a motion capture system. We will focus especially on one of the fundamental scenes of the play occurring around the end, which implies a 3D anthropomorphic figure in computer-graphic interacting with the real dancer (Fig. 2). This section regards the dancer initially interacting with a not defined human-like figure with a shiny and silver facet appearing in the background canvas. After a short interaction, in which the virtual character moves confusedly, it gets synchronised with the performer in a pre-set choreography.
The database for making the virtual character move is constituted of 40 recording sessions – for a
total of two hours and a half – stored through the Vicon motion capture system at 60 fps [
The scene underlines the relevance of the neural network processing over the entire show. Indeed, dramaturgy is rooted in the mutual relationship between the performer and virtual character, as the latter gets increasingly similar to the former. For most of the play, AI is used only for the previous management of data but still affects the actual representation at both a computational and conceptual level. Indeed, the core aspect of the performance is the interaction occurring between dancers and referring to a pre-set choreography, whereas virtual characters or figures are processed and recorded to simulate the live relationship between the real and virtual entities.
Discrete Figures is one example of algorithms implemented through a dataset directly referring to the scene, in this case regarding dancers’ movement. Additional data might be employed, always suggesting the discretisation of physical objects and subjects on stage. For example, Convergence (2020) by Alexander Schubert proposes a performance in which sounds generated by a string quartet and videos of some of their actions are processed by neural networks both before the show, for training, and during it [46]. The outputs, partially generated in real-time, are rendered on stage via projection canvas and speakers. The audio and the video streams are treated with autoencoders, in separate ways: in both cases, an autoencoder is used to reconstruct the original sound or image. About the audio stream, there are several autoencoders trained on different sounds (string sounds, voice singing, voice speaking, screams, and others). During the performance, a live audio input (e.g. the voice of a singer) is passed to an autoencoder (e.g. the one trained with string sounds), that returns new reconstructed audio (e.g. the voice is reconstructed based on string sounds); sometimes some sound transformation is applied when the audio sample is in the latent space, between the encoding and the decoding phases. For the video steam, there are three different autoencoders trained with three datasets: the faces of musicians and musicians playing the instruments (stand or sit, depending on the instrument). During the performance, the autoencoder reconstructs the image of the musicians, sometimes with some transformation (e.g. changing expression, face orientation, gesture on the instrument, or other details); the system is also employed for obtaining a morphing effect from an image to another one with high-level interpolation, that generates fake images between the start and the end ones.
On the other hand, Corpus Nil (2016) by Marco Donnarumma is also a dance play but employs
microphones and electrodes applied on the performer’s limbs to capture sounds and voltages from
moving muscles and internal organic elements [
In all these cases, the AI relates to the very elements it processes from the learning stage until the live play, and so does the author through the entire creative process. This approach embeds the stage setting and enactment as a single concept. As such, it shows a direct association between the algorithm and the scene: the learning process is visible as an outcome of the stage itself. That is, the inputs and the outputs are seen at once, even if the programming and the learning process are unknown and previously structured. Moreover, an emotional link between the computational artefact and whoever relates to it at a performative level is fostered where people are immediately engaged with the result they have contributed to achieving. 2.2.
The second category concerns data not strictly regarding the stage but autonomously built by the
author. To better describe this case, we will analyse the implementation and usage of the neural
network provided in Δnfang (2019) by the Fronte Vacuo collective [
As discussed, AI strongly contributes to scenic design. The AI algorithm is based on deep reinforcement learning and employs a neural network composed of two fully connected layers with 24 nodes per layer, followed by a third fully connected layer for output actions.5 At each step the network makes a prediction; such prediction is compared with the target array and then the weights of the neural networks are updated according to the distance between the prediction and the target array. However, the target is an array of ten decimal numbers from 0 to 1 arbitrarily assigned with no meaning: the software is not programmed to fulfil a particular task, but to constantly try and start over.6 Numerical values outputted by the machine learning are filtered through the OSC protocol and then sent to Pure Data and TouchDesigner, two software respectively employed for converting numbers in sound and light patterns. The consequent triggering of these patterns, roughly occurring every second for lights and every 13 seconds for sounds, conceptually represents the AI learning process. 4 Δnfang is the first and prototypical performance of the Humane Methods cycle. As the authors stated in the interview that occurred on May 9, 2022, subsequent performances use more complex systems but are based on the same process. For this reason, and because of the richer presence of data, Δnfang is the only one here considered. 5 The code is available at https://github.com/bcaramiaux/humane-methods, the Github account of the programmer Baptiste Caramiaux. 6 In this case, the database is to be considered, in a more abstract way, as a sequence of random numbers generated in real time. The performance has been assigned to the present category because the AI still uses autonomous data, even if they are not archived.
In Δnfang, human beings and computational artefacts do not share the same data. Their dichotomy is also represented on stage, insofar as AI autonomously runs its methods and conceptually influences human behaviour. Other cases regard other plot developments and AI functions. For example, Asterism (2021) by Alexander Schubert is a 36-hour performance installation in which AI, conceived as an oracle, generates textual elements recited by a pre-recorded voice and affects the generation of part of the sound components [45]. Musicians and performers play within this partially extemporaneous environment and a forest expressly built within the hall.
Pieces employing an autonomous database are more likely to enact a detachment between AI and human beings, in the mentioned cases as tools that control, or make prophecies. Indeed, data through which the algorithm is implemented bring outputs not strictly regarding the stage and relating to the performance as an external and self-standing entity. Since data is manually generated, the performance is still enclosed within the author’s management, but the stage setting and the software encoding are separated as two distinct steps. Consequently, also the creative process evolves within a conceptual dichotomy which is eventually reflected in the dramaturgical asset. 2.3.
The third category involves datasets that are not gathered from the scene and neither independently built but refer to specific pre-existing platforms. The Great Outdoors (2017) by Annie Dorsen reflects this approach, insofar as information is gathered from discussion websites [29]. The performance takes place in a planetarium and provides the audience laying on yoga mattresses (Fig. 4). While looking at the night sky on the ceiling, the live-mixed music by Sébastien Roux plays in the background, and a performer recites comments scraped through AI from Reddit and 4chan from the previous 24 hours. Even if data is not processed in real time, the performance is based on automated processes that still determine its dramaturgical outcome. The aim is indeed to compare the Internet and networked technologies, articulated in countless comments, to an infinite, poetic, and celestial landscape to individually explore.
The algorithm is based on the natural language processing word2vec [
Another example of this approach is Frankenstein AI: A Monster Made by Many (2018) by
Columbia University. The performance installation consists of three acts in which AI learns from data
gathered from the text corpus and participants’ inputs to conceptually learn information about
humanity [
On the other hand, Improbotics (2018) by Piotr Mirowski & Kory Mathewson is an improvised
theatrical performance in which a neural network trained on film scripts receives textual inputs
through an operator and processes sentences on the fly [
Employing a pre-existent database as in the mentioned cases fosters a direct relationship with the selected platforms. This is the case of Reddit and 4chan comments in The Great Outdoors; of the reference to the rising monster portrayed by Shelly in Frankenstein AI; of the movie database built for developing an artificial actor. The author selects those databases according to the scenic purpose and defines their role in relation to the dramaturgical goal. Therefore, the reference dataset significantly influences the play regardless of whether a scene is explicitly structured on the relationship with the original content – as in The Great Outdoors – or only employs information for defining the overall enactment – as in Improbotics. 2.4.
The last category regards datasets previously built to define an autonomous operative function (e.g.
face tracking, or text analysis) that are unknown or subsidiary to the author. DoPPioGioco (2019) by
Rossana Damiano, Vincenzo Lombardo, and Antonio Pizzo is a leading example of this trend [
Another example of this approach is Sight Machine (2017) by Trevor Paglen, which also works
through face tracking. It regards a string quartet playing pieces from the 20th-century repertoire, while
an algorithm recognises the visages of performers and audience through a camera and projects them
on the background screen [35]. Though making them visible, it also highlights faces with rectangles,
modifies their connotation, and describes their facets, thus underlying the real-time recognition
performed by the algorithm as a surveillance mechanism. The performance uses digital motion
capture and facial recognition, which is mapped by open-source software that runs neural systems
based on AI engines from Google and other tech companies [
The cases mentioned in this section imply the software elaboration of visual data, providing an enactment focused on the algorithm processing. Besides the representation of specific actions and analogue media rendering, the authors’ role here mainly concerns the software processing and not the database selection, thus the employment of previously programmed operative functions eventually edited. Even if AI is still relevant in designing the development of the dramaturgical goal, the author’s role is limited to the scene definition independently from the dataset, thus not implying an extensive relationship with the software. Therefore, the scenography results as defined not so much by the database definition but by pre-set functions that the algorithm performs during the performance.
In the last section, we will move from the authors’ preliminary setting to the stage enactment of AI,
thus to how the computational process is manifested to the audience. Indeed, authors aim to set the
scene according to their own representational purposes, concealing the ground architecture of the
software through different kinds of interfaces. That is, AI must be somehow manifested on stage as a
‘digital agent’– as showing autonomous behaviour but still according to pre-set instances programmed
by the author [
The first and most frequent case takes into account the algorithm as an abstract entity, underlying unseen or even super-human properties that question its socio-political role. The machine here reflects an unbalanced relationship with humans as governing them or communicating like an omniscient or superior being. This is the case of Convergence, in which musicians’ actions are scanned and rendered as mechanical elements while a voice-over describes the occurring events; Sight Machine, showing the algorithm analysis of human bodies as a surveillance mechanism; Δnfang and Asterism, manifesting a divine entity which, also through a ritual representation, constricts humans or reveals unexpected horizons; Frankenstein AI, portraying the software as an abstract figure that speaks with the audience through screens and speakers and is eventually embodied in a human dancer; The Great Outdoors, where the social nature of websites is depicted as a celestial and infinite reality.
Secondly, AI might be represented as an anthropomorphic figure, enacting a dramaturgical development in a peer relationship with human beings. This case provides an exchange and a mutual influence between humans and machines. Discrete Figures, for example, enacts a choreography fulfilled by human and virtual characters and shows the progressive evolution of the mutual interaction; Improbotics, on the other hand, provides the interplay between a computer voice expressed in common language and actors improvising on those stimuli; Ultrachunk implies the singer interacting with the algorithm embodied in her own digital ego and shown on the background canvas.
Finally, the algorithm can operate as a technical tool whose processing is not directly relatable to staged outputs. The extemporaneous development of these pieces is still strongly defined by AI, but the rendering of its processing relates to other factors than AI itself. Thus, the software operates a dramaturgical mediation, not the enactment of the machine concept. In DoPPioGioco, for example, the algorithm mediates between audience and performer by categorising expressions of the former and providing four possibilities to the latter to continue the narrative action, but face tracking is not explicitly shown; in Corpus Nil, the software processes movements and sounds in real-time towards a self-standing sonic output which relates expressly to the performer’s body.
These three classes can be considered exemplary of the observed forms given to algorithms in
contemporary theatre. The representations employed in the first two categories often refer to the
horizon of speculative and science fiction, as interrogating normative notions about reality with a
focus on imaginary technology [33, 38, 39] – e.g., the AI might be depicted as a 3D virtual character,
speaking through a computerised voice, or moving as geometrical or pulsating digital shapes.
Specifically, the unseen nature of AI might evoke mythical properties [
Merging all the categories outlined in the article, as done in Table 1, provides a basis for better overviewing the AI conception in both the preliminary stages and its further representation. Being these steps strictly related, we believe that splitting and re-framing them might enlighten the specificity of each piece. For example, both Δnfang and The Great Outdoors refer to AI as an abstract entity but from different angles: the former depicts a power relationship of an autonomous creature whereas the latter describes a specific online environment as celestial space. On the other hand, Discrete Figures and Improbotics both imply a peer exchange between humans and digital entities but, respectively, as characters directly shaped from the performers’ facet or referring to movie scenes. Furthermore, Convergence and Discrete Figures both use data directly gathered from the scene, but the former depicts AI as a ubiquitous and omniscient entity exploiting performers’ actions whereas the latter as an anthropomorphic character trying to learn from humans. It must be noticed that this framework does not aim to be exhaustive but opened to future research. It would be possible to investigate possible sub-classes or compare other perspectives – for example, focusing on the analogue media interfaces or deepening the human-machine relationships. We still believe that this model allows a useful basis for further comparison and experimentation in both academic and artistic fields.
In the present paper, we have proposed some criteria to overview how AI has been implemented and interpreted within contemporary intermedia theatre. The section concerning AI employment has been dedicated to the preliminary creative process which does not only involve a script or score definition but a whole programming environment which is computational and follows its own rules. Hence, the various ways in which the database and learning processes are set strongly influence the dramaturgy and the creative approach, being aimed to fulfil the staging purpose through automated instances. It has been highlighted that the various pieces mentioned in the article show different applications of the four basic principles regarding the algorithms implementation. These recurrent factors are thus indicative of common approaches highlighting technical and aesthetic intersections between informatics and art. On the other hand, AI representation involves the theatrical codes of enactment and the emulation of its presence on stage. The many forms acquired by the software are frequently inherited from speculative fiction but also regard other interpretations of the digital media such as the enactment of pre-existent digital platforms, the rendering of renewed entities, or the simple exploitation of computational instances. This overall framework does not aim to be exhaustive but to give some hints for comprehending the role of performance art with respect to the increasing social relevance of AI. We believe that this knowledge can stimulate the discussion in the academic field for theoretical aims and propose useful knowledge to artists for further experimentation with new expressive possibilities.
The present article has been realised thanks to the kind availability of authors, composers, and programmers involved in the pieces analysed, specifically regarding Δnfang – the interview with Marco Donnarumma and Andrea Familiari that occurred on May 9, 2022 – Discrete Figures – the email exchange with Kyle McDonald that took place on April 24, 2022 – The Great Outdoors – the source code provided by Annie Dorsen for research purposes and the email exchange with the programmer Marcel Schwittlick that occurred on September 26, 2022. We are sincerely grateful for the attention devoted to us.
[28] A. M. Monteverdi, Leggere uno Spettacolo Multimediale: La Nuova Scena tra Videomapping,
Interaction Design e Intelligenza Artificiale, Dino Audino, Rome, 2020. [29] Museum of Contemporary Art of Chicago, Annie Dorsen: The Great Outdoors, 2019. URL: https://mcachicago.org/Calendar/2019/03/Annie-Dorsen-The-Great-Outdoors. [30] S. Natale, Deceitful Media: Artificial Intelligence and Social Life after the Turing Test, Oxford
University Press, New York, 2021. [31] K. L. Norman, J. Kirakowski (Eds.), The Wiley Handbook of Human Computer Interaction, John
Wiley & Sons, Hoboken, 2018. [32] U. Otto. “Theatres of Control: The Performance of Algorithms and the Question of Governance.”
The Drama Review 63.4 (2019): 121–138. doi: 10.1162/dram_a_00879. [33] M. Oziewicz, Speculative Fiction, Oxford Research Encyclopedia of Literature, 2017. doi: 10.1093/acrefore/9780190201098.013.78. [34] R. Packer, K. Jordan, Multimedia: From Wagner to Virtual Reality, Norton & Company, New
York, 2001. [35] T. Paglen, Sight Machine, 2020. URL: https://paglen.studio/2019/07/01/sight-machine/. [36] R. F. Paloutzian, R. J. Seitz, H.-F. Angel, The Processes of Believing and Communicating with the Unseen, in: T. G. Plante, G. E. Schwartz (Eds.), Human Interaction with the Divine, the Sacred, and the Deceased Psychological, Scientific, and Theological Perspectives, Routledge, New York, 2022, pp. 213–133. [37] J. Parker-Starbuck, Cyborg Theatre: Corporeal/Technological Intersections in Multimedia
Performance, Palgrave Macmillan, Houndmills, 2011. [38] P. Parrinder, Science Fiction: Its Criticism and Teaching, Routledge, London, 2003. [39] S. Parry, Science in Performance: Theatre and the Politics of Engagement, Manchester
University Press Manchester, 2020. [40] A. Pizzo. “Performing/Watching Artificial Intelligence On Stage.” Skenè: Journal of Theatre and
Drama Studies 7.1 (2021): 91–110. doi: 10.13136/sjtds.v7i1.308. [41] A. Pizzo, V. Lombardo, and R. Damiano. “Algorithms and Interoperability between Drama and
Artificial Intelligence.” The Drama Review 63.4 (2019): 14–32. doi: 10.1162/dram_a_00872. [42] Rhizomatiks, Discrete Figures, 2019. URL: https://research.rhizomatiks.com/s/works/discrete_figures/en/. [43] D. Rokeby. “Perspectives on Algorithmic Performance through the Lens of Interactive Art.” The
Drama Review 63.4 (2019): 88–98. doi: 10.1162/dram_a_00876. [44] C. Salter, Entangled: Technology and the transformation of performance, MIT Press, Cambridge, 2010. [45] A. Schubert, Asterism, 2021. URL: https://www.alexanderschubert.net/works/Asterism.php. [46] A. Schubert, Presentation ‘Convergence’ [@Eclat Festival Presentation Series], 2020. URL: https://www.youtube.com/watch?v=laoV7cGXUNo. [47] A. Schubert, Switching Worlds, Wolke Verlag, 2021. [48] I. Sutskever, O. Vinyals, V. Le Quoc, Sequence to Sequence Learning with Neural Networks, in: Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, K. Q. Weinberger (Eds.), Proceedings of the 27th International Conference on Neural Information Processing Systems, MIT Press, Cambridge, 2014, pp. 3104–3112. doi: 10.48550/arXiv.1409.3215. [49] A. Tinwell, The Uncanny Valley in Games and Animation, Taylor & Francis, New York, 2015. [50] J. Zylinska, AI Art: Machine Visions and Warped Dreams, Open Humanities Press, London, 2020.