Tangible Narrative: The Intersection Interactivity, and Narrative-A Design Case

Abstract

This paper expands on previous work on tangible narratives. It briefly reviews the design case of Letters to José, from which stemmed a narrative architecture for tangible narratives and a typology that frames the qualities of tangible artifacts with narrative meaning. Based on this architecture and exemplified in the typology, the paper introduces The Non-myth of the Noble Red, a tangible narrative which expands upon the learnings gained from the design of Letters to José. This paper presents some of its technological considerations and discusses how it integrates Brechtian theatre concepts to explore alternative digital storytelling approaches.

Keywords

• 1 Tangible Narratives
• interactive storytelling
• tangible interaction
• Brechtian theatre
• The Nonmyth of the Noble Red

1. Introduction: Tangible Narratives

Tangible Narratives belong to a type of interactive media, commonly referred to as Interactive Digital Narratives (IDN) [ 1 ], which integrate interactivity and narrative. While in existing literature, Tangible Narratives are widely described as interfaces [ 2,3 ], platforms [ 4,5 ], or environments [ 6 ], indisputably, they are more than software. What distinguishes them from other manifestations of Tangible Interaction is that they integrate narrative content with interfaces, objects, and a system environment [ 7 ]. As a narrative, they present a plot and feature a story world inhabited by characters and objects, while its computational aspects discreetly serve to support the narrative experience [ 7,8 ]. The field of Tangible Interaction has contributed to the development of theoretical principles that can be applied to the study of Tangible Narratives in tandem with other principles borrowed from narratology.

This short paper expands on previous work on tangible narratives. It introduces the design case from which stems a narrative architecture and a typology that aids in framing and designing tangible narratives. Considering these concepts, the paper presents The Non-myth of the Noble Red. This tangible narrative applies the concepts above while integrating aspects brought from the study of performative arts, particularly Brechtian theatre, to find alternative ways of authoring tangible narratives. 1.1.

The interface of an interactive digital narrative creates a space for active performance where both the user and the computer have a role. The software provides a consistent illusion of the story's reality. Creating human-computer experiences intends to construct imaginary worlds that can be extended, amplified and enriched [ 9 ]. This statement is critically relevant to the authoring of tangible narratives. Letters to José, a tangible interactive narrative, is a demonstrative case. In the form of three interactive, physically unfolding storyworlds, the narrative presents the story of two brothers and the letters they exchanged during the late 1940s. Every physical expression of Letters to José sought to “transport” [ 10 ] the person into the illusion of the story by representing its likeness. Each story world was comprised of separate panels that were both the interface and the stage of the story. They were an interface because they allowed the person to control how the narrative was presented (for instance, triggering sound after a specific gesture). They were also a stage because they allowed the enactment of various narrative aspects. Interactions like posing a cardboard puppet in a particular way allowed the person to assume the character's narrative position by imitating multiple actions. This way, the story was not only told by a narrator, but simultaneously read, played, observed, and performed by the user. The design of Letters to José, and the subsequent study of this narrative [ 11 ], revealed multiple considerations for authoring tangible interactive narratives [ 7,11,12 ]. Among them are an authoring architecture and a typology for what Echeverri and Wei [ 7,11 ] framed as artifacts for storytelling, as discussed below. 1.2.

Narrative Architecture

Based on the experience of designing and authoring Letters to José, as suggested in previous work [ 7 ], structuring a tangible digital narrative focuses on two primary levels: the content level and the operations level. On the one hand, the content level, based on a non-linear narrative structure and grounded on the hypertextual tradition, is composed of two sub-levels: the story content and the exposition content sub-levels. The story content sub-level includes the major arcs—interconnected chronological events that tell causality—and the secondary arcs—side sequences that provide different layers of depth in the story. The exposition content sub-level, while it does not directly narrate events, provides context to what is being narrated, generates expectations, formulates dilemmas, presents choices, establishes a mood, or introduces new elements.

On the other hand, the operations level refers to the non-narrative content that outlines the procedures performed both by the system and the person. It supports the unfolding of the plot by generating physical and cognitive affordances that tell the person how to navigate, the options available, and the actions and possible outcomes caused by their actions through the manipulation of physical artifacts with narrative meaning, or artifacts for storytelling. 1.3.

Artifacts for Storytelling

The concept of artifacts for storytelling departs from Kendall Walton’s objects of imaginings and the make-believe theory of representation [ 13 ]. According to him, objects of imaginings are objects imagined as something else, like a broom that children imagine to be a pony. In Letters to José, for instance, posing a puppet, or opening a door, allowed people to imagine aspects of the narrative related to those actions [ 11 ]. Considering that these artifacts have a critical role beyond aesthetical expressions in a tangible narrative, they can be described by three varying dynamic qualities and interaction methods [ 12 ] (Fig. 1):

Diegesis [ 8,14,15 ] refers to whether the artifact and its representation exist in the space and time of the narrative. It considers three values: diegetic, where the artefact and its representation are part of the storyworld and are presented in the same way in the real world (e.g., a glove worn by the protagonist of a story but also worn by the user), transdiegetic, where the artefact exists in the storyworld and realworld, however, while represented differently, bridges both worlds (e.g., change of pace in the music before a fight scene), and extra-diegetic, where the artefact exists in the space of the narrative system but resides outside of the storyworld (e.g., a light switch).

Embodiment [ 16,17 ] describes how close the action performed and the outcome of that action is. This quality responds to different values conditioned by the actions made by the person. Its values are full, where the artefact is both the means of performance and the space where the result of the performance is manifested (e.g., a Rubik's cube). Coupled, where the result of the performance happens very near to the artefact where the action was initiated (e.g., connecting two objects), and nongraspable: where the artefact has a relationship with another detached artefact, away from the original action (e.g., a buzzer that goes off when the player makes a mistake).

Function [ 8,18 ] explains how the actions made through the artefact impact the story. These actions guide the interactor into different paths in the narrative. This quality considers two values: ontological: where actions lead the interactor into different paths and allow altering the plot. In contrast, the value of exploratory describes actions that can change in the narrative's perspective or examine new relationships in the narrative without changing the plot.

While their qualities condition the way people interact with these artifacts, the embodied and tangible methods used also play an essential role [ 19–22 ]. These methods can be space-directed, which refers to how a person negotiates and navigates a space by manipulating an object (e.g., moving an artefact from one point to another). Artifact-directed methods indicate how the person manipulates, interacts, and creates relationships between one or more artifacts (e.g., tapping an artifact against another). Lastly, body-directed methods relate to the person’s awareness of their bodies; it considers their range of motion, senses, and other body-related skills.

Artifacts have a crucial role in connecting the storyworld and the real world. The configurations of these artifacts mediate between the narrative and the person conditioned by the relationships between the space, the body, or other artifacts. Through these everchanging interconnections between values, an artifact tells a story and supports the person’s imagination, fosters new motivations to identify and feel empathetic to the story or the characters and feel rewarded about the experience.

2. A Design Case: The Non-myth of the Noble Red

Grounded on the learnings from the design of Letters to José, The Non-myth of the Noble Red expands upon the architecture presented in the previous section [23]. Inspired by Paolo Uccello’s “St. George and the Dragon” and U.A Fanthorpe’s poem “Not my best side” [24], the narrative borrows principles from Bertolt Brecht’s Epic Theatre [25] as a way of exploring alternative approaches to the authoring of interactive digital narratives. While the myth of St. George can be described as the formulaic fight between good and evil, Fanthorpe’s poem critiques established gender and mythological roles. Informed by Brecht’s Verfremdung effect, V-Effekt, also known as estrangement, distancing, or the alienation effect [25], many representative aspects of the original story are questioned through interruptions and inconsistencies, breaking the illusion of the story, and as encouraged by Brecht, facilitate reflection and political action of the persons engaging with the story. For instance, the Dragon, St. George, and the Maiden from the original story were abstracted in The Non-myth of the Noble Red to become the Villainous Yellow, the Heroic Blue, and the Noble Red, each represented by a paper puppet and played/performed by different users. As a form of distancing, the resulting characters are non-gendered beings that retain only a few traits from the original myth [23].

The Non-myth of the Noble Red plot is relatively simple: a community is under the threat of the Villainous Yellow. They decide to offer a sacrifice to deal with its anger, so the Noble Red is eventually sent to its lair. The Heroic Blue hears about this and decides it must do its part and save everyone. At this point, the story breaks away from tradition and offers more questions than answers: Does the Noble Red want to be saved? Can it save itself without help from the Heroic Blue? Does the Villainous Yellow genuinely want to be a villain? Is the Heroic Blue indeed needed? Is the Noble Red really the weakest link in the story? It is up to the users to answer these questions. The story presents three physical spaces where each act unfolds: the first space is an individual but connected space where users interact with others; the second space hints at the cave where the conflict happens; a collaborative space where the users work together to defeat the villain (or not), and the third space, a shared one, is inhabited only by certain characters depending on the outcome of the conflict.

The Non-myth of the Noble Red places the confrontation from the original myth at the centre of the experience, not as its starting point but as the arriving point of individual yet interconnected storylines. From an architectural perspective, these storylines lead to conflict resolution in multiple ways depending on the actions and decisions taken by each user during the first act (Figure 2). With the resolution of the conflict, two users can follow a single path, take divergent paths, or arrive at no ending at all. The major arcs describe each character's path until the conflict and subsequent resolution. The secondary arc focuses on other supporting characters (i.e., the ruler and the helper) that facilitate the arrival of the users to points where they diverge or converge, changing the conditions for the conflict.

The paper puppets in The Non-myth of the Noble Red are worn in the user’s hand, like traditional glove puppets [26], with mechanisms inspired by traditional Indonesian puppetry and Eastern European marionettes but computationally enhanced (Figure 3). This way, it is possible for the puppets (and the users) to interact with other puppets, spaces, and objects. Puppets, as interfaces, have positive cognitive effects: they facilitate self-recognition [27], improve digital/physical interactions [28], strengthen the sense of agency in the users, and make the storytelling process more expressive and dynamic [ 11 ].

In the latest prototype of The Non-myth of the Noble Red, the puppets are tethered to the wrist of the user for structural support and balance, where a Wi-fi enabled ESP32 microcontroller, and a LiPo battery is housed and connected to sensors distributed across the puppet (Figure 3b). These sensors are a PN532 NFC/RFID module, a TCS34725 RGB colour sensor, two capacitive sensors, and an MPU 6050 3-axis accelerometer and a 3-axis gyroscope sensor. Additionally, puppets can “talk” between themselves and the physical spaces using a client/server architecture based on NodeRed and p5js.

Considered as artifacts for storytelling, the puppets are transdiegetic artifacts, that can make ontological changes to the plot based on decisions and actions performed by the user. Because of their diegetic nature, the puppet can interact with objects and other puppets using artifact-directed methods such as tapping and “touching”, thanks to the NFC reader placed in the puppet’s hand. By relying on space-directed methods, the location of the puppet is tracked thanks to a matrix of capacitive surfaces laid on the space, along with a series of coloured shapes placed over these surfaces. When one of the capacitive elements is triggered, the colour sensor placed on the foot of a puppet, reads the colour it is standing on. If the colour matches the one of the puppet, then the system assumes its location. This way, the puppet can “walk” and be tracked when moving from one place to another (Figures 3c and 3d). While this solution is not precise—although fine accuracy is not required—it removes the need for complex tracking hardware. Body-directed methods are critically relevant because the puppet is controlled through hand movement. For instance, the user mimics walking with the index and middle fingers as they are placed on the puppet's legs while it controls the movement of the puppet’s hand through a rod connected to the user’s thumb (Figure 3a). With the MPU 6050 sensor placed in the puppet's body, the system can track movement to its left or right. Combining these movements adds naturality and widens how the user interacts with different narrative aspects.

However, while the puppet, in a certain way, requires the user to behave and act as the character, the user is intentionally distanced by the way it performs through the puppet. For instance, the user does not see her performance but the back of the puppet—although it can see at the performances of others. In a Brechtian manner, all technological means are visible to the user but hidden to others, while certain intentional limitations on how the puppet can move make certain movements uncomfortable or challenging to perform. These limitations break the immersion and illusion of the narrative. Finally, the two capacitive sensors on the microcontroller housing allow the user to make "a" or "b" choices. (Figure 3b) The text is authored to distance the user by presenting these choices not as if it was addressing the character of the story but as someone who is consciously controlling a character.

3. Conclusions

This short paper focused on a particular type of interactive media: Tangible Narratives. These narratives combine aspects, principles, and concepts brought from the fields of Tangible Interaction and the study of narratives. By briefly introducing previous research on a tangible narrative, namely Letters to José, the author reviewed design considerations that guided the formulation of an authoring architecture and a typology of tangible artifacts with narrative purpose—or artifacts for storytelling [ 7,12 ]. Notably, this typology unifies concepts from tangible interaction and narratology.

Based on this architecture and exemplified in the typology, the paper presented The Non-myth of the Noble Red. This tangible narrative expands upon the learnings gained from the design of Letters to José and integrates concepts from the study of performative art, particularly Brechtian theatre. This pairing aims to look at performative arts as the source for new and divergent ways to design and author interactive media, as suggested by other authors [ 9,29–31 ]. This approach places interest on the critical discourse of the narrative and the exploration of alternative aesthetical approaches to storytelling. The paper also discussed the technological considerations in the latest prototype of The Non-myth of the Noble Red and related them to the artifacts for storytelling typology.

While world-building in Letters to José was about representing a reality and interactions happened according to that reality, in The Non-myth of the Noble Red, world-building is about hinting and symbolizing a secondary world where people can perform through physical puppets but also observe and reflect while others perform during short alienating pauses. In Letters to José, the purpose of authoring the story was about allowing `people to enact Jesús's life. In contrast, in The Non-myth of the Noble Red, authoring was about making visible, questioning, and challenging assumed gendered roles inherited from prototypical western literature by providing ways to bend, break, or follow them. Although there is still work ahead in developing a fully working version of The Non-myth of the Noble Red, the author hopes that the considerations and the design cases discussed here can inform future work in the design of tangible narratives.

4. References

[20] A. Valli, The Design of Natural Interaction, Multimedia Tools and Applications. 38 (2008) 295– 305. https://doi.org/10.1007/s11042-007-0190-z. [21] R.J. Jacob, A. Girouard, L.M. Hirshfield, M.S. Horn, O. Shaer, E.T. Solovey, J. Zigelbaum, Reality-based Interaction: A Framework for Post-WIMP Interfaces, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, Florence, Italy, 2008: pp. 201–210. http://dl.acm.org/citation.cfm?id=1357089 (accessed October 1, 2017). [22] A. Mazalek, O. Shaer, B. Ullmer, M.K. Konkel, Tangible Meets Gestural: Gesture Based Interaction with Active Tokens, in: ACM CHI 2014 Workshop on Gesture-Based Interaction Design, ACM CHI, 2014. [23] D. Echeverri, The Non-myth of the Noble Red: Exploring Brechtian Principles of Storytelling and Performance in the Authoring of a Tangible Narrative, in: Proceeding of CC’22, ACM Press, Venice, Italy, 2022: p. 6. https://doi.org/10.1145/3527927.3535207. [24] U.A. Fanthorpe, Not My Best Side, in: Selected Poems, Penguin Books, London, UK, 1986: pp.

28–29. [25] B. Brecht, Brecht on theatre, Bloomsbury Academic, London ; New York, 2019. [26] A. Tomanek, Forms of Puppets, Akademie múzických umění, Prague, Czech Republic, 2016. [27] A. Mazalek, M. Nitsche, S. Chandrasekharan, T. Welsh, P. Clifton, A. Quitmeyer, F. Peer, F.

Kirschner, Recognizing self in puppet controlled virtual avatars, in: Proceedings of the 3rd International Conference on Fun and Games - Fun and Games ’10, ACM Press, Leuven, Belgium, 2010: pp. 66–73. https://doi.org/10.1145/1823818.1823825. [28] M. Sakashita, T. Minagawa, A. Koike, I. Suzuki, K. Kawahara, Y. Ochiai, You as a Puppet: Evaluation of Telepresence User Interface for Puppetry, in: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, ACM, Québec City QC Canada, 2017: pp. 217–228. https://doi.org/10.1145/3126594.3126608. [29] C. Fernández-Vara, Play’s the Thing: A Framework to Study Videogames as Performance, in: 2009 DiGRA International Conference: Breaking New Ground: Innovation in Games, Play, Practice and Theory, Digital Games Research Association, 2009 DiGRA, 2009: p. 10. [30] H.Y. Nam, M. Nitsche, Interactive installations as performance: inspiration for HCI, in: Proceedings of the 8th International Conference on Tangible, Embedded and Embodied Interaction - TEI ’14, ACM Press, Munich, Germany, 2013: pp. 189–196. https://doi.org/10.1145/2540930.2540976. [31] H. Pötzsch, Playing Games with Shklovsky, Brecht, and Boal: Ostranenie, V-Effect, and SpectActors as Analytical Tools for Game Studies, Game Studies. 17 (2017). http://gamestudies.org/1702/articles/potzsch (accessed December 22, 2021).

Annex: Complementing images from the ETIS’22 presentation

References

[1] H. Koenitz , Towards a Specific Theory of Interactive Digital Narrative , in: H. Koenitz , G. Ferri, M. Haahr , D. Sezen , T. Sezen (Eds.), Interactive Digital Narrative: History, Theory and Practice , Routledge, London, UK, 2015 : pp. 91 - 105 . https://doi.org/10.4324/9781315769189.

[2] J.H. Chu , Designing Tangible Interfaces to Support Expression and Sensemaking in Interactive Narratives , in: Proceeding of TEI 2015 , ACM Press, Stanford, CA, US, 2015 : pp. 457 - 460 . https://doi.org/10.1145/2677199.2693161.

[3] C. Sylla , P. Branco , C. Coutinho , E. Coquet , D. Skaroupka , TOK: A Tangible Interface for Storytelling , in: Proceedings of CHI 2011 , ACM Press, Vancouver, BC, Canada, 2011 : pp. 1363 - 1368 . https://doi.org/10.1145/1979742.1979775.

[4] D. Petrelli , A. Soranzo , L. Ciolfi , J. Reidy , Exploring the Aesthetics of Tangible Interaction: Experiments on the Perception of Hybrid Objects , in: Proceedings of the TEI '16 , ACM Press, Eindhoven, Netherlands, 2016 : pp. 100 - 108 . https://doi.org/10.1145/2839462.2839478.

[5] C. Sylla , S. Gonçalves , P. Brito , P. Branco , C. Coutinho , A Tangible Platform for Mixing and Remixing Narratives , in: ACE 2013: Advances in Computer Entertainment, Springer, Cham, Twente, the Netherlands, 2013 : pp. 630 - 633 .

[6] A. Alves , R. Lopes , P. Matos , L. Velho , D. Silva , Reactoon: Storytelling in a Tangible Environment , in: 2010 Third IEEE International Conference on Digital Game and Intelligent Toy Enhanced Learning , IEEE, Kaohsiung, Taiwan, 2010 : pp. 161 - 165 . https://doi.org/10.1109/DIGITEL. 2010 . 28 .

[7] D. Echeverri , H. Wei , Letters to José: A Design Case for Building Tangible Interactive Narratives , in: A. -G. Bosser , D.E. Millard , C. Hargood (Eds.), Interactive Storytelling. ICIDS 2020 , Springer, Cham, Bournemouth, UK, 2020 : pp. 15 - 29 . https://doi.org/10.1007/978-3- 030 -62516- 0 _ 2 .

[8] D. Harley , J.H. Chu , J. Kwan , A. Mazalek , Towards a Framework for Tangible Narratives , in: Proceedings of TEI '16 , ACM Press, Eindhoven, Netherlands, 2016 : pp. 62 - 69 . https://doi.org/10.1145/2839462.2839471.

[9] B. Laurel , Computers as Theatre, Second Edition, Addison-Wesley Professional , Boston, MA, US, 2013 .

[10] M. Green , J.K. Donahue , Simulated Worlds: Transportation Into Narratives , in: K.D. Markman , W.M. Klein , J.A. Suhr (Eds.), Handbook of Imagination and Mental Simulation , Taylor & Francis Group, New York, NY, USA, 2014 : pp. 241 - 256 . https://doi.org/10.4324/9780203809846.ch16.

[11] D. Echeverri , H. Wei , Exploring the Experience with Tangible Interactive Narrative: Authoring and Evaluation of Letters to José ., Entertainment Computing Journal. ( 2022 , in review) 19 .

[12] D. Echeverri , H. Wei , Designing Physical Artifacts for Tangible Narratives: Lessons Learned from Letters to José , in: Proceeding of TEI 2021 , ACM Press, Salzburg, Austria, 2021 . https://doi.org/10.1145/3430524.3446070.

[13] K.L. Walton , Pictures and Hobby Horses: Make-Believe beyond Childhood , in: Marvelous Images: On Values and the Arts , Oxford University Press, Oxford ; New York, 2008 : pp. 63 - 78 .

[14] G. Genette , Narrative Discourse: An Essay in Method , Cornell University Press, Ithaca, New York, 1983 .

[15] K. Jørgensen , On Transdiegetic Sounds in Computer Games, Northern Lights: Film and Media Studies Yearbook . 5 ( 2007 ) 105 - 117 . https://doi.org/10.1386/nl.5.1. 105 _ 1 .

[16] K.P. Fishkin , A Taxonomy for and Analysis of Tangible Interfaces , Personal and Ubiquitous Computing . 8 ( 2004 ) 347 - 358 . https://doi.org/10.1007/s00779-004-0297-4.

[17] B. Ullmer , H. Ishii , Emerging Frameworks for Tangible User Interfaces , IBM Systems Journal. 39 ( 2000 ) 915 - 931 . https://doi.org/10.1147/sj.393.0915.

[18] M.-L. Ryan , Beyond Myth and Metaphor: Narrative in Digital Media, Poetics Today. 23 ( 2002 ) 581 - 609 .

[19] E. Hornecker , J. Buur , Getting a Grip on Tangible Interaction: A Framework on Physical Space and Social Interaction , in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI 2006 , ACM, Montreal, Quebe, Canada, 2006 : pp. 437 - 446 . http://dl.acm.org/citation.cfm?id= 1124838 ( accessed October 7 , 2017 ).