Usability and engagement of a digital and multisensorial tool for immersive storytelling: a pilot study *

Abstract

Storytelling is a crucial practice to help children to develop linguistic, cognitive, emotional and social skills. New methodologies for interactive storytelling, as Tangible User Interface (TUI), have promoted innovative, educative and cognitive interventions to listen and stimulate narratives. The aim of the study is to analyze and evaluate the usability and the engagement with respect to a TUI-based interactive storytelling for preschool and school aged children. 11 typically developing children aged between 5 and 9 years old participated in the study. The procedure involved a TUI pre-training phase and the storytelling phase, in which participants listened to and interacted with a TUI-augmented story. Data, such as participants' behavior, satisfaction, engagement and TUI's usability, were collected through observations and specific questionnaires adapted to the participants. Results of the pilot study showed a good level of usability and overall engagement, with differences that vary according to age. Many of the participants considered the activity as challenging. We also highlight issues and technical notes that we need to address for future studies. Innovative interactive systems allow children to use physical objects immersed in the environment, interacting together with the digital world, supporting their physical and cognitive development. Future studies' directions are discussed.

Keywords

• TUI
• Storytelling
• Usability
• Engagement
• Multisensory approach

-

Storytelling is the art of telling stories and composing narratives, a means to report real or fictitious events through words, images, sounds. Since ancient times narrating and building stories was a common practice, through which create shared identities, positive values and an active and participatory spirit [ 1 ]. The act of narrating, in story*

Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). telling, is found in human experience and can be represented in various forms (individual or collective) that connect thought and culture.

Storytelling practice enhances children development: first, it can sustain language development. Several studies have shown how repeated exposure to stories listening contributes to the development and enhancement of some key skills in language acquisition such as lexical, phonetic, morphological, syntactic, as well as pragmatic skills [ 2 ] [ 3 ] [ 4 ]. Storytelling exposure can enhance the representation of the structure of events and of the memory processes, the organization and planning of narrative discourse [ 5 ], but also listening skills, attention and memory [ 6 ] [ 7 ], social skills [ 8 ], fantasy and creativity [ 9 ].

For young children, both typically and atypically developing ones, there has always been and there will be a need to target explicitly narrative skills that will prepare them for academic outcomes, emotional expression, social interaction. Narrative comprehension and narrative production, which includes retells, generating personal or fantasy stories, can help establish an adequate language foundation and serve to support the acquisition of other important skills for life. 1.1

Innovative tools for storytelling

Technologies are becoming more and more present in our lives and this stimulates reflections and questions regarding the potential impact that they can have on children's minds today. New methodologies and tools for storytelling have promoted innovative educative and cognitive interventions that exploit new technologies to stimulate narrative production [ 10 ]. Studies, see [ 11 ], found that traditional storybook reading intervention has a positive effect compared to screen-based intervention on improved vocabulary and decreased functional connectivity, so the hyperconnectivity during screen-based stories listening is associated with lower narrative comprehension. A review [ 12 ] highlights how technology provides a small but significant enhancement to story comprehension and expressive word learning for young children, however interactive elements were especially distracting for these children.

In the field of new technologies, hybrid and innovative tools have been created, namely Tangible User Interfaces (TUIs): TUIs allow users to interact with the digital world through the manipulation of objects belonging to the physical world, and therefore tangible [ 13 ]. Objects are augmented and made "intelligent" through the exploitation of Internet of Things technologies [ 14 ] such as touchscreens, networks and sensors wireless, sensors, actuators and RFID or NFC systems. The interaction with these tangible interfaces by the user causes consequences in the virtual world.

Since multisensorial learning extends the richness of an interaction by engaging the haptic, sound and visual senses, a TUI for multisensorial learning can generate haptic sensation, due to its connection to a meaningful object, and can also enhance media content. TUI’s approach is part of the Embodied and Situated Cognition Theory (ESCT) framework [ 15 ] and has its roots in the classical psycho-pedagogical theories [ 16 ] [ 17 ], which highlight the benefits of using and manipulating both concrete and abstract objects. When a technology is developed for the enhancement of all the senses, such as auditory, visual, tactile, olfactory, it stimulates our neuro-cognitive structures delegated to the perception and elaboration of those stimuli.

A solution to a digital only intervention could be that of exploiting the potential of Tangible User Interfaces mixed with multisensorial storytelling. Multisensory training can recreate natural settings and are more effective for learning; moreover, multisensory and multimodal stimulation could better enhance the acquisition of information then unisensory stimulation alone [ 18 ].

Tangible User Interfaces prototypes and environments for augmented storytelling are numerous in the literature [ 19 ] [ 20 ], showing the potential of these tools for fostering developmentally advanced storytelling in children. However, few TUIs applications aimed at storytelling practice exploit multisensory augmented objects, the number radically decreases for the involvement of the sense of smell and taste. If TUIs can be an excellent learning tool in typically developing children, they could be considerably effective for children with disabilities, such as blind or visually impaired children, as TUIs allow learning through direct experience, manipulation and multisensory approach that includes residual senses for those users [ 21 ].

Accordingly, we developed a prototype of TUI technology aimed at learning and enhancement of storytelling through multisensorial manipulation, based on previous experiences [ 22 ]. A Tangible User Interface for the educational and psychological intervention experience needs to be appropriately designed for young children’s developing physical and cognitive skills [ 23 ] [ 24 ]. To benefit from embodied storytelling through a TUI methodology, first children need to engage with the technology itself. Young children’s physical and cognitive development constantly evolve and, consequently, the ability to use technology also evolve.

Therefore, the objective of this pilot study is to analyze and evaluate the usability of our methodology based on TUIs for preschool and school aged children, as well as the satisfaction and engagement with respect to the proposed activity. 2 2.1

Materials and methods

The tool development

Our tool for immersive storytelling is composed of software and some hardware components. The software used to create the story is STELT platform [ 25 ] that combines the management of hardware components (as sensors and antennas) and software components (programming and developing environments, programs and activities for users, database for tracking user behavior and an adapting tutoring system). STELT implements augmented reality systems based on RFID (Radio-Frequency Identification) and NFC (Near Field Communication) technology. The RFID/NFC tags are thin and can be applied to any type of object and are detected by small readers. Our reader, the MagicBoard [ 26 ], is a board that can be connected to a computer with either a wired or wireless connection. This recognizes tangible objects equipped with RFID/NFC tags, allowing users to drive the exercise/scenario, based on the requests shown in the screen and with an aural output. STELT combines communication protocols with the various hardware devices (readers and output devices). Then the device provides feedback based on the user activity with objects. 2.2

Participants and procedure

To carry out the pilot study we recruited 11 typically developing children (2 boys and 9 girls) aged between 5 and 9 years old. 6 children were based in Rome (Italy) and 5 children were based in the province of Naples (Italy). Every child was right-handed. No participant used a Tangible User Interface before the pilot study. Prior to the study, informed consent was obtained from all parents of the children involved.

One of our first objectives was to create a methodology and a technology that was flexible and adaptable to the most age-related educational and psychological necessity. We focused on the story listening and telling practice for preschool and school aged children. The age when children start school is a delicate phase of cognitive, emotional and social development and, on one hand, appropriate tools can enhance the acquisition of cognitive skills like comprehension, reading and storytelling, on the other hand they can be motivating since the innovative nature of new technology.

The Storytelling Kit (see Fig. 1) was composed of: a bracelet, a fairy doll, four lizards (all covered with different skins: normal lizard, feather, synthetic grass and wool), three jars with smells (apple, soap, burnt), two candy containers (one with cherry and one with strawberry candies). The Magic Board and the laptop used previously were always placed on the table; the software was developed with the STELT platform. Due to the current Covid-19 security measures, each child participated in the study in their own houses, ensuring parents that any object used for the study had been cleaned and sterilized before each administration. Before each study session, parents were asked to fill in a questionnaire regarding general information of the child and previous experience with TUIs. Particularly we asked whether the children made use of technological tools and how long per day.

Each session took place in a silent room in the participants' house. In the room there were the child and two researchers who were the experimenters, one observer and one who interacted with and tutored each child if needed. At the beginning of the session, participants carried out a pre-training phase on the use of TUI. This was made to ensure that all children were able to interact with the technology in the right way, therefore, to make them understand how it worked. Particularly, children were asked to try a game of the “Block Magic” software [ 27 ], in which the goal is to find the right shape among all the physical logical blocks presented; the user places various shapes on the board until he finds the right one, guided by the software feedback.

Children were seated in front of a table where only the Block Magic kit, a laptop and the MagicBoard were placed. They were first introduced to the technology, then let observe the material and freely interact, and finally told that they were going to play a game that would have helped them understand the TUI functioning.

After a while from the pre-training phase, which lasted approximately 5 minutes, the Block Magic kit was replaced with the Storytelling Kit. The story was created specifically for the study. The tale title is “Smilzon and the Sugar Fairy”. As in the pre-training phase children were let observe the material and told that they were going to listen to a story and that throughout the story they would also be asked to use the objects placed in front of them. Each child wore the bracelet on the right hand as the application started only when children placed the bracelet on the board. The structure of the story included 10 different scenes and for each scene the story reproduced an image of a background, an audio and a request (i.e. “Find Smilzon and put it on the Magic Board”). To go forward in the tale children had to explore the objects available, the various textures, the various smells and flavors made available in the playground and position the key object on the board to unlock the scenes. The story was therefore lived, in line with the hybrid approach, both digitally (through audio and video stimuli reproduced from the PC) and through exploration and manipulation with touch, smell and taste with the physical learning environment. The requests made by the story were aimed at enriching the understanding of the single scene of the story told by the audio-video. At the end of the session children were asked to answer simple questions about usability and engagement (for details see section below). 2.3

Usability and engagement measures

The parameters evaluated in our pilot study were the following: participants’ behaviors, interactions and facial expressions; satisfaction; usability of the TUI and engagement. We also decided to ask 4 questions related to the story.

For recording most of all participants’ behaviors and facial expressions during children's interaction with the story enhanced by the TUI, the observer experimenter took notes on an observation checklist created for the present pilot study. The experimenter observed whether the child took items and placed them on the board when required, whether the child looked at the objects before placing them and kept the attention focused on the activity. Moreover, those behaviors were observed: if the child stopped the activity without notifying or asked for a break, unsolicited actions, facial expressions, mistakes during the interaction with the story, whether the story was completed and finally the setting elements where the pilot test was conducted. Everything expressed verbally by the participants was recorded and transcribed.

We decided to add simplified questionnaires to be administered to the children once the activity ended for bringing out any additional considerations from children. As one aim of the study was to measure the overall pleasantness of the activity and use of the TUI, participants were also asked to answer to a questionnaire adapted to their age [ 28 ] both with visual and literacy stimuli, with the help of the experimenter, who also read the questions. The questions asked to measure acceptance of the activity were: “How much fun did you have?” which answers were recorded on a 5 item Likert Scale from “little” to “highly”, and “What did you like most?” which answers were “Listen to the story”,” Touch the objects” and “Smell and taste”.

Usability is the main criterion to evaluate the degree to which interactive systems the technology meets the needs of the target for which it was designed [ 29 ]. To measure the usability of the Storytelling TUI, children were asked to answer a question, "What was it like listening to the story and choosing together the objects?" and choosing between 3 different answers: “easy”, “so and so”, “difficult”.

Read and colleagues [ 30 ] observed children’s facial expressions to identify signs of engagement and fun with respect to a novel interface, and linked engagement with the “endurability” of the experience. Endurability is the likelihood of remembering enjoyable situations and intending to perform them again. Therefore, the question “Would you like to do this again?” was asked to children to measure their interest in repeating the activity. Children were asked to motivate their answers. Finally, to briefly evaluate participants' understanding of the story, some story-related questions were asked to children. Therefore, we divided the questions in two categories: context questions and action questions. The first ones refer to events that happen during the story (such as ”what does Smilzon ask the fairy?”) to evaluate the attention to the story itself, excluding the direct intervention of the user; the second ones refer to tasks that the user has to achieve to interact with the story, for example choose from smell jars (such as “what odor can you smell near the fairy house?”). The possible answers are three and are divided in: the right one and two distractors. 3

Results

The qualitative analysis was based on the analysis of questionnaires, observational notes and digitized audio recordings of the sessions. 3.1

Questionnaires

Overall, all the children enjoyed the activity, and almost all, 10 out of 11, responded "very" or "very much". Only a 5-year-old boy replied that he did not find it pleasant, during the activity he was restless and not very attentive to requests. After the question "what did you like most?", more than a half of the children, 7 out of 11, replied "smelling and tasting" showing great enthusiasm, half of the rest chose "listen to the story" and the other half "touch the objects". Positive responses also emerged regarding the use of objects to unlock the story; all the children responded from "enough" to "very much". All the children considered the activity easy and almost all expressed their willingness to repeat it again, except for 2 children, a male of 5 who had previously expressed that he did not like the activity, and a girl who claimed not to want to repeat it at that moment because she would have preferred to repeat the training activity with the logic blocks.

With respect to the questions related to story comprehension (see Fig. 2), almost all children, 9 out of 11, correctly answered the context questions, that is, those concerning the plot of the story; only the youngest girl incorrectly answered both the questions. Instead, some children, 5 out of 11, gave the wrong answer to the action questions, especially 4 of them incorrectly answered the one concerning the requested smell, confusing it with the taste tasted during the story.

Fig. 2. Users answers for the story comprehension. In the upper figures correct/incorrect answers for the two context questions; in the lower figures the two action questions. 3.2

Observation analysis and additional notes

The qualitative observations reported by the researchers highlight that all the children completed the storytelling activity without stopping it, nobody asked for a break or interrupted the activity without warning. Most of the children kept their attention focused on the activity and managed to shift the attention between the digital story and the observation and exploration of objects. Some children, precisely younger children (5 years) paid more attention to objects than to the story itself, because they were very active in observing and manipulating the tangible characters; sometimes they lost focus on the progress of the story and made some mistakes in choosing the objects to be placed. 4 children tended to place the objects on the board before the instruction, as they wanted to demonstrate that they already knew the answer; this led them to make some mistakes.

Regarding emotional expression, the children all showed expressions of fun and pleasure, some showed calm and focused expressions. Only the child who stated he did not like the activity showed expressions and vocalizations of frustration since he made numerous errors in choosing the objects, probably due to his little attention to the story. The expressions of surprise were expressed when children saw tangible objects for the first time, especially the characters with different textures.

Some useful technical notes on usability have been highlighted and they will guide the development of an improved version of the tool. Among these, the bracelet annoys almost all children, therefore, should be avoided. There is a need to introduce a longer pause or stronger feedback when the correct object is placed, as children often did not immediately understand that it was the correct choice. Furthermore, since the digital graphic part only included a change of backgrounds, the children expected to see something more happening on the screen when they placed a correct object. It was therefore thought to also add the digital version of the characters in the background.

Finally, the “Block Magic” training activity was challenging and extremely engaging, so children were attracted to the game; someone also asked to repeat the activity. Therefore, participants could have perceived the storytelling activity less challenging than the first one. This could also have reinforced their tendency to position objects before the instruction, to show they already knew the answer. 4

Discussions and conclusions

Storytelling is a valid learning and capacity-building tool and can enhance communication and language, social intelligence and emotional expression. The present study was aimed at evaluating usability and engagement of a digital and multisensorial tool for immersive storytelling. Results of the pilot test showed a good level of usability and overall engagement, with differences that vary according to age. However, few males were included in the pilot study, due to limitations in the recruitment process. Differently from adults, young children interaction with technology may influence their development, especially in a period of rapid growth in cognitive abilities. New technological tools such as tablets and smartphones have opened an attractive and stimulating world which, however, could lead to a shutdown in the digital system and affect the development of thinking and cognitive processes. Innovative interactive systems allow children to look up from the screens towards the physical objects of our world (tactile, smells, flavors) and to enhance the ability to reflect on the real environment by interacting with it together with the digital world, supporting their physical and cognitive development [ 29 ].

Following this path, one future direction of our research is to evaluate how a TUI that provides narrative stimuli could be effective for children’s narrative comprehension and consequently story-retell quality, comparing the impact of different narrative-stimuli presentation methods. Another hypothesis is that the effect could vary between specific ages, particularly preschool age and school age. Any observed effects on comprehension and retell will be studied and analyzed in relation to abilities that could influence them, such as attention, memory, inference making skills.

The essential TUI’s features make it suitable for people with visual disabilities, especially for the multisensory nature of the objects and the possibility of exploiting different sensory channels, so that residual senses and skills are sustained and compensated. Tangible technologies have been developed for children with visual impairments, but there are still very few, especially those for storytelling [ 31 ] [ 32 ] [ 33 ], and nobody has systematically studied the effects that a practice with TUI can have on enhancing storytelling. Thus, another important goal will be to evaluate the effects of our TUI technology in storytelling interventions for children with visual disabilities, as interventions to strengthen autonomy with everyday life stories but also to allow them to tell augmented personal stories.

References

1. Pellowski , A. : The world of storytelling . H.W. Wilson, Bronx, NY ( 1990 ).

2. Collins , F. : The use of traditional storytelling in education to the learning of literacy skills . Early Child Development and Care 152 , 77 - 108 ( 1999 ).

3. Craig , S. , Hull , K. , Haggart , A.G. , Crowder , E.: Storytelling: Addressing the literacy needs of diverse learners . TEACHING Exceptional Children 33 , 46 - 51 ( 2001 ).

4. Miller , S. , Pennycuff , L. : The power of story: using storytelling to improve literacy learning 1, 8 ( 2008 ).

5. Speaker , K.M. , Taylor , D., Kamen , R.: Storytelling: enhancing language acquisition in young children . Education 125 , 3 ( 2004 ).

6. Kromka , S.M. , Goodboy , A.K. : Classroom storytelling: using instructor narratives to increase student recall, affect, and attention . Communication Education 68 , 20 - 43 ( 2019 ).

7. Gallets , M.P. : Storytelling and story reading: a comparison of effects on children's memory and story comprehension . Electronic Theses and Dissertations ( 2005 ).

8. Koivula , M. , Turja , L. , Laakso , M. L. : Using the storytelling method to hear children's perspectives and promote their social-emotional competence . Journal of Early Intervention 42 ( 2 ), 163 - 181 ( 2020 ).

9. Phillips , L. : Storytelling: the seeds of children's creativity . Australian Journal of Early Childhood . 25 , 1 - 1 ( 2000 ).

10. Wu , J. , Chen , D. T. V. : A systematic review of educational digital storytelling . Computers & Education 147 , 103786 ( 2020 ).

11. Farah , R. , Meri , R. , Kadis , D.S. , Hutton , J. , DeWitt, T., Horowitz-Kraus , T. : Hyperconnectivity during screen-based stories listening is associated with lower narrative comprehension in preschool children exposed to screens vs dialogic reading: An EEG study . PLOS ONE 14 , e0225445 ( 2019 ).

12. Takacs , Z. K. , Swart , E. K. , Bus , A. G. Benefits and pitfalls of multimedia and interactive features in technology-enhanced storybooks: A meta-analysis . Review of educational research , 85 ( 4 ), 698 - 739 ( 2015 ).

13. Shaer , O. , Hornecker , E.: Tangible user interfaces: past, present, and future directions . HCI 3 , 4 - 137 ( 2010 ).

14. Angelini , L. , Couture , N. , Khaled , O.A. , Mugellini , E.: Internet of Tangible Things (IoTT): challenges and opportunities for tangible interaction with IoT . Informatics 5 ( 1 ), 7 ( 2018 ).

15. Clark , A. : Supersizing the Mind: Embodiment, Action, and Cognitive Extension . Oxford University Press, Oxford, New York ( 2008 ).

16. Montessori , M.: The absorbent mind . Holt Paperbacks ( 1995 ).

17. Piaget , J.: La representation du monde chez l'enfant . Paris, Alcan ( 1926 ).

18. Shams , L. , Seitz , A.R. : Benefits of multisensory learning . Trends in Cognitive Sciences 12 , 411 - 417 ( 2008 ).

19. Ryokai , K. , Cassel , J.: Computer support for children fantasy play and storytelling . Computer Supported Cooperative Learning , 99 ( 1999 ).

20. Yilmaz , R.M. , Goktas , Y. : Using augmented reality technology in storytelling activities: examining elementary students' narrative skill and creativity . Virtual Reality 2 , 75 - 89 ( 2016 ).

21. Brock , A. : Tangible Interaction for Visually Impaired People: why and how . In World Haptics Conference - Workshop on Haptic Interfaces for Accessibility , pp. 3 . Fuerstenfeldbruck, Germany ( 2017 ).

22. Di Fuccio , R. , Ponticorvo , M. , Ferrara , F. , Miglino , O. : Digital and multisensory storytelling: narration with smell, taste and touch . In European conference on technology enhanced learning , pp. 509 - 512 . Springer, Cham. ( 2016 ).

23. Radu , I. , MacIntyre, B. : Using children's developmental psychology to guide augmentedreality design and usability . In IEEE International Symposium on Mixed and Augmented Reality (ISMAR) ( 2012 ).

24. Radu , I. , Antle , A. : Embodied learning mechanics and their relationship to usability of handheld augmented reality . In IEEE Virtual Reality Workshop on K-12 Embodied Learning through Virtual & Augmented Reality (KELVAR) , pp. 1 - 5 ( 2017 ).

25. Miglino , O. , Di Ferdinando , A. , Schembri , M. , Caretti , M. , Rega , A. , Ricci , C. : STELT (Smart Technologies to Enhance Learning and Teaching): una piattaforma per realizzare ambienti di realtà aumentata per apprendere, insegnare e giocare . Sistemi intelligenti 25(2) , 397 - 404 ( 2013 ).

26. Di Fuccio , R. , Siano , G. , De Marco , A. : The activity board 1.0: RFID-NFC WI-FI multitags desktop reader for education and rehabilitation applications . In World Conference on Information Systems and Technologies , pp. 677 - 689 . Springer, Cham ( 2017 ).

27. di Ferdinando , A., di Fuccio , R. , Ponticorvo , M. , Miglino , O. : Block magic: a prototype bridging digital and physical educational materials to support children learning processes . In Smart Education and Smart e-Learning , pp. 171 - 180 . Springer, Cham ( 2015 ).

28. Read , J. C. : Validating the Fun Toolkit: an instrument for measuring children's opinions of technology . Cognition, Technology & Work 10 ( 2 ), 119 - 128 ( 2008 ).

29. Antle , A.N. : Research opportunities: Embodied child-computer interaction . International Journal of Child-Computer Interaction 1 , 30 - 36 ( 2013 ).

30. Read , J. , Macfarlane , S. : Endurability, engagement and expectations: measuring children's fun . In: Interaction Design and Children , Shaker Publishing, pp. 1 - 23 . Shaker Publishing ( 2002 ).

31. Edirisinghe , C. , Podari , N. , Cheok , A. D.: A multi-sensory interactive reading experience for visually impaired children; a user evaluation . Personal and Ubiquitous Computing , 1 - 13 ( 2018 ).

32. Cullen , C. , Metatla , O.: Co-designing inclusive multisensory story mapping with children with mixed visual abilities . In Proceedings of the 18th ACM International Conference on Interaction Design and Children . pp. 361 - 373 ( 2019 ).

33. Koushik , V. , Guinness , D. , Kane , S.K. : StoryBlocks: A tangible programming game to create accessible audio stories . In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems , pp. 1 - 12 ( 2019 ).