Storytelling with physical artefacts provides young children with opportunities to explore and express their creativity. Phygital artefacts-physical objects imbued with embedded physical computing technology- further expand children's expressive potential, providing multisensory experiences and opportunities for storytelling. This research focuses on enabling children aged 5 to 8 years to engage in storytelling using phygital artefacts, while empowering educators to tailor the interactions of those phygital artefacts via a companion software application. In parallel, the study examines the usage context and the physical computing materials necessary for creating such a toolkit. This involves identifying and empathising with primary users and understanding how storytelling activities are conducted with diferent artefacts in educational settings. A literature review underpins the research, and expert-based studies were conducted with educators and other domain experts, followed by user studies with pre-school and primary school children in educational settings. Insights from the literature and user studies yielded user requirements, which informed the technical decisions made for the toolkit, including the selection of an appropriate microcontroller as well as the necessary sensors, actuators, and power requirements, along with the development of its companion software application.
Storytelling is an “important and developmental arena for children” highlighted by Cassell and Ryokai
[
Rapid advancement in technology, including the development of physical computing boards, for
instance BBC Micro: Bit, Adafruit Circuit Playground, and Espressif ESP32 has expanded the spectrum
of children’s expressive potential, allowing for the construction of various phygital artefacts for their
storytelling [
Phygital artefacts open doors to multi-sensory experiences, for instance, Mobeybou and I-Theatre
use screens that allow young children to create and narrate stories open-endedly, leveraging tangible
blocks coupled with RFID readers [
In the work by Raposo et al., children animated a given story by animating puppet characters of the
story, “indirectly, apprehending the fundamental concepts of input, output and processing of physical
computing” [
In other words, phygital artefacts allow children to use touch, sound, and movement to create engaging, open-ended storytelling experiences [14]. Aligned with constructionism, which maintains that young children can grasp concepts better through the creation and sharing of meaningful objects, children can hone not only their open-ended storytelling abilities by playing with these phygital artefacts but also their Science, Technology, Engineering, and Mathematics (STEM) skills [15].
End-user development (EUD) research focuses on designing software applications for nonprogrammers, including both children and educators [16, 17]. Teachers and educators often lack the programming competencies–and the time and motivation– required to code the behaviour of phygital artefacts [17, 18]. Providing a dedicated software application that allows them to modify the behaviour of the phygital artefacts without coding can lower technological barriers and increase its adoption [19, 20].
In line with the existing literature that emphasises open-ended storytelling and EUD, this work focuses on investigating the design of a toolkit that allows children to express their stories through phygital artefacts open-endedly. In parallel, the work hereby reported examines the usage context and the physical computing materials necessary for creating such a toolkit, along with a companion app, to be adopted by teachers and educators. This involves identifying and empathising with the primary users of the toolkit, and understanding how storytelling activities are conducted with diferent artefacts in pre-school and primary school settings.
This paper is organised to provide a structured overview of the research process and expected outcomes. Section 2 outlines participants and the main research questions guiding this work, followed by Section 3, which provides an overview of the overall research methodology that will be employed to answer these questions. Lastly, Section 4 outlines completed and future work.
The overall goal of this research is to explore the impact and opportunities of storytelling phygital artefacts-based toolkits. Children using the toolkit are early-year primary school children (5–8 years old in general). Answering the following research questions will help meet the goal. RQ1. What are the available toolkits for storytelling? • What are their strong points? • What are their limitations? RQ2. What are children’s interactions with storytelling phygital artefacts? • What is children’s experience with storytelling phygital artefacts? • What are children’s expectations for storytelling phygital artefacts? RQ3. What are design guidelines for future storytelling phygital artefacts-based toolkits?
In addition to researchers, several users are involved in the research. The primary users of the storytelling phygital artefacts-based toolkits are children from the last years of pre-school and first years of primary school (5–8 years old). The other secondary users are their teachers.
In future, other users may be school children, older than 8 years old, who learn how to program the behaviour of the phygital artefacts. Further, young children with intellectual disabilities (ID) may also be considered in future studies.
This work plans to use a Research through Design (RtD) methodology. The basic idea of RtD is to understand the problem and its context and use design skills to create phygital artefacts-based toolkits that solve the problem. It is a cyclical process of creating, reflecting, and refining until a satisfactory solution is found [21, 22].
The common goal of RtD is to promote the value of design activities to produce knowledge, specifically those activities involving prototypes. The emphasis is on the importance of creating prototypes, reflecting on their impact, and discussing and analysing their efects, sometimes even their creation process [23, 24]. In RtD, it is important to make sure that the design process is thoroughly documented, encompassing all stages involved [24].
Research through design consistently generates knowledge through prototypes [25]. Historical examples were highlighted by Zimmerman et al. and complemented by Giaccardi in their studies, such as “Keller’s Cabinet”, “Grove’s Drift Table” and “Wensveen’s Alarm Clock”, explain the concept of producing knowledge leveraging prototypes [26, 27, 23].
The RtD process in my PhD begins with a thorough context-of-use analysis, aimed at answering the RQ1 research question. An initial systematic literature review was conducted to explore the available toolkits, their strengths and limitations, and to understand how storytelling activities can drive the evolution of phygital artefacts for the primary users of the research [28]. Furthermore, the initial literature review was extended to cover for most recent works.
The RtD process continues with knowledge acquired through the context-of-use analysis, guiding the design of phygital artefacts for the research. Specifically, insights from the literature review enable us to generate initial design concepts that support the iterative evolution of phygital artefact-based toolkit through storytelling-driven interaction. These concepts were then converted into phygital artefacts and deployed in real environments such as schools and workshops; children first experienced the phygital artefacts and later imagined their future phygital artefacts. The phygital artefacts were then iteratively refined based on feedback and analysis gathered from the studies, along with suggestions from the ifeld experts. In other words, the phygital artefacts were used in the field to answer the RQ2 research question.
Furthermore, understanding the context of usage and identifying necessary materials for constructing the toolkit, facilitated its practical implementation and provided insights for future research guidelines, thus addressing the RQ3 research question.
The work started with an initial systematic literature review [28], followed by an extended literature review to account for recent and relevant publications. The design and development of phygital artefacts used in this research work have been shaped through a series of participatory workshops with pre-school teachers and children, listed in Table 1. Initially, two phygital artefacts were designed and developed based on the recommendations from the literature and two short-term field studies. Further phygital artefacts were refined in terms of technology and aesthetics based on the insights from the field studies in a summer camp with pre-school children. Finally, a phygital artefact was co-designed by pre-school children in a series of four bi-weekly workshops at a local pre-school and was named “Strichpiano”. Further, phygital hats were developed based on the insights and recommendations of the field studies and used in children’s storytelling workshop at the atelier [29].
Date June 2024 July 2024 Aug 2024 Sept–Oct 2024 Nov 2024
Context Short-term workshop 1 at pre-school Short-term workshop 2 at pre-school Summer-camp workshop 4 bi-weekly workshop at pre-school 4-day “Hat Atelier” workshop
Children 4 — 18 11 (avg) 15 (avg)
The analysis of the literature and data from the field studies–including children’s interactions with phygital artefacts and their ideas for future artefacts–provides the basis for designing and developing a toolkit that supports open-ended storytelling. The toolkit has been developed and is currently in the testing phase. In parallel, a software companion app was designed to allow users, such as teachers and educators, to adopt the toolkit and modify the behaviour of the phygital artefacts without writing code. Once the initial tests are concluded, it will be evaluated in the field with pre-school children and their teachers. Subsequently, it will be extended to test and evaluate with young children with intellectual disabilities and their therapists and caregivers.
Currently, we have had three papers accepted and published, one more is under review, and we are drafting another two. Looking forward, we will continue to share our research through reputable conferences and journals.
This research work is supervised by Prof. Rosella Gennari and Prof. Alessandra Melonio. It is funded by the PNRR scholarship.
During the preparation of this work, the author used Chat-GPT-4 in order to: Grammar, spelling check and paraphrase. After using these tool/service, the author reviewed and edited the content as needed and takes full responsibility for the publication’s content. smart-thing design with children through action research, International Journal of ChildComputer Interaction 31 (2022) 100359. URL: https://www.sciencedirect.com/science/article/pii/ S2212868921000660. doi:https://doi.org/10.1016/j.ijcci.2021.100359. [14] J. P. Hourcade, Interaction Design and Children, Found. Trends Hum.-Comput. Interact. 1 (2008) 277–392. URL: http://dx.doi.org/10.1561/1100000006. doi:10.1561/1100000006. [15] S. Papert, I. Harel, Situating constructionism, Ablex Publishing Corporation, ???? [16] H. Lieberman, F. Paternò, M. Klann, V. Wulf, End-User Development: An Emerging Paradigm, Springer Netherlands, Dordrecht, 2006, pp. 1–8. URL: https://doi.org/10.1007/1-4020-5386-X_1. doi:10.1007/1-4020-5386-X_1. [17] P. Markopoulos, J. Nichols, F. Paternò, V. Pipek, Editorial: End-user development for the internet of things, ACM Trans. Comput.-Hum. Interact. 24 (2017). URL: https://doi.org/10.1145/3054765. doi:10.1145/3054765. [18] T. McGill, C. Klisc, End-user perceptions of the benefits and risks of end-user web development, Journal of Organizational and End User Computing 18 (2006) 22–42. URL: https://services.igi-global. com/resolvedoi/resolve.aspx?doi=10.4018/joeuc.2006100102. doi:10.4018/joeuc.2006100102. [19] M. Andrao, F. Gini, F. Greco, A. Cappelletti, G. Desolda, B. Treccani, M. Zancanaro, "react", "command", or "instruct"? teachers mental models on end-user development, in: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI ’25, Association for Computing Machinery, New York, NY, USA, 2025. URL: https://doi.org/10.1145/3706598.3713234. doi:10.1145/3706598.3713234. [20] S. Wiedenbeck, Facilitators and inhibitors of end-user development by teachers in a school, in: 2005 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC’05), 2005, pp. 215–222. doi:10.1109/VLHCC.2005.36. [21] H. W. Rittel, M. M. Webber, Dilemmas in a general theory of planning, Policy sciences 4 (1973) 155–169. [22] D. A. Schön, The Reflective Practitioner: How Professionals Think in Action, Basic Books, 1983. [23] P. Stappers, E. Giaccardi, Research through design, Retrieved April 24, 2024, from https://www.interaction-design.org/literature/book/ the-encyclopedia-of-human-computer-interaction-2nd-ed/research-through-design, 2014. [24] J. Zimmerman, E. Stolterman, J. Forlizzi, An analysis and critique of research through design: towards a formalization of a research approach, in: Proceedings of the 8th ACM Conference on Designing Interactive Systems, DIS ’10, Association for Computing Machinery, New York, NY, USA, 2010, p. 310–319. URL: https://doi.org/10.1145/1858171.1858228. doi:10.1145/1858171. 1858228. [25] D. Fallman, Design-oriented research versus research-oriented design (2004). [26] T. Djajadiningrat, S. Wensveen, J. Frens, K. Overbeeke, Tangible products: redressing the balance between appearance and action, Personal Ubiquitous Comput. 8 (2004) 294–309. URL: https: //doi.org/10.1007/s00779-004-0293-8. doi:10.1007/s00779-004-0293-8. [27] E. Giaccardi, Histories and futures of research through design: From prototypes to connected things, International Journal of Design 13 (2019) 139–155. [28] R. Gennari, M. B. Khan, A. Melonio, Storytelling with technology-enhanced artefacts: A literature review of toolkits for children, in: C. Herodotou, S. Papavlasopoulou, C. Santos, M. Milrad, N. Otero, P. Vittorini, R. Gennari, T. Di Mascio, M. Temperini, F. De la Prieta (Eds.), Methodologies and Intelligent Systems for Technology Enhanced Learning, 14th International Conference, Springer Nature Switzerland, Cham, 2024, pp. 243–254. [29] R. Gennari, M. B. Khan, A. Melonio, Storytelling and phygital artefacts for preschools: the case study of the hat atelier, in: Proc. of the 15th International Conference on Methodologies and Intelligent Systems for Technology Enhanced Learning, Springer, 2025.