Addressing User Engagement With an Interactive Reading Model by Innovative Digital Expansion Nicola Barbuti, Mauro De Bari University of Bari Aldo Moro, Umberto I, Bari, Index, Italy Abstract The paper presents the research results aimed to design and realize an innovative prototype of user’s interactions method, based on the digital impact that Digital Transformation (DT) has on society. The experimentation presented takes as an example some good practices identified in the Victoria & Albert Museum (V&A), London: challenging, unconventional cultural goods could attract new users. Based on this assumption, we have observed a target users’ impact on a cultural good generally considered little appeal, transforming the user into a prosumer. The first tests were made in 2019. The University of Bari Aldo Moro and the De Sanctis Classical Linguistic High School of Human Sciences in Trani started a collaboration as part of a PON project titled "Comunicare e promuovere un evento culturale". High School students have been identified as an important target of study. Subsequently, in the paper, we examined a project created by D.A.BI.MUS S.r.l. and Quorum Italia S.r.l, working on niche cultural goods also in this case. In this project, an anastatic printed book became a potential endless book capable of intercepting the interest of even large groups of digital users. Keywords 1 Digital Transformation (DT), digital expansion, prosumer, user-centric, user perspective 1. Introduction Recently, designing and implementing pilot experiences has been consolidated in the diverse cultural heritage and culture sectors. Pioneering experiences can stimulate emotional impacts on users [1]. Moreover, in this period of Digital Transformation (DT) [2], digital technology offers a wide range of possibilities [3] with almost unlimited potential. This trend was accentuated during the pandemic caused by COVID-19, where the closure of cultural institutions generated a dichotomy in supply and demand. For several months, people have been thinking about dealing with the limits imposed by the impossibility of physically enjoying cultural experiences. The possibility of having ‘alternative’ interactive experiences available, therefore, has been transformed into a need. However, equally, this need has manifested a reasonable opportunity to develop new interactions with heritage [4]. The study and analysis of the digital proposals of some European leader institutions12show that among the various solutions promoted -calibrated according to the potentially affected users- the use of digital technologies maintains a first-level role in the creation of interactive experiences [5]. As an excellent digital innovation practice, we consider the proposals of the Victoria & Albert Museum (V&A) [6] in London, which presents some interesting user-centric approaches [7]. IRCDL 2022: 18th Italian Research Conference on Digital Libraries, February 24–25, 2022, Padova, Italy nicola.barbuti@uniba.it (N. Barbuti); mauro.debari@uniba.it (M. De Bari) 0000-0003-0817-4235 (N. Barbuti); 0000-0003-2730-2204 (M. De Bari) ©️ 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) 1 Some interactive proposals created by cultural institutions to promote user-centric digital experiences: https://www.rijksmuseum.nl/en/masterpieces-up-close; https://artsandculture.google.com/streetview/british-museum/AwEp68JO4NECkQ?sv_lng=- 0.1266024509257022&sv_lat=51.51905368906714&sv_h=129.83421644322323&sv_p=- 15.93602158329513&sv_pid=JeKwUFYAMWXNWPh3IOg3jw&sv_z=2.95691280674487; https://3d.smb.museum/pergamonaltar/. Katy Price, Head of Digital Media at V&A, argues that it is essential to create a compelling online experience that does not attempt to recreate the museum experience but instead offers something indeed distinctive [8]. It seems that the “traditional perception of heritage” could be overturned and modified towards a touch-free approach [9]. A solution initially conceived to encourage the flow of people with visual disabilities into museums [10] could be modified and adapted to different cultural aims, such as the emergency determined by the pandemic. Furthermore, a touch-free approach could be an offer comparable to the “traditional” one in the future. Nevertheless, the premise remains to generate an intensely emotional impact in users by involving them as participants in the experience of interaction with goods [11]. This approach is how some international cultural institutions operate more than before [12]. However, the promising scenario does not always consider an indispensable prerequisite correctly for activating exciting and sustainable solutions: digital as an “activator of interest” for visit-actors increasingly oriented towards active interaction. In this, the large digital platforms have taught us a fundamental rule: to put the user at the centre [13]. This need seems particularly crucial in the librarian field, considering the decreasing number of people who read. For this reason, it becomes essential to “define who the users are, including everything about work roles, sub-roles, user class definitions, and personas” [14] to address an issue that is affecting contemporary society. 2. State-of-the-art, how the digital affects users and reading Some recent industry analyses have confirmed that, in the new generations, reading tends to decrease more and more, in part, due to the DT [15]. Although, since the early 2000s, experiments have been started on printed texts mixed with digital solutions, it seems that expanding content with graphics, animations, and digital audio in Augmented Reality (AR) transforms printed books into a dynamic and engaging experience. This improvement could foster a return of interest in the paper-printed book. However, as suggested by Virginia Clinton after several studies, initial experimentations on digital books missed something. “My findings were not fair to screens because the screens could not offer everything they could… they were just a shiny piece of paper” [16]. Clinton, a strong supporter of digital books, has changed her mind because many students refused to study with digital support. Students considered those lacked suggestions and expansions. While by contrast, the introduction and the use of AR content are shifting this interpretation, trying to improve books perception and restoring the paper’s interest. As McKenzie and Darnell stated, the research on applications of AR to library contents was initially conducted mainly to design resources that fostered the learning of determinate categories of users [17]. The experiments would facilitate the understanding of two-dimensional printed contents by visualizing three-dimensional objects that expand them. From the studies made, researchers have deduced that the use of AR in the education system improves students’ learning by fixing abstract concepts [18]. In this perspective, “one of the most interesting uses in recent years among the promising educational applications are AR books” [19]. In the first decade of the new century, the first experiments of integrated texts with content in AR were implemented in mathematical publishing and biochemical [20]. Effectively, these pioneering experiments have initiated the research that led to the definition of the book model. Furthermore, the aim was to motivate the experience of users positively. The first somewhat evolved solution of this model was studied, in 2010, in the context of research on texts usable by subjects suffering from deafness [21]. However, all these experiments were based on AR configurations for desktop PCs. Severely limited production fallout from several critical issues: high costs, unattractive product appeal, limited user capabilities. In fact, despite an augmented book being considered, often by opponents, like a printed book, advocates sustain hardware and software implementations transform augmented books into complex digital cultural entities [22]. The breakthrough came with the move to AR solutions for mobile devices, well known and widely used by the different age groups of the new generation of digital natives. The model AR Book that launched this new phase of the research was the “MagicBook”, realized with the functions of Education and edutainment for children and young people of school age, which allowed users to read the physical book as a regular text and display 3D content via handheld [23]. It has been possible to create immersive and innovative solutions starting from this moment. The user is progressively more involved than in the past because of the spread of increasingly accessible digital technological solutions. Recently, the V&A has launched a project titled “Immersive Dickens” [24] that synthesizes the aim of this paper. The museum conceived the project considering manuscripts challenging objects. The V&A has identified a good that is generally little appreciated and usable by a non-specialized user, a manuscript, to make it expandable and attractive through mixed reality. The experience generated has promoted a double return for the museum in terms of user perception, creating unconventional attractiveness and bringing a different, more expansive audience closer2.3 Subsequently, the museum has identified the user target: “We want to involve the next generation in the development of a ‘next generation’ experience. We have chosen a demanding audience, in the shape of teenage students. By helping them discover and understand Dickens’s creative process, we aim to inspire young people’s self-expression” [25]. Finally, it has created the immersive experience to shift users from watchers to proactive prosumers. The analysis of this model and the previous research experiences are the assumptions that substantiate the methodological approach used in the research that will be described later. In particular, the goal is to show user interaction models, using an experiment conducted in some school libraries and presenting an innovative platform for the marker and the interactive use of paper books. Therefore, the methodologies of digitizing artefacts books for paper editions are integrated with the creative use of digital technology that favours interaction with dynamic content personalized by users. 2.1. Methodology The methodological proposal described results from some experiments that confirm the previous assumptions. We tested our research into two different contexts, the educational and the CCIs. 2.1.1. Expanded Library project In 2019, we started by looking for a target user to test. Finally, based on the excellent practice carried out by the V&A, considering that “The advent of augmented reality technologies often promises great potential for improving education”, we tested high school students as ideal users. In this orbit, the DISUM Department of the University of Bari Aldo Moro, in the course of 2019, collaborated with the De Sanctis Classical Linguistic High School of Human Sciences in Trani as part of the PON titled “Comunicare e promuovere un evento Culturale”3.4 The purpose of this project was divided into two phases. 1. The first was to implement the search for digital technologies capable of expanding the books identified by the students. 2. The second is to modernize the old school library using AR interaction modes. As described above, it emerges that the paper book approach is “difficult” for young people. Students expressed their perplexity in considering books as entertainment or cultural expansion elements. Instead, the high school students have expressed their rejection of books regarded as compulsory study tools. Hence the idea of making the school library more attractive. However, “when planning to design 2 “It can’t be handled or viewed in full; it’s full of complex handwritten revisions and not easy to read; it doesn’t have immediate visual impact and the connection with the rich imaginative world of Dickens’s fiction isn’t always obvious. But the manuscripts are the best evidence we have of the creative process of one of the UK’s most important and renowned authors, whose work has continuing relevance. Dickens was an observer of urban material culture and life in a period of rapid technological innovation and widespread financial insecurity, a writer who explored themes that still resonate today”, https://www.vam.ac.uk/blog/projects/dangerously-immersive-dickens. 3 http://www.liceodesanctis.edu.it/attachments/article/1984/1.pdf. an Augmented Reality Learning Object (ARLO), researchers together with educators need to take into consideration how the content will be presented to students based on what technology is available at hand or what can be procured” [26]. 56 We have tried to bring students closer to the paper book and expand it through AR. Through co- working, students approached a contemporary reality in the cultural debate, becoming prosumers themselves and changing their minds on their previous beliefs. Students sampled the pages of some volumes, then scanned and digitized them. At the end of this step, the digital expansion activities began. These, through platforms such as BLIPPAR4 and Genial.lly5, have expanded the paper-book with digital content, allowing a different perception and immersion of cultural contents. The critical points of the methodology applied are summarized below: • Identification of a target study: Twenty-five, high school students were divided into work teams for a total of five groups of 5 students each. • Training: Two training sessions were organized to introduce the topic of expanded books to the students. • Practical tasks: 1. In the school library, each workgroup has chosen one or more volumes relating to topics chosen from the available materials, for a total of 10 volumes. 2. The selected volumes were analyzed, and the contents of potential interest for the connection of the extensions were identified. 3. The research was carried out both online and in other paper volumes of the library to verify the availability of the contents. 4. Pages and contents to be used have been scanned. The scanned contents and the expansions have been located in a dedicated online repository. 5. In correspondence with the digital reproduction of the topical pages identified by the students, the marking software for recognizing the contents of interest was applied. 6. Subsequently, the contents were expanded. 7. Finally, a prototype web link was created with everything necessary to make the expansions usable through mobile technology. • Test: Students have verified that by framing the chosen book on the marked page, the display showed the reproduction of the page with the access keys to the highlighted expansions and the types of related contents (images, audio, video, audio-video). In this way, the students approached little-known resources, including the “Libro Rosso della città di Trani” (collection and transcription of ancient diplomatic documents), appreciating its contents, and re-evaluating the book as an entity is no longer static. 7 2.1.2. EXEBook In parallel, research was carried out to develop an innovative editorial process to produce printed books integrated with “digital expansions” usable through mobile devices. D.A.BI.MUS. S.r.l6, (a spin- off of the University of Bari, specialized in the design and development of digitization services and 4 https://www.blippar.com. 5 https://genial.ly. 6 https://www.dabimus.com. digital products for the cultural heritage) and the publishing house Quorum Italia S.r.l. 7 (a company focused on the fruition of ancient and historical books) has practically realized this aim. 8 The lack of knowledge about the Apulian authors’ book production of the past centuries has generated interest in implementing the research. The aim is to show their importance as protagonists in the cultural and scientific debates of the different eras. Hence, the suggestion to experiment with the anastatic reprint of an ancient text linked to Apulia curated in the typographic design. This supposition has based a first research question: how to make analogues qualitatively attractive and, equally, made interactive using the innovative digital solutions on the model of ARBooks, expanding the textual content, to intercept the interest of a young audience. The experimental methodology was divided into three main activities. 1. The first focused on the choice of analogue materials: a collection of printed poems from the first half of the nineteenth century was chosen. They represent the oldest known testimony of the original Barese vernacular idiom. Next, the volume was scanned in high- definition BITMAP format for printing. Finally, the images were post-processed by working on the contrast between text background and the definition of the graphic stroke to favour the best application of the markers. 2. The second activity has included the typographic design (Figure 1) and printing the paper prototype to apply the AR markers. The aesthetic quality of the book plays a fundamental role in the impact with the user, as a factor that helps arouse interest and stimulate physical interaction with the object. In this regard, it was considered crucial to associate with the classical manipulation of the physical object modes of use of the mobile device based on Tangible user Interface (TUI), using the finger as a pointing element to interact with the contents augmented. Figure 1: First design of the book cover 3. The third activity was focused on the development of the application in AR. Several web apps were analyzed, and for the development of the first version of the pilot, the open-source Blippar2 and Unity, provided with the necessary tools to develop the application model and usable with any system for mobile devices (iOS and Android). However, both platforms: practical, functional, and sufficient for the students, were unstable and unsuitable for the needs of this creation. For this reason, a platform for uploading content has been built in house. 7 https://quorumedizioni.it. The configuration of the AR application was based on four main components: • the mobile device for using the content in AR; • the mobile device camera for framing and scanning the object in real-time; • the analogue book; • Markers for tangible interaction and tracking targets. The template’s design integrates the concepts of “History page” and “marker page”. The history page covers the content of the pages of the volume. The marker page consists of the markers that animate the AR content linked to each page. The prototype to view and interact with contents uses the display of mobile devices that, pointed directly at the page in reading, scans it and highlights the increased interface, allowing the user to explore the contents with dedicated tools and to enjoy them while keeping intact the standard reading of the book (Figure 2 and 3). Figure 2: First experimentations Figure 3: Scanned image Digital content was then created to expand the printed text: digital text files, audio/video files with interactive reading, image files, avatars for interactive reading by users. DA BI.MUS. S.r.l is implementing this proposal launching: EXEBooK, precisely. A mobile app (for both Android and iOS operating systems) capable of taking a photo on any page (in the main screen) of a “unique identifier” code associated with a book (among those that provide interactive content) and to automatically “land” on the “target” resource of the platform. The application differs because it can mark content of interest, generating a link, which allows you to access additional digital content, both variable and augmented. Furthermore, the reader, interacting in real-time with the marker page through the dedicated tools and having access to augmented content, can also live the emotional experience of becoming an avatar protagonist of the text reading. In particular, when starting the mobile app, the user will enter a code that he will find associated with the paper book. Subsequently, the mobile app will prepare itself in “camera preview” to allow the user to frame and photograph any page of the analogue book in his possession (obviously one enabled to use the mobile app). Upon successful acquisition, the image will be sent to the appropriate REST service, displayed by the server. Next, the relative alphanumeric “template” will be extracted from the picture, which, given as input to the indexer, will allow you to determine which page (and which book) it is, to retrieve the “target” URL address to be returned to the mobile app (as the output of the called REST service). The mobile app will view the digital/interactive resource, from the platform, directly in the app itself after the user has received the “target” URL address. The user always could repeat the action on other pages of the book or other books (by entering, in this case, a new “unique identifier” code). Also, in this case, users actively participate in the perception of the cultural asset, being free to enjoy the contents of their interest. The user is not trapped in a guided choice of the expanded contents to be enjoyed. Furthermore, this approach allows continuous and “n” expansions. 3. Conclusion Today, cultural content cannot have the same interaction as it did in the past, especially considering user demand and how people want to approach the heritage. For this reason, those involved in producing cultural content could entice the user with user-centric experiences to have positive feedback. Furthermore, the user could actively participate in the cultural production, as this is the last “evaluator”, as demonstrated before with the case studies reported. Hence, cultural institutions seem to understand how people are precious to the life-cycle of experience, primarily digital or phygital. Therefore, as previously reported, choosing content that is little known and perhaps not customarily considered can become a challenge for the supplying institutions and who is called to expand elements digitally. Peculiar goods could be more attractive than others typically considered the praxis, becoming generators of unique and cutting-edge experiences. 4. References [1] E. Bonacini, “Il museo partecipativo sul web: forme di partecipazione dell’utente alla produzione culturale e alla creazione di valore culturale”, Il capitale culturale, Studies on the Value of Cultural Heritage, 2012, 5: 93-125. [2] I-scoop: Digitization, digital and digital transformation: the differences; https://www.i- scoop.eu/digital-transformation/digitization-digitalization-digital-transformation-disruption/. [3] European Commission website: Digital cultural heritage; https://ec.europa.eu/digital-single- market/en/digital-cultural-heritage. [4] ICCROM: Heritage and Pandemics: Accessing heritage during a pandemic; https://www.iccrom.org/events/heritage-and-pandemics-accessing-heritage-during-pandemic [5] P. Callet, “3D Reconstruction from 3D Cultural Heritage models”, in: Münster, Sander / Pfarr- Harfst, Mieke / Ioannides, Marinos (eds.): 3D Research Challenges in Cultural Heritage. Lecture Notes in Computer Science, vol 8355. Springer, Berlin, Heidelberg, 2014: 132-145. DOI:10.1007/978-3-662-44630-0_10136-145. [6] Victoria and Albert Museum website: Welcome to the V&A – the world’s leading museum of art and performance; https://www.vam.ac.uk. [7] Victoria and Albert Museum blog: Museum interactives; https://www.vam.ac.uk/blog/digital/museum-interactives. [8] Artribune: Musei e digitale. L’esempio del Victoria & Albert Museum di Londra; https://www.artribune.com/professioni-e-professionisti/who-is-who/2020/05/musei-digitale- victoria-albert-museum-londra/. [9] Science Museum Group Digital Lab: Is this the end of touchscreens in museums? The use of touchless gesture-based controls; https://lab.sciencemuseum.org.uk/is-this-the-end-of- touchscreens-in-museums-the-use-of-touchless-gesture-based-controls-ee3f3c3f37ce. [10] A. S. Vassillos, K Charikleia, “Re-imagining the museum through “touch”: Reflections of individuals with visual disability on their experience of museum-visiting in Greece”, Alter, 2015, 9, 2: 130-143. [11] Victoria and Albert Museum blog, How can technology improve the museum experience?; https://www.vam.ac.uk/blog/digital/how-can-technology-improve-the-museum-experience. [12] I. Ten, G. Arbelaez-Garces, L. Dupont, “Designing interactive museum visitors’ experience using a Fab Living Lab platform: the Museum-Aquarium of Nancy case”, in: 2020 IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC), Jun 2020, Cardiff, United Kingdom, 1-8, 10.1109/ICE/ITMC49519.2020.9198641. hal-02941618. [13] Agenda Digitale, La digital transformation immersiva dei Beni Culturali: l’esempio di M9; https://www.agendadigitale.eu/cultura-digitale/la-digital-transformation-immersiva-dei-beni- culturali-lesempio-di-m9/ [14] R. Hartson, P. S. Pyla, The UX book. Process and guidelines for ensuring a quality user experience. Morgan Kaufmann, 2016, p. 222. [15] C. Faggiolani, A. Federici, L’affordance della biblioteca: Modalità d’uso emergenti dalle indagini Istat, Biblioteche, 2018, 3:36, pp. 5-16. [16] J. Barshay, “Evidence increases for reading on paper instead of screens”, https://hechingerreport.org/evidence-increases-for-reading-on-paper-instead-of-screens/. [17] J. McKenize, D. Darnell, The eyemagic book: A report into augmented reality storytelling on the context of a children’s workshop. NZ: Centre for Children’s Literature, Christchurch College of EducationAuthor, 2016. 2(5), pp. 99–110. [18] G. Martgetis, A. Ntelidakis, X. Zabulis, S. Ntoa, P. Koutlemanis, C. Stephanidis, Augmented physical books towards education enhancement. User-Centerd Computer Vision (UCCV), 2013 1st IEEE Workshop on, 2013, pp.43-49. [19] H. Altinpulluk, M. Kesim, The classification of Augmented Reality books: a literature review, in Proceedings of INTED2016 Conference 7th-9th March 2016, Valencia, Spain, 2016. [20] H. Kaufmann, D. Schmalstieg, Mathematics and Geometry in Education with Collaborative Augmented Reality, ACM SIGGRAPH 2002 Conference, New York, 2002, pp. 37-41. [21] E. Medina, Y. C. Chen, S. Weghorst, Understanding biochemistry with Augmented Reality, in C. Montogomerie & J. Seale (Eds.), Proceedings of Word Conference on Educational Multimendia, Hypermedia and Telecommunications, 2007, pp.4235-4239. [22] M. Billinghurst, H. Kato, I. Poupyrev, The magic book: a transitional AR interface. Computers & graphics, 2001, 25(5), pp. 745-753. [23] A. B.Tomia, R. Dayang, A. Rambli, An Interactive Mobile Augmented Reality Magical Playbook: Learning Number with the Thirsty Crow, International Conference on Virtual and Augmented Reality in Education, Procedia Computer Science 25, 2013, pp. 123-130. [24] V&A Research Projects, Immersive Dickens, https://www.vam.ac.uk/research/projects/immersive-dickens. [25] K. Price, Dangerously immersive’ Dickens, https://www.vam.ac.uk/blog/projects/dangerously- immersive-dickens. [26] M. Kljun, G.Vladimir, K. Č. Pucihar, Augmented Reality in Education: Current Status and Advancement of the Field, in Augmented Reality in Education, 2020.