Multimedia storytelling is an efective and engaging method to convey information in multiple domains. Specifically, book trailers -video advertisements for books- positively influence the desire to learn and the motivation to read. This work describes the design and implementation of Storybook, a tool for the semi-automatic creation of book trailers aiming to support storytelling for digital libraries. Storybook supports an expert by gathering relevant crowd-sourced multimedia content, which, arranged as stories, can be used to showcase a book in the form of video clips. Crucially, the expert controls how the content is finally combined and edited rather than ofering a fully automated process. In an early informal evaluation, experts consider the method favourably.
Storytelling is the act of narrating, using the principles of narratology in the audiovisual or literary field to communicate and facilitate learning and understanding. For example, in school books ( to make a concept simple), a story with characters is used; similarly, in language courses, even for adults, the contents are organized with characters who show an aspect of the language through dialogue or a text. This methodology is an experiential resource, as it promotes a generative development between experience, its observation and the resulting insights.
Numerous researches [
The design, implementation, and early evaluation of Storybook are described in this work.
Storybook is a software tool to support the creation of book trailers by collecting and organizing relevant video content. The system users retain control on how to edit and compose the ifnal content. The proposed process aims to semi-automatically build digital trailers that allow the viewer, generically interested in a specialized topic but not expert, to appreciate better the topic, both for their own cultural/professional enrichment and a possible purchase.
The remaining sections are organized as follows. Section 2 presents related work about data-driven storytelling systems. Section 3 describes the pipeline of the proposed system and the technologies used. Section 4 reports an early evaluation. Finally, Section 5 discusses future research directions.
This section briefly surveys relevant literature for relevant works, starting from data-driven
storytelling systems. Storytelling and narrative visualization techniques are two lines of research
that intertwine and have led to new methods for enhancing data understanding, information
dissemination and displaying information [
Users usually encounter dificulties creating a data-driven story due to technical barriers,
which motivate the design and development of various automated and simplified generation
tools. The creation of a data-based narrative can be divided into two branches [
The second branch of data narration research, which is one of our interests, seeks to break down technical barriers in creating data-based stories, focuses more on automated systems to save users’ eforts that take data as inputs and generate organizations of history components for users. For instance: • VoxPopuli [13] is developed to generate video documentaries based on interviews about controversial topics. Via a Web interface, the user selects one of the possible topics and a point of view, and the engine assembles video material from the repository to satisfy the user request. • Fotoinmotion [14] is designed to produce automatic videos based on a still image. The tool considers semantic information to produce storytelling videos, focusing on the relevant features of the input image. • NewsViews [15] is developed to create interactive, annotated maps from news articles.
The NewsViews’s maps support trend identification and data comparisons relevant to a given news article.
Summarizing, existing storytelling tools focus on how to tell a story but rarely base the story on original data. Moreover, these tools assume that the story content is created manually, resulting in ineficiency. In contrast, Storybook supports automatic story generation and flexible story editing features, which ensure quality, reduce the barrier and improve the eficiency of visual storytelling.
A pipeline to support making book trailers has been designed and developed. The process is schematized in the Algorithm 1. The following subsections introduce the deployed pipeline’s details and discussion and motivations of choices.
Input : Data about a book: content (PDF) and metadata ( and ).
Input : A set of , web services ofering video search capability.
Input : Set of keywords ℎ to add to the search.
Input : Set of keywords to exclude from the search.
Input : Parameters (number of videos), (number of named entities extracted), (number of video results), (number of labels from image classification).
Output : A set of videos relevant to the book content. 1 ← top named entities found in content (by named-entity recognition) 2 ← { } ∪ {} ∪ {} × ( ∪ ℎ) 3 ← {} 4 for ∈ do 5 for ∈ do 6 ← ifrst results from searching while excluding 7 ← ∪ 8 end for 9 end for 10 ← largest images of content 11 ← {} 12 for ∈ do 13 ← top labels assigned to image by image classification 14 rank by overlap of their tags with 15 ← video with maximum overlap 16 ← ∪ { } 17 end for 18 return
The process is designed to work on books with meaningful visual content in images extracted from the PDF. The idea is to find videos relevant to the domain and subject of the book that can be associated with a set of images found in the book. The underlying intuition elicited working with content experts is that the visual content included in the book, like images, is relevant to identify appropriate video content to create a book trailer.
The algorithm has four main parameters: , number of images extracted from the book, it is also the number of videos provided as output; , number of named entities extracted from the book; , number of video results for each keyword and each video search service; , number of labels gathered from the classification of each image. We assume that all the considered books have at least images.
The first important step in the pipeline is to retrieve/extract metadata related to the book itself (see input metadata and line 1 of Algorithm 1).
We already have implemented an automatic semantic extractor of information from a digital library called Arca [16, 17, 18] that produces a knowledge graph (KG) [19, 20] with all extracted information and integrated with KGs of the web, so we decided to use that one. However, it can be easily replaced or extended with other semantic enrichment tools as long as they process and extract relevant information from the book. In particular, we perform a SPARQL query1 to the KG and retrieve existing and extracted metadata for a specific book, including the top concepts found as named entities in the book.
It is essential to notice that the whole script can run in batch mode. A web-server has been built to manage the execution of the whole Storybook process, while also providing metadata related to the book before, during, and after the process.
We consider the book metadata gathered in the previous step during this step, and we run a web craw looking for relevant videos (lines 2–9 of Algorithm 1). After experimenting with diferent sources of video content, we decided to focus on a single source, YouTube 2. YouTube is currently the most extensive database of videos in the world.
In the interface for Storybook, the curator can also specify a keywords’ whitelist and blacklist. In this way, the curator can try to steer the research of the crawler in diferent directions. The crawler uses the whitelist to extend the top concepts retrieved in the previous steps, extending the diferent keywords and searches that the crawler will perform. Instead, the blacklist is used to filter out more videos by excluding certain keywords. For example, depending on the specific search, sometimes it happens to find videos that explain a particular concept with slides (not helpful in building a trailer). Using the blacklist, the curator can cancel out some of these videos.
It is crucial to note that some retrieved videos are too long to suit our purpose. Even if a long video contains a shot valuable for the trailer, searching that shot in hours of videos would be dificult for the curator. So, we decided to cap the length of the retrieved videos. By default, the limit is set to be 480 seconds, but the curator can easily change this parameter in the application’s settings.
We also encountered another problem in searching in such a vast and mixed database. A lot of the videos uploaded on Youtube have licenses that prohibit reuse; hence they cannot be used for our purpose of building a trailer. Therefore, we implemented another filter for retrieving only videos with the Youtube Creative Common license that allows for commercial use. Nevertheless, we can search and retrieve many videos that fit within the search parameters. The gathered video content and related metadata will be used in the last step to compose the draft of a trailer for the book.
Another fundamental step in the pipeline of Storybook is to extract images from the PDF files of the books (line 10 of Algorithm 1).
Dificulties arise from the fact that many book publishers compress the PDF, hence losing the information related to the images. Thus, it is necessary for the algorithm first to detect whether the PDF file was compressed or not. This detection can be done by looking into the metadata and searching for the compressor’s sign (often, it leaves a signature). As a result, we can separate the compressed PDF to pass through an OCR process that includes rebuilding the images as well; the one we found most eficient was the Adobe Acrobat one 3.
Subsequently, whether the PDF was not compressed or had been rebuilt from the OCR, the next step was to understand the color space in which the image is stored inside the PDF. In some rare cases, it is not possible to rebuild the color space, so it is impossible without analyzing the image to know if a 0 means a white or black pixel4. In these cases, we extract both versions of the image.
Finally, we select the best images of the book. After trying diferent methods, the easiest one that yielded good results was sorting the images by their dimensions. We observed that book publishers tend to let essential images take up more space on the page (thus being more significant in size once extracted).
An image classification process is run on each of the best images identified at the previous step (line 13 of Algorithm 1). In order to speed up the development, instead of implementing a custom image classification, we went for using an external service, Google Vision AI 5. For each image, we retrieve up to labels, along with their confidence level. In the next step, this information is used when generating the suggestions for the trailer.
An interesting piece of information is the SafeSearch detection: how likely the image contains adult, spoof, medical, violent or racy themes 6. We do not use this information directly; however, it is available for the curator inside a comprehensive JSON file generated at the end of the process.
The last step of the process consists in filtering videos (lines 14–16 of Algorithm 1) and organising them in a draft of the final trailer (line 18 of Algorithm 1).
The algorithm assigns each video a multidimensional score (one dimension for each image). The score increases when there is a match between the image’s labels and the video’s metadata, such as the description. The confidence of the image-keywords association gives a score’s weight (as assigned during the classification step). Once each video has a score, the algorithm matches the highest score per single image, associates that video with the specific image, and 3https://www.adobe.com/acrobat.html 4The opposite applies to 255. 5https://cloud.google.com/vision/ 6https://cloud.google.com/vision/docs/detecting-safe-search discards all the others. Afterwards, the trailer is generated by interleaving the extracted images, and their correspondent retrieved videos. This draft of the trailer is generated for compatibility reasons as an annotated PowerPoint presentation, thus allowing the curator to manipulate the content as they see fit before the actual video creation. Furthermore, all the data generated along the pipeline is packaged inside a JSON file that the curator can access to check the intermediates results and other details on the process.
The system was designed and implemented based on requirements iteratively refined with a team of curators of a publishing house. In respect to the algorithm parameters, satisfactory results were reached by choosing = = = 10 and = 12.
Furthermore, the curators gave some general feedback on the system paradigm. They identiifed several perceived strengths and potential of the system: • the transformation of the images of books into information nodes; • the control of the automatic creation process of a book trailer, through accessible configuration parameters; • the free access and management of the information output generated by Storybook. They also expressed some concerns for perceived weaknesses: the scarce availability of videos of niche topics on the web and the prolonged extraction process duration for searches involving longer crawler queries.
Storybook is a tool for the semi-automatic generation of book trailers. It has been designed with experts in the field, implemented, and tested on the book catalog of a publishing house. From a high-level perspective, the main novelty of the approach is that it supports the creation process while not replacing the human contribution and final choice. An early evaluation shows that expert users appreciate this aspect and react positively to how relevant videos are collected. A potential limiting factor has been found in the limited availability of online videos on niche topics, significantly restricting the search to videos with licenses that allow unrestricted reuse. For the future, we have planned a formal evaluation of the system with a larger sample of users and the integration of Storybook into the Arca system mentioned in section 3.1.
This work has been partly supported by projects ARCA (POR FESR Lazio 2014–2020 - Avviso pubblico “Creatività 2020”, domanda prot. n. A0128-2017-17189), STORYBOOK (POR FESR Lazio 2014-2020 - Avviso Pubblico “Progetti di Innovazione Digitale”, domanda prot. n. A03492020-34437), and PON R&I 2014–2020 – Attraction and International Mobility (AIM) – project n. 1852414 of the Italian Ministry of University and Research. [13] S. Bocconi, F. Nack, L. Hardman, Vox populi: a tool for automatically generating video documentaries, in: Proceedings of the sixteenth ACM conference on Hypertext and hypermedia, 2005, pp. 292–294. [14] I. Teixeira, P. Viana, M. Andrade, P. Carvalho, L. Vilaça, J. Pinto, T. Costa, S. Rapanakis, P. Jonker, Creating automatic storytelling videos from still images: a semantically-aware approach, 2021. [15] T. Gao, J. R. Hullman, E. Adar, B. Hecht, N. Diakopoulos, Newsviews: an automated pipeline for creating custom geovisualizations for news, in: Proceedings of the SIGCHI conference on human factors in computing systems, 2014, pp. 3005–3014. [16] E. Bernasconi, M. Ceriani, M. Mecella, T. Catarci, M. C. Capanna, C. D. Fazio, R. Marcucci, E. Pender, F. M. Petriccione, Arca. semantic exploration of a bookstore, Proceedings of the International Conference on Advanced Visual Interfaces (2020). URL: http://dx.doi.org/10. 1145/3399715.3399939. doi:10.1145/3399715.3399939. [17] M. Ceriani, E. Bernasconi, M. Mecella, A streamlined pipeline to enable the semantic exploration of a bookstore, in: Digital Libraries: The Era of Big Data and Data Science - 16th Italian Research Conference on Digital Libraries, IRCDL 2020, Bari, Italy, January 30-31, 2020, Proceedings, volume 1177 of Communications in Computer and Information Science, Springer, 2020, pp. 75–81. [18] E. Bernasconi, M. Ceriani, M. Mecella, T. Catarci, M. C. Capanna, C. Di Fazio, R. Marcucci, E. Pender, F. M. Petriccione, Arca. semantic exploration of a bookstore, in: Proceedings of the International Conference on Advanced Visual Interfaces, AVI ’20, Association for Computing Machinery, New York, NY, USA, 2020. URL: https://doi.org/10.1145/3399715. 3399939. doi:10.1145/3399715.3399939. [19] Q. Guo, F. Zhuang, C. Qin, H. Zhu, X. Xie, ..., A survey on knowledge graph-based recommender systems, . . . on Knowledge and . . . (2020). URL: https://ieeexplore.ieee.org/ abstract/document/9216015/. [20] X. Zou, A survey on application of knowledge graph, Journal of Physics: Conference Series (2020). URL: https://iopscience.iop.org/article/10.1088/1742-6596/1487/1/012016/ meta. doi:10.1088/1742-6596/1487/1/012016.