The StoryBook project stems from the cooperation between Sapienza Università di Roma, the company Aton IT and the historic publishing house L'Erma di Bretschneider. The project deals with the broad area of Digital Humanities and has favored the development of a prototype system for semi-automatic video trailer generation of books.
StoryBook is a research project in the field of digital humanities, which proposes the knowledge
extraction and management of information of a digital library to crate semi-automatically video
trailers of books. StoryBook has been originally conceived to meet the needs of “L’Erma di
Bretschneider” publishing house that deals with topics related to ancient history and archaeology.
From publishers’ point of view, promotional trailers respond to a changing market with a high
focus on digital and visual media. The goal of a digital presentation of a book is nevertheless
broader than selling it and includes providing helpful information to the potential future reader.
A publishing house wants to disclose the contents of its digital library not only to experts in the
sector but also to interested people attracted by the contents of their books shown on the Web in
the form of searching tools or advertising such as video trailers. Numerous researches show that
a book trailer fosters the desire to learn and the level of motivation to read [
With the StoryBook project, we aimed at demonstrating that querying a knowledge graph about the content of a digital library can semi-automatically generate a book’s trailer based on this information, and can favour the difusion of the contents of a digital library and enhance the cultural heritage contained in those digital libraries.
StoryBook potential is based on technologies such as: • Linked Data to allow disambiguation of concepts and the connection of the information with the Web; • Named entity recognition (NER) to identify people, cities, organizations and things in unstructured text like that of books; • Knowledge graphs to organize information around concepts with their relations; • Computer Vision to detect objects in the images of books allowing their searchability.
In Figure 1, you can see the whole process that leads to the semi-automatic book trailer
creation. The process starts from point 1 with a digital library (a collection of books in PDF
format). Through the Arca platform [
With the StoryBook project, we started extracting images from books (point 2.b.1). The process is designed to work on books with meaningful visual content in images extracted from the PDF. We selected the “k” best images of the book. After trying diferent methods, the easiest one that yielded good results was sorting the images by their dimensions to detect the best images. We observed that book publishers tend to let essential images take up more space on the page (thus being more significant in size once extracted).
In point 2.b.2, there is object detection. In order to speed up the development, instead of implementing a custom image classification, we went for using an external service, Google Vision AI2. We retrieve up to “o” labels for each image, along with their confidence level. The concepts and top “l” concepts associated with a book and its metadata coming from point 2.a, and the “o” labels, point 2.b.2 associated with images of the book point 2.b.1 were sent as linked data to the linked data container (point 3).
1https://dbpedia.org/ 2https://cloud.google.com/ • all the concepts; • the most relevant concepts; • the metadata (like title, author, topics, etc.); • the objects contained in the book’s images.
We introduced some parameters to allow users who are curators of a publishing house to iflter the video search. The parameters are: • the video duration; • the type of video licence; • a white list containing words relevant for curators; • a black list containing the words that mustn’t be in the search; • the book’s metadata.
We use this information to Web crawl looking for relevant videos. After experimenting with diferent sources of video content, we decided to focus on a single source, YouTube 3. YouTube is currently the most extensive database of videos in the world. We can start the query to YouTube ifltered with all the elicited parameters.
The next step (point 5) of the process consists of filtering the “k” videos resulting from the previous step, and organising them in a draft of the final trailer. The algorithm assigns each video a multidimensional score (one dimension for each image). The score increases when there is a match between the image’s labels and the video’s metadata, such as the description. The confidence of the image-keywords association gives a score’s weight (as assigned during the classification step). Once each video has a score, the algorithm matches the highest score per single image, associates that video with the specific image, and discards all the others. Afterwards, the trailer is generated by interleaving the extracted images, and their correspondent retrieved videos.
This trailer draft is generated (point 6) for compatibility reasons as an annotated PowerPoint presentation, thus allowing the curator to manipulate the content as s/he think fitting before the actual video creation. Furthermore, all the data generated along the pipeline is packaged inside a JSON file that the curator can access to check the intermediates results and other details of the process.
We carried on a preliminary evaluation with a domain expert. Satisfactory results were obtained by choosing: • “l” number of top concepts = “k” number of books’ images = “o” number of objects detected in one image = 10 • “m” number of resulted video’s query = 12
Furthermore, the domain expert gave some general feedback on the system paradigm. She identified several perceived strengths and potentialities of the system: • the transformation of the images of books into information nodes; • the control of the automatic creation process of a book trailer, through accessible configuration parameters; • the free access and management of the information output generated by StoryBook.
• the scarce availability of videos of niche topics on the Web; • the prolonged extraction process duration for searches involving longer crawler queries.
With StoryBook, a tool for the semi-automatic generation of book trailers, we aimed at introducing an innovation in the scientific publishing market for the historical, historical-artistic and archaeological disciplines, a specific context characterized by users such as students, researchers, scholars, academics, cultural bodies and institutions, universities operating in the sector of Cultural Heritage and antiquity sciences. By revolutionizing the traditional approach to access and use of texts and images of a digital library, this is an operation that hopefully will have a strong impact in the reference context. For the future, we have planned a formal user study to further evaluate the system.
This work has been partly supported by the project STORYBOOK (POR FESR Lazio 2014-2020 Avviso Pubblico “Progetti di Innovazione Digitale”, domanda prot. n. A0349-2020-34437).