In todays digital age, the ability to access, analyze and (re) use large amounts of data is a strategic asset of fundamental importance for the broadcast and media industry. The challenge lies in the ability to search, organize and access multimedia assets in a fast and semantically relevant way. Visual search is the new frontier to achieve these objectives, by allowing users to match image and video contents depicting the same objects, such as buildings, paintings and logos, based on visual similarities and without the need of querying for manually generated metadata. This paper presents the implementation of a Content Retrieval Architecture (CRA) for visual content analysis and search. Experimental results to demonstrate the feasibility of the proposed architecture on cultural TV programme archives are discussed as well.
In todays digital age, television content life cycle has a very long span: after being produced and broadcasted, a copy of the content, possibly enriched with metadata, is stored in the archive to be reused when needed or to be published online. Multimedia asset management (MAM) systems provide tools to store and retrieve media files. Pioneer systems used by the industry employed text-based queries to search over textual information and metadata, typically associated to each stored file using either semi-automatic or handmade annota tions. While this procedure is still in practise these days, due to its overall reliability and robustness, it presents some critical weaknesses. In fact, metadata extraction is an expensive and time consuming process, which requires human supervision and needs to be done both for audiovisual content that is already produced digitally, as well as for vintage footage that is converted from analog to digital formats. New technologies are needed to increase documentation efficiency, as well as access and (re)use of video archives.
Content-based retrieval (CBR) relies on the idea of indexing and ma tching
image and video contents based on visual characteristics, in addition to manually
generated metadata. Many methods have been developed to achieve this goal.
Despite the considerable effort, almost all the available CBR systems still suffer
from the semantic gap issue, being based on low-level features, e.g . color, shape
and motion, rather than on high level concepts. To overtake this issue, efficient
algorithms for object recognition, such as those for key-point fea ture detectors and
descriptors, have been proposed. For this purpose, the Scale-I nvariant Feature
Transform (SIFT) algorithm is considered a pioneer work [
Being a public broadcaster, RAI has the promotion of Italy’s historic al,
artistic and cultural heritage among its mission objectives. For this purpose, several
hours of programmes are produced and broadcasted daily, as well as archived
for preservation and future access. In order to maximize the reuse of those
assets, the ability to efficiently search, organize and access content in a fast and
semantic-driven way is an asset of fundamental importance. A nov el approach
for image retrieval and automatic annotation of cultural heritage images is
proposed in [
In this paper, an automatic Content Retrieval Architecture (CRA) for video analysis and search is presented. The architecture is designed to meet requirements given by handling large volume of video contents. The paper is organized
Fig. 1: Content Retrieval Architecture (CRA) overview. as follows. Section 2 describes the proposed architecture. Section 3 presents preliminary experimental results, and finally, Section 4 concludes the paper. 2
The proposed architecture extracts local and global features from video and performs retrieval operations based on content similarities. It has been designed to strongly reduce video redundancy and concentrate processing resources on informative contents. Its functionalities extend the MPEG CDVS technology to video content keeping the standard interoperability. The architecture is made of three main modules, as illustrated in Fig. 1: (i) The Summarizer segments the video in shots and extracts the most representative key-frames ; (ii) The Selector extracts CDVS descriptors from key-frames, gathers similar sho ts in clusters and performs key-frame ranking and cluster ranking by relevance; (iii) The database (DB) stores information and metadata about video structure, ranking lists and visual descriptors. The architecture can work in two modalities, namely extraction and retrieval. In extraction mode, a reference (input) video is segmented into shots and representative key-frames are selected from each sh ot. CDVS descriptors are computed from each key-frame and stored in the databa se for retrieval operations. In retrieval mode a query video is processed in the same way as a reference video. CDVS descriptors are then extracted and matched against those stored in the database. Matching videos are returned as lists of key-frames, shots and videos sorted according to the matching score. 2.1
very long GOP (Group Of Pictures) settings and motion vectors are extracted. Then unreliable motion vectors on the edges and within smooth regions are discarded and the remaining vectors are accumulated. Shot transitions are detected when abrupt changes in the motion vectors field happen and the percentage of intra coded motion vectors exceeds a defined threshold τintra. Each frame is classified as “zooming” for zoom and “dolly” or “panning” for pan, tilt, pe destal and truck accordingly to the global camera movement. Furthermore a two pass filter, based on a temporal window of size Tw = 5, evens spurious zooming or panning frames. Next a frame is selected as key-frame when the following co nditions are met: (i) It is the first of a shot; (ii) It is the last frame of a zoom; (iii) The distance crossed during a panning exceeds the size of the frame height. 2.2
In this section, the datasets for our experiments are first introduced and
subsequently more details about our experimental settings are provided. Three
datasets have been selected among those provided as part of the CDVA
evaluation framework [
(b) Selector processing blocks. (a) Shot clustering and cluster weight ranking creation. (b) Key-frame graph and key-frame ranking list. buildings.3 The David Daniels (DD) dataset is focused on London historical buildings recorded in non-professional way. The “RAI Monuments o f Italy” dataset (RAI-IT-MON) includes about 2,000 clips depicting about 20 0 monuments from all over Italy, mainly acquired from RAI regional newscasts.4 Furthermore, a new dataset, “RAI Viaggio nella Bellezza” (RAI-BELLEZ ZE), made of over 20 hours of video material showing monuments, paintings, sculptures and historic locations belonging to the artistic heritage of Italy, was acquired from RAI’s cultural TV programmes. A visual example of keyframes extr acted from the experimental datasets is shown in Fig. 4. Three types of experiments have been executed aimed to evaluate (i) shot boundaries detection effectiveness, (ii) content compression efficiency and (iii) content retrieval performance.
Shot boundary detection (SBD) has been tested using part of the RAI-ITMON dataset manually annotated with ground truth. Despite this is not the focus of the system, we performed this test to verify that content retrieval performance were not affected by possible wrong shot segmentations. Considering as true positives the shot boundaries correctly detected, false positives the 3 https://pacific.tilab.com/www/datasets/ (last accessed July 2017) 4 The R“AI Monuments of Italy” dataset can be made available on request Fig. 4: Example of keyframes from the experimental datasets. From top to bottom, left to right: Royal Palace (TI-CTurin180); Santa Giulia ch urch (TICTurin180); St. Paul (DD); London Bridge (DD); Florence, Palazzo Vecchio (RAI-IT-MON); Milan Cathedral (RAI-IT-MON); Agrigento, Valley of the Temples (RAI-BELLEZZE); Mantua, Palazzo Te (RAI-BELLEZZE).
Sub-sampler
CRA Summarizer
shot boundaries wrongly detected and false negatives the missing ground truth
boundaries, the shot boundaries detector achieved average Precision, Recall and
F-measure of 0.86, 0.89 and 0.88, respectively. This is comparable wit h state of
the art, where F-measure ranging from 0.84 and 0.9 is reported [
In the second experiment, the content compression provided by the Summarizer with the key-frame selection has been tested using single-s hot video, as reported in Table 1. The number of extracted key-frames is compa red with the results of a uniform subsampling algorithm (4 frames per shot). Then CDVS Test Model pairwise matching is used to compare a query image with the extracted key-frames and obtain an average score sc. The quality of the extracted key-frames is evaluated as q = #sck , where #k is the average number of extracted key-frames. The efficiency (i.e. compression ratio) of our key-fra me selection algorithm is comparable with uniform subsampling. However, the selected keyframes are more representative (i.e. higher quality in Table 1) of the video shots. Furthermore, the loss of CDVS matching accuracy is not significant since empirical studies demonstrated that CDVS matching scores higher than 3 result in near-100% matching precision.
The last experiment is aimed at evaluating the performance of the CRA architecture when used for search and retrieval. Two tests were conducted for videoto-video search and image-to-video search, respectively. Video- to-video search was performed according to the following steps: (i) The datasets have been randomly split in two parts, the former used as query dataset and the latter as reference dataset; (ii) All the videos have been processed in order to detect keyframes, shots and clusters as previously described; (iii) Reference videos have been stored in the database according to the Extraction mode of the architecture; (iv) Query videos have been matched to the reference database according to the Retrieval mode of the architecture. Results have been collected and analyzed in terms of Precision, Recall and F-measure. Precision and recall ar e measured considering retrieved videos as true positives when related to the query video and false positives when unrelated to the query. Furthermore expected videos that are not retrieved, are considered as a false negative. Results are reported in Table 2. The TI-CTurin180 dataset got optimal performance in te rms of both precision and recall. The peak of false positives in the RAI-IT-MON da taset is due to some elements appearing in most of the videos, such as logos, graphics or studio settings. This behavior might be mitigated applying some pre filtering heuristics to the input data (e.g. frame cropping). Recall significantly drops down for both DD and RAI-IT-MON datasets. However, this issue m ainly depends on the datasets themselves rather than on the CRA architecture. In fact, many buildings in these datasets are captured from different sides. The semantic gap between non superimposable view of the same object can not be overcome because of the lack of common elements. This behavior may be mitigated e.g. including more views of the same object in the reference dataset. Furthemore, precision is preserved by all the analyzed dataset. Image-to-vide o search was performed similarly as for video-to-video search, using the RAI-BE LLEZZE as reference dataset and Web images as query items. Images concerning some of the artistic subjects (i.e. monuments, paintings, statues, buildings and archeological sites) depicted in the reference videos have been automatically collected from Google Image search. Achieved precision is 1, meaning that no false positive results were returned by the system. Finally, we examined processing time spent at each pipeline stage using the TI-CTurin180 dataset and an Ubunt u 14.04 LTS virtual machine configured with dual Intel Xeon E5-2690@2.9GHz 1 6 cores (32 threads, maximum number of running threads limited to 12) and 32GB RAM DDR3L@1.6GHz (maximum memory usage limited to 8GB). The longest operation was performed by the Summarizer (∼ 15′). The extraction process took ∼ 2′. Even if both summarization and extraction are performed only one time for each video and are normally server side jobs, a faster summarization algorithm may be investigated in future developments. Retrieval took ∼ 2′′.
This paper presented an end to end video retrieval architecture based on global and local feature descriptors. Due to its flexibility, this architecture may be implemented in different demanding application, from cultural applications on smartphones to professional catalogs management on servers. Experimental results demonstrated the feasibility of the system, in particular when the objective is to achieve high precision, while lower recall is acceptable. Processing times demonstrated that the architecture implementation is compatible with an asymmetric client-server implementation, where the core jobs (su mmarization and extraction) are performed in the background on server side. Future work will include the analysis of the impact of different video summarization techniques.
Acknowledgments. This work was partly funded by the European Union’s Horizon 2020 programme (grant No 731667, MULTIDRONE).