In today's digital age, the ability to access, analyze and (re)use ever-growing amounts of data is a strategic asset for the broadcasting and media industry. Despite the growing interest around new technologies, archive's search and retrieval operations are still usually done by means of text-based search over tags and metadata of manually pre-annotated material. This is particularly true because of its reliability and the broad availability of powerful full-text search platforms. However, this approach still does not completely meet the requirements that a search over huge multimedia archives poses, such as the need for semantic-driven indexing and retrieval, or the possibility to access contents based on visual features. In this paper, we describe a framework currently under development in Rai that enables visual search over the company's archive, which includes still images as well as annotated broadcast contents and raw footages, totaling over 1.5 million hours of video material. The current architecture's core is based on LIRe (Lucene Image Retrieval), an open source Java Library for content-based image retrieval, and Apache Solr, an enterprise full-text search platform. Possible extensions of the framework to include new technologies such as deep learning or semantic learning are also discussed.
For modern broadcast and media companies, the proper organization and management of contents, including archives of footage and production material, constitutes a strategic asset. Furthermore, e cient search and retrieval methodologies are equally important to quickly and e ectively access those contents.
Multimedia asset management (MAM) systems attempt to address this problem by providing solutions to easily store and retrieve media les. Pioneer systems used by the industry employed text-based queries to search over textual information and metadata, typically associated to each stored le using either semi-automatic or handmade annotations. While this procedure is still in practice these days, due to its overall reliability and robustness, it presents some critical weaknesses.
First, metadata extraction is an expensive and time consuming process, which requires human supervision and needs to be done both for audiovisual content that is produced digitally in the rst place, as well as for vintage footage that is converted from analog to digital formats. Second, search and retrieval based on handmade metadata annotation usually does not involve semantics or analytical representations of the media contents, thus does not allow visual query tasks such as query-by-example (e.g. image queries) or near duplicate detection. A clever use of metadata helps to mitigate these issues, but does not solve the problem.
To address these and other shortcomings, Content-Based Image Retrieval
(CBIR) systems have been developed. These systems tackle some of the issues
related to the use of textual metadata by representing multimedia items in terms
of features automatically extractable from the contents themselves, rather than
in terms of metadata (manually) associated to the les. Nowadays, there is a
considerable and always growing number of CBIR systems available on the market,
with di erent features and licensing options tailored to address speci c needs in
image search. For a comprehensive review of state of the art in CBIR system,
interested readers may refer to [
Despite the considerable e ort, almost all the available CBIR systems still
su er from the semantic gap issue, being based on low-level features rather than
on high level concepts. To overtake this issue, e cient algorithms for object
recognition, such as SIFT and SURF, have been proposed in [
The attention is also moving from still images to the video domain. The
LIvRE project [
In the broadcast domain in which we operate, the target tasks are mainly
focused on image-to-video or video-to-video search. Since, as stated above, there
is plenty of options to choose from for image search but there are fewer
ready-tomarket solutions for video-to-video search, we started developing a new
framework based on ready-to-use solutions, compatible with our enterprise
infrastructure. This choice was motivated by the need of integrating such novel search and
retrieval framework in the existing archival and production work ows while
ensuring compatibility with the software used within our company. Since Apache
Solr is widely adopted in Rai, among all the options we decided to use LIRe
(Lucene Image Retrieval) [
The remain of the paper is organized as follows. Section 2 reveals more details about the use-case at the core of this paper. Section 3 describes the work ow at the heart of our framework. Section 4 provides some preliminary considerations about performance measurement. Section 5 concludes the paper with a brief sum up and future directions. 2
Being Rai a broadcasting company, there are di erent scenarios within the company's departments that could greatly bene t from a proper visual information retrieval engine. To provide some examples, our real-world use-cases include: { In the news department, being able to link an edited news/reportage to its raw footage and, viceversa, being able to retrieve all the news/reportages that used a speci c footage (video-to-video search); { In the archives department, aiding the employees during semi-automatic annotation tasks (video-to-video search) by correlating non-annotated material with similar pre-annotated contents; { In the archives department, being able to retrieve a speci c video or image in the multimedia catalog from a clip, a single frame or a similar image (image/video to image/video search); { For online content, allowing the user to nd a speci c show from an image/clip (images/video to video search).
It can be noted that almost all of the use-cases mentioned above fall within two main categories: image-to-video search and video-to-video search. In this paper we will examine mainly the former category (image-to-video search) as it will serve as a starting point for the more complex video-to-video search.
Since our goal was to implement a CBIR framework, we decided to start the
development process by adopting one of the already available image retrieval
solutions and build our framework on top of that. From our preliminary research
regarding the state-of-the-art, though, we spotted some possible obstacles that
separate us from our goal. In fact, cutting-edge solutions usually o er solid
absolute performance [
LIRe was our rst choice because, as stated above, is a valid CBIR platform that can be integrated with Apache Solr, an enterprise search server widely used in Rai, by means of a ready-to-use plugin (currently used in this project) that ensures compatibility to the Solr indexing format. The adoption of Solr allows distributed search and index replication and scalability, making it a much better and e cient enterprise solution. 3
In this section we will explain in details the work ow at the core of the whole project. 3.1
One of the main advantages of the proposed framework is its modularity. The whole architecture was planned and designed to make the fundamental logic blocks of the work ow as independent as possible. This will enable us to easily develop code in parallel and swap the blocks in case we nd out more e cient solutions in the future, other than making the whole framework easier to debug and maintain.
The main modules composing the framework (and their current implementations) are:
2. Scene detector/key-frames extractor (FFMpeg) 3. Feature extractor (LIRESOLR Plugin) 4. Indexer (LIRESOLR Plugin) 5. Retriever (LIRESOLR Plugin)
In Fig. 1 a diagram of the architecture is represented. It is worth noting
that the scene detection and key-frames extraction blocks are completely
separate from feature extraction and indexing blocks. This will allow us to replace
the basic scene detection we used with more sophisticated algorithms (such as
those described in [
The starting point of the whole process is the creation/addition of a JSON token le within the watch folder, which triggers a Listener application that, in turn, acts like a supervisor of the whole chain. 3.2
The rst step in our work ow consists in indexing the reference videos for our database. Those videos are the references that will be matched during the retrieval phase.
Rai owns a great amount of documents which di er both in format (image/video, analog/digital) and geographic storage location. To make our framework e ective, we planned to provide various entry-points for video indexing and we opted to o er a two-way approach to input les into the chain: TV-Radio digitisation Flow Tape/others digitisation Flow JSON token (trigger)
Listener application Keyframe extractor
Feature extractor
Indexer Watch Folder
LireSolr
Index
(Solr)
{ Shared folder: used to integrate easily our work ow inside pre-existing
company's work ows such as the digitisation process of the DIGIMASTER
[
The Listener process is developed to run in background and watch a shared
folder that acts as a container for the les to be indexed. When new les are
added to that folder, the Listener is triggered and its execution follows the steps
below:
1. Wait for a JSON token le creation/addition in the shared folder.
2. Create an output folder associated with each input video to be processed.
3. Perform scene detection with FFMPEG and save the selected frames with
their time-stamp.
4. Extract CEDD [
to. This le contains an array of parameters needed by the indexing process to run properly and each element in the array is composed, in turn, by two main elements VideoInfo and MetaData.
To run the process in batch on multiple video les, these elements must be speci ed for each video and, for each video in the JSON le, a corresponding output folder is created at runtime using a structure based on the current datetime.
This con guration allows to easily manage video les with multiple formats and resolutions and to control the status of the execution via the JSON tokens. Moreover, this architecture allows to have multiple clients populating the folder simultaneously with heterogeneous sources.
Scene detection and subsampling: Step 3 of the work ow consists in the generation of the images whose features will form the retrieval index. Since the whole work ow is mainly targeted at image search on video les, a proper scene detection methodology has to be used to extract signi cant images from video les. In the current implementation scene detection and key-frame extraction are both performed using FFMpeg lters. This allows to execute these tasks with a good precision in acceptable processing times. To be more speci c, the command chain currently used by FFMpeg is: 1. Selection of Intra-Frames with select='eq(pict type,I)' : this option makes the extraction phase much faster without penalizing the performance. 2. Selection of scene-change frames with that select='gt(scene,d)' : selects the frames whose new scene probability value is greater than the threshold The scene probability value used by FFMpeg is evaluated using a LGPL algorithm within libav lter library.
Feature extraction: After the key-frame extraction, the obtained images have
to be indexed to allow visual search. During the indexing process, global
features corresponding to the "Color and Edge Directivity Descriptor"(CEDD) are
extracted using the corresponding LireFeature class. This descriptor, despite
being slightly obsolete and not state-of-the-art, was selected because it
incorporates color and texture information in a histogram and performs well for many
use cases, according to [
The index creation is performed by LIRESOLR plugin to ensure compatibility with Solr indexing format. Solr cores: As just mentioned, our work ow is currently based on Apache Solr search platform for the indexing/retrieval module. In the current implementation we instantiated two separate Solr cores: { ImageCore: it stores the index with the global features of the input frames (extracted using LIRESOLR plugin). { MetaCore: it stores the metadata informations related to the videos indexed in the ImageCore (wherever available).
It is worth noting that this second core extends the capabilities of the ImageCore by giving the possibility to keep track of the source and original metadata of the input les besides allowing to update the metadata in a successive moment (after a manual annotation, for example). Moreover, theoretically, it enables the user to retrieve indexed videos using a more traditional text search approach. 3.3
During the retrieval process, the same features selected in the indexing phase
are extracted from the query image. Results are then collected from the index
after evaluating the distance between each entry and the query image using the
distance metric speci ed for the selected feature (Tanimoto coe cient [
At query time, LIRe allows to set two parameters to tweak the speed and accuracy of the retrieval process: { Accuracy is a parameter used to choose a trade-o between runtime complexity and precision of results. An accuracy parameter below 1 means that the results are approximate but the search is performed faster. { Number of candidates is another settings parameter aimed at reducing runtime complexity. Lower values means faster searches but less accurate results.
The results obtained after this operation are then sorted by relevance using the same distance measurement as score and presented to the user with a GUI.
The retrieval process described so far is very simple and a lot of e ort has been
made in the past to improve the indexing structures and retrieval performances
(as it can be read in [
In the preliminary stage of our work, image-to-video search was considered as the starting point. Regarding the datasets involved, Rai archives store a humongous amount of documents in di erent locations, with 1.540.032 hours1 of video material only. Ideally, performing image search on all these les would be a massive long-term achievement. In this initial development phase, though, it would be impractical, to say the least, to process and perform image search on all those les. To tackle this problem we selected two speci c datasets, that cover pretty well most of the use-cases mentioned above, in order to evaluate and test our platform. These datasets are: { TG Leonardo (set of 2200 episodes, approx. 360 hours of material): a thematic, scienti c focused, newscast, suitable for news/reportage and raw footage retrieval but also to nd similar videos for the recommendation system. { Medita (set of 2000 episodes, approx. 2000 hours of material): an educational show aired both on TV (Rai Edu 1) and online. It represents the greatest online educational media library and each episode is aimed to be a supporting material for teachers and students. This dataset is well suited to test pure image search and tagging-aid capabilities of our framework.
The proposed work ow has been tested using images extracted from the same videos used for indexing with two di erent key-frame extraction techniques. This type of visual search was chosen because is close to the use case of video search on video database, the next step in our roadmap. The dataset involved was the TG Leonardo, while the key-frame extraction techniques were: { FFMpeg shot detection { Rai's Shot nder
The latter is a proprietary software, developed by Rai within a bigger
framework aimed to aid news annotation and called Automatic Newscast Transcription
System (ANTS) [
The reason why we decided to test the framework with two di erent keyframe extractors is because we wanted to test the robustness of the global descriptor of choice when the query image is slightly di erent to one or a small subset of indexed images.
Let's see now an example of a test performed using the di erent key-frame extractors mentioned above. In Fig. 2 is shown a comparison between the best match retrieved (on the right) and the query image (on the left). Is quite obvious that the reference video is the same but the images are slightly di erent because they were extracted using di erent algorithms. In this particular example, the descriptor seems to be robust enough, anyway, the retrieval performance is not always as good as in this case.
In the following example (Fig. 3) we can see that the best match is not always found among the very rst results: this could be related to the fact that CEDD is a very compact descriptor (good for fast retrieval times) and, hence, images with similar colours and textures may have very similar descriptors. Changing the accuracy does not guarantee a substantial improvement of the results but increases retrieval time.
Regarding the quantitative evaluation of the framework, due to the nature of the datasets (high frame-to-frame di erence, motion blur in shots, small number of shots representing a scene, etc) it was very di cult to give an evaluation in terms of precision and recall for query images di erent than the indexed images. In facts, when the query shot is the same as an already indexed shot, the pat(1)'1 and the correct shot is retrieved in the rst position every time. Otherwise, the descriptor of choice does not prove to be robust enough and the rst result has a distance value signi cantly higher than the matching case previously described.
This result, though, is satisfactory enough for our rst use case (raw footage/ nal edit match) because, if the same piece of footage is present in two videos and the key-frames are extracted in the same way, there's a high chance that the query key-frame and indexed key-frame will be the same. Further considerations regarding video quality di erences between nal edits and raw footages will be investigated in the future.
Our benchmarks also targeted retrieval times as we wanted to give an insight about the speed of the framework. The setup that we used was the following: { Local web-server for request handling (based on java Spring framework). { Single Solr index used for queries. { Solr core and web-server both hosted on the same virtual machine with 4 cores and 8GB of RAM dedicated.
We tested query times using three di erent accuracy/candidates con gurations, as it can be seen from the Tab.1.
A=0.33, C=10000 A=0.5, C=50000 A=0.8, C=80000
Query time evaluation (ms) rawDocsSearchTime reRankSearchTime 91.2 90.5 264 224.3 355.4 375.8
The accuracy parameter (A) in uences just the raw documents search time as well as the retrieval precision, while the number of candidates (C) a ects the time needed to re-rank the results in a similar fashion. 5
In the previous chapters we discussed brie y about CBIR system and where our work is trying to t in today's scenario. We also described the current development stage of our framework and we presented some very early results to back up our approach.
In the current state our framework seems to con rm expectations that we are not able to nd instances of same objects within di erent videos and under di erent conditions (e.g. di erent video quality, framing, etc..). One reason for this may be the choice of the CEDD descriptor, and, in general, global descriptors. On the other hand, those compact global descriptors may give good results for speci c tasks like searching the exact same videos segments inside di erent dataset, useful in our case to match raw and edited footage.
The quantitative tests we presented are not mature yet, one reason for that
is the lack of copyright-free datasets and evaluation framework that targets our
speci c use-case to use as reference. In fact, almost all open datasets available up
to date are either very generic (ImageNet [
For the future developments of the retrieval core we plan to evaluate the performance of more sophisticated feature extraction algorithms, including local features, bags of visual words and deep-learning generated feature vectors. Most likely, this could also lead to the adoption of di erent retrieval solutions than LIRe. Regarding the deep-learning, we also wish to integrate this technology within the framework, for example with DCNN features or enriching the MetaCore with automatically-extracted scene informations (e.g. object/face recognition, image captioning, . . . ).
Regarding the other functional blocks of the framework, our goal is to investigate further on key-frame extraction and shot detection algorithms in order to reduce the number of extracted key-frames and, possibly, weighting them according to their relevance within the related sequence. By doing this we hope to improve retrieval performances, decrease index size and, therefore, reduce disk occupation and speed-up search times.