These working notes paper present the contribution of the UPC team to the Hyperlinking sub-task of the Search and Hyperlinking Task in MediaEval 2013. Our contribution explores the potential of a solution based only on visual cues. In particular, every automatically generated shot is represented by a keyframe. The linking between video segments is based on the visual similarity of the keyframes they contain. Visual similarity is assessed with the intersection of bag of features histograms generated with the SURF descriptor.
These working notes describe the algorithms tested by the
UPC team in the MediaEval 2013 Search and Hyperlinking
Task. The reader is referred to the task description [
The Hyperlinking sub-task aims at linking anchors related to a temporal segment of a video. In this sub-task, as well as the Search one, one of the main challenges is the uncertainty regarding what criteria are to be followed to generate these links. There is ambiguity about what the user expectations are regarding these links, as well as little information about what is considered relevant to the user in the video segment.
In addition, the search within the video collection can consider three di erent types of media information: visual, audio and textual. Previous editions of the Hyperlinking subtask have especially focused on the textual data, whether in the form of closed captions or audio transcripts. The visual modality, though, has received less attention despite its great potential. In our work, we study the results of the visual cues in trying to capture the user expectations about the links. In particular, we adopt the popular Bag of Features model to generate links to video segments which are visually similar to the provided anchor.
The generation of links between video segments can be
solved with techniques developed for image retrieval. This
eld was greatly in uenced by the Video Google [
Once each image is represented by the interest points it contains, Video Google proposes the construction of a visual vocabulary of features (words) by clustering a large amount of them, extracted from a training dataset. After this learning stage, every feature point of the test dataset can be quantised by assigning it to the most similar word within a vocabulary. As a result, every image in the test dataset can be described with a signature, where each bin corresponds to the word occurrences. The representation of each image as a Bag of Features (BoF) allows its e cient indexing with inverted indexes. In addition, the resulting signatures tend to be sparse, allowing e cient storage and similarity assessment.
Currently there are several implementations of the
algorithms available. We have adopted the solutions included in
OpenCV [
The Hyperlinking sub-task is addressed to temporal video
segments, but dealing with all the visual information
contained in the collection requires a very large amount of
computing power not available to the participants. Nevertheless,
the MediaEval organisers published the shot boundaries and
associated keyframes generated by [
The provided anchors were de ned by a start and end time codes. The temporal boundaries of an anchor typically covered several shots. As every shot was also represented with a keyframe, every anchor corresponded to a collection of keyframes. Each of these keyframes was used to formulate an independent image query, which was compared with all the keyframes of the video collection to generate a ranked list of retrieved keyframes. The similarity assessment was performed with the histogram intersection, an operation which can be e ciently assessed on sparse signatures.
The sub-task organisers required that, in addition to strictly adjusting to the anchor boundaries, one of the runs should consider the context of the general segment. Following the spirit of our exclusively visual solution, we simply expanded the time codes of each anchor to 5 more shots backward and forward. That is, in the context run, 10 more shots were considered as part of the query frames. 3.2
The retrieved keyframes associated with an anchor require a further processing to generate a ranked list of temporal video segments. Our approach is based on the assumption that the user appreciates diversity within the resulting links. In particular, the implemented solution limits to one the appearance of a video in the retrieved list of video segments.
The rst step to create video segments is the merging of all ranked lists of keyframes associated to an anchor into a uni ed one. If a keyframe appears in more than one ranked list, only the highest score is considered to represent this keyframe.
The algorithm scans in increasing order the ranked list of keyframes and generates a new and sorted entry in the output list whenever a new video is referred to. Taking as a base time code the one of the rst appearance of the video in the ranked list, the rest of the uni ed ranked list is explored searching for more keyframes from the considered video. Whenever a new keyframe is found, it is considered to set a new start or end time code to the entry under process. If expanding the time span does not exceed the limit of 2 minutes set by the organisers, the new time code is accepted. Otherwise, this keyframe is discarded and the rest of the list is explored. When the end of the keyframes ranked is reached, the end time code is expanded so that the nal duration of the linked video segments is also 2 minutes. This strategy was adopted because it was assumed that the user would especially appreciate a precise start time, but that it would not be very selective about the end time. On the other hand, expanding the segment may, even by chance, provide the user with some valuable information that they would appreciate.
It must be noticed that our results did not exclude the video source of the anchor from the retrieved video segments. We considered that the results may provide a di erent segment from the anchor one, although probably containing it.
The UPC participated in the Hyperlinking sub-task,
providing 1,000 video segments for each of the 98 anchors
provided. Finally, only 30 of them were assessed by users. The
Bag of Features model relied on a vocabulary of 10,000
codewords extracted with the SURF descriptor [
Results indicate that using an expanded temporal segment as a context is not advisable, probably because the newly considered frames are not relevant for the user. The decreasing precision values point that the upper part of the ranked list contains a higher proportion of relevant targets,
0.0282 0.2600 0.2000 0.1233
0.0260 0.2400 0.1967 0.1217 which decreases when considering a larger amount of hits. 5.
The presented technique has explored the potential of solving for the Hyperlinking sub-task based exclusively on visual similarity. Our proposal has considered videos as collections of shots, each of them represented by a single keyframe. A classic image retrieval solution has been adapted to work with temporal segments, based on the popular Bag of Features approach. In our work we have used a given shot segmentation to map video segments into keyframes, and viceversa.
We consider the obtained results as promising, however, we understand this rst participation of our team in the Hyperlinking sub-task as exploratory. The experience has shown has the challenges of dealing with large amounts of visual data, which requires high computational power and an indexing strategy. The results obtained might be improved by considering a larger amount of keyframes to build the vocabulary, as well as allowing multiple targets from the same video. 6. 7.
This work has been partially funded by the Spanish project TEC2010-18094 MuViPro.