This paper aims at presenting the results of LinkedTV's rst participation to the Search and Hyperlinking task at MediaEval challenge 2013. We used textual information, transcripts, subtitles and metadata, and we tested their combination with automatically detected visual concepts. Hence, we submitted various runs to compare diverse approaches and see the improvement when adding visual information.
This paper describes the framework used by the LinkedTV
team to tackle the problem of Search and Hyperlinking
inside a video collection [
Concept detection was performed on the key-frames of
the video, following the approach in [
We indexed all available data in a Lucene index at
different granularities: video level, scene level, shot level and
segments created using sliding window algorithm [
Text search is straightforward with Lucene, by using the
default text search based on TF-IDF values. In order to
incorporate visual information to the search, we mapped
keywords extracted from the visual cues query (using Alchemy
API2) to visual concepts using a semantic word distance
based on Wordnet synsets [
We concatenated textual and visual queries to perform the text query. Two strategies were adopted: we either used segments indexed in the Lucene engine, or performed queries creating segments on the y, by merging video segments based on their score.
Performing a text query on the video index often returned the relevant video in the top of the list. Hence, some runs rst restrict the pool of videos that are going to be searched to a small number, and then perform additional queries for smaller segments inside this pool.
We submitted 9 runs in total:
scenes-C : Scene search using textual and visual cues.
scenes-noC : Same as previous using textual cues only (no
visual cues) for comparison purposes.
part-sc-C : Partial scenes search from shot boundary using
textual and visual cues following three steps: ltering of
the list of videos; querying for shots inside each video;
ordering them by score. As a shot is a unit that is too small
to be returned to a viewer, we completed the segment with
the end of the scene that includes this shot.
part-sc-noC : Same as previous using textual cues only.
cl10-C : Temporal clustering of shots within a video using
text and visual cues in the following manner: ltering out
the set of videos to search; computing scores for every
shot in the video; clustering together shots closer than 10
seconds apart (scores were added to form the nal score).
cl10-noC :Same as previous using text search only.
scenes-S or scenes-U or scenes-I : Scene search using only
textual cue from transcript or subtitle, no metadata.
SW-60-I or SW-60-S : Search over segments created by
the sliding window algorithm for LIMSI/Vocapia [
SW-40-U : Same as above for LIUM transcript with sliding window size of 40.
A rst approach consisted in reusing the search component with the scene approach and the shot clustering approach. A query was crafted from the anchor: the text query was made by extracting keywords from the subtitle aligned at start time and end time of the anchor. Visual concepts scores were extracted from the keyframes of shots contained in the anchor. If the anchor was constituted by more than one shot, we took for each concept the highest score over all shots.
A second approach made use of MorelikeThis Solr
component (MLT) combined with Entityclassi er.eu annotation
[
The results of the search task are listed in Table 1. We rst notice than given the same conditions, subtitles perform signi cantly better than any of the transcripts, which is an expected outcome. It is also interesting to note that using the visual concepts in the query slightly increases the results for all measures (e.g., clustering10-C vs clustering10-noC).
Overall, the best approaches are those using scenes and sliding windows. Scene based approaches retrieve a higher number of correct relevant segments within a time window of 60 seconds (higher MRR), but they are not the most precise in terms of start and end time, compared to the sliding windows approach (as suggested by mGAP and MASP). 4.2
The results are listed in Table 2. For both LA and LC condictions, runs using scenes outperform other runs for all metrics. The MoreLikeThis/Entityclassi er.eu approach comes second. As expected, using the context increases the precision when hyperlinking video segments. It is also notable that the precision at rank n decreases when n increases.
This paper presented our framework and results at the MediaEval Search and Hyperlinking task. From our runs, it is clear that scene segmentation is the approach with the best performances. Therefore, this approach should be studied more in depth, a potential improvement being to re ne the segmentation using semantics or speakers information. Also, we see here that this task bene ts from the use of visual information present in the video. Hence, those two axes should be the next steps to study for a future challenge. 6.
This work was supported by the European Commission under contracts FP7-287911 LinkedTV and FP7-318101 MediaMixer.