In this paper, we describe our participation in the Search part of the Search and Hyperlinking Task in MediaEval Benchmark 2014. In our experiments, we compare two types of segmentation: fixed-length segmentation and segmentation employing Decision Trees on a set of various features. We also show usefulness of exploiting metadata and explore removal of overlapping retrieved segments.
The main aim of the Search sub-task is to find video
segments relevant to a given textual query. This problem is
an important part of the Spoken Content Retrieval [
All experiments presented in this paper were conducted
on the BBC Broadcast data. A total of 1335 hours of video
was available for training and 2686 hours for testing. We
exploited subtitles, automatic speech recognition (ASR)
transcripts by LIMSI [
Based on the results of our previous experiments [
Two strategies were used for segmentation of the
recordings: 1) we divided the video recordings into segments of
fixed length and 2) we used segmentation system which
employed Decision Trees (DT) [
Based on our previous experiments, we set the segment length in the fixed-length segmentation to 60 seconds and the shift between the overlapping segments to 10 seconds. The segment length applied in the segmentation system was tuned on the training data and set to 50 seconds and 120 seconds for the Search sub-task. We also experimented with post-filtering of the retrieved segments – we either used all the retrieved segments or we removed segments which partially overlapped with another higher ranked segment.
We also employed the metadata provided for the task. For each recording we extracted the title, episode title, description, short episode synopsis, service name and program variant and appended the text to each segment from that recording. 4.
The results for the Search sub-task are given in Table 1.
We present scores of six evaluation measures: Mean Average
Precision (MAP), Precision at 5 (P5), Precision at 10 (P10),
Precision at 20 (P20), Binned Relevance (MAP-bin), and
Tolerance to Irrelevance (MAP-tol) [
Unsurprisingly, the best results are achieved in experiments using subtitles. Generally, most of the results obtained with the LIMSI transcripts are higher than the corresponding results with the LIUM and NST-Sheffield transcripts. The only exception are the experiments employing overlapping segments. The results with the NST-Sheffield transcripts are higher than the corresponding results with the LIUM transcripts.
In most of the cases, the concatenation of the segment with metadata improved the results, despite the drop in the P5 score for all types of transcripts. Apart from several values of P and MAP-bin for the LIUM transcript, the fixedlength segmentation outperforms the Decision Trees-based segmentation with 120-seconds-long segments. Though the 50-seconds-long segments created using Decision Trees notably outperform the fixed-length segments measured by MAP and precision-based measures, they are outperformed by the fixed-length segmentation using the MAP-bin and MAP-tol measures.
All measures, except the MAP-tol measure, are notably higher in the experiments in which we did not remove partially overlapping segments from the list of the retrieved segments. Due to the nature of these measures, it is not possible to distinguish, whether a user had already seen the retrieved segment or not. Therefore, all the relevant segments, which frequently overlap each other, increase the score. The MAP-tol measure is not influenced by this behavior as it takes into account only the relevant content which had not been already seen by a user. Therefore, the highest MAP-tol scores are achieved for the fixed-length segmentation when the overlapping retrieved segments are removed.
In our experiments in the Search sub-task, we have experimented with subtitles and three ASR transcripts. The subtitles outperformed all used ASR transcripts. However, the LIMSI transcripts also generally scored well and they slightly outperformed the NST-Sheffield transcripts. The LIUM transcripts achieved the lowest scores in most of the cases. Moreover, we have confirmed usefulness of the metadata and effectiveness of simple segmentation into fixedlength segments.
We have also pointed out the problems with partially overlapping segments occurring in the results. Such segments can greatly increase MAP scores, however they could not be expected to be helpful for the users. Therefore, the MAP-tol measure could be preferred in such cases.
This research is supported by the Czech Science Foundation, grant number P103/12/G084, Charles University Grant Agency GA UK, grant number 920913, and by SVV project number 260 104.