This paper describes the participation of the University of Twente team at the Rich Text Retrieval Task of the Media Eval Benchmark Initiative 2011. The goal of the task is to find entry points of relevant parts of videos to reduce the browsing effort of searchers. This is our first participation, therefore our main focus is to create a baseline system which can be improved in the future. We experiment with different evidence sources (ASR and meta data) together with a basic score combination function. We also experiment with different entry points relative to the segments found by the contained evidence.
When searching in videos, it is especially important to return an entry point where the relevant part for a searcher begins. The reason is that videos can be multiple hours long, and unlike our ability to quickly scan text for relevant parts, scanning a video is much more time consuming. In this paper, we describe the methods we used for the participation in the Rich Text Retrieval Task of the Media Eval Benchmark Initiative 2011 [2].
This paper is structured as follows: Section 2 describes the evidences we considered to calculate the likelihood for the relevance of a segment, the combination of those evidences, and the alternative entry points. Section 3 details the experiments we undertook to evaluate our approach. Section 4 concludes this paper.
In this section, we describe the methods we used to search for segments in videos which are likely to contain relevant information, and given one such segment, how to determine the entry point into the video presented to the user.
We used the following types of evidence to identify suitable entry points. First, we used the meta data which was provided on a video level. Second, we used transcripts from provided automatic speech recognition. The transcripts are divided into speech segments which the recognition system believed to originate from a speaker. Since the returned speech segments were relatively short, we also considered speaker turns, the transcripts of all consecutive speech segments of the same speaker, as an alternative evidence (we always used either speech segments or speaker turns). Finally, words in a speech segment also influence the likelihood that an entry point can be found in the remaining speech segments [5], therefore we also considered the transcript for the whole document as evidence.
For each of the three evidence sources, meta data, speech or speaker turn segments, and the transcript of the whole document, we create a ranking using two standard retrieval models, see Section 3. We refer to scores for the speech segment or the speaker turn as sbase because we determine the entry point relative to these segments. To the scores on the meta data, we refer to by smeta, and to the score on the transcript for the whole video as sdoc. Therefore, while sbase can be be used to find entry points, smeta and sdoc provide general evidence about the relevance of a video and could be combined with sbase to promote segments within a video with many relevant entry points. 2.2
There are good reasons for the combination of the above scores. For example, an important query word might be said only in neighboring segments of the segment close to the ideal entry point. Furthermore, the language of the searcher and the speaker in the video might be different, and different sources, such as the available meta data can be useful to enrich the findings. Therefore, it is desirable to combine the findings from all evidence sources. For our first participation in this task, we choose a heuristic-based approach, on which we plan to improve in future work: because scores between evidences are not comparable, we scale the scores for each evidence to the interval [0 : 1], see [3]. We combine the evidence scores linearly. The final ranking function is defined as the following:
sbase max(sbase) +λ2
sdoc
max(sdoc)
+(1−λ1−λ2)
sd = λ1
with
smeta
max(smeta)
(
The ranking based on Equation 1 provides a ranking of intervals where suitable entry points could be. However, they do not necessarily need to be the beginning of this segment. Here, we investigate the following four entry points (EP) relative to the found speech or speaker turn segment: 1. the beginning of the retrieved speech segment (SS), 2. the beginning of the retrieved speaker turn (ST), 3. the beginning of the shot that contains the beginning of the segment (SHOT), 4. the time of the key frame of the latter shot, which is usually close to the middle of the shot (KF).
In this section, we describe the runs we performed to evaluate the evidence sources, parameter settings of the combination method in Equation 1, and methods to select an entry point based on obtained ranking.
For the experiments, we used the search engine PFTijah [1]. We used two different retrieval models (RM): the language models (LM) by the author of the engine, and the okapi retrieval model (BM25), see [4]. We considered the two different versions of the automatic speech recognition (ASR) output from 2010 and 2011, referred to by this number, which both provided a segmentation into speech segments, and inferred speaker turns therefrom. For the retrieval function in Equation 1, we performed a grid search with a step-size of 0.1 for the parameters λ1 and λ2. In the performance figures below, we used either only the segment evidence (λ1 = 1.0) or the best performing combination of the grid search on the development dataset. We performed our evaluation using the three prescribed window sizes of 10, 30 and 60 seconds and a granularity factor of 10 according to [2], but only report results of window size 60 because of space requirements.
Submitted runs.
In the following we list the results of our official runs:
No
(
RM BM25 BM25 BM25 BM25 BM25
Run (
This paper described a basic approach to combine evidence to find segments in a video which might contain relevant information to a user’s query. The ranking function for segments linearly combined a normalized score of the evidence found for each segment by a text retrieval model together with evidence found in the transcript of the whole video and its meta data. Relative to a found segment, we investigated different alternatives for the entry point returned to the user. Among a large set of combinations, we found that the entry point alternative is the most influential. Overall, using the BM25 retrieval model together with the beginning of shot which contained the start of the found segments as an entry point produced the strongest performance. The performance produced by our system was low compared to other systems, which we plan to investigate in the future.