We describe our approach to the Search Subtask of the Search and Hyperlinking Task at MediaEval 2013. We experiment with various methods for segmentation of the recordings into shorter segments which are then used in a standard retrieval setup to search for relevant passages. We use regular segmentation into equilong segments and experiment with machine-learning based segmentation expanding our approach to Similar Segments in Social Spech Task [4].
The main focus of the Search and Hyperlinking Task is
to enable users to nd information relevant to the
submitted query in a collection of audio-visual recordings (Search
Subtask) and to nd more segments similar to the retrieved
ones to enable easier navigation in the collection
(Hyperlinking Subtask). The collection consists of TV programmes
provided by BBC. The video recordings, audio track,
metadata, synopsis, cast, detected shots, detected faces, visual
concepts, subtitles and two automatic (ASR) transcripts {
provided by LIMSI [
We participated only in the Search Subtask of the Search and Hyperlinking task. In our approach, we divided the recordings into shorter segments and applied the Terrier IR system1 to search for relevant passages within the recordings. As we searched in the set of the known-boundary segments, we were able to determine the beginnings and ends of the retrieved segments.
As the training set only consists of four queries, we collected other 30 queries and used the whole set consisting of 34 queries for training. We randomly selected recordings from the collection, identi ed short passages somehow interesting to us in these recordings, and formulated the queries to search for those passages. We formulated the queries to imitate the formulation of the original given queries (e.g. \how to prepare Vietnamese spring rolls", \Thomas Tallis signature", and \a di erence between a hare and a rabbit").
We applied the Hiemstra Language Model with the
parameter set to 0.35 as it achieved good results in our
experiments in the Search and Hyperlinking Task in 2012 [
We also employed metadata information (title, source, variant, description, service name, episode name, and short episode synopsis) and synopsis, which improved the results on the training data. We simply concatenated each segment (in all recordings) with the metadata information available for the corresponding recording. We also tried to use the cast information, but it slightly decreased the results.
Segmentation of the recordings is realized in two ways: regular segmentation and segmentation based on machine learning (ML). 2.1
The regular segmentation achieved good results in last
year's experiments [
The ML-based segmentation is adopted from our
experiments conducted for the Similar Segments in Social Speech
Task [
As the training set of the Search Subtask is very small (even after the additional training data is included), we dement in Social Speech Task to segment the transcripts in the Search and Hyperlinking Task. However, this approach did not bring any positive results and needs to be investigated in more depth.
This research is supported by the Charles University Grant Agency (GA UK n. 920913) and the Czech Science Foundation (grant n. P103/12/G084). cided to use the model trained on the Similar Segments in Social Speech Task.
We at rst identi ed all possible beginnings of the segments and all possible ends of the segments, both tuned for high F-measure. Then, for each possible beginning, we identi ed the segment end (from the set of possible segment ends) which lay closest to 50.111 seconds (which is the average segment length in the Similar Segments in Social Speech Task) from the beginning.
This approach (when the model is trained on a di erent data set) enables us to examine the possibility of creating a universal model for ML-based segmentation. However, it also carries potential problems. The sets of the cue words collected on the student dialogues may di er from the cue words used in TV programmes, and the silence between the words in dialogues may have di erent distribution as the silence between words in the TV programmes.
The Search Subtask is evaluated using Mean Reciprocal
Rank (MRR), mean Generalized Average Precision (mGAP),
and Mean Average Segment Precision (MASP). Details about
these measures can be found in the task description [
Our results for the Search Subtask are presented in the Table 1. The highest result is achieved on the subtitles, using the regular segmentation and employing metadata and synopsis. The subtitles, unsurprisingly, outperform both ASR transcripts in all measures; the LIUM transcripts outperform LIMSI transcripts in all measures, except the MASP score for the regular segmentation if the metadata are employed. The ML-based segmentation is applied on the LIUM transcripts only, and it did not improve the results.
In our approach to the Search Subtask, we employed regular segmentation into 50 seconds long passages on three transcripts. The highest score was achieved on the subtitles. The utilization of metadata slightly improved the results in all applied measures.
We also proposed another solution which interlinks this task and our solution used in the Similar Segment in Social Speech Task and use models trained in the Similar Seg5. 6.