Evaluation tracks o er valuable opportunities to measure scienti c and technological advances. Spinque approaches challenges as the MediaEval Rich Speech Recognition task with the additional goal of developing solutions that can easily transferred from academic labs to industry. The system used during this evaluation was obtained with minimal e ort and no manual optimisation and yet it provides a reasonably good baseline to improve upon. More importantly, it is by nature an extensible approach, based on the concept of declarative search strategies, rather than an ad-hoc search system.
Our participation in the MediaEval Rich Speech
Recognition task, described in [
We modelled and executed our runs as search strategies within the Spinque framework. This is a prototype environment where search processes are divided into two phases: the search strategy de nition and the actual search.
Modelling search strategies in this framework corresponds to designing graph structures, where edges represent dataows consisting of terms, documents (e.g. speech-transcripts), and document-sections. The nodes connected by such edges are pre-de ned, general-purpose operational blocks, that either provide source data (the speech transcripts and the topics) or modify their input data- ow applying operations such as extraction of speci c sections from documents or ranking of sections and documents, to name a few.
Search strategies de ned in this framework are automati3.
The speech transcripts were indexed at two levels of granularity: as whole documents as well as individual SpeechSegment sections. We did not use the tags and the video keyframes provided, nor any other source of evidence.
Our runs can be described as follows: run1 First, all words from title (weight 0:2) and all words from short-title (weight 0:8) are used to search all documents in the collection. Then, all the SpeechSegment sections within those documents are searched using the same keywords. The start of the section is returned as the result. This strategy is depicted in Figure 1. run2 the same as run1, except that all terms from title get a weight of 0:0 and all terms from short-title get a weight of 1:0. This basically discards the terms from title. run3 the same as run1, except that all terms from title get a weight of 1:0 and all terms from short-title get a weight of 0:0. This basically discards the terms from short-title. Run3 should be considered as the \required run".
Textual ranking is performed with the BM25 [
The average time for retrieving results for a topic was 230ms. This time includes \compiling" the search strategy (i.e. translating it into SQL queries) out-of-the-box and without manual optimisations, and the overhead for generating the run- les. A glitch later found in our indexer may have altered results marginally: a few documents have not been included in our index and therefore not retrieved.
The evaluation scores for the 3 submitted runs are shown in Table 1. Scores have been measured with window sizes of 10, 30, and 60 seconds. Overall scores are reasonably satisfying for a simple keyword-search approach. As expected, the combination of both the title and the short-title yield a better result than the individual runs. Best results were title 0.2 0.0 1.0
Weights for short-title 0.8 1.0 0.0
Window size (seconds) 10 30 0.1320 0.2210 0.1164 0.1816 0.1054 0.1630 found on the test-set assigning a larger weight to short-title keywords, which suggests that full titles may carry o -topic words which yield lower precision.
We found that searching short sections produced disappointing rankings, probably due to a non ne-tuned documentlength normalisation. Both parameter con gurations used (for BM25 and for the title / short-title keyword mixture) could be improved with a more exhaustive exploration of their search space. The simplicity of the strategies used and the small size of the corpus at hand would make this approach feasible indeed, which is not the case in general.
One more direction for possible improvements is to experiment with a more ne-grained zooming in, with search windows of e.g. entire documents followed by 10 minute, 1 minute and 5 seconds speeches. Such a multi-stage strategy would likely retain recall and improve precision at every iteration. 5.
The main contribution of this paper is to show how a speci c search engine for speech transcripts of reasonable quality can be instantiated with minimal e ort. While outof-the-box text search is not unique to Spinque's framework, the ability to play with retrieval units of di erent granularities and combine query and/or data sources easily is not common.
We plan to improve on our rst speech retrieval evaluation in two ways: rstly, by automating as much as possible the optimisation of search strategies' free parameters, including the choice of unit retrieval granularities; secondly, by building on top of this optimised baseline with the addition of more sources of evidence that may be available (such as tags and video material). 6.