Rich Speech Retrieval performance improves when general query-language words are ltered and both speech recognition transcripts and metadata are indexed via BM25F(ields).
Our Rich Speech Retrieval (RSR) approach lters words
in the query into two categories and treats each separately.
RSR is a known-item task that involves returning a ranked
list of jump-in points in response to a user query
describing a segment of video in which someone is speaking. The
queries are given in two formulations: a long form consisting
of a natural language description of what the known item is
about (ca. one sentence in length) and a short form
consisting of a keyword version of the query as it might be issued
to a general-purpose search engine. The video corpus used
contains Creative Commons content collected from blip.tv
and the spoken channel is a mixture of planned and
spontaneous speech. Although visual features might prove
helpful for some RSR queries, here, we investigate only the use
of ASR-transcripts and metadata. Note that although the
know-items targeted in the RSR task correspond to
particular speech acts, we did not investigate this aspect here. More
details on the RSR task are available in [
We conjecture that users queries are a mixture of two
distinct types of language: general query language and primary
language. General query language is language the users
always use when formulating queries for videos during a search
session with a general search engine (e.g., video, episode,
show ). Our conjecture is based on informal observation of
user query behavior. It is supported by a user study of
At the time the work presented here was done the author
was a liated with Novay, Enschede (The Netherlands) and
Delft University of Technology.
podcast search behavior [
In this section, we describe our approaches to RSR. For all runs, we produce our ranked list of jump-in points using a standard IR algorithm to retrieve video fragments that have been de ned on the basis of the ASR-transcripts. We return the start point of each fragment as a jump-in point. Fragments are de ned as a sequence of sentences of about 40 non-stop-words. Sentences are derived on the basis of punctuation (full-stop = sentence end), which is hypothesized by the recognizer and included in the output of ASR-system. If a sentence is less than 40 words in length, subsequent sentences are added until it approximately meets this target.
Mark Hepple's [
We carry out ranking using BM25 [
N
dft + 0:5 dft + 0:5 (1) Here, N is the total number of fragments, and dft is the number of fragments in which term t occurs. The weight of each term in each fragment-document is given by w(d; t), w(d; t) = idf(t)
(k + 1) fdt fdt + k (1 b + b avlgddl ) ; (2) where fdt is the number of occurrences of term t in document d, ld is the length of d, and avgdl is the average document length. In our experiments, we set k = 2 and b = 0:75.
The retrieval status value (RSV) of a document for query consisting of more than one word is de ned as, w(d; Q) =
X w(d; t): t2Q (3) Note that each query word contributes once to the sum, i.e., repetition of query words is ignored.
We create an initial ranking by ordering all fragments by their RSV values (Eq. 3). In order to generate our nal results list, we remove all fragments with a starting time within a window of 600 seconds of a higher ranked fragment.
The approaches used by our runs are shown in Table 1.
In runs 4 and 5 we use metadata (descriptions, title and
tags) along with the ASR-transcripts. These runs make use
of the BM25 extension known as, BM25F(ields) [
(4)
X t2Q;f2F Here, F is a set of elds, df the part of document d labeled as eld f , and where wf is the weight for eld f . In our experiments we use wf = 1 for the ASR and wf = 0:5 for all other elds. Tests on the development set showed that results are not particularly sensitive to the exact value and we used 0:5 since it gave the best results.
In runs 5, we applied a query word lter built using a
corpus of 3,400 requests for video made by users on Yahoo!
Answers, cf. [
N
dft + 0:5 dft + 0:5 (5) +(1 ) log Nreq reqft + 0:5 reqft + 0:5 ; where Nreq is the number of requests in the corpus and reqft is the number of requests in which term t occurs. In the reported runs we have set = 0:5.
Our results our reported in Table 2 and given in terms
of the mean Generalized Reciprocal Rank (mGRR) [
We can see that ltering or downweighting general querylanguage words (e.g., video and tv ) can indeed improve results. Downweighting has a larger impact, suggesting that general query-language words should not be treated by extending a conventional stop word list for application in video retrieval. No appreciable di erence was observed between using ASR transcripts alone and using both ASR transcripts and metadata in the conventional case in which query words are all treated the same (cf. run 2 vs. run 4). Apparently, separate treatment for di erent types of query words is particularly important to fully exploit the contribution of metadata (cf. run 2 vs. run 5b). In the experiments, we nd that adding the query-language downweighting slightly improves the results of very many queries, as long as they already performed reasonably well without downweighting. However, a number of queries fail completely. An investigation of these cases carried out by hand revealed that failure was in most cases due to vocabulary mismatch between query and target item, suggesting that performance would bene t from the use of conventional techniques for query expansion.
Future work will focus on developing more sophisticated models for general-language query words. Additionally, we will attempt to model of query words that are `primary', i.e., more likely to occur in spontaneously produced and/or direct speech and less likely to occur in the descriptive or indirect descriptions of the video in the metadata.
Acknowledgments The research leading to these results has received funding from the European Commission's 7th Framework Programme (FP7) under grant agreement no. 216444 (EU PetaMedia Network of Excellence).