This paper describes the experiments and observations for Query-by-Example Search on Speech Task (QUESST) at MediaEval 2014. In this paper, we describe two di erent representations of speech that were explored for the task. We also show the capabilities and limitations of non-segmental dynamic time warping (NS-DTW) technique for searching various types of queries. This paper mainly focuses on the experiments and analysis of the existing NS-DTW algorithm for various types of queries. The observations show that for a speci c representation of speech, the algorithm is capable of detecting partial matches.
Some of the approaches for query-by-example spoken term detection rely on building models from resource rich languages, and use these models to convert the speech data into sequence of symbols. Building models for multi-lingual data is a challenging task as phone classes are not language universal. Another way is relying on dynamic time warping (DTW) based techniques for matching two time series vectors. Here, speech data is usually represented as Gaussian posteriorgrams (GP) of various acoustic features.
For MediaEval 2014 QUESST task [
A three step process to generate the features for queries
and the audio references is described here. (a) 39
dimensional frequency domain linear prediction (FDLP) features
along with delta and acceleration coe cients were extracted
for every 25 ms window and a shift of 10 ms. An all-pole
model of order 160 poles/sec and 37 lter banks were
considered to extract FDLP features. (b) Bottle neck (BN)
features were derived from Multi-layer perceptron (MLP)
trained with articulatory features (AF) (c) Gaussian
posteriorgrams were computed for speech parameters (FDLP)
in tandem with articulatory bottle neck features. Bottle
neck features are a form of compressed features which are of
lower dimension and also capture the classi cation
properties of the target classes. These features were obtained from
the MLP trained on 24 hours of labeled Telugu database
[
We used a variant of DTW called non-segmental DTW
(NS-DTW) [
All the experiments were performed on a single HP SL230 node which is equipped with two Intel E5-2640 processors with 12 cores each and 64 GB of main memory. The peak memory usage (PMU) was approximately 12 GB. The searching speed factor (SSF) was 3.46.
To increase the search speed, the distance computation was parallelized on a GPU (NVIDIA GT 610 with 48 cores and 2 GB of GPU memory). The SSF was reduced to 0.85. 1https://sites.google.com/site/bosaristoolkit/
MinCnxe
Cnxe
All 0.8070 0.9121 0.2263 0.2261
All dev dataset
Type of queries Type 1 Type 2 0.6734 0.8739 0.8032 1.0121 eval dataset
Type of queries Type 1 Type 2 100
We have analyzed the cases of false alarms and misses for all types of queries. The analysis on false alarms helped us in enforcing a slope constraint on the aligned path which was described in Section 3. The results in Table 1 show that the NS-DTW algorithm is able to detect some of the type 2 queries, but fails in detecting type 3 queries. Fig. 1(a) shows the similarity matrix plot for a multi-word query with ller content present in the reference. The dark bands represent the match between the query and the reference. In Fig. 1(a) there are multiple dark bands, each showing a match between parts of the query (word) to the speci c locations (words) in the reference. The peaks in the alignment scores in Fig. 1(b) re ects the partial matches. This shows that for this speci c (FDLP + AF-BN) feature representation of speech, the algorithm is capable of detecting smaller/partial matches. Even though the scores re ect the partial matches, we have observed that the poor performance of the system is due to the number of false alarms. Further investigation is required to nd the methods that can penalize the false alarms.
(a) (b) global hypotheses was considered as the reference in computing the phone confusions. Next, the queries and the audio references were decoded using the bootstrapped models, and the search was performed using the NS-DTW. The phone confusion matrix was used in the computation of similarity matrix in the NS-DTW framework.
If i and j are the indices of phones and N is the number of phones in the dictionary, then the similarity between them is given by,
d(i; j) = c(i; j) 8 0 i; j N where c(i; j) is the confusion matrix of i being the reference phone and j being the query phone.
The SSF in this case was 0.38 and the PMU was approximately 2 GB. The results for various types of queries on development dataset are shown in Table 2. 6.
In this work, we have explored two di erent representations of speech. We have observed the capabilities and limitations of NS-DTW algorithm for various types of queries. We have also observed that the same algorithm is able to detect some of the type 2 queries in the reference documents. The future work is focused on improving the NS-DTW algorithm for detecting type 2 and type 3 queries and also in developing robust cross-lingual phone decoders. 00 20 40 60 80 100
Query (frames) 10
In this work, we have also built a cross-lingual phone
decoder and used NS-DTW for search. The cross-lingual
decoder was built in a two step process. As the rst step,
we trained acoustic models on 24 hours of Telugu database
[
We have built a phone confusion matrix in an unsupervised way which is as follows: (a) We divided the SWS 2013 database into 4 parts and 4 acoustic models were built (b) 4 hypotheses (local hypotheses), each corresponding to a di erent part of the database were obtained (c) A string alignment was done between the global hypotheses and each of the local hypotheses to obtain the phone confusions. The