In this paper, we present an approach to detect spoken keywords according to a given query, as part of the MediaEval benchmark. The proposed approach is based on a concept of modelling the speech query as a concatenation of language-independent quasiphoneme models, which are derived by unsupervised clustering on various audio data. Since only an initial version of the system is presented, issues concerning further system improvements are also discussed.
The developed system generates stochastic models of “elementary
sounds” (ES) derivated from the provided speech data in various
languages. These ESs are used as building blocks for speech
modelling instead of conventional phoneme based models. We
have recently adopted this approach to the task of generic sound
modelling and retrieval [
System layout is depicted in Figure 1. Each utterance (query) is
modelled by a HMM-based statistical model. Due to lack of
information about language and linguistic structure of the
utterances, obviously, models of phonemes (or any other
linguistic units) could not be built in advance. Instead, we have
proposed to build models of language-independent quasi
phoneme-like ES units with aid of unsupervised clustering. Then,
the HMM for each query is built up by concatenation of such ES
models (or fillers). Searching the query over the database is
performed by Viterbi decoding [
We applied a 30 ms sliding window with 10 ms shift for computation of MFCCs (Mel Frequency Cepstral Coefficients), which may be considered as a standard in speech recognition. Each frame is represented by the 39–dimensional feature vector that is composed of 13 MFCCs along with their delta and delta– delta derivates (incl. 0th cepstral coefficient), followed by Cepstral Mean Normalisation. The feature vectors form observations for further acoustical modelling by HMM.
The overall performance of our proposed approach for the SWS task relies on how precisely the developed models of ES units mimic real phoneme-based units. Thus they should preserve the linguistic information as much as possible while the non-linguistic features (related to speaker/gender variability, emotion, background noise, channel characteristics, etc) should be suppressed. This is a very challenging problem and can be solved only to a certain extent. We are aware of the fact that the developed ES units are only a rough approximation of linguistic units if no information about language structure is taken into account.
In these initial experiments, we developed the ES models by the
following procedure:
1) All 20 hours of audio of the SWS2013 dataset were
parameterized as a sequence of MFCC vectors.
2) Means and variances of Gaussian components of
semicontinuous Gaussian Mixture PDF were estimated by
unsupervised clustering of all MFCC vectors from the dataset.
The clustering was performed by the K–variable K–means
procedure [
Audio data Feature extraction C luster analysis - estim ation of PD F G auss. com ponents
D ata stream segm entation
E S m odel estim ation
C luster analysis
- reduction of m odels num ber
The queries were represented by concatenation of these ES
models as it is shown in Figure 1. For each query, the best
sequence of the ES models was obtained by Viterbi decoding
according to [
Maximal confidence obtained by Viterbi decoding was chosen as
the score in submission. The segments with score about the given
threshold (the threshold was tuned on the development queries)
were labelled as YES decision. With an effort to decrease high
false alarm rate, the segments with duration D > 2Q or D < 0.5Q
(where Q is the duration of the query) were filtered out.
The performance of the submitted system was evaluated in terms
of several measures, as defined by the SWS2013 task [
We reckon that the following bottlenecks caused very low performance of this first version of the submitted system: - Since the ES models are obtained by unsupervised clustering, a very important issue is that the features belonging to the same linguistic unit should be grouped together in the feature vector space. The problem with MFCC based features is that they are advisable for both speech recognition as well as speaker discrimination tasks, what is disadvantageous in our case. Some discriminant analysis transform on MFCC, to obtain the speaker– independent features, might be applied prior to clustering; - We haven’t investigated the impact of the size of the ES inventory on the search performance. Due to lack of time also only a simplified HMM training without re-estimation was applied for ES model development; - After inspection of the results we have noticed that ES models of noise and silence were very often found in the decoding sequences, and overall confidence was affected by these “non linguistic” models. Prior proper speech activity detection on queries as well as audio content (to avoid modelling of nonspeech events) would also help.
The system was run on a workstation with 32-core CPU. All the programming was made in Matlab and it was not optimized for speed. In pre-processing (indexing) phase, only MFCC features were precomputed. It took about 1/60 of real-time, that means ISF = 1205/ (71839+696 sec.) = 0.017 for the evaluation data. Note that ES models computation is not considered in ISF. In the submitted version of the system, audio content-based adaptation of ES models is not considered, thus the development of ES models might be seen as an extra part. The processing time employed in searching (recomputed for single CPU) is approx. SSF = 1.08 x 106 / (71839 x 696 sec.) = 0.022. The peak memory usage in searching is very low because only the Viterbi algorithm is performed. Rough estimation is about 1 – 10 MBytes. Hence, the proposed system, if it is tuned (and compiled in other more suitable programming environment), might be computationally very efficient.