We submitted a system composed of 26 subsystems as the required run. 13 subsystems are based on Acoustic Keyword Spotting and 13 on DTW. All of them were using three state phoneme posteriors as input. The underlaying phoneme posterior estimators were both in-language (Czech, English) and out-of-language (other 12 languages). We also performed unsupervised adaptation of the artificial neural network (ANN) on the target data and fusion based on binary logistic regression. ∗Igor Szo¨ke was supported by Grant Agency of Czech Republic post-doctoral project No. GP202/12/P567.
Our motivation was to use many (mostly) out-of-targetlanguages systems which can be combined by fusion at the detection level. The goal was to re-use as many trained systems available at BUT as possible. Please bear in mind that reusing all these systems (so-called Atomic Systems) lead to several inconsistencies among them — for example feature extraction and sizes of the ANN.
We performed unsupervised adaptation of ANN on the target data (utterances). Our goal was also to test combination two approaches in the query-by-example task – Acoustic Keyword Spotting (AKWS) and Dynamic Time Warping (DTW). We also explored system calibration with respect to the TWV metric.
All our subsystems use ANN to estimate per-frame
phonestate probabilities (so-called posteriorgrams). The
subsystems based on DTW use the posteriorgrams as features
for calculating distances between query and test segment
frames. The subsystems based on AKWS uses the
phonestate posteriors as HMM output probabilities. We re-use
ANNs, which were trained for different projects as
acoustic models for phone or LVCSR recognizers. One DTW
and one AKWS system were built for each of the 13 ANNs
trained on the following datasets: 3× Speechdat (Czech,
Hungarian and Russian; monolingual LCRC systems [
The Acoustic Keyword Spotting (AKWS) systems follow
our paper [
For standard keyword spotting tasks (in-language task and textual input), the query model is built using a pronunciation dictionary. In SWS task, however, we need to generate the phoneme sequence for each of the query acoustic examples – query-to-text step. This is achieved by decoding each example using free phoneme loop. We cut-off initial and final silence labels (if present) and omit queries having less than three non-silence phones, as these short queries could generate huge amounts of false alarms. We experimented with phoneme insertion penalty in the queryto-text step with the conclusion that it has no significant impact and set it to −1 consistently. 4.
In our implementation, we follow the standard query-byexample recipe – subsequence DTW. Single DTW is run for each combination of query and test segment, where the query is allowed to start at any frame of the test segment. When selecting the locally optimal path in the standard DTW algorithm, transition from the smallest accumulated distance is chosen. In our implementation, we compare the accumulated distances (including the current local distance) normalized by the corresponding path lengths on-the-fly. This is to avoid the preference for shorter paths. As the distance metric, we used the usual negative logarithm of the dot product of phone-state posterior vectors.
In our late submission, we further improved the DTW systems by applying VAD to cut off the initial and the final silence from the query examples. As can be seen in Table 1, it improved the overall system by 10% relative. 5.
For both DTW and AKWS systems, the local maxima of frame-by-frame detection scores are selected as candidate detections. For overlapping detections, only the best scoring ones are preserved. There might be significant differences between the score distributions corresponding to the different queries and it is important to normalize (calibrate) the scores for each query to allow for a single common threshold maximizing the TWV metric. We adopted a new normalization approach, m-norm, which is motivated by the observation that score distributions have very long tails towards the small scores, which significantly differ in shape from query to query. In m-norm, for each query, score corresponding to the mod (maximum) of the score histogram (denoted SM ) is found for each query and subtracted from the original scores – all mods are thus aligned to 0. The scores are further divided by standard deviation estimated only on scores larger than SM , to unify the terms’ variances. 6.
Normalized scores from the individual subsystems were
fused similarly to [
Approach AKWSDTW-vad (late) DTW-vad (notsub) AKWS AKWSDTW AKWSDTW-treefus AKWSDTW-notarlang AKWS-notarlang eval MTWV 0.3776(0.4835) 0.3557(0.4585) 0.3041(0.4165) 0.2969(0.4081) 0.2787(0.3934) 0.2562(0.3685) 0.2778(0.3840) 8.1
We have experimented with three types of NN adaptation
using BABEL system NN. This network was initially trained
with 4 independent softmax non-linearities in the output
layer – one softmax per language (Cantonese, Pashto,
Tagalog, Turkish). The original network had 1065 phoneme-state
outputs (355 phonemes for all the 4 languages). We decoded
SWS-dev data [
In terms of MTWV (UBTWV), our results on SWS dev with BABEL AKWS subsystem are as follows : orig 0.0443 (0.1154), adapt 0.0569 (0.1355), and trfs 0.0769 (0.1630). 8.2
As the TWV metric was set to drastically penalize false
alarms [
Next, we tried to calibrate the scores using binary
logistic regression, where the input to the logistic regression
was a vector of z-normed scores augmented with different
per-term side-information scores [
Finally, we tested m-norm (see section 5), which we found to be superior to z-norm-sideinfo. Furthermore, the additional side information based calibration resulted in no further MTWV improvements.
In terms of MTWV (UBTWV), our results are: raw 0.000 (0.1012), z-norm 0.0330 (0.1434), z-norm-sideinfo 0.0603 (0.1436), m-norm 0.0769 (0.1611). 9.
We successfully built a QbE system making use of a high number of already trained phoneme posterior estimators and applied unsupervised adaptation of ANNs. DTW with application of VAD (VAD is really important) is still superior to AKWS approach. On the other hand, AKWS is level-ofmagnitude faster compared to DTW. Adaptation of ANN is also important, so it makes sense to take as much as possible “black-boxed” phoneme posterior estimators, label the target data and train a new ANN. Finally, we found m-norm calibration is promising in the area of high FA rates and non-posterior scores.