In this paper, we attempt to solve the Spoken Term Detection (STD) problem for under-resourced languages by a phone recognition approach within the Automatic Speech Recognition (ASR) paradigm, with multilingual acoustic models from six languages (Albanian, Czech, English, Hungarian, Romanian and Russian). The Power Normalized Cepstral Coefficients (PNCC) features are used for improved robustness to noise, along with Phone Posteriorgrams in order to obtain content-aware acoustic features as independent as possible from speaker and acoustic environment.
In our approach, one thing we want to compare is the effect
of using PNCC features vs MFCC, along with the improvements a
robust phone recognizer based on neural network classifiers can
bring to our STD task. For the first comparison, we have built four
acoustic models, using internal audio resources for training, as
described in Table 1. The AM training and the phoneme
recognition are made in a conventional way, using Hidden
Markov Models (HMMs), in CMU Sphinx [
We have built an AM for each language, (AM1 - AM3).
AM1 is trained with 8.7 hours of read speech. We could have
trained the Romanian AM with more data, but as we stated in the
introduction, we wanted to have balanced training data among
different languages, for an under-resourced task. AM2 is trained
with 4.1 hours of Albanian read speech and broadcast news. AM3
is trained with 3.9 hours of native English read speech from the
standard TIMIT database [
No. phones 34 36 75 98
Training data [h] 8.7 4.1 3.9 16.7
For our second approach, we used phone posteriorgrams that
are output by the robust phoneme recognizer from BUT. This
phone recognizer uses a split temporal context (STC) based
feature extraction, with neural network classifiers [
In order to use additional languages to build features for the
phoneme recognizer, we used the pre-trained systems available at
[
No. phonemes 45 61 52
The languages used for training the systems described in
Table 2 are from the SpeechDat-E Eastern European Speech
Database [
The STC approach is based on the theoretical study that
significant information about phoneme is spread over few
hundreds milliseconds and that an STC system can process two
parts of the phoneme independently. The trajectory representing a
phoneme feature can then be decorrelated by splitting them into
two parts, to limit the size of the model, in particular the number
of weights in the neural-net (NN). The system uses two blocks of
features, for left and right contexts (the blocks have one frame
overlap). Before splitting, the speech signal is filtered by applying
the Hamming window on the whole block, so that the original
central frame is emphasized. Dimensions of vectors are then
reduced by DCT and results are sent to two neural networks. The
posteriors from both contexts are, in the final stage, merged, after
the front-end neural networks are able to generate a three-state per
phoneme posterior model [
In ideal conditions, if an ASR system should output utterances with a 100% accuracy and precision, then the STD would be reduced to a simple character string search of a query within a textual content. As the experimental results show, we are far from the ideal case, hence we have to find within a content a string which is similar to the query.
Our proposed DTW String Search (DTWSS) uses the Dynamic Time Warping to align a string (a query) within a content. The search is not performed on the entire content, but only on a part of it by means of a sliding window proportional to the length of the query. The term is considered detected if the DTW scores above a threshold. This method is refined by introducing a penalization for the short queries and the spread of the DTW match. The formula for the score s is given by equation (1): − )(1 + − = (1 − ℎ )(1 +
−
where Lq is the length of the query, LQM = 18 and LQm = 4 are
the maximum and the minimum query lengths found in the
development data set, LW is the length of the sliding window, LS is
the length of the matched term in the content, while α and β are
the tuning parameters. For this task, α and β are set to 0.6, from
previous evaluations [
For our first comparison, the results obtained on the development database with the first two type of features (MFCC and PNCC) are shown in Table 3.
All our runs used our proposed DTW algorithm, described in
section 1.2. The metric used is the normalized cross entropy cost
(Cnxe) [
The results obtained by the official runs on the evaluation database are shown in Table 5. We selected the first five best models, sorted after the Actual Cnxe metric. Because no tuning is made based on the development data set, the results on the evaluation data set are quite similar and the same conclusions can be drawn. Table 5 shows also the results per query type. *PNCC model
It can be noticed that better results are obtained by query type 2, as these queries are longer, which may have affected the results. Posterior models (AM6/AM7) seem to offer minimal performance improvements, so we cannot draw a conclusion that posteriors are better suited for the STD task in difficult acoustical environments. Going further, we think improvements can be obtained by preprocessing the audio first, before extracting features, to remove noise or possible reverberation.
We have approached STD with a two-step process. A multilingual ASR is used as a phone recognizer for indexing the database, while a DTW based algorithm is used for searching a given query in the content database. We tested three types of features (MFCC, PNCC and Posteriorgrams) with two approaches to the phoneme recognizer (statistical HMMs and a neural STC approach). The results show no big improvement between each approach and feature types, in part because this year’s database features very challenging acoustic environments, and our phonetizers return a lot of “noise” tokens or repeated phonemes, which reflected further upon our DTW algorithm.
The work has been funded by the Sectoral Operational Programme Human Resources Development 2007-2013 of the Ministry of European Funds through the Financial Agreements POSDRU/159/1.5/S/132395 and in part by the PN II Programme "Partnerships in priority areas" of MEN - UEFISCDI, through project no. 332/2014.