TUKE at MediaEval 2013 Spoken Web Search Task Jozef Vavrek, Matúš Pleva, Martin Lojka, Peter Viszlay, Eva Kiktová, Daniel Hládek, Jozef Juhár Technical University of Kosice, Letna 9, 04200 Košice, Slovakia {Jozef.Vavrek, Matus.Pleva, Martin.Lojka, Peter.Viszlay, Eva.Kiktova, Daniel.Hladek, Jozef.Juhar}@tuke.sk TRAINING PHASE ABSTRACT Dev. VAD Segmentation K-means GMM Acoustic MFCC (13) utterances VAMFCC based training This paper provides a rough description of zero resource PCA based clustering models Query-by-Example retrieving system for the MediaEval 2013 spoken web search task. The proposed solution firstly imple- RETRIEVING PHASE ments the voice activity detection (VAD) utilizing variance Posteriorgram Audio query of acceleration MFCC (VAMFCC) rule-based approach. A VAD MFCC (39) Scoring using for query Posteriorgram SDTW VAMFCC based Mah. distance PCA-based segmentation, K-means clustering and GMM Test utterance for utterances training are then used in order to built the posteriorgrams. Transducer Final match GMM based Finally, two searching architectures based on posteriorgram matching (SDTW) and GMM modeling (GMM-FST) are evaluated. Results show that none of our systems is able to Figure 1: SWS framework architecture achieve the positive Actual Term Weighted Value, because of high number of insertions. We suppose that chosen clus- tering scheme caused generation of too many false alarms. query enters the block of model-based scorer, where scoring Only provided data were used and no other resources were is performed by computing the Mahalanobis distance be- examined in any system component during the development. tween MFCC vectors (frames with length 25 ms and 10 ms shift) and GMM acoustic model. The product of that are 1. MOTIVATION posteriorgrams. They are defined as probability vectors with The main purpose of our experiments was to check the the length of N, where N represents the number of GMMs. proposed approaches for the language independent audio Segmental Dynamic Time Warping (SDTW) algorithm is query detection and new speech feature analysis compo- then used for comparing these posteriorgrams and finding nents. The mentioned approach is used in the MediaEval a possible occurrence of a query in the test utterance. The activity [1] and could be also applied in various speech [4] other solution GMM-FST (GMM-Finite State Transducers) or non-speech [5] Query-by-Example applications. implements a Viterbi algorithm to create a model for each query and state sequence network to find occurrences in test utterance by using this model. 2. SYSTEM OVERVIEW Proposed solution for SWS task uses posteriorgram term 2.1 Segmentation and Clustering matching and audio segment GMM modeling. The overall In order to identify and to distinguish the speech segments architecture of proposed system is depicted on Fig.1. in the i-th utterance, PCA (Principal Component Analysis) At first, a training phase is carried out using available de- was applied as follows. Each 13-dimensional MFCC vec- velopment utterances. The VAMFCC-based silence detector tor xj was reshaped to matrix Xj with row dimension nr , performs the initial discrimination of silent parts in audio where j ∈ h1; ni i is the number of vectors in i-th record- stream. The block of feature extraction is implemented af- ing. In the next step, the covariance matrix C1 was com- ter VAD utilizing 13 MFCCs. The phase of segmentation puted from the first matrix X1 and its eigenvectors and and clustering creates the audio segment units (ASU). ASU eigenvalues were computed. The eigenvalue spectrum Λ1 = is represented as a small audio part (phoneme for example) {λ1j }n r j=1 was used to determine the significance ∆(λ1max ) with some spectral and temporal characteristics, different for λ each ASU. Then the training of acoustic models is performed of the dominant eigenvalue of C1 as ∆(λ1max ) = Pn1rmax λ , j=1 1j using these ASU, where each ASU represents one class. La- where λ1j are the eigenvalues of C1 . Then the matrix X1 bels for these classes (ASUs) are assigned according to the was spliced together with X2 and the covariance matrix C12 number of GMM. Only the process of voice activity detec- and ∆(λ12max ) were computed again. If ∆(λ12max ) com- tion and feature extraction is then performed in preprocess- pared to ∆(λ1max ) changed significantly, a new speech seg- ing stage within the retrieving phase. Each utterance and ment was created and PCA started from the current frame. In the other way, if ∆(λ12max ) did not change significantly, the current matrix X12 was spliced together with X3 and Copyright is held by the author/owner(s). the process was repeated automatically until a new segment MediaEval 2013 Workshop, October 18-19, 2013, Barcelona, Spain was indicated. The created segments corresponded to ASUs. Table 1: Evaluation results of the tested algorithms Table 2: Processing resources measures min query system ATWV Cnxe Cnxe system ISF SSF P M UI P M US PL dev GMM-FST -0.1371 0.980745 0.976363 GMM-FST 0.0054 0.0048 2GB 1.8GB 0.009 eval GMM-FST -0.1372 0.98505 0.979883 SDTW 0.0054 0.0046 2GB 2.2GB 0.01 dev SDTW -0.4176 0.998421 0.989649 eval SDTW -0.4252 0.998494 0.988404 3. EXPERIMENTAL RESULTS AND CON- CLUSIONS In the next phase, the segments with similar acoustic The official results for SWS task are listed in Tab. 1. and statistical properties were grouped together into sev- Two metrics were used to asses the overall performance of eral speech clusters using k-means clustering with k = 50 GMM-FST and SDTW on dev and eval queries: the ac- clusters and squared Euclidean distance metrics. As the in- tual AT W V , normalized Cnxe and minimal cross-entropy min put data for clustering the means of the segments were used. Cnxe . The score normalization for both systems was per- Each mean vector obtained an index (label) of the specific formed only on development data. A minimum-cost align- cluster. This label was assigned to the original feature vec- ment (MCA) for each segment was used as detection score at tors corresponding to the specific mean vector. first level of search in case of SDTW. Final detection of re- trieved query was then carried out utilizing score parameter 2.2 Searching techniques defined in (2.2.2). A threshold for this parameter was set to 0.0819. A decision threshold for score parameter was set to 2.2.1 GMM approach 2.8 in case of GMM-FST based system, while Θ = 3 (2.2.1). Both systems produced a huge amount of false alarms (FA) A retrieving process uses Weighted Finite State Transduc- during the evaluation. Regarding the evaluation results, the ers (WFST) that allow us to find the most probable path GMM-FST system is more appropriate solution for SWS (state sequence) in search network [2]. The search of a query task, because of its lower tendency to detect spurious terms. consists of two steps. At first, query alone is recognized us- All the experiments were mainly done using 2x IBM Sys- ing search network, created from the trained acoustic model tem x3650 servers, 2x Intelr Xeonr QuadCore E5530 CPU so that all GMM states are arranged in parallel. The re- @ 2.4 GHz Hyper-threading enabled (16 threads), 28 GB sult is a sequence of states that model the particular query. RAM, 1TB SAS HDD (RAID5), running Debian OS. The process of recognition is done repeatedly with differ- The Speed Factors (the ratio of the total time employed ent insertion penalties in order to obtain multiple states in searching{indexing} the set of queries in{and} the set of sequences with different lengths. It helps to improve the audio documents to the product{sum} of their total dura- model representation of retrieving query. The sequences are tions) and Peak Memory Usage during Searching{Indexing} labeled and added to the previous search network in paral- tasks are presented in Tab. 2. The Performance Load equals lel. The second step involves the recognition of a test ut- 0.9 · SSF · P M US + 0.1 · ISF · P M UI is derived from them. terance using Viterbi algorithm. The final score for decision In the future, an improved clustering and segmentation al- is computed as a difference between modeled likelihoods of gorithm will be investigated in order to decrease the overlap- query and utterance using lambda acoustic model where ping between individual ASUs. A minimal length of warping score = (P (Ooccurence |λ) − P (Oquery |λ)) + Θ. Score is then path algorithm will be integrated in SDTW approach, too. shifted by predefined value Θ and then results with score below zero are removed. 4. ACKNOWLEDGMENTS 2.2.2 SDTW detection This research was supported by the ERDF funded projects A simplified SDTW searching algorithm was utilized in ITMS-26220220155 (50%) & ITMS-26220220182 (50%). our system, similar to that used in [3]. The adjustment win- dow condition was set to |(ik − i1 ) − (jk − j1 )| ≤ R, where i1 5. REFERENCES and j1 are starting coordinates of warping path in each seg- [1] X. Anguera, F. Metze, A. Buzo, I. Szoke, and L. J. ment, ik and jk define the k-th coordinates and R represents Rodriguez-Fuentes. The Spoken Web Search Task. In the constraint parameter, set to M/2, where M is the length MediaEval 2013 Workshop, page 4, Barcelona, Spain, of query. The range of starting coordinates was conditioned 18-19 October 2013. by the constraint parameter and length of each utterance: [2] M. Lojka and J. Juhár. Finite-state transducers and N −1 speech recognition in Slovak language. In SPA 2009 ((2R + 1)k + 1, 1), where 0 ≤ k ≤ 2R+1 . The process of Conference, pages 149–153, art. no. 5941305, 2009. finding the optimal local alignment between each utterance [3] A. Park and J. Glass. Unsupervised pattern discovery and query produces a set of local warp paths, equal to the in speech. IEEE T Audio Speech, 16(1):186–197, 2008. number of diagonal regions. A score s parameter was then set [4] J. Vavrek, M. Pleva, and J. Juhár. TUKE MediaEval P 2n n warpDist 1 2012: Spoken Web search using DTW and in the following form score = N +M n+1 ,where unsupervised SVM. In MediaEval 2012 Workshop, Pisa n is the number of steps in local alignment, - CEUR Workshop Proceedings, volume 927, 2012. P N is the length of utterance and M the length of query, n [5] E. Vozáriková, M. Pleva, S. Ondáš, J. Vavrek, J. Juhár, 1 warpDist rep- resents a summation of components in each warping path, and A. Čižmár. Detection and classification of audio where components are computed from Bhattacharyya dis- events in noisy environment. Journal of Computer tance matrix. Science and Control Systems, 3(1):253–258, 2010.