=Paper=
{{Paper
|id=None
|storemode=property
|title=GTTS Systems for the SWS Task at MediaEval 2013
|pdfUrl=https://ceur-ws.org/Vol-1043/mediaeval2013_submission_83.pdf
|volume=Vol-1043
|dblpUrl=https://dblp.org/rec/conf/mediaeval/Rodriguez-FuentesVPBD13
}}
==GTTS Systems for the SWS Task at MediaEval 2013==
https://ceur-ws.org/Vol-1043/mediaeval2013_submission_83.pdf
GTTS Systems for the SWS Task at MediaEval 2013
Luis J. Rodriguez-Fuentes, Amparo Varona, Mikel Penagarikano,
Germán Bordel, Mireia Diez
Software Technologies Working Group (http://gtts.ehu.es), DEE, ZTF/FCT
University of the Basque Country UPV/EHU, Barrio Sarriena, 48940 Leioa, Spain
{luisjavier.rodriguez, amparo.varona, mikel.penagarikano, german.bordel, mireia.diez}@ehu.es
ABSTRACT Finally, the posteriors of the three non-phonetic units are
This paper briefly describes the systems presented by the added and stored as a single non-speech posterior. Thus, the
Software Technologies Working Group (http://gtts.ehu.es, size of the frame-level feature vectors is 43, 59 and 50 for the
GTTS) of the University of the Basque Country (UPV/EHU) Czech, Hungarian and Russian BUT decoders, respectively.
to the Spoken Web Search (SWS) task at MediaEval 2013. 2.2 Speech Activity Detection
GTTS systems consist of four main modules: (1) feature Given an audio signal, Speech Activity Detection (SAD) is
extraction; (2) speech activity detection; (3) DTW-based performed by discarding those phone posterior feature vec-
query matching; and (4) score calibration and fusion. The tors for which the non-speech posterior is the highest. The
most remarkable contributions are the use of phone log- remaining vectors, along with their corresponding time off-
likelihood ratio features, the normalization of the DTW dis- sets, are stored for further use, but the component corre-
tance matrix and the calibration/fusion approach (which is sponding to the non-speech unit is deleted. If the number of
imported from language/speaker verification). speech vectors is too low (in this evaluation, that threshold
was arbitrarily set to 10, that is, 0.1 seconds), the whole
1. INTRODUCTION signal is discarded, to save time and to avoid false alarms.
The MediaEval 2013 Spoken Web Search (SWS) task con-
sists of searching for a spoken query within a set of audio 2.3 DTW-based query matching
documents [4]. The locations and durations of all the oc- Given two SAD-filtered sequences of feature vectors corre-
currences of spoken queries in the audio documents must sponding to a spoken query q and a spoken document x, the
be obtained. System performance is primarily measured in cosine distance is computed between each pair of vectors,
terms of the Average Term-Weighted Value (ATWV) [5], q[i] and x[j] as follows:
but also in terms of a normalized cross-entropy metric and q[i] · x[j]
the processing resources (real-time factor and peak mem- d(q[i], x[j]) = − log (2)
|q[i]| · |x[j]|
ory usage) required by the submitted systems [6]. For more
details on the SWS task at MediaEval 2013, see [2]. Note that d(v, w) ≥ 0, with d(v, w) = 0 if and only if v
and w are perfectly aligned and d(v, w) = +∞ if and only
2. SYSTEM OVERVIEW if v and w are orthogonal. The distance matrix computed
according to Eq. 2 is further normalized with regard to the
2.1 Feature extraction spoken document x, as follows:
The Brno University of Technology (BUT) phone decoders
d(q[i], x[j]) − dmin (j)
for Czech, Hungarian and Russian [7] are applied to de- dnorm (q[i], x[j]) = (3)
dmax (j) − dmin (j)
code both the spoken queries and the audio documents.
BUT decoders are trained on 8 kHz SpeechDat(E) databases with dmin (j) = min d(q[i], x[j]) and dmax (j) = max d(q[i], x[j]).
i i
recorded over fixed telephone networks, containing 12, 10 In this way, matrix values are all comprised between 0 and
and 18 hours of speech and featuring 45, 61 and 52 units for 1, so that a perfect match would produce a quasi-diagonal
Czech, Hungarian and Russian, respectively (three of them sequence of zeroes.
being non-phonetic units that stand for short pauses and The best match of a query q of length m in a spoken
noises). document x of length n is defined as that minimizing the
Given an input signal of length T , the decoder outputs average distance in a crossing path of the matrix dnorm . A
the posterior probability of each state s (1 ≤ s ≤ S) of each crossing path starts at any given frame of x, k1 ∈ [1, n],
unit i (1 ≤ i ≤ M ) at each frame t (1 ≤ t ≤ T ), pi,s (t), then traverses a region of x which is optimally aligned to
where M is the number of units and S the number of states q (involving L vector alignments), and ends at frame k2 ∈
per unit. The posterior probability of each unit i at each [k1 , n]. The average distance in this crossing path is:
frame t are computed by adding the posteriors of its states: L
1X
X davg (q, x) = dnorm (q[il ], x[jl ]) (4)
pi (t) = pi,s (t) (1) L
l=1
∀s
where il and jl are the indices of the vectors of q and x
Copyright is held by the author/owner(s). in the alignment l, for l = 1, 2, . . . , L. Note that i1 = 1,
MediaEval 2013 Workshop, October 18-19, 2013, Barcelona, Spain iL = m, j1 = k1 and jL = k2 . The minimization operation
Table 1: Results of GTTS on-time and late systems submitted to the required (single-example) condition.
development queries evaluation queries
MTWV/ATWV Cnxe (act/min) SSF PMUs MTWV/ATWV Cnxe (act/min) SSF PMUs ISF PMUi
p 0.4174 / 0.4078 0.7962 / 0.6605 0.2509 0.325 0.3992 / 0.3806 0.8159 / 0.6570 0.2350 0.226 0.8015 0.027
c1 0.3601 / 0.3586 0.9976 / 0.6877 0.1219 0.325 0.3457 / 0.3430 1.0229 / 0.6838 0.1187 0.226 0.8015 0.027
c2 0.2726 / 0.2687 1.4365 / 0.7559 0.0399 0.298 0.2586 / 0.2538 1.5588 / 0.7543 0.0311 0.200 0.2473 0.023
c3 0.2704 / 0.2466 1.0274 / 0.7710 0.0457 0.302 0.2408 / 0.2221 0.9514 / 0.7665 0.0449 0.204 0.2862 0.027
c4 0.2491 / 0.2437 1.2606 / 0.7716 0.0434 0.300 0.2418 / 0.2372 1.2125 / 0.7534 0.0403 0.202 0.2680 0.024
p-late 0.4186 / 0.4164 0.6659 / 0.6603 0.2509 0.325 0.3994 / 0.3989 0.6582 / 0.6567 0.2350 0.226 0.8015 0.027
c1-late 0.3601 / 0.3590 0.6878 / 0.6877 0.1219 0.325 0.3457 / 0.3438 0.6848 / 0.6838 0.1187 0.226 0.8015 0.027
c2-late 0.2726 / 0.2722 0.7561 / 0.7559 0.0399 0.298 0.2586 / 0.2570 0.7645 / 0.7543 0.0311 0.200 0.2473 0.023
Table 2: Results of the GTTS late system submitted to the extended (multiple-example) condition.
development queries evaluation queries
MTWV/ATWV Cnxe (act/min) SSF PMUs MTWV/ATWV Cnxe (act/min) SSF PMUs ISF PMUi
c2-late 0.3038 / 0.3004 0.6845 / 0.6844 0.0192 0.298 0.2970 / 0.2939 0.6943 / 0.6942 0.0173 0.200 0.2473 0.023
is accomplished by means of a dynamic programming proce- evaluation)— is applied. The BOSARIS toolkit [3] is used
dure, which is Θ(n · m · d) in time (d: size of feature vectors) to estimate and apply the calibration/fusion models.
and Θ(n · m) in space. The detection score is computed as
1 − davg (q, x). The starting time and the duration of each
3. RESULTS
detection are obtained by retrieving the time offsets corre- Tables 1 and 2 show the results (performance and pro-
sponding to frames k1 and k2 in the SAD-filtered spoken cessing resources) for GTTS systems in the required and
document. extended conditions, respectively. All the experiments have
This procedure is iteratively applied to find not only the been carried out on a 2× Xeon E5-2450 (×8 core, ×2 HT)
best match but also less likely matches in the same docu- @2.10GHz, 64GB, under Linux Fedora 3.3.4-5.fc17.x86 64.
ment. To that end, a queue of search intervals is defined The indexing phase involves just applying BUT decoders to
and initialized with (1, n). Let us consider an interval (a, b), extract phone posterior features. ISF, SSF and PMU val-
and assume that the best match is found at (a0 , b0 ), then ues have been computed as if all the computation had been
the intervals (a, a0 ) and (b0 , b) are added to the queue (for performed sequentially in a single processor (see [6]). Cali-
further processing) if: (1) the score of the current match is bration and fusion costs have been neglected.
greater than a given threshold (in this evaluation, 0.85); (2) The contrastive systems 2, 3 and 4 (c2, c3 and c4) use the
the interval is long enough (in this evaluation, half the query BUT decoders for Czech, Hungarian and Russian, respec-
length); and (3) the number of matches (already computed tively. The contrastive system 1 (c1) uses the concatenation
+ pendant) is less than a given maximum (in this evaluation, of phone posteriors from the three decoders as features (and
7). Finally, the list of matches for each query is truncated to the average of non-speech posteriors for SAD). The primary
the N with the highest scores (in this evaluation, N = 1000). system (p), which is the fusion of the four contrastive sys-
Under the extended (multiple examples) condition, only tems, increases MTWV in 5 absolute points (15% relative)
the examples passing SAD filtering (i.e. with enough speech with regard to the best contrastive (c1). In all cases, cali-
samples) are considered for each query. The longest example bration and fusion parameters have been estimated on the
is taken as reference and DTW-aligned to the other avail- development set. Late submissions fixed a bug in the fusion
able examples. Finally, the vectors aligned at each frame script (which did not count missed detections), thus leading
are averaged and a single average example is obtained and to better calibrated systems. Note also that a 15% rela-
processed as in the required (single example) condition. tive MTWV increase (nearly 4 absolute points) is obtained
by using multiple examples under the approach described
2.4 Score calibration and fusion above (system c2-late).
System scores are transformed according to [1], which is an 4. REFERENCES
[1] A. Abad, L. J. Rodriguez Fuentes, M. Penagarikano, A. Varona,
adaptation of the discriminative calibration/fusion approach M. Diez, and G. Bordel. On the calibration and fusion of
commonly applied in speaker and language recognition. heterogeneous spoken term detection systems. In Interspeech
First, the so-called q-norm (query normalization) is ap- 2013, Lyon, France, August 25-29 2013.
plied, so that zero-mean and unit-variance scores are ob- [2] X. Anguera, F. Metze, A. Buzo, I. Szoke, and L.-J.
Rodriguez-Fuentes. The Spoken Web Search Task. In MediaEval
tained per query. Then, if n different systems are fused, 2013 Workshop, Barcelona, Spain, October 18-19 2013.
detections are aligned so that only those supported by n/2 [3] N. Brümmer and E. de Villiers. The BOSARIS Toolkit User
or more systems are retained for further processing (this is Guide: Theory, Algorithms and Code for Binary Classifier Score
Processing. Technical report, 2011.
known as majority voting validation). Let us consider one https://sites.google.com/site/bosaristoolkit/.
of such validated detections, corresponding to a query q; if a [4] MediaEval Benchmarking Initiative for Multimedia Evaluation.
system A does not provide a score for it, we use instead the The 2013 Spoken Web Search Task, June 2013.
minimum score that A has output for q. The same value http://www.multimediaeval.org/mediaeval2013/sws2013/.
[5] NIST. The Spoken Term Detection (STD) 2006 Evaluation
is assigned to missed detections and non-target trials. In Plan, September 2006.
this way, a complete set of scores is prepared, which besides http://www.itl.nist.gov/iad/mig/tests/std/2006/.
the ground truth (target/non-target labels) can be used to [6] L.-J. Rodriguez-Fuentes and M. Penagarikano. MediaEval 2013
discriminatively estimate a linear transformation that pro- Spoken Web Search Task: System Performance Measures.
Technical report, GTTS, UPV/EHU, May 2013.
duces well-calibrated scores that can be linearly combined http://gtts.ehu.es/gtts/NT/fulltext/rodriguezmediaeval13.pdf.
to get fused scores. Under this approach, the Bayes opti- [7] P. Schwarz. Phoneme recognition based on long temporal
mal threshold —given by the effective prior (0.0148 for this context. PhD thesis, FIT, BUT, Brno, Czech Republic, 2008.