Prosody is important in spoken language, and especially in dialog, but its utility for search in dialog archives has remained an open question. Using prosody-based measures of similarity, which also roughly correlate with dialog-activity similarity and topic similarity, we built support for “retrieve more like this” searches. Performance on the Similar Segments in Social Speech Task at MediaEval 2013 was well above baseline, showing the value of prosody for search.
In most cases people searching in audio are probably not really interested in finding words. What people want is often information of some type, which may be characterized in part by dialog process or activity, for example recommending, answering a question, agreeing, forming a decision, telling life stories, making plans, hearing surprising statements, giving advice, explaining, and so on. In dialog, such activities and topics often are associated with characteristic prosodic features and patterns.
Our basic idea is to use a vector-space model of dialog
activity, where each moment in time maps to a point in this
space. This representation is obtained by applying Principal
Component Analysis to 78 local prosodic features computed
every 10ms calculated over a 6 second sliding window [
In previous work [
The similar segments task is based on regions, but the dialog-space model is based on timepoints. For simplicity, the middle point of the query region is used as the characteristic point. The most similar (proximal) timepoints, across the entire corpus, are then found and returned, in order, as the ranked list of jump-in points.
We started with a similarity metric using simple Euclidean
distance in the vector space, as described in [
First we tried this with uniform weights, giving the “dissim” results in the tables. We then tried optimized weights, trained using linear regression, where the target was a distance of 0 if x and y were similar, and 1 if they were not similar. Thus, for example, if two selected timepoints x and y both were located in regions that had been tagged as talk about “favorite movies,” then x and y were counted as similar. If x and y shared no tags, they were counted as not similar. This is of course not ideal, since a point-pair might be similar even if not belonging to regions that were felt to be worth tagging. Sets of similar and non-similar timepointpairs were obtained by random sampling over the training set.
For sampling we experimented with various more
restrictive definitions of similar. One type of constraint was to
require agreement by at least some number of annotators in
order to consider a timepoint pair as similar. For this the
label names, were ignored (as always), and so the annotators
might have considered the points to be similar in die↵rent
ways entirely. The second type of constraint relied on the
utility values (“weights”) assigned by the annotators to their
tags, higher the more informative and cohesive they thought
the tagset was. For example, in one sampling we included
only pairs whose connecting tag was rated 3, excluding those
rated 0, 1, or 2 [
We also experimented with pruning the dimensions, using model
Expl. raw s.u.r model
Dissim. all+ good+ all-p+ good-p+ raw s.u.r naive raw prec. recall norm.
s.u.r norm. recall two feature selection methods. This was prompted by the observation that linear regression consistently gave negative weights to some of the dimensions, for example 67, which, when we listened to it, seemed to encode the di↵erence between calm, indie↵rent speech and energetic explaining. The first method was to try to leave a dimension out of the model (set its weight to zero), and if that improved performance on a held-out subset of the training data, to drop it from the set. This was iterated, typically resulting in dropping about a third of the dimensions. The second approach was to simply drop any dimension to which regression assigned a negative weight.
The tables show the results1. for the four models which performed best on the training set and four reference models: the baseline, where the jump in points for each query are randomly selected; a tagset-exploiting model, where jump in points are found by considering tags by other annotators with regions that overlap the query region; the Euclidean distance model; and a model based on uniform-weight dissimalarity, that is, like distance but using absolute-value instead of squared di↵erences. We used the tagset-exploiting model as a likely upper bound on performance, as it is akin to how a second human might themselves perform the search task. For the best models, performance is far above baseline, showing that information retrieval can indeed benefit by using prosodic information.
These results are, however, weaker than those that can be obtained by using lexical features. Perhaps in this corpus topical similarity was more relevent then functional similarity, and perhaps lexical models are better for topic similarity. Thus prosodic models may still be of value, as is, for languages for which speech recognizers are not available or perform poorly. We further conjecture that the prosody is capturing dimensions of similarity not seen in lexical similarity, and therefore that a combined model could do even 1From the point of view of the competition, these results are all unocial, since the authors, being also the competition organizers, had privileged access to the data. better. Exploring this is a priority for future research.
The ee↵cts of using higher quality training data varied with the testset: on the training set, using the good quality set gave the best performance, but on the test set the model trained using all the data performed best. Pruning was generally beneficial, with dropping dimensions with negative weights being the most useful, with some additional benefit from also selectively dropping dimensions.
Looking at robustness to changes in the data, the picture is clouded by the fact that the test set was harder, in terms of recall (because the target regions, like all regions in this set, tended to be shorter and thus harder to find). Nevertheless, on the training set the best model’s performance was still far above baseline, showing a degree of generalizability.
Although the potential utility of prosody for search has
been long discussed [
We thank the National Science Foundation for support via a REU supplement to Award IIS-0914868, and Olac Fuentes.