This paper describes the participation of the University of Washington Computational Linguistics Laboratory (UWCL) in the Similar Segments in Social Speech task at MediaEval 2013. Participants in this task develop systems that, given a span of speech from a recorded conversation, aim to identify all and only highly similar regions in other recordings. As this was a new task for this year, the goal was to establish a baseline and a framework for future experimentation. The approach aimed to address two particular challenges posed by the task: the lack of prior segmentation of the conversations and the limited material provided by a single brief example segment. To this end, the system employed a query-by-example information retrieval framework using passage retrieval to identify segments dynamically and query expansion to support robust retrieval. Query expansion provided substantial gains when applied to both manual and automatic transcriptions; results using automatic transcripts were competitive with those using manual ones.
Recent years have seen a dramatic growth in the use of social media as well as the sharing of multimedia materials in venues ranging from Facebook to YouTube. Users increasingly share personal content in these social media settings. However, exible search and targeted access to this material remains challenging, relying largely on manually assigned metadata, such as titles and tags, to identify content, rather than directly indexing the content of these multimedia materials themselves. Furthermore, not only is extraction of content from multimedia streams more challenging than from text, but skimming or browsing in a media stream is slower and more di cult than in text.
The Similar Segments in Social Speech (SSSS) Task
developed for MediaEval 2013 aims to overcome these limitations
in information access. As described in the task overview
paper [
In contrast to the signi cant prior work on spoken
document retrieval [
The remainder of the paper is organized as follows. Section 2 presents key challenges in the task and UWCL's approach to addressing them. Section 3 describes the experimental con guration, o cial runs, and results, along with discussion. Section 4 concludes with plans for future work. 2.
The SSSS task posed a wide range of interesting challenges. These issues included:
Task modeling: would the task be best modelled as retrieval, clustering, ranking, or something else? Sources of similarity: How should similarity be assessed |through lexical or acoustic information or some combination? Segmentation: how should segments be identi ed |via xed segmentation, agglomeration, or other means? Generalization: given a simple example segment, how can we overcome di erences across speakers? Transcription: what is the e ect of transcription type |manual or automatic |on task e ectiveness? For this challenging new task, UWCL's approach built on and extended existing methodologies. In particular the approach adopts an information retrieval perspective, using the text transcriptions of the spoken data. From this perspective, system design focused on the latter three issues identied above: segmentation, generalization, and transcription.
Segmentation Much prior work on spoken document
retrieval has either provided a gold standard segmentation or
assumed its existence. In contrast, the SSSS task does not
provide a segmentation, and one could imagine di erent
segmentations based on di erent notions of similarity. Thus,
the strategy aimed to create segments and jump-in points
sensitive to the similarity measure and to the exemplar
segment. The UWCL system exploits passage retrieval [
Generalization Di erences in lexical choice between materials being searched and the searcher's speci cation of their uwclman uwclauto uwclmanexp uwclautoexp uwclauto2exp man auto man auto auto no no yes yes yes n.a. n.a. man man auto
Exp. set
information need are a well-known issue in information
retrieval. The segments in the SSSS task, which average about
50 seconds in the training set and about 30 seconds in the
test set, are not particularly short. However, it seems likely
that variation between speakers and the broad notion of
similarity will make lexical match highly challenging. To
address this issue, the UWCL system investigates the use
of query expansion [
Transcription Both manual and automatic transcripts of the spoken data are employed. 3.1
The UWCL system employed the INDRI/LEMUR
information retrieval engine (http://www.lemurproject.org) for
indexing and retrieval with default settings [
Five o cial runs on the test data were submitted and
scored. As shown in Table 1, contrasting conditions explored
the impact of transcription (manual/automatic), query
expansion (yes/no), and expansion corpus (manual/automatic).
The o cial results are also tabulated, for the primary
metrics, Normalized Searcher Utility Ratio (NSUR) and F-measure,
as described in the task overview [
We nd that, although the baseline query formulation achieves modest e ectiveness, query expansion using pseudorelevance feedback based on a matched corpus yielded substantially increased e ectiveness. With the mismatched expansion corpus, the divergence between manual and automatic transcription led to a smaller, but still noticeable, improvement. Finally, it is interesting to note that, with suitable query expansion, a con guration based on automatic transcription greatly outperformed one using manual transcripts without query expansion and was highly competitive with one using manual transcripts with query expansion. 4.
UWCL's approach to the MediaEval 2013 SSSS task employed a text-based information retrieval approach, using passage retrieval to create segments dynamically. Automatic query expansion yielded strong improvements for both manual and automatic transcripts. While these approaches showed promise, many avenues for improvement remain. In addition to tuning retrieval factors, such as passage length and retrieval models, I plan to explore the integration of acoustic, especially acoustic-prosodic, evidence into measures of segment similarity, in addition to the lexical evidence already in use. Such measures could be particularly helpful in recognizing segments with similarity based less on topical content than on emotional or attitudinal content.
Many thanks to the task organizers and also to Steve Renals for providing the high-quality automatic transcriptions. 5.