We describe our experiments for the Similar Segments in Social Speech Task at MediaEval 2013 Benchmark. We mainly focus on segmentation of the recordings into shorter passages on which we apply standard retrieval techniques. We experiment with machine-learning-based segmentation employing textual (word n-grams, tag n-grams, letter cases, lexical cohesion, etc.) and prosodic features (silence) and compare the results with those obtained by regular segmentation.
The main aim of the Similar Segments in Social Speech
Task is to nd segments similar to the given ones (query
segments) in the collection of audio-visual recordings containing
English dialogues of a university student community. In
addition to the human and automatic (ASR) transcripts (both
transcripts are given separately for each speaker), the
collection also contains prosodic features and metadata. The
training data consists of segments manually assigned to
similarity sets of the query segments. The details of the task
and data are described in the task description [
In our experiments, the queries are created from the
human transcripts of the query segments. The recordings are
segmented into overlapping passages (identi ed by their
starting and ending times) which are then indexed using the
Terrier IR Platform [
For the indexing, we use the default settings, which
outperformed our most successful setting from previous
experiments in the Search and Hyperlinking MediaEval
Benchmark [
As both transcripts are given in separated tracks for each speaker, we join these tracks into a single one. In the human transcripts, we sort sentences from both transcripts according to their beginnings to acquire single sequential transcript. Similarly, we sort the speakers' segments given in the ASR transcripts. While in the ASR transcripts the exact playback time is given for each word, in the human transcripts such information is available only on sentence level and therefore we approximate it by assuming equal duration of words in a sentence. 2.1
The query segments are speci ed by their starting and ending time. The queries are constructed by including all words lying within the boundaries of the query segment in both tracks.
We tried to expand the queries by adding words appearing in the vicinity of the query segment (allowing 5, 10, 15, 20, 30, and 60 seconds) but none of these experiments improved the results.
We also attempted to generate the queries from both the human and ASR transcripts and apply them to search in both types of transcripts. The queries created from the human transcripts achieved higher scores when applied on both the human and ASR transcripts, therefore they are used in the experiments presented in this paper. 2.2
In this work, we mainly focus on segmentation of the
recordings, which appears to be crucial for segment retrieval
[
In regular segmentation, the recordings are divided into
equilong segments of 50 seconds (which is approximately
equal to the average segment length in the collection). The
shift between the segments (and the overlap) is also regular,
set to 25 seconds, since according to our experience from
the 2012 Search and Hyperlinking task, the shift of 10 to 30
seconds achieves optimal results [
In the rst ML approach, we identify segment boundaries
using classi cation trees [
In the second ML approach, we apply a similar process to detect beginnings of segments which are then set to be 50
beginings ends REG REG ML REG ML ML Normalized
SUR 0.57 0.65 0.59 F-measure 0.58 0.67 0.61
beginings ends
REG REG ML REG ML ML 0.90 0.72 0.67 seconds long (naturally, the segments can overlap). In this case, we aim at higher recall of the decision process to nd all possible segment beginnings, but still keep the number of created segments reasonable. We set the weight of the segment boundary misclassi ed as segment continuation in the loss matrix to 61, the weight of the segment continuation misclassi ed as segment boundary to 1, and the complexity parameter to 0.
For comparison, the classi cation models trained and tuned on the human transcripts are also applied on the ASR transcripts despite their mutual inconsistency. The transcripts di er in the length of silence (which is in human transcripts only approximated as the duration between the imprecise word beginnings), tokenization, and letter capitalization. Therefore, our future plans include to train the classi cation model on the ASR transcripts too. 2.3
Our classi cation model exploits the following features:
cue words and cue tags, letter cases, length of the silence
before the word, division given in transcripts, and the output
of the TextTiling algorithm [
The cue words are the words that appear frequently at
the segment boundary and often do not carry special
meaning. Based on the training data, we have identi ed words
which frequently stand at the segment boundary and words
which are the most informative for the segment boundary
(the mutual information between these words and the
segment boundary is high). We have also de ned our own set
of words which might occur at such boundary and created
sets for unigrams, bigrams and trigrams, for words and tags
(obtained by Featurama tagger [
As the TextTiling algorithm is based on calculating similarity between adjacent regions, utilizing its output, we can also employ lexical cohesion into our decision process.
We employ three automatic evaluation measures:
Normalized Searcher Utility Ratio (SUR), Normalized Recall, and
the F-measure (for details, see the task description [
In the experiment utilizing human transcripts, the MLbased segmentation outperforms the regular segmentation. However in the experiments with the ASR transcripts, the regular segmentation wins. In both cases, ML-based segmentation searching for segment beginnings outperforms MLsegmentation searching for entire segments.
In the overall results, the ASR transcripts surprisingly outperform human transcripts. This is probably caused by the approximation of word timing and duration in the human transcripts { in the ASR transcripts, we are able to determine precise segment beginning and end times but the times in the human transcripts are inaccurate. 4.
The overall best result is achieved using regular segmentation on the ASR transcripts. For the human transcripts, however, the proposed ML-based segmentation outperformed the regular segmentation, which is very promising and we will attempt to project this results into experiments using the ASR transcripts. In our future work, we would also like to employ a joint model for identi cation of both segment beginnings and the segment ends. 5. 6.
This research is supported by the Charles University Grant Agency (GA UK n. 920913) and the Czech Science Foundation (grant n. P103/12/G084).