This paper describes an experimental investigation of data fusion techniques for spoken document retrieval. The e®ectiveness of retrievals solely based on the outputs from automatic speech recognition (ASR) is subject to the recognition errors introduced by the ASR process. This is especially true for retrievals on Malach test collection, whose ASR outputs have average word error rate (WER) of 35explored data fusion techniques for integrating the manually generated metadata information, which is provided for every Malach document, with the ASR outputs. We concentrated our e®ort on the post-search data fusion techniques, where multiple retrieval results using automatic generated outputs or human metadata were combined. Our initial studies indicated that a simple unweighted combination method (i.e., CombMNZ) that had demonstrated to be useful in written text retrieval environment only generated signi¯cant 38in retrieval e®ectiveness (measured by Mean Average Precision) for our task by comparing to a simple retrieval baseline where all manual metadata and ASR outputs are put together. This motivated us to explore a more elaborated weighted data fusion model, where the weights are associated with each retrieval results, and can be speci¯ed by the users in advance. We also explored multiple iteration of data fusion in our weighted fusion model, and obtained further improvement at 2nd iteration. In total, our best run on data fusion obtained 31baseline, and 4which is a signi¯cant di®erence.
technologies. However, comparing to the studies performed on text base documents, the achievements on retrieving spoken documents are still far less. Recent remarkable advances in the automatic recognition of spontaneous conversational speech makes it even urgent to study e®ective spoken document retrieval techniques. This is the reason that we participated CLEF Spoken Document Retrieval (SDR) track, and our goal is to leverage technologies developed for text retrieval into retrieving spoken documents.
Retrieving spoken documents from Malach test collection, a test collection developed by
University of Maryland for retrieving spontaneous conversational speech [
Our major interests for this year's experiments, however, lie on the several forms of human generated metadata associated with the spoken documents. For each document, there are a list of person names mentioned in the documents, the human assigned thesaurus keywords and a brief summary in 2-3 sentences written by the human catalogers during their cataloging process.
We view ASR outputs and human generated metadata as two types of information that are complimentary of each other in retrieval process. On the one hand, ASR outputs provide full and detailed information about the content of the documents, which often could not be totally covered by human generated data. On the other hand, human generated metadata provide focused, humanprocessed, and high quality information that can be relied on for the accuracy of the retrieval. If we can develop a reliable retrieval method that can combine both information into the retrieval process in such way that their complimentary features can be fully explored, the achieved retrieval e®ectiveness would be greatly superior than that of any one of them. This is the goal of our studies in this year's CLEF-SDR experiments, and two derived research questions are: 1. how the manual metadata and ASR outputs in Malach collection can be integrated for improving the retrieval e®ectiveness of the ¯nal run? 2. what are the parameters that we can utilize to make the data fusion techniques more e®ective for our task?
In the rest of this report, we will ¯rstly review some existing data fusion methods in Section 2; discuss in detail the experiment settings in Section 3; then talk about the fusion techniques we developed for the CLEF-SDR experiments in Section 4. Finally we will discuss some further studies in Section 5. 2
In the literature, the techniques for combining multiple queries, document representations or
retrieval results is called \data fusion" [
In post-search data fusion approaches, to properly merge retrieval results that are commonly
ranklist of documents, the score associated with each document has to be normalized within that
list. A often used normalization scheme (see Equation (1)) utilizes the maximum and minimum
scores of a ranklist (i.e., M axScore and M inScore) in the normalization process [
U nnormalizedScore ¡ M inScore N ormalizedScore = (1)
M axScore ¡ M inScore
Fox and Shaw [
minimum of all scores of a document maximum of all scores of a document summation of all scores of a document
Malach Test Collection
Malach Test Collection was developed by University of Maryland as part of their e®ort in Malach
project [
There are total 63 search topics, 38 of which were available for training, and 25 were held as the testing topics. Each topic is designed in TREC style, which has a title, a description and a narrative (see Figure 3). The topics are available in English, Spanish, Czech, German and French, however, we only used English topics for our studies. 3.2
Indri search engine Our search engine was Indri 1.0, which was a collaboration e®ort between the University of Massachusetts and Carnegie Mellon University [6]. Its retrieval model is a combination of language model and inference network. We chose it not only because of its state-of-art retrieval e®ectiveness, but also because it is an open source system, on which we can easily integrated our modi¯cations. Its powerful query syntax is another attraction to us, since we want to specify which index ¯elds should a retrieval be based on for our studies of manual metadata only or automatic data only searches. 3.3
Measures To study the retrieval results in as wide scenarios as possible, instead of choosing on single measure, we employed a set of evaluation measures, each of which tells us some aspect of the retrieval e®ectiveness of the search results: ² mean average precision (MAP), the measure aim at giving an emphasis view of precision in a ranklist. Since the ranks of the relevant documents are considered in the measure, this measure gives a reasonable overview of the quality of the ranklist for a given retrieval topic. <ASRTEXT2004A> oh i'll you know are yeah yeah yeah yeah yeah yeah yeah the very why don't we start with you saying anything in your about grandparents great grandparents well as a small child i remember only one of my grandfathers and his wife his second wife he was selling flour and the type of business it was he didn't even have a store he just a few sacks of different flour and the entrance of an apartment building and people would pass by everyday and buy a chela but two killers of flour we have to remember related times were there was no already baked bread so people had to baked her own bread all the time for some strange reason i do remember fresh rolls where everyone would buy every day but not the bread so that was the business that's how he made a living where was this was the name of the town it wasn't shammay dish he ours is we be and why i as i know in southern poland and alisa are close to her patient mountains it was rather mid sized town and uhhuh i was and the only child and the family i had a governess who was with me all their long from the time i got up until i went to sleep she washed me practice piano she took me to ballet lessons she took me skiing and skating wherever there was else that i was doing being non reach higher out i needed other children to players and the governors were always follow me and stay with me while ours twang that i was a rotten spoiled care from work to do family the youngest and the large large family and everyone was door in the army </ASRTEXT2004A> <AUTOKEYWORD2004A1> family life | family homes | grandparents | living under false identity | forced labor in the ghettos | extended family members | family businesses | clandestine economic activities | smuggling | barter | refugee labor experiences | life under communism | communities, Jewish | cultural and social activities | postwar visits home | flight from Belgium | Poland 1918 (November 11) - 1935 (May 12) | Poland 1935 (May 13) - 1939 (August 31) | Poland 1939 (September 1) - 1945 (May 7) | Tashkent (Uzbekistan, USSR) </AUTOKEYWORD2004A1> ² Precision at 10 (P10) is a useful measure to examine how much relevant documents are there in the ¯rst result screen, which is often the only results viewed by a user. ² R-Precision (R-PREC) emphases on precision but also avoids the arti¯cial cut-o® e®ect imposed by pre-de¯ned cut-o® point, like in P10. The \R" varies according to the number of relevant documents of a given topic. ² Average Recall at top 1000 returned documents. It tells the quality of the ranklist from the point of recall. 3.4
Baselines We established three baselines for evaluating our methods. The ¯rst two represent the scenario that no data fusion is performed. We selected a run on ASRTEXT 2004 as the baseline for searching on ASR output, since ASRTEXT 2004 is the better one among the two ASR outputs. This baseline is referred as \asr04", and is treated as the lower baseline. Ideally, we should use <INTERVIEWDATA> Sidonia Lax | 1927 | Shaindl | Lewin | Sydzia </INTERVIEWDATA> <NAME></NAME> <MANUALKEYWORD> grandfathers | socioeconomic status | Przemysl (Poland) | Poland 1926 (May 12) - 1935 (May 12) | Poland 1935 (May 13) - 1939 (August 31) | cultural and social activities </MANUALKEYWORD> <SUMMARY>SL remembers her grandfather. She talks about her town. SL recalls her family's socioeconomic status and her social and cultural activities.</SUMMARY> <top> <num> 1148 <title> Jewish resistance in Europe <desc> Provide testimonies or describe actions of Jewish resistance in Europe before and during the war. <narr> The relevant material should describe actions of only- or mostly Jewish resistance in Europe. Both individual and group-based actions are relevant. Type of actions may include survival (fleeing, hiding, saving children), testifying (alerting the outside world, writing, hiding testimonies), fighting (partisans, uprising, political security) Information about undifferentiated resistance groups is not relevant. </top> the search on manual transcripts as the upper baseline. Since Malach collection does not provide manual transcripts, we used all manually generated data in the segments as the surrogate. It acts as a proximate upper bound baseline, and is referred as \manual-only" baseline. We did not apply blind relevance feedback (BRF) over \manual-only" run since our explorations using Indri's BRF function over \manual-only" baseline generated inferior results. The third baseline is a search on all manual and ASR outputs being put together as if they are di®erent parts of the same document. This represents the simplest data fusion method, and is referred \simple-fusion" baseline.
The search e®ectiveness of the three baselines are presented in Table 2
runs manual-only
asr04 simple-fusion
MAP 0.2312 0.0693 0.1842
P10 0.4603 0.2111 0.3635
4 4.1
Data Fusion with CombMNZ
The ¯rst data fusion method studied in the CLEF-SDR experiments was our implementation of
CombMNZ method since Lee demonstrated its superior over the other three methods[
A close exam of the retrieval runs in Table 3 shows that the retrieval e®ectiveness of \manualonly" run is greatly higher than that of the two automatic runs. For example, the MAP of \manual-only" increases about 200% over \asr04", the better one of the two automatic runs. Therefore, it makes no sense to assume that their contribution to the ¯nal fused ranklist is the same, which is the assumption in CombMNZ model. We need a data fusion model that considers the retrieval di®erence.
runs manual-only
asr04 autowa1
The failure of CombMNZ on our data fusion task motivated us to explore a weighted scheme for data fusion based on CombMNZ. A natural place to insert a weight in CombMNZ is to assign a weight of belief for each retrieval run as it is possible to obtain such evidence or belief. In our weighted CombMNZ model (called WCombMNZ model), such belief is used in calculation of the ¯nal combined score for a document (see Equation 2).
W CombM N Zi = n X (wj £ N ormalizedScorei;j ) £ n j=1 (2) where wj is a prede¯ned weight associated with a search result to be combined, n is the number of nonzero scores of document i, and the N ormalizedScorei;j is calculated using Equation 1.
runs manual-only
asr04 autowa1
MAP 0.1494 0.0525 0.0239
Various methods can be used to obtain the weight wj for a given ranklist j. In this year's experiment, we ¯rstly used the retrieval e®ectiveness of those pre-combined runs on the 38 training topics as the weights (the details of the pre-fused runs on the training topics are in Table 4). Therefore, we have four di®erent \WCombMNZ" runs (see Table 5), and their retrieval e®ectiveness evaluated by the four measures are in Table 6.
use the MAP values of the three runs on training topics as the weights use the R-PREC values of the three runs on training topics as the weights use the P10 values of the three runs on training topics as the weights use the Avg-Recall values of the three runs on training topics as the weights All four WCombMNZ runs are signi¯cant higher than the non-weighted \CombMNZ" run (paired T-test with P < 0.05) when looking at MAP, R-PREC, and P10 as the measures. However, they are still signi¯cant lower than the \manual-only" run using the same measures.
Since the weights of \WCombMNZ-1" generated the best MAP, R-PREC and P10 results, we used those weights to help us explored the e®ect of di®erent combinations of the weights. As the di®erence of the retrieval e®ectiveness between \manual-only" run and the two automatic runs is signi¯cant higher than that between the two automatic runs, we ¯rst explored the change of ratio between the weight of \manual-only" run and that of the \asr-04" and \autowa1" runs. The ratio we tested were 2:1 (that is the weight for \manual-only" run is 2, and the weights for the two automatic runs were both assigned to be 1 in WCombMNZ model), 5:1, 10:1, 15:1, and up to 1000:1 (the results are presented in Table 7). We then change the weight ratio between the \asr04" and that of \autowa1" to 2:1, which is closer to the weight ratio in \WCombMNZ-1", and varied the weight ratio of the three runs from 4:2:1, 6:2:1, and up to 50:2:1. As shown in Table 7, the ratio between the weight of the \manual-only" and that of two automatic runs is the dominate factor in a®ecting the retrieval performance, and once the ratio between the manual run and the automatic runs is larger than 10, there is not much di®erence in the retrieval e®ectiveness evaluated by all measures. However, still none of the fused runs achieves better MAP, R-PREC, and P10 than \manual-only", although they are much closer to the performance of the \manualonly" than the two automatic runs, and at the same time, many of they have achieved signi¯cant better Avg-Recall than the \manual-only" run. 4.3
Multiple Iteration of Data Fusion One exploration within the data fusion framework is `does multiple iterations of data fusion make sense?" To answer this, we conducted several experiments in WCombMNZ model. Total ¯ve retrieval runs were used in the second iteration of data fusion. We kept the \manual-only" run since it is the best run so far, and we used the four runs listed in Table 6 \WCombMNZ-1" to \WCombMNZ-4". We used the similar scheme to vary the weight ratios among the runs, and we also set all weights to 1 to make WCombMNZ model fall back to CombMNZ so that we can study CombMNZ too. The ratio \2-1" in Table 8 means that the weight for \manual-only" is 2, and that for the other four runs is 1.
As shown in Table 8, the \manual-only", in our current retrieval setting, still deserves more weights than the other runs, and the best retrieval results are achieved with the ratio around 10:1. Statistical tests (paired T-test) between the results of the 2nd round fusion runs and that of the \manual-only" run demonstrate that all the 2-iteration fusion data generated signi¯cant improvement on average recall, but only the runs with ratio above 10:1 generated signi¯cant improvement on P10, and only runs with ratio 10:1 and 15:1 generated signi¯cant improvement on MAP. No signi¯cant improvement can be achieved on R-PREC.
We then generated various 3rd round fusion runs using the similar scheme, which include the \manual-only" run, and the four 2nd round runs with the ratio 5:1, 10:1, 15:1 and 20:1. The results are shown in Table 8. None of the 3rd round runs could generate statistical signi¯cant improvement over the 2nd round runs. It seems that fusions with iteration more than 2 does not justify the extra costs involved comparing to the 2nd round fusion runs.
Our multiple iteration experiments tell us that it is usually di±cult to obtain a better fusion results over the best pre-fusion run when the retrieval e®ectiveness of the pre-fusion runs are greatly di®erent to each other. The fusion experiment on \manual-only" and the other automatic runs is an example of such fusion. However, signi¯cant improvement over the best pre-fusion run could be achieved via multiple iterations of fusion. For example, we achieved signi¯cant improvement over the best pre-fusion run in two iterations. Of course, we need further experiments to test the general e®ectiveness of the multiple iteration fusion.
In this paper, we have described an experimental investigation of data fusion techniques for spoken
document retrieval. Because of the various characteristics of the documents in the Malach test
collection, retrieval solely based on the outputs from automatic speech recognition (ASR) is well
below retrievals on manual generated data. To overcome the problem, we have explored data fusion
techniques for integrating the manually generated metadata information with the ASR outputs.
We concentrated on the post-search fusion approach, and explored weighted CombMNZ model with
di®erent weight ratios and multiple iterations. Our initial results indicate that a simple unweighted
combination method that has demonstrated to be useful in written retrieval environment [
Our future work include further experiments on the general e®ectiveness of the multiple iteration fusion, and another future work is to explore the usage of WCombMNZ in other retrieval tasks, where multiple retrieval results can be obtained from one retrieval engine, or even di®erent engines. The third further study we want to work on is to answer the question what is the minimum human generated data to ASR output if the goal is to combine the human generated data with the ASR output to achieve a comparable retrieval e®ectiveness to a retrieval on manual transcripts.
The authors would like to thank Doug Oard, Gareth Jones and Ryen White for their tireless e®orts to coordinate CLEF-SDR. audio collections, [2] N.J. Belikn, C. Cool, W.B. Croft, and J.P. Callan. The e®ect of multiple query representations on information retrieval system performance. In Proceeding of ACM-SIGIR'93, pages 339{346, 1993. [6] http://newhaven.lti.cs.cmu.edu/indri/. [9] Luis M. Rocha. Combination of evidence in recommendation systems characterized by distance functions. In Proceedings of the 2002 World Congress on Computational Intelligence, FUZZIEEE'02, pages 203{208. IEEE Press, 2002.