Brown's entry to the Cross-Language Speech Retrieval (CL-SR) track at the 2007 Cross Language Evaluation Forum (CLEF)1 was based on the language model (LM) paradigm for retrieval [17]. For English, our system introduced two minor enhancements to the basic unigram: we extended Dirichlet smoothing (popular with unigram modeling) to bigrams, and we smoothed the collection LM to compensate for the small collection size. For Czech, time-constraints restricted us to using a basic unigram model, though we did apply Czech-specific stemming. While our English system performed well in the evaluation and showed the utility of our enhancements, several aspects of it were rushed and need to be addressed in future work. Our Czech system did not perform competitively but did provide us with a useful first experience in non-English retrieval.
shown Dirichlet smoothing to outperform linear interpolation for unigram modeling by varying the mixture weight per-document according to document length. We extended document-collection unigram Dirichlet smoothing to include bigram mixing as well, as described in §3.
The second extension we added was collection smoothing. In the LM paradigm, the document LM is initially estimated by maximum likelihood, meaning any query word not observed in the document is assigned zero probability. This is problematic because there are likely many words related to the document yet which do not appear in it due to its brevity (i.e. chance). Assigning zero probability to a single term would zero-out the probability assigned to the entire query string, which is likely to be a poor estimate of document relevance. Consequently, it is common practice to smooth a document’s LM with the collection LM (as a prior) to make the LM more robust. However, the collection LM is also estimated by maximum likelihood and so may also suffer from sparse data problems in the case of small collections. To investigate whether collection smoothing could help, we tried mixing the collection with larger corpora, and results show that collection smoothing substantially improved performance (§4).
Retrieval experiments were conducted in English and Czech (i.e. English queries/documents and Czech queries/documents), but due to time constraints we gave much less attention to Czech, evaluating only a unigram model without the above extensions. While our English system performed well in the evaluation, our Czech system was not competitive. §2 describes the data used in our experiments. In §3, system methodology is presented. Results are presented in §4, with additional details given in the appendix (§7). Concluding remarks appear in §5. 2
This section introduces the English and Czech data used in our retrieval experiments. Our
description is brief since this data has been previously described in detail [
The collection consists of interviews conducted by the Survivors of the Shoah Visual History
Foundation (VHF) to record the memories of Holocaust survivors, rescuers, and witnesses. In
terms of information management, cultural heritage archives like this one can be expected to
become more frequent as recording and storage technology continues to become ever more widely
accessible. Such archives are also just the tip of the iceberg in terms of the sorts of spontaneous
speech being increasingly archived: debates, meetings, classroom discussion, talk shows, telephone
conversations, online chat, etc.. In regard to information retrieval, previous work has considered
broadcast news in detail [
Compared to modern-sized retrieval collections, the VHF data set is quite small: the English collection consists of just 8,104 manually segmented interview passages to rank. However, whereas the size of the early text collections like Cranfield was limited by cost and human effort, a cultural heritage archive like this one may in fact be naturally small: in the case of VHF, there are only a limited number of people alive today with first-hand experience of the Holocaust. As for other forms of archived spontaneous speech, a particular individual may be interviewed only so many times, a course or talk shows series eventually terminates, etc. As such, we may increasingly see a practical need for effective search techniques on smaller collections such as this one; optimal methods and parameters on a terabyte collection may not yield the best performance here. Of course, one may have a broad information need and not care about which archived talkshow contained the relevant information. With regard to this scenario, the VHF collection does for spontaneous speech what early collections did for text, and likely larger collections of spontaneous speech are on the horizon.
Topics used were written in usual TREC style with three fields of increasing length: title, description, and narrative. These topics were based on actual information requests received by VHF from interested parties. Manual transcriptions of the interviews were not available in English or Czech, unfortunately, making it difficult to evaluate the impact of recognition errors on retrieval accuracy. Several variant one-best ASR transcripts were available for comparison. Both interviewer and interviewee were recorded on the same microphone/channel; while dialogue from the interviewee certainly dominates the interview, interviewer questions and comments are seen mixed into the same transcript. 2.1
Each segment contained a two to three sentence manual summary as well as a set of manually
assigned keywords following a careful ontology developed by VHF. It is worth noting that these
“manual” segments are far shorter than text and spoken documents commonly retrieved today;
they are perhaps most akin to scientific paper abstracts in terms of previous retrieval experiments.
Each interviewee also filled out a pre-interview questionnaire with some additional information
that could also be used in the “manual” retrieval condition. The ontology developed by VHF
could also be exploited for synonym expansion, etc. To date, the interview data and ontology
have rarely been used [
Czech interviews were not manually segmented as with English. Instead, track organizers supplied a set of scripts allowing the interviews to be automatically segmented for a fixed duration and overlap size with neighboring segments. They also supplied one such “quickstart” segmentation generated by their scripts, which had 11,377 segments of three-minute passages in which the first and last minute overlapped the neighboring segments. The idea of the scripts was to allow participants to explore the effects of various segmentations on retrieval accuracy. Unfortunately, problems with the scripts were found during the evaluation and led the organizers to suggest teams use the quickstart segments and avoid use of the scripts.
No manual summary or keywords available were available for the interviews. There also were no automatically recognized keywords; those used in the 2006 evaluation were removed for 2007 due to an unspecified problem with them. 3
As described in the introduction, our retrieval system was based on the language model (LM)
paradigm for retrieval [
P (Q|D) =
Y P (w|D) w∈Q (1) One challenge of the LM paradigm is estimating the parameters of the underlying LMs given the brevity of the observed documents; if one simply takes the maximum likelihood estimate (MLE), a single query term unobserved in the document would zero-out the entire probability of observing the query given the document, making the entire framework exceedingly fragile. Instead one commonly employs smoothing to discount the probability mass assigned to observed terms and reserve some probability mass for all unseen terms. Previous work has shown Dirichlet smoothing of the form below to be most effective in practice [26]. This form of smoothing adds a hyper-parameter number of pseudo-counts distributed fractionally according to the prior collection model. N is the total number of words in the document.
P (w|D) =
CD(w) + μP (w|C)
N + μ
The LM approach has been shown to have a strong theoretical connection to TF-IDF [26] and
perform comparably to vector similarity and probabilistic approaches in practice [
A well-known improvement to the basic unigram LM is to model word pairs in a unigrambigram mixture model [23]. While previous work linearly mixed bigram and unigram models using a fixed mixture weight over all documents, this misses the key idea of Dirichlet smoothing that longer documents provide more evidence for MLE and so should require less smoothing. For this reason, we extended document-collection unigram Dirichlet smoothing equation above to include bigram mixing. The Dirichlet smoothed bigram is given by:
P (wi|wi−1, D) =
CD(wi−1, wi) + μ1P (wi|wi−1, C)
CD(wi−1) + μ1 and can be mixed with the unigram by adding an additional hyper-parameter, μ2.
P (wi|wi−1, D) =
CD(wi−1, wi) + μ1P (wi|wi−1, C) + μ2P (w|D)
CD(wi−1) + μ1 + μ2 This leaves three hyper-parameters for tuning: μ, μ1, and μ2.
Our second extension to the basic LM approach was to incorporate collection smoothing. While
it is common practice to smooth a document’s LM with the collection LM (as a prior) to make
the LM more robust, as shown above, the collection LM is also estimated by maximum likelihood
and so may also suffer from sparse data problems in the case of small collections. To investigate
whether collections smoothing could help, we tried linearly mixing the collection with two larger
text corpora: 40,000 sentences from the Wall Street Journal as found in the Penn Treebank [
The importance of sentence boundaries is that bigram statistics were not collected across them,
which also differs from previous work. Phrase-based statistics can be expected to perform best
when not collected or applied across sentential boundaries, especially as phrase length increases.
This issue has largely been ignored in previous work since sentences tend to be rather long in text
(maybe around 30 words in a typical newspaper), and so error introduced for short phrase statistics
by approximating the entire document as a single sentence is somewhat limited. Our attention to
sentence boundaries stems primarily from the fact that we are eventually interested in comparing
the efficacy of bigrams with syntactic bi-lexical dependencies induced from sentences, effectively
revisiting previous work [
We also applied pseudo-relevance feedback and found it significantly improved our results. Experimental parameters included the number of documents to use for feedback, and a multiplicative scaler for the original query counts. We never performed more than one iteration of feedback, nor did we try restricting term harvesting to a subset of terms rather than the entire document. This feedback scheme was developed in a short amount of time and leaves much room for improvement.
As indicated in the introduction, time constraints caused us to give far less attention to Czech
than to English. We did not model bigrams or perform collection smoothing, and we used as
the off-the-shelf Indri system as our model (with one minor change to its parsing code to not
split tokens on accented characters). Indri actually implements a multi-Bernoulli model rather
than a true unigram model, but results are roughly comparable [
This section describes our evaluation on English and Czech, including evaluation framework, parameter settings, and results. 4.1
The same 63 training topics and 33 evaluation topics used in 2006 were used again in 2007. Relevance assessments for the test set were distributed to participants following the 2006 evaluation, so some participants may have run error analysis experiments on the test data then without realizing it would be reused this year. Since relevant assessments for all queries were available, 2007 track organizers clearly indicated which queries comprised the test set to help ensure no one accidentally tuned on it this year. In order to preserve the test set for future use, we have performed no error analysis of our results on it. While working on the development set, we found a small problem with the distributed relevance assessments (across all topics) that some documents marked as relevant were not actually in the distributed collection, but these cases were automatically filtered out without compromising the assessments.
For the “manual” case, we used the manual summaries and keywords; we made no use preinterview questionnaire information, or the VHF ontology. For “automatic” retrieval, we used the ASR2006B transcripts and both sets of automatic keywords. As with the manual case, we found use of the automatic keywords to improve retrieval, although we did not evaluate use of one set of keywords versus the other. We also did not evaluate other versions of the automatic transcripts, and this may be worth revisiting since some teams have reported better retrieval accuracy with the 2004 transcripts. Following previous work [26], μ was fixed at 2000 for all runs, manual and automatic. In hindsight, it would have been interesting to explore alternative μ settings since the manual and automatic “documents” used here are rather different from one another, and both differ significantly from previous retrieval experiments which varied μ on text collections. For both manual and automatic runs, best performance was almost always seen with μ1 set to 1, and so this parameter was also largely fixed. Additional detail on parameter settings used and corresponding results on the development set can be found in the appendix (§7).
Results in Table 1 show performance of our five submitted runs on development and test sets; queries used were: title-only (T), title and description (TD), and title, description, and narrative (TDN). Representative strong results achieved the CL-SR tracks are also shown, though it should be noted that our results on the development set correspond to optimal tuning on those queries whereas the CL-SR’05 numbers do not. Retrieval accuracy was measured using mean-average Auto
TDN TD T TDN TD precision (MAP) as reported by the trec_eval tool version 8.12.
After submitting our official runs, we discovered a system bug which caused some query topic fields to be prematurely truncated. The bug did not affect the development set but did affect the narrative field of three test queries. After fixing the bug, we re-ran our two submissions affected by it (one manual, one automatic). We then made our only use of the test set relevance assessments to evaluate the resulting retrieval accuracy with the bug fix. The difference was substantial, and results given in the body of Table 1 show system performance with the bug fix. Without the fix, Manual-TDN on the test set was .2577 and Auto-TDN was .0831.
Regarding the impact of our enhancements to the basic unigram (bigram mixing and collections smoothing), we refer the reader to results on the development set shown in the appendix (§7). To broadly summarize, while the best unigram result was often not too far below the best bigram result, bigram results were more robust across parameter settings. Bigram statistics also appeared to have greater impact with pseudo-relevance feedback than without. As for collection smoothing, it clearly provided a substantial improvement. WSJ smoothing always helped, and NANC smoothing almost always helped, though less so in the case of pseudo-relevance feedback.
Overall, we are fairly pleased with our system’s relative performance in the evaluation, but room is certainly left for improvement. 4.2
Czech retrieval accuracy was measured using the mGAP metric and tool, which mapped retrieved
segments to replay times [
Brown’s participation in the Cross-Language Speech Retrieval (CL-SR) track at the 2007 Cross Language Evaluation Forum (CLEF) represented our group’s first effort in developing and applying an information retrieval (IR) system for human language. Our goal was to develop a strong baseline system which we could build on and compare to in future research. Our English system applied a novel form of Dirichlet bigram smoothing and showed the importance of collection smoothing
Stemmer none aggressive light none aggressive light
no yes with a small collection. This system performed well, though we expect there is still room for improvement through more tuning of existing methodology. Our Czech system applied a simple unigram model under two forms of stemming and provided us with initial experience in non-English retrieval. 6
This work was initiated at the Institute of Formal and Applied Linguistics ( U´FAL)3 at Charles University in Prague, where the first author was graciously hosted for the first half of 2007. Support for this work was provided by NSF PIRE Grant No OISE-0530118 and DARPA GALE contract HR0011-06-2-0001. Any opinions, findings, and conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the supporting agencies. [6] David Graff.
LDC95T21. [7] Mary Harper, Bonnie Dorr, John Hale, Brian Roark, Izhak Shafran, Matthew Lease, Yang Liu, Matthew Snover, Lisa Yung, Anna Krasnyanskaya, and Robin Stewart. 2005 Johns Hopkins Summer Workshop Final Report on Parsing and Spoken Structural Event Detection. [9] Matthew Lease. Natural language processing for information retrieval: the time is ripe (again).
In Proceedings of the 1st Ph.D. Workshop at the ACM Conference on Information and Knowledge Management (PIKM), 2007. To appear. [21] Amit Singhal. Modern information retrieval: A brief overview. Bulletin of the IEEE Computer
Society Technical Committee on Data Engineering. [22] Amit Singhal, Chris Buckley, and Mandar Mitra. Pivoted document length normalization.
In Proceedings of the 19th annual international ACM SIGIR conference, pages 21–29, 1996. [23] Fei Song and W. Bruce Croft. A general language model for information retrieval. In Proceedings of the eighth international conference on Information and knowledge management (CIKM), pages 316–321, 1999. [24] K. Sparck Jones, S. Walker, and S.E. Robertson. A probabilistic model of information retrieval: development and comparative experiments (parts i and ii). Information Processing and Management, 36:779–840, 2000. [25] Chengxiang Zhai. A brief review of information retrieval models. Technical report, Department of Computer Science, University of Illinois at Urbana-Champaign, 2007. [26] Chengxiang Zhai and John Lafferty. A study of smoothing methods for language models applied to information retrieval. ACM Trans. Inf. Syst., 22(2):179–214, 2004. 7
This section provides some additional detail on English retrieval experiments, including parameter settings and results on the development set. 7.1
MAP retrieval accuracy on development corpus for top 10 manual runs without pseudo-relevance feedback. μ2 unigram weight was either 500 or 100000 (the latter effectively nullifying bigram statistics). Use of manual key words in addition to manual summaries was varied. Collection smoothing was varied between: none, WSJ, and WSJ+NANC (simple concatenation). Overall, use of manual keywords is seen to help though not dramatically. Collection smoothing, in contrast, appears to be much more important.
MAP retrieval accuracy for top 10 parameter settings on development corpus for each query condition given manual summary and keywords with pseudo-relevance feedback. Fields are: query condition, μ2 unigram weight, number of documents to use for feedback, and mixing weight of original query relative to feedback set. μ1 bigram weight was fixed at 1. The relative rarity of μ2 unigram weight parameter setting of 100000 suggests the bigram model provides more robust, strong performance for medium and long queries. 0.3829 0.3829 0.3823 0.3821 0.3821 0.3817 0.3812 0.3812 0.3811 0.3811 ... 0.3443 0.3440 0.3437 0.3435 0.3431 0.3431 0.3430 0.3425 0.3425 0.3425 ... 0.3131 0.3120 0.3116 0.3116 0.3110 0.3110 0.3107 0.3102 0.3100 0.3098
TDN.2000.20.8 TDN.100000.20.4 TDN.2000.15.4 TDN.2000.15.3 TDN.1000.20.8 TDN.1500.20.8 TDN.1500.15.4 TDN.100000.15.4 TDN.2000.20.4 TDN.1500.15.3 TD.2000.15.8 TD.2000.10.8 TD.1500.10.8 TD.100000.15.8 TD.2000.10.4 TD.1500.15.8 TD.1000.10.8 TD.750.10.8 TD.2000.15.16 TD.2000.10.3 T.2000.20.16 T.1500.20.16 T.1500.15.16 T.100000.20.16 T.2000.15.16 T.1000.20.16 T.2000.20.8 T.100000.20.8 T.100000.15.16
T.100000.20.4 7.3
MAP retrieval accuracy for top 10 parameter settings on development corpus for each query condition given ASR06B recognizer output plus automatic keywords and no pseudo-relevance feedback. NANC-smoothing consistently appeared in the two shorter query conditions while rarely appearing for verbose queries. Fields are: query condition, μ1 bigram weight, μ2 unigram weight, collection weight, WSJ weight, and NANC weight.
TDN.1.750.coll+akw-1.wsj-16.nanc-0 TDN.1.750.coll+akw-1.wsj-8.nanc-0 TDN.1.750.coll+akw-1.wsj-8.nanc-1 TDN.1.500.coll+akw-1.wsj-8.nanc-0 TDN.1.1250.coll+akw-1.wsj-16.nanc-0 TDN.1.1500.coll+akw-1.wsj-16.nanc-0 TDN.1.1000.coll+akw-1.wsj-8.nanc-0 TDN.1.1000.coll+akw-1.wsj-16.nanc-0 TDN.1.1250.coll+akw-1.wsj-8.nanc-0 TDN.1.750.coll+akw-1.wsj-4.nanc-1 TD.1.1500.coll+akw-2.wsj-1.nanc-1 TD.1.1250.coll+akw-2.wsj-1.nanc-1 TD.1.2000.coll+akw-1.wsj-1.nanc-1 TD.1.1500.coll+akw-1.wsj-1.nanc-1 TD.1.1250.coll+akw-1.wsj-2.nanc-1 TD.1.2000.coll+akw-1.wsj-2.nanc-1 TD.1.1250.coll+akw-1.wsj-1.nanc-1 TD.1.1500.coll+akw-1.wsj-2.nanc-1 TD.1.1000.coll+akw-1.wsj-4.nanc-1 TD.1.1000.coll+akw-1.wsj-2.nanc-1 T.1.5000.coll+akw-2.wsj-1.nanc-1 T.1.2000.coll+akw-2.wsj-1.nanc-1 T.1.5000.coll+akw-1.wsj-2.nanc-1 T.1.5000.coll+akw-1.wsj-1.nanc-1 T.1.5000.coll+akw-1.wsj-4.nanc-1 T.1.2000.coll+akw-1.wsj-4.nanc-0 T.1.1500.coll+akw-2.wsj-1.nanc-1 T.1.10000.coll+akw-1.wsj-1.nanc-1 T.5.5000.coll+akw-2.wsj-1.nanc-1
T.1.2000.coll+akw-1.wsj-1.nanc-1 7.4
MAP retrieval accuracy for top 10 parameter settings on development corpus for each query condition given ASR06B recognizer output plus automatic keywords using pseudo-relevance feedback. WSJ-smoothing was consistently useful while NANC smoothing was not. Fields are: query condition, μ2 unigram weight, number of documents to use for feedback, mixing weight of original query relative to feedback set, collection weight, WSJ weight, and NANC weight. μ1 bigram weight was fixed at 1.