The University of Maryland participated in the English and Czech tasks. For English, one monolingual run using only ¯elds based on fully automatic transcription (the required condition) and one (otherwise identical) cross-language run using French queries were o±cially scored. Three contrastive runs in which manually generated metadata ¯elds in the English collection were indexed were also o±cially scored to explore the applicability of recently developed \meaning matching" approaches to cross-language retrieval of manually indexed interviews. Statistical translation models trained on European Parliament proceedings were found to be poorly matched to this task, yielding 38% and 44% of monolingual mean average precision for indexing based on automatic transcription and manually generated metadata, respectively. Weighted use of alternative translations yielded an apparent (but not statistically signi¯cant) 7% improvement over one-best translation when bi-directional meaning matching techniques were employed. Results for Czech were not informative in this ¯rst year of that task, perhaps because no accommodations were made for the unique characteristics of Czech morphology.
speech [
This year's track also introduced a new task, searching unsegmented Czech interviews. Unlike more traditional retrieval tasks, the objective in this case was to ¯nd points in the interviews that mark the beginning of relevant segments (i.e., points at which a searcher might wish to begin replay). A new evaluation metric, Generalized Average Precision (GAP), was de¯ned to evaluate system performance on this task. The GAP measure takes into account the distance between each system-suggested start time in a ranked list and the closest start time found by an assessor|the greater the time between the two points, the lower the contribution of that match to a system score. A more detailed description of GAP and details of how it is computed can be found in the track overview paper. In this study, we were interested in evaluating retrieval based on overlapping passages, a retrieval techniques that has proven to be useful in more traditional document retrieval settings, while hopefully also gaining some insight into the suitability of the new evaluation metric. 2
In this section, we brie°y describe the techniques that we used in our study. 2.1
Both test collections provide several types of data that are associated with the information to be retrieved, so we tried di®erent pre-indexing combinations of that data. For the English test collection, we combined the ASR text generated with the 2004 system and the ASR text generated with the 2006 system, and we compared the result with that obtained by indexing each alone. Our expectation was that the two ASR engines could produce di®erent errors, and that combining their results might therefore yield better retrieval e®ectiveness. Thesaurus terms generated automatically using a kNN classi¯er o®er some measure of vocabulary expansion, so we added them to the ASR text combination as well.
The English test collection also contains a rich set of metadata that was produced by human indexers. Speci¯cally, a set of (on average, ¯ve) thesaurus terms were manually assigned, and a three-sentence summary was written for each segment. In addition, the ¯rst names of persons that were mentioned in each segment are available, even if the name itself was not stated. We combined all three types of human-generated metadata to create an index for a contrastive condition.
The Czech document collection contains, in addition to ASR text, manually assigned English thesaurus terms, automatic translations of those terms into Czech, English thesaurus terms that were generated automatically using a kNN classi¯er that was trained on English segments, and automatic translations of those thesaurus terms into Czech. We tried two ways of combining this data. Czech translations of the automatically generated thesaurus terms were combined with Czech ASR text to produce the required automatic run. We also combined all available keywords (automatic and manual, English and Czech) with the ASR text, hoping that some proper names in English might match proper names in Czech. 2.2
In our previous study, we found that using several translation alternatives generated from
bidirectional statistical translation could signi¯cantly outperform techniques that utilize only the most
probable translation, but that using all possible translation alternatives was harmful because
statistical techniques generate a large number of very unlikely translations [
We used the same bilingual corpus that we used in last year's experiment. Speci¯cally, to
produce a statistical translation table from French to English, we used the freely available GIZA++
toolkit [
Due to the time and resource constraints, we were not able to try a similar technique for Czech this year. All of our processing for Czech was, therefore, monolingual. 3
The required run for the CLEF-2006 CL-SR track called for use of the title and description ¯elds as a basis for formulating queries. We therefore used all words from those ¯elds as the query (a condition we call \TD") for our ¯ve o±cial submissions. Stopwords in each query (as well as in each segment) were automatically removed (after translation) by InQuery, which is the retrieval engine that we used for all of our experiments. Stemming of the queries (after translation) and segments was performed automatically by InQuery using kstem. Statistical signi¯cance is reported for p < 0:05 by a Wilcoxon signed rank test for paired samples. 3.1
1http://www-i6.informatik.rwth-aachen.de/Colleagues/och/software/GIZA++.html umd.auto umd.asr04a umd.asr06a umd.asr06b MAP
As Table 1 shows, our one o±cially scored automatic CLIR run, umd.auto.fr0.9 (with a threshold of 0.9) achieved only 38% of the MAP of the corresponding monolingual MAP, a statistically signi¯cant di®erence. We suspect that domain mismatch between the corpus used for training statistical translation model and the document collection might be a contributing factor, but further investigation of this hypothesis is clearly needed. A locally scored one-best contrastive condition (not shown) yielded about the same results.
Not surprisingly, combining ¯rst names, segment summaries, and manually assigned thesaurus terms produced the best retrieval e®ectiveness as measured by MAP. Among the three runs with manually created metadata, umd.manu.fr0.9 and umd.manu.fr0 are a pair of comparative crosslanguage runs: umd.manu.fr0.9 had a threshold of 0.9 (which usually led to the selection of multiple translations), while umd.manu.fr0 used only the most probable French translation for each English query word. These settings yielded 44% and 41% of the corresponding monolingual MAP, respectively. The 7% apparent relative increase in MAP with a threshold of 0.9 (compared with one-best translation) was not found to be statistically signi¯cant. 3.2
In additional to the o±cially scored monolingual run umd.auto that used four automatically generated ¯elds (ASRTEXT2004A, ASRTEXT2006A, AUTOKEYWORD2004A1, and AUTOKEYWORD2004A2), we scored three additional runs based on automatically generated data locally: umd.asr04a, umd.asr06a, and umd.asr06b. These runs used only the ASRTEXT2004A, ASRTEXT2006A, or ASRTEXT2006B ¯elds, respectively (ASRTEXT2006A is empty for some segments; in ASRTEXT2004B those segments are ¯lled in with reportedly less accurate data from ASRTEXT2004A). Table 2 shows the MAP for each of these runs and (again) for umd.auto. Combining the four automatically generated ¯elds yielded a slight apparent improvement in MAP over any run that used only one of those four ¯elds, although the di®erences are not statistically signi¯cant. Interestingly, there is no noticeable di®erence among umd.asr04a, umd.asr06a and umd.asr06b even when average precision is compared on a topic-by-topic basis, despite the fact that the ASR text produced by the 2006 system is reported to have a markedly lower word error rate than that produced by the 2004 system. 3.3
To investigate the factors that could have had a major in°uence on the e®ectiveness of runs with automatic data, we conducted a topic-by-topic comparison of average precision. To facilitate that analysis, we produced two additional runs with queries that were formed using words from the title ¯eld only, with one run searching the ASRTEXT2006B ¯eld only (AUTO) and the other the three metadata ¯elds (MANU). We focused our analysis on those topics that have a MAP of 0.2 or larger in the MANU run for consistency with the failure analysis framework that we have applied previously. With this constraint applied, 15 topics remain for analysis. Figure 1 shows the topic-by-topic comparison of average precision. As we saw in 2006, the di®erence in average precision between the two conditions is quite large for most topics.
Using title-only queries allowed us to look at the contribution of each query word in more detail. Speci¯cally, we looked at the number of segments in which query word appears (a statistic normally referred to as \document frequency"). As Table 3 shows, there are some marked di®erences in the prevalence of query term occurrences between automatically and manually generated ¯elds. In last year's study, poor relative retrieval e®ectiveness with automatically-generated data was most often associated with a failure to recognizing at least one important query word (often a person or location name). This year, by contrast, we see only two obvious cases that ¯t that pattern (\varian" and \dp"). This may be because proper names are less common in this year's topic titles. Instead, the pattern that we see is that query words actually appears quite often in ASR segments, and in some cases perhaps too often. Said another way, our problem last year was recall; this year our problem seems to be precision. We'll need to actually read some of the ASR text to see if this supposition is supported, of course. But it is intriguing to note that the failure pattern this year seems to be very di®erent from last year's. 4
The University of Maryland was also one of three teams to participate in the ¯rst year of the Czech evaluation. Limited time prevented us from implementing any language processing techniques that were speci¯c to Czech, so all of our runs were based on string matching without any morphological normalization. Automatic segmentation into overlapping passages was provided by the organizers , and we used that segmentation unchanged. Our sole research question for Czech was, therefore, whether the new start-time evaluation metric yielded results that could be used to compare variant systems.
We submitted three o±cially scored runs for Czech: umd.all used all available ¯elds (both automatically and manually generated, in both English and Czech), umd.asr used only the Czech ASR text, and umd.akey.asr used both ASR text and the automatic Czech translations of the automatically generated thesaurus terms. Table 4 shows the resulting mean Generalized Average Precision (mGAP) values. About all that we can conclude form these results is that they do not serve as a result for making meaningful comparisons. We have identi¯ed four possible causes that merit investigation: (1) our retrieval system may indeed be performing poorly, (2) the evaluation metric may have unanticipated weaknesses, (3) the scripts for computing the metric may be producing erroneous values, or (4) the relevance assessments may contain errors. There is some reason to suspect that the problem may lie with our system design, which contains two known weaknesses. Most obviously, Czech is a highly in°ected language in which some degree of morphological normalization is more important than it would be, for example, for English. Good morphological analysis tools are available for Czech, so this problem should be easily overcome. The second known problem is more subtle: overlapping segmentation can yield redundant highly ranked segments, but the mGAP scoring process penalized redundancy. Failing to prune redundant segments prior to submission likely resulted in a systematic reduction in our scores. It is not clear whether these two explanations together su±ce to explain the very low reported scores this year, but the test collection that is now available from this year's Czech task is exactly the resource that we need to answer that question. 5
Earlier experiments with searching broadcast news yielded excellent results, leading to a plausible conclusion that searching speech was a solved problem. As with all such claims, that is both true and false. Techniques for searching broadcast news are now largely well understood, but searching spontaneous conversational speech based solely on automatically generated transcripts remains a very challenging task. We continue to be surprised by some aspects of our English results, and we are only beginning to understand what happens when we look beyond manually segmented English to unsegmented Czech. Among the things that we don't yet understand are the degree to which the di®erences we observed this year in English are due to di®erences in the topics or di®erences in the ASR, how best to a®ordable analyze retrieval failures when the blame points more to precision than to recall, whether our unexpectedly poor CLIR results were caused solely by a domain mismatch or principally by some other factor, and whether our Czech test collection and retrieval evaluation metric are properly constructed. In each case, we have so far looked at only one small part of the opportunity space, and much more remains to be done. We look forward to this year's CLEF workshop, where we will have much to discuss!