The Dublin City University participation in the CLEF CL-SR 2005 task concentrated on exploring the application of our existing information retrieval methods based on the Okapi model to the conversational speech data set. This required an approach to determining approximate sentence boundaries within the free-flowing automatic transcription provided. We also performed exploratory experiments on the use of the metadata provided with the document transcriptions. Topics were translated into English using Systran V3.0 machine translation.
The Dublin City University participation in the CLEF CL-SR 2005 task concentrated on exploring the application of our existing information retrieval methods based on the Okapi model for this data set, and exploratory experiments on the use of the provided document metadata. Our official submissions included both the English monolingual and French bilingual runs. This paper reports additional results for German and Spanish bilingual runs. Topics were translated into English using Systran V3.0 machine translation system. The resulting English topics were used for retrieving from the English document collection.
Our standard Okapi retrieval system incorporates a summary-based pseudo relevance feedback (PRF) stage. This PRF system operates by selecting topic expansion terms from document summaries, full details are described in [1]. However, since the automated transcriptions of the conversational speech documents do not contain punctuation, we needed to develop a method of selecting significant document segments. Details of this method are described in Section 2.1.
The document transcriptions are provided with a rich set of metadata. It is not immediately clear how best to exploit this most effectively in retrieval. This paper reports our initial exploratory experiments in making use of this additional information.
The remainder of this paper is structured as follows: Section 2 overviews our retrieval system and describes our sentence boundary creation technique, Section 3 presents the results of our experimental investigations, and Section 4 concludes the paper with a discussion of our results.
The basis of our experimental system is the City University research distribution version of the Okapi system [2]. The documents and search topics were processed to remove stopwords from a list of about 260 words, suffix stripped using the Okapi implementation of Porter stemming [3] and terms were indexed using a small standard set of synonyms. None of these procedures were adapted for the new CL-SR document set.
Document terms were weighted using the Okapi BM25 weighting scheme developed in [2] calculated as follows,
where cw(i,j) represents the weight of term i in document j, cfw(i) is the standard collection frequency weight, tf(i,j) is the document term frequency, and ndl(j) is the normalized document length. ndl(j) is calculated as ndl(j) = dl(j)/avdl where dl(j) is the length of j and avdl is the average document length for all documents. k1 and b are empirically selected tuning constants for a particular collection. k1 is designed to modify the degree of effect of tf(i,j), while constant b modifies the effect of document length. High values of b imply that documents are long because they are verbose, while low values imply that they are long because they are multi-topic. Our values are tuned based on the training topics.
We apply PRF for query expansion using a summary-based method described in [1] which has been shown to be effective in our previous submissions to CLEF, including [4] and elsewhere. The main challenge for query expansion is the selection of appropriate terms from the assumed relevant documents. Our query expansion method selects terms from summaries of the top ranked relevant document. All non-stopwords in the summaries are ranked using a slightly modified version of the Robertson selection value (rsv
where r(i) = number of relevant documents containing term i, and rw(i) is the standard Robertson/Sparck Jones relevance weight
where n(i) = the total number of documents containing term i, r(i) = the total number of relevant documents term i occurs in, R = the total number of relevant documents for this query, and N = the total number of documents
The top ranked terms are then added to the topic. In our modified version of the rsv(i), potential expansion terms are selected from the summaries of the top ranked documents, but ranked using statistics from top a large number of assumed relevant ranked documents from the initial run.
Our standard process for summary generation is to select representative sentences from the document. Since the document transcriptions do not contain punctuation marking we needed an alternative approach to identifying significant units in the transcription. We approached this using a method derived from Luhn's word cluster hypothesis. Luhn's hypothesis states that significant words separated by not more than 5 non-significant words are strongly related. Strongly related word clusters were identified in the running document transcription by searching for word groups separated by not more than 5 insignificant words, as shown in Figure 1. Note that words appearing between clusters are not included in clusters, but can be ignored for the purposes of query expansion since they are by definition stop words. Luhn's Keyword Cluster Method Luhn's method assigns a sentence score for highest scoring cluster within a sentence. We adopted this method to assign a cluster score as follows:
where SS1 = the sentence score SW = the number of bracketed significant words (in this case 6) TW = the total number of bracketed words (in this case 14)
Query-Bias Method This method assigns a score to each sentence based on the number of query terms in the sentence as follows:
where SS2 = the sentence score TQ = the number of query terms present in the sentence NQ = the number of terms in a query
The overall score for each sentence (cluster) was then form by summing these two measures for each sentence.
This section describes the establishment of the parameters of our experimental system and gives results from our investigations.
In order to set the appropriate parameters for our feedback runs, we carried out development runs using the CLSR 2005 training topics. The Okapi parameters were set as follows k1=1.4 b=0.8. For all our PRF runs, 5 documents were assumed relevant for term selection and document summaries comprised the best scoring 4 clusters. The rsv values to rank the potential expansion terms were estimated based on the top 20 or 40 ranked assumed relevant documents. The top 20 ranked expansion terms taken from the clusters were added to the original query in each case. Based on results from our previous experiments in CLEF, the original topic terms are up-weighted by a factor of 3.5 relative to terms introduced by PRF. For our runs we used either the Title section (dcu*tit) or the Title and Description (dcu*desc) section of each topic. Our official runs are marked *. Initial baseline monolingual results using English without query expansion are also given for comparison.
For our experiments the document fields were combined as follows, dcua2 -combination of ASRTEXT2004A and AUTOKEYWORDA1 dcua1a2 -combination of ASRTEXT2004A, AUTOKEYWORDA1 and AUTOKEYWORDA2 dcusum -combination of ASRTEXT2004A, AUTOKEYWORDA1 and AUTOKEYWORDA2 and the SUMMARY dcuall -combination of NAME, MANUALKEYWORD, SUMMARY and ASRTEXT2004A section of each documents.
Tables 1-4 show results of our experiments using these different data combinations for the 25 test topics released for the CLEF 2005 CL-SR task. Results shown are Mean Average Precision (MAP), total relevant retrieved (Rr), and precision at cutoffs of 10 and 30 documents. Topic languages used are English, French, German and Spanish. Topics were translated into English using the Systran V3.0 machine translation system. The upper set of results in each table shows combined Title and Description topic queries and the lower set Title only topic queries. 1: Results using a combination of ASRTEXT2004A and AUTOKEYWORDA1, the Title or Title and Description topic fields. Expansion terms ranked for selection using statistics of 40 top ranked documents.
Results in Table 1 show results for combination of ASRTEXT2004A with AUTOKEYWORDA1. It can be seen that the PSF method improves results for the English topics. Also that the results using Title only topics is better than using the combined Title and Description topics with respect to MAP. This result is perhaps a little surprising since the latter are generally found to be perform better and we are investigating the reasons for the results observed here. However, the number of relevant documents retrieved is generally higher when using the combined topics. Cross language results using French topics are shown to perform better than monolingual English for both MAP and relevant retrieved. This is again unusual, but not unprecedented. Results for translated German and Spanish show a reduction on the monolingual results. 4: Results using a combination of NAME, MANUALKEYWORD, SUMMARY and ASRTEXT2004A section of each document, the Title or Title and Description topic fields. Expansion terms ranked for selection using statistics of 40 top ranked documents.
Table 4 shows a final set of experiments combining the NAME, MANUALKEYWORD and SUMMARY fields with ASRTEXT2004A. These results show large improvements over the results shown in previous tables. Performance for Title only and Title and Description combined topics is now similar with neither clearly showing an advantage. Monolingual English performance is clearly better than results for translated French topics, while our PRF method is still shown to be effective.
Our initial experiments with the CLEF 2005 CL-SR task illustrate a number of points: PRF can be successfully applied to this data set, that the different fields of the document set make varying contributions to information retrieval effectiveness. In general in can be seen that manual assigned fields are more usefully than the automatically generated ones.
These experiments only represent a small subset of those that are possible with this dataset. In order to better understand the usefulness of document fields and retrieval methods more detailed analysis of these existing results and further experiments are planned.