In this paper we describe the UNED's participation in the CLEF CL-SR 2005 track. We have tested several strategies to clean the automatic transcriptions and we have performed 84 different runs mixing these strategies with a proper noun recognition and different pseudo-relevant feedback approaches, in order to study the influence of each method in the retrieval process both in monolingual and cross-lingual environments.
We noticed that the influence of proper noun recognition is higher on the crosslingual environment, where MAP scores double when we use our entity recognizer. The best pseudo-relevance feedback approach was the one using the MANUALKEY-WORDS field. The effects of the different cleaning strategies were very similar, except for character trigrams, which obtained poor scores compared with the full word approaches.
The goal of the CLEF CL-SR 2005 track is to develop and evaluate systems for ranked retrieval of spontaneous conversational speech, over a collection of 8104 segments of interviews with different people.
Our participation in the track is focused on testing and mixing different techniques to improve the retrieval effectiveness: strategies to clean the documents, proper-noun recognition and different pseudo-relevance feedback approaches.
The effects of the cleaning strategies are very similar for all methods using full words (morpho, pos and clean). The manual keywords show to be the best field to use in a pseudo-relevance feedback approach. When using our entity recognizer, we also improve the MAP scores in both monolingual and cross-lingual environments. However, we also noticed that the influence of proper noun detection is bigger on the cross-lingual environment, because we use the entity recognizer to detect nouns that should not be translated.
The remaining sections of this paper are divided as follows: in Section 2 we describe out testbed, the design of our submitted runs, and the new strategies used in our additional experiments. In Section 3 we present the results of our 84 runs and we analyze the influence of: proper noun recognition in monolingual and cross-lingual environments (3.1), cleaning strategies (3.2) and the different pseudo-relevance feedback methods used (3.3). Finally, in Section 4 we present some conclusions.
Following the CL-SR CLEF 2005 guidelines we submitted five different runs, and we perform several other experiments after receive the results and the relevance assessments.
The test collection consists of 8104 segments from interviews of Holocaust survivors. Each document has several fields with different pieces of information about the segment. In our experiments we have used the following information:
• We compared the two different automatic transcriptions (ASR2003A and ASR2004A) of the segment, and we find that there are no big differences between both transcriptions. So, we decide to use only the transcription field ASR2004A.
• The human written summary of the segment (SUMMARY) to detect Proper Nouns which not need to be translated.
• Three different sets of keywords: one set of manually selected keywords (MANUALKEY-WORD) and two different automatic sets of keywords (AUTOKEYWORD2004A1 and AUTOKEYWORD2004A2).
We have tried different strategies to clean the automatic transcriptions of the documents:
1. We noticed that several words that are not recognized by the automatic transcription, in particular proper nouns, appear split in their characters, like "l i e b b a c h a r d ". We have searched all the occurrences of a list of single characters into the documents, and we have joined them assuming that these characters form a whole unrecognized word.
2. When one speaks, it is usual to repeat some words to emphasize a specific part of the conversation or to show to the other person a thinking process ("let me think, yes, yes, yes..."). Automatic Speech Recognizers transcribe these duplicated words and when performing a retrieval process, the results can be affected by these words. So we have removed all extra occurrences of the duplicated words.
The resulting documents after these two first steps have been indexed in a collection that we will refer as clean.
3. In Information Retrieval, the most informative words seems to be nouns, adjectives and verbs. Our next step was to clean the documents, removing all words except nouns, adjectives and verbs.
We used two different approaches to perform this cleaning:
• We used the FreeLing [6] set of linguistic tools to perform a morphological analysis of the documents from the clean collection. The output of this analysis indicates the possible Part of Speech for a given word. If, according to this analysis, a word can act as a noun, an adjective or a verb, the word remains in the document and was removed otherwise. This technique was used successfully to extract noun-phrases from a Spanish collection [3] and it proved to be very useful in an Information Retrieval environment. We built a new collection called morpho with the documents obtained after this process.
• We perform a full Part of Speech tagging using freeling again. This process is more complex than the previous one, and includes a PoS Disambiguation phase in order to select only one of the possible Part of Speech for a given word. Like in the previous process, only words that act as nouns, adjectives or verbs remain in the resulting documents, which have been indexed in a collection that we will refer as pos.
4. Additionally, we split the cleaned documents in character 3-grams to compare the performance of this simple approach with the performance of more complex cleaning process. This collection will be referred as 3grams.
We have used the English and the Spanish topics, provided by the organizers, from those we have only used TITLE and DESCRIPTION fields in all our runs. For each topic we have removed the usual stopwords and, in the 3-grams runs, we have split each word into character 3-grams individually.
For our cross-lingual runs, we used a query translation approach following Pirkola's proposal [4], where alternative translations for a term were taken as synonyms, giving them equal weights.
Finally we have used INQUERY [2] as a search engine.
Following the CL-SR CLEF guidelines, we submit five different runs:
1. A monolingual run using the 3grams collection and the English topics expressed as 3-grams (mono-3grams).
2. A monolingual run using the morpho collection (mono-morpho).
3. A cross-lingual run using the morpho collection and the Spanish topics translated into English (trans-morpho).
4. A monolingual run using the pos collection (mono-pos).
A cross-lingual run using the pos collection and the Spanish topics translated into English (trans-pos).
The results of the submitted runs are shown in Table 1, where we can compare our results with the best monolingual and Spanish cross-lingual runs.
Regarding these numbers we can draw some preliminary conclusions:
• Our runs are far from the best monolingual and Spanish cross-lingual runs, so there is room for improvement.
• MAP scores of morpho and pos runs are very similar in both monolingual and crosslingual environments. As expected pos scores slightly better than morpho, but with only two different runs we don't have enough data to conclude that the PoS Disambiguation helps to clean these documents.
• Character 3-grams run scores worse than full word retrieval (75.6%). Again we have too few data to conclude that it's better to use full word retrieval than a 3-grams approach. • Just using a bilingual dictionary and a Pirkola's approach, our cross-lingual runs reach 40% MAP of their respective monolingual counterparts. In some cases (see section 3.1) a bad translation of some proper nouns difficult the cross-lingual search.
With only five different runs it's very difficult to obtain clear conclusions. According the suggestion of CL-SR CLEF organizers, we have run more experiments after the official submission. These experiments are described in the next section.
With these additional experiments, our intention was to test the effects of two new strategies (proper noun identification and pseudo-relevance feedback) and compare all possible combination of all our different approaches.
We have used our entity recognizer [5] in order to improve the query structure identifying possible proper nouns in the topics. These entities have been used in different ways in our monolingual and cross-lingual experiments:
• Monolingual: we just identify the proper nouns contained in the topics and, using the #phrase operator of INQUERY, we have structured the query.
• Cross-lingual: if we use the same strategy as above, the identified proper nouns in Spanish topics maybe will not appear in the English documents. For instance, in the topic #1133 ("The story of Varian Fry"), we identify these proper nouns:
-Varian Fry -Comité de Rescates de Emergencia -Marsella
We used the recognizer in order to identify possible proper nouns in the SUMMARY field of the documents too. Only proper nouns that appear in both lists have been used to structure the query.
In the given example, the only proper noun that appear in the SUMMARY field of the documents was "Varian Fry". So our strategy was to translate the topic, except these names.
Once we have the proper nouns identified, we use the INQUERY's operator #phrase with each one to structure the query. Below we can see the topic #2012 unstructured and structured using the proper noun identified on it.
• Topic #2012 unstructured: #sum( collaboration local population information collaboration local population german authorities east central europe holocaust );
• Topic #2012 structured: #sum( #phrase(German Authorities) #phrase(East Central Europe) #phrase(Holocaust) collaboration local population information collaboration local population );
We decide to test a Pseudo-Relevance Feedback [1] (prf) approach to check the utility of the keyword fields of the documents. We built five different relevance feedback methods:
• A collection (AK1) using the AUTOKEYWORD2004A1 field.
• A collection (AK2) using the AUTOKEYWORD2004A2 field.
• A collection (AK12) mixing the keywords present in both autokeyword fields.
• A collection (MK) using the MANUALKEYWORD field.
• A collection (MKAK12) mixing the keywords from the three keyword fields.
In order to mix the keywords from different fields, we used the following process:
1. Each automatic keyword was scored according to his order of appearance in the field: the first keyword obtains a score of 20 (there are 20 keywords in each autokeyword field), and the last one just 1.
2. If a keyword appears in both autokeyword fields, his final score was the sum of the scores obtained in each field.
3. In order to create the AK12 collection, we selected the 20 keywords with higher score.
4. When building the MKAK12 collection, we first selected the n manual keywords, and then, we added the 20 − n first keywords from AK12 collection.
When searching, we select the top 10 ranked documents and we combine their keywords using the same method as described above to mix the keywords from different fields. Finally we refine the query adding the top 20 keywords.
Our intention was to test all possible combinations of each feature: topic language, proper noun identification, cleaning method and relevance feedback. However, we notice that the meaning of cross-lingual runs using 3-grams might be confusing. The translated query is structured with the #syn operator of INQUERY. If we split the translated query into 3-grams, we can't use the #syn operator to structure it, because all 3-grams would be considered as synonyms even if they came from the same word. So we have not used 3-grams on our cross-lingual runs.
proper noun cleaning method relevance feedback
Each run is named with the labels of the different features used on it. For instance "mononoent-morpho-AK2 " represents a monolingual run without proper noun identification, over the morpho collection and using the AUTOKEYWORD2004A2 field for the Pseudo-Relevance Feedback process.
On figure 1 we can see all possible values for each feature: 96 different combinations. But, if we exclude the cross-lingual runs when using 3-grams, there are 84 different runs.
The results of all our runs are shown in Table 2.
We can draw some preliminary points:
• Our best run, mono-ent-morpho-MK, scores a 25.95% MAP, a 82.9% of the best submitted monolingual run, from the University of Maryland. We have obtained a 277.8% of improvement respect our best submitted monolingual run.
• Our best cross-lingual run trans-ent-pos-MK scores about a 131.2% MAP respect the best submitted Spanish run from the University of Ottowa. In this case, the improvement respect our best submitted cross-lingual run is 545.8%.
• The best strategy seems to be pseudo-relevance feedback using the MANUALKEYWORDS field, followed by the combination of the three keywords fields.
• The monolingual 3-grams runs score poorly, reaching only a 30% MAP of our best run.
Let analyze more carefully the influence of the different approaches:
On table 3 we can compare the effects of the proper noun detection in both monolingual and cross-lingual runs. The numbers on the ent and noent columns show the percentage of the MAP of the cross-lingual runs compared with the MAP of the monolingual. The numbers on the mono and trans represent the increment of the MAP when using proper noun detection technique.
Regarding these numbers we can infer some interesting points:
• Using proper nouns, the MAP of cross-lingual runs reach 75% of the monolingual runs. Without proper nouns, cross-lingual runs reach only 35 − 40% MAP of monolingual runs.
• Influence of proper nouns is higher on cross-lingual runs, increasing MAP more than twice respect to noent runs. On monolingual runs the increment is worthless and, probably, statistically not relevant.
• The influence of proper nouns is also worthless on 3-grams runs, even the best 3-grams run is mono-noent-3grams-MK that not uses proper noun detection.
For instance, on topic #1113 ("The story of Varian Fry"), the influence of proper noun detection is very important, because in Spanish the word "Varian" can be identified as a verbal form of "Variar" (to vary, change) and is wrongly translated:
Regarding the influence of the cleaning method we can conclude that the best cleaning strategy seems to be morpho, but the differences respect pos and clean are minimal and, probably statistically not relevant.
Using just a morphological analyzer to identify possible part of speech for a given word, proves to be a very useful strategy in information retrieval if we don't have a full part of speech tagger.
Again, character 3-grams show to be a bad cleaning strategy when compared with full words approaches.
On table 4 we can compare the differences between the different pseudo-relevant feedback strategies tested. Each column represent the MAP percentage of one prf method respect to another. For instance MKAK12/MK column represent the percentage of MKAK12 prf respect MK prf. • The best prf method is MK (average increment of MAP using the prf over manual keywords field respect no relevance feedback is about 271.6%), nearly followed by MKAK12 (an average MAP of 90.5% respect MK).
• There are no big differences between the use of each automatic keyword field, but prf using AUTOKEYWORD2004A1 field seems to obtain high MAP score. And, when combining both fields (AK12), MAP scores 41.51% of MKAK12 on average.
In this paper, we have shown different techniques to improve retrieval of automatic speech transcriptions in both monolingual and cross-lingual environments.
• We have tested four different cleaning techniques. Differences between full word techniques (clean, morpho and pos) are worthless, but a character 3-grams approach seems to be worse.
• Pseudo-relevance feedback using manually generated keywords shows to be the best option to increment the performance of the retrieval, with an average percentage of 271.6% respect no relevance feedback.
• The use of an entity recognizer to identify proper nouns, proves to be very useful, specially on a cross-lingual environment, where the MAP scores twice when using them.
Our intention is to perform further analysis over the results, including statistical relevance tests to determine the influence of the different methods we have tried.
We also want to test a different approach to identify proper nouns in the automatic transcriptions, or in the automatic keyword fields, instead of using the manual summary of the documents. Maybe the big improvement detecting proper nouns in the cross-lingual environment is due to the use of a manually generated field, similarly to the best scores obtained when using MANU-ALKEYWORDS field in pseudo-relevance feedback.
This work has been partially supported by the Spanish Government under project R2D2-Syembra (TIC2003-07158-C04-02). Víctor Peinado holds a PhD grant by UNED (Universidad Nacional de Educación a Distancia).