This paper describes the participation of MIRACLE research consortium at the VideoCLEF track at CLEF 2008. We took part in both the main mandatory Classification task that consists in classifying videos of television episodes using speech transcripts and metadata, and the Keyframe Extraction task, whose objective is to select keyframes that represent individual episodes from a set of supplied keyframes (one from each shot of the video source). For the first task, our system is composed of two main blocks, the first in charge of building the core system knowledge base, and then the set of operational elements that are needed to classify the speech transcripts of the topic episodes and generate the output in RSS format. For the second task, our approach is based on the assumption that the most representative fragment (shot) of each episode is the one whose distance to the whole episode is the lowest, considering a vector space model. 4 runs were submitted in all. Regarding the classification task, we ranked 3rd (out of 6 participants) in terms of precision and 2nd in terms of recall.
MIRACLE team is a research consortium formed by research groups of three different universities in Madrid
(Universidad Politécnica de Madrid, Universidad Autónoma de Madrid and Universidad Carlos III de Madrid)
along with DAEDALUS, a small/medium size enterprise (SME) founded in 1998 as a spin-off of two of these
groups and a leading company in the field of linguistic technologies in Spain. MIRACLE has taken part in
CLEF since 2003 in many different tracks and tasks, including the main bilingual, monolingual and cross lingual
tasks as well as in ImageCLEF [4] [
We have participated in the main mandatory Classification task that consists in classifying videos of television episodes using speech transcripts and metadata, and in the Keyframe Extraction task, whose objective is to select keyframes that represent individual episodes from a set of supplied keyframes (one from each shot of the video source). 2.
The objective of the mandatory Classification task is to perform the speech recognition transcript-based topic classification (i.e., classify the videos of the television documentary episodes using the speech recognition output only). Videos include dual language (Dutch + English) episodes. Output is 10 topic-based feeds, each containing the episodes that have been classified in that topic category. The defined topic classes are Archeology, Architecture, Chemistry, Dance, Film, History, Music, Paintings, Scientific research and Visual arts. While classification task I is based on episode speech transcripts only, classification task II allows to use episode metadata for classification. The first step is to obtain the necessary training data for the classifier, as part of the task is, given the description of the subject class, to gather the necessary data to train the classifier. Our knowledge base for training the classifier was generated from Wikipedia articles. In order to do so, we first established a matching between the topic classes provided for the task and the classification topics that Wikipedia uses for articles, encoded in metadata. The next step was to obtain a list of Wikipedia articles belonging to each of the 10 topic class, for each task language, i.e. English and Dutch. Table 1 shows the number of articles for each topic class and language. 1. 2.
Text extraction: Ad-hoc scripts are run to obtain the actual text content of articles, filtering out Wikipedia tags.
Diacritics removal and conversion to lowercase: all terms are normalized by removing diacritics (in
the case of Dutch) and changing all letters to lowercase.
3. Filtering: All words recognized as stopwords are filtered out. Stopwords in the two target languages
were initially obtained from [
The processed corpus is indexed with Lucene retrieval engine [1] to allow a fast and efficient access to information needed for the classification. Two different indexes are built, one for each language. The classifier is based on the k-Nearest Neighbour algorithm [7]. To find the class for a given episode, the content is first processed as explained before. Then the whole set of resulting terms is used to build a query that is given to the Lucene search engine to obtain the list of the top k most relevant (i.e., most similar) articles in the Wikipedia-based corpus. Finally, the class of the given episode is the most frequent class in the top k results. After some preliminary experiments, a value of k set to 10 was chosen for our runs.
Our system is based on the assumption that, in the context of a vector space model representation [2], the most representative fragment (shot) of each episode (represented by a vector) is the one whose distance to the whole episode (also a vector) is the lowest. The contents of both each shot and the whole episode are first processed as explained before, after extracting the text from the speech transcription. Based on the vector space model, a weighted vector is built for each episode and set of shots, representing the term frequency of the main most significant terms in the given episode. Finally the keyframe extraction module selects the keyframe belonging to the most representative shot in the episode, which is the shot whose vector has the lowest distance from (i.e., is nearest) the vector of the whole episode. The metric used here was the cosine distance [2], although Euclidean distance could also be valid. An overview of the system architecture is shown in Figure 2.
We have submitted different runs for each proposed subtask: three for the classification task and one for the keyframe extraction. Table 2 shows the list of submitted runs. In short, “CNL” run only uses the index for the Dutch corpus, “CNLEN” uses both indexes and gathers together results from any of them, and “CNLMeta” uses the Dutch index but also includes the episode metadata to build the query for the retrieval engine.
It can be observed that the best precision is achieved with the “CNL” run in which only the Dutch transcription is used. When the knowledge base and the transcription in English are involved, results are noticeably and significantly worse. This could be directly motivated by the fact that the dominant language of the episodes is Dutch. However, the best modelled class is “Music”, corresponding to one of the classes that own a higher number of Wikipedia articles in the training set. This may suggest other explanations, such as the fact that training set (the knowledge base) for English is much smaller than the one available for Dutch. Another possible explanation could be that the voice recognition system for English is not as good as for Dutch. Obviously, these issues have to be further studied. 1 By definition, macroaverage values are computed as the mean of values of all classes that are first individually calculated. In contrast, the microaverage measures first obtain the aggregate counts for all classes and then calculate the values of precision and recall. Although results for the classification task II, i.e. including metadata, seem to be worse than results for classification task I, this is misleading. For all classes except Scientific research and Visual arts, precision values are higher if metadata is used, but the average value is worse specially due to the very low value for Visual arts. This issue still has to be analyzed.
Comparing to other groups, we successfully ranked 3rd out of 6 participants in terms of precision, 2nd in terms of recall and also f-score (not shown in the table).
Regarding the keyframe extraction task, MIRACLE was the only participant who submitted results. Thus, the evaluation has been manually made. Five native Dutch speakers (in the age range 20-50) were presented with the title and the description of each video episode along with two keyframes, one manually extracted and one automatically extracted provided by us. They were asked to choose which keyframe they preferred. Of the 40 videos, 1 did not have a keyframe. In two cases, the keyframe chosen manually and that chosen by the system was the same. Thus, each subject was asked about 37 different pairs of keyframes.
On average, the subjects chose the automatic over the manually selected keyframe in 15.2 cases (41.08%) and the manually over the automatic in 21.8 cases (58.92%). Table 4 shows the results of the evaluation process. The “Automatic keyframe” and “Manual keyframe” column represent the number of episodes that the evaluator has selected as more adequate between the automatically or manually extracted keyframe. The last column tries to show the number of correctly extracted keyframes, assuming that the evaluator has chosen correctly. These promising figures indicate that the automatically extracted keyframes may be strong competitors with the manual ones in the short- or middle-term future.
After a preliminary analysis of results obtained in the classification task, we can conclude that there seems to be a direct relationship between the knowledge base associated to a given class and results achieved in it. This probably indicates that the architecture of the system and provided algorithms are useful, but more effort must be invested to improve the knowledge base, both in its volume (coverage) and the pre-processing activities. Despite the subjectivity of the keyframe extraction task and lack of any reference experiment to which compare our own system, we can say that these results are promising and encourage us to keep on this line of research for future participations.
This work has been partially supported by the Spanish R+D National Plan, by means of the project BRAVO (Multilingual and Multimodal Answers Advanced Search – Information Retrieval), TIN2007-67407-C03-03 and by Madrid R+D Regional Plan, by means of the project MAVIR (Enhancing the Access and the Visibility of Networked Multilingual Information for the Community of Madrid), S-0505/TIC/000267. [1] Apache Lucene project. On line http://lucene.apache.org [Visited 10/08/2008]. [2] Baeza-Yates, R., Ribeiro-Prieto B.: Modern Information Retrieval. Addison Wesley (1999).