This paper describes the participation of the Chemnitz University of Technology at Grid@CLEF 2009. We integrated the CIRCO framework into our Xtrieval framework and performed 15 runs in the three languages German, English, and French. For each language we used two different stemmers and two different retrieval models. One run one was a fusion run combining the results of the four other experiments. Whereas the different runs demonstrated that the impact of the used retrieval technologies is highly depending on the corpus, the merged approach produced the best results in each language.
information retrieval, audiovisual media, data fusion, merging, Xtrieval, AMOPA In 2006 we started participating at CLEF. We hoped to gain in-depth insight into the effects of the different retrieval techniques in order to apply them to our real world problem: an archive for audiovisual media. Thus, we participated at several different tasks and achieved results from acceptable to very good. By now we can claim we got some kind of gut instinct how to configure our system in order to produce good results. But we did not get much closer to gain knowledge about the impact of retrieval techniques based on hard facts.
This contribution begins with explaining our motivation, the retrieval of audiovisual media in a TV-archive. It then provides a summary of the experiments done by the Chemnitz University of Technology at CLEF during the last four years. The final section discusses our results at the Grid-task and gives an outlook to future work.
Saxony has a unique TV landscape in Europe. With 60 broadcasting stations 30% of the German local TV stations reside in Saxony. In particular, the district Chemnitz is very strong here. With 165 stations, altogether, the eastern German states are covered considerably dense. In the western German states 37 local TV stations reside.
The local TV stations are an important producer and deliverer of information which are covered by stations broadcasting nationwide. For example, the public broadcasting station “Mitteldeutscher Rundfunk” (MDR) covers three federal states - Saxony, Saxony-Anhalt and Thuringia - at once, and produces only half an hour broadcasting time per day for information from Saxony. According to several media studies, local television stations are the most important suppliers of local news and information – ahead of local radios, newspapers and local Internet offers.
In order to enable the local TV stations to cooperate the project sachsMedia - Storage, Retrieval and Distribution of Audiovisual Media1 is creating tools for an archive of audiovisual media which can be jointly used by the TV stations. Within this archive both raw material as well as produced and broadcasted material is stored by every cooperating TV station. This material needs to be described as comprehensively as possible in order to be easily searchable. On the one hand, the description - or annotation - of the material is carried out intellectually according to principles of documentation. On the other hand automatic annotation will shift the load of annotation from human to the machine as far as possible. Here, sophisticated methods of multimedia retrieval will be implemented like object recognition and automated speech recognition.
It is commonplace in multimedia retrieval, that the actual search is not done using the original material but done using some kind of textual representation of the material. This textual representation can be produced intellectually: Ideally, some documentation specialist watches a video and describes it using a given vocabulary or classification which can be used for retrieval purposes afterwards. Reality looks a bit different: somebody describes a video by his/her own words. This is far from the ideal world but it is better than nothing. In the world of local TV storage and retrieval it looks like this: the producer of a video stores it on a common tape and puts the tape in a huge cupboard. Then he/she types some words describing the tape in a huge Excel-sheet. This annotation process is not designed for retrieval purposes but due to financial thoughts: proper intellectual annotation is expensive.
Another way to get textual annotations is an automatic analysis of the material. Within textual retrieval, this is not too complicated and every days practice. Just have a look at the web search engines. Nevertheless, intellectual annotation usually leads to better retrieval results even in the strictly textual domain. The major problem we face here is the switch between different media. Especially visual media are extremely hard to describe textually. For example, an article about the painting Mona Lisa will certainly contain the word “painting” and “Mona Lisa”. They are easy to be extracted and therefore easy to be searched for successfully. But, the painting itself does not tell us anything about the person displayed. Thought we know the name of the painting, we only know because somebody told us – in words. Somebody annotated it for us. Here, the expensive intellectual annotation seems to be way superior to automated approaches.
In order to enable optimal retrieval and inexpensive but complete annotation, the gap between intellectual and automated annotation needs to be overcome. Our approach is holistic. We use every kind of annotation and combine them to a full description. Figure 1 demonstrates the approach:
Fig 1: Video annotation approach used in sachsMedia The video stream is taken and analyzed using text and speech recognition. This produces a large amount of textual information as it is also used in some CLEF tasks which are based on the outputs of automatic speech recognition (ASR) systems. On the other hand visual features of different levels are extracted. They range from low level features like color histograms to high level features like face detection. The third source of metadata is intellectual annotation. Here, user interfaces are created which support the user in providing annotations without being distracted from his original work – which is in the case of local TV-station producing videos.
For some well developed elements of the annotation process we rely on commercial and open source products. Software for speech recognition for example is well developed and there is neither need nor capacity to 1 The project sachsMedia is funded by the Unternehmen Region-program of the German Federal Ministry of Education and Research. For more information about the project see: http://www.tu-chemnitz.de/informatik/ Medieninformatik/Sachsmedia/ build up an own system. Others like the low-level visual features can easily be implemented in our system. Yet others like speaker recognition are well enough described in literature to guarantee a useful implementation. Thus, for the single annotation tasks we rely on previous work, open source and commercialized products as far as possible. Nevertheless, there is much research left to do. For high-level features like recognition of people and places suitable processes are still needed. Text recognition in video streams needs to be implemented. And the proper way of intellectual annotations needs to be defined.
The final task is to configure the interplay of the single annotations. Speaker recognition by audio for example can support the people recognition by video and vice versa. Buildings found by picture recognition can be named by intellectual annotation or text recognition. The ways of metadata becoming interwoven are manifold. In order to have a closer look at the impact of the different metadata to the retrieval process and the dependencies and interactions between these metadata we are developing a highly flexible retrieval framework which will described in the following. 3
In 2005 we started conceptualizing and implementing a flexible framework for information retrieval purposes. It is a general finding in information retrieval, that the performance of retrieval systems highly depends on hardly generalizable aspects like for example corpora: Retrieval methods that perform well with one corpus do not necessarily work at all when applied to another corpus. After all that is the reason for installing different tracks in evaluation campaigns like CLEF2 and TREC3.
The general idea was to create a framework which is highly flexible and adjustable concerning information
retrieval technologies. The framework needed to provide interfaces to combine different state-of-the-art text
retrieval techniques on the one hand and to evaluate and integrate new methods for multimedia retrieval on the
other hand. An in-depth description of the framework design is given in [
The framework, named Xtrieval, implements a Java-based object-orientated API specification providing interfaces to all methods necessary for all possible designs of retrieval systems. By this, the framework is able to exchange, evaluate, and combine different components of other retrieval systems. In a first implementation Apache Lucene4 was integrated but by now also Terrier5 and Lemur6 are included in practice. The framework supports not only the integration of these and other toolkits but also allows combining their retrieval results on the fly.
Thus, the framework provides a realm of possible configurations. In order to conveniently adjust the system to different corpora we created a Graphical User Interface (GUI) (see figure 2).
This GUI provides a general configuration interface that supports the user in setting all parameter driven classes. Thus, all parameters of each class can be changed during runtime without any changes in the source code of the project. A second interface incorporates methods for calculating and visualizating recall-precision graphs.
Additional functions to load and save relevance assessments in popular formats (e.g. TREC) are provided as well.
The GUI can be used to configure the three main components: indexing, retrieval and evaluation (see figure 3). A general programming interface is able to convert every structured data Fig 2: GUI for Evaluation collection into an internal representation which is then used for the application of transformation and tokenization procedures like for example different stemming algorithms. The pre-processed data is than passed forward to a programming interface which allows connecting indexing libraries like Lucene. In order to integrate the full amount of metadata of audiovisual data we created the framework AMOPA which is presented later on.
Probably the most important interface of the Xtrieval framework allows the flexible use of retrieval algorithms. Queries are pre-processed according to the needs of different toolkits. It is also possible to combine searches in different indexes and to fuse these results into one result set by for example Sum-RSV, ProductRSV, and Z-Score. 2 http://www.clef-campaign.org/ 3 http://trec.nist.gov 4 http://lucene.apache.org/ 5 http://ir.dcs.gla.ac.uk/terrier/ 6 http://www.lemurproject.org/
Finally the evaluation component is capable to store and reload experiments and their complete parameter sets. This enables us to repeat experiments at a later date. It provides several measures to compare retrieval output to assessments. Additionally, it is possible to load and store relevance assessments in the TREC format. Figure 3 demonstrates the basic architecture of Xtrieval:
Graphical User Interface queries results assessments artefacts
AMOPA
indexing preprocessing feedback search
index evaluation relevances g n i x e d n i
Fig 3: The Xtrieval framework
For practical reasons (video analysis tool are written in C, Xtrieval in Java) we built for the automated annotation
tasks a separate framework called AMOPA-Automated MOving Picture Annotator. AMOPA uses the FFMPEG7
library to read video stream and perform first low level methods. Access for Java code to the C library FFMPEG
is provided by the library FFMPEG-Java, which is part of the Streambaby8 project. The actual analysis is
performed by AMOPA and organized in process chains. This concept allows us to exchange and reorder
processes very easily. A detailed description of AMOPA is given in [
Process Engine
(C-Library)
Î Java
Fig 4: The AMOPA framework [
In 2009 we participated the 4th time at CLEF. Table 1 gives a summary of our experiences with different CLEF tasks and provides short insight into the experiences of other participating groups. As one can see our system performed quite different over the years. Performance seems to be highly depending on the underlying corpus. 7 http://ffmpeg.org/ 8 http://code.google.com/p/streambaby/
The Xtrieval framework was used to prepare and run our text retrieval experiments for the Grid Experiments Pilot Track. The core retrieval functionality is provided by Apache Lucene, the Lemur toolkit, and the Terrier framework. This allowed us to choose from a wide range of state of the art retrieval models for all kinds of text retrieval experiments. Our main goal in this first Grid experiment was to provide strong baseline experiments, which could be used as reference for evaluation of sophisticated new retrieval approaches.
In order to participate at the Grid@CLEF track the CIRCO framework [25] had to be integrated into Xtrieval. Since one of the main design concepts of the Xtrieval framework was flexibility towards enhancements only a small number of classes had to be rewritten: two classes that are used to process the token streams during indexing and another class that writes the processed token stream in the index format of the used retrieval core. Since the integration of the Lemur and Terrier retrieval toolkits into Xtrieval had been done lately we did not have the time to test and debug the integration. Thus, we decided to adapt the Lucene indexing class only. Ten collections in five European languages, namely Dutch, English, French, German and Italian were provided for the Grid Experiment Pilot Track. For our participation we chose to run experiments on the English, French and German collections, which included six text collections in total. Table 2 shows the used collections and the provided fields which were taken for indexing. Table 3 shows some indexing statistics. We performed 15 runs, five for each language German, English, and French. For each language we used two different stemmers and two different retrieval models. One run one was a fusion run combining the results of the four other experiments. Table 4 provides the general configuration of each experiment as well as the retrieval performance in terms of mean average precision (MAP) and geometric mean average precision (GMAP). Please note the French run cut_fr_3. This run was corrupted while submitting. We did a separate evaluation for this run: cut_fr_3* is not part of the official statistics but shows the correct results.
All in all, merging models and stemmers brings the best results for all three languages. Comparing the models and stemmers leads to the following conclusions: • German: BM25 performs better than VSM. N-gram performs better than Snowball. • English: The results in English are vice versa: VSM performs (slightly) better than BM25. Snowball performs (slightly) better than Krovetz. • French: Here the results are even more confusing: VSM performs (especially in conjunction with Savoy) better than BM25. In conjunction with VSM Snowball performs better but in conjunction with BM25 Savoy is superior. Thus, some results demonstrate a better performance for VSM, some results show superiority of BM25. The results for the stemmers are similarly unpredictable. But it seems that this uncertainty can be overcome by data fusion: As table 4 demonstrates, for each language the merging of the retrieval models produced the best results. In our framework, merging is done by the z-score operator [28]. The results for the merged experiments are shown in figure 5:
Fig 5: Results for the merged experiments
9 http://www.unternehmen-region.de 10 http://www.bmbf.de/en/