In this paper, we describe the TNO approach to the Finding Related Resources or linking task of VideoCLEF09. Our system consists of a weighted combination of o -theshelf and proprietary modules, including the Wikipedia Miner toolkit of the University of Waikato. Using this cocktail of largely o -the-shelf technology allows for setting a baseline for future approaches to this task.1 The Finding Related Resources or linking task of VideoCLEF'09 consists of relating Dutch automatically transcribed TV speech to English Wikipedia content. For a total of 45 video episodes, a total of 165 anchors (speech transcripts) have to be linked to related Wikipedia articles. Technology emerging from this task will contribute to a better understanding of Dutch video for non-native speakers. The TNO approach to this problem consists of a cocktail of o -the-shelf techniques. Central to our approach is the use of the Wikipedia Miner toolkit developed by researchers at the University of Waikato2 (see [4]). The so-called Wiki er functionality of the toolkit detects Wikipedia topics from raw text, and generates cross-links from input text to a relevance-ranked list of Wikipedia pages. We investigated two possible options for bridging the gap between Dutch input text and English Wikipedia pages: translating queries to English prior to the detection of Wikipedia topics, and translating Wikipedia topics detected in Dutch texts to English Wikipedia topics. In the latter case, the use of Wikipedia allows for an abstraction of raw queries to Wikipedia topics, for which 1This work is supported by the European IST Programme Project FP6-0033812. This paper only re ects the authors' views and funding agencies are not liable for any use that may be made of the information contained herein. 2See http://wikipedia-miner.sourceforge.net
the translation process in theory is less complicated and error prone. Speci c to our approach is a weighted combination of various modules, and the use of a specially developed part-of-speech tagger for uncapitalized speech transcripts. 2
In this section, we describe the setup of our system. In subsections 2.1, 2.2 and 2.3, we describe the essential ingredients of our system. In subsection 2.4, we de ne a number of linking strategies based on these basic ingredients, which are combined into scenarios for our runs (section 3). 2.1
For the translation of Dutch text to English (and following [
Since people, organizations and locations often have entries in Wikipedia, accurate proper name
detection is important for this task. Erroneous translation to English of Dutch names (e.g. 'Frans
Hals' becoming 'French Neck') should be avoided. Proper name detection prior to translation
allows for exempting the detected names from translation. A complicating factor is formed by
the fact that the transcribed speech in the various broadcastings is in lowercase, which makes the
recognition of proper names challenging, since important capitalization features can no longer be
used. In order to address this problem, we trained a maximum entropy part-of-speech tagger: an
instance of the Stanford tagger4 (see [
3http://babelfish.yahoo.com/ 4http://nlp.stanford.edu/software/tagger.shtml 2.2
The imperfect English translation by Babel Fish was observed to be the main reason for erroneous Wikify results. In order to omit the translation step, we ported the English Wiki er of the Wikipedia Miner toolkit to Dutch, for which we used the Dutch Wikipedia dump and Perl scripts provided by developers of the Wikipedia Miner toolkit. The resulting Dutch Wiki er ('NL Wiki er' in Figure 3) has exactly the same functionality as the English version, but unfortunately contains a lot less pages than the English version (a factor 6 less). Even so, the translation process now is narrowed down to translating detected Wikipedia topics (the output of the Dutch Wikify step) to English Wikipedia topics. For the latter, we implemented a simple database facility (to which we shall refer with 'The English Topic Finder') that uses the cross-lingual links between topics in the Wikipedia database for carrying out the translation of Dutch topics to English topics.
An example of the output of the English and Dutch Wiki ers for the query in Figure 1 is given in Figure 2. The di erent rankings of the various detected topics are represented as a tag cloud with di erent font sizes, and can be extracted as numerical scores from the output. 2.3
In order to be able to entirely by-pass the Wikipedia Miner toolkit, we deployed the Lucene
search engine ([
The set of techniques just described leads to a total of four basic linking strategies.
Of the various combinatorial possibilities of these strategies, we selected 5 combinations for our submitted runs. The basic linking strategies are: Strategy 1: proper names only (the top row in Figure 3) Following proper name recognition, a quasi-document is created that only consists of all recognized proper names. The Dutch Wikify tool is used to produce a ranked list of Dutch Wikipedia pages for this quasi-document.
Subsequently, the topics of these pages are linked to English Wikipedia pages with the English Topic Finder.
Strategy 2: proper names preservation (second row in Figure 3) Dutch text is translated to English with Babel sh. Any proper names found in the part-of-speech tagged Dutch text are added to the translated text as untranslated text, after which the English Wiki er is applied, producing a ranked list of matching Wikipedia pages.
Strategy 3: topic to topic linking (3rd row from the top in Figure 3) The original Dutch text is wiki ed using the Dutch Wikify tool, producing a ranked list of Wikipedia pages. The topics of these pages are subsequently linked to English Wikipedia pages with the English Topic nder.
Strategy 4: text to page linking (bottom row in Figure 3) Lucene matches queries with Dutch Wikipedia pages. The English topic nder tries to nd the corresponding English Wikipedia pages for the Dutch topics in the pages returned by Lucene. This strategy omits the use of the Wiki er and was used as a fall-back option, if none of the other modules delivered a result. A thresholded merging algorithm removes any results below an estimated threshold and blends the remaining results into a single ordered list of Wikipedia topics, using estimated weights for the various sources of these results. Several di erent merging techniques were used for di erent runs; these will be discussed in subsection 3. 3
In this section, we describe the con gurations of the 5 runs we submitted. We were speci cally interested in the e ect of proper name recognition, the relative contributions of the Dutch and English Wiki ers, and the e ect of full-text Babel sh translation as compared to a topic-to-topic translation approach.
Run 1 All four linking strategies were used to produce the rst run. A weighted merger ('Merger' in Figure 3) was used to merge the results from the di erent strategies. The merger works as follows: 1. English Wikipedia pages referring to proper names are uniformly ranked before all other results. 2. The rankings produced by the second linking strategy (rankEN ) and third linking strategy (rankDU ) for any returned Wikipedia page p are combined according to the following scheme: rank(p) = ((rankEN (p) 0:2) + (rankDU (p) 0:8)) 1:4 (1) The Dutch score was found to be more relevant than the English one (hence the 0.8 vs. 0.2 weights). The sum of the Dutch and English score is boosted with an additional factor of 1.4, awarding the fact that both linking strategies come up with the same result. 3. Pages found by linking strategy 2 but not by linking strategy 3 are added to the result and their ranking score is boosted with a factor of 1.1. 4. Pages found by linking strategy 3 but not by linking strategy 2 are added to the result (but their ranking score is not boosted). 5. If linking strategies 1 to 3 did not produce results, the results of linking strategy 4 are added to the result.
Run 2 is the same as run 1 with the exception that linking strategy 1 is left out (no proper name recognition).
Run 3 is similar to run 1, but does not boost results at the merging stage, and averages the rankings of the second and third linking strategy. This means that the weights used by the merger in run 1 (0.8, 0.2 and 1.4) are resp. 0.5, 0.5 and 1.0 for this run.
Run 4 only uses linking strategy 1 and 3. This means that no translation from Dutch to English is performed. In the result set, the Wikipedia pages returned by linking strategy 1 are ordered before the results from linking strategy 2.
Run 2 Run 3 Run 4 Run 5 Run 5 uses all linking strategies except linking strategy 1 (it omits proper name detection). In this run a di erent merging strategy is used: 1. If linking strategy 2 produces any results, add those to the nal result set and then stop. 2. If linking strategy 2 produces no results, but linking strategy 3 does, add those to the nal result and stop. 3. If none of the preceding linking strategies produces any results, add the results from linking strategy 3 to the nal result set. 4
For VideoCLEF'09, two groups submitted runs for the linking task: Dublin City University and TNO. Two evaluation methods were applied by the task organizers to the submitted results. A team of assessors rst achieved consensus on a primary link (the most important or descriptive Wikipedia article), with a minimum consensus among 3 people. All queries in each submitted run were scored for Mean Reciprocal Rank5 for this primary link, as well as for recall. Subsequently, the annotators agreed on a set or related resources that necessarily included the primary link, in addition to secondary relevant links (minimum consensus of one person). Since this list of secondary links is non-exhaustive, for this measure only MRR is reported, and not recall. As it turns out, the unweighted combination of results (run 3) outperforms all other runs, followed by the thresholded combination (run 1). This indicates that the weights in the merging step are suboptimal. Omitting proper name recognition results in a noticeable drop of performance under both evaluation methods, underlining the importance of proper names for this task.
In addition to the recall and MRR scores, the assessment team distributed the graded relevance
scores (see [
In this contribution, we have taken a technological and o -the-shelf-oriented approach to the problem of linking Dutch transcripts to English Wikipedia pages. Using a blend of commonly available rank of answer ri with respect to the correct answer.
Q 5For a response r = r1; : : : ; rQ to a ranking task, the MRR would be MRR = jQ1 j X i=1
1 ranki , with ranki the ’dcu_run1.plot’ ’dcu_run2.plot’ ’dcu_run3.plot’ ’dcu_run4.plot’ 20 40 60 80 100 120 140 160
180 Queries 80
Queries 20 40 60 100 120 140 160 software resources (Babel sh, the Waikato Wikipedia Miner Toolkit, Lucene, and the Stanford maximum entropy part-of-speech tagger), we demonstrated that an unweighted combination produces competitive results. We hope to have demonstrated that this low-entry approach can be used as a baseline level that can inspire future approaches to this problem. A more accurate estimation of weights for the contribution of several sources of information can be carried out in future benchmarks, now that the VideoClef annotators have produced ground truth ranking data.