This paper presents our work in the field of semantic multimedia annotation and indexing with the use of complementary textual resources analysis. We describe the advantages of complementary sources of information as a support for annotation and test whether these data can be used for automatic annotation and event detection.
Paul Buitelaar, Thierry Declerck
In this paper we present our work using the complementary textual resources in video analysis. This, for the selected domain (soccer in our case) concerns various textual sources such as structured data (match tables with teams, player names, score goals, substitutions, etc.) and semi-structured, textual web data (minute-byminute match reports – unstructured text accompanied with temporal information). Events and entities detected in these sources are marked up with semantic classes derived from an ontology on soccer by use of information extraction tools. Since the target audience comes from various research areas, this text will be focused on the potential use of this approach rather than on the description of the details. Temporal alignment of primary video data (soccer match videos) with semantically organized events and entities from the textual and structured complementary resources can be used as indicator for video segment extraction and semantic classification; e.g. the occurrence of a 'Header' event in the complementary resources will be used to train and later classify the corresponding video segment accordingly. This information can then be used for semantic indexing and retrieval of events in soccer videos, but also for the targeted extraction of audio/visual (A/V) features (motion, audio-pitch, field-line, close-up). We denote such extraction of A/V features based on textual evidence "cross-media feature extraction".
There is quite a lot research effort carried out in the field of
semantic annotation and indexing in the sports domain.
Some of them, such as the work by [
The exploitation of related (complementary) textual
resources, especially when these are endowed with temporal
references can largely increase the quality of the video
analysis, indexing and retrieval. Of course the number of
domains containing freely available detailed temporal
descriptions is limited, but those where this information is
available in a large scale can be very effectively used.
Multiple parallel descriptions of one event will further
increase the coverage and eliminate false events. Good
examples can be found in the sports domain. Current
research in sports video analysis focuses on event
recognition and classification based on the extraction of
low-level features and is—when based solely on the
lowlevel features—limited to a very small number of different
event types, e.g. 'scoring-event' [
Major sports events, such as the FIFA Soccer World Cup
Tournament that was held in Germany in 2006, provide a
wide range of available textual resources, ranging from
semi-structured data in the form of tables on web sites to
textual summaries and other match reports. The video
material was analyzed independently of the research
described here, see [
A dataset for ontology-based [
Minute-by-minute reports are usually published at soccer
web sites and enable people to ’watch‘ the match in textual
form on the web. Processing several such reports in parallel
increases the coverage of events and eliminates false
positive events. We therefore rely on the following 6
different sources in this case: ARD, bild.de, LigaLive (all in
German), and Guardian, DW-World, DFB.de (all in
English); we apply the SProUT tool [
The aim of the semantic annotation is to allow
(semi)automatic detection of events in the video based on
previously learned examples. The aim of this experiment
was to test whether the general detectors are able to serve as
sufficient source of information. For this experiment we
used two manually-annotated soccer match videos, one as a
training set and the other for the tests. We created
additional derived features describing the previous and the
next values of the detectors in the same time range as the
event instance itself, providing us with better chance to
capture the behavior of the detector in time. We used
decision trees as machine learning algorithm and built up a
binary classifier for each of the observed events. The task of
the classifier was to decide whether the particular segment
is or is not an event. By our observation, the detectors we
used are too generic for fine-grained event detection but
they can help detect a certain event in a given (usually one
minute long) time range where the event was identified in
the text. The table below shows that different detectors are
important for different event types. This potentially allows
detecting instances of event types based on observing only
those detectors that are discriminative for them (this
assumption is also used by the decision tree algorithm). The
letters P, C, N represent Previous, Current or Next Value of
the detector for particular event type. More details can be
found in [
Results of the cross-media feature selection
We presented an approach to the use of resources that
are complementary to A/V streams, such as videos of
football matches, for the semantic indexing of such streams.
We further presented an experiment with event detection
based on general A/V detectors supported by textual
annotation. In [
This research was supported by the European Commission under contract FP6-027026 for the K-Space project. We thank D Sadlier and Noel O’Connor (DCU, Ireland) for providing the A/V data and analysis results.