Advanced semantic description of multimedia data signi¯cantly improves representing, labeling, and retrieving multimedia-based contents. In this paper we present an intelligent framework for attaching semantic annotations to image contents based on the extraction of elementary low-level features, user's relevance feedback and the usage of ontology knowledge. This approach facilitates image annotation by proposing a set of most likely relevant content descriptors as a result of extracted image features and the prior annotation of similar images. We illustrate how the speci¯c components of our architecture interact in order to provide a °exible annotation schema and a learning-based annotation mechanism.
Since the amount of unstructured image and multimedia content is increasing nowadays, e±cient methods for indexing, querying and browsing such information, and the recognition of relevant patterns become more and more essential. In comparison to text retrieval techniques there are even more problems with image retrieval. Particularly, the semantic gap between low-level visual features of images and high-level human perception of inferred semantic contents decreases the performance of traditional content-based image retrieval systems.
Various content-based image retrieval (CBIR) approaches have been
introduced in past years, e.g. in [
On the other hand, users are highly interested in querying images at the
conceptual and semantic level, not only in terms of features like color, texture,
or shape [
Presently, most of the image database systems utilize manual annotation [
In this paper we propose an intelligent framework for image annotation which satis¯es the requirements of advanced multimedia information systems by combining the analysis of visual content and the manually performed description of image data. In Section 2 we give a motivation of our work and introduce levels of image representation. Section 3 introduces the architecture of our system and describes the interaction of the components. The annotation schema and steps required to generate a semantic annotation template are detailed in Section 4. 2
The core element of an image retrieval system is the underlying knowledge
representation model qualifying the structure and contents of the underlying data. An
image object I is modeled as a composition of two layers: the physical and the
logical layer. Physical image representation RP (I) is related to raw image data
obtained during the image input or storage and includes the image described
by a bitmap, that is stored as an array of pixel values. The logical image
representation RL(I) serves as an abstraction of the physical image representation,
which is subdivided into multiple representation levels: At the bottom of the
hierarchy an image object I is represented by a set FI = ffig of primitive visual
features. For every given feature fi, there exists a corresponding set RI = frij g
of representations [
The principal objective of our annotation system is to provide users a retrieval system with the capacity to evaluate image classi¯cation, assign the data to prede¯ned categories and thus its association with extensive descriptors from existing ontologies. Furthermore, by analyzing the logical structure of already annotated images and interactive user's feedback it will be feasible to provide a semi-automatic annotation which generates an object description template and thus proposes the membership of the data to a prede¯ned category. The proposed architecture is illustrated in Figure 1 and consists of the following components: Visualization Component. This component comprises the graphical user interface. It consists of a image data display and a results display, which creates thumbs from a subset of images belonging to one category. Furthermore, the graphical user interface provides a visualization of the semantic knowledge tree and the properties of its nodes used in the image annotating process. Object Ontology n o itt a n e m g e s l a u n a m
Retrieval Component keyword description region description
Annotation
Visualization SemSapnactiec
Meta Data
Description Component Logical
Feature Extraction Segmentation Component
Image Regions Retrieval Component. This component is responsible for the whole retrieval process. Beginning with query formulation and interpretation, which is performed by parsing and compiling the query into an internal format, the component provides also functions for similarity computation between the query object and the underlying data items stored in the database. Feature Extraction Component. Methods for extracting primitive visual characteristics of an image are provided by this component.
Segmentation Component. In order to divide images into objects, a set of segmentation algorithms is provided by the Segmentation Component. Since our system is arranged to involve user's perception, this component proceeds either interactively or automatically.
Description Component. Content descriptions of the images is stored in a logical database. This component also provides methods for description matching which compute the overall similarity between the content description of a query image and the content descriptions of images in our collection. Annotation Component. The annotation component provides a template for interactively attaching images with semantic descriptions. This template covers records for object description with a structured set of object properties, object activities and relations between objects.
Semantic Concept Space. The semantic base results from a projection of the image feature space into a variable set of concepts and their qualitative characteristics from existing ontologies which can be used for generating suitable annotation patterns. This ¯xed ontology tries to obviate the inconsistency of keyword assignments among di®erent indexers. These subject concepts are stored in a C-dimensional concept space which represents their weighted properties and the weighted relationships to other subject concepts in di®erent application domains.
Knowledge Base. The semantic knowledge of di®erent application domains are captured in this module. Furthermore, the visual characteristics associated with the used concepts are required in order to compute the mapping between low-level and high-level semantics, and for the prediction of a suitable description to be suggested by the annotation component.
Basic Metadata. Metadata contains standard information of the image raw data, like date, the photographers name, or the ¯lename.
Description Component. In this module, methods for similarity computation between di®erent content descriptions are provided. 4
In order to capture all required information about the semantic meaning of an image, multi-level descriptors, the so-called keywords have to be utilized. Keywords appear on several abstraction levels: the visual appearance and structure of the image is described in terms of regions and their spatial relations. For that purpose, the image is partitioned { automatically or manually { into content bearing segments comprising objects including their type, identity and possible activity. The resulting semantic classi¯cation of the image, later named as semantic class, is recorded as the root of the hierarchical description structure. The annotation AI of an image I consists of a sequence of keywords ci; : : : cn, which selection depends on the presence of the concepts ci in the image. In addition, the sequence contains d implizite descriptors (imdescriptors) specifying the meaning of image contents recognized by humans.
The semi-automatic annotation mapping can be formulated as follows:
Input: Set of training examples T = ft1; t2 : : : trg where ti = (FI ; AI ) are tuples representing low-level features FI and the corresponding annotation AI of an image.
Output: Suitable template for labeling the image I with a set of keywords ci; : : : cn which are ordered by the relevance in the image.
The functionality of the algorithm should include the determination and update
of correspondences between low-level features of image segments and their
annotations. Afterwards, information about the derived semantic classes and their
representative low-level characteristics should be attached to the Semantic
Concept Space. The clustering of image data is performed both at the low level
and the semantic level. For clustering images on the semantic level additional
knowledge about the characteristics of image features is used. Since there are
many low-level features for every image, an appropriate set of relevant features
has to be chosen. For this purpose, we use a modi¯cation of Subspace Clustering
[
In our approach the initial semantic categories of images are speci¯ed by supervised classi¯cation using the training set T . As a result, each semantic category is speci¯ed by a representative vector (prototype vector p^) which is updated during the relevance feedback loop. This vector can be considered to accurately represent overall characteristics of the images that belong to the same category. The i-th component of a prototype vector p^ of the category c is computed by p^i = j1cj Px2c fi(x); where fi denotes the i-th component of the feature vector of an image x 2 c and jcj denotes the number of images in the category c. To perform a selection of a subspace of the feature components, a weighting of the components relevant for the distinction between other categories is needed. As a general rule, local and global criteria are combined for weighting. Let the image database consist of N images, and let xj be the j-th image. Let fi be one feature that is essentially representative for a category of images or for a class cm. The weighting wi of the component i of a prototype vector p^ is computed as follows: wi = freq(fi; cm) log¡
N occ(fi; C) ¢: (1) where the feature frequency freq(fi; cm) represents the occurrence of feature fi in images of class cm and occ(fi; C) denotes the occurrence of this feature fi within other classes C = fc1; :::; cm¡1; cm+1; :::; cng. During the retrieval and annotation session the weights wi of the prototype vector p^ are updated by taking into account a newly classi¯ed image xnew. If required, additional factors ® and ¯ can be assigned manually by the user in order to give important features a stronger weighting or eliminate non-relevant features.
The relevance feedback technique is used to bridge the gap between low-level
features and high-level semantics in retrieval systems. The user can re¯ne results
by using negative and positive examples and update the knowledge about image
classes in the semantic space. Each time a feedback or a new annotation is
provided by the user, the prototype vector p^, the concept space (subjects concepts
and their relationships), and a semantic template for image annotation have to
be updated. Let us assume, that an initial concept space of c1 : : : cn concept
classes and their low-level characteristics have been created interactively. The
next step is to de¯ne rules for mapping each semantic class to a Semantic
Annotation Template, which has the following properties: in should provide entries
for general entities like agents and objects, their relations, time, place, and
activity. During the training of the retrieval system, correspondences between
concept classes and the layout of the template are captured. Furthermore, the
template ¯elds are associated with a concept thesaurus (ontology) derived from
WordNet [
Currently, the annotation schema is extended within some students' projects for attaching image data with lexical information from ontologies. In future work we plan retrieval performance experiments (precision vs. recall) for the semantic query level and the investigation of the accuracy of the semantic description. Furthermore, the de¯nition of ontologies for speci¯c application domains and the adaptation of existing ones in order to enhance the semantic description of our image data is another important aspect.