The semantic annotation of images can bene t from representations of useful concepts and the links between them as ontologies. Recently, several multimedia ontologies have been proposed in the literature as suitable knowledge models to bridge the well known semantic gap between low level features of image content and its high level conceptual meaning. Nevertheless, these multimedia ontologies are often dedicated to (or initially built for) particular needs or a particular application. Ontology matching, de ned as the process of relating di erent heterogeneous models, we will argue, is a suitable approach to solve interoperability issues in semantic image annotation and retrieval. We propose a generic instance-based ontology matching approach, applied to an important semantic image retrieval issue: the bridging of the semantic gap by matching a multimedia ontology against a common-sense knowledge resource.
The fast growth of shared digital image and video collections together with
the intensive use of visual information for decision making in many domains
(medicine, geosciences, etc) require new e ective methods for search and
retrieval in these collections. In order to enable and improve the communication
and the interface between humans and computers, it is necessary to understand
the semantic content of images and to built linguistic descriptions of their
content in an automatic way. Following decades of research on Content Based Image
Retrieval (CBIR), automatic image annotation is nowadays an active research
topic which aims at bridging the semantic and the perceptual levels of
abstraction, known as the Semantic gap problem [
With the growth of the application of ontology-based solutions in the
multimedia domain, a lot of interoperability issues have arisen: (a) At the semantic
level { between di erent representations of the same domain knowledge; (b) At
the visual level { between di erent multimedia ontologies; (c) Between the visual
level and the semantic level, i.e. the semantic gap problem. Ontology matching,
widely used for semantic web applications and rarely in the context of image
sharing and retrieval, that we de ned as the process of relating heterogeneous
knowledge models, can be used to solve these kinds of interoperability issues.
This paper proposes a generic approach to address the question of lling the
semantic gap by matching an ontology at the semantic level (Wordnet1
associated to the image database LabelMe[
Next section is a short review of existing multimedia ontologies and related approaches. Section 3 describes the ontology matching framework which forms the methodological background of our approach, presented in turn in Section 4. Results of our preliminary experiments are discussed in Section 5; Section 6 concludes. 2
In the past few years, concept-based multimedia retrieval has been a very active
research eld with a major e ort in the automatic detection of semantic concepts
from low level features with machine learning approaches. Despite these e orts,
the semantic gap problem is still an issue for the semantic understanding of
multimedia documents. Recently, many knowledge models have been proposed
to improve multimedia retrieval and interpretation by the explicit modeling of
the di erent relationships between semantic concepts. Indeed, many generic large
scale multimedia ontologies or multimedia concept lexicons together with image
collections have been proposed to improve multimedia search and retrieval by
providing an e ective representation and interpretation of multimedia concepts
[
This paper proposes to situate these problems in an O[ntology] M[atching]
framework. The OM-approach presented in next section is much in line with the
tradition of extensional matching. This comprises a set of techniques which base
the similarity of concepts on characteristics of the instances that these concepts
contain [
An ontology is based on a set of concepts and relations de ned on these concepts, which altogether describe the knowledge in a given domain of interest. Due to the fact that di erent communities, independently from one another, tend to conceptualize di erently the same domain of interest, a growing number of heterogeneous ontologies, describing similar or overlapping parts of the world are created. An OM procedure aims at reducing this heterogeneity by linking the correspondent elements of two ontologies in an automatic or semi-automatic manner.
Formally, a populated ontology will be de ned by O = fC; is_a; R; I; gg; where C is a set whose elements are called concepts, is_a is a partial order on C; R is a set of other (binary) relations holding between the concepts from the set C, I is a set whose elements are called instances and g : C ! 2I is an injection from the set of concepts to the set of subsets of I:
We note that the sets C and I are compulsorily non-empty, in contrast to R: Thus, the de nition above describes an ontology which, although not limited to subsumptional relations, necessarily contains a hierarchical backbone, de ned by the partial order. The set I may contain text documents, images or other (real world data) entities. By assumption, every instance can be represented as an n-dimensional real-valued vector, de ned by n input variables of some kind which are the same for all instances in I.
In the context of semantic image annotation, WordNet together with the
LabelMe database [
Often the outcome of an OM-procedure is a set of cross-ontology concept alignments, issued from a measure of concept similarity. The measures used in the current study are based on variable selection and we will describe them in more detail.
Variable selection techniques (reviewed in [
We de ne a binary classi cation training set SOc for each concept c from an ontology O by taking I; the entire set of instances assigned to O and labeling all instances from the set g(c) as positive and all the rest (Ing(c)) as negative. By the help of a variable selection procedure performed on SOc, we obtain a representation of the concept c as a list
L(c) = (sc1; sc2; :::; scn);
(1)
where sic is the score associated to the ith variable. To compute a score per
variable and per concept, we apply the S[upport] V[ector] M[achine]-based
variable selection technique introduced in [
By following the described procedure, given two source ontologies O1 and
O2; a representation as the one in (1) is made available for every concept of each
of these ontologies. The similarity of two concepts, A 2 O1 and B 2 O2 is then
assessed in terms of their corresponding representations L(A) and L(B): Several
choices of a similarity measure based on these representations are proposed and
compared in [
6 qPn i=1(siA sAmean)2qPn
i=1(siB Pn i=1(siA sAmean)(siB sBmean) sBmean)2
; P d2
i i n(n2 1) ;
nc simKendall = 12 n(n nd : 1) (2) (3) In the formulae above, sAmean and sBmean are the means of the scores over all input variables, di is the di erence of the ranks calculated for the ith variable w.r.t. the two concepts, and nc and nd are the numbers of concordant and discordant pairs among the lists of scores L(A) and L(B):
As noted in the introduction, many challenging issues in the eld of image
retrieval stem from the semantic gap problem. Two examples are the construction
of robust high level concept detectors and the creation of user oriented
annotations with high level semantics. In this section, we propose an attempt to
ll the semantic gap by matching two complementary resources: a visual and a
semantic thesaurus. Contrary to [
On one hand, we chose LSCOM [
In our setting, the instances that extensionally de ne a concept are images
whose annotations contain the name associated to this concept. An image is
represented as a vector of descriptors. We use a codebook built on a bag-of-features
model and histograms of codewords which is, nowadays, the best approach in
the state-of-the-art [
Aligning LSCOM to WordNet allows to infer knowledge about the LSCOM concepts (dedicated to the multimedia document annotation) with regard to the concepts of WordNet and the alignment could be used to build a linguistic description of the concepts of LSCOM, or, in other words, to answer the question \What is an LSCOM concept in WordNet?" in an automatic manner. This improves the retrieval process in several ways: (1) through query expansion and
reformulation, i.e. retrieving documents annotated with concepts from an ontology O1 using a query composed of concepts of an ontology O2, (2) through a better description of the documents in the indexing process. However, note that this relation is not symmetric: alignments in the other sense are prompt to fail to be of any help, since WordNet concepts are rather atomic (such as \car") as compared to the more complex LSCOM concepts (e.g. \Natural Disaster Scene"). 5
We use a part of the LSCOM ontology, LSCOM Annotation v1.03, which is a subset of 449 concepts from the initial LSCOM ontology, and is used for annotating 61,517 images from the TRECVID2005 development set. Since this set contains images from broadcast news videos, the choosen LSCOM subpart is particularly adapted to annotate this kind of content, thus contains abstract and speci c concepts (e.g. 196 Science Technology, 330 Interview On Location). To the contrary, our sub-ontology de ned from WordNet populated with LabelMe (3676 concepts) is very general considering the nature of LabelMe, which is composed of photographs from the daily life.
In this way, to provide a preliminary evaluation of the suggested approach, we chose three concepts from the LSCOM ontology and ve concepts from the WordNet ontology. The choice of the selected concepts was made on several criteria: (1) the number of associated instances, (2) for every selected concepts there is no semantic ambiguity in our dataset, (3) for WordNet only: a high con dence (arbitrarily decided) in the discrimination of the concept using only perceptual information.
To construct image features, we use a bag-of-features model with a visual codebook, built classically using the well known SIFT descriptor and a K-Means algorithm. The quanti cation of the extracted SIFT features was investigated in two ways: (1) over all the instances associated to the selected concepts (LSCOM and WordNet), (2) only over the LabelMe images and quanti cation per concept. The two experimentations gave very similar results, and the results of the experiment based on the rst codebook are resumed in Table 1.
The values in the rst three matrices are correlations indicating high similarity for positive values (low for non-positive). As we can see, the concept WordNet:TV is weakly correlated to the chosen LSCOM concepts, and the concept WordNet:House is highly correlated with LSCOM:Natural Disasters and LSCOM:Single Familly Homes but not with LSCOM:US Flags. This is coherent with the TRECVID2005 data considering that the images annotated with LSCOM:US Flags are mostly images from speeches of politicians during presidential elections. An example of an LSCOM image annotation that could be extended to WordNet by the help the concept mapping is given in Fig. 1. 6
The paper proposes an ontology matching technique to solve interoperability issues in the area of semantic image annotation and retrieval. In particular, we have addressed the problem of bridging the semantic gap by the help of a generic instance-based ontology matching approach which aims at automatically producing concept-based annotations enriched with a lexical description of the concepts. In preliminary experiments, we have tested a concept similarity measure on two small sets of concepts taken from the LSCOM ontology and WordNet/LabelMe. Our results are in good agreement with the nature of the instances associated to the selected LSCOM concepts. However, the e ciency of the approach has to be tested on larger sets of concepts (currently in progress). A large-scale application would also allow us to bene t from all the semantic relations in WordNet, like hypernymy, meronymy, antonymy. In the future, we plan to investigate the qualities of our automatic approach in terms of retrieval e ciency as compared to approaches that solely rely on manual mappings. Acknowledgments. This work is funded by the French National Research Agency (ANR) through the COSINUS program (project COLLAVIZ ANR-08-COSI-003) and by the region ^Ile de France through the SEBASTIAN2 project (Cap Digital cluster).