=Paper=
{{Paper
|id=Vol-1172/CLEF2006wn-ImageCLEF-ChangEt2006
|storemode=property
|title=Approaches Using a Word-Image Ontology and an Annotated Image Corpus as Intermedia for Cross-Language Image Retrieval
|pdfUrl=https://ceur-ws.org/Vol-1172/CLEF2006wn-ImageCLEF-ChangEt2006.pdf
|volume=Vol-1172
|dblpUrl=https://dblp.org/rec/conf/clef/ChangC06a
}}
==Approaches Using a Word-Image Ontology and an Annotated Image Corpus as Intermedia for Cross-Language Image Retrieval==
https://ceur-ws.org/Vol-1172/CLEF2006wn-ImageCLEF-ChangEt2006.pdf
Approaches of Using a Word-Image Ontology and an Annotated Image
Corpus as Intermedia for Cross-Language Image Retrieval
Yih-Chen Chang and Hsin-Hsi Chen
Department of Computer Science and Information Engineering
National Taiwan University
Taipei, Taiwan
E-mail: ycchang@nlg.csie.ntu.edu.tw; hhchen@csie.ntu.edu.tw
Abstract
Two kinds of intermedia are explored in ImageCLEFphoto2006. The approach of using a word-image ontology
maps images to fundamental concepts in an ontology and measure the similarity of two images by using the
kind-of relationship of the ontology. The approach of using an annotated image corpus maps images to texts
describing concepts in the images, and the similarity of two images is measured by text counterparts using BM25.
The official runs show that visual query and intermedia are useful. Comparing the runs using textual query only
with the runs merging textual query and visual query, the latter improved 71%~119% of the performance of the
former. Even in the situation which example images were removed from the image collection beforehand, the
performance was still improved about 21%~43%.
ACM Categories and Subject Descriptors: H.3.3 [Information Storage and Retrieval]: Information Search and
Retrieval---Retrieval models, Relevance feedback
Keywords: Cross language image retrieval, cross-media translation, image annotation, ontology
1. Introduction
In recent years, many methods (Clough, Sanderson, & Müller, 2005; Clough, Müller, Deselaers, Grubinger,
Lehmann, Jensen, & Hersh, 2006) have been proposed to explore visual information to improve the performance
of cross-language image retrieval. The challenging issue is the semantic differences among visual and textual
information. For example, using t hev isuali nfor ma ti
on“ redc i
rcl
e”may retrieve noise images containing “ red
flower” , “red balloon”, “red ball”,an ds oon, rather than the desired ones containing “ sun” . The semantic
difference between visual concept “ redc i
rc l
e ”a n dtextual symbol “sun”is called a semantic gap.
Some approaches conducted text- and content-based image retrieval separately and then merged the results
of two runs (Besançon, et al., 2005; Jones, et al., 2005; Lin, Chang & Chen, forthcoming). Content-based image
retrieval may suffer from the semantic gap problem and report noise images. That may have negative effects on
the final performance. Some other approaches (Lin, Chang & Chen, forthcoming) learned the relationships
between visual and textual information and used the relationships for media transformation. The final retrieval
performance depends on the relationship mining.
In this paper, we use two intermedia approaches to capture the semantic gap. The main characteristic of
these approaches is that human knowledge is imbedded in the intermedia and can be used to compute the
semantic similarity of images. A word-image ontology and an annotated image corpus are explored and
compared. Section 2 specifies how to build and use the word-image ontology. Section 3 deals with the uses of
the annotated corpus. Sections 4 and 5 show and discuss the official runs in ImageCLEFphoto2006.
2. An Approach of Using a Word-Image Ontology
2.1 Building the Ontology
A word-image ontology is a word ontology aligned with the related images on each node. Building such an
ontology manually is time consuming. In ImageCLEFphoto2006, the image collection has 20,000 colored
images. There are 15,998 images containing English captions in and <DESCRIPTION> fields. The
vocabularies include more than 8,000 different words, thus an ontology with only hundreds words is not enough.
Instead of creating a new ontology from the beginning, we extend WordNet, the well-known word
ontology, to a word-image ontology. In WordNet, different senses and relations are defined for each word. For
simplicity, we only consider the first two senses and kind-of relations in the ontology. Because our experiments
in ImageCLEF2004 (Lin, Chang & Chen, 2006) showed that verbs and adjectives are less appropriate to be
represented as visual features, we only used nouns here.
Before aligning images and words, we selected those nouns in both WordNet and image collection based
on their TF-IDF scores. For each selected noun, we used Google image search to retrieval 60 images from the
web. The returned images may encounter two problems: (1) they may have unrelated images, and (2) the related
images may not be pure enough, i.e., the foci may be in the background or there may be some other things in the
images. Zinger (2005) tried to deal with this problem by using visual features to cluster the retrieved images and
filtering out those images not belonging to any clusters. Unlike Zinger (2005), we employed textual features. For
each retrieved image, Google will return a short snippet. We filter out those images whose snippets do not
exactly match the query terms. The basic idea is: “ themor et hing sa snippet mentions, the more complex the
image is.”Finally, we get a word-image ontology with 11,723 images in 2,346 nodes.
2.2 Using the Ontology
2.2.1 Similarity Scoring
Each image contains several fundamental concepts specified in the word-image ontology. The similarity of two
images is measured by the sum of the similarity of the fundamental concepts. In this paper we use kind-of
relations to compute semantic distance between fundamental concepts A and B in the word-image ontology. At
first, we find the least common ancestor (LCA) of A and B. The distance between A and B is the length of the
path from A through LCA to B. When computing the semantic distance of nodes A and B, the more the nodes
should be traversed from A to B, the larger the distance is. In addition to the path length, we also consider the
weighting of links in a path shown as follows.
(1) When computing the semantic distance of a node A and its child B, we consider the number of children
of A. The more children A has, the larger the distance between A and B is. In an extreme case, if A has only one
child B, then the distance between A and B is 0. Let #children(A) denote the number of children of A, and
base(A) denote the basic distance of A and its children. We define base(A) to be log(#children(A)).
(2) When computing the semantic distance of a node A and its brother, we consider the level it locates.
Assume B is a child of A. If A and B have the same number of brothers, then the distance between A and its
brothers is larger than that between B and its brothers. Let level(A) be the depth of node A. Assume the level
of root is 0. The distance between node A and its child, denoted by dist(A), is defined to be clevel ( A ) base( A) .
Here C is a constant between 0 and 1. In this paper, C is set to 0.9.
If the shortest path between two different nodes N0 and Nm is N0, N1, …,NLCA, …, Nm-1, Nm, we define the
distance between N0 and Nm to be:
m -1
dist ( N 0 , N m ) dist ( N LCA ) dist ( N i )
i 1
The larger the distance of two nodes is, the less similar the nodes are.
2.2.2 Mapping into the Ontology
Before counting the semantic distance between two given images, we need to map the two images into nodes of
the ontology. In other words, we have to find the fundamental concepts the two images consist of. A CBIR
system is adopted. It regards an image as a visual query and retrieves the top k fundamental images (i.e.,
fundamental concepts) in the word-image ontology. In such a case, we have two sets of nodes S1={n11, n12,
n13, …,n1k} and S2={n21, n22, n23, …, n2k}, which correspond to the two images, respectively. We define the
following formula to compute the semantic distance:
k
SemanticDistance( S1 , S2 ) min(dist (n1i , n2 j )), where j 1,..., k
i 1
Given a query with m example images, we regard each example image Q as an independent visual query,
and compute the semantic distance between Q and images in the collection. Note that we determine what
concepts are composed of an image in the collection before retrieval. After m image retrievals, each image in the
collection has been assigned m scores based on the above formula. We choose the best score for each image and
sort all the images to create a rank list. Finally, the top 1000 images in the rank list will be reported.
3. An Approach of Using an Annotated Image Corpus
An annotated image corpus is a collection of images along with their text annotations. The text annotation
specifies the concepts and their relationships in the images. To measure the similarity of two images, we have to
know how many common concepts there are in the two images. An annotated image corpus can be regarded as
a reference corpus. We submit two CBIRs to the reference corpus for the two images to be compared. The
corresponding text annotations of the retrieved images are postulated to contain the concepts embedded in the
two images. The similarity of text annotations measures the similarity of the two images indirectly.
The image collection in ImageCLEFphoto2006 can be considered as a reference annotated image corpus.
Using image collection itself as intermedia has some advantages. It is not necessary to map images in the image
collection to the intermedia. Besides, the domain can be more restricted to peregrine pictures. In the
experiments, the <DESCRIPTION>, <NOTE>, and <LOCATION> fields in English form the annotated corpus.
To use the annotated image corpus as intermedia to compute similarity between example images and
images in image collection, we need to map these images into intermedia. Since we use the image collection
itself as intermedia, we only need to map example images in this work. An example image is considered as a
visual query and submitted to retrieve images in intermedia by a CBIR system. The corresponding text
counterparts of the top returned k images form a long text query and it is submitted to an Okapi system to
retrieval images in the image collection. BM25 formula measures the similarity between example images and
images in image collection.
4. Experiments
In the formal runs, we submitted 25 cross-lingual runs for eight different query languages. All the queries with
different source languages were translated into target language (i.e., English) using SYSTRAN system. We
considered several issues, including (1) using different intermedia approaches (i.e., the text-image ontology and
the annotated image corpus), and (2) with/without using visual query. In addition, we also submitted 4
monolingual runs which compared (1) the annotation in English and in German, and (2) using or not using visual
query and intermedia. At last, we submitted a run using visual query and intermedia only. The details of our runs
are described as follows:
(1) 8 cross-lingual and text query only runs:
NTU-PT-EN-AUTO-NOFB-TXT, NTU-RU-EN-AUTO-NOFB-TXT,
NTU-ES-EN-AUTO-NOFB-TXT, NTU-ZHT-EN-AUTO-NOFB-TXT,
NTU-FR-EN-AUTO-NOFB-TXT, NTU-JA-EN-AUTO-NOFB-TXT,
NTU-IT-EN-AUTO-NOFB-TXT, and NTU-ZHS-EN-AUTO-NOFB-TXT.
These runs are regarded as baselines and are compared with the runs using both textual and visual
information.
(2) 2 monolingual and text query only runs:
NTU-EN-EN-AUTO-NOFB-TXT, and NTU-DE-DE-AUTO-NOFB-TXT.
These two runs serve as the baselines to compare with cross-lingual runs with text query only, and to
compare with the runs using both textual and visual information.
(3) 1 visual query only run with the approach of using an annotated image corpus:
NTU-AUTO-FB-TXTIMG-WEprf.
This run will be merged with the runs using textual query only, and is also a baseline to compare
with the runs using both visual and textual queries.
(4) 8 cross-lingual runs, using both textual and visual queries with the approach of an annotated corpus:
NTU-PT-EN-AUTO-FB-TXTIMG-T-WEprf, NTU-RU-EN-AUTO-FB-TXTIMG-T-WEprf,
NTU-ES-EN-AUTO-FB-TXTIMG-T-WEprf, NTU-FR-EN-AUTO-FB-TXTIMG-T-WEprf,
NTU-ZHS-EN-AUTO-FB-TXTIMG-T-WEprf, NTU-JA-EN-AUTO-FB-TXTIMG-T-WEprf,
NTU-ZHT-EN-AUTO-FB-TXTIMG-T-Weprf, and NTU-IT-EN-AUTO-FB-TXTIMG-T-Weprf.
These runs merge the textual query only runs in (1) and visual query only run in (3) with equal
weight.
(5) 8 cross-lingual runs, using both textual and visual queries with the approach of using word-image
ontology:
NTU-PT-EN-AUTO-NOFB-TXTIMG-T-IOntology,
NTU-RU-EN-AUTO-NOFB-TXTIMG-T-IOntology,
NTU-ES-EN-AUTO-NOFB-TXTIMG-T-IOntology,
NTU-FR-EN-AUTO-NOFB-TXTIMG-T-IOntology,
NTU-ZHS-EN-AUTO-NOFB-TXTIMG-T-IOntology,
NTU-JA-EN-AUTO-NOFB-TXTIMG-T-IOntology,
NTU-ZHT-EN-AUTO-NOFB-TXTIMG-T-IOntology, and
NTU-IT-EN-AUTO-NOFB-TXTIMG-T-IOntology.
These runs merge textual query only runs in (1), and visual query runs with weights 0.9 and 0.1.
(6) 2 monolingual runs, using both textual and visual queries with the approach of an annotated corpus:
NTU-EN-EN-AUTO-FB-TXTIMG, and NTU-DE-DE-AUTO-FB-TXTIMG
These two runs using both textual and visual queries. The monolingual run in (2) and the visual run
in (3) are merged with equal weight.
5. Results and Discussions
Table 1 shows experimental results of official runs in ImageCLEFphoto2006. We compare performance of the
runs using textual query only, and the runs using both textual and visual queries (i.e., Text Only vs. Text +
Annotation and Text + Ontology). In addition, we also compare the runs using word-image ontology and the runs
using annotated image corpus (i.e., Text + Ontology vs. Text + Annotation). The runs whose performance is
better than that of baseline (i.e., Text Only) will be marked in bold. The results show all runs using annotated
image corpus are better than the baseline. In contrast, only two runs using word-image ontology are better.
Table 1. Performance of Official Runs
Query Language MAP Description Runs
Portuguese 0.1630 Text Only NTU-PT-EN-AUTO-NOFB-TXT
0.2854 Text + Annotations NTU-PT-EN-AUTO-FB-TXTIMG-T-WEprf
0.1580 Text + Ontology NTU-PT-EN-AUTO-NOFB-TXTIMG-T-IOntology
Russian 0.1630 Text Only NTU-RU-EN-AUTO-NOFB-TXT
0.2789 Text + Annotations NTU-RU-EN-AUTO-FB-TXTIMG-T-Weprf
0.1591 Text + Ontology NTU-RU-EN-AUTO-NOFB-TXTIMG-T-IOntology
Spanish 0.1595 Text Only NTU-ES-EN-AUTO-NOFB-TXT
0.2775 Text + Annotations NTU-ES-EN-AUTO-FB-TXTIMG-T-Weprf
0.1554 Text + Ontology NTU-ES-EN-AUTO-NOFB-TXTIMG-T-IOntology
French 0.1548 Text Only NTU-FR-EN-AUTO-NOFB-TXT
0.2758 Text + Annotations NTU-FR-EN-AUTO-FB-TXTIMG-T-WEprf
0.1525 Text + Ontology NTU-FR-EN-AUTO-NOFB-TXTIMG-T-IOntology
Simplified Chinese 0.1248 Text Only NTU-ZHS-EN-AUTO-NOFB-TXT
0.2715 Text + Annotations NTU-ZHS-EN-AUTO-FB-TXTIMG-T-Weprf
0.1262 Text + Ontology NTU-ZHS-EN-AUTO-NOFB-TXTIMG-T-IOntology
Japanese 0.1431 Text Only NTU-JA-EN-AUTO-NOFB-TXT
0.2705 Text + Annotations NTU-JA-EN-AUTO-FB-TXTIMG-T-Weprf
0.1396 Text + Ontology NTU-JA-EN-AUTO-NOFB-TXTIMG-T-IOntology
Table 1. Performance of Official Runs (Continued)
Traditional Chinese 0.1228 Text Only NTU-ZHT-EN-AUTO-NOFB-TXT
0.2700 Text + Annotations NTU-ZHT-EN-AUTO-FB-TXTIMG-T-Weprf
0.1239 Text + Ontology NTU-ZHT-EN-AUTO-NOFB-TXTIMG-T-IOntology
Italian 0.1340 Text Only NTU-IT-EN-AUTO-NOFB-TXT
0.2616 Text + Annotations NTU-IT-EN-AUTO-FB-TXTIMG-T-Weprf
0.1287 Text + Ontology NTU-IT-EN-AUTO-NOFB-TXTIMG-T-IOntology
Since the example images in this task are in the image collection, the CBIR system always correctly maps
the example images into themselves at mapping step. We made some extra experiments to examine the
performance of our intermedia approach. In the experiments, we took out the example images from the image
collection when mapping example images into intermedia. Table 2 shows the experiment results. Although the
performance of Table 2 is lower than that of Table 1, the runs using the approach of annotated image corpus are
still better than the runs using textual query only.
Table 2. Performance of Runs by Removing Example Images from the Collection (Unofficial Runs)
Query Language MAP Description Runs
Portuguese 0.1630 Text Only NTU-PT-EN-AUTO-NOFB-TXT
0.1992 Text + Annotations NTU-PT-EN-AUTO-FB-TXTIMG-T-WEprf-NoE
Russian 0.1630 Text Only NTU-RU-EN-AUTO-NOFB-TXT
0.1880 Text + Annotations NTU-RU-EN-AUTO-FB-TXTIMG-T-Weprf-NoE
Spanish 0.1595 Text Only NTU-ES-EN-AUTO-NOFB-TXT
0.1928 Text + Annotations NTU-ES-EN-AUTO-FB-TXTIMG-T-Weprf-NoE
French 0.1548 Text Only NTU-FR-EN-AUTO-NOFB-TXT
0.1848 Text + Annotations NTU-FR-EN-AUTO-FB-TXTIMG-T-WEprf-NoE
Simplified Chinese 0.1248 Text Only NTU-ZHS-EN-AUTO-NOFB-TXT
0.1779 Text + Annotations NTU-ZHS-EN-AUTO-FB-TXTIMG-T-Weprf-NoE
Japanese 0.1431 Text Only NTU-JA-EN-AUTO-NOFB-TXT
0.1702 Text + Annotations NTU-JA-EN-AUTO-FB-TXTIMG-T-Weprf-NoE
Traditional Chinese 0.1228 Text Only NTU-ZHT-EN-AUTO-NOFB-TXT
0.1757 Text + Annotations NTU-ZHT-EN-AUTO-FB-TXTIMG-T-Weprf-NoE
Italian 0.1340 Text Only NTU-IT-EN-AUTO-NOFB-TXT
0.1694 Text + Annotations NTU-IT-EN-AUTO-FB-TXTIMG-T-Weprf-NoE
Table 3 shows the experiment results of monolingual runs. Using both textual and visual queries are still
better than runs using textual query only. The performance of the runs by taking out the example images from
collection beforehand is still better than the runs use textual query only. Comparing with Tables 1 and 2, we can
find some cross-lingual runs that use textual and visual queries are even better than monolingual run that use
textual query only. That means visual information is very important when doing cross language image retrieval.
Table 4 shows the experiment of runs using visual query only. When example images were kept in the
image collection, we can always map the example images into the right images. Therefore, the translation from
visual information into textual information will be more correctly. The experiment shows the performance of
visual query runs is better than that of textual query runs when the transformation is correct.
Table 3. Performance of Monolingual Image Retrieval
Query Language MAP Description Runs
English 0.1787 Text Only NTU-EN-EN-AUTO-NOFB-TXT
(+example images) 0.2950 Text + Annotations NTU-EN-EN-AUTO-FB-TXTIMG
(-example images) 0.2027 Text + Annotations NTU-EN-EN-AUTO-FB-TXTIMG-NoE (unofficial)
German 0.1294 Text Only NTU-DE-DE-AUTO-NOFB-TXT
(+example images) 0.3109 Text + Annotations NTU-DE-DE-AUTO-FB-TXTIMG
(-example images) 0.1608 Text + Annotations NTU-DE-DE-AUTO-FB-TXTIMG-NoE (unofficial)
Table 4. Performance of Visual Query
MAP Description Runs
0.1787 Text Only (monolingual) NTU-EN-EN-AUTO-NOFB-TXT
0.2757 Visual Only (+example images) NTU-AUTO-FB-TXTIMG-Weprf
0.1174 Visual Only (-example images) NTU-AUTO-FB-TXTIMG-Weprf-NoE (unofficial)
6. Conclusion
The experiments show visual query and intermedia approaches are useful. Comparing the runs using textual
query only with the runs merging textual query and visual query, the latter improved 71%~119% of performance
of the former. Even in the situation which example images are removed from the image collection, the
performance can still be improved about 21%~43%. We find visual query in image retrieval is important. The
performance of the runs using visual query only can be even better than the runs using textual only if we
translate visual information into textual one correctly. In this year the word-image ontology built automatically
still contain much noise. We will investigate how to filter out the noise and explore different methods.
Acknowledgments
Research of this paper was partially supported by National Science Council, Taiwan, under the contracts
NSC94-2213-E-002-076 and NSC95-2752-E-001-001-PAE.
References
1. Besançon, R., Hède, P., Moellic, P.A., & Fluhr, C. (2005). Cross-media feedback strategies: Merging text and image
information to improve image retrieval. In Peters, C.; Clough, P.; Gonzalo, J.; Jones, G.J.F.; Kluck, M.; Magnini, B.
(Eds.), Proceedings of 5th Workshop of the Cross-Language Evaluation Forum, LNCS 3491, (pp. 709-717). Berlin:
Springer.
2. Clough, P., Sanderson, M. & Müller, H. (2005). The CLEF 2004 cross language image retrieval track. In Peters, C.;
Clough, P.; Gonzalo, J.; Jones, G.J.F.; Kluck, M.; Magnini, B. (Eds.), Proceedings of 5th Workshop of the Cross-Language
Evaluation Forum, LNCS 3491, (pp. 597-613). Berlin: Springer.
3. Clough, P., Müller, H., Deselaers, T., Grubinger, M., Lehmann, T.M., Jensen, J., & Hersh, W. (2006). The CLEF 2005
cross-language image retrieval track, Proceedings of 6th Workshop of the Cross Language Evaluation Forum, Lecture
Notes in Computer Science, 2006.
4. Jones, G.J.F., Groves, D., Khasin, A., Lam-Adesina, A., Mellebeek, B., & Way, A. (2005). Dublin City University at
CLEF 2004: Experiments with the ImageCLEF St Andrew's Collection. In Peters, C.; Clough, P.; Gonzalo, J.; Jones,
G.J.F.; Kluck, M.; Magnini, B. (Eds.), Proceedings of 5th Workshop of the Cross-Language Evaluation Forum, LNCS
3491, (pp. 653-663). Berlin: Springer.
5. Lin, W.C., Chang, Y.C. and Chen, H.H. (forthcoming) .“ Integ rat
ingTe xtualandVi sua lInfor
ma t
ionf orCr oss-Language
Image Retrieval: A Trans-Me diaDi c ti
onaryAppr oach.
”Information Processing and Management, Special Issue on Asia
Information Retrieval Research.
6. Zing er, S. ( 2005).“ Ext
ra cti
n g a n Ont ology of Por tabl
e Obj ectsf r
om Wor dNet.
” Proceedings of the
MUSCLE/ImageCLEF Workshop on Image and Video Retrieval Evaluation.
</pre>