In this paper we proposed an approach to deal with the Chinese-English cross-language image retrieval problem. Text-based image retrieval and query translation were adopted in the experiments. A similarity-based backward transliteration model with candidate filter was proposed to translate the proper nouns. The experimental results showed that using similarity-based backward transliteration increased the retrieval performances.
2.1 that they are of equal length and denoted as S1’ and S2’. S1’ and S2’ are aligned when every character in either string is opposite a unique character or space in the other string. The similarity score of an alignment is measured using the following formula.
l Score = ∑ s(S1 ' (i), S2 ' (i)) (1) i=1 where s(a, b) is the similarity score between the character a and b in Σ’,
S’(i) is the ith character in the string S’, and l is the length of S1’ and S2’.
The similarity score s(a, b) can be manually assigned or automatically learned.
The optimal alignment of two strings S1 and S2 can be computed efficiently using dynamic programming. Let T is an n+1 by m+1 table, where n and m are the length of S1 and S2 respectively. By filling table T row by row, we can obtain the optimal alignment and the similarity score of S1 and S2. The base condition is defined as follows.
T (i,0) = ∑ s(S1 (k ), ' _' ) 1≤k≤i T (0, j) = ∑ s(' _', S2 (k ))
1≤k≤ j The recurrence formula is defined as follows.
where
1≤ i ≤ n, 1 ≤ j ≤ m.
T (i, j) = max[T (i −1, j −1) + s(S1(i), S2 ( j), T (i −1, j) + s(S1(i),' _'), T (i, j −1) + s(' _', S2 ( j)] (2) (3) 2.2
Candidate Filter Similarity based backward transliteration with automatically learned phonetic similarities works well, but it will cost too much time if there are a lot of candidate words. That is not suitable for some applications like online IR systems. To reduce processing time, we take a pre-process to decrease the number of candidates. A transliterated word and its original word should contain the same or the similar phonemes, and the order of the phonemes are the same. In other words, if two IPA strings contain more identical or similar characters, their similarity may be higher. A vector space IR model is adopted to select the appropriate candidates for a transliterated word. The document set is the IPA strings of a list of proper nouns in source language. Each proper noun is treated as one document. The query is the IPA string of the transliterated word. After retrieving, the top ranked documents (candidate words) are selected as the appropriate candidates of the transliterated word.
The transliterated word and its original word do not always contain the same phonemes due to the different
pronunciation of different languages. For example, the English phoneme ‘g’ is usually transliterated into
Chinese phoneme ‘k’1. If only the phonemes of the transliterated word are used as the query terms, the original
word may not be retrieved. Thus, the query has to be expanded with the most co-transliterated phonemes. The
co-transliterated Chinese-English phoneme pairs are trained from a Chinese-English person name corpus, which
has 51,114 pairs of Chinese transliterated names and the corresponding English original names. A variance of
Mutual Information is adopted to measure the strength of co-transliteration of two phonemes. The variant
Mutual Information can solve the preferring rare terms problem that traditional Mutual Information
MI ( x, y) = log p( x, y)
× log( f ( x, y)) p( x) p( y) p(x) is the occurrence probability of phoneme x in Chinese names, p(y) is the occurrence probability of phoneme y in English names, p(x, y) is the probability of x and y that occur in a pair of transliterated and original name, and f(x, y) is the frequency of x and y that occur in a pair of transliterated and original name. (4) 1 All phonemes are represented in SAMPA, which can represent IPA in ASCII.
A phoneme x in a transliterated word will be expanded with the phonemes that have positive MI values with x. The augmented phonemes are weighted by MI(x, y)/the number of augmented terms. 3
In the experiments, Chinese queries were used as the source language queries. The Chinese queries are
translated from English by native speakers. We adopted query translation to unify the languages of queries and
documents. First, the Chinese queries were segmented by a word recognition system, tagged by a POS tagger
and name entities were identified
(1) CO model (Chen, et al., 1999)
CO model employed word co-occurrence information trained from a target language text collection to
disambiguate the translations of query terms. We adopted Mutual Information (MI)
The first two translation equivalents with the highest occurrence frequency in the English image captions were considered as the target language query terms.
There are 150 distinct Chinese query terms in 50 topics. Total 16 of the 150 query terms could not be found
in our dictionary. Among the 16 terms, 7 terms were tagged as person names, and 5 terms were location names.
These names are Chinese translations of foreign names. We can use backward transliteration scheme to translate
these names. First, we adopted the transformation rules
(1) The person names and the location names in the English image captions were extracted. We collected a list of English names that contained 50,979 person names and 19,340 location names. If a term in the captions can be found in the name list, it was extracted. Total 3,599 names were extracted from the image captions. (2) For each Chinese name, 300 candidates were selected from the 3,599 English names by using the candidate filter described in Section 2.2. (3) The similarity-based backward transliteration approach described in Section 2.1 was adopted to translate the Chinese name. Top 6 candidates with the highest similarities were considered as the translations of the Chinese name.
In the segmentation and name identification stage, some terms were segmented or tagged incorrectly. These errors propagated to the translation stage and affected the performance of backward transliteration. In order to evaluate the real performance of similarity-based backward transliteration, we conducted manual runs in which the Chinese queries were segmented and tagged manually. In the manual runs, there are 136 distinct terms and 18 terms have no translations. Among the 18 terms, 5 terms were tagged as person names, 9 terms were location names and 1 term was an organization name.
Run NTUiaCo NTUiaCoP NTUiaF2hf NTUiaF2hfP NTUimCo NTUimCoP NTUimF2hf NTUimF2hfP Run NTUiaCo NTUiaCoP NTUiaF2hf NTUiaF2hfP NTUimCo NTUimCoP NTUimF2hf NTUimF2hfP
Manually 0.1712 0.1892 0.2635 0.2888 0.1985 0.2241 0.2821 0.3143
In the experiments, we adopted text-based approach. The captions were used to represent the images. Okapi IR
system
We submitted eight runs in CLEF 2003 image track. The performances of two query translation methods with or without similarity-based backward transliteration were compared. The details of the submitted runs are shown in Table 1. The performances are shown in Table 2.
The results show that using similarity-based backward transliteration to translate proper nouns increases performances. In the automatic segmentation runs, twelve topics have proper nouns that were not contained in our dictionary. After applying similarity-based backward transliteration model, the proper nouns in six topics were translated correctly, and the average precisions of these topics were increased dramatically. The performances of the twelve topics are shown in Table 3. Terms in square brackets are keywords extracted by transformation rules. In Topic 16, “丹地” (dan di), transliterated from “Dundee”, was not translated correctly due to the error of keyword extraction. “丹地” was tagged as a location name, and “地” (di) was identified as a keyword according to the transformation rules. By looking up the dictionary, “地” (di) was translated into “field” and “ground”. Only “丹” (dan) was transliterated by similarity-based backward transliteration and the similarity between “丹” (dan) and “Dundee” is low. Five terms were segmented incorrectly, so that they were transliterated incorrectly. In manual segmentation runs, segmentation error problem is excluded. Total 14 topics contain proper nouns, which were not in our dictionary. The performances of seven and ten topics were increased after applying similarity-based backward transliteration model followed CO and first-two-highestfrequency models, respectively. The performances are shown in Table 4. Examining the results of backward transliteration, the original English words of about 50% Chinese proper nouns were in the top 6 ranks. Recall that the top 6 ranked terms were added to the query. At most one term was the correct word and the others were noises. Although noises were introduced, the performances of the topics in which the proper nouns were backward transliterated correctly were improved. If the performance of backward transliteration is improved and fewer incorrect terms are added to the queries, the retrieval performance should be better. We also found that the original words of four Chinese transliterated words were not included in the name list. Thus, these original English words were not contained in the candidate lists. How to enlarge the coverage of the name list is also an important issue.
Comparing the performances of two query translation models, surprisingly, CO model was worse than the first-two-highest-frequency model. In CO model, only one translation equivalent is selected for a query term. Since the captions of the images are very short, the suggested English translation may be not used in the captions. If we expand queries or captions, the performance may be better. On the other hand, the first-two-highestfrequency model selects the translations with the highest frequency in the target documents. Most of the English translated query terms present in the captions. The term usage is more consistent in the first-two-highestfrequency model.
[大]亞茅斯 (Great Yarmouth) Has no entry in the name list 卡羅斯 (Culross) nHaams en olisetntry in the 亨利耶塔 (Henrietta) nHaams en olisetntry in the 吉爾摩 (Gilmour) Correct (rank 2) 勃恩斯 (Burns) Correct (rank 3) 尼維[峰] (Nevis ben) 湯普森 (Thompson) 瓦倫坦 (Valentine)
Keyword extraction error Correct (rank 2)
In this paper we proposed an approach to deal with the Chinese-English cross-language image retrieval problem. Text-based image retrieval and query translation were adopted in the experiments. A similarity-based backward transliteration model with candidate filter was proposed to translate the proper nouns. The experimental results showed that using similarity-based backward transliteration increased the retrieval performances. The average precisions of about 50% topics consisting of proper nouns were increased. The performances of the rest topics were decreased due to the failure of backward transliteration. The errors were caused by segmentation error, named entity identification error, keyword extraction error, and the coverage of the name list. Several methods such as learning phoneme similarity from larger data, extracting more named entities from target document set, and improving the performance of candidate filter and keyword extraction will be further investigated to improve the performance of the similarity-based backward transliteration.
The consistency of term usages is also an important issue. The image captions are usually short and the words used in captions are limited. Query expansion or document expansion could resolve this problem. We will experiment with various expansion approaches in the future.