This paper describes the participation of the Web Science Laboratory of Meiji University in the ImageCLEF2010 photo annotation task. We devised a system that combines conceptual fuzzy sets with visual words that have become popular in the fields of image retrieval and recognition. The utility of the system was verified by comparing it with the ordinary Bag of Visual words technique. In addition, we constructed a system using Flickr User Tags that was based on the same idea. Because there are many images without tags, the system integrates visual word and tag-based methods. The utility of the integrated system was verified in an experiment.
A Conceptual Fuzzy Set (CFS) [
First, the keypoints are extracted from an image by using SIFT [
The Visual Words Combination Approach (VWCA) is a model that the CFS applies to visual words. Several visual words are combined in order to decrease their vagueness. The Bag of Visual words technique was originally proposed as an analogy of the Bag of Words approach, and visual words have the same meaning as words in sentences. Thus, CFS, which is useful for sentence retrieval, might also be useful for visual words. Moreover, this idea considers human psychology; someone visually recognizes an object by seeing many points at the same time. For instance, we recognize someone’s face by seeing the eyes, nose, and mouth at the same time.
The Visual Words Combination (VWC) is a word set generated by combining several visual words. For example, ten pairs (combinations) can be generated from five visual words. Because we cannot judge which visual words are related to each other, we generated various combinations by using this method. 5
5.1
This section explains the system that we made for this task. First, we will explain the system using only visual information. After that, we will explain the system that uses Flickr User Tags.
The Visual Dictionary (VD) was our dictionary that contains visual words. Visual words were generated by
quantizing keypoints extracted from images. We randomly chose 1,200 images from the 8,000 training images,
and made 4,000 visual words from about 830,000 keypoints [10][
The CFS system was constructed by using VWCs. The procedure, from preprocessing to annotation, is as follows.
1 ) The test images are converted into a frequency histogram of visual words. 2 ) The VWCs are generated from the result of step 1). 3 ) Compare the test image’s VWCs with the visual words in the casebase made from the training images, and identify similar images.
4 ) The test images are annotated.
The test images were converted into a frequency histogram of visual words. The procedure is as follows and is illustrated in Figure 1.
1 ) The keypoints are extracted from the test image by using SIFT.
2 ) Each keypoint is assigned into a visual word in the Visual Dictionary by using the nearest neighbor
algorithm [
Next, the VWCs of the test image were generated. We needed to use a small number of visual words. For instance, the VWCs generated from 300 visual words in combinations of 3 would amount to 300C3. As a result, 4,455,100 combinations would be generated. The calculation cost increases when using a lot of visual words. Hence, we used only 20 visual words that had high frequencies in the histogram and made 1,140 VWCs from them.
The casebase for the CFS system to annotate the 10,000 test images was made from 8,000 training images. Figure 2 shows an example of the casebase. The casebase stored relations such as "If this image has these visual words, then this image has these concepts". The training images were also converted into histograms in the same way and stored as combinations of 20 visual words.
We retrieved similar images in the casebase by matching VWCs generated from the test image and the training images. If a training image had a VWC of the test image, the matching produced a hit. If it didn’t have a VWC, the matching was considered to be a miss. If the number of hits for a certain training image exceeded ten, the images were considered to be similar. An example of matching is shown in Fig. 3.
The confidence was the word frequency in the retrieval results divided by the number of retrieval results. In addition, when confidence was converted into a binary value of either 0 or 1, if the concept had a confidence of 0.2 or more, the concept's binary value was taken to be 1. If the value was less than 0.2, the binary value was 0. 5.3
The Bag of Visual words system was used as a baseline for evaluating the CFS system. The procedure to annotate the test images is almost the same as in the CFS system. The difference is that left side of the casebase of the Bag of Visual Words system in Fig. 2 is not 20 visual words but the frequency histogram in 5.2.1. Twenty training images were retrieved from the casebase, and each test image was annotated in the same way as described in section 5.2.5. The cosine measure was used to determine the degree of similarity between the test image and the training image. 5.4
We describe about a system using Flickr User Tag. The procedure from preprocessing to annotation is as follows.
1 ) The casebase is constructed in advance. 2 ) The confidence between the test image and concept is calculated.
3 ) The test images are annotated.
The casebase here is different from the casebase explained in 5.2.3. It was constructed as follows. 1 ) The training images for each concept are collected. 2 ) The tags of the collected images are collected. 3 ) The term frequency (TF) and the document frequency (DF) of each tag are calculated, and the tags are weighted by using the TF-IDF method. 4 ) The casebase (Fig. 4) paraphrasing concepts with tags is constructed by using tags having the calculated
TF-IDF values. In the figure, the numerical values in parentheses are the TF-IDFs.
We shall illustrate this procedure by using the concept "Dog" as an example. The training images come annotated with concepts from ImageCLEF. Of the 8000 images, there were 211 images with the concept "dog" attached to them. Moreover, the training images had tags plus the concept. The tags of the 211 images were collected. The TF-IDF value of each tag was then calculated and the tag and its value were stored.
The casebase described in sec. 5.2.3 stored the 8,000 cases, but the casebase here stored 93 cases. The idea behind this casebase comes from the theory of meaning propounded by Wittgenstein. According to the theory of meaning, the meaning of a word can be represented by another word. In the example of Figure 4, “partylife” is paraphrased by “concert” and “fireworks”.
Next, the confidence between the test image and each concept is calculated. The procedure is as follows (an example is shown in Fig. 5).
1 ) The tags of the concept and the test image are matched, and the TF-IDFs of the tags that become hits are added together. The total TF-TDF is stored in the concept. 2 ) Step 1) is repeated for all concepts. 3 ) All concepts are regularized by the maximum value of each concept, and this value is assumed to be the confidence. 4 ) The steps from 1) to 3) are repeated for all test images.
When the tag of the test image and concept were compared, combinations like those in section 5.2.2 were not always generated because there were many unsuitable tags. For instance, there were spelling mistakes like "Mweeting" and sentences like "Girlecstaticallydancingondnbtuneswhileholdingaglass ofbeerinherhand".
Moreover, about 1,000 images of the 10,000 test images didn’t have any tags; i.e., only about 9,000 images were annotated.
Every concept, not every image, was considered when the confidence was converted into a binary value.
For every concept, the confidences of all test images were sorted in descending order. The binaries of the test images that had the top n confidences became 1 because these images may have had a strong relationship with each concept. The binaries of the other test images were 0.
The value of n was determined stochastically. For instance, in the example described in 5.4.1, the concept "Dog" was annotated on 211 of the 8000 training images. Note that when calculating the number of test images that are annotated to 9,000 images stochastically, the concept "Dog" is sure to be annotated to 237 test images in 9000 training images.
Figure 6 shows an example of converting the concept "1: partylife". In this case, n is 2. 6
6.1
Below, we describe the five systems that we submitted and their results. 1. VisualOnly_CFS_of_meiji
This system is the CFS system. 2. VisualOnly_BagOfVisual words_of_meiji
This system is the Bag of Visual words system. 3. Mixed_CFS_and_Tags_of_meiji 9,000 images that have tags attached to them are annotated by using the Flickr User Tag System. The remaining 1,000 images are annotated with the CFS system. This system integrated the results of systems 1 and 2. 4. Mixed_Tag_Based_CFS_of_meiji.txt
This system attempts to improve the result of system 1 by using the Flickr User Tag System. 5. Mixed_Text_Based_BagOfVisual_words_of_meiji
This system attempts to improve on the result of system 2 by using the Flickr User Tag System.
The improvements to system’s 4 and 5 were to refine the annotation results of systems 1 and 2 by using the results of the Flickr User Tag System. The confidence of each concept was calculated by using the Flickr User Tag System on all test images. Seven concepts that had high confidence were employed to improve the results of systems 1 and 2. The confidences of these seven concepts of systems 1 and 2 added up to 0.5. They were converted into a binary value of 1 because they exceeded 0.2. 6.2
First, we can see that system 2, which uses the original Bag of Visual words, is more accurate than system
1 that uses CFS. The CFS result was not as good as we expected because the number of visual words was
limited in order to reduce the processing cost. In short, the image was represented by only 20 visual words,
and this is not enough information. Moreover, there is a research result showing that visual words whose
frequencies are high are not so important [
The system that improved the results of the Bag of Visual words method by using Flickr User Tags was the most accurate one that we submitted. This means that annotations using text information like tags are effective. Unfortunately, the system did not have as good a result as we expected. We think the problem was that the selection and the number of visual words were not suitable (see sec. 6.2). It will be necessary to verify which visual words to select in the future. [3] MIR Flickr 25.000 image dataset, http://press.liacs.nl/mirflickr/