The University of Amsterdam participated in the photo annotation task and the concept-based retrieval task of ImageCLEF 2011. In the per-image evaluation of the photo annotation task, we achieve the highest score overall. For the concept-based retrieval task, we submitted the best visual-only run. For the concept-based retrieval task, we considered three ways to perform visual retrieval: fully automatic, human topic mapping and human topic inspection. For a fully automatic system, including more random negatives to train a topic model improves results. For a human selecting relevant concepts to the topic, multiplication fusion works better than summation. For human topic inspection, a relevance feedback scheme on the train data gives an 8-fold increase in the number of positive examples per topic. Depending on the topic, the human topic mapping (best for 21 topics) and inspection (best for 17 topics) give the best results. An oracle fusion of the di erent methods would increase MAP from 0.100 for our best run to 0.128 overall.
The University of Amsterdam participated in the photo annotation task and the
concept-based retrieval task of ImageCLEF 2011. The Large-Scale Visual
Concept Detection Task [
Last years experiments [6{9, 11] emphasize in particular the role of visual
sampling, the value of color invariant features, the in uence of codebook
construction, and the e ectiveness of kernel-based learning parameters. This was
successful, resulting in the best visual only run for the photo annotation task in
terms of MAP. Speedups using parallel computing were investigated in [
A new task for this year is the concept-based retrieval task. By extending the test set to 200,000 images, this ensures that systems need to have reasonable computation times. Another di erence in this task is that there are no prede ned concepts, but a collection of 40 topics. These topics are typically combinations of several existing ImageCLEF concepts, but can have complex boolean expressions within them. They come in the form of a textual description and up to 5 example images. 2
Our concept detection system is an improved version of the system from the
ImageCLEF book [
To achieve better results in the per-image evaluation, where we need to perform a binary assignment of a tag to an image, we use the probabilistic output of the SVM. In a cross-validation experiment, we have found a threshold of 0.3 to be good for most concepts: the default threshold of 0.5 would be too conservative when evaluating with an example-based F-measure where precision and recall are weighted equally. Optimizing the threshold on a per-concept basis instead of a single threshold was found to be less stable. Instead of a single parameter, 99 parameters need to be chosen (one per concept), and this estimation is done on the data of a single concept (instead of over 99 concepts).
New this year is our inclusion of textual information based on the image tags. As a textual representation of the image, we use a binary vector signaling whether a tag is present or absent among the provided Flickr tags. We select all words which occur at least 25 times. Tags consisting of multiple words, split by spaces are turned into multiple words. Also, words consisting of only digits are discarded. This gives us a lexicon of 1008 words. The binary feature vectors are L2-normalized.
1Papers available from http://www.colordescriptors.com
CoreFast. Harris-Laplace and dense sampling every 6 pixels (multi-scale)
with 3-SIFT and fast intersection kernel [
Multimodal-CoreA. Combination of the CoreA visual features with our text features; equally weighed at the SVM kernel level.
Multimodal-CoreA-MKL. Combination of the CoreA visual features with our text features; weighed at the kernel level by multiple kernel learning. 2.2
In table 1, the overall scores for the evaluation of concept detectors are shown.
The features with sampling at every pixel instead of every 6 pixels perform
better (0.375 versus 0.368), which is similar to the result obtained in [
The use of topics in the concept-based retrieval task, instead of concepts, poses a new problem to concept detection: what do we use as a starting point? Each topic has up to 5 example images, which could also be used to start visual retrieval. Since the topics are primarily combinations of several existing ImageCLEF concepts, we could use existing concept detectors. However, to do the latter fully automatic, we would need language parsing tools with support for boolean logic. An alternative is to add a `manual' component to the system where a human maps topics to existing topics. But, a human can go a step further in their inspection of the topic. The concept-based retrieval task states that the training set of the annotation task (8,000 images annotated with 99 visual concepts) can be used to train the concept detectors. Therefore, we have extended the formulation of the topic by using relevance feedback on this training set.
Overall, we have explored 3 approaches:
Fully automatic retrieval. We use only the provided example images as positive examples to train a new concept detector. We combine these positive examples with either 10, 33 or 100 random negatives from the photo annotation train set. These are runs auto10, auto33 and auto100. Human topic mapping. A human reads the topic and then selects relevant concept(s). For run 1concept, the human can only select a single concept. For 2conceptsum and 2conceptmul, the human can select two concepts. The probability scores of these concepts are then combined using either summation or multiplication.
Human topic inspection. A human can give quick feedback on whether images are relevant for a certain topic. Therefore, we have taken the concept models trained for the fully automatic retrieval, and applied them to the training set. A human was then given up to 7.5 minutes per topic to check the top ranked images for additional positive examples, and allowed to mark negative examples as well. Besides the output from the fully automatic system, the human was also allowed to look at the positive examples for one of the 99 existing concepts, and get additional positives from there. We also include a run with 100 negatives randomly added besides the negatives selected by a human.
The concept detectors used for concept-based retrieval are trained using the Core system from the photo annotation task, unless the word fast is in the name. In the latter case, the CoreFast system was used. It is of interest to note that we have only used visual information for the concept-based retrieval, where other participants have also included information from the tags. 3.1
In Figure 1, we show results for our 3 concept-based retrieval approaches. For
the fully automatic system, including more random negatives improves results.
The fully automatic system achieves 0.043 MAP with 100 negative examples.
Additional negative examples might improve results further, but this also
increases the chances that there are true positives among the random negatives.
For the human concept mapping, selecting two concepts (where possible) results
in a large improvement over selecting a single concept. This is expected, as the
topics are designed to be boolean combinations of existing concepts Topics which
directly map to a single concept have been left out on purpose. When
combining two concepts, the multiplication fusion (0.089 MAP) works better than the
summation fusion (0.080 MAP). For the human topic inspection, results are
much better than the automatic system: the number of positives has increased
to 42 on average, and 228 negatives have been selected. We nd that including
100 random negatives still improves results; apparently the negatives selected
by a human are not su cient. To check whether selecting negatives is necessary
at all, an interesting experiment would be to leave out the negatives selected by
the human completely, and to only use random negatives. See also [
The human concept mapping achieves the best results for 21 out of 40 topics. The human concept inspection achieves the best results for 17 out of 40 topics. Had we used the best approach per topic (oracle fusion), we would have increased MAP from 0.100 for our best run to 0.128 overall. Further analysis is needed to determine the relationship between how closely the topic maps to existing concepts, accuracy and the speci city of the topic.
The submissions from our visual concept detection system in the ImageCLEF 2011 photo annotation task have resulted in the best run in the per-image evaluation. In the concept-based retrieval task, it was the best visual-only system. For the concept-based retrieval task, we considered three ways to perform visual retrieval: fully automatic, human topic mapping and human topic inspection. For a fully automatic system, including more random negatives to train a topic model improves results. For a human selecting relevant concepts to the topic, multiplication fusion works better than summation. For human topic inspection, a relevance feedback scheme on the train data gives an 8-fold increase in the number of positive examples per topic. Depending on the topic, the human topic mapping (best for 21 topics) and inspection (best for 17 topics) give the best results. An oracle fusion of the di erent methods would increase MAP from 0.100 for our best run to 0.128 overall. 5
This work is supported by STW SEARCHER, FES COMMIT, and the IARPA via Department of Interior National Business Center contract number D11PC20067. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the o cial policies or endorsements, either expressed or implied, of IARPA, DoI/NBC, or the U.S. Government.