Our group within the University of Amsterdam participated in the large-scale visual concept detection task of ImageCLEF 2010. The submissions from our visual concept detection system have resulted in the best visual-only run in the per-concept evaluation. In the per-image evaluation, it achieves the highest score in terms of example-based F-measure across all types of runs.
Our group within the University of Amsterdam participated in the large-scale
visual concept detection task of ImageCLEF 2010. The Large-Scale Visual
Concept Detection Task [
Last years experiments [3{8] emphasize in particular the role of visual sampling, the value of color invariant features, the in uence of codebook construction, and the e ectiveness of kernel-based learning parameters. This was successful, resulting in the top ranking for the large-scale visual concept detection task in terms of both EER and AUC. Both these measures do a per-concept evaluation. The per-image evaluation based on the ontology score suggested that the assignment of concept tags to images leaves room for improvement. Therefore, for this year, we focus on the per-image evaluation. The primary evaluation metric used in 2010 for the per-image evaluation is the average example-based F-measure.
Our concept detection system is an improved version of last years system [
To achieve better results in the per-image evaluation, where we need to perform a binary assignment of a tag to an image, we have modi ed the probabilistic output of the SVM. We have disabled Platts conversion method to probabilities, and instead use the distance to the decision boundary. The decision boundary lies at 0, positives are trained to lie at 1 and negatives are trained to lie at -1. In a cross-validation experiment, we have found a threshold of -0.3 to be good for most concepts: the default threshold of 0 would be too conservative when evaluating with an example-based F-measure where precision and recall are weighted equally. Optimizing the threshold on a per-concept basis instead of a single threshold was found to be less stable. 3
We have submitted ve di erent runs. All runs use both Harris-Laplace and dense sampling with the SVM classi er. We do not use the EXIF metadata provided for the photos nor the provided text tags.
Harris-Laplace and dense sampling every 6 pixels (multi-scale)
with 4-SIFT: from ImageCLEF 2009 [
Harris-Laplace and dense sampling every pixel (single-scale) with
4-SIFT: from ImageCLEF@ICPR 2010 and the ImageCLEF book [
1Papers available from http://www.colordescriptors.com
In table 1, the overall scores for the evaluation of concept detectors are shown. The features with sampling at every pixel instead of every 6 pixels perform better (0.4026 versus 0.3963), which is similar to the result obtained in ImageCLEF@ICPR 2010. Optimizing the parameters of soft assignment using Multiple Kernel Learning did not have the desired e ect. A possible explanation is that the slack parameter for MKL was set to 1, whereas the normal SVM runs optimize this parameter and tend to select 10 as a good slack setting. The two nal runs perform better than the two `baseline' runs from the ImageCLEF@ICPR 2010. However, the color descriptors present in these two runs have not yet been documented.
Compared to other ImageCLEF participants, our runs are the best visualonly submissions. However, combinations of text and visual methods do get a higher overall AP. For concepts like Birthday and Party, an attached tag with the words party or birthday implies presence of that concept, whereas the visual presence might be more ambiguous. The best visual+text method scores 0.4553, compared to 0.4073 for our best visual-only run and 0.2338 for the best text-only run. 5
For the per-image evaluation, overall results are shown in table 2. Our emphasis on optimizing the threshold for tag assignment has resulted in the best overall run in terms of example-based F-measure, i.e. this visual-only run outperforms visual+text methods.
The submissions from our visual concept detection system in the ImageCLEF 2010 large-scale visual concept detection task have resulted in the best visualonly run in the per-concept evaluation. In the per-image evaluation, it achieves the highest score in terms of example-based F-measure across all types of runs.