This paper provides baseline results for the medical automatic annotation task of CLEF 2007. Therefore, the algorithms initially used for the corresponding tasks in 2005 and 2006 are applied, using the same parameterization. Three classifiers based on global image features are used and combined within a nearest neighbor approach. In 2007, a hierarchical code is introduced to describe the image contents, with the evaluation scheme allowing a finer granularity of the classification accuracy. We therefore evaluate some techniques for estimating the confidence in the classifier decision, which stop or alter classifier reports at code levels with uncertain classifier reports.
This paper provides baseline results for the medical automatic annotation challenge of CLEF 2007. By using the same methods and the same parameterization, the obtained results allow to roughly estimate the complexity of the annotation tasks from 2005, 2006, and 2007 relative to each other. In 2007, the evaluation scheme addresses the hierarchical structure of the IRMA code [1] by allowing the classifier to decide don’t know at any level of the code, independently for each of the four axes [2]. To effectively support this scheme, models which estimate the classifier’s confidence in its decision are required. 2.1
The image content is described by global features, i.e. the intensity information from the images
is drastically reduced to few (numerical) values [
Tamura et al. proposed histograms of the coarseness, directionality, and contrast to capture
texture properties [
The three classifiers are combined within a nearest neighbor classifier. The overall distance between a sample q and a reference r is computed by
dc(q, r) = λIDM · d′IDM(q, r) + λCCF · d′CCF(q, r) + λTTM · d′TTM(q, r) where d′(q, r) denotes a normalized distance between q and r. Normalization is done by dividing the individual distance by the sum of distances between the sample q and all references. For CCF, distances are obtained by transforming similarity s to d = 1 − s. The nearest neighbor classifier then decides based on the class information of the k best references. 2.2
To address the modified evaluation scheme of the 2007 challenge, the nearest neighbor decision rule is modified. Three options are implemented: 1. From the k neighbors, a common code is generated by setting differing parts (and their subparts) to don’t know, e.g. two neighbors with codes 1121-120-434-700 and 1121-12f-466700 result in a common code of 1121-12X-4XX-700. 2. For the k neighbors, a threshold td is applied to the majority vote decision. If the distance for the best neighbor from the decided class is greater than td, the decision is rejected, i.e. the reported code is XXXX-XXX-XXX-XXX. 3. A threshold tn is applied to the k neighbors. A neighbor is excluded from the decision if its distance is greater than tn. For the remaining neighbors, majority vote is used to obtain the decision. If all neighbors are excluded, XXXX-XXX-XXX-XXX is reported.
To keep the number of combinations at bay, only combinations 1+2 and 1+3 are evaluated. 3
The classifier weights are the same as 2005 and 2006: λIDM = 0.42, λCCF = 0.18, λTTM = 0.40. To obtain estimates for td and tn, the development set is used: inspecting the classifiction for this set, the results are sorted based on the best distances from neighbors from the decision class for each sample. Based on this sorted list, the thresholds are chosen from the 1st, 5th, 10th, 25th, and decision majority vote common code k 1 5 5 50th worst distances encountered. Both thresholds are evaluated with the normal majority vote decision rule first and afterwards with the policy to obtain the common code parts.
The evaluation is done using the scheme described in [2]. For each image from the test set, an error value e ∈ [0..1] is obtained, based on the position of classification errors in the hierarchy. By summation over all 1,000 test images, the overall value is obtained. Constantly answering don’t know yields a value of 500.0, the worst possible value is 1, 000.0.
Results for applications of the decision threshold td are summarized in Tab. 1. The neighbor threshold tn is used in combination with k ∈ {1, 5, 10, 25, 50, 100}, because otherwise the number of considered neighbors would be so high that small classes are never reported. Tab. 2 contains the results for the application of tn. The proposed mechanisms for the estimation of classifier results and the modification of reported codes do not improve the baseline results of 51.29 for 1-NN and 52.54/rank 18 for 5-NN. This seems to have been observed by the other groups as well [2]. In our case, common code performs generally worse than the majority vote decision. The results become drastically worse for bad parameter sets, especially when the number of considered neighbors is too high.
Comparing the baseline error rates to the ones from the previous year, the medical automatic annotation task in 2007 is a bit easier than 2006. This can be taken into account when comparing methods by groups who participated in only one of the past years. [1] Lehmann TM, Schubert H, Keysers D, Kohnen M, Wein BB. The IRMA code for unique classification of medical images. Procs SPIE 2003; 5033: 109-117 [2] Mu¨ller H, Deselaers T, Kim E, Kalpathy–Cramer J, Deserno TM. Overview of the ImageCLEFmed 2007 Medical Retrieval and Annotation Tasks. Working Notes of the 2007 CLEF Workshop (in this book)