In this paper we present the methods and techniques performed by the IPL Group for the concept detection task of ImageCLEF 2017. A probabilistic k-nearest neighbor approach was used for automatically detecting multiple concepts in medical images. The visual representation of images was based on the well known, bag of visual words and bag-of-colors models. Detection performance was further enhanced by applying late fusion on the results obtained using di erent image representations. Our best results were ranked 2nd compared with runs under the same conditions.
Automatic Image annotation is an important and challenging task within the eld of computer vision with applications in several domains. In the medical domain it plays an important role in supporting image search, browsing and organization for clinical diagnosis and treatment. Image retrieval based on semantic information has many advantages and is more robust than using only low-level visual features. In the case of absence of semantic information a typical method to bridge the gap between low level visual features and high level semantics is through the automatic image annotation. This is achieved by applying machine learning techniques to learn a mapping of visual features to textual words. The learned model is then used to assign semantic concepts to new unseen images.
The ImageCLEFcaption 2017 task, [
Details of this task can be found in the overview paper [
The following sections, present the image representation methods and our algorithm for concept detection. Finally we report on our results and conclude with possible venues for further research. 2
Three di erent visual representation models were used in our experiments: 1. Localized compact features 2. Bag of Visual Words (BoVW) 3. Bag of Colors (BoC) 2.1
Compact visual descriptors have been used extensively in the past years to e
ciently represent images in a dataset. For this task, two kinds of visual features
were extracted:
1. Color and Edge Directivity Descriptor (CEDD)[
However, in their original form, these descriptors are extracted globally from an image. In order to include a degree of spatial information, features are extracted over a spatial 4x4 grid. The image is rst resized into 256 x 256 pixels and then is split into a 4x4 grid of non-overlaping image blocks. The visual features are then extracted for each block and their corresponding vectors are concatenated to form a single feature vector. The nal vector size for the CEDD and FCTH is 4 4 144 = 2; 303 and 4 4 192 = 3; 702 respectively. 2.2
Inspired from text retrieval, the Bag-of-visual Words (BoVW) approach has shown promising results in the eld of image retrieval and classi cation. Here the BoVW model was implemented using the DenseSIFT visual descriptor.
The Dense SIFT algorithm [
The QBoC representation was successfully used for representing images in
previous works [
Probabilistic k-NN concept detection (PKNN) In this section we brie y present our baseline algorithm for automatic concept detection in medical images. The algorithm is divided into two main phases, namely, the visual retrieval step and the annotation step.
In the visual retrieval phase, for a given test image, a sample of the k most visually similar images from the training dataset is retrieved. Several experiments on the validation set helped to determine the optimum value for k (k=100).
In the annotation step, the concepts associated to the k retrieved images form the candidate concepts for a test image. The nal assigned concepts are determined by a probability score based on the occurrence of concepts in the selected sample. For every distinct concept, w, present in the retrieved training subset, we calculate ConceptScore(w) as:
D combSU M (w) = X ConceptScored(w)
d=1 where D is the number of descriptors to combine.
ConceptScore(w) = k X P (j) P (wjj) j=1 where j is an image from the top-k results and P (wjj) is calculated by: P (wjj) = count(w; j)
jWj j where count(w; j) is the number of times concept w is found in image j, and jWj j is the total number of concepts in image j. P (j) is considered uniform for all images and thus it is ignored.
The top 6 concepts with the highest ConceptScore are selected for the test image. This number was determined by calculating the average number of concepts per image in the training set (i.e. 5:58). 3.1
As an alternative concept scoring method, a Random Walk with Restart (RWR) algorithm [11] was tested. First, from the set of the top k retrieved images an adjacency matrix A of size [c c] is constructed, where c is the number of distinct concepts in the retrieved images. Matrix A de nes the graph whose nodes correspond to concepts and edges connect concepts if they are assigned in the same image of the train set. Next, the RWR algorithm is applied to matrix A resulting in a vector r of size [c 1] representing the most probable concepts for the test image. Similarly, the top 6 concepts with the highest r(w) value are selected as the concepts for the test image. 3.2
In order to improve results, late fusion was applied to the ranked lists of concepts for each test image. In late fusion, the ranked concept score lists from di erent visual descriptors are combined. A new score is calculated based on the combSUM function: (1) (2) (3)
To determine our system's optimal parameters and best visual features we experimented with the validation set provided by the organizers. Trying di erent visual descriptors and concept detection algorithms, helped us conclude on the submitted runs. Table 1 presents some of the top results obtained from these experiments on the validation set. The "PKNN" pre x in run id corresponds to experiments using the probabilistic concept detection algorithm and "RWR" the random walks with restarts algorithm. The "LFS" pre x corresponds to runs using the late fusion method described in Section 3.2. Table 2 presents the runs submitted to clef and their corresponding results. The same pre xes are also used to describe the individual runs.
Run ID PKNN CEDD PKNN CEDD 4x4 PKNN FCTH PKNN FCTH 4x4 PKNN GBOC PKNN DSIFT RWR GBOC RWR DSIFT LFS PKNN (FCTH 4x4 DSIFT GBOC) LFS PKNN (CEDD 4x4 DSIFT GBOC) In this report, we presented the image concept detection methods used by the IPL Group for the medical concept detection subtask at ImageCLEF 2017. For our runs, we used a simple Probabilistic k-Nearest Neighbor approach. Experiments show that using late fusion on BoVW and QBoC performs best and that the image representation plays an important role in performance. Furthermore, the Random walks with Restarts algorithm seemed to perform slightly less, however, a more systematic research is currently underway for this method. Our best run was ranked 2nd in the top performing runs compared to algorithms that don't rely on any external data sources. This results are encouraging and lead to further research on improving the concept detection algorithm with additional textual meta-data. 11. Wang, C., Jing, F., Zhang, L., Zhang, H.J.: Image annotation re nement using random walk with restarts. In: Proceedings of the 14th ACM International Conference on Multimedia. pp. 647{650. MM '06, ACM, New York, NY, USA (2006), http://doi.acm.org/10.1145/1180639.1180774 12. Wengert, C., Douze, M., Jegou, H.: Bag-of-colors for improved image search. In: Proceedings of the 19th International Conference on Multimedia 2011, Scottsdale, AZ, USA, November 28 - December 1, 2011. pp. 1437{1440 (2011), http://doi. acm.org/10.1145/2072298.2072034