In this paper we present the methods and techniques performed by the IPL Group for the Concept Detection subtask of the ImageCLEF 2018 Caption Task. In order to automatically predict multiple concepts in medical images, a k-NN based concept detection algorithm was used. The visual representation of images was based on the bagof-visual-words and bag-of-colors models. Our proposed algorithm was ranked 13th among 28 runs and our top run achieved F1 score 0.0509.
Visual Concept Detection or Automated Image Annotation of medical images has become a very challenging task, due to the increasing number of images in the medical domain. This immense number of medical images leads to a situation where automatic image annotation becomes more and more important. It has gained a lot of attention through various challenges and researchers that try to automate the understanding of the image and its content and provide useful insights that could be beneficial for clinicians. Detecting image concepts can be particularly useful in Content Based Image Retrieval (CBIR) because it allows us to annotate images with semantically meaningful information bridging the gap between low level visual features and high level semantic information and improving the efectiveness of CBIR. The main idea of image annotation techniques is to learn automatically semantic concept from a large number of training samples, and use them to label new images. As the amount of visual information increases, the need for new methods for searching it, also increases.
This year our group participated only in the concept detection task. Details
of this task and the data can be found in the overview papers [
kNN is one of the simplest and very efective algorithm in classification problems. Given an unknown item, J , (testing image) we calculate its distance from all images in a training set and assign the item to its nearest category. The decision is based on the number of neighbours we take to assign a score to categories. This is a dificult decision and a weak point of the algorithm.
In the following we give the definition of the annotation problem and a brief description of the algorithm we used in our submitted runs.
Let X = fx1; x2; :::; xN g a set of images and Y = fy1; y2; :::; ylg the set of
concepts (labels or annotations). Consider a train set T = f(x1; Y1); :::; (xN ; YN )g
where Yi Y is a subset of concepts assigned to image xi. For each concept yi we
define its semantic group by the set Ti T , such that Ti = fxk 2 T : yk 2 Yig.
The annotation problem is then defined by the probability p(yijJ ). Given an
unannotated image, J , the best label will be
(1)
(2)
(3)
were dist(:) can be any distance (L1, L2 etc) between the visual representations
of the images J and xj . If we sort the images inside Tk, with the highest score
on the top, we can take any number of images in the summation 2. Due to the
imbalance property of the images between the semantic groups of the concepts
usually we consider only a subset of the nearest neighbours in the summation
of equation 2. The scores in (2) are converted to probabilities using a soft-max
function with a decay parameter, w, which nullifies the distances of most of the
images and only few of them, the nearest ones, contribute in the summation.
In our experiments the value of the parameters, like the parameter w, were
estimated on experiments with the CLEF 2017 data set. The algorithm we have
described in matrix form is written by the matrix multiplication:
score(yk; J ) = J T XvT Y
were Xv is a matrix N mv with N the size of the train set and mv the number of
visual features and Y a N numOf Concepts binary and very sparse matrix. The
entry Y fig(j) denotes the j th conceptID of the image i. For computational
eficiency, eq. 3 was implemented with a very fast Matlab function using the
cosine distance -equivalent to Euclidean distance- for normalized to unity vectors.
Several relations have been used to define the probability
following it is defined by the score:
p(yijJ ) [
i score(yk; J ) = ∑ dist(J; xj ) xj2Tk
As we already mention due to the imbalance property of the annotation problem the algorithm benefits those concepts with high frequency occurrence in the training images. From several experiments with smaller data, plotting the distribution of relevance of the concepts, sorted by their values of DF (Document Frequency) versus the distribution of retrieval we observed that, concepts with low frequency in the train set are downgraded while concepts with high frequency are benefited by the algorithm. Thus the algorithm was modified by normalizing the outcome from eq. 3 by the value DF (yk)=avgDF . This last step improved the performance of the algorithm significantly. 3 3.1
One important step in the process of concept detection is the visual
representation of images. Images are represented using two models, the Bag-of-visual
Words (BoVW) model and a generalized version of the Bag-of-Colors (QBoC)
model based on the quad tree decomposition of the image. The BoC model was
used for classification of biomedical images in [
BoC model was proposed in [
The TF-IDF weights of visual words were calculated for each model separately and the corresponding visual vectors were normalized using the Euclidean norm. The similarities between test and train images were combined for both representations of the images in a late fusion manner using weights w1=0.65 for the DenseSIFT descriptor and w2=0.35 for the QBoC. The values of these parameters were chosen based on our experiments on several other image collections. Finally another parameter of our algorithm is the decay parameter (w) of the softmax function. Several values of w were used in our runs based on experimentation with the CLEF-2017 data set. This year’s data contain a set of 223305 training images and a test set of 10000 images with 111156 discrete concepts. The training data are represented with two matrices, one of 223305 4096 for the dense SIFT representation and the other of 223305 3000 for the QBoC representation. Similarly are represented the images in the test set. These data demand more that 13GB of memory. To overcome our memory limitations we implemented a parallel knn algorithm splitting the matrices into 10 blocks that are accommodated in RAM. 3.3
To determine the algorithm’s optimal parameters we experimented with the ImageCLEF 2017 caption task dataset. In this year’s contest we submitted eight visual runs for the concept detection task. For all runs we used Dense Sift with 4.096 clusters and GBoC with 200 clusters. The parameter w is between 200 and 300. The parameter annot denotes the number of predicted concepts. The results are presented in table 1.
Run_ID F1 Score Annot Parameter DET_IPL_CLEF2018_w_300_gboc_200 0.0509 70 DET_IPL_CLEF2018_w_300_gboc_200 0.0406 40 DET_IPL_CLEF2018_w_300_gboc_200 0.0351 30 DET_IPL_CLEF2018_w_200_gboc_200 0.0307 30
The choice of the parameters w, and number of concepts (annot) is a matter for further investigation. It seems that there is a trade of between the values of these parameters which are set experimentally. A large value of w, may lead the model to over-fitting while a large value of annot reduces the accuracy. Our choice of annot was based on the observation that on average each image in the train set contains 30 concepts.
In this paper we presented the automated image annotation experiments performed by the IPL Group for the concept detection task at ImageCLEF 2018 Caption task. A k-NN based concept detection algorithm was used for the automatic Image Annotation of medical images. A normalization step was proposed on the scores in eq. (3) which improved significantly the performance of kNN. The results so far with our new knn approach are encouraging and several new directions have emerged which are currently under investigation.