In this paper, we describe the participation of the Mami team at ImageCLEF 2017 for the Image Caption task. We participated to the concept detection subtask which aims at assigning a set of concept labels to a medical image. We used transfer learning method with VGG19 model for feature extraction to solve this task, and apply those features as input of a new neural network.
In this paper, we describe the participation of the Mami team to ImageCLEF 2017 for the Image Caption task. This team results from a collaboration between IRIT SIG team from the Universite de Toulouse (France) and MISA from the Universit d'Antananarivo (Madagascar).
The Image Caption task consists in two subtasks: concept detection and
caption prediction. The main goal of the concept detection task is to identify
the existence of relevant biomedical concepts in the medical images delivered
by the organizers (see [
This problem can be see as a classi cation problem where each class corresponds to a concept label. As each image may contain multiple concepts, we tackled this task as a multi-label classi cation problem. The multi-label classi cation consists to associate a given instance xi 2 X to a set of labels Yi = yi1; ::; yili 2 Y . Where xi is the instance of the i-st image and Yi the set of concepts xi belongs to. li is the number of concepts in Yi.
We used deep learning method to solve this problem. Deep convolutional
neural networks are considered among the best classi ers for single-label image
classi cation [
In this work, we adapt a convolutional neural network with transfers learning to multi-label classi cation task.
In the next section we present in Section 2 a brief analysis of the dataset used in the concept detection subtask. We then present in Section 3 the method we developed. Section 4 presents the results while Section 4 concludes the paper.
The dataset we used was delivered by the image CLEF Lab and contains 3 subsets: the training set is composed of 164,614 images and 20,463 labels; the validation set consists of 10,000 images and 7,070 labels from which 309 are not in the training set nor in the test set. The test set contains 10,000 images.
Table 2 presents some characteristics of the training set.
mean std min 25% 50% 75% max
Number of labels per image 5.58 4.47 0.00 3.00 4.00 7.00 75.00 mean std min 25% 50% 75% max
Number of images per label 44.95 320.06 1.00 1.00 3.00 13.00 17,998.00
We found that 3.9% of the training images and 3.79% of the validation images have no labels. 3
As said previously, we used a convolutional neural network. For the transfer
learning [
As illustrated in Figure 1, the Oxford VGG19 model consists in 19 layers. To adapt the pre-trained model 3 to the imageCLEF Concept detection task, we froze the parameters of the 18 rst layers and add 2 new layers above it. Our learning consists to train the 2 new layers to map the images to the corresponding concepts. The goal of the training is to assign a positive score to concepts the given image belongs to. The method consists in 2 steps that are described in the following sub-sections. 3.1
The rst step of the process we developed is to forward each image to the VGG19
model and extract the output of the second to last layer of the VGG19 CNN
3 The VGG19 model trained on ILSRVC-2014 http://www.robots.ox.ac.uk/
%7Evgg/research/very_deep/
(fully connected layer 7(fc7)). This output, named feature vector, is a 4,096
dimensional vector. This feature vector summarizes the representation of the
forwarded image. This representation helps to gain time and space on disk, as
suggested by Sharif et al. [
Then, the feature vector is sent to the rst new layer we add for multi-label classi cation ; this process results in scores associated to each concept or label. To train our model for assigning positive scores to the relevant concepts, we adopted the sigmoid cross entropy as loss function.
Let us de ne: p^n = (xn) 2 [0; 1],
the sigmoid function, xn the predicted score for the concept n, pn = 1 if the image belongs to the concept n and 0 otherwise.
Formally, the sigmoid cross entropy loss [
E =
N 1 X[pn log p^n + (1 n n=1 pn) log(1 p^n)]: (1) 3.2
We used the test/validation method to train our model performance and its capacity to generalize on new data. For that, we kept the data sets as split by the organizers. We trained our model using the training set. The validation set serves then as a new data set to validate the model performance and to tune the hyper parameters of the model. We trained our model with a stochastic gradient descent method using 0,0001 as learning rate (which corresponds to 1/10 of the pre-trained model initial learning rate) and a batch size of 128.
We preserved the momentum default value: 0.9 and set the weight decay 5:10 4. After 150 epochs, training stopped and the resulting model was stored although it converges to E=42.547 . The delivered model is applied to predict the concepts in the validation and test data sets; the results are presented in the following section. 4
The task organizers suggested F1-score [
We also added 2 other measures to check the performance of our model on the validation dataset : exact matching ratio measure, Hamming loss. The results are detailed in Table 3. As we predicted a vector, the exact matching ratio corresponds to the percentage of correctly predicted vector elements. This measure does not take into account partially correct predictions. The formula is: M R = 1 N
X I(yi = xi):
N i=1
where N is the number of samples, yi the ground truth vector, and xi is the
prediction vector. The Hamming loss (HamLoss) computes the percentage of
labels that are misclassi ed in a multi-label classi cation, i.e., relevant labels
that are not predicted or irrelevant labels that are predicted [
HamLoss =
N i=1 1 XN xor(xi; yi) :
L (2) (3) where N is the number of samples, L the number of labels, yi the ground truth, and xi is the prediction.
Table 2 presents the results for our model when evaluated on the validation data set with the di erent measures as mentioned before. The same con guration of the model was used for our unique run named DET CORRECTED mami resulat.txt that we submitted to ImageCLEF and that was evaluated by the organizers on the test set (see Table 3 for the results).
Measure Names
F1-score
Exact matching ratio
Hamming loss
The model we developed to participate to ImageCLEF 2017 uses a transfer
learning based on VGG19 and that we adapt for multi-label classi cation. Using
this model, we obtain a low F1-score (0.0462) on the test set. Our result is far
from the other teams. We expect to improve this score by using more appropriate
machine that will make it possible to make a better training. We would like
to make a ner tuning and deeper network. Alternatively, we could experiment
with another classi er like SVM to classify the feature vectors that we extracted.
Also, one of the main issue we faced was to handle the medical compound images
(like formula, graphs, ) that is more complex to process. We envisage to tackle
this problem using Region based CNN (R-CNN) [