In these notes we present an image classi cation method which has been submitted to the ImageCLEF 2015 Medical Classi cation challenge. The aim is to classify images from 30 heterogeneous classes ranging from diagnose images coming from di erent acquisition techniques, to various biomedical publication illustrations. The presented work is intended to be a proof of concept of how our method, which uses only visual information, performs in the modelling of such image classes. Our approach uses 1st and 2nd order color features obtained at a whole image level. These features are considered as samples of a multidimensional statistical distribution, and a distinctive signature of the represented image can be built in the form of a Covariance-matrix based descriptor. The Riemannian manifold structure of such descriptors can be exploited in order to formulate an image classi cation methodology. Despite the challenging task due to unbalanced classes and image homogeneity, the obtained results in the task place our method on the top of the most accurate ones using purely visual features. This asserts the feasibility of our methodology and proves that its performance can be on par with other methods which use also complementary textual features for complex image retrieval.
Medical image classi cation provides a challenge on the identi cation of similar
medical images: this is an interesting problem due to the subtle changes between
di erent image sources. For instance, inside the range of microscopy images there
exist di erent acquisition devices (light, electron, uorescence or transmission)
which are able to capture di erent tissue details. Despite of that, the resemblance
between image cues is high and poses a challenging problem from a classi cation
perspective [
The ImageCLEF Medical Classi cation challenge [
In the Computer Vision research area, many pattern recognition methods have been developed for image classi cation and retrieval. Most of them include the development of content and feature selection functions, or the usage of keypoint extractors and associated descriptors which can be later categorized by supervised classi cation methods (Support Vector Machines, Boosting, Neural Networks, etc). Our presented approach is based in our ongoing research in Covariance-based descriptors, and we are specially motivated by the demanding conditions found in the di erent images of the medical classi cation subtask. Our method provides a simplistic formulation, which provides a discriminative signature for a whole image according to the variation of di erent features at its pixels. We are particularly interested in seeing if this proposed description, using purely visual information, is discriminative enough.
The following sections of this report present an overview of our methodology, the results obtained on the train data and the own challenge, and a nal discussion about some aspects of the presented approach and associated future work. 2
An inspection of the provided images of this medical classi cation task makes evident that class separation from purely visual cues is not a trivial task. Di erent image sources might share visual features, or su er from a lack of discriminative salient cues (see Fig. 1). Nevertheless, this also yields to our rst intuition of what should be taken into account. First of all, there are several information cues that are equally important: not only texture patterns, but also color, sparsity, structure features... And in a second place, even more important than the features themselves: the modelling must take into account all the feature interactions together. That is, a diagram gure in a medical publication can be in grayscale just as an electron microscopy image, but structural features in a diagram contain pure lines or geometrical shapes which are not present on a biological tissue captured by the microscope. At the same time, di erent microscopy devices might capture similar natural tissue patterns, but for instance a visible light microscope can capture a di erent range of color spectra than a transmission microscope. Therefore, in an analogy with a natural visual perceptual system, our goal is to model the space of di erent visual cues and their joint relationships, and correlate them to the wide range of image classes. 2.1
Visual features Covariance-based descriptor An ideal image representation must encode all images in a common compact, size invariant notation regardless of the di erent image sizes. The description must DRCO DSEC GCHE GMAT DRCT DSEE GFIG GNCP DRMR DSEM GFLO GPLI DRPE DVDM GGEL GSCR DRUS DVEN GGEN GSYS DRXR DVOR GHDR GTAB also be robust to intraclass spatial transformations, such as rotations, and if possible it should not depend on computationally loading intermediate stages, such as keypoint extractions. The intuition behind our method is not to use image features themselves, but rather than that observe the features along the complete image and consider them as unstructured samples of a multidimensional statistical distribution, using their covariance as a descriptive signature.
Covariance matrices were introduced as descriptors in the Computer Vision
domain by Tuzel et al. in [
In order to formally de ne this 2D color feature based Covariance Descriptors, we denote a feature selection function (I) for a given image I as: (I) = f x;y 8x; y 2 Ig ; (1) which provides a set of feature vectors x;y for each one of the pixel coordinates fx; yg inside all the image I. These 11-dimensional feature vectors are expressed as: and include the pixel coordinates, the di erent RGB color values, rst and second order image intensity derivatives and their magnitude and pixel curvature. These cues provide information about the color distribution of a given image class, as well as their texture patterns and visual structure {as found in the rst and second order gradient and curvature features. Then, for a given color image I the associated Covariance Descriptor can be obtained as:
Cov ( (I)) = N 1 1 Xi=N1 ( x;y ) ( x;y )T ; (3) where is the vector mean of the set of vectors f g within the image I.
The resulting 11 11 matrix Cov is a symmetric matrix where the diagonal entries will represent the variance of each feature channel, and the non-diagonal elements represent their pairwise covariance, as seen in Fig. 2. This provides a signature of how feature behave in a characteristic way for each one of the images of the di erent classes. Covariance Descriptors have the form of covariance matrices which, besides providing a compact and exible representation, causes them to lie in the Riemannian manifold of symmetric de nite positive matrices Symd+. This has a major impact on their interest as descriptive units, as their spatial variety is geometrically meaningful: samples of classes sharing similar feature characteristics will remain under close areas in this descriptor space. Nevertheless, it is important to bear in mind that this spatial distribution is non Euclidean and has to be treated with its particular Riemannian metric in order to perform analytic operations with the descriptors.
According to [
For computational simplicity in certain problems, the projection point can be established to the Identity matrix, and therefore the tangent mapping becomes: (4) (5) (6) log(X) = U log(D)U 0; where U and D are the elements of the single value decomposition (SVD) of X 2 Symd+.
In an analogous manner, the exponential mapping of a point y 2 TY returns its original point representation Y in the Symd+ manifold:
exp(y) = U exp(D)U 0;
One property of the projected symmetric matrices in the tangent space TY is that they contain only d(d+1)=2 independent coe cients, in their upper or lower triangular parts. Therefore it is possible to apply the vectorization operation in order to obtain a linear orthonormal space for the independent coe cients: x^ = vect(x) = (x1;1; x1;2; :::; x1;d; x2;2; x2;3; :::; xd;d); (7) where x is the mapping of X 2 Symd+ to the tangent space, resulting from Eq. (4). The obtained vector x^ will lie in the Euclidean space Rm, where m = d(d + 1)=2 {R66 in the current approach.
This set of operations is useful for data visualization, feature selection, and for developing Machine Learning and classi cation techniques on top of the particular geometric space of the proposed Covariance Descriptors, specially taking into account the following Riemannian metric which expresses the geodesic distance between two points X1 and X2 on Symd+: (X1; X2) = s
1 1 2 T race log X1 2 X2X1 2 ; (8) or more simply (X1; X2) =
1 1 values of X1 2 X2X1 2 .
qPd
i=1 log( i)2, where i are the positive eigen2.3
Classi cation via a Manifold-regularized sparse representation
For the classi cation of the proposed 2D color feature based Covariance
Descriptors we propose a manifold-based sparse classi cation method which is part of
our research as presented in previous approaches [
The topological layout of the proposed Covariance Descriptor yields to focus
on a geometrically sensitive classi cation method which can exploit the
Riemannian manifold distribution. Sparse representation based methods [
The base intuition is that an unknown sample should be ideally represented, as accurate as possible, by using the smallest group of most similar samples from a learning set A. Then, a test sample in the form of a new vectorized Covariance descriptor C 2 R66 can be expressed as a linear combination on top of the tangent space T of the Symd+ manifold of the available set of training samples: C = A .
Let A be the whole set of n training samples, in its vectorized form according to eq. 7, from K di erent classes: A = [A1; A2; :::; AK ] 2 R66 n, where each Ai = fvect(i)g is the set of vectorized Covariance descriptors which form the subset of training samples for the class i. And let = [ 1; 2; :::; K ] be a vector of weights corresponding to each one of the training samples in A. Then, the sparsity restriction on can be achieved via its L2 norm minimization, proposing a manifold-aware minimization constraint which relaxes the computational expense of the method and adds numerical stability: . . .
0 where D is a diagonal matrix of size n n which allows the imposition of prior knowledge on the solution with respect to the training set, using the Riemannian metric de ned in eq. (8). This term contributes also on making the least squares solution stable, and on introducing forehand sparsity conditions to the vector ^ as well. D is de ned as: (9) (10) where A0i and C0 are the unvectorized covariance descriptors for training and test samples respectively. The solution to the sparse collaborative representation, ^, can be calculated by the following derived expression according to [10]: ^ =
AT A + DT D 1 AT C
Finally, the classi cation label of the test sample C can be obtained by observing the regularized reconstruction residuals from the resulting sparse vector ^: class(C) = argmin i kC
Ai ^ik2 k ^ik2 (11) (12) 3
The evaluation score used on the task performance assessment is the classi cation accuracy ratio for all the classes, computed as the ratio of true positives and negatives over the total number of samples. We collect the top results in Table 3, which are also publicly available on the challenge website 1.
Before the submission of the task, we tested our method on the training data set, using a 10-fold cross-validation. Each fold was adapted so at least 20% of samples of each class were kept in each subset. In classes with a very low number of samples which would cause to have some folds without class representation, some samples where duplicated. Therefore, classes with very few samples where guaranteed to be balanced and represented on the training set of our classi cation method. After iterating the cross-validation runs, we obtained an average accuracy of 73.24 %. As we have commented in section 2.3, the presented classi er arises as a method for expressing unknown samples as the best sparse representation regarding to a learning set. Therefore, we explain this 1 http://www.imageclef.org/2015/medical increase on the accuracy as a direct e ect of the balancing preprocessing of those classes with very few elements.
Once the groundtruth annotations of the testing set have been made publicly available, we can analyse the di erent Precision and Recall values for each class as presented in Table 2, and observe if there is a particular correlation between these values and the di erent cardinality of each class or their visual nature.
Class Class # Precision Recall Class Class # Precision Recall
These results assert our hypothesis of a mandatory class balancing stage in order to boost the accuracy performance of our proposed sparse classi er. 4
The presented approach provides two main outcomes: on one side, a Covariancebased descriptor which uses only low-level visual features and requires very low computational cost for its construction. On the other side, a classi cation method which takes into account the geometric properties of such representation. All together, the system provides an image retrieval method which is fast and has demonstrated to be of similar accuracy levels to other methods using complementary textual information.
Despite of that, we rmly believe that this method can be further extended in the future, in many directions. Descriptor features could be extended with a codi cation of medical terms associated to di erent image classes. Thus, visual and textual feature fusion would take place within the nature of our descriptor. On the other side, after analysing the results and the available groundtruth annotations, we have observed a major dependency of our method on class cardinality due to its sparse representation formulation. Classes with minor representation can lead to higher classi cation error as a consequence of the minimization formulation of our method. Therefore, we have observed that this can be solved by incorporating a class balancing stage before the sparse regularization.
So far, the participation on the ImageCLEF Medical Classi cation task has provided an interesting benchmark which has contributed to test our ongoing research and identify some improvements for our methodology thanks to the particular nature of the provided testing data. 10. D Zhang, Meng Yang, and Xiangchu Feng. Sparse representation or collaborative representation: Which helps face recognition? In International Conference on