The rapid increase of CT scans and the limited number of radiologists present a unique opportunity for computer-based radiological Content-Based Image Retrieval (CBIR) systems. However, the current structure of the clinical diagnosis reports presents substantial variability, which signi cantly hampers the creation of e ective CBIR systems. Researchers are currently looking for ways of standardizing the reports structure, e.g., by introducing uniform User Express (UsE) annotations and by automating the extraction of UsE annotations with Computer Generated (CoG) features. This paper presents an experimental evaluation of the derivation of UsE annotations from CoG features with a classi er that estimates each UsE annotation from the input CoG features. We used the datasets of the ImageCLEF-Liver CT Annotation challenge: 50 training and 10 testing CT scans with liver and liver lesion annotations. Our experimental results on the ImageCLEF-Liver CT Annotation challenge exhibit a completeness level of 95% and accuracy of 91% for 10 unseen cases. This is the second best result obtained in the Liver CT Annotation challenge and only 1% away from the rst place.
About 68 million CT scans are performed in the USA each year. Clinicians are struggling under the burden of diagnosis and follow-up of such an immense amount of scans. This phenomenon gave rise to a plethora of methods to improve and assist clinicians with the diagnosis process.
Content based image retrieval (CBIR) is a growing and popular research
topic [
Besides the known problem with diagnosis and follow-up of this huge number of scans, there is substantial variability in the structure of the clinical reports provided by the clinicians for each case. This variability hampers the ability to establish an e cient and consistent CBIR system, as uniform reports structure is a major need for such an application.
The task of standardization the clinical reports for liver and liver lesions has
recently been proposed by Kokciyan et al. [
The goal of this work is therefore to use the CoG features to automatically
generate the UsE annotations. A major part of this work deals with designing
and building a machine learning algorithm that link CoG features to UsE
annotations. Training datasets provided by the ImageCLEF-Liver CT challenge
[
Method
A major part of this work would deal with developing a machine learning
algorithm that would best link CoG features to UsE annotations based on training
data-sets. Developing a machine learning algorithm involves four main steps [
Data Collection The input of our algorithm is a set of 50 datasets provided by the imageCLEF 2014 Liver Annotation Task data and has been collected by CaReRa Project (TUBITAK Grant 110E264), Bogazici University, EE Dept., Istanbul, Turkey (www.vavlab.ee.boun.edu.tr). Each dataset includes: 1. A CT scan that consists the liver region and liver tumors 2. A segmentation of the liver 3. The lesion's bounding box 4. A set of 60 Computer Generated features (CoG) features 5. A set of 73 User Express (UsE) annotations Fig. 1. Four main steps in designing and building a machine learning algorithm for the estimation of UsE from CoG and CT images: (a) Data collection: the input to our system consists of CT scans and CoG features and UsE annotation ; (b) Feature extraction: only the most informative CoG features are selected; (c) Model selection: Estimating the most appropriate model for the UsE annotations from CoG features. The model is selected after testing a variety of prediction models and; (d) Training the selected model: The selected models parameters are trained using all 50 cases.
The CoG features can be divided to global image descriptors and pathology descriptors. The global image descriptors cover the basic and liver-wide global statistical properties, such as the mean and variance of the gray-level values, and the liver volume. They are extracted directly from the CT scans and the associated segmentation. The pathology descriptors are computed for each liver lesion. They re ect ner levels of visual information related to individual lesions.
UsE annotations are also divided into global and pathology descriptors. Global descriptors are divided into Liver and Vessel groups. These two groups include annotations about the liver itself and its hepatic vasculature. Pathology descriptors include two groups: Lesion and Lesion Component and contain annotations about the selected lesion in the liver.
A complete list of all UsE and CoG features with their associated descriptor
type can be found at [
First, we reduce problem dimensionality by omitting 21 high dimension features from the 60 provided CoG (e.g. FourierDescriptors, BoundaryScaleHistogram, BoundaryWindowHistogram etc.). Thus, our analysis includes only CoG features with scalar values. This results in 39 features divided between global and pathology related features.
The 18 global CoG features are: LiverVolume , LiverMean, LiverVariance, VesselRatio, VesselVolume, MinLesionVolume, MaxLesionVolume, LesionRatio, AllLesionsMean, AllLesionsVariance, AllLesionsSkewness, AllLesionsKurtosis, AllLesionsEnergy, AllLesionsSmoothness, AllLesionsAbcssia, AllLesionsropy, AllLesionsThreshold, NumberofLesions
The 21 pathology related CoG features are: LesionMean, LesionVariance, LesionSkewness, LesionKurtosis, Lesionnergy, Lesionmoothness, Lesionbcssia, LesionEntropy, LesionThreshold, Lesion2VesselMinDistance, Lesion2VesselTouchRatio, VesselTotalRatio, VesselLesionRatio, Volume, SurfaceArea, MaxExtent, AspectRatio, Sphericity, Compactness, Convexity, Solidity
We added 9 features to the 21 pathology features to describe the statistics of the lesion itself. The new features are derived from a re ned segmentation of the liver obtained by thresholding the given lesion bounding box with its mean gray level followed by morphological operations. The added CoG features are: 1. The average gray level intensity values of the healthy part of the liver.
(LiverGrayMean) 2. The standard deviation of gray level intensity values of the healthy part of the liver. (LiverGrayStd) 3. The average gray level intensity values of the lesion. (LesionGrayMean) 4. The standard deviation of gray level intensity values of the lesion. (LesionGrayStd) 5. The lesion's contour mean gray levels. (LesionBounderyGrayMean) 6. The standard deviation of the lesion's contour gray levels.
(LesionBounderyGrayStd) 7. The average gray level di erence between the healthy part of the liver and the lesion. (LesionLiverGrayDi ) 8. The average gray level di erence between the healthy part of the liver and the lesion' contour. (BounderyLiverGrayDi ) 9. The average gray level di erence between the the lesion and its contour. (lesionBounderyGrayDi )
The result is a modi ed CoG list with 18 global image descriptors and 30 pathology descriptors. 2.3
Model Selection In this section the classi cation algorithm to be evaluated is presented: Predictive models can be characterized by two properties: parametric/non-parametric and generative/discriminative. Parametric models have a xed number of parameters and have the advantage of often being faster to use. However, they tend to rely on stronger assumptions about the nature of the data distributions. In non-parametric classi ers, the number of parameters grows with the size of the training data. Non-parametric classi ers are more exible but are often computationally intractable for large datasets.
As to the generative/discriminative property, the main focus of generative
generative models, is not the classi cation task but to correctly model the
probabilistic model. They are called generative since sampling can generate synthetic
data points. Discriminative models, however, do not attempt to model the
underlying probability distributions, but rather focus on the given task, i.e. the
classi cation itself. Therefore, they may achieve better performance in terms
of overall accuracy of the classi cation task. In general, when the probabilistic
distribution assumptions made are correct, the generative model requires less
training data than discriminative methods to provide the same performance,
but if the probabilistic assumptions are incorrect, discriminative methods will
do better [
For real world datasets, so far there is no theoretically correct, general
criterion for choosing between the di erent models. Therefore, we examined four
classi ers, representative of the 4 di erent families of models [
We used the Python scikit-Learn Machine learning tool [
For simplicity, each model was tested with a set of default parameters as
de ne by scikit-learn package [
Training Once the classi er which produced the highest classi cation was found in the previous step, we trained it using all 50 cases. As a result, for each UsE, we obtain a trained model, consisting of classi er along with optimized sets of CoG features (i.e. the selected features) 2.5
Evaluation The evaluating phase of the challenge consists of the estimation of UsE annotation from the given CoG features and the images. Unlike the training phase, UsE annotation are not given here to examine our accuracy.
To apply the resulting classi er in the testing phase on an unseen dataset, we rst extract and extend the CoG features according to a scheme described in Section 2.3. Then, for each test case, we apply the prediction model { the one with the highest score { according to the training phase results. The result is a UsE annotation for each unseen case. 2.6
Estimation of the Lesion Lobe, Lesion Segment, and Cluster Size As noted in Section 2.3, the Cluster Size (i.e. the number of lesions inside the lesion bounding box), the Lesion Lobe and the Lesion Segment containing the lesion were not part of the general learning process but were rather estimated from the image itself. The estimation of these elds was performed as follows:
For the Lesion Lobe: we measure the center of the lesion and the liver. The lesion lobe is estimated as the right lobe if the lesion center is on the right part of the liver, and as the left lobe if the lesion center is on the left part of the liver. In case they were overlapping we estimated it to be the Caudate Lobe.
The Lesion Segment is estimated as follows: If at a previous stage the estimation was that the lesion is on the right lobe, we assessed that the lesion is located on the fourth segment. Alternatively, if at the previous stage the estimation was that the lesion is on the left lobe, we analyzed whether the lesion is located above or below the center of the liver. If it is located above the center of the liver, we assessed that the lesion is in segment 5-6; and if it is located below the center of the liver - we assessed that the lesion is on segment 7-8.
For the Cluster Size: We de ne the Cluster Size to the number of lesion { the value listed in the CoG eld NumberofLesions. Except in cases when the value listed in number of lesion is higher than 6, and then we de ne that the number of lesions to be 6. For the Cluster Size: We de ne the Cluster Size to the number of lesion { the value listed in the CoG eld NumberofLesions. Except in cases when the value listed in number of lesion is higher than 6, and then we de ne that the number of lesions to be 6. Experimental results of applying our method to the ImageCLEF-Liver CT Annotation challenge datasets results with estimation of UsE annotations at a completeness level of 95% and accuracy of 91% for 10 unseen cases.
Training and test results for each of the classi ers are shown in Table 2.6. Due to the simplicity of estimating the global UsE annotations, we present their result per group. A detailed presentation is provided for the pathology related features, which is indeed a much more challenging task. It can be seen that all classi ers successfully estimated the global features.
As mentioned, three additional features were estimated from the images and were not part of the learning process. These features are: ClusterSize, LesionLobe, LesionSegment and their accuracy on the training datasets was 0.75, 0.9, 0.7 respectively. The completeness level of our method is 0.95 due to omitting 3 UsE annotations from the analysis: LesionComposition, Shape and the MargenType.
The optimized sets of CoG features (i.e. the selected features) that were obtained by the model with the highest score after the leave-one-out procedure are shown in Table 5. Note that the set of added features (Section 2.2) were indeed selected by the model, which proves their necessity. Lesion-Lesion
Contrast Pattern
KNN Lesion-Lesion
Other Lesion-Area Lesion-Area Lesion-Area Lesion-Area
Is Contrasted Density Density Type Is Peripherical
Localized Lesion-Area
Is Central Localized KNN Lesion-Area Other Lesion-Component All
Any Any LDA Any LDA LDA LDA KNN Any Any
Selected CoG
Features All All BounderyLiverGrayDi , LesionGrayStd, Entropy, SurfaceArea LesionGrayMean, LesionBounderyGrayMean, BounderyLiverGrayDi , LesionBounderyGrayStd, All LesionLiverGrayDi , Solidity, Entropy, Kurtosis, lesionBoundryGrayMean LesionBounderyGrayMean, Solidity, LesionGrayStd LiverGrayStd, LesionGrayStd, lesionBounderyGrayStd LiverGrayStd, LesionGrayStd, LesionBounderyGrayStd All All We have presented an approach to estimate UsE annotations from CoG features and associated CT scans. We extended the CoG features with 9 additional features to enhance the learning process. In the ImageCLEF-Liver CT Annotation challenge. Our approach provides an average accuracy level of 91% with completeness level of 95% when applied on 10 unseen test cases. This work provides reliable estimation of uniform clinical report from imaging features and therefore it constitutes another step toward automatic CBIR system by enabling e cient search in clinical reports. Future work consists of examining an additional set of classi ers and extending the completeness of our algorithm to estimate the full UsE annotations set and values (e.g. estimating segment 1-3 of the Lesion Segment feature )