Recent research has suggested that there is no general similarity measure which can be applied parameter free on arbitrary databases. In contrast the optimal combination of similarity measures and parameters must be identified for each new image repository. This optimization loop is time consuming and depends on the experience of the designer as well as the knowledge of the medical expert. It would be useful if results that have been obtained for one dataset could be transferred to another image repository without extensive re-design of all relevant components. Transfer of data corpora is vital if image retrieval is integrated into complex environments such as picture archiving and communication systems (PACS). Image retrieval in medical applications (IRMA) is a framework that strictly separates data administration and application logic. This permits an efficient transfer of the data abstraction from one database to another without re-designing the software. It supports the loop of estimating a combination of distance measures, parameter adaption and result visualization, which is characteristic if an image retrieval application is used for varying data corpora. In this work the casimage dataset has been added as a data corpus to the IRMA system. Thereon the query performance has been evaluated without optimization of the currently applied feature combination. It consists of scaled representations of the images, global texture, aspect ratio, and an evaluation of deformation between pixels of different images.
Classical architectures of image retrieval systems consist of an image repository, along with
visualization tools and query functionality. The principle of data storage and visualization does not
vary significantly for different databases whereas classes of retrieval approaches are differentiated
by the query principle. In medical applications this becomes a fundamental question since several
requirements for data entry, retrieval time, and content representation must be considered [
The first class of approaches associates and stores the secondary annotated content descriptions with each image. In general this means textual information, which makes the retrieval task a text search in the descriptions. Since the information is added by a human observer it is likely to be correct and considers the semantical context of the image content. However inter- and intra-individual variance in perception, knowledge and capability of expression leads to different descriptions for a single issue, as do the homonyms and synonyms in medical terminology. Due to the fact that each image has to be categorized manually, the effort at data entry time is high, which is infeasible in clinical applications. Furthermore there is no mean of objective verification for the added data.
The second class of retrieval approaches overcomes the data entry problem by making use of information that is exclusively contained in the image, i.e. image content. Here the retrieval task is the detection of the nearest-neighbors to the query image in the image database. This is based on the similarity of abstract representations of images in a feature space. Consequently content based image retrieval depends on an appropriate selection of the similarity measure which again depends on the considered image features. In general, selection of features and similarity measures cannot be done by a physician in clinical routine. Thus an abstraction layer from the feature handling is required.
Besides the data entry cost and the content representation a third problem is the state of the image database: In a clinical environment the set of available images continuously grows. This must be considered since it is one aim of content based image retrieval to make clinical routine data available as a source of knowledge for education and diagnosis. Consequently the feature selection and similarity computation must be as flexible as possible to adopt to the current state of the database.
The image retrieval in medical applications (IRMA) project integrates these three aspects of
query design into a single framework [
As stated in the introduction a content based query consists of an automated feature extraction combined with a similarity measure. 2.1
Existing literature describes numerous features, which can be extracted from an image. They are
roughly categorized in shape descriptions such as edges [
In this concept a query corresponds to a nearest-neighbor classification of the query image to all images in the currently considered image corpus. Consequently it is designed by defining a similarity measure as a metric in the images’ feature space. Result of this computation is a representation of the corpus as a sorted list, where the most similar images with the smallest distance form the head. It must be considered how many of the first images in the list are relevant images matching the query. This question is task specific and cannot be determined in advance. Furthermore it depends on the actual number of relevant images in the database. Thus a sufficient set of results must be presented to the user who has to take the final decision. For this purpose the IRMA system offers a set of database processing sequences which enable the sequential or parallel access to the stored image features via iteration or fan-in/fan-out over the corpus respectively. Those processing sequences are combined on a binary execution level by abstract methods. Consequently the data handling is also hidden from the application by special data flow interfaces. 2.3
The component based software architecture provides a platform where new data is integrated without re-implementing the available features and distance measures. New images typically require the adoption of existing feature extractions and similarity computations. Yet when introducing a new image corpus there is no knowledge on the classes of images and their sizes. Thus finding the appropriate features and similarity measures is similar to an optimization task, without knowing the target function. However, brute force learning approaches for the optimal query parameters can only be performed if the ground truth is known. Alternatively a manual optimization of query methods to a distinct database is inapplicable for clinical routine solutions, since there is simply no time to supervise the learning process. Thus the designer of an application combines a set of features and similarity methods in advance and then hands it over to the medical expert who has to verify the results. Once such a retrieval engine is integrated for instance into a picture archiving and communication system (PACS) it can hardly be changed, since the database continuously evolves. In contrast the IRMA framework allows a hot swap of the feature extraction, similarity computation and database without affecting each other.
In case of the casimage data set there was no ground truth given, so the results could only be generated by transfer of query settings successfully applied in other applications. This was the choice of query design for the ImageCLEF task. Main objective was the transfer of already implemented code and associated experience from recent experiments onto a new database without considering parameter adaption. 3
The methods applied here have been taken without optimization for the casimage image set from
recent applications on the IRMA database. This database consists of 10,000 images from clinical
routine which have been categorized by medical experts and is used to train parameters and verify
query quality [
32768
Furthermore recent experiments have indicated that spatial and intensity features must be
considered coevally to obtain reasonable results [
Tamura et al. use coarseness, contrast and directionality to capture an image’s texture properties
[
To compare the Tamura histograms of two images T H(Q) and T H(R) with M = 384 bins
each, the Jensen-Shannon divergence is used [
JST(Q, R) =
T Hm(Q) log 1 XM · 2 m=1
2T Hm(Q) T Hm(Q) + T Hm(R) + T Hm(R) log
2T Hm(R)
T Hm(Q) + T Hm(R)
¸
Comparing the aspect ratio of images is an unspecific measure yet it is useful if the dimension of
images has to be considered. Since normalization for some texture features requires the
deformation of the image dimensions to a square shape the aspect ratio of an original image is a mean of
image comparison. Furthermore the aspect ratio is characteristic for different classes of medical
images. MRI slices have identical edge dimensions while radiographs of limbs are rectangular
elongated in direction of the principal bone. The aspect ratio is compared by:
While histogram-based methods provide invariance against some transformations, e.g. translation,
scaled representations of the original images can preserve spatial properties, which are especially
important to recognize medical images [
X Y IDM(Q, R) = X X min X x=1 y=1 |x′|,|y′|≤W1|x′′|,|y′′|≤W2 ||R(x + x′ + x′′, y + y′ + y′′) − Q(x + x′′, y + y′′)||2 Where I(x, y), I ∈ {R, Q} indicates the pixel value of an image R, Q at position (x, y). The results are improved if the image gradient is used instead of the intensity values. For our experiment, we used W1 = 2 (5 × 5 pixel-sized search window for corresponding pixels) and W2 = 1 (3 × 3 pixels of local context). The images were scaled to a fixed height of 32 pixels keeping their original aspect ratio. 3.4
Classifier combinations can be grouped into three main categories [
ρ(Q, R) = α · J ST (Q, R) + β · IDM(Q, R) + γ · AR(Q, R).
As a matter of fact α, β and γ are parameters of the function ρ. Yet for the retrieval application described in this paper they are considered as constants that were empirically determined beforehand on the IRMA medical image corpus. DB denotes the entire image repository that is to be searched and n = |DB| is the number of images in the repository. Relevance determination applies the classifier to all elements of the database. Consequently time consuming ρ-functions are computed for many irrelevant comparisons. For this purpose a sieve is applied to reduce the number of potentially relevant references. In this work the sieve is applied to the IDM classifier by the following steps. First, compute a neighbor list using Euclidian distance on 16 × 16 representations of the query image and the database images. Afterwards, the IDM is applied to the closest k database images. Consequently, the computation time is reduced by the factor n/k. On the other hand, the IDM can only reorder the results. The sieve is defined by: 1. DS(I, S), Compute a S × S representation for I 2. δ(A, B) = ||DS(A, 16), DS(B, 16)||2 3. DB′(Q, DB, k) = (R1, ..., Rk)|δ(Q, R1) ≤ δ(Q, R2) ≤ ... ≤ δ(Q, Rn), R1..Rn ∈ DB, n = |DB| Based on the sieve DB′(Q, DB, k), the most relevant images are now selected by application of
REL(Q, DB′(Q, DB, k)) 4
Aim of the experiments is the verification whether the query design that yielded good results on the IRMA database could be transferred to another database with minimal changes to the parameterization. 4.1 The casimage database consists of 8,723 images and represents a mixture of images from clinical routine and drawings from medical education. Furthermore there are images with secondary added contents such as pseudo-colorings of segmented ultrasound images or manually placed marker arrows for operation planning, which do not represent original data.
From this dataset 26 samples have been arbitrarily selected as query images. The experimental task was to extract similar images to each of the 26 samples and provide a list of query results for manual evaluation. Thus the ground truth for each query was a-priori unknown and could not be optimized. This setting corresponds to free database exploration without any knowledge about its structure and not to mention the relevance of the results. On behalf of that there is no specific quantization and threshold computation to cut off the list from REL(Q, DB) with respect to each Q. Since the actual amount of relevant images to a query Q that are contained in the database is unknown, a fixed set of possible results is to be presented. In the experiments the cutoff is set to 100 images since this number is manually manageable in a browser view. These are lifelike values for free database exploration as used to understand the structure of a database. For comparable quantitative evaluation, a ground truth is mandatory. It was provided by three medical experts form the Geneva University Hospital for the casimage dataset for analysis after deadline for result submission. 4.2
The capacity of the concept is best verified by examination of the effect of different query designs on the results. Since the function ρ(Q, R) provides the central query functionality it is manipulated AR JST
IDM Comb 1 Comb 2 Comb 1’ Comb 2’ to optimize the results. As explained in section 3.4 this means the adaptation of the parameters α, β and γ. Table 1 lists the settings for the combined classifier weights, that have been used for the experiments.
For each of the 26 query images Q the first 100 images from the sequence REL(Q, DB) have been considered. They have been compared to the ground truth provided for the casimage data by the usual measures of precision and recall. Precision quantifies the percentage of relevant images that were among the 100 images returned. Recall denotes the ratio of relevant images to all images that should have been returned. Due to the restriction to 100 replies, recall will never reach 100 percent for queries with more than 100 relevant results in the database. Also precision will be low for query images which have significantly less than 100 images among the dataset. 4.3
Besides the quality of results the runtime of the algorithms must be considered. Especially if parameters must be adjusted in a feedback cycle. Furthermore the retrieval system must be capable of handling several queries at a time with the respective response times. And finally the setup of the parameters has to be efficient for efficient verification cycles. For this purpose the extraction of features at the image entry time (Sec. 2.1) must be considered separately from the feature comparison (Sec. 2.2). Furthermore runtime is reduced by preliminary application of the sieve DB′(Q, DB, k) to reduce the number of necessary comparisons. In the conducted experiments the cutoff value k was set to 500. For quality comparison the the combinations Comb 1 and Comb 2 are also applied on DB′(Q, DB, k) which extends the experiment set by Comb 1′ and Comb 2′. 5 5.1
The results for precision and recall for each of the classifier combinations are listed in Table 2 and Table 4. For the combined classifiers Comb 1 and Comb 2 the best precision was obtained for image 24 and the worst precision for image 14. While best recall for the combined measures was also for image 14 the worst recall for Comb 1 and Comb 2 was for image 23. Overall Comb 2 yielded the highest average precision. The results for Comb 1′ and Comb 2′ on the reduced datasets are only slightly inferior in the average precision. Several single results are even better such as for query 5. Figure 1 illustrates excerpts from the results for three queries. For four query images, precision is perfect or near perfect, whereas several query images yielded unsatisfactory results, especially queries 4 (43 relevant images), 11 (9), 14 (11), 17 (31), and 23 (74).
AR 0.10 0.01 0.08 0.09 0.03 0.20 0.00 0.09 0.01 0.04 0.00 0.23 0.02 0.01 0.14 0.02 0.00 0.20 0.01 0.06 0.02 0.10 0.03 0.15 0.38 0.13 0.08 Computation of all requires feature representations takes approximately 7.5 hours while the query computation for the combined measures for a single image takes about 5 minutes (Tab. 3) on a standard Pentium PC running at 2.4 GHz. The sieve based computation of the combined measures Comb 1′ and Comb 2′ yields a significantly faster runtime of 18.7 seconds for a single query. 6
The application of image features and their respective distance measures, including a parameter set, which was optimized on the IRMA database, yields useful retrieval results on the previously unknown casimage image data set. However, the casimage data set also includes color images, which demand additional features to capture their specific properties.
The variability of the precision computed for all queries is high, since the actual number of relevant images for all queries varies. For 16 query images there are less than 100 relevant images in the database. Consequently the number of false positive results is at least the difference between the relevant images and the preset threshold. On the other hand the recall is biased, for the ten images for which more than 100 relevant results exist. Thus methods for an automated threshold estimation are needed to improve query results.
AR JST
IDM Comb 1 Comb 2 Comb 1′ Comb 2′
By application of the query related sieve on the database the number of costly IDM comparisons is significantly reduced. On the other hand the query results have only slightly lost in average precision and recall. This encourages the use of more sophisticated classifiers for online retrieval applications such as PACS integration. 7
The presented approach considers the entire application cycle of an image retrieval system. By separating the application logic from the storage concept the transfer of new features classifiers and image data in the existing system requires no changes of existing implementations. Yet the quality of the results is not as good as it could be. The transfer of existing retrieval solutions onto new data repositories still requires some adaptive efforts. On the other hand the results can be compared to middle-rate results of manually optimized approaches. This is encouraging since optimization is always time consuming with respect to verification times of good classifiers such as the IDM. It is not necessary to start from scratch if a retrieval concept is transferred to another database. In contrast if the system is well defined the actual optimization is efficiently prepared.
However this conclusion has to be verified on various image repositories. Furthermore there is still the gap between fast computable query designs and good retrieval results. Powerful classifiers as required by medical applications still require long computation times. This remains a field of ongoing research where the IRMA system provides a supporting framework for efficient verification and also application.