The rst Liver CT annotation challenge was organized during the 2014 Image-CLEF workshop held in Sheeld, UK. This challenge entailed the annotation of Liver CT scans to generate structured reports. This paper describes the motivations for this task, the training and test datasets, the evaluation methods, and discusses the approaches of the participating groups. abstract environment.
2,
ImageCLEF [1] was part of the Cross Language Evaluation Forum (CLEF) 2014 consisting of four main tasks: Robot Vision, Image Annotation, Liver CT Annotation, and Domain Adaptation. It was the rst time that the automatic annotation of Liver CT images was provided as a challenge.
The purpose of the Liver CT annotation task was to automatically generate structured reports with the use of computer generated features of liver CT volumes. Structured reports are highly valuable in medical contexts due to the processing opportunities they provide, such as reporting, image retrieval, and computer-aided diagnosis systems. However, structured reports are cumbersome and time consuming to create. Furthermore, their creation requires domain expertise who is time constrained. Consequently, such structured medical reports are often not found or are incomplete in practice. This challenge was designed to aid the generation of structured reports.
The datasets provided for this challenge consisted of 50 training and 10 test
datasets. Participants were asked to answer a xed set of multiple-choice
questions about livers. The questions were automatically generated from an
opensource ontology of liver for radiology (onlira) [2]. The answers to the questions
describe the properties of the liver, the hepatic vasculature of the liver, and
a specic lesion within the liver. During this task, the user is presented with
the following training data: (
The rest of the paper is organized as follows, Section 2 gives a detailed description of the task and introduces the participants. Section 3 presents the main results of the task and the results of the participants, and Section 4 concludes the paper. 2
This section describes the task and introduces the participants. 2.1
Task Denition and Datasets The Liver CT annotation task is proposed towards the generation of structured reports describing the semantic features of the liver, its vascularity, and the types of lesions in the liver. The goal of proposing this task is to develop automated mechanisms to assist in the dicult and practically infeasible task of annotating medical records.
The training dataset includes 50 cases, each consisting of: a cropped CT image of the liver a 3D matrix with the same size as cropped CT image, a liver mask that species the part corresponding to the liver a 3D matrix indicating the liver areas with a 1 and nonliver areas with a 0, a bounding box (ROI) corresponding to the region of the selected lesion within the liver as a vector of 6 numbers corresponding to the coordinates of two opposite corners, a set of 60 computer generated ( CoG) features obtained from an interactive segmentation software a 60 4 array, and A set of 73 user expressed features ( UsE) manually entered by a radiologist and stored in a 73 6 array.
In training dataset 50 .mat les, each consisting of all the above data were given to the participants. The format of the test dataset is the same except that the UsE features are missing, which the participants were expected to predict. The participants were allowed to extract and use their own image features. It is important to note that the resolution of CT images may vary ( x : 190 308 pixels, y : 213 238 pixels, and z : 41 588 slices). The spacing may also vary in the range of (x; y : 0:674 1:007 mm, slice : 0:399 2:5 mm). Computer Generated Features For each case, there is a set of 60 CoG image descriptors. Table 4 provides the list of all CoG features for a case. Some of them have only one value and the rest are vectors with dierent dimension. For example, the size of "HistogramOfAllLesions" is 67, while the size of "LiverVolume" is 1. The total dimension of all features is 454. The CoG features were extracted after interactively marking the liver, vessels, and lesions on a CT image. The CoG features describe the characteristics of the liver, vessels, and lesions. Lesion descriptors are categorized into ve types: geometric, locational, gray-scale, boundary, and texture features. The reader is referred to [3] for the details of CoG features. In both training and test dataset, CoG features are shown in a 60 array where each columns is stands for:
Column Feature Type 1 group string 1 name string 3 type string 4 value type of the feature User Expressed Features Imaging observations of a radiologist for the liver domain are represented with ONLIRA(Ontology of the Liver for Radiologists). A web based data collection application, called CaReRa-Web 4.
For each case, there are 73 user expressed ( UsE) features represented in a 73 6 cell array. These features clinically characterize the liver, hepatic vascularity, and liver’s lesions. In the training dataset, the UsE features are manually entered by an expert radiologist. Every UsE feature corresponds to a question answered by a radiologist. Some UsE features may take on more than one value. Such features are represented with a multi-selection answers.
In the test phase, the participants were expected to predict the UsE features. The format of the 73 6 UsE data is:
Column Annotation Features Type 1 Group string 2 Concept string 3 Properties string 4 Indices bar separated list of integers 5 Values bar separated list of strings 6 "Free text" related to value Text
The "Group" and "Concept" are the ONLIRA-based concepts. Each concept may have several properties. Each property may have multiple values whose indices and meaning are given in "Indices" and "Values" columns, respectively. Properties deemed irrelevant are marked as NA by the radiologist. UsE features are grouped as: Liver, Vessel, General and Lesion. Table 5 lists every group and its corresponding concepts, properties, possible values and their assigned indices. 4 The CaReRa-Web is a tool that can accessed at https://vavlab.ee.boun.edu.tr: 5904/CareraWeb2. It is available for academic use from the CaReRaproject (Case Retrieval in Radiological Databases) website http://www.vavlab.ee.boun.edu.tr 2.2
Evaluation methodology The evaluation is performed on the basis of the completeness and accuracy of the predicted annotations with reference to the manual annotations of the test dataset. Completeness is dened as the number of predicted features divided by total number of features, while accuracy is the number of correct predicted features divided by total number of predicted features.
For answers that allow multiple values to a question, the correct prediction of a single feature is considered as a correct annotation.
Completeness = number of predicted U sE f eatures
T otal number of U sE f eatures Accuracy = number of correctly predicted U sE f eatures
N umber of predicted U sE f eatures T otalScore = pCompleteness
Accuracy
(
Participation Among 20 groups, which registered for the task and signed the license agreement to access the datasets, only 3 of them submitted at least one run. The number of runs per group was limited to ten. Tables 1 describes these runs. The groups that submitted their results based their prediction on classiers, image retrieval, and generalized coupled tensor factorization (GCTF).
The BMET group, achieved the best results using the image retrieval techniques with total score of %94:7. Classier-based methods were used by both BMET and CASMIP groups. Only piLabVAVlab used a GCTF method. Table 2 shows the completeness, accuracy and total score achieved by each run on the test dataset.
The BMET group [4] submitted 8 runs, of which 4 of them used a classierbased approach and the remaining used an image retrieval algorithm. They used two dierent feature sets: the prepared CoG features from the database and a bag of visual words (BoVW). In the classier-based approach, they used twostage classication, where each stage consists of a bank of several support vector machines (SVM), which is used for each UsE feature. A two-stage classication is proposed to solve the unbalanced training dataset. For each UsE feature, the rst stage is composed of the 1-vs-all classiers and the second stage is consisted of the 1-vs-1 classiers. Second classier is activated only if the result of the rst step is more than one label. In the second stage they run 1-vs-1 classier for the set of labels resulting from the rst step and a majority voting scheme is used to select the nal answer. In their rst and second runs, they used linear kernels while in their third and fourth runs, they employed radial basis function (RBF) kernel. They examined these two kernels with both sets of features.
In the image retrieval based approach, they used the most similar training images to select the UsE features for the test image. Similarity is calculated by computing the Euclidean distance between image feature vectors. Finally, they applied a weighted voting scheme to select the label assigned to each UsE features using the "n" most similar images to the test image, where n = 10 in this scenario. Basically, this algorithm votes images more similarity to the test image with higher values. Results of this approach with two dierent sets of features are seen in 5th and 6th runs. In 7th and 8h runs, they applied a sequential feature selection method to use the most discriminating features for each question during the similarity calculation, in order to use the most suitable one. As mentioned above, BMET group used two kernels for SVM classication, however, there is no signicant performance dierence in the results, which the participants attribute to the unbalanced training dataset. Their classication methods performed best when they employed their expanded feature set. Their retrieval method performed best when the given CoG features were employed. This observation suggests that the nature of feature sets are important for utilizing dierent methods.
CASMIP group [5] submitted one run to the task, which achieved the second best performance. They tried four dierent classiers in the learning phase: linear discriminant analysis (LDA), logistic regression (LR), K-nearest neighbors (KNN), and nally SVM to predict UsE features. An exhaustive search of every combination of image features is done using leave-one-out cross validation method on training data for every UsE feature and classier. As the result, for each UsE feature the best classier and its related features are learned. They used only a certain part of provided CoG features, which was achieved by ignoring 21 high-dimension features, i.e. they ignored features with dimensionally more than one. Instead, 9 additional features have been added to individual lesion features extracted in the lesion ROI describing the gray level features of liver, lesion, and boundary of lesion properties. The learning step was performed using all UsE features of the training dataset except cluster size, lobe and segments, which were obtained directly from image features. Python scikit-Learn Machine learning toolbox was used for implementing each classier with the default parameters. As the result, for most of the UsE features they got same performance using any classier and any combination of image features. Hence they assigned any classier and all image features for them. For 6 of the UsE features that describe the density, contrast and location of lesions, one of the LDA or KNN classiers was chosen along with their selected features.
piLabVAVlab group [6] considered the dataset as heterogeneous data and GCTF approach was applied to predict UsE features. They considered both KLdivergence and Euclidean-distance-based cost functions as well as the coupled matrix factorization models using GCTF framework. They tried to predict approximately half of the UsE features. In order to achieve this, UsE features with only 4 indices whose values vary from 0 to 3 were considered as the rst group and UsE features with binary indices were considered as the second groups. The reason for this was that the threshold selection needed to be specied for each type of question. Thus, they considered questions with similar answer ranges in a study and ignore question with varied answer ranges. Basically, they provide three matrices for 50 training and 10 test cases:
X1: A 60 X2: A 60 Z1: A 60 21 matrix (UsE features of rst group). 13 matrix (UsE features of second group).
447 matrix (all CoG features).
They estimated the latent matrices: Z2 and Z3 by using coupled matrix factorization models according to the following formula:
X1 = Z1 X2 = Z1
Z2
Z3
(
The UsE features of the 10 test cases are predicted with Z2 and Z3. Since the predicted values are not discrete, a binary thresholding method has been proposed to extract the labels of UsE features.
This group submitted one run during the submission period, which had the accuracy of %45. However, after the submission deadline, they claimed that they had improved their thresholding method and requested that we evaluate their new results (see Table 2 run2 and run3).
Among 73 UsE features, 7 of them were excluded from the evaluation because of their unbounded labels (numeric continuous values). The BMET group achieved the highest scores with completeness of %98 (See Table 2. In terms of accuracy, BMET group has also attained the best performance by using an image retrieval method. In terms of classier-based methods, BMET and CASMIP groups both obtained the total score of %93.
Table 3 compares the results of dierent runs in predicting dierent groups of UsE features. We divide UsE features into 5 groups: liver, vessels and three lesion groups with area, lesion and component concepts. Results show that all the groups have completed the vessel UsE features with high accuracy. The BMET and CASMIP groups completed liver features in full with accuracy more than %80. None of the groups can completely annotate the area related concepts of lesions. Components related concepts of lesion are completed fully and annotated with accuracy higher than %72 by both BIMET and CASMIP groups. Lesion related concepts of lesions are annotated completely by only BIMET group with accuracy more than %72. This was the rst time the liver CT annotation task was proposed. We provided liver patient data collected via a hybrid patient information entry system whose liver characteristics are based on the ONLIRA ontology. The challenge presented to the participants was to predict UsE features of patient records, given CoG features. As this was the rst time for this challenge, it was not surprising that few groups were able to submit their runs for this complex task. out of 20 teams 3 teams submitted at least 1 run. The approaches and results were reviewed and documented in this paper.
The main challenge of the task was due to the unbalanced dataset and participants tried to overcome this issue with dierent methods. Among all methods image retrieval obtained the best performance. It was observed that feature selection is important for the best performance of the prediction method. Acknowledgments This work is in part supported by CaReRa Project (TBTAK Project No: 110E264) and Bogazici University B.A.P (Project No: 5324) Group Vessel General Patient
Concept Properties Possible values(assigned indices)
Left Hepatic Left Hepatic V. Lu- decreased(0), increased(
Left Hepatic V. Lu- obliterated(0), open(
Middle Hepatic V. obliterated(0), open(
Right Hepatic V. Lu- obliterated(0), open(
Diagnosis Lesion
Lesion is Leveling served? Leveling Type
Diagnosis of given image using ICD10 codes (bar separated) and in the free text MD’s comments are written (bar separated).
Cluster Size 1(
For simple cases this value shows number of lesions inside
the ROI, but in case of having more than one lesions of a
certain type, the biggest lesion is annotated as a sample of
that cluster and number of lesions with same properties
is written here
Contrast Uptake NA(-1), dense(0), heterogeneous(
NA(-1), uid uid(0), uid gas(
SegmentVIII(8)
width a number in mm which represents width of the lesion
height a number in mm which represents heigth of the lesion
is Gallbladder Adja- True(