The second Liver CT (Computed tomography) annotation challenge was organized during the 2015 Image-CLEF workshop, hosted by the Institut de Recherche en Informatique de Toulouse (IRIT), University of Toulouse, France. This challenge entailed the annotation of Liver CT scans to generate structured reports. This paper describes the motivations for this task, training and test datasets, the evaluation methods, and discusses the approaches of the participating groups.
ImageCLEF [7] was part of the Cross Language Evaluation Forum (CLEF) 2015 consisting of four main tasks: Image Annotation, Medical Classification, Medical Clustering, and Liver CT Annotation. It was the second time that the automatic annotation of Liver CT images was provided as a challenge. However, there are two changes compared to the last edition of the challenge. First, the format of the given UsE (user expressed) features has been changed. Furthermore, there is no CoG (Computer generated) features provided this year. In the term of UsE features, LiCO (liver case ontology) is used instead of ONLIRA (ontology of liver for radiology)[4], which has additional patient and study information.
The purpose of the Liver CT annotation task is to automatically generate structured reports with use of computer generated features of liver CT volumes. Structured reports are highly valuable in medical contexts due to the processing opportunities they provide, such as reporting, image retrieval, and computeraided diagnosis systems. However, structured report generation is cumbersome and time consuming. Furthermore, their creation requires administration of the domain expert, who is time constrained. Consequently, such structured medical reports are often not found or are incomplete in practice. This challenge has been designed to aid the fill a pre-prepared structured report using the imaging information derived from CT images.
The data provided for this challenge consists of 50 training and 10 test
datasets. Participants were asked to answer a fixed set of multiple-choice
questions about livers. The questions were automatically generated from an
opensource liver case ontology (LiCO) [4] and provided in files with RDF (resource
description framework) format. The answers to the questions describe the
properties of the liver, the hepatic vasculature of the liver, and a specific lesion within
it. During this task, the user is presented with the following training data: (
The rest of the paper is organized as follows, Section 2 gives a detailed description of the task and introduces the participants. Section 3 presents the main results of the task and the results of the participants, and finally, Section 4 concludes the paper. 2
This section describes the task and introduces the participants. 2.1
Task Definition and Datasets The Liver CT annotation task is proposed towards the generation of structured reports describing the semantic features of the liver, its vascularity, and the types of its lesions. The goal of this task is to develop automated mechanisms to assist medical experts in difficult and practically infeasible task of annotating medical records.
The training dataset includes 50 cases, each consisting of: – a cropped CT image of the liver – a 3D matrix with the same size as cropped
CT image, – a liver mask that specifies the part corresponding to the liver – a 3D matrix indicating the liver areas with a 1 and nonliver areas with a 0, – a bounding box (ROI) corresponding to the region of the selected lesion within the liver – as a vector of 6 numbers corresponding to the coordinates of two opposite corners, – An RDF file generated using LiCOrepresenting manually entered imaging observations by a radiologist. In total, there are 73 UsE features. If a feature is not applicable for a case, it will not be represented in the corresponding RDF file.
In the training set, 50 ".mat" files, each containing the first three of above data, as well as an RDF file representing the imaging observations, have been given to the participants. In the test set, there is no RDF file and imaging observations are missing and participants are asked to predict them. The participants have been asked to extract and use their own image features to complete the task. RFD files include information of patient, study, and imaging observation. Participants are expected to predict the only imaging information, same as the last year’s challenge.
The resolution of CT images may vary in the range of (x : 190 308 pixels, y : 213 238 pixels, and z : 41 588 slices). The spacing may also vary in the range of (x; y : 0:674 1:007 mm, slice : 0:399 2:5 mm).
It is important to note that, this dataset is partially available through imageCLEF2015 system (http://medgift.hevs.ch:8080/CLEF2015/faces/Login. jsp), for academic use only. If you are interested in using this dataset, you need to properly cite this paper.
User Expressed Features Imaging observations of a radiologist for the liver domain are represented with LiCO. A web-based data collection application, called CaReRa-Web (case retrieval in radiological databases), which can be accessed for academic use from the CaReRa project website http://www.vavlab. ee.boun.edu.tr.
For each case, there are 73 UsE features represented in RDF format. These features clinically characterize the liver, hepatic vascularity, and liver’s lesions. In the training set, the UsE features are manually entered by an expert radiologist. Every UsE feature corresponds to a question answered by a radiologist. Some UsE features may take on more than one values. Such features are represented with a multi-choice answers. Features with value marked as "NA" are not included in the RDF file.
In the test phase, the participants are expected to predict the UsE features in the following format (73 4 UsE data):
Column Annotation Features Type 1 Group string 2 Concept string 3 Properties string 4 Values bar separated list of strings
The "Group" and "Concept" are the LiCO-based concepts. Each concept may have several properties. Each property may have multiple values, whose indices and meaning are given in Table 4 under "Possible values" column. Properties deemed irrelevant are marked as NA by the radiologist. Note that, UsE features are grouped as: Liver, Vessel, General, and Lesion. Table 4 lists every group and its corresponding concepts, properties, possible values, and their assigned indices.
In every RDF file, there is a patient which has a study and a set of data properties same as Name, age and gender, as well as a set of object properties same as disease and drugs. Each study has a set of data properties including ID and dates as well as a set of object properties same as laboratory results and physical examinations. Also each study has a liver, which has a set of imaging observation. Each liver also has a lesion, which has relevant imaging observation. 2.2
Evaluation methodology The evaluation is performed on the basis of completeness and accuracy of the predicted annotations with reference to the manual annotations of the test dataset. Completeness is defined as the number of predicted features divided by the total number of features, while accuracy is the number of correctly-predicted features divided by the total number of predicted features.
For answers that allow multiple values to a question, the correct prediction of a single feature is considered as the correct annotation.
Completeness = number of predicted U sE f eatures
T otal number of U sE f eatures Accuracy = number of correctly predicted U sE f eatures
N umber of predicted U sE f eatures T otalScore = pCompleteness
Accuracy
(
Participation Among 32 groups, which registered for the task and signed the license agreement to access the datasets, only 1 of them submitted his results. The group name is "CREDOM", from Biomedical Engineering Laboratory, Tlemcen University, Algeria. They have submitted three runs to the task. 3 3.1
Runs submitted in 2014 Last year in 2014 [1], three groups submitted their results. Their prediction were based on classifiers, image retrieval, and generalized-coupled tensor factorization (GCTF). Last year a set of computer generated (CoG) features were also provided for the participants as an optional data.
The best performance was achieved by the BMET group [5] submitting 8 runs using two different methods: classifier-based approach using SVM (support vector machine) and image retrieval algorithm. They also used two different sets of feature: the prepared CoG features from the database and a bag of visual words (BoVW). Their classification methods outperformed the other methods, when they employed their expanded feature set. However, their retrieval method gave the best results, when the given CoG features were employed. This observation suggests that the nature of feature sets are important for utilizing different methods.
The second best performance was achieved by CASMIP group [2] submitting only one run to the task. They tried four different classifiers in the learning phase: linear discriminant analysis (LDA), logistic regression (LR), K-nearest neighbors (KNN), and finally SVM to predict UsE features. For each UsE feature the best classifier and its related features were learnt by using exhaustive search. They used only a certain part of provided CoG features, as well as 9 additional features extracted in the lesion ROI. As a result, for most of the UsE features, they achieved the same performance using any classifier and any combination of image features.
piLabVAVlab group [3] considered the dataset as heterogeneous data and GCTF approach was applied to predict UsE features. They considered both KLdivergence and Euclidean-distance-based cost functions, as well as the coupled matrix factorization models using GCTF framework.
The BMET group achieved the highest scores with completeness of %98 (See Table 3. In terms of accuracy, BMET group has also attained the best score by using an image retrieval method. 3.2
Runs submitted this year in 2015 This year in 2015, three runs has been submitted by the "CREDOM" group [6]. They used two different methods: classification by using random forest (RF) classifier, and retrieval by considering the specific signature of the liver (See Table 1.
For the classification-based method (run 1 and run 2), they have employed two different sets of features. The first set contains 115 liver texture features in addition to 9 lesion geometric features, and the second set includes 214 lesion texture features, in addition to 9 lesion geometric features. Classification is performed by using supervised multi-class RF classifier. In this method, they divided the UsE features into two groups. For the first group, they have used the RF classifier, but for the second group, they have used a retrieval based method with their proposed similarity metric. The reason of proposing this separation is the unbalanced dataset.
Second method (run 3) is a retrieval-based method. Basically, they have encoded the 2D image extracted from the central slice of the lesion by applying 1D Log-Gabor filter, and then break the output of the filter into small blocks and quantize the dominant angular direction of each block to four levels by using Daugman method. Afterward, the Hamming distance has been employed as the similarity metric to retrieve the five most similar images to the test image. Finally, for each UsE feature, they have used majority voting between retrieved images.
Among 73 UsE features, 7 of them were excluded from the evaluation because of their unbounded labels (numeric continuous values). Table 1, compares the results of three submitted runs in terms of completeness, accuracy and total scores.
Table 2 compares the results of different runs in predicting different groups
of UsE features. We divide UsE features into 5 groups: liver, vessels and three
lesion groups with area, lesion and component concepts. Results show that all
runs have completed UsE features of the liver and vessel with high accuracy.
None of the runs can completely annotate the component-related concepts of
lesions. Lesion-related concepts of lesion are fully completed, while showing a
very low accuracy. Results show that the third run (retrieval based method),
outperforms the other runs.
This was the second time that the liver CT annotation task was organized.
We provided liver patient data collected via a hybrid patient information entry
system, whose liver characteristics are based on the LiCO ontology. The challenge
was to predict UsE features of patient records, given in RDF format. This year
in 2015, in contrast to last year’s edition, users have not been provided with
CoG features. Also, they were free to use any set of image features to perform
the task. As this was the first time that UsE features were provided in RDF
formats, it was not surprising that few groups were finally able to submit their
runs for this complex task. Out of 32 teams, 1 team submitted its runs. The
approaches and results were reviewed and documented in this paper. Since the
dataset was exactly the same as last year’s, comparison of the results of all runs
submitted to the liver CT annotation task in both 2014 and 2015 is also provided
in this paper. The main challenge of the task was due to the unbalanced dataset
and participants tried to overcome this issue with different methods. Among all
methods, image retrieval obtained the best performance. It was observed that
feature selection is important for the best performance of the prediction method.
Acknowledgments The Liver CT Annotation task is supported by TÜBİTAK
Grant # 110E264 (CaReRa project), and in part by COST Action IC1302
(KEYSTONE).
Group Concept Properties Possible values(assigned indices)
Liver Placement downward displacement(0), normal placement(
ward displacement(
Density Change decreased(0), increased(
Change Left Lobe dLaelftDLiombeenCsiroann(imocma)u- The amount change in size of left lobe(mm)
Left Lobe Size decreased(0), increased(
Change
Caudate Lobe CCraaundiaotceaudal LoDbie- The amount change in size of caudate lobe(mm)
mension(mm)
Caudate Lobe Size decreased(0), increased(
Hepatic Artery Lu- decreased(0), increased(
Hepatic Artery Lu- obliterated(0), open(
Hepatic Portal V. obliterated(0), open(
Portal Vein)
Left Portal Left Portal V. Lumen decreased(0), increased(
Left Portal V. Lumen obliterated(0), open(
Vein)
Right Portal Right Portal V. Lu- decreased(0), increased(
Right Portal V. Lu- obliterated(0), open(
Vein)
Hepatic Vein HDeiapmat.ic V. Lumen decreased(0), increased(
Diagnosis of given image using ICD10 codes (bar separated) and in the free text MD’s comments are written (bar separated).
Cluster Size 1(
For simple cases this value shows number of lesions inside
the ROI, but in case of having more than one lesions of a
certain type, the biggest lesion is annotated as a sample of
that cluster and number of lesions with same properties
is written here
Contrast Uptake NA(-1), dense(0), heterogeneous(
minimal(
PureSolid(
Properties Possible values(assigned indices)
Area Margin Type sgeeropgirgaipnhioiucas(l(40)),,spililcudleafitinveed(5(
ovoid(
other(
Density Type NA(-1), heterogeneous(0), homogeneous(
Capsule Calcification NA(-1), coarse(0), focal(
Type tate(
Pseudocapsule Calc. NA(-1), coarse(0), focal(
Type tate(
Thickness NA(-1), thick(0), thin(
Solid Component NA(-1), coarse(0), focal(
Calcification Type tate(