Automatic segmentation of organs-at-risk (OAR) in computed tomography (CT) is an essential part of planning effective treatment strategies to combat lung and esophageal cancer. Accurate segmentation of organs surrounding tumours helps account for the variation in position and morphology inherent across patients, thereby facilitating adaptive and computer-assisted radiotherapy. Although manual delineation of OARs is still highly prevalent, it is prone to errors due to complex variations in the shape and position of organs across patients, and low soft tissue contrast between neighbouring organs in CT images. Recently, deep convolutional neural networks (CNNs) have gained tremendous traction and achieved state-of-the-art results in medical image segmentation. In this paper, we propose a deep learning framework to segment OARs in thoracic CT images, specifically for the: heart, esophagus, trachea and aorta. Our approach employs dilated convolutions and aggregated residual connections in the bottleneck of a U-Net styled network, which incorporates global context and dense information. Our method achieved an overall Dice score of 91.57% on 20 unseen test samples from the ISBI 2019 SegTHOR challenge.
Organs at risk (OAR) refer to structures surrounding tumours,
at risk of damage during radiotherapy treatment [1]. Accurate
segmentation of OARs is crucial for efficient planning of
radiation therapy, a fundamental part of treating different types
of cancer. However, manual segmentation of OARs in
computed tomography (CT) images for structural analysis, is very
time-consuming, susceptible to manual errors, and is subject
to inter-rater differences[1][
In this paper, we propose a fully automatic 2D
segmentation approach for the esophagus, heart, aorta, and trachea, in
CT images of patients diagnosed with lung cancer. Accurate
multi-organ segmentation requires incorporation of both local
and global information. Consequently, we modified the
original 2D U-Net [
Segmentation tasks generally benefit from incorporating local
and global contextual information. In a conventional U-Net
[
We propose a 2D U-Net+DR (refer to Fig.3.) network
inspired by our previous studies [
The ISBI SegTHOR challenge1 organizer provided the
computed tomography (CT) images from the medical records of
60 patients. The CT scans are 512 512 pixels in size, with
an in-plane resolution varying between 0.90 mm and 1.37 mm
per pixel. The number of slices varies from 150 to 284 with
a z-resolution between 2mm and 3.7mm. The most common
resolution is 0.98 0.98 2.5 mm3. The SegTHOR dataset
(60 patients) was randomly split into a training set: 40
patients(7390 slices) and a testing set: 20 patients(3694 slices).
The ground truth for OARs was delineated by an experienced
radiation oncologist [
Due to low-contrast in most of CT volumes in the SegTHOR dataset, we enhanced the contrast slice-by-slice, using contrast limited adaptive histogram equalization (CLAHE), and normalized each volume with respect to mean and standard deviation. In order to retain just the region of interest (ROI), i.e. the body part and its anatomical structures, as the input to our network, each volume was center cropped to a size of 288 288 along the x and y axes, while the same number of slices along z were retained. We trained the model using the provided training samples via five-fold cross-validation (each fold comprising 32 subjects for training and 8 subjects for validation). Moreover, we applied off-line augmentation to increase the number of subjects within the training set, by flipping the volumes horizontally and vertically.
In order to train our model, we formulated a modified version
of soft-Dice loss [
N P ynky^nk k=1 puts are five probabilities with classes of k = 0; 1; 2; 3; 4, such that Pk y^n;k = 1 for each pixel. Correspondingly, if yn;k is the one-hot encoded ground truth of that pixel, then the multiclass soft Dice loss is defined as follows: dc(y; y^) = 1 1 (X N k
Pn ynky^nk Pn ynk + Pn y^nk ) (1) In Eq. (1) y^nk denotes the output of the model, where n represents the pixels and k denotes the classes. The ground truth labels are denoted by ynk.
Furthermore, in the second stage of the training (described
in detail in the next section), we used Tversky Loss (TL)[
Also by adjusting the hyper-parameters and (refer to Eq. 2) we can control the trade-off between false positives and false negatives. In our experiments, we set both and to 0.5. Training with this loss for additional epochs improved the segmentation accuracy on the validation set as well as on the SegTHOR test set, compared to training with the multiclass Dice loss alone.
The adaptive moment estimation (ADAM) optimizer was used to estimate network parameters throughout, and the 1st and 2nd-moment estimates were set to 0.9 and 0.999 respectively. The learning rate was initialized to 0.0001 with a decay factor of 0.2 during training. Validation accuracy was evaluated after each epoch during training, until it stopped increasing. Subsequently, the best performing model was selected for evaluation on the test set. We first trained our model using five-fold cross-validation without any online data augmentation and using only multiclass Dice loss function. In the second stage, in order to improve the segmentation accuracy, we loaded the weights from the first stage and trained the model with random online data augmentation (zooming, rotation, shifting, shearing, and cropping) for 50 additional epochs. This lead to significant performance improvement on the SegTHOR test data. As the multiclass Dice loss does not account for class imbalance, we further improved the second stage of the training process, by employing the TL in place of the former. Consequently, the highest accuracy achieved by our approach employed the TL along with online data augmentation. The network was implemented using Keras, an open-source deep learning library for Python, and was trained T L(y; y^) = 1
N P ynky^nk + k=1 on an NVIDIA Titan X-Pascal GPU with 3840 CUDA cores and 12GB RAM. On the test dataset, we observed that our model predicted small structures in implausible locations. This was addressed by post-processing the segmentations, to retain only the largest connected component for each structure. As the segmentations predicted by our network were already of good quality, this only lead to marginal improvements in the average Dice score, of approximately 0.002. However, it substantially reduced the average Hausdorff distance, which is very sensitive to outliers.
Two standard evaluation metrics are used assess segmentation accuracy, namely, the Dice score coefficient (DSC) and Hausdorff distance (HD). The DSC metric, also known as F1score, measures the similarity/overlap between manual and automatic segmentation. DSC metric is the most widely used metric to evaluate segmentation accuracy, and is defined as: DSC(G; P ) =
2T P (F P + 2T P + F N ) = 2jGi \ Pij jGij + jPij
The HD is defined as the largest of the pairwise distances from points in one set to their corresponding closest points in another set. This is expressed as:
HD(G; P ) = max g2G max p2P ( ( pg2 p2 )) (3) (4) In Eq. (3) and (4), (G) and (P ) represent the ground truth and predicted segmentations, respectively.
The average DSC and HD measures achieved by 2D
UNet+DR across five-fold cross-validation experiments are
summarized in Table 1. The DSC scores achieved by the
2D U-Net+DR without data augmentation for the validation
and test sets were 93.61% and 88.69%, respectively. The
same network with online data augmentation significantly
improved the segmentation accuracy to 94.53% and 91.43%
for the validation and test sets, respectively. Finally, on
finetuning the trained network using the TL we achieved DSC
scores of 94.59% and 91.57%, respectively. Compared to
[
The images presented in Fig.3 help visually assess the segmentation quality of the proposed method on three test volumes. Here, the green color represents the heart, and the red, blue and yellow colors represent the esophagus, trachea, 2D U-Net + DR 2D U-Net + DR (Augmented) 2D U-Net + DR (Augmented) + TL
Train Data Validation Data
DSC [%] DSC [%]
0.9784 0.9361
0.9741
0.9749
and aorta respectively. We obtained the highest average DSC
value and HD for the heart and Aorta because of its high
contrast, regular shape, and larger size compared to the other
organs. As expected, the esophagus had the lowest average DSC
and HD values due to its irregularity and low contrast,
making it difficult to identify within CT volumes. However, our
method achieved a DSC score of 85.8% for the esophagus on
test data set, demonstrating better performance in
comparison to the method proposed in [
Fig. 3. 3D surface segmentation outputs of proposed model for three subjects from ISBI SegTHOR challenge test set.
In this study, we presented a fully automated approach, called 2D U-Net+DR, for automatic segmentation of the OARs (esophagus, heart, aorta, and trachea) in CT volumes. Our approach provides accurate and reproducible segmentations, thereby aiding in improving consistency and robustness in delineating OARs, relative to manual segmentations. The method uses both local and global information, by expanding the receptive-field in the lowest level of the network, using dilated convolutions. The two-stage training strategy employed here, together with the multi-class soft Dice loss and Tversky loss, significantly improved the segmentation accuracy. Furthermore, we believe that including additional information, e.g. MR images, may be beneficial for some OARs with poorly-visible boundaries such as the esophagus.
[1] I. Bulat and L. Xing, “Segmentation of organs-at-risks in head and neck ct images using convolutional neural networks,” Medical Physics, vol. 44, no. 2, pp. 547–557, 2017.