This paper presents a neural network architecture for segmentation of medical images. The network trains from manually labeled images and can be used to segment various organs and anatomical structures of interest. We propose an efficient reformulation of a 3D convolution into a series of 2D convolutions in different dimensions. A loss function that directly optimizes intersection-over-union metric popular in medical image segmentation field is proposed. applications in image segmentation, but did not gather
Medical image is a visual representation of the interior of a body; it reveals internal anatomical structures and thus can be used for clinical analysis, intervention planning etc.
Volumetric
medical images are obtained from
various medical image acquisition technologies, such as
computed
tomography
(CT),
magnetic
resonance
tomography (MRT), etc. These images are represented
by a stack of 2D image slices thus forming a 3D
representation of a body [
Medical image segmentation is an automatic or semi-automatic process of splitting a medical image into regions, which
may correspond to an organ, a tissue, a tumor, or any other anatomical structure of interest.
Some of the
applications of medical image segmentation are surgical planning, virtual simulation of surgeries, tumor detection and segmentation, brain development study, functional mapping, automated classification blood cells, mass detection in mammograms, image registration, heart segmentation and analysis of cardiac images, border detection in angiograms of coronary, etc.
medical image segmentation techniques were based on low-level processing of image data (comparing gray level values of voxels to one or multiple thresholds, edge detector filters, unsupervised clustering algorithms etc.).
Later, supervised techniques, where training data (manually labeled examples) is used to train a model, became increasingly popular.
Examples of such methods are maximization maximum likelihood and expectation methods, maximum a posteriori and
methods, deformable models (active contour models, level set models), atlas-based models, conditional random field, graph cut algorithms.
the “Data
in (DAMDID/RCDL’2017), XIX
Convolutional
momentum until various new techniques and computing
architectures were developed. In December 2012 CNNs
won ImageNet challenge for the first time. AlexNet [
margin. In subsequent years,
further progress has been made [
A supervised learning algorithms experience a dataset, consisting of examples, each of which contains features xi and a target yi. For example, popular Iris dataset contains measurements of various species of iris plants. A supervised learning algorithm can study the Iris dataset and learn to classify iris plants into three different species based on their measurements. In our task, xi can be a computed tomography medical image, while yi can be a segmentation of that image done by an experienced radiologist.
An artificial neural network consists of many simple units called neurons.
Neurons receive and send information via
weighted connections. Each neuron calculates weighted sum of inputs and applies nonlinear activation function f to them: ℎ( ; , ) =
∗ + .
Historically popular choices for activation functions were ℎ, where ( ) =
1 1 + − .
Recently, one of the most popular activation functions used in computer vision are rectified linear units (ReLU) defined as: ( ) = , ≥ 0, ℎ 0
In a simple feed forward architecture, neurons are organized into groups called layers. Neurons in the first layer (called input layer) process information from the environment, while neurons in subsequent layers process information from previous layers. Neurons in the last layer (called output layer) produce information of interest. Because of this multi-layered structure, neural networks show very complex behavior: = ℎ(… ℎ(ℎ( ; 1, 1); 2, 2) … ; , ).
The universal approximation theorem states that a
feed-forward network with a single hidden layer
containing a finite number of neurons, can approximate
continuous functions. Thus, the theorem states that
simple neural networks can represent a wide variety of
interesting functions when given appropriate number of
parameters [
Convolutional neural networks (CNNs) are type of
artificial neural networks specialized for processing
data that has grid-like topology. Examples of such data
domains include 1D time-series data or 2D or 3D
images. Given two-dimensional image I and kernel K,
convolution operation can be defined as [
( − , − ) ∗ ( , ).
, Combining outputs of convolutions from earlier layers with new convolutions on later layers, a neural network can learn very complex features. Usually, first layers of convolutional neural network detect edges and angles, while later layers detect more complex features like eyes, hair, wheels, and even deeper layers detect human faces, cars etc. depending on the task at hand.
If we need to transform data in the direction
opposite to convolution, i.e., from something that has
the shape of the output of some convolution to
something that has the shape of its input while
maintaining a connectivity pattern that is compatible
with said convolution, we can use so called transposed
convolutions or deconvolutions [
In classification task, the final layer of a neural network computes scores applying convolutions to the output of previous layer. The scores are not bounded, but we would like to model probability distribution from them. In order to convert them to probabilities, a softmax function is used: e− ( ) = ∑ e− .
In order to train a neural network, we minimize a loss function ( , , ; ) with respect to , where = { 1, 2 … , 1, 2 … }, x and y are elements of training set, and is a prediction of the network. The loss function used in this paper will be presented in section 2.
The minimization of the loss function is achieved by calculating partial derivatives of the loss function with respect to parameters of the neural network, and then applying small changes to the parameters. The most basic optimization algorithm is stochastic gradient descent algorithm (SGD), which uses following update rule: where is a small constant called learning rate. is a hyper parameter of learning algorithm, usually good values for are between 0.01 – 10−6.
= + ( , , ; ) ∗ ,
As neural network may have many layers, gradients of the loss function are computed using backpropagation algorithm: first we computed gradients for last layer, and then we compute gradients of preceding layer using chain rule:
−1 −1
If neural network has many layers gradient information may be lost during this process (this problem is called vanishing gradients problem). One solution to this problem is skip connections: we sum outputs of deeper layers with outputs of more shallow layers, e.g.: = ℎ1 = ℎ( ; 1, 1), ℎ2 = ℎ(ℎ1; 2, 2), ℎ3 = ℎ(ℎ2; 3, 3) + ℎ1.
One of the most useful (and most popular) metrics in medical image segmentation is intersection-overunion metric (IoU). For volumes A and B, IoU is defined as: | ∩ | ( , ) = .
| ∪ |
CNNs can directly classify each pixel of an image
individually, given that we present to it a patch of image
around pixel of interest. However, a drawback of this
naïve sliding window approach is that input patches of
neighboring pixels have a huge overlap, and thus some
convolutions would be computed many times [
A significant speedup can be achieved if we present
many pixels to a CNN simultaneously. One of the first
implementations of this idea, that were successful in
medical image segmentation, were Fully Convolutional
Neural Networks (fCNN) [
The next iteration of fCNN idea is U-Net
architecture, where a typical convolutional network
architecture (contracting path) is followed by an
upsampling layers (expanding path) where the size of
an image is increased with upconvolutions. The
resulting network forms a U-shape giving the name of
the architecture. Another major improvement are
skipconnections which directly connect contracting and
expanding layers. The architecture showed very good
performance on a different biomedical segmentation
applications. Thanks to use of data augmentation with
elastic deformations, it only needs a very few annotated
images and has a very reasonable training time [
The 3D U-Net architecture developed ideas of
UNet further to construct a network for volumetric image
segmentation that learns from sparsely annotated
volumetric images. The implementation replaced all 2D
convolutions of U-Net by 3D convolutions. The authors
showed a successful application of the proposed method
on difficult data set of the Xenopus kidney [
We used convolution-deconvolution network based on U-Net architecture. The input is processed by blocks of convolutional operations. The data is downscaled with maxpool operations and fed to a next convolutional block in which we would increase number of channels twice to alleviate for loss of resolution. We upscale images with upconvolutions and concatenate data with signals from inner blocks before processing with another convolutional block. Thus, the neural networks forms U-shape with skip connections. Figure 1 summarizes the overall architecture of the neural network.
The skip connections were introduced as it is known that they reduce gradient vanishing problem.
A convolutional block (see Figure 2) consists of four 2D convolutions along different axis. This was done to optimize processing time, as even single 3x3x3 convolution has 27 parameters, while 4 3x3 convolutions have only 36 parameters. Each convolution is followed by ReLU nonlinearity and a dropout.
An upconvolution layer has 1x1 kernel which upscales the data, after which we concatenate upscaled data with output of convoltuion block with same size.
Our experiments showed that more popular softmax
cross-entropy function is harder to tune, as it optimizes
a metric (accuracy) that we’re not interested in and
needs tuning of weights of examples. In our setting, IoU
metric is much more informative. We extend loss
function presented in [
The proposed method was implemented using
TensorFlow library in Python 3 language [
A machine with Intel Core i7 6700K CPU, 32 Gb RAM, and NVidia GeForce GTX 1070 GPU was used to train a neural network and perform all experiments.
One of the problems we faced was limited memory of the video card. During training, we were not able to process a full image thus we had to split an image into blocks. This could potentially decrease accuracy for voxels close to the edges of the split because they would have less information about their neighbors. Our experiments showed that this is not a significant problem. To segment an image with trained model, we used TensorFlow’s ability to apply convolutions to inputs of variable size to speed up segmentation. However, the architecture of our network forced us to use images with dimensions 23 ∗ , as otherwise dimensions of upscaled images would not match original images.
The neural network showed strong signs of overfitting. We tested various regularization methods and obtained best results by using dropout right before output layer, as well as l2 regularization of convolutional filters’ weights.
We performed an extensive search for optimal
hyperparameters. Our program would select previous
best hyperparameters, randomly generate new ones in
interval [0.1*pbest, 10*pbest], perform 5000 training steps,
and select the network which showed higher IoU score
on validation set. We summarized final
hyperparameters that were used in Table 1.
not beneficial in our case and leads to numerical
instabilities [
In order to minimize numerical problems gradients were clipped to be less than or equal to 1.0.
In order to confirm ability of the neural network to
produce segmentation, we conducted an experiment to
segment heart’s left ventricle using the Cardiac Atlas
Project dataset [
Images in dataset were split to training set, validation set and test set. Validation set was used to tune the parameters of the neural network.
Figure 6 shows segmentation that was obtained using our convolutional neural network. Our model showed quality segmentation with IoU = 0.63.
Our experiment showed that convolutional neural network is capable of segmenting visually distinguishable anatomical structures on medical images. We plan to extend presented model to more medical image segmentation datasets.
Support. This research was supported by the Russian Foundation for Basic Research (grant 16-07-01028).