The covid-19 pandemic has quickly spread all over the world, overwhelming public healthcare systems in many countries. In this situation demand for automatic assistance systems, to facilitate and accelerate a doctor's job has rapidly increased. Antibody tests were introduced for diagnosing covid-19, but physicians still need tools for quantification of disease severity, since treatment choice strongly depends on it. To estimate the severity of the disease physicians use computer tomography scans. It provides physicians with information about lung lesions and their types and they use this information to determine proper treatment. In this paper we made an attempt to build a system that uses patients' computer tomography scans for lung and lesion segmentation and for segmentation of specific types of lesions (i.e. pulmonary consolidation and “crazypaving”). Models for lung, lesions, consolidation, and “crazy-paving” segmentation performed with 0.96, 0.65, 0.48, 0.45 Dice coefficients respectively. Also it was shown that removing images with inaccurate ground-truth from the training subset can improve the quality of models trained on it.
Covid-19 is an infectious disease caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) that has a considerable mortality rate. Quick spread of this disease caused a pandemic, which overwhelmed healthcare systems in a large number of countries. As of 1 July 2020, 10.5M cases were confirmed worldwide and 512 thousands of patients died, so the average mortality rate was about 5%. In Russia it was 1.5% (9.5 thousands deaths for 654 thousands cases).
Early diagnosis can reduce the time and intensity of medical treatment. This can be achieved by the use of computer tomography (CT). CT is a radiography in which a three-dimensional image of a body structure is constructed by computer from a series of plane cross-sectional images made along an axis. CT provides physicians with information about the lesions in lungs. Then they can calculate the percentage of lung opacity and identify the type of the lesion so they can select appropriate treatment for the patient and monitor the course of the disease.
Computed tomography (CT) is considered to be a primary tool for giving diagnosis of covid-19 and evaluation of the disease progression. CT is a radiography in which a three-dimensional image of a body structure is constructed by computer from a series of plane cross-sectional images made along an axis. CT provides radiologists with information about disease features, or radiographic findings. These features are examined in order to identify their type and volume. Then this information is used for selection of appropriate treatment for the patient and monitoring the course of the disease.
The primary CT findings of covid-19 have been reported in [
The aim of this study was to create a solution for segmentation of lungs and abnormal regions of lungs and for detecting regions with pulmonary consolidation and “crazy-paving” pattern. 2
As covid-19 has quickly been spreading, computer vision scientists started to search for solutions that could help physicians diagnose their patients. There were several areas to research.
Before the mass use of antibody tests some scientists would try to find out if a
patient had covid-19, using only their CT scans. Linda Wang [
But those studies could not be applied for tracking patients’ condition. To achieve that, some scientists concentrated on a segmentation task, detecting lungs and areas with lesions. This information could then be used for measuring the percentage of lung opacity to quantify severity of the disease. Segmentation could be performed on 2D
Segmentation of Lungs and Lesions on Chest CT Scans 3
horizontal slices of CT or on full 3D scans. Lu Huang [
The dataset was provided by Third Opinion Platform [
The dataset was split into training and testing subsets, containing 85% and 15% studies respectively.
Radiologists used an assessment tool to draw polygons around regions of interest. As they tend to draw areas with smoothed boundaries, while models for segmentation calculate more precise masks, it is not expected to get results close to ideal according to quality metrics. 4
For model evaluation three metrics were used: mean AP over test studies, IoU and Dice coefficient.
Model’s recall is calculated as the number of true positive pixels divided by the number of all positive pixels. Model’s precision is calculated as the number of true positive pixels divided by the number of pixels, predicted as positive. AP is calculated as an area under precision-recall curve, where each point on the curve plot represents precision and recall of results with a threshold corresponding to the point.
IoU measures how much ground truth and predicted areas overlap: (1) (2) IoU =
T P
F N + T P + F P Dice =
2T P
F N + 2T P + F P Dice coefficient is often used to evaluate the quality of segmentation of medical images: TP is the number of true positive pixels, FN – number of false negative pixels, FP – number of false positive pixels. 5
Due to the fact that only 20% horizontal slices from each CT were assessed, it was
necessary to use a neural network for 2D segmentation of images. In this work U-net
[
At first, image normalization is performed on each slice image:
Inorm =
Iorig (3)
Segmentation of Lungs and Lesions on Chest CT Scans 5 where mu is the mean of the image and sigma is its standard deviation. It was done to unify information from different X-ray machines, as due to different settings they can produce information in different ranges of values. Then random spatial transformations (horizontal flipping, shifting, scaling, rotating) were applied on every slice to augment images.
so it could be possible to calculate the percentage of lung opacity. Diseased areas and lungs are segmented on each horizontal slice.
Several losses were used to train U-net. For training with binary cross entropy loss and Dice loss models predicted two masks: probability of lung in the pixel and probability of affected lung (either pulmonary consolidation or “ground-glass” opacity). Another model was trained using softmax cross entropy loss. It predicted 3 classes: background, healthy lung and affected lung, so the mask with background was added and lung class was replaced by healthy lung.
In the table 2 results for lung segmentation and lesion segmentation are shown. The best result for lesion class by all metrics was shown by the model that was trained with Dice loss. Models with BCE loss and Dice loss showed similar results for lung class.
second task was to identify different types of pathologies. Experiments showed that the use of samplers was necessary in this task, as the majority of images did not contain consolidation or “crazy-paving”, so training was unstable as the model received an enormous amount of negative examples.
Several models were trained. The first one (U4) predicted four binary masks for the following classes: lungs, consolidation, “ground-glass” opacity, and “crazy-paving”. During training batch size was divisible by 4. In every four elements of batch the first element contained image with consolidation, the second — “crazy-paving”, the third — “ground-glass” opacity and the fourth contained image with healthy lungs with 95% probability and image without lungs (i.e. slices from the top or the bottom of CT) with 5% probability.
The second model (C) predicted two binary masks: mask for lung and for consolidation probabilities. Sampler was used so each sample in a batch during training contained consolidation with 95% probability, lungs without consolidation with 2.5% probability and no lungs with 2.5% probability. The third model (CP) was similar to the second, but made predictions for “crazy-paving”.
All three models were trained with binary cross entropy loss, Dice loss, and softmax cross entropy loss. For training with CE loss, the model was supposed to predicted the background class and unaffected lung class instead of the lung class.
Experiment U4 BCE U4 Softmax CE C BCE C Dice C Softmax Experiment U4 BCE U4 Softmax CE CP BCE CP Dice CP Softmax For U4 BCE model confusion matrices for train and test subsets were calculated.
As figure 4 shows, on both train and test subsets the model confuses lesion classes and healthy lungs. This can be explained by the fact that some assessors tend to draw inaccurate masks, while models produce precise result.
Also figure 4 shows, that on both train and test subsets the model significantly confuses “ground-glass” opacities that are not “crazy-paving” and the “crazy-paving”. It also confuses consolidation and “ground-glass”. This could be explained by the presence of ambiguous examples in the dataset and/or noisy assessment, examples are presented on figure 5. To check the latter hypothesis three models (U4 BCE, C BCE, C Dice) were applied to the train dataset. Images from the train dataset were blocklisted if IoU of consolidation class was lower than 0.2 in any of those models. Then new model C BCE (*) was trained on a new dataset.
As results show even rough cleaning of the dataset could improve the result of training.
Training of a “crazy-paving” model and a united model was not performed, because after dataset reduction too few examples of “crazy-paving” remained. 7
In this study we presented a solution for segmentation of lungs, lesions, pulmonary consolidation and “crazy-paving” with reasonable quality. Experiments showed that binary cross entropy loss works better for training models that could distinguish different types of pathologies, but gives slightly inferior results than the model trained with Dice loss for segmentation of general classes such as lungs and lesions. Then it was shown that usage of noisy data during training can decrease the quality of a model and discarding such data from the training subset can improve the quality of a model.