Deep learning has shown excellent performance in analysing medical images. However, datasets are di cult to obtain due privacy issues, standardization problems, and lack of annotations. We address these problems by producing realistic synthetic images using a combination of 3D technologies and generative adversarial networks. We use zero annotations from medical professionals in our pipeline. Our fully unsupervised method achieves promising results on ve real polyp segmentation datasets. As a part of this study we release Synth-Colon, an entirely synthetic dataset that includes 20 000 realistic colon images and additional details about depth and 3D geometry: https://enric1994.github.io/ synth-colon ? This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 765140. This publication has emanated from research supported by Science Foundation Ireland (SFI) under Grant Number SFI/12/RC/2289 P2, co-funded by the European Regional Development Fund.
Colorectal cancer is one of the most commonly diagnosed cancer types. It can be treated with an early intervention, which consists of detecting and removing polyps in the colon. The accuracy of the procedure strongly depends on the medical professionals experience and hand-eye coordination during the procedure, which can last up to 60 minutes. Computer vision can provide real-time support for doctors to ensure a reliable examination by double-checking all the tissues during the colonoscopy.
The data obtained during a colonoscopy is accompanied by a set of issues that prevent creating datasets for computer vision applications. First, there are privacy issues because it is considered personal data that can not be used without the consent of the patients. Second, there are a wide range of cameras and lights used to perform colonoscopies. Every device has its own focal length, aperture, and resolution. There are no large datasets with standardized parameters. Finally, polyp segmentation datasets are expensive because they depend on the annotations of quali ed professionals with limited available time.
We propose an unsupervised method to detect polyps that does not require
annotations by combining 3D rendering and a CycleGAN [
The contributions of this paper are as follows: { To the best of our knowledge, we are the rst to train a polyp segmentation model with zero annotations from the real world. { We propose a pipeline that preserves the self-generated annotations when shifting the domain from synthetic to real. { We release Synth-Colon (see Figure 1), the largest synthetic dataset for polyp segmentation including additional data such as depth and 3D mesh.
The remainder of the paper is structured as follows: Section 2 reviews relevant work, Section 3 explains our method, Section 4 presents the Synth-Colon dataset, Section 5 describes our experiments, and Section 6 concludes the paper." 2
Here we brie y review some relevant works related to polyp segmentation and synthetic data. 2.1
Early polyp segmentation was based in the texture and shape of the polyps.
For example, Hwang et al. [
With the rise of convolutional neural networks (CNNs) [
The limitation of using large neural networks is that they often require large amounts of annotated data. This problem is particularly acute in medical imaging due to problems in privacy, standardization, and the lack of professional annotators. Table 1 shows the limited size and resolution of the datasets used to train and evaluate existing polyp segmentation models. The lack of large datasets for polyp segmentation can be addressed by generating synthetic data.
Thambawita et al. [
Synthetic images combined with generative networks have also been widely
used in the depth prediction task [
Unlike previous works, our method is entirely unsupervised and does not require any human annotations. We automatically generate the annotations by de ning the structure of the colon and polyps and transferring the location of the polyps to a 2D mask. The key di erence between our approach and other state-of-the-art is that we combine 3D rendering and generative networks. First, the 3D engine de nes the structure of the image and generates the annotations. Second, the adversarial network makes the images realistic.
Similar unsupervised methods have also been successfully applied in other
domains like crowd counting. For example, Wang et al. [
Our approach is composed of three steps: rst, we procedurally generate colon images and annotations using a 3D engine; second, we feed a CycleGAN with images from real colonoscopies and our synthetic images; nally, we use the realistic images created by CycleGAN to train an image segmentation model. 3.1
The 3D colon and polyps are procedurally generated using Blender, a 3D engine that can be automated via scripting.
Our 3D colons structure is a cone composed by 2454 faces. Vertices are randomly displaced following a normal distribution in order to simulate the tissues in the colon. Additionally, the colon structure is modi ed by displacing 7 segments as in Figure 3. For the textures we used a base color [0.80, 0.13, 0.18] (RGB). For each sample we shift the color to other tones of brown, orange and pink. One single polyp is used on every image, which is placed inside the colon. It can be either in the colon's walls or in the middle. Polyps are distorted spheres with 16384 faces. Samples with polyps occupying less than 20,000 pixels are removed.
Lighting is composed by a white ambient light, two white dynamic lights that project glare into the walls, and three negative lights that project black light at the end of the colon. We found that having a dark area at the end helps CycleGAN to understand the structure of the colon. The 3D scene must be similar to real colon images because otherwise, the CycleGAN will not translate properly the images to the real-world domain. Figure 4 illustrates the images and ground truth generated by the 3D engine. 3.2
A standard CycleGAN composed by two generators and two discriminators is trained using real images from colonoscopies and synthetic images generated using the 3D engine as depicted in Figure 6. We train a CycleGAN for 200 epochs and then we infer real images in the \Generator Synth to Real" model, producing realistic colon images.
Figure 5 displays synthetic images before and after the CycleGAN domain adaptation. Note that the position of the polyps is not altered. Hence, the ground truth information generated by the 3D engine is preserved. 3.3
After creating a synthetic dataset that has been adapted to the real colon textures,
we train an image segmentation model. We used the HarDNeT-MSEG [
We publicly release Synth-Colon, a synthetic dataset for polyp segmentation. It
is the rst dataset generated using zero annotations from medical professionals.
The dataset is composed of 20 000 images with a resolution of 500×500.
SynthColon additionally includes realistic colon images generated with our CycleGAN
and the Kvasir training set images. Synth-Colon can also be used for the colon
depth estimation task [
{ 3D meshes of the colon and polyp in OBJ format. 5
5.1
We use two common metrics for evaluation. The mean Dice score, given by: and the mean intersection over union (IoU): mDice =
2 × tp 2 × tp + f p + f n mIoU =
tp tp + f p + f n (1) (2) where in both forumlae, tp is the number of true positives, f p the number of false positives, and f n the number of false negatives. 5.2
We evaluate our approach on ve real polyp segmentation datasets. Table 2 shows
the results obtained when training HarDNeT-MSEG [