This work aims to research an automatic method for detecting Age-related Macular Degeneration (AMD) lesions in RGB eye fundus images. For this, we align invasively obtained eye fundus contrast images (the “golden standard” diagnostic) to the RGB ones and use them to hand-annotate the lesions. This is done using our custom-made tool. Using the data, we train and test five diferent convolutional neural networks: a custom one to classify healthy and AMD-afected eye fundi, and four well-known networks: ResNet50, ResNet101, MobileNetV3, and UNet to segment (localize) the AMD lesions in the afected eye fundus images. We achieve 93.55 % accuracy or 69.71 % Dice index as the preliminary best results in segmentation with MobileNetV3.
lesion zone detection and evaluation “golden standard” is fluorescein angiography – a
diagnostic method where sodium fluorescein dye is injected intravenously for pathological
neovascularization visualization. However, this injection can have adverse efects, is costly, and
invasive to the patient [
Superposition of registered fluorescence eye fundus images of AMD patients with RGB eye fundus images of the same patients can produce a training set for deep learning neural networks. The network eventually can be trained to detect and localize the lesion zone in RGB images.
There are successful examples of classification between AMD disease-afected and healthy
eye fundi using OCT images [
This work aimed to assess the possibility of AMD lesion zone detection and evaluation in non-invasively registered RGB fundus images by using deep learning neural network architectures.
The objectives of our work are: 1. Prepare the data: a) Collect the AMD afected fundus images (RGB and contrast photos); b) Create the data preparation tool; c) Align RGB and contrast images; d) Annotate the lesion zone on the contrast images; e) Prepare comparable images without AMD. 2. Investigate automated algorithms: a) Train algorithms that can classify between images with and without AMD; b) Train algorithms that can segment the lesion area from the RGB image; c) Investigate performance of the algorithms and their applicability in real life.
We present our materials and methods in Section 2, introduce our dataset and its preparation in Section 2.1, present our algorithm investigations in Section 2.2, the obtained preliminary results in Section 3, and conclude with a discussion in Section 4.
We have gathered a collection of RGB and gray-scale contrast images of eye fundi. We discuss the details of this data and its preparation in Section 2.1.
44 sets of age-related macular degeneration examination images were collected from the Lithuanian University of Health Sciences Department of Ophthalmology. All these sets contain AMDafected cases of varying degrees.
Any personal information was removed and made sure that no identifiable details were left in file names or possible additional text files. The pictures from the same patient were grouped, making preliminary sets that contain a single RGB image and several contrast-enhanced images for damaged area detection.
The set of images containing AMD had to be prepared for training. The RGB and contrastenhanced images that were taken are misaligned by position, rotation, and scaling. In our case, we assumed that the tilt of the camera was not changed so only 3 latter parameters were used to match the images. The main reference points in matching images were the most visible vascular structures. To achieve a more convenient and efective image matching a specialized tool was created which is discussed in Section 2.1.1.
To investigate automatic classification between healthy and AMD-afected images, we
additionally took 15 healthy fundus images from Pattern Recognition Lab online database [
Before training data preparation, a convenient tool was created (Figure 1) to eficiently annotate and process raw data into training datasets for machine learning. The tool has the following functions: 1. Preview raw images; 2. Create a training set; 3. Annotate the training set; 4. Upload saved training sets to the online database; 5. Download other training sets from the online database for preview and editing. The main task of this tool is to produce a matched and uniformly sized collection of training data sets for later use in neural network model training. The training data must have RGB images where all pixels are marked with “ground truth” information as being in a lesion or unafected zone.
To create a dataset first the RGB and contrast-enhanced image (contrast image) are loaded into our dataset preparation tool. The tool lets a user add reference points in both images and applies contrast image transformation according to the reference points. If the resulting image is matched the user can continue painting the mask of the lesion area according to the matched contrast image. After this step, the tool produces an annotated and prepared training set which can be uploaded for the machine learning algorithm to train.
A single training dataset featured in Figure 2 contains an RGB image, a contrast-enhanced image, and a mask image which is in grayscale. The RGB image is the input data for algorithm Perform classification
of RGB image Is AMD lesion zone found?
No
Output the scalar
result Yes
Perform segmentation of RGB
image Output image and
scalar results training. The mask image corresponds to the lesion area of the retina and also is the desired output result after algorithm training. All dataset images are in preferred PNG format and 512x512 pixels resolution.
For the classification task, we needed to have both healthy and AMD-afected images. Since the two classes came from diferent sources, we took some steps to make the images more similar.
We scaled, centered, and cropped the images to make them the same resolution. One obvious diference that remained, was the diferent dominating tints of the two datasets. To eliminate this, we included the experiments with color histogram equalization of the images as the preprocessing step.
In our evaluation process, we investigate two diferent algorithms for classification and segmentation respectively. The classification algorithm detects whether AMD is present in the given RGB image at all. The segmentation algorithm detects the location of the AMD lesion zones by producing a gray-scale output image highlighting them. The segmentation part of the process is performed only if the classifier detects AMD.
The evaluation process flowchart is displayed in Figure 3. Each algorithm is described in more detail in Sections 2.2.1 and 2.2.2 respectively. 16@126x126 Convolution
Max-Pool
Convolution
For degeneration classification, a custom convolutional neural network was created which consists of 5 layers: 2 convolutions, 1 max-pooling, and 2 fully connected layers. The network takes the RGB image as the input and has a single output estimating the probability of degeneration. A detailed structure of classification neural network is presented in Figure 4.
For the degeneration segmentation algorithm, four well-known neural network architectures
were tried: ResNet50, ResNet101 [
The models were trained from scratch (random weight initialization). The training configurations of the segmentation neural networks are presented in Table 1.
The confusion matrix and its derived measures were used to assess the detection quality of both the classification and localization algorithms: specificity, sensitivity, and accuracy measures were calculated. For lesion semantic segmentation Sørensen–Dice (Dice) coeficient, a.k.a. F1 score, was also calculated.
Before computing these measurements, we binarized the gray-scale output images to only black and white pixels. We used four diferent thresholds of 0.01, 0.05, 0.1, and 0.5 to compare which threshold worked best for each semantic segmentation neural network.
The results of our classification algorithm are presented in Table 2. We can see that with histogram-normalized images, 99.93 % accuracy in the classification of images containing AMD lesions vs. healthy control can be achieved.
The results of our semantic segmentation models are presented in Table 3. We can see that the best Dice result was achieved with the lowest threshold when binarizing the output images in all neural networks except for MobileNetV3. The best Dice result of 69.71 % was achieved with it and a 0.05 threshold.
The very high classification accuracy is most likely since the images of the AMD-afected and the healthy eye fundi are taken from diferent sources - obtained from diferent populations using diferent equipment.
We tried to make them similar, but this is probably impossible to do completely. Firstly, we cropped the images to mitigate diferent framing. But the color diferences between the healthy and afected images were still visible by the naked eye. To eliminate these diferences as a likely telltale sign for the classification, we equalized the color histograms of all the images, as mentioned in Section 2.1.4. However, this produced an even better classification (Table 2).
In addition to equipment-based diferences such as diferent color grading, lighting, position, and possibly tilt, the biggest diference might be in the subject population. While this is not indicated in the control dataset, it appears that the subjects were young, and the AMD-afected images are taken from old people.
To overcome this, we would need to collect and prepare more training data including healthy eye fundus images collected under the same conditions as the afected ones. But this is dificult to achieve because the procedure of taking eye fundus images requires pupil dilation medicine and is not normally performed on healthy subjects.
Overall, the results of the classification part of our algorithm are not so important to ophthalmologists, but are useful to our lesion zone segmentation algorithm, improving its performance. We plan to further concentrate our efort on the segmentation part.
While observing typical outputs produced by our tested segmentation neural networks in Figure 5, we can see that the UNet model produces very uneven and disconnected annotations. They difer greatly from the target output and result in the worst performance. The other models produce more blurry-edged annotations but they are more homogeneous. This is most true for MobileNetV3, which gives the best performance, followed by ResNet50 and then ResNet101.
The result of the segmentation algorithm is important for periodic checkups of a patient who has been diagnosed with AMD to objectively observe the progress of the disease. Such a method could help to recognize the AMD progress without invasive contrast-enhanced angiography or costly OCT imaging. The detected AMD lesion zone can be further used to automatically compute the lesion area ratio to the healthy retina and the previous scans of the patient, to give a quantitative estimate of the progress of the disease.