Based on Neural-Symbolic Learning

Abstract

In this paper, we propose an explainable diabetic retinopathy (ExplainDR) classification model based on neural-symbolic learning. To gain explainability, a high-level symbolic representation should be considered in decision making. Specifically, we introduce a human-readable symbolic representation, which follows a taxonomy style of diabetic retinopathy characteristics related to eye health conditions to achieve explainability. We then include human-readable features obtained from the symbolic representation in the disease prediction. Experimental results on a diabetic retinopathy classification dataset show that our proposed ExplainDR method exhibits promising performance when compared to that from state-of-the-art methods applied to the IDRiD dataset, while also providing interpretability and explainability.

-

(A. H. Thiery)

1. Introduction

Diabetic Retinopathy (DR) is one of the leading causes of vision loss afecting the working age population worldwide [ 1 ]. Thanks to the success of deep learning, convolutional neural networks (CNNs) based deep learning approaches have been recently applied to DR classification problems [ 2, 3, 4 ]. Most of the research eforts devoted to CNN-based DR classification methods have been devoted to designing robust neural architectures (e.g. ResNet and DenseNet) for enhanced classification accuracy [ 5, 6 ]. Although deep-learning-based DR classification approaches have demonstrated excellent performance, understanding the decision making process remains a challenge because of the black-box nature of the deep learning methods. This lack of explainability has hindered the adoption of deep-learning based methods in clinical

To gain confidence that developed deep learning methods are robust, researchers have designed and used visually interpretable tools. For instance, gradient-weighted class activation © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

Severity Grade No DR: Mild NPDR*: Moderate NPDR: Severe NPDR:

Description No visible sign of abnormalities Presence of MAs only More than just MAs but less than severe NPDR > 20 intraretinal HEs, Venous beading, Intraretinal microvascular abnormalities, No signs of PDR

Neovascularization, Vitreous/pre-retinal HE *NPDR: Non-Proliferative DR, **PDR: Proliferative DR

PDR**: mapping (Grad-CAM) [ 7 ] is a popular approach that can highlight suspected lesions [ 8 ]. However, most of these post-processing tools generate images (e.g. attention maps) that can only be interpreted by expert ophthalmologists. To circumvent this issue, in [ 9 ], a capsule network [ 10 ] was adopted to encode visually interpretable feature scores for X-ray images in a human-level representation – importantly, these scores can also be interpreted by radiologists. However, this approach could not be considered an explainable model per se since a taxonomy style of characteristics or attributes (such as eyes, a nose, and a mouth that can be used to define a given face) was not involved in the decision making process [ 11 ].

In order to achieve interpretability and completeness for an explainable DR classification model, we have to understand how DR severity is defined clinically. Table 1 summarizes grading criteria for DR severity. Clinically, DR is diagnosed based on the presence of one or more retinal lesions such as Microaneurysms (MA), Hemorrhages (HE), Soft Exudates (SE) and Hard Exudates (EX) [ 12 ]. In addition, Diabetic Macular Edema (DME) severity is also assessed based on the presence of EXs in the macula region [ 13 ].

Neural-symbolic learning [ 15, 16 ] is a suitable approach to produce computational tools for integrated machine learning and reasoning for explainability [ 17 ]. Neural-symbolic learning uses deep neural networks to generate high-level symbolic representation that humans can understand. Logical operations are then conducted using symbolic representation for decision making. In [ 18 ], a neural-symbolic learning system for visual question answering was presented to find an answer from a structural scene representation. This system encoded an image into a compact symbolic representation and then performed symbolic program execution that included logical operations manually designed for reasoning. However, due to the manual design, updating logics for improving performances is not an easy task since the logics should consider relationships between each other.

In this paper, we propose an explainable diabetic retinopathy (ExplainDR) classification model based on neural-symbolic learning to generate a human-readable symbolic representation. The proposed symbolic representation follows a taxonomy style of diabetic retinopathy characteristics consisting of several abnormalities such as MA, HE, SE and EX via a deep neural network for segmentation. The proposed human-readable feature representation is meant to be directly interpretable by both ophthalmologists and patients.

In this paper, we aim to develop a neural-symbolic AI approach to accurately diagnose DR. Such an approach may be of clinical value, because we first generate high-level symbolic representations that are subsequently used to make a DR diagnosis. In other words, our approach has the advantage to remain easily interpretable by both clinicians and patients. The algorithm was tested on the the IDRiD dataset [ 14 ], and heavily relied on lesion segmentation and disease severity gradings.

2. Related Works

2.1. Visually interpretable based deep learning models

In order to improve the black box based deep learning models, visually interpretable tools [ 19, 20, 21, 22 ] for map generation (e.g. attention maps) have been recently applied to DR problems. In [ 23 ], an attention network was used as a clustering method to generate an attention map that can highlight the suspected lesions. This can also be achieved with Class Activation Mapping (CAM) [ 19, 24 ]. In [ 25 ], a regression based activation map was developed to include severity level information in the generated saliency map. In [ 8 ], a Grad-CAM method that can evaluate the suspected lesions without requiring architectural changes or re-training [ 7 ], was adopted to use diferent CNN architectures for improving visual interpretability. In [ 26 ], a combination of lower-layer and higher-layer saliency maps was developed to accurately locate the lesions. Although the above methods could provide clinical value, they still could not explain why and how the developed models could visually localize the suspected lesions.

2.2. Neural-Symbolic Learning

The goal of neural-symbolic learning is to provide a coherent, unifying view for logic and connectionism to contribute to the modelling and understanding of cognition and, thereby, behavior [ 15 ]. The neural-symbolic learning includes a neural network implementation of a logic, a logical characterisation of a neural network system and a hybrid learning system that profitably achieves symbolic and connectionist approaches together to artificial intelligence. Deep neural networks can learn complex input data such as images, audio and text to generate high-level representations, which are useful in decision making [27]. A logic network on top of a deep neural network to learn the relations of those abstractions, can then help systems to be able to explain itself. In [28], DeepProbLog was developed by combining an end-to-end learning with reasoning, where outputs of the neural networks were applied as inputs to ProbLog [29]. In [30], a neural-symbolic framework called logical neural networks (LNN) was designed to simultaneously provide key properties of both neural networks for learning and symbolic logics for knowledge and reasoning. LNN considers every neuron to have a meaning as a component of a formula in a weighted real-valued logic. In LNN, an idea of a 1-to-1 correspondence between neurons and the elements of logical formulae was presented by observing the weights of neurons that can act like AND or OR operations. Based on this idea, LNN has achieved a diferentiable model that can minimize a logical loss function for refutation of logical contradiction.

3. Explainable Diabetic Retinopathy Classification

In this section, we propose an explainable diabetic retinopathy (ExplainDR) classification method based on neural-symbolic learning. Fig. 1 illustrates an overview of the proposed ExplainDR method. Our proposed neural-symbolic learning method includes a U-Net segmentation network [31] used to generate a high-level symbolic representation and a fully connected network (FCN) for learning the generated symbolic representation to predict decision instead of designing logical operations [32]. The U-Net segmentation network extracts a higher-level representation in a symbolic space than the pixel-level representation. To produce the high-level symbolic representation in a taxonomy style, we train the U-Net segmentation network using four segmentation labels, namely Microaneurysms (MA), Hemorrhages (HE), Soft Exudates (SE) and Hard Exudates (EX) which are the main factors to decide about DR severity. Based on the four output images , 1 ≤ ≤ 4 produced by the segmentation network for each eye condition (i.e. = 1 for MA), we extract a human-readable feature vector as symbolic representation using a quantization technique. This feature vector counts the segmented regions in each segmentation output image by setting = { }=1 where is a set of the segmented regions in and is the number of segmented regions within each set . The human-readable feature vector is then given by (1) (2)

= [ | 1| , | 2| , | 3| , | 4| ] ∈ ℕ4, where | | is the number of segmented regions in . The human-readable feature vector is trained using the FCN instead of performing the logical operations to avoid the eforts of designing considerable logic combinations for decision making.

For instance, from an unseen test image, the human-readable feature vector is obtained from each segmented output through the trained segmentation network. Based on the trained FCN, the decision prediction is performed using the human-readable feature vector. We then generate explanation by combining the human-readable feature vector and the predicted decision as follows: • The DR diagnosis of “image 1” is “moderate NPDR” because there are 33 MA, 13 HE, 5 SE and 27 EX regions, respectively. • The DR diagnosis of “image 2” is “mild NPDR” because there are 20 MA, 5 HE, 1 SE and 3

EX regions, respectively.

Additionally, similar to other interpretable DR methods, the visually interpretable images (i.e. segmented images) are also provided. Therefore, we achieve an explainable DR classification method, which includes human-readable symbolic representation in the decision making process, whereas typical AI black-box models only address pixel-level representations.

3.1. Extension of the symbolic representation

Our proposed human-readable feature vector consists of the simple symbolic representation in only four dimensions, and for the four eye conditions (e.g. MA, HE, SE and EX). In order to improve the simple symbolic representation, we propose to consider the sizes of the segmented lesions for better symbolic representation while removing false or noisy segmented lesions. Each segmented lesion is classified into one of three subsets: small, medium or large size as follows: = { ∶ 0 < ≤ 1,∀ } , = { ∶ 1 < ≤ 2,∀ } , = { ∶ 2 < ≤ 3,∀ } , where the size is given by the number of the connected pixels in each segmented lesion . is a threshold that experimentally defines the small, medium and large sizes of the segmented lesions. The improved human-readable feature vector is then given by: = [| 1 | , | 1 | , | 1 | , … , | 4 | , | 4 | , | 4 |] ∈ ℕ12.

We note that the extended human-readable feature vector is still under a taxonomy style that can ofer logical explanation according to the diferent sizes of the segmented lesion within (3) (4) each eye condition.

4. Experiments

4.1. Experimental settings

In our experiment, we use the Indian Diabetic Retinopathy Image Dataset (IDRiD)1 [ 14 ], since this is the one public dataset that provides both lesion segmentation and disease severity gradings. The images have the resolution of 4288 × 2848 pixels. Each image is resized to 1024 × 1024 pixels. In the lesion segmentation dataset, four labels such as Microaneurysms (MA), Hemorrhages (HE), Soft Exudates (SE) and Hard Exudates (EX) are included. In the severity grading dataset, five labels for diabetic retinopathy (DR) such as no DR, mild NPDR, moderate NPDR, severe NPDR and PDR are provided. Additionally, three labels for diabetic macular edema (DME) such as no EX, presence of EX outside and within the macula center are also given. The lesion segmentation dataset has 187 training images and 95 test images in total 282 images. The severity grading dataset provides 413 training images and 103 test images in total 516 images.

In the IDRiD challenge [ 14 ], they provided a specific accuracy evaluation metric counts when the following condition is satisfied: ( == ̂ ) and ( == ̂ ) , (5) where is a true label, and ̂ is a predicted label for DR and DME. In Equation (3), the thresholds are experimentally set at 0 = 10, 1 = 500, 2 = 1000 and 3 = 10000 respectively.

In the segmentation network, the ResNet34 structure [33] is used with the Adam optimizer following a batch size of 2, a learning rate of 0.0001 and a dropout probability of 0.1 for 20 epochs with early stopping. The data augmentation of the segmentation networks includes random lfipping, gamma contrast with a range (0.5, 1.5) and a contrast limited adaptive histogram equalization. The FCN layers are given by: [ 12, 25, 50, 75, 100, 75, 50, 25, 12 ]. In the FCN layers, the Adam optimizer is adopted with a batch size of 16, a learning rate of 0.01 and a dropout probability of 0.1 for 20 epochs with early stopping. The segmentation network is ifrst trained using the lesion segmentation training set. The FCN layers are then trained using the proposed symbolic feature vectors obtained from the severity grading training set via the trained segmentation network. We split the training sets into 80% for training and 20% for validation.

4.2. Results

In order to observe the efect of our proposed ExplainDR method, we conduct an ablation study to evaluate the extension of the human-readable feature vector. We compare the proposed ExplainDR method with the state-of-the-art methods using the IDRiD dataset. Fig. 2 qualitatively shows the segmentation results for eye conditions such as MA, HE, SE and EX using six images from the severity grading dataset. According to small, medium and large (sml) size regions of each eye condition, the 6 extracted human-readable feature vectors for each image are as follows: (1) smlMA: 37, 0, 0, smlHE: 26, 2, 2, smlSE: 0, 0, 0, smlEX: 197, 5, 3 (2) smlMA: 59, 0, 0, smlHE: 54, 4, 4, smlSE: 0, 0, 0, smlEX: 96, 2, 0 (3) smlMA: 25, 0, 0, smlHE: 27, 4, 1, smlSE: 10, 0, 0, smlEX: 90, 0, 1 (4) smlMA: 8, 0, 0, smlHE: 9, 0, 0, smlSE: 0, 0, 0, smlEX: 122, 3, 1 (5) smlMA: 4, 0, 0, smlHE: 6, 0, 0, smlSE: 0, 0, 0, smlEX: 1, 0, 0 (6) smlMA: 1, 0, 0, smlHE: 5, 0, 0, smlSE: 2, 1, 1, smlEX: 0, 0, 0 The explanation along with the predicted decision using the human-readable features are generated as follows: (1) The image 1 is classified as severe NPDR because 37 small MAs, 26 small HEs, 2 medium

HEs, 2 large HEs, 197 small EXs, 5 medium EXs and 3 large EXs are detected. (2) The image 2 is classified as PDR because 59 small MAs, 54 small HEs, 4 medium HEs, 4 large

HEs, 96 small EXs, and 2 medium EXs are detected. (3) ... (4) The image 6 is classified as mild NPDR because 1 small MA, 5 small HEs, 2 small HEs, 1 medium HE, 1 large HE are detected. Here, we note that the above explanations can be compared to the severity grading criteria shown in Table 1 by summing all the numbers of the small, medium and large size regions for each eye condition. This helps non-experts to analyze the generated explanations for self-diagnosis.

To observe the impact of symbolic feature extension of the proposed ExplainDR method, Table 2 shows an ablation study for: 1) ExplainDR with 4 dimensions of the simple symbolic features and 2) ExplainDR with 12 dimensions of the extended symbolic features. The extension of the symbolic representation outperforms that of the simple symbolic representation since the detailed categorization of the simple symbolic representation provides more discriminative symbolic representation than the simple symbolic representation. For performance comparison, Table 3 summarizes accuracy performances of the proposed ExplainDR method and the state-ofthe-art methods [ 14 ]. The proposed method without utilizing any external dataset (e.g. Kaggle2, Messidor3 and DiaretDB14) shows the second-best performance with interpretable images and texts in the leaderboard on the IDRiD dataset. Whereas, the state-of-the-art methods with external datasets provide the accuracy performances without any explanation.

5. Conclusion

This paper presented an explainable diabetic retinopathy (ExplainDR) classification method based on neural-symbolic learning which generated a high-level symbolic representation via a segmentation network. The generated symbolic representation was extended according to the sizes of the segmented lesions to produce more discriminative symbolic representation. The DR severity is predicted by the fully connected network, which was trained using the extended symbolic representation. We qualitatively showed that our proposed symbolic representation was human-readable in the taxonomy style associated with the eye health conditions, as well as an explanation with the reasons of the DR severity. The proposed ExplainDR method showed promising performances to the state-of-the-art methods in terms of classification accuracies on the IDRiD dataset as well as providing interpretability and explainability.

The limitations of our works are: 1) The accuracy and explainablity performances of the proposed ExplainDR are afected by the quality of the segmentation results; 2) Diferent decision outputs can be observed due to the nature of stochastic learning (e.g. FCN); and 3) An enhanced design is needed to adopt other datasets if there is no annotation of the lesion segmentation and the DR classification together. Our future works accordingly are as follows: 1) Study of the 2https://www.kaggle.com/c/diabetic-retinopathy-detection 3https://www.adcis.net/en/third-party/messidor 4http://www2.it.lut.fi/project/imageret/diaretdb1 efect of the segmentation performance; 2) Use of least-squares based methods as a deterministic learning approach instead of the stochastic learning approach; and 3) Study of adoption of other datasets without annotation of the lesion segmentation.

Acknowledgments

The authors would like to thank the anonymous reviewers for their constructive comments to improve this paper. SI would like to thank Prof. Alexandre H. Thiery for his numerous supports. AHT acknowledges support from the Singapore Ministry of Education Tier 1 grant (R155-000-228-114). Artificial Intelligence Organization, 2020, pp. 1–1. [27] A. Garcez, M. Gori, L. Lamb, L. Serafini, M. Spranger, S. Tran, Neural-symbolic computing: An efective methodology for principled integration of machine learning and reasoning, Journal of Applied Logics 6 (2019) 611–632. [28] R. Manhaeve, S. Dumancic, A. Kimmig, T. Demeester, L. De Raedt, Deepproblog: Neural probabilistic logic programming, Advances in Neural Information Processing Systems 31 (2018) 3749–3759. [29] L. De Raedt, A. Kimmig, H. Toivonen, Problog: A probabilistic prolog and its application in link discovery., in: IJCAI, volume 7, Hyderabad, 2007, pp. 2462–2467. [30] R. Riegel, A. Gray, F. Luus, N. Khan, N. Makondo, I. Y. Akhalwaya, H. Qian, R. Fagin, F. Barahona, U. Sharma, et al., Logical neural networks, arXiv preprint arXiv:2006.13155 (2020). [31] O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical image computing and computerassisted intervention, Springer, 2015, pp. 234–241. [32] G. G. Towell, J. W. Shavlik, Knowledge-based artificial neural networks, Artificial intelligence 70 (1994) 119–165. [33] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.

References

[1] S. Garg , R. M. Davis , Diabetic retinopathy screening update , Clinical diabetes 27 ( 2009 ) 140 - 145 .

[2] U. Schmidt-Erfurth , A. Sadeghipour , B. S. Gerendas , S. M. Waldstein , H. Bogunović , Artificial intelligence in retina, Progress in retinal and eye research 67 ( 2018 ) 1 - 29 .

[3] D. S. W. Ting , L. R. Pasquale , L. Peng , J. P. Campbell , A. Y. Lee , R. Raman , G. S. W. Tan , L. Schmetterer , P. A. Keane , T. Y. Wong , Artificial intelligence and deep learning in ophthalmology , British Journal of Ophthalmology 103 ( 2019 ) 167 - 175 .

[4] D. S. Ting , L. Peng , A. V. Varadarajan , P. A. Keane , P. M. Burlina , M. F. Chiang , L. Schmetterer , L. R. Pasquale , N. M. Bressler , D. R. Webster , et al., Deep learning in ophthalmology: the technical and clinical considerations , Progress in retinal and eye research 72 ( 2019 ) 100759 .

[5] H. Pratt , F. Coenen , D. M. Broadbent , S. P. Harding , Y. Zheng , Convolutional neural networks for diabetic retinopathy , Procedia computer science 90 ( 2016 ) 200 - 205 .

[6] Y. Yang , T. Li , W. Li , H. Wu , W. Fan , W. Zhang, Lesion detection and grading of diabetic retinopathy via two-stages deep convolutional neural networks , in: International conference on medical image computing and computer-assisted intervention , Springer, 2017 , pp. 533 - 540 .

[7] R. R. Selvaraju , M. Cogswell , A. Das , R. Vedantam , D. Parikh , D. Batra , Grad-cam: Visual explanations from deep networks via gradient-based localization , in: Proceedings of the IEEE international conference on computer vision , 2017 , pp. 618 - 626 .

[8] M. Chetoui , M. A. Akhloufi , Explainable diabetic retinopathy using eficientnet , in: 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) , IEEE, 2020 , pp. 1966 - 1969 .

[9] R. LaLonde , D. Torigian , U. Bagci, Encoding visual attributes in capsules for explainable medical diagnoses , in: International Conference on Medical Image Computing and Computer-Assisted Intervention , Springer, 2020 , pp. 294 - 304 .

[10] S. Sabour , N. Frosst , G. E. Hinton , Dynamic routing between capsules , in: Proceedings of the 31st International Conference on Neural Information Processing Systems , 2017 , pp. 3859 - 3869 .

[11] D. Gunning , Explainable artificial intelligence (xai) , Defense Advanced Research Projects Agency (DARPA) , nd Web 2 ( 2017 ).

[12] J. W. Yau , S. L. Rogers , R. Kawasaki , E. L. Lamoureux , J. W. Kowalski , T. Bek , S.-J. Chen , J. M. Dekker , A. Fletcher , J. Grauslund , et al., Global prevalence and major risk factors of diabetic retinopathy , Diabetes care 35 ( 2012 ) 556 - 564 .

[13] E. Decencière , X. Zhang , G. Cazuguel, B. Lay , B. Cochener , C. Trone , P. Gain , R. Ordonez , P. Massin , A. Erginay , et al., Feedback on a publicly distributed image database: the messidor database , Image Analysis & Stereology 33 ( 2014 ) 231 - 234 .

[14] P. Porwal , S. Pachade , M. Kokare , G. Deshmukh, J. Son , W. Bae , L. Liu, J. Wang , X. Liu , L. Gao , et al., Idrid: Diabetic retinopathy-segmentation and grading challenge , Medical image analysis 59 ( 2020 ) 101561 .

[15] A. Garcez , T. Besold , L. Raedt , P. Foldiak , P. Hitzler , T. Icard , K. Kuhnberger , L. Lamb , R. Miikkulainen , D. Silver , Neural-symbolic learning and reasoning: Contributions and challenges , in: AAAI Spring Symposium Series, 2015 , pp. 23 - 03 .

[16] A. d. Garcez , L. C. Lamb , Neurosymbolic ai: the 3rd wave , arXiv preprint arXiv: 2012 . 05876 ( 2020 ).

[17] T. R. Besold , A. d. Garcez, S. Bader , H. Bowman , P. Domingos , P. Hitzler , K.-U. Kuhnberger, L. C. Lamb , D. Lowd , P. M. V. Lima , et al., Neural-symbolic learning and reasoning: A survey and interpretation , arXiv preprint arXiv:1711.03902 ( 2017 ).

[18] K. Yi , J. Wu , C. Gan , A. Torralba , P. Kohli , J. B. Tenenbaum , Neural-symbolic vqa: disentangling reasoning from vision and language understanding , in: Proceedings of the 32nd International Conference on Neural Information Processing Systems , 2018 , pp. 1039 - 1050 .

[19] B. Zhou , A. Khosla , A. Lapedriza , A. Oliva , A. Torralba , Learning deep features for discriminative localization , in: Proceedings of the IEEE conference on computer vision and pattern recognition , 2016 , pp. 2921 - 2929 .

[20] S. M. Lundberg , S.-I. Lee , A unified approach to interpreting model predictions , in: Proceedings of the 31st international conference on neural information processing systems , 2017 , pp. 4768 - 4777 .

[21] M. Sundararajan , A. Taly , Q. Yan , Axiomatic attribution for deep networks , in: International Conference on Machine Learning, PMLR , 2017 , pp. 3319 - 3328 .

[22] D. Smilkov , N. Thorat , B. Kim , F. Viégas , M. Wattenberg , Smoothgrad: removing noise by adding noise , arXiv preprint arXiv:1706.03825 ( 2017 ).

[23] Z. Wang , Y. Yin , J. Shi , W. Fang , H. Li , X. Wang , Zoom-in-net: Deep mining lesions for diabetic retinopathy detection , in: International Conference on Medical Image Computing and Computer-Assisted Intervention , Springer, 2017 , pp. 267 - 275 .

[24] H. Jiang , K. Yang , M. Gao , D. Zhang , H. Ma, W. Qian, An interpretable ensemble deep learning model for diabetic retinopathy disease classification , in: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) , IEEE, 2019 , pp. 2045 - 2048 .

[25] Z. Wang , J. Yang , Diabetic retinopathy detection via deep convolutional networks for discriminative localization and visual explanation , arXiv preprint arXiv:1703.10757 ( 2017 ).

[26] C. Lin , J. Zhu , C. Shen , P. Hu , Q. Wang , Ellg: Explainable lesion learning and generation for diabetic retinopathy detection , in: International Joint Conferences on Artificial Intelligence Workshop on Disease Computational Modeling , International Joint Conferences on