Approach to Automatic Segmentation of Atherosclerotic Plaque in B-images Using Active Contour Algorithm Adapted by Convolutional Neural Network to Echogenicity Index Computation Jiri Blahuta, Tomas Soukup, Petr Sosik Silesian University in Opava, The Department of Computer Science, Opava, Czech Republic Abstract: The presented paper is dedicated to image Principles and methods of neurosonology are described in processing of ultrasound B-images in neurosonology. [2]. Atherosclerotic plaques in in-vitro B-images are analyzed In 2011, we developed a software tool B-MODE As- in this study. The content is divided into two core parts. sist using binary thresholding algorithm to detect hyper- The first one is focused on computing Echo-Index value echogenicity of the substantia nigra. Due to general prop- inside the atherosclerotic plaque as a defined Region of erties of B-images in grayscale, the software can be used Interest. From achieved results is obvious that the Echo- also for the atherosclerotic plaques but in slightly adapted Index is well-reproducible value in general. Totally of 278 form. images were analyzed by two non-experienced observers and by an experienced sonographer to validate the result. 1.1 Input B-images Basic statistical descriptors were calculated to judge the level of agreement. In this part, the ROI were selected For this research, a set of 278 B-images (in-vitro) is pro- manually. cessed. In Fig. 1, an example of the B-image in which the The second part is focused on approach to automatic plaque is displayed, is stated. selection of the ROI. Manual drawing of the border is time-consuming. Our idea is to use active contour al- gorithm (ACM; Active Contour Model) to eliminate the black background from the displayed plaque. Using ACM can be useful way to select ROI automatically. The main issue is to separate the shape of the plaque from neigh- bor structures, especially from the bottom of the tube. To adapt ACM, the principle of the convolutional neural net- work can be used to extract the feature of the shape to se- lect a correct ROI. Thus, CNN can be trained and learnt to adapt number of iterations of the ACM (or another param- eter) based on supervised learning from properly bordered examples. The main disadvantage is time-consuming pro- cess and high performance to be needed to train and learn the CNN. In fact, in this study, there is no real design of CNN but the primary goals are defined to realize in future. Figure 1: Input B-image with atherosclerotic plaque in- 1 Motivation and Input Data vitro In recent few decades, ischemic stroke caused by Our recent research has been based on computing the atherosclerosis is one of the top causes of the mortality echogenicity index (called Echo-Index) to comparison of worldwide. In modern neurology, ultrasound B-imaging is the risk of the plaque using our developed software and a one of the diagnostic tools to detect atherosclerotic plaques visual assessment by an experienced sonographer. in general. The diagnostic ultrasound [1] is a fast, non- invasive examination which is also combined with an- 2 Echogenicity Index in A Free-Hand ROI other modalities, such as CT or MRI. The atherosclerotic plaques are well displayed in B-image but the limitation is The echogenicity index (Echo-Index) is one numeric value to understand their complex structure to find some mark- which could corresponds with the echogenicity grade. The ers to predict severe problems. Increased echogenicity of Echo-Index is computed in closed Region of Interest, in the plaque can be one of the detectable features as well. this case in the atherosclerotic plaque. To obtain the Echo- Copyright ©2020 for this paper by its authors. Use permitted under Index, we use own developed software tool B-MODE As- Creative Commons License Attribution 4.0 International (CC BY 4.0). sist, originally developed for substantia nigra echogenic area computation. Hyperechogenic substantia nigra is a detectable parkinsonic marker in B-image. The algorithm is fully described in our previous publications, e.g. [3] and [6]. The important clinical study based on this software have been published in 2014 [4] and [6]. The basic princi- ple can be described by the following steps: 1. Load an input image (in bitmap or DICOM format) which is converted into 8-bit grayscale depth auto- matically 2. Select a window in which the examined structure is shown, i.e. atherosclerotic plaque 3. Select a Region of Interest (in the case of atheroscle- Figure 2: Six different shapes of the plaque in in-vitro B- rotic plaques, ROI of a free shape); the ROI is a bi- images nary mask 4. Inside the ROI, the area is computed according to Due to the principle of binary thresholding, for lower threshold echogenicity grade, the Echo-Index should be lower and (a) The area is computed as the number of remain- for higher echogenicity the Echo-Index should be higher. ing pixels after binary thresholding algorithm This is an assumption which proceeds from the principle of binary thresholding. Thus, in the case of low echogenic- (b) For each threshold T in the range of 0 to 255, ity, for low T threshold the computed area should be very the number of pixels is computed low and vice versa. In consequence of this principle, the (c) The number of pixels is converted into real mm2 sum for low echogenicity is low and for high echogenic according to displayed scale, i.e. the window ROI the sum is higher. During the future work it should be size in step 2 (the size of 20 × 20 mm is used) confirmed whether this idea is correct or not; especially in comparison with visual assessment. 5. All values of the area for all 256 thresholds are drawn as a "curve" (256 isolated values); see Fig. 2. 2.1 General Reproducibility Assessment of the In the case of analysis of the substantia nigra, this al- Echo-Index gorithm was genuinely useful. The speed of decreasing of The goal of the pilot study [8] was to prove the repro- the area inside the ROI has been observed. For substantia ducibility of the index; total of 284 B-images were ana- nigra, an equal shape of the ROI is used; an elliptical shape lyzed using this software with this following conditions: with area of 50 mm2 . It was sufficient for clinical studies, e.g. [5] to statistical analysis of the echogenicity grade of • 2 independent non-experienced observers measured the substantia nigra. all images two times In the case of atherosclerotic plaques, the main differ- • each observer measured for 2 weeks; for one week ence is the fact that ROI is selected by a free-hand shape the first has been performed (278 images) and for and the area is different for each plaque. In Fig. 2, six dif- following week the second measuring has been per- ferent shapes of the plaque are presented as a closed ROI formed (the same set of 278 images) selected in the window 20 × 20 mm. The idea of the Echo-Index is considered as a number • all images have the same resolution but the algorithm which can describe echogenicity grade inside the plaque; can be used for different resolution inside the free-hand ROI. Let H is the brightness value of The reproducibility of the Echo-Index has been proved a pixel and T to be the threshold then AT is the computed as well-acceptable in general considering inexperience in area for each threshold T in the range of 0 ≤ T ≤ 255. The neurosonology of both of the observers. The Echo-Index sum is computed does not evince significant difference in case of the same 255 image analyzed by each observer; each of them draws ROI AREASUM = ∑ AT (1) slightly in different shape. Table 1 shows an example of T =0 computed Echo-Index for 22 images between 2 observers; and the AREASUM value is divided by 100 there are no significant differences due to similarly drawn ROI manually by each observer. AREASUM Some images from the set were also analyzed by an EchoIndex = (2) 100 experienced sonographer; ROI drawn precisely; and very and this value we called Echo-Index. similar values for Echo-index have been achieved. Table 1: An example of measured Echo-Index between Table 2: Basic statistical analysis to reproducibility assess- two observers for 15 randomly selected images ment. Image ID Echo-Index 1 Echo-Index 2 100027 1229.35 1333.48 variable value(s) 105520 1299.29 1371.01 R11 , var11 2423.65, 160215.98 128207 2036.73 1836.71 R12 , var12 2804.90, 164459.53 131283 1386.88 1385.05 R21 , var21 2215.96, 158765.22 137930 962.85 1303.37 R22 , var22 2855.31, 162005.70 144329 977.41 798.63 max(obs) 281.90 (absolute value) 146994 1180.13 1361.27 mean(obs) 59.93 153036 774.99 847.06 robs 0.947 159463 1464.88 1458.29 max(obs12w ) 312.44 (absolute value) 160177 1351.90 1318.61 max(obs22w ) 279.61 (absolute value) 160507 1330.93 1476.99 mean(obs12w ), mean(obs22w ) 64.72, 70.17 162265 484.23 493.64 robs1 , robs2 0.894, 0.912 163803 450.52 501.71 197476 747.57 825.111 198052 570.93 653.80 Obtained results are summarized in Table 2. Maximum values are stated in absolute value because the difference of the Echo-Index between observers or 2.2 Basic statistical analysis of computed Echo-Index measurement can be also negative. In the case of mea- values surement during the first phase (week), for 220 values from 278, i.e. 79.1 %, the difference under 100 between To evaluate the Echo-Index reproducibility, the following observers has been achieved. In the case of the second statistical descriptors were analyzed from 286 B-images: phase, for 214 values from 278, i.e. 76.9 %, the dif- ference under 100 between observers has been achieved. • range, variance for Echo-Index values measured by Due to achieved results, the Echo-Index can be considered observer1 - first measurement R11 , var11 as well-reproducible value between 2 independent, non- experienced observers and also between 2 measurements • range, variance for Echo-Index values measured by from the same observer. observer1 - second measurement R12 , var12 • range, variance for Echo-Index values measured by 2.3 Echo-Index Related to Real Echogenicity Grade observer2 - first measurement R21 , var21 From Visual Assessment • range, variance for Echo-Index values measured by In general, the basic idea "smaller Echo-Index means observer2 - second measurement R22 , var22 lower echogenicity" which was not confirmed from the point of view of an experienced sonographer, who com- • maximum and mean difference between two ob- pared images with different plaque risk level in which dif- servers max(obs), mean(obs) ferent echogenicity grade is obvious and there is no signif- icant correlation between visual assessment and computed • correlation coefficient between Echo-Index values Echo-Index. from 2 observers robs • maximum and mean difference between mea- 2.4 The idea of decision-making system to risk sured values from observer1 between two weeks assessment based on Echo-Index value max(obs12w ) Although the Echo-Index seems like a reproducible value, • maximum and mean difference between mea- it must be thoroughly examined if the value corresponds sured values from observer2 between two weeks with visual assessment by an experienced sonographer. max(obs22w ) The idea is to create a decision-making expert system us- ing a knowledge base of echogenicity grades determined • correlation coefficient of the Echo-Index values from by experienced sonographer. In the future, the decision- observer1 between 2 weeks robs1 making system can be developed to use as a tool to eval- uate the probability risk of the plaque in accordance with • correlation coefficient of the Echo-Index values from Echo-index value in determined intervals. A draft of the observer2 between 2 weeks robs2 system were presented in [9]. Some real case studies to use ultrasound imaging to judge risk level of the atherosclero- the edge. For example, we tried to segment the plaque sis are summarized in [10]. using threshold level segmentation with 3 different thresh- olds (T = 15, T = 25 and T = 40). You can see in Fig. 4 that higher threshold level has the influence on segmented 3 Active Contour to Detect The Plaque area. This part of the paper is focused on automatic selection of the ROI instead of manual process. A manual selection of the plaque is used up to now. However, each selection of the plaque takes up to 2 minutes, see in Fig. 2 that some shapes are relatively simple but another one are very complex to its exact selection. We need to find a way how to select the plaque from the input B-image (Fig. 3) automatically. The idea is based on removing the black solid background; B-image can be comprised of the background in the major. Put differently, it is desired to detect edges of the plaque to select the Re- gion of Interest in which the Echo-Index is computed. The active contour algorithm is one of the segmentation tech- niques which is used to detect the background and to ex- Figure 4: Three threshold levels to segment the plaque tract the foreground of the image. It is based on the iter- ative process. In many recent studies were demonstrated In the case of T = 40, the plaque is segmented as iso- that active contour segmentation technique is well appli- lated object with no bottom of the tube. To get an opti- cable for medical images against its complex morphologi- mal result, the threshold is important. In the case of active cal structure. One of many studies focused on using active contour, the number of iterations is an important parame- contour segmentation in clinical image processing is avail- ter like the threshold. At the beginning, initial mask is set able in [7]. and increasing of iterations produces better border of the The goal of the active contour is to remove the black object. background and to extract the border of the plaque. See in Fig. 1. As a result, the goal is to execute the steps 1 to 3 3.1 Using Active Contour automatically. The algorithm is based on iterative steps to acquire the edge which separates segmented objects in the In Fig, 4 is demonstrated that after thresholding there are image. many isolated objects with small area. We need to create a ROI mask which is one closed boundary (one segmented object). Using Active Contour could be a useful way to perform it. Initially, ACM with 25 iterations has been performed. See Fig. 5 in which the results are demonstrated for 2 different images. Figure 3: The idea of using segmentation principle to se- lect the plaque border There are two main limitations to discuss: 1. how to detect inner black areas (Fig. 1 right) 2. how to separate the plaque from the bottom of the tube 3. how to select a closed boundary as a mask ROI Figure 5: Using ACM with 25 iteration to detect seg- mented regions The sensitivity is the primary parameter; how sensi- tive the detection is. Thus, what is considered as back- The results can be used in general but as an experimen- ground and what is considered as foreground bordered by tal study to further improvement. Firstly, we need to sepa- rate the plaque from the tube bottom. See Fig. 6 for some examples of the segmented masks. Figure 8: The results of the ACM after click the area and set the threshold In Fig. 9 initial rectangle ROI and segmented area using ACM after only 10 iterations is shown. Figure 6: Example of using ACM with 25 iterations to segment atherosclerotic plaques Surely, we can increase the number of iterations in the algorithm but no one object is get. When higher number of iterations is used, the border is get more precisely but still more than one segment is obtained. 3.2 Area Detection Using Clickable Input Better results of the ACM were achieved with semi- Figure 9: Initial rectangle ROI (left) selected and ACM automatic input. After selection of the window with the result after 10 iterations (right) plaque, click into the plaque to understanding which seg- ment is useful for us and set the threshold level (as H inten- sity level). This level is automatically computed by Otsu 3.3 Brightness Transformation (Contrast algorithm which can be well applicable as a segmentation Enhancement) As A Pre-processing Phase method for medical images, for example in a study from 2012 [11]. The principle of the ACM is to find gradient to B-images have a complex structure with a lot of isolated separate edges and the background. In Fig. 8 a magnified pixels caused by noise and artifacts, see Fig. 7. To faster segment in which the border between plaque is displayed. and more accurate results of ACM, the brightness en- If we can set the threshold H, the sensitivity can be ad- hancement can be recommended as a pre-processing phase justed; which H is the lowest value of the background. at the beginning. The idea is to "clear" image from isolated pixels and enhance the brightness to better finding the con- tour. Let H to be brightness of the pixel. For example, all pixels with H < 20 can be set as H = 0 (as the background) and all pixels with H > 80 are transformed as H + 30. The effect is presented in Fig. 10. Figure 7: Brightness value rapidly changes between the object and the background Figure 10: Brightness transformation to enhance contours to achieve faster and more accurate ACM results In Fig. 8 there are results after for H = 20 and click into the plaque. The results are more accurate in comparison The main benefit of this transformation is to eliminate with using ACM for the whole image. noise in low echogenicity levels but some image artifacts are enhanced too. Although it can be useful pre-processing phase to achieve better ACM results. 4 Adaptation of Active Contour Using A Convolutional Neural Network Due to limitations mentioned above, the algorithm should be adaptable to find an appropriate border of the plaque. Figure 11: An example of the simplified CNN The result of the active contour algorithm depends pri- model with layers to extract segmented mask (source: marily on brightness of the image. In other words, it is https://awesomeopensource.com/project/vuptran/cardiac- necessary to adapt the algorithm to achieve the best sen- segmentation sitivity in accordance with brightness of the input image. Using a neural network is one of the ideas how to improve border finding. In 2018, we published a pilot study with In Fig, 11 there is an example of the process what we an experimental draft of a feed-forward neural network to need. No classification of the object is necessary, only to recognize the shape of the plaque using edge detection op- extract the mask to learn ACM model. erators [9]. The approach using CNN is different, based Design of the CNN is nontrivial, long-term process due on feature recognition from a training set. The principle to complexity of B-images. The goal is to interconnect of the convolution mask is designed to extract some fea- the CNN with ACM algorithm to reliable detection of the tures. Earlier, we also implemented a simple neural net plaque separately from neighbor objects (bottom of the to detection of the ROI of substantia nigra based on train- tube). To training and learning, correct examples of plaque ing and learning of the coordinates of the ROI to put the borders will be used. Interconnection with ACM could elliptical ROI mask [6]. automatically help predict number of iterations to achieve The goal of the CNN is to train and to learn the shape of better segmentation results of the plaque. Design, training, the plaque using active contour algorithm; in other words learning and testing of CNN are time-consuming tasks for to estimation of the active contour properties to find the which we need a large dataset. best contour of the plaque in B-image. In image process- ing, CNN represent an eminently suitable tool for auto- 4.2 Automatic ROI selection Related To Echo-Index matic segmentation using deep learning, not only for med- Accuracy ical images. The idea is inspired from the research [12] in which Although automatic ROI selection is useful and faster, CNN are used to adaptive learning of the active contour there is one meaningful relation between ROI and com- parameters. In this case, the CNN which sets the threshold puted Echo-Index. To Echo-Index computing, currently and the desired segmented area could be designed. Ac- only manual selection of the ROI is used. So, each ROI cording to complex structure of B-images, deep learning was precisely selected, and the difference between ob- methods can be applied with CNN which are designed to servers was minimal. In the case of automatic selection, extract feature maps for segmentation. ROI can be selected inaccurately in comparison with man- ual border drawing so Echo-Index could be different. Ob- serve the example in Fig. 12 between manual and ACM- 4.1 What We Expect from Using CNN based border. In the case of ACM, ROI is selected includ- ing black background inside; it is undesirable. Currently, CNN should work as a supervised learning model. There- manual selection is more accurate but this a starting point fore, we can use correctly drawn borders for training and to develop the ACM boosted by CNN to create an auto- learning process of the CNN. The convolutional layer of matic segmentation of the plaque. the CNN is designed to extract the features using a convo- Due to this fact, the goal is minimize difference between lution kernel operation, e.g. 5 × 5. In Fig. 11 a simplified manual drawing and using ACM with trained CNN to op- model of CNN is illustrated. The convolutional layer is timize the accuracy. Using CNN could predict an optimal based on computing 2-D convolution. Let f (x, y) is an in- number of iterations from training set of correctly selected put image, g(x, y) to be an output image and the ω to be ROI. the convolution kernel, 2-D convolution is computed by g(x, y) = ω × f (x, y) (3) 5 Conclusions, Motivation to Improve and Future Work on 2-D image. There are usually more than one convo- lutional layer in CNN to detect edges, gradient to higher This paper is divided into two main parts. The first one level of segmented objects. is focused on reproducibility of the Echo-Index between References [1] Azar, R.N., Donaldson, C. Ultrasound Imaging (Rad- cases) (1st Edition) Kindle Edition. Thieme, 2014, ASIN: B00SRLKPOU. [2] Laszlo, C., Baracchini, C. Manual of Neurosonology. Cam- bridge University Press; 1st Edition, 2016, ISBN-13: 978- 1107659155. [3] Blahuta, J., Cermak, P., Soukup, T., Vecerek, M. A repro- ducible method to transcranial B-MODE ultrasound images analysis based on echogenicity evaluation in selectable ROI. (2014) International Journal of Biology and Biomedical En- gineering, Vol. 8, pp. 98-106. ISSN: 19984510. [4] Blahuta, J., Soukup, Jelinkova, M., Bartova, P., Cermak, P., Herzig, R., Skoloudik, D.:A new program for highly reproducible automatic evaluation of the substantia nigra from transcranial sonographic images. Biomedical Papers Vol. 158 Issue 4, 621–627 (2014). [5] Skoloudík, D., Jelinkova, M., Blahuta, J., Cermak, P., Figure 12: Manual selection is more accurate with no Soukup, T., Bartova, P., Langova, K., Herzig, R. Transcra- black area inside (bold border) in comparison with ACM- nial Sonography of the Substantia Nigra: Digital Image based computed ROI (light border) Analysis. American Journal of Neuroradiology Dec 2014, 35 (12) 2273-2278; DOI: 10.3174/ajnr.A4049. [6] Blahuta, J., Cermak, P., Soukup, T. How to Detect and An- alyze Atherosclerotic Plaques in B-MODE Ultrasound Im- two non-experienced observers for atherosclerotic plaques ages: A Pilot Study of Reproducibility of Computer Anal- in B-images. In all images, the plaque was selected manu- ysis. Artificial Intelligence: Methodology, Systems, and ally. Applications: 17th International Conference AIMSA 2016, In the second part, the approach to automatic plaque se- Varna, pp.360-363. lection instead of manual drawing the border, is discussed. [7] Blahuta, J., Soukup, T., Cermak, P. An Expert System Using active contour segmentation algorithm with possi- Based on Using Artificial Neural Network and Region- Based Image Processing to Recognition Substantia Nigra bility to improve the accuracy using a convolutional neural and Atherosclerotic Plaques in B-Images: A Prospective network (CNN) could be useful. Study. 14th International Work-Conference on Artificial In general, the Echo-Index can be considered as a well- Neural Networks, IWANN 2017, Cadiz, Spain, June 14-16, reproducible value. From achieved statistical results be- 2017, Proceedings, Part I. Lecture Notes in Computer Sci- tween two non-experienced observers in sonography, there ence 10305, Springer 2017, pp 236-245. is no significant difference. However, Echo-Index is not [8] Hemalatha, R.J., Thamizhvani, T.R., Babu, B., Chan- usable from the clinical point of view because there is drasekaran, R., Josephin Arockia Dhivya, A., Joseph, E.J. satisfactory correlation between Echo-index and visual as- Active Contour Based Segmentation Techniques for Med- sessment of the echogenicity grade which should be an in- ical Image Analysis. Medical and Biological Image Anal- ysis, Robert Koprowski, IntechOpen, DOI: 10.5772/inte- dicator of the risk level in general. So, lower Echo-Index chopen.74576. value does not correspond to lower echogenicity and vice [9] Blahuta, J., Soukup, T., Skacel, J. Pilot Design of a Rule- versa. Based System and an Artificial Neural Network to Risk The second part is dedicated to automatic ROI selection Evaluation of Atherosclerotic Plaques in Long-Range Clini- to eliminate manual drawing the border. Currently, it is cal Research. ICANN 2018, Lecture Notes in Computer Sci- only idea to do it. It seems that using active contour algo- ence book series (LNCS, volume 11140), Springer, 2018, pp. rithm could be useful. In addition, it can be supported by 90-100, ISSN: 978-3-030-01420-9. CNN to smart adaptation of the ACM parameters. How- [10] Steinl, D.C., Kaufmann B.A. Ultrasound Imaging for Risk ever, this is time-consuming task to effective learn the Assessment in Atherosclerosis. Int J Mol Sci. 2015 May; CNN. It is one of the key goals for future work to improve 16(5): 9749–9769. DOI: 10.3390/ijms16059749. automatic detection of the atherosclerotic plaque without [11] Bindu, H., Prasad, K. An Efficient Medical Image Seg- manual drawing of the border. Faster and reliable solution mentation Using Conventional OTSU Method. International Journal of Advanced Science and Technology. 38. using artificial intelligence components is expected. [12] Hoogi, A., Subramaniam, A., Veerapaneni, R., Rubin, L.D. This research is supported by the project IT4Innovations Adaptive Estimation of Active Contour Parameters Using Excellence in Science - LQ1602 and data supported by Convolutional Neural Networks and Texture Analysis. IEEE Ministry of Health of the Czech Republic, grants nr. 16- TRANSACTIONS ON MEDICAL IMAGING, Vol. 36, No. 28628A. 3, March 2017, pp. 781-791.