Brightness Levels in MRI Should Correspond With Echogenicity Grade in Ultrasound B-MODE images: A Pilot Study of Reproducibility Using ROI-based Measurement Between Two Blind Observers Jiří Blahuta, Tomáš Soukup Silesian University in Opava, The Department of Computer Science, Opava, Czech Republic Abstract: In 2011, we developed a software tool to anal- 1.1 Input MR Images ysis of the echogenicity level in ultrasound B-MODE im- In this study, we have three sets of T1 and T2 MR images ages. This software is based on binary thresholding in a (two basic types of MR images) [6] with different image predefined Region of Interest (ROI). resolution to analyze using the same approach as for ul- The goal of this paper is to observe if the echogenic- trasound images; using echogenicity index as a feature to ity grade in B-MODE images corresponds with bright- distinguish different brightness level. In comparison with ness level in MR images using the echogenicity index. ultrasound B-MODE images which we have used in previ- Achieved results obtained by two, non-experienced ob- ous studies, there is no native scale how to select a window servers in radiology, shows the software can be used also 50 × 50 mm so we use the full width of the image, see Fig. for MRI images. The reproducibility of the measurement 1. evinces the high level of agreement. We use three ROI areas for which the exact position in MR image is not important at this moment. Totally of 52 images were analyzed. Achieved results show the error between measurements by two non-experienced observers does not exceed 5 %; calculated based on the range of the measurements and computed average difference for each image set. Thus, the echogenicity index can be considered as reproducible marker; a small shift of the ROI does not evince signifi- cant change. Average range of the index is computed from 28.17 up to 67.95; minimal index value was < 20 and the highest value was 101.2 due to different brightness level in the examined ROI. The range for the same ROI are almost equal, the difference does not exceed 2 %. 1 Motivation and Input Data Our software has been developed for ultrasound B- imaging [1] in neurology to detect hyperechogenicity of the substantia nigra [2], [3] which is probably one of the most common markers of Parkinson’s Disease detectable Figure 1: Input MR image with selected ROI on transcranial (TCS) ultrasound B-MODE images. The principle of the core algorithm based on binary threshold- ing enables to load not only ultrasound images. Thus, MR images could be also analyzed using this software tool. 1.2 Methodology of the Analysis Clinical studies were published since 2014. The core of We analyze three ROI areas with different size and shape, the software was improved, especially new ROI areas for see Fig. 2. For each image, all three ROI areas are placed different diagnoses. in the same position in the image. In other words, each In modern neurology and neurosurgery, MRI is one of image is analyzed thrice using three different ROI in the the most progressive medical imaging for all perioperative same position. phases [5]. MRI and diagnostic ultrasound are commonly The size and shape of the ROI were defined in the past considered as complementary diagnostic modalities; also for B-MODE images. Originally, ROI1 was used for ncl. for diagnosis confirmation. raphe analysis and ROI2 has been defined for substantia Copyright ©2021 for this paper by its authors. Use permitted under nigra area; both in B-MODE images. Square-shaped ROI3 Creative Commons License Attribution 4.0 International (CC BY 4.0). has been used to analyze medial temporal lobe (MTL) in different case; in measurement of the black/white pixel ra- images to judge general reproducibility between two non- tio in the ROI 20 × 20 mm to judge a probability of MTL experienced observers. atrophy as a marker for the dementia [7]. An example of achieved results for a selected image set of 14 images, is stated in Table 1. Table 1: Achieved differences of the echogenicity index between 2 observers ROI1 ROI1 ROI2 ROI2 ROI3 ROI3 diffROI1 diffROI2 diffROI3 86.99 85.92 71.45 72.22 57.55 58.36 1.07 -0.76 -0.81 38.69 33.97 26.12 24.10 22.08 21.14 4.72 2.02 0.94 55.71 62.77 50.43 53.18 41.61 45.33 -7.06 -2,75 -3.72 75.47 81.18 46.94 48.24 38.19 36.12 -5.71 -1.30 2.07 67.19 70.42 55.83 52.06 39.30 41.52 -3.23 3.77 -2.22 71.32 73.93 58.18 60.54 44.98 44.63 -2.61 -2,36 0.35 73.87 73.95 46.92 48.63 47.53 48.08 -0.07 -1,71 -0.55 83.47 84.27 58.22 58.45 52.00 51.06 -0.80 -0.23 0.94 74.30 74.85 47.44 47.33 53.23 54.15 -0.55 0.11 -0.92 79.27 82.01 52.18 53.58 51.89 52.36 -2.74 -1.40 -0.47 74.79 74.67 51.69 49.54 47.16 48.23 0.12 2,15 -1.07 73.57 70.13 45.88 42.32 40.60 40.22 3.44 3.56 0.38 69.43 62.74 45.01 45.98 42.20 41.78 6.69 -0.97 0.42 66.92 58.35 48.78 50.55 45.07 47.17 8.57 -1.77 -2.10 0.13 -0.12 -0.48 From achieved results we can judge the echogenicity index could be well-applicable in general as a feature in MRI analysis. In Table 2 you can see the average differences for each ROI in four image sets. It is closely related to judge level Figure 2: Three different ROI in the same position are used of agreement which is almost perfect. Table 2: Average computed differences of the echogenic- ity index between 2 observers 2 Echogenicity Index Evaluation in MR image set / avg difference ROI1 ROI2 ROI3 Images SET 1 1.00 -1.00 -0.40 SET 2 0.13 -0.12 -0.48 In the case of B-MODE images, the echogenicity index SET 3 0.89 -0.90 0.21 should corresponds with echogenicity grade of the tissue. SET 4 -0.16 0.64 0.52 We can use the same index for MR images, in which the index should corresponds with brightness level of the ex- The data in Table 2 shows that the differences between amined part inside the ROI. The index is one numerical observers are minimal in the case of the same position value computed using our software. of the ROI including a small shift. It seems that small More information about the methodology, see [8]; the ROI position changes which are not recognizable visually, paper focused on atherosclerotic plaques in B-MODE im- have no significant influence on resulted echogenicity in- ages, in which we have defined the index and its purpose. dex. From achieved results, the range of the index is from Simply, the index is computed as one number which can 28.17 up to 38.89; very similar for each image set. Ac- describe visual brightness level (echogenicity grade in US cording to the range and computed average differences be- imaging). Our software is based on computing the area of tween the observers, the difference between observers is remaining pixels after binary thresholding in the ROI. Let smaller than 5 %. we have 256 intensity levels Hi where i = 0, 1, ..., 255, the area is computed for each level. After that, the all com- puted areas are summed and the sum is divided by 100 to 2.1 Echogenicity Grade in US imaging vs MRI obtain the index given by In the case of US imaging, the image can be adjusted dynamically during examination by ultrasound probe set- ∑255 H=0 AH ECHOINDEX = (1) tings; we can increase or decrease the brightness level 100 according to examined tissue density. The echogenicity Due to the principle of binary thresholding, for lower grade displayed on acquired digitized image can be differ- echogenicity grade, the Echo-Index should be lower and ent visually for the same tissue density. Due to this fact we for higher echogenicity the Echo-Index should be higher. need to analyze the image sets with same probe (image) This is an assumption which proceeds from the principle settings to avoid incorrect echogenicity evaluation. See of binary thresholding. We have used the index in MR Fig. 3 in which three TCS B-MODE images with different global brightness level and corresponding histogram pro- files are shown. Figure 3: Three TCS B-MODE images with different brightness global levels and the histogram profile In MRI, the settings of the MR machine are deter- mined by the manufacturer. All image enhancements are set in post-processing phase in digitized MR images, like histogram equalization using different algorithms (LHE, GHE) [9]. In Fig. 5 you can see the example of three MR images which are different by visual assessment but the histogram is very similar. So, MR images should be considered as more stable from the point of brightness set- tings. The brightness settings should not be affected by set- tings during examination but there is other limitation in MR images corresponding with ROI selection in MR slices, see the following chapter. 2.2 A Limitation of the Echogenicity Index Evaluation in MRI Although it seems the echogenicity index is well repro- ducible, there is one important limitation. In Fig. 5, there is the example of using the same ROI size and shape to select a structure (it is not important from medical point of view at this moment). Due to weighted MRI, the exam- ined structure can be smaller, larger, deformed or may not Figure 4: Four MR images in which the histogram is very visible. In the case of ultrasound B-MODE images, like similar the substantia nigra in TCS images, the position and size is determined; only echogenicity grade is different corre- sponding the gain settings, angle, etc. In this case, totally different echogenicity grade can be obtained for the same patient but in different MR im- age. Thus, another ROI types will be defined in the future which should be better adapted for different MR images to examine structures. This limitation is also a barrier for au- tomatic ROI selection discussed in the following chapter. 3 Possibility of Automatic Finding Closed ROI of Examined Area Using Figure 5: An equal ROI type is used for the structure Convolutional Neural Network which is different in MR images In our previous study dedicated to atherosclerotic plaques analysis in B-MODE images, we also have discussed the possibility to automatic learning of the plaque detection using ANN [10] and also a possibility to create a decision- making expert system to evaluate the echogenicity as a risk marker of the plaque [11]. Ultrasound imaging is widely used in atherosclerosis recognition to early diag- nosis [12]. We have presented a draft of back-propagation ANN model to find a closed region of the plaque. In this field, ANN based on deep learning approach are widely used. In general, the ANN could be used to place ROI ac- cording to learning of some structure in MR image like in Fig. 1. However, the most important barrier is the fact that examined structure may vary in weighted MR images in each image due to intensity level [13]. See an example in Fig. 6 how the weighted MR images are different for the same examined patient. Figure 7: The example of active contours for atheroscle- rotic plaques boundary in B-MODE images experience. There is difficult to learn any shape and size of the ROI due to changing weighted MR images. In this field, there is an interesting inspiration how to de- velop an automatic segmentation using deep learning ap- proach in T1 and T2 weighted images [16]. The desired goal is to train the ANN to extract some features of the ex- amined structure to place the ROI to the correct position. To CNN training, the back-propagation algorithm is used; similarly as in linear feed-forward ANN architec- Figure 6: Weighted MR images example ture. Input image is represented as a single vector w × h × d where w and h represent image resolution and d to be Thus, it could be hard to apply an automatic recognition color depth, in this case d = 1 (for RGB channels d = 3). a ROI described by shape or size when is changing in the Each pixel is represented as intensity value in the range of weighted MR imaging. 0 to 255. CNN uses ReLU (Rectified Linear Unit) activa- The principle could be based on iterative learning using tion function instead of sigmoid or hyperbolic tangent in a convolutional neural network (CNN) which uses filter- traditional multi-layer backpropagation networks. In gen- ing to extract some features to recognize the region. CNN eral, CNN has the following layers and functions: are designed to work with grid-structured inputs, like 2D images. There are many advanced techniques using CNN 1. input layer (as a single vector w × h × d) in medical imaging like in [14]. 2. convolutional layer (3 × 3 or 5 × 5 convolutional masks are commonly used) to extract feature map 3.1 From a Boundary to Learn a Feature 3. activation function like ReLU In 2020, we presented an idea to automatic segmentation based on boundary recognition of atherosclerotic plaques 4. pooling (sub-sampling) layer (to reduce dimensional- in B-MODE images [15]. It could be realized using it- ity of feature maps using MaxPooling algorithm) erative boundary recognition based on active contour al- 5. fully-connected layer gorithm boosted with CNN to train corresponding pairs input-output to learn the rules how to obtain the plaque 6. softmax activation function border, see Fig. 7 in which the contours are shown and segmented plaque shapes after 25 iterations. 7. output layer In the case of MRI, there is a different task. There are no exact borders to find the ROI. In Fig. 6, the weighted Convolutional layer with ReLU and pooling layer are MR images are displayed; it could be hard to learn what to designed for feature extraction and the fully-connected consider as a feature. Let to have a structure in MR image layer with softmax function is used to classification. In which is probably located equally based on radiologist’s our case, we need to recognize a structure in MR image which is defined by a radiologist, e.g. in Fig. 5 and/or in Fig. 10. The process is illustrated in Fig. 9. Deep learning paradigm is based on learning rules from inputs and desired outputs. This is main difference from traditional programming when we have inputs, rules and we need to create outputs. Deep learning requires large data amount to efficiency. In comparison with tradi- tional neural networks and learning, deep learning should achieve better accuracy related to increasing data amount [18]. Figure 10: Six MR images with highlighted square-shaped ROI this task, deep learning could be applied to help to extract the features to recognize the structure. The background of the principle of the image convolution algorithm in CNN Figure 8: Estimated deep learning accuracy vs. conven- you can find in [17] and also in [18] which is a compre- tional paradigms hensive guide to deep learning paradigm. In 2021, a pa- per focused on multi-classification of brain tumors in MRI In a critical point, depending on data complexity and its using CNN, including deep performance evaluation, has structure, conventional paradigms could be inefficient due been published [19]. In future, automatic finding of the to overfitting so the learning rate is low or stopped. ROI could be one of the main goals in our long-term re- In MR images, we can use deep learning approach to search. learn the rules to recognize the ROI. In general, deep learn- There are many ways for practical implementation. One ing is focused on training with pairs input-output from of the most known to be Keras, a high-level modular API large datasets, e.g. thousands of images. Thus, when we developed for Python programming language using GPU need to learn a specific structure in MR images, the train- acceleration. More information, code samples (including ing is based on input-output training set to learn rules, i.e. using for CT scans) are available on Keras.io website. features, to find an appropriate structure to place a prede- fined ROI. The idea of deep learning using CNN is illus- trated in Fig. 9. 4 Conclusions and Using Results in Clinical Studies The goal of the paper is to show how to use echogenicity index, computed with ultrasound B-MODE images, in MR images. To this purpose, we have analyzed sets of T1 and T2 MR images. The principle of the analysis is equal as for B-MODE images. The core of the algorithm is based on binary thresholding of the images in grayscale. Within this MR images analysis, the main idea is also applicable in Figure 9: Deep learning idea with CNN MR images; the higher index value should correlate with higher brightness intensity and vice versa. Consider the following six MR images in Fig. 10. Achieved results show the principle of the echogenic- Consider a task to find the highlighted anatomic structure ity index could be applied for B-MODE images and MR (square-shaped ROI). It seems, there is really hard to learn images independently. It seems, the echogenicity index the features of the structure because it is from small to big- is well applicable to observe different brightness in MRI ger area in which the structure is located. equally as in the case of B-MODE images. The obtained To effective training and learning the network, a large differences are not significant, but the software is more set of images is needed to learn how to recognize the struc- sensitive than visual assessment in general. ture from input-output training set. For example, we can Finally, we can recommend using this methodology in learn the edge, the brightness difference, the shape, e.g. future clinical studies focused on the analysis of MRI us- roundness, height/width ratio, etc. and another feature. In ing different ROI shapes and sizes according to examined structure in MR image. In future, we will use a new ROI Scientific GeoConference, Proceedings SGEM, 2020 pp. 341- areas, like a circle-shaped and/or free-hand closed area, 348. DOI: 10.5593/sgem2020/2.1/s07.044. defined by an experienced sonographer. It is related to ex- [9] Senthilkumaran, N., Thimmiaraja, J. Histogram Equaliza- amined structures in MR images, see Fig. 8 as the exam- tion for Image Enhancement Using MRI Brain Images," 2014 ple. World Congress on Computing and Communication Tech- In parallel, we are working on analysis of the echogenic- nologies, IEEE, 2014, pp. 80-83, E-ISBN: 978-1-4799-2877- ity index differences between a light area and a dark area 4, DOI: 10.1109/WCCCT.2014.45. within the same ROI. [10] Blahuta, J., Soukup, T., Čermák, P. An Expert Sys- tem Based on Using Artificial Neural Network and Region- This work was supported by European Union un- Based Image Processing to Recognition Substantia Nigra and der European Structural and Investment Funds Opera- Atherosclerotic Plaques in B-Images: A Prospective Study. tional Programme Research, Development and Educa- 14th International Work-Conference on Artificial Neural Net- tion project "Zvýšení kvality vzdělávání na Slezské uni- works, IWANN 2017, Cadiz, Spain, June 14-16, 2017, Pro- verzitě v Opavě ve vazbě na potřeby Moravskoslezského ceedings, Part I. Lecture Notes in Computer Science 10305, kraje" CZ.02.2.69/0.0/0.0/18-058/0010238 and project Springer 2017, pp. 236-245. CZ.02.2.69/0.0/0.0/18-054/0014696 "Rozvoj VaV kapacit [11] Blahuta, J., Soukup, T., Skacel, J. Pilot Design of a Rule- Slezské univerzity v Opavě, "Rozvoj metod teoretické a Based System and an Artificial Neural Network to Risk Eval- aplikované informatiky" SGS/11/2019 and the image use uation of Atherosclerotic Plaques in Long-Range Clinical Re- from grant No.16-28628A. search. ICANN 2018, Lecture Notes in Computer Science book series (LNCS, volume 11140), Springer, 2018, pp. 90- 100, ISSN: 978-3-030-01420-9. References [12] Steinl, D.C., Kaufmann B.A. Ultrasound Imaging for Risk Assessment in Atherosclerosis. Int J Mol Sci. 2015 May; [1] Blahuta, J., Čermák, P., Soukup, T., Vecerek, M. A repro- 16(5): 9749–9769. DOI: 10.3390/ijms16059749. ducible method to transcranial B-MODE ultrasound images [13] Ito, S., Shirai, W., Hattori, T. Putaminal hyperintensity on analysis based on echogenicity evaluation in selectable ROI. T1-weighted MR imaging in patients with the Parkinson vari- (2014) International Journal of Biology and Biomedical En- ant of multiple system atrophy. AJNR. American Journal of gineering, Vol. 8, pp. 98-106. ISSN: 1998-4510. Neuroradiology, 2009, 30(4), pp. 689–692. DOI: 10.3174/a- [2] Blahuta, J., Soukup, T., Jelínková, M., Bártová, P., Čermák, jnr.A1443. P., Herzig, R., Školoudík, D.:A new program for highly re- [14] Blahuta, J., Soukup, T., Sosík., P. Approach to Automatic producible automatic evaluation of the substantia nigra from Segmentation of Atherosclerotic Plaque in B-MODE images transcranial sonographic images. Biomedical Papers Vol. 158 Using Active Contour Algorithm Adapted by Convolutional Issue 4, pp. 621-627 (2014). Neural Network to Echogenicity Index Computation. ITAT [3] Školoudík, D., Jelinkova, M., Blahuta, J., Čermák, P., Conference 2020, CEUR Workshop Proceedings, 2020, pp. Soukup, T., Bártová, P., Langová, K., Herzig, R. Transcra- 223-229, ISSN: 1613-0073. nial Sonography of the Substantia Nigra: Digital Image Anal- [15] Hoogi, A., Subramaniam, A., Veerapaneni, R., Rubin, L.D. ysis. American Journal of Neuroradiology Dec 2014, 35 (12) Adaptive Estimation of Active Contour Parameters Using 2273-2278; DOI: 10.3174/ajnr.A4049. Convolutional Neural Networks and Texture Analysis. IEEE [4] Azar, R.N., Donaldson, C. Ultrasound Imaging (Rad- TRANSACTIONS ON MEDICAL IMAGING, Vol. 36, No. cases) (1st Edition) Kindle Edition. Thieme, 2014, ASIN: 3, March 2017, pp. 781-791. B00SRLKPOU. [16] Zhang, F., Breger, A., Cho, K. Ning, L., Westin, C., [5] Levin, C.M. Magnetic Resonance Imaging (MRI) in Neu- O’Donnell, L., Pasternak, O. Deep Learning Based Segmenta- rologic Disorders. 2018, Merck Manual, Available online: tion of Brain Tissue from Diffusion MRI. NeuroImage, 2021, https://www.merckmanuals.com/professional/neurologic- 233. 117934. DOI: 10.1016/j.neuroimage.2021.117934. disorders/neurologic-tests-and-procedures/magnetic- [17] O’Shea, K., Nash, R. An Introduction to Convolutional resonance-imaging-mri-in-neurologic-disorders Neural Networks. 2015, ArXiv e-prints. [6] Mohan, G., Subashini, M. MRI based medical image analy- [18] Aggarwal, C.C. Convolutional Neural Networks. In: sis: Survey on brain tumor grade classification. Biomed. Sig- Neural Networks and Deep Learning. Springer, Cham. nal Process. Control., 2018, Vol. 39, pp. 139-161. Online-ISBN:978-3-319-94463-0. DOI:10.1007/978-3-319- [7] Blahuta, J., Soukup, T., Pavlík, L. The Black-White Pixels 94463-0-3. Ratio in Medial Temporal Lobe Brain Structure in Transcra- [19] Irmak, E. Multi-Classification of Brain Tumor MRI Im- nial B-Images as a Measurable Marker of Alzheimer’s Dis- ages Using Deep Convolutional Neural Network with Fully ease Probability: The Reproducibility Overview. International Optimized Framework. Iran J Sci Technol Trans Electr Eng Conference on Software, Telecommunications and Computer (2021). DOI: 10.1007/s40998-021-00426-9. Networks (SoftCOM), 2020, pp. 1-6, DOI: 10.23919/Soft- COM50211.2020.9238214. [8] Blahuta, J., Soukup, T., Pavlík, L. The classification of the progression of atherosclerotic plaques in B-MODE images between computer image analysis using echogenicity index and visual assessment. 20th International Multidisciplinary