There is an increase in medicine data quantity and image resolution requirements due to the modern medicine development, which leads to the necessity of strong computing resources and huge computer memory amount during the appropriate tasks' modeling. Semantic segmentation in medicine implements assessment functions in diagnostics.
Modern medicine needs technological solutions that are going to help doctors to diagnose the
patients [
Previous images processing is a very important stage because it depends on the image quality, accuracy, and efficiency of the results obtained during the medium and high levels of processing. It is important because different nature noises affect the images. If they are not eliminated, the next levels are about to be negatively displayed. These noises (they are divided into additive Gaussian and
Vilhutska)
2022 Copyright for this paper by its authors. impulse, which can be part of the studied object) are formed when the images are transferred on the communication channels. In addition to filtering, images are also improved by adjusting brightness levels. This is necessary in order to select individual micro-objects. There are also modern image enhancement methods, such as histogram alignment, contrast-adaptive alignment with histories, and Multi Scale Retinex.
The aim of the research is to study the effectiveness of parallel algorithms for biomedical image segmentation methods, namely the Vinegar Method, Sobel’s filter, and Canny edge detector.
The research objects are a process of the biomedical semantic segmentation of images using artificial intelligence methods and the efficiency of the segmentation methods’ parallelization on the graphic processors.
The research subjects are methods and machine learning tools for semantic segmentation of medical images, methods, and segmentation algorithms parallelization tools on graphic processors.
Scientific papers related to this topic are divided into two types. The first is the work of finding the best neural network models to solve the problem of semantic classification. The second work is related to the parallelization of various known methods of image segmentation (threshold image calculation methods, area extensions, watershed method, k-means, fuzzy c-means) using CUDA or Open Mp technologies.
There is a brief discussion of the need for CUDA GPU calculations in the analysis of medical
images in this work [
In this work [
In the research [
In the research [
In the research [
In the research [
In the research [12], there were detected that classic U-Net architecture has limiting in several aspects. That is why such modifications were applied: 1) an efficient CNN architecture was designed to replace the encoder and decoder, 2) a residual module was used to replace the connection gap between the encoder and the decoder for improvement based on the latest U-Net model. After these modifications, a new architecture was developed - DC-UNet, as a potential successor to the U-Net architecture. A new efficient CNN architecture was created and DC-UNet was built on this CNN. The model was evaluated on three data sets with complex cases and received a relative improvement in productivity of 2.90%, 1.49% and 11.42%, respectively, compared to the classic UNet.
In the research [13] algorithm to fully automatized model for segmentation pixels’ polyps ResUNet ++ was suggested, which is an updated architecture ResUNet for colonoscopical images’ segmentation. Experimental estimates show that the proposed architecture gives good segmentation results in publicly available datasets. In addition, ResUNet ++ significantly outperforms U-Net and ResUNet, two key state-of-the-art deep learning architectures, achieving high scores with a dice coefficient of 81.33% and a mIoU of 79.27% for the Kvasir-SEG and dice coefficient 79.55% and mIoU 79.62% with CVC-612 dataset.
The disadvantages of all these scientific articles are the limited research; usually scientists are stuck on one technology of parallelization or one of the methods of segmentation, without examining the subject area completely. Also, many of these studies are outdated because they use old hardware or old functionality of paralleling technologies.
In addition to a review of scientific papers, the task is to investigate the existing analogues of information systems. A ready-made review of such information systems has already been found in network resources in [14].
7 software products were selected: 1. The Pulmonary toolkit (Ptk) 2. YaDiv 3. NIH-CIDI Lung Segmentation Tool 4. TurtleSeg 5. MITK 6. ITK-SNAP 7. 3D Slicer
Ptk is a set of programs for the analysis of three-dimensional medical images of the lungs and is intended only for the use of academic and research use. Requires MATLAB and a C ++ compiler for some functional components.
TurtleSeg provides accurate 3D image segmentation based on the 2D Turtlemap algorithm. The software is not open source; however, it was provided as a 32-bit trial within 1 month of requesting the trial.
MITK is a free, open source software system that combines the Insight Toolkit (ITK) and the Visualization Toolkit (VTK) with an application structure that provides multiple sets tools.
ITK-SNAP is a software application that provides semi-automatic segmentation in threedimensional medical images using active contouring methods, as well as manual delimitation and image navigation. This is free open source software.
3D Slicer is a software platform for the analysis and visualization of medical images, as well as for research in the field of managed image therapy. It provides registration and interactive segmentation and is free and open source software.
In this study, Ptk showed the highest quality.
The paper proposes a system that will differ from existing ones by the following criteria: 1. Availability of many segmentation methods; 2. Intelligent semantic segmentation, which is performed using a neural network that can be trained on different data sets - with different image sizes and different amounts of data; 3. Parallelization of algorithms to achieve high processing speed; 4. The system is designed for research, i.e. numerical experiments, and for general use.
The task of this study can be divided into the following stages: 1. Subject area analysis. This stage includes the study of existing scientific papers on this topic; ready solutions review of information systems; biomedical data study sets, their relevance, restrictions imposed on them, the format in which they are stored; determination of the initial data to be issued by the developed information system. 2. Design of an information system that performs several tasks: accepts biomedical images, preprocesses, segments them using the Otsu method, Sobel filter, watershed, k-means, performs semantic segmentation using a trained U-Net model CNN, for the latter provides as a result not only a segmented image but also estimates of the quality of this segmentation on the basis of the reference, compares the parallel methods relative to the consistent (all but semantic segmentation). 3. CUDA technology research, its solution in research on this topic; review of mathematical and software that will be used in the development of information systems; software product testing and analysis of results.
The system developed in the work consists of several components that can be narrowed down to two main ones. The first is segmentation using the Otsu method, the Sobel filter, and the Canney boundary detector. This component, in addition to performing its functional tasks, is also needed to study the effectiveness of the use of CUDA in segmentation tasks [14]. To do this, you need to load the image, read it, perform the appropriate segmentation algorithm, output a segmented image and information about the time spent on serial and parallel algorithms, acceleration. The second component is a semantic segmentation, which is implemented using the UNet architecture. It requires downloading input data, reading images, pre-processing, training, issuing results on the network training quality and saving the model. Once the model is saved, the system can perform semantic segmentation quite accurately, on images from this data set, which previously it could not segment. The user is now able to download the image he wants to segment, and the system will output the segmentation result.
The main idea of the Vinegar method is to find a threshold value that will minimize the variance within the class (variances weighted sum of two classes) (formula (1)).
2 ( ) = 0( ) 02( ) + 1( ) 12( ), (1) where 0, 1 – two classes possibility (object and tint), – class dividing threshold, 02, 12 – classes deviation.
Weights calculation with L-histogram: 0( ) = ∑ =−01 ( ); 1( ) = ∑ =− 1 ( ).
Importantly, minimizing variance within one of the classes is the same as maximizing interclass variance (2).
2( ) = 2 − 2 ( ) = 0( 0 − )2 + 1( 1 − )2 = 0( ) 1( )[ 0( ) − 1( )]2, (2) where 0( ) = ∑ =−01 ⋅ (0), 1( ) = ∑ =− 1 ⋅ (1), = ∑ =−01 ( ).
Generalized algorithm: 1. Calculate the histogram and probability of each intensity level. 2. Set initial approximations 0 and 1. 3. Passing all possible threshold values (this means from 1 to the maximum intensity), update and , calculate 2( ). 4. Maximal values 2( ), recieved in the iteration process and is desired threshold values.
The Sobel’s filter is a method of edge detection that uses a gradient. The Sobel’s filter works by calculating the image intensity gradient at each pixel within the image. It finds the direction of greatest increase from light to dark and the rate of change in this direction. The result shows how sharply or smoothly the image changes in each pixel, and therefore how likely it is that this pixel represents the edge. It also shows how this region is likely to be oriented.
This edge selection algorithm is also a gradient method (uses a first-order derivative) and consists of five steps:
Apply a Gaussian filter.
Find the intensity gradient (Sobel filter).
Apply non-maximum suppression.
Apply a "double threshold".
Trace the edges with hysteresis.
Because edge detection results are easily affected by noise of different nature in the image, it is very important to filter out noise to prevent erroneous detection. (3) shows the equation for the size of the Gaussian filter core (2k + 1) × (2k + 1).
, =
1 2 2 (− ( −( +1))2+( −( +1))2 2 2 ), (3) where is less or equal to 1, and is less or equal to (2 + 1).
It is important to understand that the choice of Gaussian nucleus size will affect the performance of the detector. The larger the size, the lower the noise sensitivity off the detector.
Non-maximum suppression is a technique of thinning the edges which is used to find places with the sharpest changes in intensity. Algorithm of each pixel on the gradient image: 1. Comparison of the "force" of the current pixel edge with the pixel force level in the positive and negative directions of the gradient. 2. If the edge strength level of the current pixel is the highest compared to other pixels with the same direction (for example, a pixel that is directed in the y direction will be compared to the pixel above and below it on the vertical axis), the value will be preserved. Otherwise, this value will be discarded.
The previous three filters have been applied to the image to detect true edges, but there may still be noise or false edges along the edges. The two-threshold filter uses high and low thresholds in the image to identify strengths and weaknesses. If the pixel value is greater than the high threshold, it is marked as a strong edge, if the pixel value is less than the upper threshold but greater than the low threshold, it is marked as a weak edge, and if the pixel value is less than the low threshold, the pixel is completely discarded. This filter is used to eliminate the last residual noise.
In addition to detecting boundaries, all four previous filters have filtered out the noise that was present in the image. Because there may be some edges that have been very heavily filtered, the hysteresis filter makes one last pass over the image and connects the edges that should have been connected. Using the strong boundaries defined by the double threshold filter, the hysteresis filter passes through the image using a 3x3 matrix that connects the weak boundaries adjacent to the strong boundaries. This re-establishes the boundaries that should have been detected, but were rejected by aggressive strong filtration. Upon completion of this filter, the boundary detection was performed properly.
Semantic segmentation is the task of clustering parts of an image that belong to the same class of objects. It can be considered a prediction at the pixel level, because each pixel in the image is classified according to class.
Deep neural networks are one way to solve this problem.
The most important step is to train the network using a back propagation algorithm. This algorithm can be described by the following steps: Step 1. Initialization of all filters and parameters \ scales by random values.
Step 2. The network takes the training image at the input, passes it on all layers and at the output gets the probability for each of the classes.
Step 3. Calculate the total error on the source layers (sum for all classes): ( , ) = ∑( ̂ − )
2 1 parameter values .
Step 4. Use the error back propagation method to calculate the error gradients for all weights in the network and use the gradient descent (or variations) to update all parameters and weights to minimize the original error. For normal gradient descent: +1 = − ( , ) or cut:
= − ⋅ ( ).
Step 5. Repeat steps 2-4 for all the images in training set.
One of the most common architectures of convolutional neural networks for biomedical images semantic segmentation is UNet [16].
As a result, the main functionality of the information system is given in the Table 1.
The input data in the work is a set of images. There are data sets that are widely used for segmentation and are publicly available. These are sets of brain images – BRATS (brain tumor segmentation), ISLES (ischemic stroke lesion segmentation), mTOP (mild traumatic brain injury outcome prediction), MSSEG (multiple sclerosis segmentation), NeoBrainS12 (neonatal brain segmentation), MRBrainS (MR brain image segmentation) ). For the lungs – LIDC-IDRI (lung image database consortium image collection). For the liver – LiTSm (liver tumor segmentation), 3Dircadb (3D image reconstruction for comparison of algorithm database), SLIVER07. There are also prostate image datasets (PROMISE12 (prostate MR image segmentation), ASPS (automated segmentation of prostate structures)) and knees (SKI10 (segmentation of knee image)).
These image sets have many advantages and disadvantages related to accessibility, relevance, software that can handle these sets, etc. The main requirement for the set of biomedical images in this work was the presence of masks – ground segmentation (ground truth), which would be needed to perform one of the work subtasks – semantic segmentation. The ISBI Dataset (from ISIC Challenge 2016) was selected. The structure of these data is divided into three parts: training kit, validation kit and test kit. The training and validation kit includes 900 images of dermoscopic lesions and 900 appropriate masks for them. The test set contains 379 such images and 379 corresponding masks [17].
Table 2 presents the acceleration of the parallel algorithm of the Otsu method according to the number of pixels fed to the input.
It is important to note that acceleration is the ratio of the execution time of a parallel algorithm to the execution time of a sequential algorithm. In the case of the GPU used for research in this paper, the number of CUDA cores is 96.
Table 2 Results and comparative characteristics of serial and parallel Otsu algorithm Acceleration
The results for the Sobel filter are presented in Table 3. The results for the Canney boundary detector are presented in Table 4.
In the research [17] is showed that parallelization efficiency is not a good metric when using parallelization on GPUs, because they have a very high number of cores, which significantly reduces the efficiency. You can increase efficiency by increasing the complexity of the task and the amount of data, but this requires a large amount of memory, which is not available on conventional hardware, to achieve maximum parallelization results. Therefore, in this study, the main metric of parallelization is acceleration.
The Unet neural network is implemented in Python using various libraries, including Pytorch. First of all there is a stage of training. To do this, you need to divide the data set into test, validation and training, reduce the image in each of the sets to the same size, conduct a preliminary installment. Unless otherwise specified from the command line, the default number of training epochs is 5, learning_rate 0.01, the CUDA accelerator [18] is used, and binary cross-entropy is used as a loss function because class 2 (malignant and benign tumors). After each epoch, the model obtained as a result of training is stored in a separate folder, and data on the Sorensen index are displayed. After training the network in the folder with check points, you can choose the model that best fits the Sorensen index, which in fact indicates a loss.
After that, the network is ready for use and with the selected model you can perform semantic segmentation of one or more images. Figure 2 shows an example of performing semantic segmentation using a neural network. Figure 3 shows the reference segmentation for the input image. It is visually noticeable that the network works quite accurately. Thirty numerical experiments were performed on the trained model and the Sorensen index averaged 0.41. Figure 1 shows a graph of changes in the value of segmentation estimation according to each epoch of neural network training. From this we can conclude that as the value of the Sorensen index decreases, the so-called "losses" in neural network training, which means that the accuracy of segmentation increases during training, and losses approach 0. Table 5 presents a comparative characteristic of serial and parallel algorithms Unet neural network training.
Also with the help of PyQt5 [19] the graphical interface of the developed system was implemented.
Accelerations of parallel algorithms are quite high, sometimes more than 10 times. Further research on this topic is to increase the amount of input for parallelization to increase the use of GPU resources. Also, one of the elements of further research should be the comparative characteristics of the program on different hardware, as well as the use of other technologies such as OpenCL, OpenMP, MPI and compare the efficiency and acceleration of ready-made solutions with them.
Image segmentation in medicine is a challenge that needs new and better solutions over time. It is relevant because it helps to identify the basic structures of the human body with the help of images that represent them, which increases the accuracy and speed of diagnosis and reduces the risks to human life and health. The number of biomedical images is constantly growing, the size and resolution of such images is also increasing. This leads to the need for large computing and hardware resources if you use sequential algorithms. This problem can be solved or minimized by using parallel algorithms on GPUs.
The paper develops an information system that performs various tasks of image segmentation in medicine, and obtained positive results: the Sorensen index for semantic segmentation is 0.45103438198566437 (this value can be considered analogous to the loss metric), and the acceleration of algorithms in some cases exceeds 10 times. Further ways to explore this topic may be new datasets, other more sophisticated image segmentation techniques, attempts at using other neural network architectures for semantic segmentation, enhancing functionality, and optimizing existing parallelization solutions.
In the perspective of further research, it is necessary to investigate the impact of improving image quality by methods from [20-22] on the basis of parallel calculations to improve the accuracy of the system.
[12] A. Lou, S. Guan, and M. H. Loew, DC-UNet: rethinking the U-Net architecture with dual channel efficient CNN for medical image segmentation, in Medical Imaging 2021: Image Processing, Online Only, United States, Feb. 2021, p. 98. doi: 10.1117/12.2582338. [13] D. Jha et al., ResUNet++: An Advanced Architecture for Medical Image Segmentation, in 2019 IEEE International Symposium on Multimedia (ISM), San Diego, CA, USA, Dec. 2019, pp. 225–2255. doi: 10.1109/ISM46123.2019.00049. [14] A. Alnaser, B. Gong, and K. Moeller, Evaluation of open-source software for the lung segmentation, Current Directions in Biomedical Engineering, vol. 2, № 1, pp. 515–518, Sep. 2016, doi: 10.1515/cdbme-2016-0114. [15] O. Ronneberger, P. Fischer, and T. Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, vol. 9351, N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, Eds. Cham: Springer International Publishing, 2015, pp. 234–241. doi: 10.1007/978-3-319-24574-4_28. [16] “ISIC Challenge.” https://challenge.isic-archive.com/data (accessed May 10, 2021). [17] R. Shams, P. Sadeghi, R. Kennedy, and R. Hartley, A Survey of Medical Image Registration on Multicore and the GPU, IEEE Signal Process. Mag., Vol. 27, №. 2, pp. 50–60, Mar. 2010, doi: 10.1109/MSP.2009.935387. [18] “CUDA C++ Programming Guide.” http://docs.nvidia.com/cuda/cuda-c-programmingguide/index.html (accessed May 20, 2021). [19] “PyQt5 Reference Guide — PyQt v5.15 Reference Guide.” https://www.riverbankcomputing.com/static/Docs/PyQt5/ (accessed May 20, 2021). [20] I. Izonin, R. Tkachenko, D. Peleshko, T. Rak, D. Batyuk, Learning-based image super-resolution using weight coefficients of synaptic connections, In: Computer science and information technologies: proc. of X intern. scien. and techn. conf., 14–17 Sep. 2015, Lviv, Ukraine., Lviv: Lviv Polytechnic Publishing House, 2015, pp. 25–29. [21] D. Peleshko, T. Rak, M. Peleshko, I. Izonin, D. Batyuk, Two-frames image superresolution based on the aggregate divergence matrix, Data stream Mining & Processing: proc. of the 1st international scien. and techn. conf., 23–27 August 2016, Lviv, Ukraine. – Lviv: Lviv Polytechnic Publishing House, 2016. – P.235–238. [22] I. Oleksiv, H. Lema, V. Kharchuk, T. Lisovych, O. Dluhopolskyi and T. Dluhopolska, Identification of Stakeholders Importance for the Company’s Social Responsibility using the Analytic Hierarchy Process, 10th International Conference on Advanced Computer Information Technologies (ACIT), 2020, pp.573-576.