Most of the current COVID-19 smart diagnostic systems use convolutional neural networks to detect foci and classify the stages of the disease on X-rays of the lungs. The conditions for obtaining such images and their parameters vary considerably. Therefore, the direct application of convolutional neural networks to unprepared lung X-rays is characterized by low accuracy. To solve this problem, the paper describes models and algorithms for preprocessing lung X-rays in relation to cleaning, preliminary detection in areas of interest, format transformation and standardization of image presentation. The use of such algorithms can significantly improve the accuracy of the subsequent application of convolutional neural networks to detect foci and classify the stages of diseases on prepared X-ray images of the patient's lungs. The paper presents the results of theoretical and experimental studies on real data - on the open depersonalized Kaggle dataset. Based on the presented research results, a generalized model and practical recommendations are provided on the use of particular preprocessing algorithms for the effective diagnosis of COVID19 on heterogeneous chest X-rays.
Effective treatment of Covid-19, first of all, requires the development and implementation of prompt and accurate diagnostic tools for this disease. In this regard, currently the most common method of rapid diagnosis of Covid-19 on a global scale is based on the examination of X-rays of the lungs. This technique has many well-known advantages, but there are also important disadvantages. Mainly, this is poor visibility and low contrast of not expressed foci of the disease. This is especially true in the early stages of Covid-19. The analysis of such images is difficult even for an experienced radiologist. Therefore, when analyzing Xray images of the lungs, the time of data analysis can significantly increase, as well as the likelihood of erroneous conclusions and inadequate diagnosis. The aim of the work is to develop mathematical models and methods for preprocessing lung X-ray images to ensure subsequent effective diagnosis of Covid-19 in terms of time and accuracy. The objectives of the article are to develop models and methods for cleaning and improving images, preliminary detection of informative areas, converting images and gray scales to the required format for the effectiveness of Covid-19 diagnostics based on standard intelligence models, mainly based on convolutional neural networks (CNN).
Currently, an objective diagnosis of lung diseases can be made on the basis of analysis of samples of the patient's genetic material. In such a situation, Immuno assay devices, Nucleic acid devices, CE Marking devices, etc. are widely used [1, 2]. These are reliable devices (and tests), but they can take several hours, and sometimes days, before the results of the analysis are received. Therefore, this approach cannot be applied in the systems of rapid diagnostics of Covid-19.
To improve the efficiency of Covid-19 diagnostics, the development of appropriate methods and
services of computer vision [3] and artificial intelligence [
Analysis of an X-ray or CT scan of the chest can be completed in minutes. CT analysis is much more
informative, but significantly more expensive and often not as readily available in low income countries.
In contrast, the diagnosis of lung diseases based on X-ray analysis is much more accessible, especially in
low income countries. However, in the early stages of the disease, the analysis of an X-ray image is much
less informative than the analysis of CT data according to [
In the current situation, the relevant task of developing innovative mathematical and software for an
intelligent Covid-19 diagnostic system [
While papers [
To create a perspective intelligent diagnostic system, at the first stage, it is necessary to develop a
subsystem for preprocessing lung X-ray images, which will perform image cleaning, image enhancement
(primarily gradation correction), detection of informative areas (lungs, disease foci), transformation of input
images and scales brightness to the required format for the effectiveness of subsequent diagnostics based
on standard models of computational intelligence. The main requirements and promising ways to build such
a system from the point of view of using computer vision are described in [21, 22], and from the point of
view of artificial intelligence – in [
The development and implementation of efficient preprocessing algorithms will make it possible to
detect foci of Covid-19 disease on lung X-rays with high accuracy and classify disease stages using standard
computational intelligence models [
In addition to solving the main problem of diagnostics, efficient preprocessing and application of the CNN, mainly in relation to the detection of the informative part of the image, will effectively compress their images for storage and transmission over communication channels in modern information and communication systems [26]. The solution of the task, including a description of the standardization of the application of the CNN, will ensure the transfer of technologies for solving related problems in the areas of image recognition, building promising systems for machine vision and controlling robots (and drones) [27, 28], artificial intelligence, related to ICT [29].
Consider typical input data, preliminary research results, materials and methods proposed for solving the problem. 3.1.
X-ray images of an open depersonalized dataset are considered as initial data for analysis and experiments [3, 30]. All files are in DICOM format. Typical examples of images are shown in Figure 1. For the highest quality visualization of images, a unified exponential-logarithmic model proposed in [31] with the parameter = 1 by default.
As a result of the analysis of the dataset images [30], it can be seen that they are all presented in the DICOM format. However, they can differ significantly from each other:
1) by size (the size / scale of representation of some images may differ by an order of magnitude from the area of other images); 2) some images may be inverted while others are not; 3) some images may have a frame around the perimeter (the dimensions and colors of the frame are not known a priori and may vary over a wide range of values); e) f) Figure 1: Examples of images of the considered dataset (opened and rendered by author's software): а) 1ebbe0481d58 (size 3052х2899; density [0,…, 1604]); b) 7e7d3afebf5d (size 1356х1486; density [0,…, 4095]); c) f5451a98d684 (size 2880х2539; density [0,…, 16383]); d) f8ded2e15154 (size 1387х1140; density [0,…, 4095]); e) bb4b1da810f3 (size 2836х2335; density [0,…, 17708]); f) b98508598396 (size 2832х2319; density [0,…, 4095]); the names of the images are given according to the Kaggle dataset [30]; images 1ebbe0481d58 and bb4b1da810f3 are inverted to a single presentation format 4) some images can be rotated to an arbitrary angle; 5) some images may contain various objects and inscriptions of arbitrary brightness in arbitrary places; 6) images can have a different density range (single-byte / double-byte, moreover, with different significant boundaries of the density range); 7) different images are obtained using different equipment in different conditions.
It is also important to note that the dataset under consideration is a mix of images of the lungs of healthy and sick people, and at different stages of the disease. To understand the situation, we note that pneumonia is associated with the appearance of an area of infiltration in the lungs (Figure 1). The main sign of pneumonia in the pictures is the effect of frosted glass with blurred borders. It may be represented by one large spot, or a group of small spots and some other effects.
For images of the lungs of sick people, the coordinates of rectangular areas of lung damage are given (they are shown by red rectangles). Their number may vary. This markup (we are considering images of the Train category) was made for the purposes of machine learning. Due to the presence of a large number of influencing factors, the use of standard models for the detection of foci of the disease and classification (primarily SNS) to the original images leads to a low quality of the results. Experiments have shown that the classification accuracy does not exceed 0.3, Metrics Average Precision [32].
In such a situation, it’s advisable to preliminarily clean and standardize the image (its brightness / density scale, scale limits) in order to obtain acceptable results for the detection of disease foci and classification. In the framework of this work, first of all, we are trying to calculate the limiting density / brightness of the lungs and normalize the image in such a way as to differentiate the pixel density of the lungs, and turn everything else into black or white. To provide for further processing the area in the image as close as possible to the lung area. By assumption, this will give the best classification result. 3.2.
To understand the essence of the proposed preprocessing methods and algorithms, in addition to the images themselves, we also consider the density histograms (Figure 2) of these images (Figure 1).
Artificial objects in the image (perimeter frames, inscriptions, equipment fragments, etc.) give peak density / brightness, distorting the histogram.
For example, boxes give long histogram tails and pronounced peaks in the start and end frequencies of the histogram. Other objects produce frequency peaks in the interior of the histogram. All this makes it difficult to assess the limits of lung density without artificial objects.
Therefore, the basic set of preprocessing algorithms will be aimed not only at standardizing the image presentation format, but also at solving the problems of eliminating artificial objects and their influence on the image histogram (by eliminating objects / cutting off the non-informative part of the histogram in order to leave only the part that corresponds to the densities lungs). 3.3.
Let's consider the key stages and methods of X-ray image preprocessing, taking into account the above variations in the properties and parameters of images.
For standardization and resistance to negative values, we will eliminate the shift in the density range to the non-negative part in the presence of negative values as follows: – find the minimum density: ; – if: < 0, correct the density of the image like this: + | |.
Now all densities are not negative numbers. The new density matrix is saved. Based on it, we make further transformations.
Some images come in inverted form. Some are offset. For uniformity, we will eliminate inversion and bias as follows: – find the minimum and maximum density: min f, max f; – if the image is inverted (this information is in the tags of the DICOM file), we perform the inverse transformation with a shift of the left value to zero: = | − max f | ; – if the image is not inverted and > 0 shift to zero: = − min f.
The new density matrix is saved. Based on it, we make further transformations.
The input images have significantly different density ranges. This is due to different image standards (within the DICOM standard), the presence of frames and artificial objects. For the effectiveness of the subsequent analysis, it is advisable: – find the boundaries of the informative range [ , … , ], that corresponds to the densities of the lungs; – eliminate the offset relative to zero and normalize this range – bring it to a standard size with standard division and start at zero.
To do this, we will use the following algorithm.
Step 1. Clipping. All pictures have a significant background area and may have borders around the edges. These effects can significantly distort the density histogram. To construct an adequate histogram of an X-ray image (without background and frame effects), we will consider images without a cut-off edge; the width of the clipping band is given in relative form by the threshold Т1, [%].
Step 2. Building a density histogram. A histogram of densities is built in the cut window; for visual analysis, all histograms are compressed to a width of 1024 bars (0 – 1023) by aggregation.
Step 3. Histogram Truncation. All histograms have peak extreme bars. Due to clipping, in most cases they go away. But there are exceptions. For large frames and a number of other difficult situations. For insurance, we set to zero the k extreme left and right non-zero densities.
Step 4. Histogram smoothing. With a low initial width of the density range, the histogram has a striped spectrum. For example, when using a one-byte density storage format. Also, the histogram may have significant frequency beats. This has a negative impact on the robustness of data analysis. In the presence of artificial gray objects (letters, wires, electrodes, …), peak values may appear on the histogram. To minimize the influence of these effects, we will smooth the histogram with an arithmetic mean filter. The linear size of the filter mask according to the experimentally verified rule shown in Figure 3.
Density Range: Filter mask linear size: 8192 and more - 3x1 4096-8192 - 5x1 2048-4096 - 7x1 1024-2048 - 9x1
Step 5. Suppression. To find the true boundaries of the informative range of lung densities, we will set insignificant densities to zero, applying the multiplicative rule in this way. Threshold Т2, [%] is set. The histogram densities are zeroed starting from the minimum value upwards until the sum of the zeroed densities is less than the T2 threshold. Upon completion of this process, all densities that are equal to the maximum zeroed density are zeroed out.
Step 6. Selection of informative range. Find the maximum length range of densities other than zero. Its extreme values are preliminary estimates of the boundaries of the desired informative range [ , … , ]. Values outside this range are set to zero. Finding such a maximum extended range is necessary in order to eliminate large residual fragments of the frame / background, if any. This approach is unstable for a striped histogram. However, after initial histogram aggregation and histogram smoothing (step 4), no banding is observed.
Experimental results. Experiments with model parameters showed the following.
1) The largest contribution to the result (almost 100%) comes from step 1 – Clipping. To stabilize the boundaries of the histogram, it suffices to set Т1 = 23%.
2) As a rule, after clipping, the peak frequencies at the edges go away. But there were a few exceptions. There was a column on the left. To eliminate it, we will assume: k = 1.
3) Smoothing allows you to eliminate the effect of the striped spectrum and stabilize frequencies. Extrema from artificial objects remain, but their contribution is insignificant.
4) Suppression allows you to eliminate residual phenomena. Often – eliminate the histogram tails that are left after removing the frame / background of the image. To perform the suppression, we set Т2 = 0.1%. We slightly increase the threshold for resistance to unexplored negative effects.
The result is a deliberately narrowed informative range [ , … , ]. Now we need to clarify the boundaries of this informative range.
Step 7. Refinement of the boundaries of the informative range. Right border – assessment of the most dense bones: ′ = 0.1, where 0.1 – right border for Т2 = 0.1% (Т1 = 23%, k = 1).
The left side is more difficult. After the image of the frame and background – the left border is an approximation of the minimum brightness of the lung.
For the approximation of the left ′ boundary the best formula
′ = {0 <. 0, = 0.1 − 0.1 ∙ ( 0.1 − 0.1), (1) where 0.1 – left boundary for Т2 = 0.1% (Т1 = 23%, k = 1); 0.1 – right boundary for Т2 = 0.1% (Т1 = 23%, k = 1); 0.1 – coefficient of proportionality for Т2 = 0.1.
Taking into account the errors 0.1 ∈ [0.015, … ,0.107]. Value 0.1 = 0.05 is on average the best. It delivers a minimum of error for most normal intervals. We use it by default.
In most cases, the value 0.0 (left boundary for Т2 = 0.0%) gives an almost perfect approximation of the minimum brightness of the lungs. But sometimes there are significant deviations. For example, for «File: 7e7d3afebf5d» approximation error 29 intervals. In addition, there is a possibility of a very big error if a picture with a large frame comes in which has a long tail on the histogram on the left.
So, we get a new informative density interval [ ′, … , ′].
Step 8. Normalization. Let's assume that: 1) [ ′, … , ′] – boundaries of the informative range, expressed by density, not by interval number; 2) – image pixel density ∈ [ ′, … , ′]; 3) [ , … , ] – boundaries of the new (after normalization) range; 4) ′– new normalized image pixel density ′ ∈ [ , … , ].
Redefine the new densities like this (shift and scale compensation)
, < ′, ′ = { ( − ′) ( − +1) ′ ≤ ≤ ′.
( ′− ′+1) , > ′.
By default, we will use such a new range [ , … , ]: 0 – 2047.
The result is a new matrix of normalized image densities ′ ∈ [ , … , ].
This matrix is saved and further used for all types of image analysis.
Visually, now the same objects in different images have similar brightness. But not identical brightness, since the original images have individual characteristics and distortions. Background and border areas – black or white.
For machine learning, it is advisable that all features change on a scale of 0 – 1. If the CNN does not transform the source data on its own, when transferring data, we will transform the densities as follows: = /( − + 1). As well as the use of CNN at the initial stage of preprocessing, it is advisable to reduce the initial image to the required scale to optimize the complexity. Perhaps one that will subsequently be fed to the input of the SNS during classification. (2)
Section 3 of the work describes typical initial data – X-ray images of the lungs of the considered dataset, including their structure and the most important features. In addition, histograms of the distributions of these images are given and their features that are important for the subsequent analysis of histograms are described. Based on this description, a set of methods for preprocessing images of the considered dataset is proposed for cleaning images, improving their quality and bringing the image format to one standard for further processing using standard SNS for detection and classification purposes.
Based on the proposed methods, has been developed software that allows preprocessing of X-ray images of the dataset [30], as well as visualization of the original images and the results of their processing, including distribution histograms (Figure 4), for control. b) Figure 4: Visualization of the histogram of the 7e7d3afebf5d image of the dataset [30] before (a) and after (b) preprocessing of this image; you can see the effective selection of the informative range of the image with the suppression of peak frequencies corresponding to the black frame
To confirm the effectiveness of the proposed models and methods for preprocessing images, a series of experiments was carried out according to the following scenario.
Dataset images [30] are fed to the input.
Experiment 1. At the first stage, X-ray images of the lungs are not preprocessed, but are immediately fed to the input of CNN EfficientNet for classification. After that, Metrics Average Precision [32] is calculated as described above.
Experiment 2. At the second stage, X-ray images of the lungs are preprocessed according to the algorithm proposed above; produced: – shift compensation; – displacement compensation; – image inversion if necessary; – normalization of the X-ray image brightness distribution; – gradation correction.
After such preliminary processing, the boundaries of the informative range [a,…, b] were determined, which correspond to the densities of the lungs. This eliminated all the negative effects of displacement.
After preprocessing, the images are fed to the input of CNN EfficientNet for classification. After that, Metrics Average Precision is calculated [32].
Next, a comparative analysis of the results of two experiments is carried out, conclusions are drawn and recommendations are formulated regarding the prospects for further work.
In addition to numerical evaluations of efficiency (Average Precision), a visual analysis of X-ray images after their preprocessing is performed, as well as an analysis of their histograms for anomalies.
The results of the experiments are given in Section 5.
A comparative analysis of the effectiveness and prospects for further development are given in Section
The typical application results of applying the proposed algorithm are shown in Figure 5, Figure 6. Figure 5 shows pairwise histograms of the original image and the lung area after preprocessing (after the stage of extracting the informative range). Figure 6 shows X-ray images which brightness/densities have been corrected based on the distribution and boundaries of the new histogram by gradation correction.
After preprocessing, we were able to localize the informative ranges of the lung region of interest with high accuracy and improve the images themselves, make them more saturated and contrast (Figure 6) in comparison with the original images (Figure 1). The application of standard CNN models for classification to such images showed an increase in accuracy by more than 1.5 times. Which made the final accuracy in the range of 0.3 – 0.4. Metrics Average Precision (CNN EfficientNet). c) d) f) Figure 5: Density histograms of the original and processed images after the stage of extracting the informative range (x axis – density, y – frequency; images are shown in Figure 1): а) Histograms 1ebbe0481d58, b) Histograms 7e7d3afebf5d, c) Histograms f5451a98d684, d) Histograms f8ded2e15154, e) Histograms bb4b1da810f3, f) Histograms b98508598396
Algorithms proposed in paragraphs “3.3.1. Histogram Shift Compensatio” and “3.3.2. Image Inversion and Bias Compensation” are systemically important. In contrast to them, the algorithm proposed in paragraph “3.3.3. Image Density Scale Normalization” is a new author's algorithm.
With these preprocessing algorithms, the information content of lung zones is increased by identifying, marking and eliminating the influence of artificial objects (frames, text, etc.) and normalizing the image scale.
As a result of these operations, not only the probability of adequate lung extraction increases, but also the accuracy of predicting the presence of the disease and the stage of its development increases by 2-3 times.
Although we got a noticeable increase in classification accuracy, this result cannot be considered acceptable. It is necessary to further improve the model and algorithms for the preprocessing of an X-ray image of the lungs.
Model 1. The first thing that gives a significant improvement is a two-stage data analysis scheme, when at the first stage we learn to detect the region of the lungs in the image (Figure 7), and after that we use the CNN to search for disease foci and classify images.
This approach was proposed in the solution [33] at the Kaggle competition [3]. It was proposed to take a third-party labeled dataset, train the SNS to detect the lung area on an X-ray, and perform all subsequent detection and classification operations based on the analysis of the lung area only. This approach allows you to focus on the lung area, discarding frames, edges and most of the background. The use of such preprocessing made it possible to significantly increase the accuracy of the subsequent classification to values in the range of 0.5 – 0.6. Metrics Average Precision [34].
Model 2. Additionally, the classification accuracy can be increased to approximately the level of 0.635 (Metrics Average Precision) by using a stacking CNN ensemble [35]. Such an increase in accuracy is mainly due to the use of the individual features of each CNN in the ensemble.
At the stages of detection and classification, the stacking ensemble was composed of the 4 most effective SNS. Types of networks and their performance indicators can be seen in the description of the solution here [33]. Comparative efficient estimates with other CNNs presented here [33].
An analysis of modern solutions in the field of computer vision shows that the accuracy of classification of X-ray images of the lungs, as well as images of other types, directly depends on the efficiency of image preprocessing. In this regard, the best classification accuracy is achieved through the use of a hybrid preprocessing model, which combines classical algorithms with innovative artificial intelligence algorithms that have been significantly developed in the last decade. Proposed hybrid preprocessing model in a generalized form is as follows.
Stage 1. Since the DICOM format is too wide, at the first stage it is advisable to bring the image brightness / density scales to a certain standard, eliminate image inversion, eliminate scale shift, normalize brightness (most often – bring to a scale [0,…1]).
In this regard, the algorithm proposed in the work (for stage 1) is system-forming; it allows you to quickly prepare the input image for subsequent analysis in accordance with the requirements.
Stage 2. Further, it is advisable to detect and exclude from consideration the background, frames and "garbage" objects. Zeroing their brightness, marking their pixels in the picture. Such detection can be done by classical methods, or artificial intelligence methods. For example, objects of the same tone (frames, inscriptions) can be quickly localized using classical segmentation methods, followed by the use of CNN to identify and filter them.
To perform preprocessing, the methods proposed in subsection 3.3, that were applied for histogram shift compensation, image inversion and bias compensation, image density scale normalization.
In comparison with other methods of histogram improvement, proposed image density scale normalization method has significantly improved the efficiency of image analysis in terms of the stability of the selection and classification of lungs and zones of interest within them.
The information content of lung zones is increased (after preprocessing) by identifying, marking and eliminating the influence of artificial objects (frames, text, etc.) and normalizing the image scale. As a result of these operations, not only the probability of adequate lung extraction increases, but also the accuracy of predicting the presence of the disease and the stage of its development increases by 2-3 times.
Stage 3. Further, it is advisable to localize the area of interest – the region of the lungs. This can be effectively done by using detection type CNNs (Yolo, EfficientDet, etc). To improve visibility and enhance the contrast of objects in the area of interest, it may be advisable to repeat step 1 for the area of lung localization.
The solution of this problem based on an ensemble of neural networks is described in [33].
Stage 4. Now you can proceed to the detection of foci of the disease and the classification of the stages of the disease based on the consideration of the area of localization of the lungs.
To improve accuracy, it’s advisable to use verified CNNs ensembles instead of individual CNNs [33]. 8. References [1] COVID-19 In Vitro Diagnostic Devices and Test Methods Database. URL: https://covid-19diagnostics.jrc.ec.europa.eu. [2] S. Tabik et al., "COVIDGR Dataset and COVID-SDNet Methodology for Predicting COVID-19 Based on Chest X-Ray Images," in IEEE Journal of Biomedical and Health Informatics, vol. 24, no. 12, pp. 3595-3605, Dec. 2020, doi: 10.1109/JBHI.2020.3037127. [3] SIIM-FISABIO-RSNA COVID-19, Detection. URL: https://www.kaggle.com/c/siim-covid19detection/overview/description#. [20] K. Smelyakov, A. Chupryna, M. Hvozdiev and D. Sandrkin, "Gradational Correction Models Efficiency Analysis of Low-Light Digital Image," 2019 Open Conference of Electrical, Electronic and Information Sciences (eStream), 2019, pp. 1-6, doi: 10.1109/eStream.2019.8732174. [21] V. Lishchenko, T. Kalimulin, I. Khizhnyak and H. Khudov, "The Method of the organization Coordinated Work for Air Surveillance in MIMO Radar," 2018 International Conference on Information and Telecommunication Technologies and Radio Electronics (UkrMiCo), 2018, pp. 1-4, doi: 10.1109/UkrMiCo43733.2018.9047560. [22] T. Esteves et al., "AUTOMOTIVE: A Case Study on AUTOmatic multiMOdal Drowsiness detecTIon for smart VEhicles," in IEEE Access, vol. 9, pp. 153678-153700, 2021, doi: 10.1109/ACCESS.2021.3128016. [23] O. Vynokurova and D. Peleshko, "Hybrid Multidimensional Deep Convolutional Neural Network for Multimodal Fusion," 2020 IEEE Third International Conference on Data Stream Mining & Processing (DSMP), 2020, pp. 131-135, doi: 10.1109/DSMP47368.2020.9204215. [24] K. Smelyakov, D. Tovchyrechko, I. Ruban, A. Chupryna and O. Ponomarenko, "Local Feature Detectors Performance Analysis on Digital Image," 2019 IEEE International Scientific-Practical Conference Problems of Infocommunications, Science and Technology (PIC S&T), 2019, pp. 644648, doi: 10.1109/PICST47496.2019.9061331. [25] R. Chauhan, K. K. Ghanshala and R. C. Joshi, "Convolutional Neural Network (CNN) for Image Detection and Recognition," 2018 First International Conference on Secure Cyber Computing and Communication (ICSCCC), 2018, pp. 278-282, doi: 10.1109/ICSCCC.2018.8703316. [26] O. Lemeshko, O. Yeremenko and A. M. Hailan, "Two-level method of fast ReRouting in softwaredefined networks," 2017 4th International Scientific-Practical Conference Problems of Infocommunications. Science and Technology (PIC S&T), 2017, pp. 376-379, doi: 10.1109/INFOCOMMST.2017.8246420. [27] Q. Fu, S. Wang, J. Wang, S. Liu and Y. Sun, "A Lightweight Eagle-Eye-Based Vision System for Target Detection and Recognition," in IEEE Sensors Journal, vol. 21, no. 22, pp. 26140-26148, 15 Nov.15, 2021, doi: 10.1109/JSEN.2021.3120922. [28] G. Krivoulya, I. Ilina, V. Tokariev and V. Shcherbak, "Mathematical Model for Finding Probability of Detecting Victims of Man-Made Disasters Using Distributed Computer System with Reconfigurable Structure and Programmable Logic," 2020 IEEE International Conference on Problems of Infocommunications. Science and Technology (PIC S&T), 2020, pp. 573-576, doi: 10.1109/PICST51311.2020.9467976. [29] K. Wang, Y. Wang, S. Zhang, J. Zhang and S. Sun, "Automatic Label Welding Robot System for Bundled Rebars," in IEEE Access, vol. 9, pp. 160072-160084, 2021, doi: 10.1109/ACCESS.2021.3130919. [30] SIIM-FISABIO-RSNA COVID-19, Data. URL: https://www.kaggle.com/c/siim-covid19detection/data. [31] K. Smelyakov, A. Datsenko, V. Skrypka and A. Akhundov, "The Efficiency of Images Reduction Algorithms with Small-Sized and Linear Details," 2019 IEEE International Scientific-Practical Conference Problems of Infocommunications, Science and Technology (PIC S&T), 2019, pp. 745750, doi: 10.1109/PICST47496.2019.9061250. [32] MAP. URL: https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection45c121a31173. [33] 1st place solution. URL: https://www.kaggle.com/c/siim-covid19-detection/discussion/263658. [34] K. Oksuz, B. C. Cam, S. Kalkan and E. Akbas, "One Metric to Measure them All: Localisation Recall Precision (LRP) for Evaluating Visual Detection Tasks," in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2021.3130188. [35] Ensemble methods: bagging, boosting and stacking. URL: https://towardsdatascience.com/ensemblemethods-bagging-boosting-and-stacking-c9214a10a205.