Nowadays, the demand for medical image computing is exceptionally high. This growth was mostly driven by the manual development of machine learning models, in particular neural networks. However, due to the constant evolution of domain requirements, manual model development has become insufficient. The present study proposes a heuristic architecture search that can be in an excellent service for the task of medical image classification. We implemented a novel approach called network morphism to the search algorithm. The proposed search method utilizes the enforced hill-climbing algorithm and functional-saving modifications. As a result of computational experiments, the search method found the optimal architecture in 28 GPU hours. The model formed by the found architecture achieved performance of 73.2% in validation accuracy and 84.5% in AUC on the validation dataset that is competitive to the state-of-the-art hand-crafted networks. Moreover, the proposed search method managed to find the architecture that contains four times fewer parameters. Besides, the model requires almost ten times less physical memory, which may indicate the practical usefulness of our method in medical image analysis.
Over the past decade, there has been a dramatic increase in the manual development
of deep convolutional neural network (DCNN) architectures that show significant
results in image processing. The most recognized hand-crafted DCCNs are VGG [
In contrast, practical tasks usually require models that can perform well on various
data, either small or large images, with two or many classes, based on numerous or
few training data. For instance, state-of-the-art architectures [
Automated machine learning (AutoML) and its branch neural architecture search
(NAS) might be an excellent solution to the mentioned-above issues. To date,
AutoML and NAS methods have been actively developed and employed in various
fields. For instance, NAS allows deploying neural networks into classification and
segmentation tasks with little domain expertise and small datasets. The most
recognized NAS approaches employ search strategies based on genetic algorithms [
Despite significant advances in NAS, its impact on the practical tasks in healthcare is yet not clear. There is still much uncertainty about the efficiency of NAS approaches in medical screening. To address this issue, the present study examines the application of automated search methods to find optimal neural architecture medical image classification. 2
Genetic or so-called evolutionary algorithms have significantly advanced and
expanded the use of AutoML. In the field of NAS, genetic algorithms are designed to
construct new sets of neural architectures, called population [
Evolutionary algorithms have become particularly popular due to the use of a vast
search space, which has considerably improved NAS techniques. Even though genetic
algorithms allow achieving decent results in NAS [
Over the past years, researchers have proposed various methods to solve primary
drawbacks of any NAS methods: significant time costs and the considerable weight of
an output network. For example, Veniat et al. [
Other researchers moved in a slightly different direction. In [
Several studies have addressed the efficiency of NAS on different medical datasets.
Gessert et al. [
From the analysis of the literature, we assumed that network morphism could be an excellent solution to reduce the running time of the search method and facilitate the output model. Therefore, in this work, we apply network morphism to a genetic search for Chest X-ray classification. 3
The main goal of the present research is to investigate the efficiency of the network morphism approach in the medical classification task. To achieve the goal, the following tasks are due to be resolved: 1. To consider network morphism and adjust it to a genetic algorithm. 2. To select an appropriate dataset of medical images for the multiclassification task. 3. To construct a CNN as a baseline architecture and investigate its impact on the output architecture. 4. To implement a heuristic algorithm based on network morphism to search for an optimal neural architecture. 5. To evaluate an optimal architecture found automatically with objective metrics and compare it with state-of-the-art manual networks on the medical image dataset. In this paper, we address the issue of searching for the optimal neural network architecture for the target data set D = {Dtran , Dtest , Dval } , where Dtran is a training dataset, Dtest stands for a test dataset, and Dval is a validation dataset. Let us consider the target problem as a two-level multipurpose optimization problem. The function of multiobjective bilevel optimization can be presented in a general form as follows: subject to aopt = min {Lval ( A, w* ) , C ( A)}
A∈A w* ∈ arg min {Ltran ( w, a)} , w∈W (1) where aopt represents the final architecture among optimized ones A from the search space of all possible architectures of A , C ( A) is the function of the complexity of the optimized architecture and other related values, w stands for the weights of the neural network from the weight space of W , Ltran and Lval are loss functions on
Below, we describe the proposed NAS method that was designed and assessed step by step by the conceptual framework from [23]. 4
In this study, we investigate a heuristic architecture search inspired by network
morphism operators proposed in [
Network morphism allows avoiding training of each new network from scratch.
Thereby, researchers can test many models in a short time. In this study, we applied
several modifications to ensure model efficiency. Concatenation operations and add
nodes serve as in [
In this section, we briefly describe a heuristic search that executes the task (1). As a search strategy, we utilized a genetic algorithm called the enforced hill-climbing search [23]. Fig. 2 depicts the scheme of the search.
The algorithm begins with a data D input. Then, a pre-trained original neural network
serves as a baseline architecture for climbing a so-called hill. In response to the
application of functional-saving modifications, the original network produces a certain
number of descendants. These modifications ensure that each descendant performs the
same as its parent network. Moreover, due to the genetic algorithm, the descendants
suit better for training than their parents. Each descendant is trained on a small
number of epochs and then evaluated by a precision measure – the best descendent moves
to the next step, where they form new descendants. The search procedure ends after a
certain number of epochs. The algorithm presents the so-called best architecture aopt ,
which then serves for final long-term training. Fig. 3 demonstrates a heuristic
architecture search based on enforced hill-climbing algorithm inspired by [
j=1,…,nneigh end for # train the final model both on training and validation datasets modelbest := Train(modelbest , epochneigh , lrate ) return modelbest end function The algorithm does not have to choose a new model at each iteration, and can also hold the one from the previous step if others do not improve. Consequently, the current best model at iteration may also be considered a child network. It is assumed that epochneigh should be small, as the algorithm is forced to train numerous networks. Thus, to check the possibility of overfitting, we conducted two experiments with small and large numbers of epochs.
The algorithm presented above is a straightforward implementation of the hillclimbing method. It can be interpreted as a simple genetic algorithm with one organism, population size of nneigh , and without crossover. While climbing, the functionalsaving modifications of network morphism serve as mutations. Besides, the selection part considers only members of the population with the best performance as a parent for the next generation. 5 5.1
To evaluate the proposed heuristic search, we chose the CheXpert benchmark dataset [25]. The whole dataset comprises 224,316 chest radiographs with a size of 320×320 pixels excluded from 65,240 patients. The images are labeled for 14 diseases as negative, positive, or uncertain. In the original dataset, positive and negative cases were marked as ones, dubious images – as zeros. Fig. 4 presents an example of the CheXpert dataset.
For the experiment, we formed a subgroup of five diseases, namely atelectasis, cardiomegaly, consolidation, edema, and pleural effusion. Therefore, the number of classes was set to five. We split the subset into 70% training, 20% testing, and 10% validation images. Furthermore, several data augmentation techniques, such as random flips, translations, and rotations, were applied to the pictures of the training dataset. Here, we describe the baseline architecture, which serves as an original network for further optimization. According to the recent comprehensive overviews [26,27], CNNs are the type of neural architectures that suite the best for the classification of medical images. Therefore, this work is devoted to applying heuristic architecture search only to CNNs. Hence, guided by [28,29], a small baseline convolutional architecture was represented as follows
Input → {16 ⋅ 2i−1 × ConvLi →
MPLi }i=1 → 128 × ConvL → SML , 3 where ConvL stands for convolutional layer, from 16 to 128 filters of size 3×3 and stride 1, ReLU is an activation function, MPL stands for MaxPool layer of size 2×2 and stride 2, and SML represents SoftMax function for the probability distribution of the output result.
The large-scale convolutional architecture was set as (2) (3) Input → {16 ⋅ 2i−1 × ConvLi →
MPLi }i=1 3 → 128 × ConvL →
BN →
ReLU → SML, where BN represents batch normalization. All other parts of architecture (2) denote the corresponding elements in the network (1). 5.3
In this section, we describe the setup of the training parameters. In the hill-climbing algorithm, we set the number of search epochs of 10, the number of organisms in the population of 10, the number of epochs for training each element of 5, the number of mutations of 5. We also limited the possible size of the model. If the sample model became too large, the agent would reject the investigated architecture and choose another one.
For the training procedure, we employed Adam optimization method with the learning rate of 10−3 , the weight decay of 0.5 ⋅10−3 , the momentum of 0.9, and a batch size of 256. According to experimental results in [30], this setup of training parameters can assure excellent model approximation in training. The original network was pre-trained by ten epochs on one fold of the training dataset, while the final architecture was optimized by one hundred runs on each fold from scratch. Overall, the training was performed five times, and the milestone results were averaged.
All experiments were performed in Python v3.6, using the TensorFlow v.1.13 backend [31]. The hardware setup consists of 8 core Ryzen 2700 and a single NVIDIA GeForce GTX1080 GPU with 8 GB memory. The working code is opensourced and available by [32]. 5.4
In this study, we evaluate the output architecture and compare it with other networks by several statistical measures, which are recall (REC), precision (PREC), accuracy (ACC), and area under the curve (AUC). Let us consider the number of real positive (P) and real negative (N) cases in the data. As it is known from the theory of statistics [33], the classification results are distributed as true positive (TP), true negative (TN), false positive (FP) and false negative (FN) cases. Thus, the evaluation metrics used in this study are as follows
REC = PREC =
TP TP + FN
TP TP + FN , , ACC =
TP + TN TP + TN + FP + FN .
For binary classification, AUC is set as
A = 1 FP2 2 FP2 − TN2 −
FP1 × TP2 FP1 − TN1 TP2 − FN2 −
TP1 .
TP1 − FN1 In the case of five classes, AUS is as follows (4) (5) (6) (7) 5 AUC5 = ∑µ (ci ) p (ci ) ,
i=1 where µ (ci ) stands for AUC under the ROC curve of class ci , p (ci ) is the prior probability of class ci .
Also, the different architectures were evaluated and compared by network size, number of parameters, number of training epochs, and training time. 6
In this section, we describe how the initial type of network can influence the results of the heuristic architecture search. Also, we compare the result of our approach with two different numbers of epochs to the set of hand-crafted architectures. 6.1
The conducted experiments revealed that selecting a small original network lead to a lengthy search process. Moreover, small network (2) required numerous mutations, and thus, the time to evolve into an optimal architecture. In contrast, a large-scale architecture (3) could limit the search space with large structures, neglecting smaller ones that might also be suitable for search.
We conducted two representative experiments to investigate the above-mentioned hypothesis. In the first run, we checked the small original network (2), in the second – the large one (3). Fig. 5 shows a comparison of the results of two possible original architectures.
According to Fig. 5, a large-scale network provides higher model accuracy.
In this section, we examine the final architecture optimized by our NAS algorithm.
We employed three hand-crafted convolutional networks commonly used in medical
image tasks. These networks are VGG19 [
We trained the networks on the selected dataset and compared them with the final architecture of our algorithm. Besides, we conducted two separate experiments with different numbers of epochs to investigate the probability of overfitting. All networks were evaluated by metrics (4)–(7). The hyperparameters remained the same. Table 1 presents averaged evaluation results of k-fold validation ( k = 5 ). According to Table 1, the architecture found by our heuristic search demonstrates competitive results to state-of-the-art models in the classification tasks. The NAS architecture comprises various branches such as concatenates, skip nodes, adds. In contrast, hand-crafted architectures usually lack additional layers as it requires numerous experiments. While most state-of-the-art models have a regular structure, that is, they consist of multiple blocks that are repeated in the architecture, the NAS architecture has a structure without a noticeable repeating pattern. This approach allows for creating flexible architectures for different datasets. Table 2 reveals more details of the comparison. The proposed NAS algorithm allowed finding the optimal architecture in 28 GPU hours, while the manual search can require weeks of tedious attempts and experiments. Furthermore, the final model fewer number of parameters with sufficiently high accuracy compared to the state-of-the-art. Thus, the model requires less physical memory and could ensure efficient use in practice. 7
In this study, we proposed a heuristic algorithm of neural architecture search in medical image analysis. To investigate the issue of medical image classification, we employed the CheXpert benchmark dataset and considered a multiclassification task. The core of the search method was the enforced hill-climbing algorithm enhanced with network morphism.
Firstly, we composed and examined two hand-crafted CNNs as baseline architecture. As a result of numerical experiments, the large-scale network turned out to be a better solution. Secondly, we implemented a heuristic search with network morphism modifications in order to find optimal neural architecture. To check if the final network was subjected to overfitting, we conducted two separate experiments with numbers of epochs 21 and 69. As a result, even taking into account the almost three-fold difference between the number of epochs, both statistical indicators and computational costs and weight were approximately equal. This outcome could occur due to either the use of one organism within the algorithm or lack of crossover. The authors aim to investigate this knowledge gap in future work.
We evaluated the final architecture found automatically with an objective statistical metrics and compared it with recognized hand-crafted networks on CheXpert dataset. Our search algorithm managed to find the optimal architecture in total in 28 GPU hours. The optimized architecture in the second experiment achieved the performance of 73.2% in accuracy and 84.5% in AUC on the validation dataset, yielding the competitive results to the state-of-the-art hand-crafted networks. Moreover, the optimized model contained four times fewer parameters and required almost ten times less physical memory compared to other networks. In summary, these results could indicate the practical usefulness of the proposed heuristic search method.
Further work needs to be done to determine the influence of sensitivity and specificity on the proposed heuristic search. A reasonable approach to tackle this issue could be the inclusion of diverse medical datasets of both X-Ray images and computer tomography scans. Further research should also focus on the optimization of hyperparameters within the heuristic architecture search. 22. Kwasigroch, A., Grochowski, M., Mikołajczyk, A.: Neural architecture search for skin lesion classification. IEEE Access. 8, 9061–9071 (2020).
doi:10.1109/ACCESS.2020.2964424 23. Radiuk, P.M., Hrypynska, N.V.: A framework for exploring and modeling neural architecture search methods. Paper presented at the 4th International Conference on Computational Linguistics and Intelligent Systems (COLINS-2020), Lviv, Ukraine, 23–24 April 2020.
CEUR-Workshop Proceedings, vol. 2604, pp. 1060–1074. CEUR-WS.org (2020) 24. Edelkamp, S., Schrödl, S.: Memory-restricted search. In: Edelkamp, S. and Schrödl, S.B.T.-H.S. (eds.) Heuristic Search: Theory and Applications. pp. 227–281. Morgan Kaufmann, San Francisco (2012). doi:10.1016/B978-0-12-372512-7.00006-7 25. Irvin, J., et al.: CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the 33d AAAI Conference on Artificial Intelligence 2019 (AAAI-2019), Honolulu, Hawaii, USA, 27 January – 1 February 2019. vol. 33(1), pp. 590–597. Association for the Advancement of Artificial Intelligence (2019). doi.10.1609/aaai.v33i01.3301590 26. Shen, J., Zhang, C.J.P., Jiang, B., Chen, J., Song, J., Liu, Z., He, Z., Wong, S.Y., Fang, P.H., Ming, W.-K.: Artificial intelligence versus clinicians in disease diagnosis: A systematic review. JMIR Med. Informatics. 7(3), e10010 (2019). doi:10.2196/10010 27. Kim, M., Yan, C., Yang, D., Wang, Q., Ma, J., Wu, G.: Deep learning in biomedical image analysis. In: Feng, D.D. (eds.) Biomedical Information Technology. pp. 239–263. San Diego, Elsevier (2020). doi:10.1016/B978-0-12-816034-3.00008-0 28. Romanuke, V.V.: An attempt of finding an appropriate number of convolutional layers in CNNs based on benchmarks of heterogeneous datasets. Electr. Control Commun. Eng. 14(1), 51–57 (2018). doi:10.2478/ecce-2018-0006 29. Romanuke, V.V.: Smooth non-increasing square spatial extents of filters in convolutional layers of CNNs for image classification problems. Appl. Comput. Syst. 23(1), 52–62 (2018). doi:10.2478/acss-2018-0007 30. Radiuk, P.M.: Impact of training set batch size on the performance of convolutional neural networks for diverse datasets. Inf. Technol. Manag. Sci. 20(1), (2017).
doi:10.1515/itms-2017-0003 31. Abadi, M., et al.: TensorFlow: A system for large-scale machine learning. In: Keeton, K. and Roscoe, T. (eds.) Proceedings of 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI-2016), Savannah, GA, USA, 2–4 November 2016. pp. 265–283. USENIX Association (2016) 32. Heuristic NAS. GitHub, Inc. https://github.com/soolstafir/Heuristic-NAS (2020). Accessed 27 Apr 2020 33. Fawcett, T.: An introduction to ROC analysis. Pattern Recognit. Lett. 27(8), 861–874 (2006). doi:10.1016/j.patrec.2005.10.010