Role of Hyperparameters in Deep Active Learning

Abstract

Despite considerable research on pool-based deep active learning (DAL), deploying DAL into real applications still has several challenges. A frequently neglected aspect is the choice of training hyperparameters (HPs), such as the learning rate. Since these HPs determine how the deep neural network learns in each cycle, they must be chosen carefully. In this article, we analyze the role of HPs in DAL. We find that optimizing HPs reduces the performance gap between DAL strategies. Conversely, we highlight challenges in finding optimal HPs when using datasets selected via DAL strategies.

1. Introduction

CEUR 0:8 y0:6 c a r u c c0:5 A 0:4 y0:6 c a r u c c0:5 A 0:4 0:3 strategy becomes apparent. In DAL, the DNN’s predictions are typically used to select instances for annotations. Suboptimal HPs could result in less accurate model predictions, negatively afecting DAL strategies. Learning curves show a greater variance than random when using a DAL strategy. Notably, two specific HP settings resulted in a significant decrease in performance. This experiment suggests the need for carefully selecting HPs when deploying DAL strategies.

We present new studies regarding HPs’ influence on DAL with the following contributions: • We briefly review influential articles in the field of DAL, especially focusing on how these handle training HP in their experimental design. • We present experiments analyzing training HPs’ influence on DAL by evaluating the reusability of labeled pools acquired by DAL strategies for DNNs with optimized HPs. • We provide a roadmap to guide future research on choosing training HPs in DAL.

2. Related Work

Foundational DAL Research: Most highly-cited DAL research underemphasizes HP choice, despite the presence of a randomly selected validation dataset in their experiments. For example, although dropout [ 11 ] is an essential component in [ 12 ], they only optimize weight decay in each DAL cycle. In contrast, [ 13 ], [ 14 ], [ 15 ], and [ 16 ] fix training HPs and only evaluate validation accuracy to select the best-performing DNN during training. While [ 17 ] fix training HPs, they employ a grid search to optimize HPs specific to their strategy. The authors in [ 3 ] argue that DAL should simply work given fixed HPs. However, their results show significant accuracy discrepancies compared to the literature. For example, in their CIFAR-10 experiment with 10k annotations and a ResNet-18, the average test accuracy is over 50% worse than in [ 18 ].

Contemporary DAL Research: Recent research [ 4, 5 ] still primarily fixes training HP at the beginning of DAL. The authors of [ 8 ] highlight that this hinders comparability across various studies and DAL strategies. Furthermore, they empirically demonstrate that HP optimization during DAL reduces accuracy improvements over a random instance selection. Accordingly, they recommend HP optimization during DAL to foster comparability. Other studies, such as [ 19 ], argue that HP optimization per cycle requires much computational efort and does not afect the DAL strategies’ ranking. Thus, they suggest fixing HPs at the start of DAL, providing recommendations for specific HP settings.

3. Experiments

We examine image classification problems using the ResNet-18 [ 1 ] as DNN and conduct experiments on CIFAR-10 [ 10 ], which contains a predefined training and test split. Throughout this article, we employ the training split for DAL. We consider a DAL scenario with an unlabeled pool and a labeled pool ℒ. We start with 100 randomly sampled labeled instances. Afterward, each cycle consists of DNN training followed by selecting 100 instances from based on a DAL strategy. Selected instances are removed from and added to ℒ with their respective labels. The cycle is repeated until a budget of 4000 instances is reached. In addition to random instance selection, we investigate three selection strategies: i) entropy, which is uncertainty-based, ii) core-sets [ 14 ], which is representation-based, and iii) BADGE [ 3 ], which is a hybrid of both. All results are averaged over three repetitions.

To assess the efect of training HPs (here: learning rate and weight decay) on DAL’s selection, we examine the labeled pools collected by DAL strategies that use diferent HP settings. Specifically, for each strategy, we fix four unique HP settings at the beginning of DAL, each leading to a distinct labeled pool. We focus on labeled pools in two stages of the DAL process: in the middle (2k instances at 50%) and at the end (4k instances at 100%). These are then employed to evaluate if varying HP settings substantially influence the performance of strategies, e.g., if a poor-performing strategy due to bad HPs would still underperform with good HPs. To this end, once a labeled pool is collected via DAL, we subsequently optimize HPs with Bayesian optimization (BO) [ 20 ]. To mimic a realistic scenario, we randomly sample a validation split of 10% of the labeled pool. As the sizes of 2k and 4k are relatively small, we use three validation splits for each HP. For a more detailed experimental setup and additional results on CIFAR-100, we refer to our implementation at https://github.com/dhuseljic/dal-toolbox.

Table 1 details the experimental results. Focusing on DAL accuracies first, we see inconsistent improvements of DAL strategies over a random selection across diferent HP settings . Especially HP 3 & 4 demonstrate considerable performance drops of strategies to random selection, suggesting that DAL strategies may even fail with a poor HP choice. Considering the ranking of the DAL accuracies across HPs, we observe that diferent HPs can lead to varying rankings . For example, DAL accuracies of Badge are best when considering HP 1 & 2, while random is best for the other HPs. Once we consider BO accuracies, we notice that a poor HP choice during DAL does not necessarily lead to a worse selection, indicated by decreasing performance gaps to random. Notably, Badge performs robustly, achieving the best BO accuracy in 7 out of 8 cases. Comparing the ranking of DAL and BO accuracy pairs for fixed HP settings, we see that optimizing HPs after DAL impacts strategies’ ranking . For example, when considering HP 4, Badge becomes the best performing DAL strategy after BO, despite not being the best during DAL. This also suggests that a suboptimal HP choice at the beginning of DAL does not necessarily lead to a bad instance selection of DAL. Interestingly, comparing DAL and BO accuracies of HP 1 for 4k instances, we see no accuracy improvement of BO over DAL, which we believe is because a well-working HP leads to a strong sampling bias in the labeled pool. For example, the labeled pool of entropy sampling consists of uncertain instances which do not match the true data distribution leading to a validation dataset unsuitable for HP optimization.

4. Takeaways and Future Research

We provided an empirical study examining the influence of HPs on the instance selection of diferent DAL strategies. Our main takeaways are: i) Varying HP settings at the beginning of DAL can lead to inconsistent performance improvements, strategy rankings, and even failing DAL strategies. However, suboptimal HPs do not necessarily lead to a worse instance selection. ii) Choosing proper HPs at the beginning of DAL does not guarantee suitable labeled pools for real applications. The resulting datasets might not represent the data distribution due to sampling bias, making a representative validation split dificult.

As datasets persist and research continuously develops new DNN architectures, the reusability of labeled pools collected by DAL strategies needs a stronger focus. Specifically, reusing these for new DNN architectures with possibly diferent HPs is crucial for efective integration into practical applications [ 21 ]. Therefore, it is essential that future research addresses: • Research perspective: HP optimization in the context of DAL requires deeper investigations. Our study suggests that HP optimization at the end of DAL reduces the performance improvement of strategies over random instance selection, corroborating the findings of [ 8 ]. Research should be extended toward more alternatives (e.g., optimize HP after regular intervals within DAL) with various architectures and strategies. • Application perspective: There should be a stronger focus on how to proceed after a dataset was acquired via DAL, especially due to requiring a validation dataset that is suitable for HP optimization. This aspect is critical for enabling DAL’s successful deployment into real applications.

To this end, it might be necessary to design DAL strategies that select not only informative instances but also representative ones, which can be used for efective validation. Consequently, this would also minimize the validation dataset’s annotation costs.

References

[1] K. He , X. Zhang , S. Ren, J. Sun , Deep Residual Learning for Image Recognition, in: Computer Vision and Pattern Recognition , IEEE , Las Vegas , NV , USA, 2016 , pp. 770 - 778 .

[2] B. Settles , Active Learning Literature Survey , Technical Report , University of Wisconsin, Department of Computer Science, 2009 .

[3] J. T. Ash , C. Zhang , A. Krishnamurthy , J. Langford , A. Agarwal , Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds , in: International Conference on Learning Representations, 2020 .

[4] G. Hacohen , A. Dekel , D. Weinshall , Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets , in: International Conference on Machine Learning, PMLR, 2022 , pp. 8175 - 8195 .

[5] O. Yehuda , A. Dekel , G. Hacohen , D. Weinshall , Active Learning Through a Covering Lens , in: Advances in Neural Information Processing Systems , 2022 .

[6] S. Mittal , M. Tatarchenko , O. Cicek , T. Brox, Parting with illusions about deep active learning , arXiv preprint arXiv: 1912 . 05361 ( 2019 ).

[7] M. Herde , D. Huseljic , B. Sick , A. Calma , A survey on cost types, interaction schemes, and annotator performance models in selection algorithms for active learning in classification , IEEE Access 9 ( 2021 ) 166970 - 166989 .

[8] P. Munjal , N. Hayat , M. Hayat , J. Sourati , S. Khan , Towards robust and reproducible active learning using neural networks , in: Computer Vision and Pattern Recognition , 2022 , pp. 223 - 232 .

[9] Y. Li , M. Chen , Y. Liu , D. He , Q. Xu , An Empirical Study on the Eficacy of Deep Active Learning for Image Classification , arXiv preprint arXiv:2212.03088 ( 2022 ).

[10] A. Krizhevsky , Learning Multiple Layers of Features from Tiny Images , Master's thesis , University of Toronto, 2009 .

[11] N. Srivastava , G. Hinton , A. Krizhevsky , I. Sutskever , R. Salakhutdinov , Dropout: A Simple Way to Prevent Neural Networks from Overfitting , Journal of Machine Learning Research 15 ( 2014 ) 1929 - 1958 .

[12] Y. Gal , R. Islam , Z. Ghahramani , Deep bayesian active learning with image data , in: International Conference on Machine Learning, PMLR , 2017 , pp. 1183 - 1192 .

[13] W. H. Beluch , T. Genewein , A. Nurnberger , J. M. Kohler , The Power of Ensembles for Active Learning in Image Classification, in: Computer Vision and Pattern Recognition , IEEE, 2018 , pp. 9368 - 9377 .

[14] O. Sener , S. Savarese , Active Learning for Convolutional Neural Networks: A Core-Set Approach , in: International Conference on Learning Representations, 2018 .

[15] D. Gissin , S. Shalev-Shwartz , Discriminative active learning , arXiv preprint arXiv: 1907 . 06347 ( 2019 ).

[16] A. Kirsch , J. Van Amersfoort , Y. Gal , Batchbald: Eficient and diverse batch acquisition for deep bayesian active learning , in: Advances in Neural Information Processing Systems , 2019 .

[17] S. Sinha , S. Ebrahimi , T. Darrell, Variational Adversarial Active Learning , in: International Conference on Computer Vision , 2019 , pp. 5972 - 5981 .

[18] A. Lang , C. Mayer , R. Timofte , Best Practices in Pool-based Active Learning for Image Classification , OpenReview, 2022 . URL: https://openreview.net/forum?id=7Rnf1F7rQhR.

[19] Y. Ji , D. Kaestner , O. Wirth , C. Wressnegger , Randomness is the Root of All Evil: More Reliable Evaluation of Deep Active Learning , in: Winter Conference on Applications of Computer Vision , 2023 , pp. 3943 - 3952 .

[20] T. Akiba , S. Sano , T. Yanase , T. Ohta , M. Koyama , Optuna: A next-generation hyperparameter optimization framework , in: International Conference on Knowledge Discovery & Data Mining , 2019 , pp. 2623 - 2631 .

[21] D. Lowell , Z. C. Lipton , B. C. Wallace , Practical Obstacles to Deploying Active Learning , in: Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing , 2019 .