This research presents a pipeline to find the key elements to achieve high accuracy. Indeed, one of the most common tasks in machine learning is classification, and numerous loss functions have been created to maximize this non-diferentiable goal. Previous work on loss function design was mainly guided by intuition and theory before being validated by experience. Here, we use a diferent approach: we aim to learn from experiments. This data-driven method is comparable to how general laws are found from data in physics. We automatically discovered a mathematical expression on more than 260 datasets that is highly correlated with the accuracy of a linear classifier. More interestingly, this formula replicates key findings from several earlier papers on loss design and is highly explainable. We hope this research will open up novel possibilities for developing new heuristics and foster a deeper comprehension of machine learning theory.
Most machine learning (ML) research involves creating Acquiring knowledge from experimentation would be and assessing components based on theoretical intuitions. rely on neural networks or random forests, making their a distinct strategy, similar to how physicists have at- cation task has already been studied with features such as tempted to deduce the analytical laws underlying the physical processes in nature from observations. With the development of AI, a new tendency to automate and support research with ML tools is emerging. Some mathematics [1] and physics [2, 3] researchers started to use it. The most similar approach in machine learning (ML) would be meta-learning, where a model gains experience throughout numerous learning sessions to enhance its performances without human intervention. Although this paradigm has been used successfully for many tasks, including hyperparameters optimization and neural architecture search (NAS), the solutions found are generally not explainable. Thus, it is not so surprising that the use of AI as a tool to assist in theoretical findings in ML research has received so little attention.
tween the variables in a given system is a requirement of the scientific method. Symbolic regression (SR) aims to solve the problem of finding a function that explains the hidden relationships in the data without knowing the structure of the function beforehand. Given that SR is NP-hard, evolutionary approaches have been created to ifnd approximations of solutions [ 4, 5, 6].
We choose 12 datasets and 22 feature extractors using the same manner as [13] to find a general law spanning a large range of factors for a classification challenge. The amount of classes varies from 10 to 1854, and the dimension of the embeddings spans from 256 to 2048. We used datasets such as CIFAR10, CUB200, ImageNetMini, or THINGS.
levels of linear classification, varied architectures with diferent pretraining have been chosen. Some of them are ture extractors such as ResNet, MobileNet, SqueezeNet, © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License kept untrained. We used diferent variants of popular feaCLIP, etc. We construct a meta-dataset ℳ from those 264 datasets of embeddings (the combination of all datasets by all feature extractors). Our formula has a better correlation and higher predictive power with only 5 variables (all -value < 0.01).
Method Linear Regression Decision Tree Regressor Random Forest Regr. (10 trees) Our GP formula ( )
Pearson each pair of dimensions (feats_cos_sim), the cosine simi- performances and the widespread belief that those modof predicting the accuracy requires a nonlinear combination of variables. Thus, we compare nonlinear regressors such as decision trees and random forests because of their els are among the most interpretable ones. Our formula outperformed them while being more explainable.
We ran our GP pipeline 1000× on the same training set and serialized their respective solutions and scores for analysis. The formula has a complexity of 6 nodes. We will refer to this Genetic Programming Formulas as: We can easily rewrite : = − log (
√ _ ⋅ log ( _
_ 2 1 log ( _ ⋅ ) _ /
_ _ ⋅
_ _ with: _ ⋅ _ _
) (1)
) (2)
After extracting the embeddings from diverse datasets using feature extractors, we need to determine the best achievable accuracy via a softmax classifier for each dataset of embeddings. We ing sets and learned the model for 1000 epochs with a batch size of 2048. As pre-processing, all embeddings were only ℓ2-normalized. By tracking the accuracy on the test set, we can observe the best-reached accuracy , an approximation of the best accuracy reachable ∗. = Our meta-dataset ℳ = {( , ) }=1 corresponds to all the pairs of statistical representation ∈ of each dataset of the datasets and the observed optimal accuracy ∈ . These tuples contain our inputs and outputs. =
We use the gplearn implemen- = tation because of the compactness of the solutions, speed of execution, robustness to noise [16], and ease of use.
The set of primitive function used is {log, , √,+, −, ×, ÷} and the set of terminals corresponds to the statistics describing the dataset . We evolved a population of 5000 individuals for 20 steps.
We designed a fitness function ℱ such that both: pretrained and untrained curacy, independently. We split our meta-dataset in a ifxed 75/25-train/test fashion and repeat each experiment 1000×. Since ℱ only seeks for correlation, a linear trans- 4. Discussion formation of the output value is learned on the training set in order to predict the accuracy ( =̂ ⋅ (⋅) + ).
extracted embeddings have a linear correlation with ac- formulas have similar structures and variables. divided each embedding dataset into testing and train- solution having the best test 2 score was found 6×. Our
To evaluate the performance of our GP solution, we compare it with popular regression methods using the same train/test split. Performances on the test set are reported in Table. 1. The substantial gap of 2 score between the linear regressor suggests that the task may correspond to a separability criterion while may correspond to correlation information. We found those two parts to be complementary. Indeed and
have respectively a pearson of only 0.65 and −0.87. Finally, we found that other best-performing GP can be written as a summation of two components.
is close to the Fisher’s criterion used in the Linear Discriminant Analysis (LDA) [17] where the objective is to find a linear projection that maximizes the ratio of between-class variance and the within-class variance. Thus,
corresponds to a separability measure of classes. Remarkably, this criterion has been efectively applied as a loss function in deep learning [18, 19]. The choice of an LDA-based loss function remains marginal in deep learning, the crossentropy (CE) being a more popular choice. However, strong similarities between the LDA and the CE allow us classifier accuracy evaluation?, in: CVPR, Procs., to swap this first separability measure with the latter one. 2021, pp. 15069–15078.
Indeed, [20] noticed that one of the most widely studied [10] E. Collins, N. Rozanov, B. Zhang, Evolutionary technical routes for the CE-based losses is to encourage data measures: Understanding the dificulty of text stronger intra-class compactness and larger inter-class classification tasks, in: CoNLL, 2018. separability such as the Fisher’s criterion. The second [11] F. Scheidegger, R. Istrate, G. Mariani, L. Benini, part, , is negatively correlated to the accuracy. The C. Bekas, C. Malossi, Eficient image dataset clasifrst variable is the number of classes ( _ ). Indeed, sification dificulty estimation for predicting deepit is natural to expect scores to decrease as the number of learning accuracy, The Visual Computer 37 (2021). classes grows. For example [21] observed a drop in accu- [12] Y. Yamada, T. Morimura, Weight features for preracy on the CUB200 dataset when changing the number dicting future model performance of deep neural of classes from a coarse level to a fine-grained one. networks., in: IJCAI, 2016, pp. 2231–2237.
In defense of the weights decorrelation term (proto- [13] B. Chamand, O. Risser-Maroix, C. Kurtz, P. Joly, types_cos_sim), [22] found on several state-of-the-art N. Loménie, Fine-tune your classifier: Finding corCNN that they could achieve better accuracy, more stable relations with temperature, in: ICIP, Procs., 2022. training, and smoother convergence by using orthog- [14] T. K. Ho, M. Basu, Complexity measures of superonal regularization of weights. Previous works on fea- vised classification problems, TPAMI 24 (2002). tures decorrelation heavily justify the presence of our fea- [15] A. C. Lorena, L. P. Garcia, J. Lehmann, M. C. Souto, tures decorrelation variable (feats_corr ) [23, 24, 25, 20, 26]. T. K. Ho, How complex is your classification probIndeed, [25] found that correlated input variables usu- lem? a survey on measuring classification complexally lead to slower convergence. Thus several proposi- ity, ACM Computing Surveys 52 (2019). tions were developed to better decorrelate variables such [16] W. La Cava, P. Orzechowski, B. Burlacu, F. O. as PCA, or ZCA. More recently, decorrelation played de França, M. Virgolin, Y. Jin, M. Kommenda, J. H. an important role in the performance increase of self- Moore, Contemporary symbolic regression methsupervised methods [23, 24, 26]. ods and their relative performance, in: J. Van
In this paper, we showed that a simple pipeline could schoren, S. Yeung (Eds.), Proceedings of the NIPS help us to extract theoretical intuitions from experimenta- Track on Datasets and Benchmarks, 2021. tion. Our formula is highly explainable and is consistent [17] R. A. Fisher, The use of multiple measurements in with decades of research. While this work is still ongoing, taxonomic problems, Annals of eugenics 7 (1936). we are working on an extended version [27]. [18] M. Dorfer, R. Kelz, G. Widmer, Deep linear discriminant analysis, in: ICLR, Procs., 2016. [19] B. Ghojogh, et al., Fisher discriminant triplet and References contrastive losses for training siamese networks, in: IJCNN Procs, IEEE, 2020. [1] A. Davies, et al., Advancing mathematics by guiding [20] W. Wan, Y. Zhong, T. Li, J. Chen, Rethinking feature human intuition with ai, Nature 600 (2021) 70–74. distribution for loss functions in image classifica[2] M. R. Douglas, Machine learning as a tool in theo- tion, in: CVPR, Procs., 2018.
retical science, Nature Reviews Physics (2022). [21] D. Chang, K. Pang, Y. Zheng, Z. Ma, Y.-Z. Song, [3] M. Schmidt, H. Lipson, Distilling free-form natural J. Guo, Your” flamingo” is my” bird”: Fine-grained, laws from experimental data, Science 324 (2009). or not, in: CVPR, Procs., 2021. [4] D. A. Augusto, H. J. C. Barbosa, Symbolic regression [22] N. Bansal, X. Chen, Z. Wang, Can we gain more via genetic programming, in: SBRN, Procs., 2000. from orthogonality regularizations in training deep [5] J. R. Koza, Genetic programming - on the program- networks?, in: NIPS, Procs., 2018. ming of computers by means of natural selection, [23] A. Ermolov, A. Siarohin, E. Sangineto, N. Sebe, Complex adaptive systems, MIT Press, 1993. Whitening for self-supervised representation learn[6] Q. Lu, J. Ren, Z. Wang, Using genetic programming ing, in: ICML, Procs., PMLR, 2021. with prior formula knowledge to solve symbolic [24] T. Hua, et al., On feature decorrelation in selfregression problem, Computational Intelligence supervised learning, in: ICCV, Procs., 2021. and Neuroscience (2016). [25] Y. A. LeCun, L. Bottou, G. B. Orr, K.-R. Müller, Efi[7] R. Istrate, et al., Tapas: Train-less accuracy predic- cient BackProp, 2012.
tor for architecture search, in: AAAI Procs, 2019. [26] S. Zhang, et al., Zero-cl: Instance and feature decor[8] W. Wen, et al., Neural predictor for neural architec- relation for negative-free symmetric contrastive ture search, in: ECCV, Procs., 2020. learning, in: ICLR, Procs., 2022. [9] W. Deng, L. Zheng, Are labels always necessary for [27] O. Risser-Maroix, B. Chamand, What can we learn by predicting accuracy?, 2022. a r X i v : 2 2 0 8 . 0 1 3 5 8 .