Residuals for Equation Discovery

Abstract

Residuals for equation discovery (RED) is a simple, universal, yet efective way to improve pre-trained equation discovery systems by disentangling the original problem into simpler problems. Based on an initial equation, we compute for a subequation the residual that this subequation should have yielded so that the entire formula predicts the output correctly. By parsing the initial equation to a syntax tree, we can use node-based calculation rules to compute the residual for each subequation of the initial equation. Using this residual as new target values, the equation discovery system predicts a new subequation, which can be merged with the initial equation. We show the advantage of using residuals for equations from the Feynman benchmark.

Keywords

• eol>AI for Science
• Equation Discovery
• Decomposition

-

We calculate the residuals by representing the equation as a syntax tree. The root is a node connected to a child node. This child node can be an operator node with child nodes or a leaf node. Leaf nodes are constants or variables, and if they are called they return the corresponding value or the column from the data set. For an operator node, the mathematical operation it performs depends on which adjacent node is calling. An overview of the operator nodes is in Figure 1 II.

To evaluate the residual for a node, the node calls its parent node. Operators that are not bijective (e.g., ) cannot be inverted. Thus, for their child nodes, the residual cannot be computed.

We use NeSymReS [ 3 ] to test RED on the Feynman equations as reported in SRBench [ 2 ]. We only examine equations with a maximum of two independent variables. We first run NeSymReS on the problem once; if the mean squared error (MSE) of the equation is > 0.001, the predicted equation is parsed to a syntax tree, and for each node except the node and its child node, an alternative subequation is predicted with RED. Subsequently, we rerun the NeSymReS as many times again on the original problem as we calculated residuals. In Figure 1 III, the best results are reported for the Classic method with a median value of 0.89 (IQR 0.06-9.21) and RED with a median value of 0.003 (IQR 0.001 - 0.08).

While RED is independent of the functionality of the pre-trained equation discovery system, it depends on an initial solution, which has to enable the disentanglement. In future work, we want to analyze this constraint and perform experiments comparing multiple equation discovery systems, data set dimensionalities, and noise levels.

Acknowledgments

This research project was partly funded by the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK) within the projects The Third Wave of Artificial Intelligence (3AI) and hessian.AI

Declaration on Generative AI

The author(s) have not employed any Generative AI tools.

References

[1] S.-M. Udrescu , M. Tegmark, AI Feynman: A physics-inspired method for symbolic regression , Sci. Adv . 6 ( 2020 ) eaay2631 . doi: 10 .1126/sciadv.aay2631.

[2] W. G. L. Cava , P. Orzechowski , B. Burlacu , F. O. de França , M. Virgolin , Y. Jin , M. Kommenda , J. H. Moore , Contemporary symbolic regression methods and their relative performance , in: J. Vanschoren , S. Yeung (Eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1 , NeurIPS Datasets and Benchmarks 2021 , December 2021 , virtual, 2021 .

[3] L. Biggio , T. Bendinelli , A. Neitz , A. Lucchi , G. Parascandolo, Neural symbolic regression that scales , in: M. Meila , T. Zhang (Eds.), Proceedings of the 38th International Conference on Machine Learning , ICML 2021 , 18 - 24 July 2021 , Virtual Event , volume 139 of Proceedings of Machine Learning Research, PMLR , 2021 , pp. 936 - 945 .

[4] P. Kamienny , S. d'Ascoli, G. Lample , F. Charton , End-to-end symbolic regression with transformers , in: S. Koyejo, S. Mohamed , A. Agarwal , D. Belgrave , K. Cho , A . Oh (Eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022 , NeurIPS 2022 , New Orleans, LA, USA, November 28 - December 9, 2022 , 2022 .

[5] P. Kamienny , G. Lample, S. Lamprier , M. Virgolin , Deep generative symbolic regression with montecarlo-tree-search , in: A. Krause , E. Brunskill , K. Cho , B. Engelhardt , S. Sabato , J. Scarlett (Eds.), International Conference on Machine Learning , ICML 2023 , 23 - 29 July 2023 , Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, PMLR , 2023 , pp. 15655 - 15668 .

[6] P. Shojaee , K. Meidani , A. B. Farimani , C. K. Reddy , Transformer-based planning for symbolic regression , in: A. Oh , T. Naumann , A. Globerson , K. Saenko , M. Hardt , S. Levine (Eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023 , NeurIPS 2023 , New Orleans, LA, USA, December 10 - 16 , 2023 , 2023 .

[7] M. Valipour , B. You , M. Panju , A . Ghodsi, SymbolicGPT: A Generative Transformer Model for Symbolic Regression , 2021 . doi: 10 .48550/arXiv.2106.14131, arXiv: 2106 .14131 [cs] version: 1 .