Neurosymbolic artificial intelligence is a growing field of research aiming at combining neural networks with symbolic systems, including their respective learning and reasoning capabilities. This hybridization can take many shapes which adds to the fragmentation of the field and makes it dificult to compare the existing approaches. If some eforts have been made in the community to define archetypical means of hybridization, many elements are still missing to establish principled comparisons. Amongst those missing elements are formal and broadly accepted definitions of neurosymbolic tasks and their corresponding benchmarks. In this paper, we start from the specific task of multi-label classification with the integration of propositional background knowledge to illustrate how such a benchmarking framework could look like. Based on the benchmarking of one granular task we zoom out and discuss important elements and characteristics of building a full benchmarking suite for more than just one task.
Neurosymbolic artificial intelligence (AI) is a trending research topic [
The potential of the field is based on the “best of both worlds” perspective, i.e., that by
combining neural and symbolic, the respective strengths are maintained while the weaknesses
are minimized. Thus, the objectives are extensive and include, amongst others, improved
performance [
Contrasting its promise of generalization, the field of neurosymbolic AI exhibits a
progresshampering level of fragmentation, e.g. in the evaluation and the architectural landscape. There
have been several attempts to structure the architectural approaches in the neurosymbolic AI
ifeld [
Previous researchers have highlighted the fragmentation problem and emphasized the need
for a more systematic approach to evaluating neurosymbolic AI [
In this position paper, we seek to complement the prior work tackling the fragmented neurosymbolic benchmark landscape by facing the challenge on a higher-level, focusing on the question of how to think about benchmarking neurosymbolic AI. We give an example for the setup of a specific benchmark on the task of multi-label classification with symbolic background knowledge. We include the thought process of coming up with a formal definition of the task, a suitable dataset, and a selection of metrics. Additionally, we discuss the implications for adding further benchmarks using our proposed thought process and thus contribute to a more principled benchmarking landscape for neurosymbolic AI.
A neurosymbolic benchmark can be designed to answer two main questions: “What performance level can neurosymbolic systems reach on a given task?” and “How does hybridization of neural and symbolic components help on a given task?”. The first question takes an outside view focusing on observable behavior while the second question takes an inside view focusing on the design of agents. The inside and the outside view are two well-known and deeply grounded perspectives in AI research [30]. We agree with Russell that in general artificial intelligence should be measured taking an outside view. However, answering the second question with the inside view can give further insights on how to design AI agents by understanding how and when to use neurosymbolic architectures. Additionally it might help directing the research eforts of the neurosymbolic community because advancement in task performance can be better linked to the architectural setup of the agent.
Hence, in this section, we describe the challenges of benchmarking the task of multi-label classification with symbolic background knowledge so that the two questions (internal and external) can be answered. We cover the formal definition of the task, the underlying dataset and the metrics. Although a task is not per se neurosymbolic, our chosen task covers elements that are linked to a neural (image classification) and to a symbolic (background knowledge) domain. This setup makes relatively straight-forward to use agents with a neurosymbolic architecture, and is suitable for a neurosymbolic benchmark that answers both questions.
Setting a formal definition of the task is a necessary preliminary step to compare neurosymbolic systems in a principled way. To be practical, the formalism also has to be comprehensive enough to incorporate diverse datasets (in terms of modality and background knowledge structure) and avoid a fragmentation of the field into multiple narrower tasks definitions.
Multi-label classification with background knowledge is mapping inputs ∈ ℝ to binary labels y ∈ {0, 1} such that these labels satisfy some background knowledge. This background knowledge is expressed as a propositional formula using symbols from the signature ∶= { }1≤≤ and logical connectors {¬, ∧, ∨} with their standard semantics. For lighter notations, we identify a label y ∈ {0, 1} with the propositional valuation mapping each to . Therefore, we note y ⊧ if the corresponding valuation models . A dataset for that task is ∶= ( , y )1≤≤ with ∈ ℝ , y ∈ {0, 1} such that all labels in the dataset satisfy the background knowledge, i.e. ∀1 ≤ ≤ , y ⊧ .
This formalism encompasses standard classification tasks like independent binary classification (where = ⊤ since every combination is valid) and multi-category classification (where = ( ⋁1≤≤ ) ∧ (⋀1≤<≤ (¬ ∨ ¬ )) enforces that one and only one atom is true at a time).
Since we formally introduced our task we need to discuss the dataset and the metrics to complete our benchmark.
Building an appropriate dataset for multi-label classification with background knowledge poses a substantial challenge. It must contain large amounts of data amenable to neural processing and whose labels present some significant structure expressible in the language of propositional logic. Eforts to build such datasets were often led by researchers trying to measure the performance of their neurosymbolic system, meaning that diferent systems are rarely evaluated on the same datasets and that datasets are often custom built to fit the capacity of a given system.
We observed three patterns in how datasets were created: symbolic datasets where a symbolic reasoning task is turned into a learning task (e.g. finding the shortest path in a weighted graph [31]), compositional datasets where instances are tuples of a base sub-symbolic classification dataset constrained to respect a given structure (e.g. the MNIST SUDOKU dataset [26]) and hierarchical datasets where classes of a sub-symbolic classification dataset are chosen in a hierarchy of concepts (e.g. classes in ImageNet [32] are chosen amongst synsets of the WordNet hierarchy [33]).
To turn this collection of datasets into an eficient benchmark for multi-label classification with background knowledge, further aspects need to be considered. On a fundamental level, we observe an inverse relation between the complexity of the sub-symbolic features and the complexity of the symbolic structure of the dataset, which means that the zone of complex sub-symbolic features and complex symbolic structure is not well covered by existing datasets. [25] is a dataset of trafic videos (complex sub-symbolic features) where labels satisfy a rich set of constraints (complex symbolic structure). It constitutes a first step to cover that void and more eforts should be invested in that direction. On the practical side, we need to set up a standard on how to represent, store and operate neurosymbolic datasets and their corresponding background knowledge, to allow rapid testing of any system on any dataset.
To evaluate neurosymbolic systems inside our benchmark we use a combination of performance metrics (the outside view) and control metrics (the inside view). Examples of standard performance metrics are cross-entropy loss, individual accuracy, f1-score, collective accuracy, top-k accuracy. Likewise, for standard control metrics we have number of trainable parameters, number of hyper-parameters, number of FLOPS.
Besides, new control or performance metrics specific to neurosymbolic tasks might be beneficial. One example for such a performance metric is semantic consistency which tracks how many predictions of a given system match the constraints expressed by the background knowledge (see [34] or [25] for instance).
To settle on a limited set of metrics for the multi-label classification with background knowledge task (which also can be used for other tasks), we suggest to use collective accuracy and semantic consistency as performance metrics and network size (number of trainable parameters) as a control metric. The semantic consistency metric helps us understand how much the system integrates background knowledge. Collective accuracy is a very demanding metric that is robust to imbalanced datasets: we generally observe a strong correlation between collective accuracy and f1-score for instance. Eventually, network size is a good first order approximation for model capacity.
To extend the thoughts on the specific task from the previous section to cover more of the neurosymbolic AI field, a natural next step is to focus on transferring the approach on more tasks. We draw confidence in the transferability of the proposed thought process from the observation that benchmarks cited in the preceding sections have already incorporated some of our suggestions (e.g. implementing control metrics). Furthermore, existing benchmarks may benefit from our thought process to improve their comparability. For instance, in visual reasoning, CLEVR [35] and CLEVRER [36] benchmarks do not provide a formal definition of the task, which makes comparisons between systems and with other datasets hard to establish. Moreover, both underlying datasets lack sub-symbolic complexity compared to classic computer vision datasets: the community could greatly benefit from filling that void.
Expanding the focus from a specific task to a collection of tasks, i.e., creating a benchmarking suite, raises another critical question: Which tasks should be included? The diversity of tasks has been identified as a key consideration for a benchmarking suite by the discussion panel in [ 29]. Potential tasks should have ranging dificulties for both the neural and the symbolic architecture. Also similar to the Glue [37] or GlueCon [38] benchmarking suites, several diferent capabilities and skills should be needed to solve the tasks.
This position paper contributes an example thought process for designing neurosymbolic AI benchmarks. Of course a single position paper cannot fully solve all questions around building a unified benchmarking system, but, in contrast to other papers in the field so far, we refrained from marketing an individual dataset and focused more on the questions around the design phase of a benchmark. We also discussed the implications for broadening the approach to multiple tasks which will be a valuable starting point for future benchmarking discussions and designs. Next steps could include to validate our approach on more tasks and add further thoughts to the discussion around important characteristics of a more holistic benchmarking suite. revision for natural language inference, Transactions of the Association for Computational Linguistics 10 (2022) 240–256. [12] K. Zheng, K.-Q. Zhou, J. Gu, Y. Fan, J. Wang, Z. xiao Li, X. He, X. E. Wang, Jarvis: A neuro-symbolic commonsense reasoning framework for conversational embodied agents, ArXiv abs/2208.13266 (2022). [13] Y. Liang, J. Tenenbaum, T. A. Le, S. N, Drawing out of distribution with neuro-symbolic generative models, in: S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh (Eds.), Advances in Neural Information Processing Systems, volume 35, Curran Associates, Inc., 2022, pp. 15244–15254. URL: https://proceedings.neurips.cc/paper_files/paper/2022/ file/6248a3b8279a39b3668a8a7c0e29164d-Paper-Conference.pdf. [14] B. Finzel, A. Saranti, A. Angerschmid, D. Tafler, B. Pfeifer, A. Holzinger, Generating explanations for conceptual validation of graph neural networks: An investigation of symbolic predicates learned on relevance-ranked sub-graphs, KI - Künstliche Intelligenz 36 (2022) 271–285. doi:1 0 . 1 0 0 7 / s 1 3 2 1 8 - 0 2 2 - 0 0 7 8 1 - 7 . [15] M. B. Ganapini, M. Campbell, F. Fabiano, L. Horesh, J. Lenchner, A. Loreggia, N. Mattei, F. Rossi, B. Srivastava, K. B. Venable, Combining fast and slow thinking for human-like and eficient decisions in constrained environments, in: International Workshop on Neural-Symbolic Learning and Reasoning, 2022. [16] X. Chen, C. Liang, A. W. Yu, D. Song, D. Zhou, Compositional generalization via neuralsymbolic stack machines, in: Advances in Neural Information Processing Systems, volume 2020-December, 2020. [17] S. Bader, P. Hitzler, Dimensions of neural-symbolic integration - a structured survey, 2005.
URL: https://arxiv.org/abs/cs/0511042. doi:1 0 . 4 8 5 5 0 / A R X I V . C S / 0 5 1 1 0 4 2 . [18] H. A. Kautz, The third ai summer: Aaai robert s. engelmore memorial lecture, AI Mag. 43 (2022) 93–104. [19] A. d’Avila Garcez, L. C. Lamb, Neurosymbolic ai: the 3rd wave, Artificial Intelligence
Review (2023). doi:1 0 . 1 0 0 7 / s 1 0 4 6 2 - 0 2 3 - 1 0 4 4 8 - w . [20] F. V. Harmelen, A. ten Teije, A boxology of design patterns for hybrid learning and reasoning systems, Journal of Web Engineering 18 (2019) 97–124. [21] L. d. Raedt, S. Dumančić, R. Manhaeve, G. Marra, From statistical relational to neurosymbolic artificial intelligence, in: C. Bessiere (Ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, International Joint Conferences on Artificial Intelligence Organization, 2020, pp. 4943–4950. URL: https: //doi.org/10.24963/ijcai.2020/688. doi:1 0 . 2 4 9 6 3 / i j c a i . 2 0 2 0 / 6 8 8 , survey track. [22] P. Madhyastha, Towards a benchmark suite for neural-symbolic approaches for learning and reasoning, 2022. URL: https://ijclr22.doc.ic.ac.uk/program/index.html, 16th International Workshop on Neural-Symbolic Learning and Reasoning. [23] Ö. Yılmaz, A. S. d’Avila Garcez, D. L. Silver, A proposal for common dataset in neuralsymbolic reasoning studies, in: NeSy@HLAI, 2016. [24] J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, R. B. Girshick, Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016) 1988–1997. [25] E. Giunchiglia, M. C. Stoian, S. Khan, F. Cuzzolin, T. Lukasiewicz, Road-r: the autonomous driving dataset with logical requirements, Machine Learning (2023). doi:1 0 . 1 0 0 7 / s 1 0 9 9 4 - 0 2 3 - 0 6 3 2 2 - z . [26] E. Augustine, C. Pryor, C. Dickens, J. Pujara, W. Y. Wang, L. Getoor, Visual sudoku puzzle classification: A suite of collective neuro-symbolic tasks, in: International Workshop on Neural-Symbolic Learning and Reasoning, 2022. [27] A. D. Lindström, S. S. Abraham, Clevr-math: A dataset for compositional language, visual and mathematical reasoning, volume 3212, 2022. [28] C. Cornelio, V. Thost, Synthetic Datasets and Evaluation Tools for Inductive Neural Reasoning, in: N. Katzouris, A. Artikis (Eds.), Inductive Logic Programming, Springer International Publishing, Cham, 2022, pp. 57–77. [29] F. Rossi, H. Kautz, G. Marcus, L. Lamb, L. Kaelbling, Closing, 2022. URL: https://video.ibm.
com/recorded/131288165, iBM Neuro-Symbolic AI Workshops. [30] S. J. Russell, Artificial intelligence a modern approach, Pearson Education, Inc., 2010. [31] J. Xu, Z. Zhang, T. Friedman, Y. Liang, G. V. D. Broeck, A semantic loss function for deep learning with symbolic knowledge, volume 12, International Machine Learning Society (IMLS), 2018, pp. 8752–8760. [32] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, L. Fei-Fei, Imagenet large scale visual recognition challenge, International Journal of Computer Vision 115 (2015) 211–252. doi:1 0 . 1 0 0 7 / s 1 1 2 6 3 - 0 1 5 - 0 8 1 6 - y . [33] G. A. Miller, Wordnet, Communications of the ACM 38 (1995) 39–41. URL: https://dl.acm.
org/doi/10.1145/219717.219748. doi:1 0 . 1 1 4 5 / 2 1 9 7 1 7 . 2 1 9 7 4 8 . [34] K. Ahmed, S. Teso, K.-W. Chang, G. Van den Broeck, A. Vergari, Semantic probabilistic layers for neuro-symbolic learning, in: S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh (Eds.), Advances in Neural Information Processing Systems, volume 35, Curran Associates, Inc., 2022, pp. 29944–29959. URL: https://proceedings.neurips.cc/paper_ files/paper/2022/file/c182ec594f38926b7fcb827635b9a8f4-Paper-Conference.pdf. [35] J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, R. Girshick, Clevr: A diagnostic dataset for compositional language and elementary visual reasoning, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2901–2910. [36] K. Yi*, C. Gan*, Y. Li, P. Kohli, J. Wu, A. Torralba, J. B. Tenenbaum, Clevrer: Collision events for video representation and reasoning, in: International Conference on Learning Representations, 2020. URL: https://openreview.net/forum?id=HkxYzANYDB. [37] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, S. Bowman, GLUE: A multi-task benchmark and analysis platform for natural language understanding, in: Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Association for Computational Linguistics, Brussels, Belgium, 2018, pp. 353–355. URL: https://aclanthology.org/W18-5446. doi:1 0 . 1 8 6 5 3 / v 1 / W 1 8 - 5 4 4 6 . [38] H. R. Faghihi, A. Nafar, C. Zheng, R. Mirzaee, Y. Zhang, A. Uszok, A. Wan, T. Premsri, D. Roth, P. Kordjamshidi, Gluecons: A generic benchmark for learning under constraints, ArXiv abs/2302.10914 (2023).