The AAAI 2023 Spring Symposium on Challenges Requiring the Combination of Machine Learning and Knowledge Engineering brought together researchers and practitioners from machine learning and knowledge engineering. The goal was to explore how combining these two fields can help address future AI challenges. The symposium included a joint keynote presentation by AI pioneers, over 25 presentations by contributors and authors who shared their research findings, and two challenges for the community to tackle in a follow-up event. This paper reports on the symposium and focuses on the current trend of generative AI and large language models (LLMs) and its possible synergy with knowledge-based systems (KBS), as the keynote speakers and the symposium chair emphasized. The discussions highlighted the potential of combining KBS's knowledge representation capabilities with LLMs' language generation capabilities.
Meanwhile, Douglas B. Lenat, the visionary behind the Cyc project and founder of Cycorp, is a Fellow of the AAAI. During their captivating joint keynote presentation, they shared invaluable insights and perspectives on AI’s historical milestones and future trajectory, shedding light on the challenges and opportunities of integrating machine learning and knowledge engineering. Additionally, they provided valuable commentary on the growing prevalence of generative AI and the trends surrounding large language models (LLMs), prompting thoughtful reflections within the symposium’s audience.
The symposium featured over 25 presentations by contributors and authors, showcasing their papers, datasets, ontologies, and initial research findings. These presentations encompassed a wide array of topics, including but not limited to hybrid (human-artificial) intelligence and the concept of human-in-the-loop interactions. Moreover, the discussions delved into commonsense reasoning and explainable AI, highlighting the importance of imbuing AI systems with the ability to provide transparent and interpretable outputs. The symposium explored research directions, such as hybrid AI approaches and neuro-symbolic AI, which combine the strengths of both symbolic reasoning and neural networks. In addition, human-centered AI, dialogue systems, and conversational AI were explored, recognizing the significance of designing AI systems that efectively interact and communicate with humans. Lastly, the symposium incorporated valuable insights from industry experts, who shared real-world application scenarios and the specific requirements that industries expect from AI technologies.
In addition, the symposium featured two captivating challenges that engaged the community and encouraged their active participation in a follow-up event. The first challenge, presented by Paulo Shakarian from Arizona State University, centered around the independent evaluation of ChatGPT’s performance on mathematical word problems. Shakarian proposed a benchmark dataset specifically designed for assessing the capabilities of chatbot systems in solving mathematical word problems in natural language. This challenge aimed to push the boundaries of AI in the realm of mathematical problem-solving.
The second challenge, by Maaike de Boer and Roos Bakker from TNO, revolved around the dynamic ontology matching challenge. Recognizing the pressing challenges faced by the labor market, they proposed developing a novel ontology-matching approach for aligning ontologies related to labor market dynamics. This challenge addresses the labor market’s friction between demand and supply, highlighting the potential of knowledge engineering and machine learning in ofering innovative solutions.
By incorporating these challenges, the symposium anticipates fostering knowledge exchange and collaboration and provides a platform for researchers, practitioners, and students to tackle real-world problems and explore the potential of combining machine learning and knowledge engineering in practical applications.
Edward A. Feigenbaum [6] remarked that most AI research and development is done in perception/recognition (such as statistical, data-oriented, and deep learning approaches), which is a highly competitive field. In addition, he pointed out that the cognition (such as reasoning and logic-based approaches) field has the potential to yield significant breakthroughs as it is mainly unsolved and less crowded. Feigenbaum also raised an intriguing question about the boundary between perception and cognition in AI, wondering if behaviors currently considered “cognitive” could become “perceptual.” He proposes exploring how much “thinking” is actually “recognizing” could be a promising research theme. Lastly, Feigenbaum suggests that young AI researchers should focus on less crowded areas, particularly investigating the boundary between perception and cognition.
Douglas B. Lenat [7] highlighted, in particular, the question of why current LLMs seem untrustworthy and brittle. Lenat elaborated on using a knowledge-based system as a source of truth to bias LLMs towards correctness and showcased the results of experiments on using LLMs, in this case GPT-3 [8], to generate CycL [9] “sentences.”
The possible utilization of LLMs in collaboration with reasoning systems has been stressed along the same lines in the opening symposium speech by Andreas Martin [10]. As depicted in Figure 1, Martin showcased the possible utilization of a probabilistic language model (LM) with instruction training [11], e.g., ChatGPT [12], that generates RDF(S) triples [13, 14, 15] from textual statements, with the potential to perform RDF(S) reasoning and constraint verification, as a new way to accomplish knowledge engineering.
The illustrated approach has been represented using a boxology [16, 17] that describes a hybrid intelligence [18] use case where text data (a prompt) is fed into a machine learning (ML) component to generate symbolic RDF(S) code. Subsequently, machine reasoning is performed as part of knowledge engineering (KE) to further infer explicit knowledge as RDF(s) triples.
The resulting knowledge can then be transformed using an ML component for text generation and verbalization, converting it into natural language text data. This allows the gained inferences to be expressed in human-readable text. Figure 1 shows a prompt with common-sense knowledge on the top left as schema definition and instance description, which has been engineered by a human and then sent to ChatGPT. Below the prompt in Figure 1, the response of ChatGPT is presented. In this straightforward use case with this particular prompt given, the results of this not representative experiment were correct in all cases. However, further investigations in this field with more complex use cases are needed. Additional experiments on inferring types resulted in randomness, with the language model occasionally generating hallucinations by making up triples and rules.
As these LMs are probabilistic next-token predictors [19] and the requested reasoning seems to be simulated, it can be doubted whether reasoning at the level of RDF(S) can be achieved at all. It appears that the used LM is just trying to replicate and adapt RDF(S) triples that were already present in the training dataset, which were originally obtained from publicly accessible code repositories. Moreover, even in this simple, seemingly harmless example, it can be discussed that biases and stereotypes, e.g., about names and who is having children, through LLMs in particular [20], can be injected here. This also speaks for a possible inclusion of a general common-sense knowledge base verified ethically and under diversity aspects.
In conclusion, the experiment in Figure 1 demonstrates how machine learning and knowledge engineering can work together. The generated triples can be verified through constraint checking, ontology matching/alignment, RDF(S) reasoning, or human input in a human-in-theloop setting [17].
The participants unanimously regarded the AAAI-MAKE symposium 2023 as an exceptionally successful, inspiring, and thought-provoking event, which efectively showcased the cuttingedge advancements and exemplified the immense potential and value derived from the integration of machine learning and knowledge engineering in the realm of AI research and practice. Moreover, the symposium served as a vibrant platform facilitating networking opportunities, nurturing meaningful interactions, and fostering fruitful contributions among participants from diverse backgrounds and domains.
As the symposium drew to a close, an engaging discussion transpired regarding a follow-up event, which could take the form of a subsequent symposium focusing on assessing approaches that combine knowledge engineering and machine learning and featuring captivating challenge presentations. The participants expressed their unwavering interest and enthusiasm for further collaboration and the ongoing exchange of ideas surrounding this crucial topic.
Overall, the AAAI-MAKE symposium 2023 left a lasting impression, igniting a collective commitment among the participants to sustain their collaborative eforts and propel advancements in the fusion of machine learning and knowledge engineering, ultimately charting new frontiers in AI.
Firstly, the AAAI-MAKE community would like to thank the two keynote speakers, Edward A. Feigenbaum and Douglas B. Lenat, for their exceptionally inspiring joint keynote. Further, thanks go to Maaike de Boer (TNO), who delivered a short talk on behalf of AAAI-MAKE during the 2023 AAAI spring symposia plenary session.
We want to acknowledge the work of the organizing committee consisting of Reinhard Stolle, Doug Lenat, Hans-Georg Fill, Aurona Gerber, Knut Hinkelmann, Frank van Harmelen, and the symposium chair, Andreas Martin.
We would also like to thank the session chairs, Maaike de Boer, Reinhard Stolle, Thomas Schmid, Emanuele Laurenzi, and Wilfrid Utz, and our various program committee members acting as reviewers. Finally, we would like to acknowledge Emanuele Laurenzi, who actively helped with the organization, and Charuta Pande, who compiled the proceedings. Airport, California, USA, 2023. URL: https://zenodo.org/record/8015313. doi:10.5281/ zenodo.8015313. [7] D. Lenat, AAAI-MAKE 2023 - Keynote - Personal Perspectives on AI. In Zenodo [Keynote].
AAAI Spring Symposium on Challenges Requiring the Combination of Machine Learning
and Knowledge Engineering
Media, Inc, Champaign, Illinois, 2023. [20] E. M. Bender, T. Gebru, A. McMillan-Major, S. Shmitchell, On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?, in: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, Association for Computing Machinery, New York, NY, USA, 2021, p. 610–623. URL: https://doi.org/10.1145/3442188. 3445922. doi:10.1145/3442188.3445922.