On Teaching Logic Programming in the Era of Generative AI Paul Tarau1 1 University of North Texas, Denton, USA Abstract After fact finding on the disruption bought by today’s Generative AI tools to the education system, we outline the advantages of joining the disruption, motivated by the natural synergies between today’s Generative AI and Logic Programming Languages, derived from both their similar and complementary reasoning mechanisms. We explore engagement strategies for AI assistants with specifics on Prolog teaching and also in the wider context of online teaching and automation of teaching processes, including AI-assisted evaluation strategies for student success. In particular, we overview and explore in some detail AI-based Prolog code generators, gamification, computational thinking in logic-focussed natural language, and the use of multi-agent AI systems. Keywords AI as a disruptor of education systems, AI as a disruptor of Logic Programming teaching process, engagement strategies forAI assistants, synergies between Logic Programming and Generative AI, AI- assisted Prolog coding 1. Introduction We will focus on technical aspects for engaging the AI disruption. It is however of paramount importance to parallel and evaluate our pragmatic fact finding and entailed reasoning steps with alignment to legal, ethical and moral values, integrity, fairness and equity which are not covered in this paper. 2. The Facts on the Generative AI Disruption of Education 2.1. The Domain-independent disruption Our current educational technologies have over the last 50 years incrementally refined the closely interrelated aspects of training, testing and student evaluation and, to a much lesser extent, the real-world outcomes of the teaching process. At the same time, leftovers for its 2000+ years baggage focussed on replicating memorized content rather than actionable generalization and creativity are still present in the evaluation mechanisms of of educational processes. With the focus on verifiable metrics, our often over-automated grading processes, driven by market-dictated cost reduction on human grader involvement, have created a perfect breading ICLP 2024: N11 Workshop PEG 2.0, October, 2024, Dallas, USA. © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings ground for Generative AI induced gaming opportunities against typical grading systems1 . Some key facts on the impact on evaluation of student success can be summarized as follows: • The advanced capabilities of AI models make it nearly impossible to detect AI-generated essays or multiple choice problems, rendering traditional plagiarism detection methods ineffective. • Educators might need to reconsider the purpose and structure of assignments, including essays and problem sets, in light of AI’s ability to generate content and solve complex problems. • The rapid changes brought by AI in education demand immediate adjustments from instructors and the creation of clear policies regarding AI use to preserve educational integrity while embracing its potential benefits. Let’s follow up with an instance of a pragmatic (notwithstanding its ethical aspects!) decision that a typical average student faces. The cheating paradox The following case study (with kind assistance from GPT-4o on obvious details) puts on the table an aspect of the a commonly used grading process, that an average student focusing on passing a course with as high grades as possible faces: Example 1. After submitting the following set of assumptions to the GPT-4o assistant, actually as a simple math word problem to solve: Assume that we grade courses with 40% home work, 30% Midterm Exam and 30\% Final Exam. It is known that AI-assisted cheating on homework reduces exam performance by around 5%. Assuming that AI-assisted cheating on home works gives 95% on them and that under honest hard work the student would obtain a 70% grade. Which of the AI-assisted cheating and the honest strategy would give the highest grade to the student? Give me a short final answer just citing the grades and their difference in %. we get: With AI-assisted cheating: - Homework: - Exams (Midterm and Final): 65 % (due to a 5% reduction) - Final grade: 0.40 * 95 + 0.30 * 65 + 0.30 * 65 = 76% 1 https://www.oneusefulthing.org/p/post-apocalyptic-education With honest hard work: - Homework: 70% - Exams (Midterm and Final): 70% - Final grade: 0.40 * 70 + 0.30 * 70 + 0.30 * 70 = 70% Difference: - AI-assisted strategy gives 6% higher final grade than the honest strategy. 2.2. The Domain Specific Disruption : Prolog Let us now focus, by taking advantage of GPT-4o’s Prolog programming skills, to solve a beginner-level Prolog assignment testing list processing. Example 2. We start with the query: Write a Prolog program that removes each element in an odd position of a list of terms. Sort the result in reverse order. Show me how it works on [l, o, g, c, is, fun] we get a first shot correct program: % Base case: An empty list has no elements % to remove, so return an empty list. remove_odd_positions([], []). % Recursive case: If the list has at least two elements, % remove the first element and keep the second. remove_odd_positions([_, X | T], [X | Rest]) :- remove_odd_positions(T, Rest). % If the list has only one element left, % skip it (since it's in an odd position). remove_odd_positions([_], []). % To get the desired result, first remove odd positions, % then sort in reverse order. remove_and_reverse(List, Result) :- remove_odd_positions(List, ListWithoutOdds), reverse(ListWithoutOdds, Result). with the results on the given example list, as returned by the LLM: You can use the program on the list [l, o, g, c, is, fun] as follows: ?- remove_and_reverse([l, o, g, c, is, fun], Result). Result = [fun, c, o]. Next, let’s push ahead with a midlevel problem. Example 3. This example shows clear, efficient and well explained Prolog code on a less trivial problem: Write a Prolog program that finds the first 5 pairs of twin primes and show the Result it returns. The LLM returns the following code: % Define a predicate to check if a number is prime is_prime(2). is_prime(3). is_prime(N) :- N > 3, N mod 2 =\= 0, \+ has_factor(N, 3). % Helper predicate to check if a number has a factor has_factor(N, F) :- N mod F =:= 0. has_factor(N, F) :- F * F < N, F2 is F + 2, has_factor(N, F2). % Define a predicate to find twin primes twin_prime(X, Y) :- is_prime(X), Y is X + 2, is_prime(Y). % Find the first N pairs of twin primes find_twin_primes(0, _, []) :- !. find_twin_primes(N, X, [(X, Y)|Rest]) :- N > 0, twin_prime(X, Y), N1 is N - 1, X2 is X + 1, find_twin_primes(N1, X2, Rest). find_twin_primes(N, X, Result) :- X2 is X + 1, find_twin_primes(N, X2, Result). % Predicate to get the first 5 pairs of twin primes first_5_twin_primes(Result) :- find_twin_primes(5, 2, Result). with the correct results computed by the AI’s implicit Prolog reasoning engine: ?- first_5_twin_primes(Result). Result = [(3, 5), (5, 7), (11, 13), (17, 19), (29, 31)]. Note also a limitation here: the program will not manage a very large number of twin prime pairs as AI’s math skills are still imperfect on computations with large values. The AI-assisted code generation quality extends to puzzles stated in natural language, reacha- bility problems, transitive closure, graph coloring and combinatorial problems, with often first shot correct and elegant, well documented Prolog programs. Our conclusion here is that today’s top Generative AI models (e.g. OpenAI GPT-4o, Anthropic’s Claude, Google’s Gemini, etc.,) know more than enough Prolog coding for covering at least the teaching domain and specifically its student evaluation subdomain. We encourage the reader to try out similar examples for intuition on the domain coverage of the Prolog Generative AI coder, both when restricted to teaching and also on snippets of industrial strength code covering Prolog based knowledge graphs, Definite Clause Grammars, networking, planning and constraint-solving problems, Prolog-based Web apps, and so on. 2.3. On the logic of joining the disruption Generative AI-denial is a natural, partly rational, partly emotion-driven reaction of practitioners of symbolic AI, Natural Language Processing and classic Machine Learning that are prime disruption targets of it, as it suddenly deprecates years of hard work on significant academic and industrial achievements. While keeping this in mind, we will explore here a rational and pragmatic path for joining the disruption motivated by the natural synergies that Logic Programming and Generative AI have together. In fact, it became obvious to this author much earlier, at the time of pure text completion models like GPT3 and and declarative image builders like DALL.E that the disruption enables the use of Prolog tools like Definite Clause grammars as generators for prompt and image synthesis [1]. As a recent proof of concept, illustrating the fast progress in the field, we refer to [2] as a prototypical example for the impressive performance gains brought by interleaving Prolog and AI steps on a set of difficult reasoning benchmarks. The more than 100 entries of the leaderboard at lmsys.org of a competition between LLMs, based on an ELO tournament that pairs them to be judged by thousands of user votes on the quality of their output, testifies on the accelerated evolution of today’s Generative AI systems2 . 3. Engagement strategies with AI Assistants We will next explore several engagement strategies with Generative AI-based teaching assistants. 3.1. A basic AI engagement strategy • Learn underlying concepts, try to solve the problem. • If stuck, iterate a few times with alternative ways to look at it. • If still stuck, call the AI-assistant using the instructor-provided prompt or your own. 2 https://lmarena.ai/ • After reading the solution, answer to a set of multiple choice questions about the AI’s code. The instructor can use the results of your answers for grading. Note that this could be still automated by the instructor’s code asking the LLM to grade the resulting code if submitted and grading the answers to the follow-up AI generated question. See our WebApp3 for a (recursive) follow-up question generator. In the case of Prolog, this AI-assisted coding pipeline can be refined by requiring the students to run the AI’s code and explain the results of a few spy points by tracing the execution. On more interactive online engagement on platforms like Canvas4 , engaging in discussion groups on the AI generated code can provide a richer learning experience that is also gradable by the instructor or an AI assistant. 3.2. Gamification Gamification5 relies on borrowing elements from the gaming world students are naturally exposed to. An example of AI assisted gamified learning process looks as follows: • an assignment requests a small group of students to compare a Prolog problem given by each to a different AI of their choice • the students will be timed on testing and possibly debugging the AI-provided solutions and gain points, banners etc. for their performance • the students discuss the merits and demerits of each in a Canvas forum with a partial grade originating from their participation in the forum, possibly evaluated by an AI-agent if we want a fully automated grading process. A refinement to the grading would be that allowing AI-assistance (seen as the equivalent of “magic spells” in a game) will have a point cost. Thus, it will need, like in the gaming world, some extra work to refill the player’s “strength level”, with clear learning benefits in this case. Another approach, easy to implement in practice, would be organizing student coding contests with participation of (possibly artificially weakened) AI antagonists. This is particularly effective when the problems’ solution is is already known to the learner or it has even been previously implemented in a different language. 3.3. Engagement with Natural Language and Logic Programming Given its syntactic and semantic features, it makes sense, when teaching Prolog, to focus on closeness to natural language while keeping in mind the importance of learnability, presence of flexible execution mechanisms and exposure to Prolog’s highly expressive language constructs. Good barometers for learnability are the learning curves of newcomers (including children in their early teens), the hurdles they experience and the projects they can achieve in a relatively short period of time. Another observable to watch is how well one can pick up the language inductively, simply by looking at coding examples. 3 https://auto-quest.streamlit.app/ 4 https://www.instructure.com/canvas 5 https://uwaterloo.ca/centre-for-teaching-excellence/catalogs/tip-sheets/gamification-and-game-based-learning When thinking about what background can be assumed in the case of newcomers, irrespectively of age, natural language pops up as a common denominator. As logic notation originates in natural language, there are conspicuous mappings between verbs and predicates and nominal groups as their arguments, especially relevant when solving knowledge representation problems. That hints toward learning methods and language constructs easily mapped syntactically and semantically to natural language equivalents. Along these lines, our Natlog system6 [3, 4] enforces closeness to Natural Language by a simplified, flat Prolog syntax while opening access to Python’s rich finite set and coroutining primitives, while ensuring its smooth integration in the Python-based deep learning ecosystem. The reader is invited to try out Natlog programs online7 . The following code snippets give a hint about Natlog’s code focusing on learnability while keeping in mind a focus on expressiveness as a key programming language feature. Example 4. Natlog as a syntactically lighter Prolog equivalent: mother of X M: parent of X M, female M. father of X M: parent of X M, male M. grand parent of X GP: parent of X P, parent of P GP. ancestor of X A : parent of X P, parent or ancestor P A. parent or ancestor P P. parent or ancestor P A : ancestor of P A. Example 5. Natlog as expressive code defining at source level a the primitive findall/3: stack S : `list S. push S X : #meth_call S append (X). pop S X : `meth_call S pop () X. findall X G Xs: listof X G S, to_cons_list S Xs. listof X G S: stack S, collect_ X G S. collect_ X G S : call G, `copy_term X CX, push S CX, fail. collect_ _X _G _S. As saliently emphasized in [5] there’s an important distinction to be made between Prolog as a programming language and Prolog as knowledge representation language. This will become relevant both for deciding the content of teaching materials and the AI-assistant teaching strategies. 6 https://github.com/ptarau/natlog 7 https://natlog.streamlit.app When focusing on the latter, AI-assistance will express its results in the form of a Propositional Horn Clause program, with facts and the rules connecting them consisting of natural language statements or noun groups describing concepts and their details. This involves a recursive descent processes starting with a knowledge exploration initiator expressed as a sentence or a key concept, that returns executable Prolog programs that can be evaluated with a simple fixpoint iteration or in tabling-enabled Prolog system like XSB [6] or SWI-Prolog [7]. We refer to [8] and [9] for details of this process and invite the reader to interact with our online deepllm streamlet app8 illustrating it. Figure 1: Exploring a concept as a Propositional Horn Clause Program Fig.1 shows an example exploring the concept of “inductive definition”. Note that, besides the uses for actually introducing a student to Prolog programming, the close connection to natural 8 https://deepllm.streamlit.app/ language exposes the student to a Prolog-inspired computational thinking processes, involving task decomposition, with its complementary aspects of refinement and exploration of alternatives. As an additional instructional tool, the deepllm app also builds an extended concept graph to expose deeper, LLM-generated relations to similar concepts. Figure 2: Visualization of the concept graph related to “inductive definitions” Fig.2 illustrates this relation graph for the concept of “inductive definition” as extracted from the minimal model of the Propositional Horn Clause program. 3.4. Teaching with multi-agent AI systems A strong trend in today’s large scale, industrial strength AI API based applications is the the use of multi-agent architectures9 chaining together custom code, vector store and database calls as well as LLM API calls. An AI-driven multi-agent online course architecture could involve a society of specialized AI agents, that in its Prolog instance would look like the following: • Prolog code generator assistant • Prolog code critique and bug finder • natural language to/from executable Prolog translator • conversation animator AI bot as part team projects or forums • instructor representative as performance evaluator 9 https://python.langchain.com/v0.1/docs/modules/agents/ The multi-agent society’s members could watch and react to unblock students based on the Agents’ specific skills and available tools using a publish-subscribe architecture while keeping track of each student progress in a student-specific memory track. 3.5. AI-assisted teaching with human in the loop We have focussed so far on scenarios involving AI-assisted automated and scalable online teaching tools that are (slowly but firmly) becoming prevalent in the education system. However even in the context, the presence of the human in the loop10 is of paramount importance. Real time or asynchronous presence of the human instructor can be naturally integrated when unexpected events are likely to occur in the process. Ideally, in a large scale online teaching system this could be an instant real-time intervention by an instructor, possibly assisted by an AI-generated summary of the blockers a given student faces. At a smaller scale, help ticket-style event queues and blackboards could be made available soliciting instructor intervention with well-defined status details. 3.6. Some “out of The box” ideas on on reshaping upcoming AI-assisted Teaching Methodologies We sketch here a few possible refinements and adaptations given the availability of large scale generative AI to the two sides of the educational process: students and their instructors. Transition from penalty-based to reward-based teaching systems Similarly to the success of Reinforcement Learning in the training loops of the today’s user-facing LLMs, possibly also seen as part of a gamification process, reward-based engagement has plenty of room to replace or complement today’s penalty-based student-evaluation mechanisms. Spontaneous social network study groups among remote online friends exposed to a diversity of teaching systems With physical colocation loosing some of its relevance to joint interests on social network co-presence and exposure to a wider diversity of teaching methods, often crossing country borders, is an opportunity to acquire intuitions about key concepts learned in the conventional school systems and synchronize terminological barriers. Facilitate curiosity-driven AI-assisted self study With personalized, user aware generative AI (e.g., Character AI’s agents11 ), individual curiosity on a school topic can be explored directly and turned into a learning process while interacting with the customized agent aware of past interactions with its owner. Industry-inspired project-driven learning with AI assistance It is by now standard practice in industrial code development to use tools (e.g., Github Copilot12 ) that revise or even create and inject new code in the coder’s development pipeline. 10 https://hai.stanford.edu/news/humans-loop-design-interactive-ai-systems 11 https://character.ai/ 12 https://github.com/features/copilot This is particularly useful in project-based courses like terminal undergrad classes or in-depth graduate classes involving software artifacts as deliverables. Focus on spending teaching time on the “why” besides the “how” in problem solving The “why” part has often been skipped over in the teaching process partly to keep the focus on the topic at hand and partly because its extent is often too wide to be presented under the time-constraints of the teaching time and the student evaluation process. Nevertheless, presenting the “ontology” covering not only the taxonomical relations between concepts but also their historical origins can be enlightening and Generative AI assistants can complement the course content with them, without a significant burden on instructor time. Extend the narrow domain-specific teaching to encompass the spectrum of related concepts to which students might have been already exposed This ranges from text simplification and a jargon removal using classic prompts like “explain me X such that a 9 years old can understand it” to using AI-generated relation graphs exposing similarities, generalizations and analogies to concepts in nearby fields. In particular, one can use AI-assistants to help “connect the dots” by forming solid intuitions of the concepts hiding beyond domain-specific taxonomies. Moving from canned routines and memorized factoids to actionable knowledge out- comes This is an uphill battle against a 2000+ years tradition, but it might be facilitated by appropriate AI-assistance to encourage immersive content-driven learning. It also involves rethinking of deadline-driven assignments, with focus on quality vs. quantity, as well as prioritizing of the learning outcomes vs. the convenience of assigning and grading the work. There are also the usual market pressures exercised on “teaching as a business”, but dropping the AI-cheatable canned multiple-choice questions in favor of AI-assisted engagement in actual content creation looks feasible given the technical competence of today’s a generative AI systems. Design course materials to include video presentations of AI-assisted teaching Finally, the wide availability of high quality multi-media teaching materials suggest producing content that describes not just the topic at hand, but also the instructor’s interaction with Generative AI agents, to build, clarify and present the content creation process and its outcomes. 4. A brief Related Work Overview The domain is fast shifting to the point that even a few months old related work, covering different tools or assuming deprecated limitations and features of now out of use AI models loses its informational value. We focus here on some work on Logic Programming and Generative AI that show a long term vision and development plans or some possibly out of the box thinking about the topic. Goal-directed ASP systems like s(CASP) [10], have provided forms of non-monotonic and co-inductive reasoning. With enhanced constructive constraint programming algorithms, going back to [11] and presenting also explanations covering the reasons of negative outcomes, they show synergies between these extensions to Prolog and Generative AI induced logic program snippets [12]. A more direct approach is recursion on LLM queries, by chaining the LLM’s distilled output as input to a next step and casting its content and interrelations in the form of logic programs, to automate and focus this information extraction with minimal human input [8, 9]. Like in the case of typical RAG architectures [13, 14], this process can rely on external ground truth but it can also use new LLM client instances as “oracles” deciding the validity of the synthesized rules or facts. We refer to [8] for an extensive list of LLM-generated Horn clause programs and [9] for a wider variety of them, including DCG encodings of recursive descents driven by AI-generated follow-up questions on a given topic as well as nicely visualized knowledge representation outputs. 5. Conclusion and Future Work We have, hopefully, made it clear that it is widely beneficial to join the disruption that Generative AI assistance brings into teaching processes and also discussed the specifics of teaching Prolog in the context. In fact, we can see Prolog teaching as a “Goldilocks” case, given the natural, reasoning-induced synergies between Prolog and Generative AI-based tools. With this in mind, automated online teaching and student-testing tools based on Prolog code generation and natural language dialog threads expressed as logic representations ranging from propositional Horn clause or Datalog forms to specialized Prolog code and Definite Clause Grammars have the potential to be extended to similar teaching and student-testing techniques for logic languages like ASP and s(CASP) as possibly also refined to support other programming paradigms. References [1] P. Tarau, Reflections on Automation, Learnability and Expressiveness in Logic-Based Programming Languages, Springer Nature Switzerland, Cham, 2023, pp. 359–371. doi:10. 1007/978-3-031-35254-6\_29. [2] N. Borazjanizadeh, S. T. Piantadosi, Reliable reasoning beyond natural language, 2024. URL: https://arxiv.org/abs/2407.11373. arXiv:2407.11373. [3] P. Tarau, Natlog: a Lightweight Logic Programming Language with a Neuro-symbolic Touch, in: A. Formisano, Y. A. Liu, B. Bogaerts, A. Brik, V. Dahl, C. Dodaro, P. Fodor, G. L. Pozzato, J. Vennekens, N.-F. Zhou (Eds.), Proceedings 37th International Conference on Logic Programming (Technical Communications) , 20-27th September 2021, 2021. doi:10.4204/EPTCS.345.27. [4] P. Tarau, Natlog: Embedding Logic Programming into the Python Deep-Learning Ecosys- tem, in: E. Pontelli, S. Costantini, C. Dodaro, S. Gaggl, R. Calegari, A. D’Avila Garcez, F. Fabiano, A. Mileo, A. Russo, F. Toni (Eds.), Proceedings 39th International Conference on Logic Programming, Imperial College London, UK, 9th July 2023 - 15th July 2023, volume 385 of Electronic Proceedings in Theoretical Computer Science, Open Publishing Association, 2023, pp. 141–154. doi:10.4204/EPTCS.385.15. [5] M. Genesereth, Prolog as a Knowledge Representation Language The Nature and Impor- tance of Prolog, in: D. S. Warren, V. Dahl, T. Eiter, M. Hermenegildo, R. Kowalski, F. Rossi (Eds.), Prolog - The Next 50 Years, number 13900 in LNCS, Springer, 2023. [6] T. Swift, S. Warren, David, XSB: Extending Prolog with Tabled Logic Program- ming, Theory and Practice of Logic Programming 12 (2012) 157–187. doi:10.1017/ S1471068411000500. [7] J. Wielemaker, T. Schrijvers, M. Triska, T. Lager, Swi-prolog, Theory and Practice of Logic Programming 12 (2012) 67–96. [8] P. Tarau, Full Automation of Goal-driven LLM Dialog Threads with And-Or Recur- sors and Refiner Oracles (2023) arXiv:2306.14077. doi:10.48550/arXiv.2306.14077. arXiv:2306.14077. [9] P. Tarau, System description: Deepllm, casting dialog threads into logic programs, in: J. Gibbons, D. Miller (Eds.), Functional and Logic Programming, Springer Nature Singapore, Singapore, 2024, pp. 117–134. doi:10.1007/978-981-97-2300-3\_7. [10] J. ARIAS, M. CARRO, E. SALAZAR, K. MARPLE, G. GUPTA, Constraint answer set programming without grounding, Theory and Practice of Logic Programming 18 (2018) 337–354. doi:10.1017/S1471068418000285. [11] P. Stuckey, Negation and Constraint Logic Programming, Information and Com- putation 118 (1995) 12–33. URL: https://www.sciencedirect.com/science/article/pii/ S0890540185710486. doi:10.1006/inco.1995.1048. [12] Y. Zeng, A. Rajashekharan, K. Basu, H. Wang, J. Arias, G. Gupta, A reliable common-sense reasoning socialbot built using llms and goal-directed asp, 2024. URL: https://arxiv.org/abs/ 2407.18498. arXiv:2407.18498. [13] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al., Retrieval-augmented generation for knowledge-intensive nlp tasks, Advances in Neural Information Processing Systems 33 (2020) 9459–9474. [14] P. Sarthi, S. Abdullah, A. Tuli, S. Khanna, A. Goldie, C. D. Manning, Raptor: Recursive abstractive processing for tree-organized retrieval, 2024. URL: https://arxiv.org/abs/2401. 18059. doi:10.48550/arXiv.2401.18059. arXiv:2401.18059.