The emergence of large language models has significantly advanced the feasibility of automated problem-solving using agents. However, despite promising results, these systems often function as “black boxes”, raising concerns about their ability to comply with requirements due to opaque decision-making processes. To mitigate these issues, we introduce a multi-agent system powered by language models. This system segments the decisionmaking process into three agent-driven stages: proposing queries, identifying norms, and retrieving facts, while delegating final judgment to a logical reasoner. We evaluated our system in simulated driving scenarios governed by a limited set of trafic regulations. Results indicate that our approach markedly enhances compliance with decision-making accuracy and ofers a more interpretable and traceable method compared to methods that rely solely on language models.
The rapid advancement and widespread adoption of Artificial Intelligence (AI) are revolutionizing
various industries and reshaping human society [
Actions Observations
False
True
assesses these inputs, ensuring eficient and efective compliance with established rules. Our principal contributions include: • A rule-compliant decision-making system: Integrates multiple LLM agents, each specialized in diferent aspects of the decision-making process from query generation to fact retrieval, thereby streamlining the logical reasoning workload. • Evaluation in simulated driving scenarios: Our system outperformed those relying solely on language models, achieving not only higher decision accuracy but also greater interpretability and transparency.
Aligning the behavior of automated agents with established norms is crucial for safe deployment in
real-world applications [
We define a scenario by a set T that includes distinct objects {1, 2, . . . , }. Each object is associated with a set of properties P = {1, 2, . . . , }, where represents the number of properties each object possesses. An automated agent operating within this scenario can perform actions from a predefined set A = {1, 2, .., }. These actions are either explicitly or implicitly regulated by a set of legal norms R. Each rule in R takes the form Φ → Ψ , stipulating that the occurrence of condition Φ mandates or prohibits the outcome of actions Ψ . The task is to determine a subset of actions ⊆ A that, when executed by the agent, complies with all the regulations in R. Specifically, the problem can be formally expressed as:
∀(Φ → Ψ) ∈ R, (T |= Φ) ⇒ ( |= Ψ) , where |= signifies the satisfaction relation, indicating that if the scenario T satisfies the condition Φ , then the chosen actions must ensure the outcome Ψ , thus adhering to the stipulated legal norms. The primary challenge in this task is the indirect evaluability of rule conditions based on the available facts. For instance, a rule for overtaking requires that the oncoming lane be clear, which involves assessing the number of vehicles visible to the ego vehicle. These essential facts are not directly available from the properties of objects; they must be inferred using domain-specific knowledge and mathematical and physical principles. Leveraging the extensive knowledge encoded in large language models can aid in making rule-compliant decisions. However, this approach risks generating hallucinations, which are unacceptable in safety-critical tasks. In contrast, traditional rule-based methods are more robust and deterministic but face significant computational challenges due to the processing of numerous rules, predicates, predicate arguments, and relevant facts. Therefore, a multi-agent decision-making system that combines the strengths of both methodologies is needed to enhance the efectiveness, robustness, and safety of the decision-making process.
The concept of “Intelligent Agents” [
F = ⟨P1, P2, . . . , P||⟩ P = ⟨T1, T2, . . . , T|P|⟩ T = (⟨I, A⟩) (1)
The limited context length of LLMs often restricts maintaining a complete communication history
among all agents and phases. To address this, agents’ context memories are segmented into short-term
and long-term memory [
3.3.1. Legal Norms Formalization
Formalizing legal norms is essential for enabling rule-based logical reasoning. However, the inherent
vagueness, abstract expressions, exceptions, and potential conflicts within established norms pose
significant challenges. Westhofen et al. [
Ontological Representation Ontology is a fundamental method for modeling domain-specific
knowledge, providing a formal and explicit specification of shared conceptualizations that facilitate
consistent and unambiguous knowledge exchange and management [
We structure the rule-compliant decision workflow into three agent-driven phases, = ⟨( ), ( ), ( )⟩, where each phase handles a single task , collaboratively aiming to create a streamlined logic program that consists of queries, facts, and rules for eficient processing by a logical reasoner. The output from the reasoner provides iterative feedback, ensuring that decisions conform strictly to established norms. As illustrated in Figure 2 , the initial phase involves two agents that interpret the trafic scene and suggest an action, such as an overtaking maneuver denoted by the query
Scene-based Query Proposing
TBox ABox
(), through mutual communication. The extracted scene features and the proposed query are then processed in the second phase by a semantic search agent, which identifies applicable rules for the case from the perspective of a legal expert. These rules are evaluated based on the available facts; any missing or unknown facts are forwarded to the third phase. In this phase, agents consider the evaluation context and the predicates of facts to generate SPARQL queries. These queries are executed on the ontology to retrieve necessary information through reasoning. The logical reasoner then processes the queries, rules, and facts to produce a decision, which either reenters the loop for further refinement or stands as the final decision. For more details about our algorithm, we refer to Appendix A.1.
To connect diferent components within a system, we have integrated three distinct types of data streams.
The first type encompasses environmental data sourced from vehicles for environment perception and
communication, such as trafic conditions and road infrastructure. This data is assumed to be accessible
via the Controller Area Network (CAN) from Electronic Control Units (ECUs) within the vehicle. The
second type involves semantic data, which is understandable and executable by ontologies and logical
reasoners. The third type comprises natural language, generated by LLMs to provide instructions or
answers. The environmental data, characterizing a trafic scene as key-value pairs, lacks the semantic
meaning required for direct evaluation of trafic rule predicates, such as hasOncomingTraffic(, ).
To enable the ontology to infer new facts, such as spatial relationships and object counts, we map the
environmental data to semantic data using the OpenX ontology [
Scene-based Query Generation In the first phase, basic facts are available from the ontology and
represented as a list of predicates that characterize the trafic scene. We employ a dual-agent system to
complete the scene interpretation and action proposal. This system follows an instruction-following
cooperation model, which has been shown to advance the progression of productive communications
and achieve meaningful solutions [
(, ) = ⟨ → , ← ⟩loop
(2)
We prompt the instructor agent to describe the scene based on the available predicates and the feedback
from the solver if available. The assistant agent is then instructed to interpret this scene and propose a
driving action. To reduce communicative hallucination [
Rule Formalization and Search Working closely with legal experts, we gather German
trafifc rules from written legislation, legal precedents, and court decisions. We then analyze these rules and
convert them into a normal form structure [
1. Zero-Informed: This method focuses on unary predicates, specifically designed for class hierarchical reasoning, characterized by a invariable and consistent query structure. It generates queries aimed at searching for instances that belong to the class required by the rule evaluation. 2. Rule-Informed: This method generates queries based on the context of the evaluated rule, which can be answered by the ontology reasoning. For example, given the context of the rule that states a solid lane marking on the left that connects to the ego lane and requires generating a query about the predicate LeftConnectedTo(X, Y), this method would incorporate the information about X as the ego lane type and Y as the solid line type into the query construction. This method limits the range of possible answers derived from the ontology. 3. Ontology-Informed: This method targets queries that can not directly inferred from the ontology.
It incorporates additional ontological information, including comments about the predicate and available predicates, to construct the query.
These three query generation methods ofer increased context and flexibility but reduced semantic correctness. In our work, most queries use the first two methods, while only two queries use the third (see examples in Appendix B.1).
Logical Reasoning We use Prolog [
Providing explicit rules as extended context in prompts is a common method for managing the behavior
of language models. In contrast, our method restricts its output by employing ontology-based fact
retrieval to evaluate rules via query generation. While agent-based language models [
four classes based on primary actions, where each class encompasses various scenarios. These include diferences in the number, size, type, location, and speed of vehicles, as well as road markings, trafic signs, weather conditions, and congestion levels, all of which may influence driving actions. Our data generation follows principles of flexibility, extensibility, and scalability through random sampling for variables considering physical and rule constraints, supplemented by thorough manual review. While our dataset, programmatically configured, may not fully reflect real-world driving complexities, it aims to test our hypotheses and serve as an example for collaboration between legal experts and computer scientists in dataset creation.
Language models primarily trained on general natural language corpora [62], have limited understanding of less common key-value data structures. While targeted instructions can help, the results may not always be reliable. To maximize the reasoning potential of our baseline models, we implement a rule-based approach that programmatically generates narratives for each scenario from key-value pairs, enabling language models to derive rule-compliant actions from these textual descriptions. We use Few-shot-CoT [63, 64] as our baseline method, enabling complex reasoning by prompting detailed intermediate reasoning steps. To guide decision-making, we provide four representative examples that follow diferent reasoning paths involving trafic rules. For a fair comparison of reasoning abilities, we exclude the rule search part by specifying applicable trafic rules as natural language text in the baseline models and as logical forms in our method. We use GPT-3.5-Turbo and GPT-4o as language models for both methods. To evaluate the accuracy of derived actions, we use precision , recall , and the 1 score as our metrics in each scenario. A True Positive is counted when both the predicate and its argument are correctly predicted. Subsequently, we calculate the average and standard deviation for each metric across all classes and methods.
As presented in Table 1, our method, NeSy-LAD (Neuro-Symbolic Legal Guidely Automated Decisionmaking System), outperforms the Few-shot-CoT baseline models across GPT-3.5 and GPT-4o, with significant gains in precision, recall, and F1 Score. The NeSy-LAD with GPT-4o achieved the best performance, exhibiting a 13.75% increase in precision, a 7.25% increase in recall, and a 10.54% increase in F1 score compared to the Few-shot-CoT with GPT-4o. Remarkably, even when utilizing GPT-3.5, NeSy-LAD still outperforms the Few-shot-CoT with GPT-4o by 0.75%, which indicates that integrating ontology-based fact retrieval with a symbolic solver ofers a significant advantage over the approach directly prompting rules for the decision-making. GPT-4o generally outperforms GPT-3.5 across both methods. Notably, in the Few-shot-CoT approach, GPT-4o demonstrates a recall that is 12.5% higher than the GPT-3.5 implementation.
To explore the performance of these two methods across various scenario categories, we plotted the average F1 scores for both in Figure 4 (left). Our method demonstrates higher F1 scores in the “Wait”, “Pass”, and “MakeUTurn” classes, with the exception of the “Overtake” class. The Few-shot-CoT approach tends to favor the “Overtake” action to escape these challenging scenarios, whereas our method relies more on the rule evaluation. Particularly, in the “Wait” and “Pass” classes, which involve a higher number of rules and predicates (Figure 4, left), our method achieved very high F1 scores. including both semantic and syntactic inaccuracies. Semantic errors are more common, largely due to the misuse of the rule context and available predicates from the ontology for query generation. The errors in the Few-shot-CoT approach arise from several sources: omitted trafic rules, inconsistencies between the reasoning process and conclusions, and an inability to accurately capture the semantic meaning of the rules. For example, the action ”Wait” is mostly regulated implicitly by the prohibition of ”Overtake” or ”MakeUTurn” in trafic rules, which Few-shot-CoT may not capture. With respect to interpretable and traceable reasoning, our approach provides detailed insights into evaluated rules and generated queries for unary and binary predicates (Figure 4, right), ofering a more reliable and trustworthy process than Few-shot CoT.
As demonstrated in our experiments, compared to providing rules as the context to language models, our method, which segments the decision process into three agent-driven phases and delegates the ifnal reasoning to a symbolic logical reasoner, exhibits significant advantages in rule-compliant decison making accuracy, transparency, and interpretability. Despite these achievements, we acknowledge certain limitations in our experiment and approach. First, we tested only a limited set of rules and predicates, which may not fully represent the language model’s query generation capabilities. To scale the approach, future work should explore more eficient mechanisms that utilize various contexts or consider fine-tuning the model for SPARQL query generation. Secondly, our system is heavily dependent on formalized rules in an executable format. However, we argue that legislation regarding automated agents should address both implementability for systems and interpretability for humans, which calls for collaboration between computer scientists and legal experts. This lays the groundwork for the safe and lawful deployment of agents in real-world applications. Our LAD system integrates trafic rules explicitly into a symbolic solver, providing an interpretable and traceable decision-making process for humans. This setup allows for flexible and rapid rule updates as regulations evolve. Additionally, our system’s modular design allows diferent modules to be replaced with varying techniques. For instance, the ontology query part can be replaced with knowledge graph embeddings, and the symbolic solver can be substituted with a neural-based reasoner. Though our system was originally designed for decision-making in autonomous driving, it can be adapted to other domains requiring scene recognition and rule compliance. In conclusion, our approach provides a scalable and adaptable framework that can serve as a foundational solution for a wide range of applications, enabling reliable and interpretable rule-compliant decision making in diverse contexts.
In this paper, we present a framework that combines language model agents with a symbolic logical reasoner for rule-compliant decision-making in autonomous driving. Our approach regulates automated agents with formalized rules, providing an adaptable solution for safer and more interpretable automated decision-making. Despite its efectiveness in trafic scenarios, our system has limitations when dealing with complex rules and probabilistic reasoning. Future work should explore more implementable rule formats and develop more scalable query methods.
During the preparation of this work, the author(s) used GPT-4o in order to: Grammar and spelling check. After using this tool, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content. et al., Metagpt: Meta programming for multi-agent collaborative framework, arXiv preprint arXiv:2308.00352 (2023). [61] Z. Ma, Y. Mei, Z. Su, Understanding the benefits and challenges of using large language model-based conversational agents for mental well-being support, in: AMIA Annual Symposium Proceedings, volume 2023, American Medical Informatics Association, 2023, p. 1105. [62] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, S. Anadkat, et al., Gpt-4 technical report, arXiv preprint arXiv:2303.08774 (2023). [63] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, L. Zettlemoyer, Rethinking the role of demonstrations: What makes in-context learning work?, 2022. arXiv:2202.12837. [64] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., Chain-of-thought prompting elicits reasoning in large language models, Advances in neural information processing systems 35 (2022) 24824–24837.
◁ Mapping data into ABox ◁ Build knowledge base with rule sets ready for query
◁ Retrieve basic available facts ◁ Iterate N trials for LLM primary action proposal ◁ Output candidate action and scene features
◁ Search for all related rules ◁ Identify rule predicates absent from facts
◁ Generate queries for absent predicates ◁ Retrieve new facts with ontology reasoning ◁ Verify action consistency via backwards chaining
We propose the Rule-Compliant Decision-Making Algorithm, which combines multiple language models with a symbolic solver to derive rule-compliant actions. As presented in Algorithm 1, it takes as input a TBox , a scene represented by key-value pairs, a set of formalized rules together with their corresponding logical representations , and possible actions . It outputs primary and secondary actions that comply with the corresponding rules and . The process begins by mapping data to an ABox . Together with the formalized rules, this forms a knowledge base , which is then ready for querying. Basic facts about the current scene are then extracted from this knowledge base. In the first loop of N trials, the language model agents collaboratively proposes a primary action ˆ and identifies relevant scene features ˆ based on the blockage of the front vehicle. It then searches for all rules related to the proposed action and the identified features. In the second loop, for each relevant rule ˆ, the algorithm evaluates which rule predicates are not currently supported by the available facts. For each absent predicate, the language model generates queries, which are used to retrieve new facts from the knowledge base through ontology reasoning. Upon identifying a candidate primary action ˆ, the symbolic solver verifies the action’s consistency with the rules through backward chaining. If the action is found to be compliant, the loop terminates, marking the action as the primary compliant action. If not, the process iterates, updating the prompts for the language model agents to refine the action proposal. After determining the primary action, the algorithm employs the symbolic solver to derive secondary actions that are compliant with the rules. Our algorithm reduces the workload for the symbolic solver by suggesting candidate actions, pinpointing the most relevant rules, and extracting the necessary facts though context-based query generation. This approach significantly narrows the search space for rule evaluation, while ofering more interpretable and traceable results compared to purely language-based models.
We begin by collecting trafic regulations from various sources, then analyze these rules and convert them into a normal form structure with detailed descriptions across five dimensions. At last, we employ large language models as tools to help formalize the rules in predicate logic. In total, we’ve formalized 25 rules, with examples shown as follows. { "id": 5, "R": "Motorcycle(X), Overtake(X)", "T": "None", "S": "Driver(J)", "E": "None", "Q": "KeepSafeLateralDistance(1.50)", "condition": "driver overtaking a motorcycle", "consequence": "maintain a minimum lateral distance of 1.5 meters" }, { }, { "id": 11, "R": "TrafficLight(X), Red(Y), hasColor(X, Y), Vehicle(V), inFrontOf(V, J)", "T": "None", "S": "Driver(J)", "E": "None", "Q": "Overtake(V), !Pass(V), !MakeUTurn(V)", "condition": "driver approaching a red traffic Light", "consequence": "must not overtake, pass or make a U-turn" "id": 18, "R": "EgoLane(G), SolidWhiteLine(Y), LeftConnectedTo(G, Y), Vehicle(X), Inoperative(X), OnComingLane(Z),
InFrontof(X, J), !hasOncomingVehicle(J, W), Vehicle(W), block(X, G)", "T": "None", "S": "Driver(J)", "E": "None", "Q": "Overtake(X), Cross(Y), LaneChangeTo(Z)", "condition": "obstacle or stationary vehicle on road partially blocking the ego lane with solid white line, not predictable when cleared", "consequence": "permitted to cross the solid white line and use oncoming lane for overtaking, if safe and no oncoming traffic"
Zero-informed method. This method rarely uses any context for query generation. It primarily focuses on the evaluation of unary predicates, involving class hierarchical reasoning, which generated less diverse query structures. For example (see Figure 5), when evaluating the predicate AdverseWeather(Y) within a rule, this method generates a SPARQL query to the ontology, which retrieves the answer w0 because the facts indicate that the weather is snow, which is defined as adverse weather in the ontology.
LM Generated SPARQL Queries AdverseWeather(Y) PREFIX : <http://www.semanticweb.org/> SELECT ?y WHERE {?y a/rdfs:subClassOf* :AdverseWeather .}
Rule-informed method. This method generates queries based on the context of the rules being evaluated, which can be answered by ontology reasoning, as illustrated in Figure 6. Given a rule under evaluation, the language model uses predicates such as Vehicle(X), Vehicle(Y), EgoVehicle(X), OnComingLane(Z), locatedOn(Y, Z), and InFrontOf(Y, X) within the rule to construct a detailed query for the InFrontOf predicate. It reduces the number of instances retrieved from the ontology.