Combining Large Language Models (LLMs) with graph-based retrieval augmented generation (GraphRAG) in multi-agent Artificial Intelligence (AI) systems promises new levels of process automation, even in highly restricted and regulated domains such as financial audits. Since trustworthiness is essential, we introduce an architecture designed with it in mind: Every agent is either inherently explainable or its decision-making mechanism is rendered transparent through post-hoc Explainable AI techniques. Procedural knowledge, agent outcomes, and their explanations are represented as nodes in a knowledge graph accessible via GraphRAG, while LLMs are confined as semantic translators, bridging graph and natural-language representations. However, when user prompts involve multiple agent subgraphs, their individual agent attribution is still opaque. We introduce an occlusion-based agent importance metric that quantifies the relative attribution of each serialized agent subgraph. Our evaluation demonstrates that the quantification of agent importance is feasible, while the presence of systematic agent interactions or narrative context efects requires further investigation.
Related Work. While LLMs have been used for several downstream tasks such as text generation [
Can you explain your inference?
The company explanation highlights a strong influence of […]. In contrast, the sector explanation places greater emphasis on […].
Where does this information originate?
Agent 1 is primarily responsible for the explanation, whereas […].
chat
III prompt + <<ssuubbggrraapphh aaggeenntt 12>> verbalization baseline Problem. Despite the decision-making mechanism behind each agent has become tractable by XAI techniques, the LLM result does not disclose the attribution of agents to the verbalization (Figure 1, left). Consequently, an auditor in our running example might still be confronted with over- or under-amplified risks, undermining the organization of the entire examination strategy.
Solution. We quantify agent importance in three steps (Figure 1, right): (i) We extract the relevant subgraph sequence from the knowledge graph. (ii) Then, we generate verbalizations: a baseline representing the entire subgraph sequence and separate variants for each subgraph occlusion. (iii) Finally, we quantify the agent importance by semantic similarity metrics. The application of our approach enables auditors to understand the origin of information and evidence in multi-agent system outcomes. Contributions. Our core contributions are twofold: First, we present a multi-agent GraphRAG architecture in a case study using open-source finance data sets that mimic the dynamics of annual audits (Section 2). Second, we formalize the proposed agent importance approach (Section 3). Research Questions. We pose two research questions: (R1) Does agent importance eligibly quantify the attribution of agent contexts retrieved from knowledge graphs to LLM verbalizations? And, (R2) does agent importance account for biases in the narrative task specification? The research questions are preliminary evaluated by an ablation study (Section 4).
To address the domain-specific requirements of financial auditing without disclosing proprietary information, we validate our architecture in a controlled scenario using open-source datasets1: Specifically, we model the influence of macroeconomic indicators on the relative annual return of individual company stocks and the average annual return of stocks within the same industry sector. The primary goal in this setting is to compare the explanatory factors for the company prediction with the reasons driving the sector prediction. We employ labeled property graphs for knowledge representations2.
The agents graph (Figure 2, left) encodes the procedural task knowledge and is responsible for the task execution. Among multiple properties, agents have an API address and Pydantic models for input and output validation3. Natural language queries are converted to an input model and trigger API execution. The outcome of which is verified using the output model. We distinguish between two types of agents (Figure 3): static agents, which rely on deterministic computations (write) or classical probabilistic methods (predict_company, predict_sector), and LLM agents, which generate natural language responses (compare). Figure 3 depicts their functionality in detail. 1The following data sets are merged wrt. the date and the company symbol: economic indicators: https://www.kaggle. com/datasets/alfredkondoro/u-s-economic-indicators-1974-2024?select=cpi_data.csv, stock prices: https://www.kaggle.com/ datasets/camnugent/sandp500?select=all_stocks_5yr.csv, stock information: https://www.kaggle.com/datasets/paytonfisher/ sp-500-companies-with-financial-information?select=financials.csv, all 6 th June 2025. 2https://memgraph.com/, 6th June 2025. 3https://docs.pydantic.dev/latest/, 6th June 2025.
properties
name description address input model output model
The results graph (Figure 2, right) contains nodes generated with Cypher queries [
We combine LLMs4 and knowledge graphs as follows: Each natural language prompt is mapped to a predefined Cypher query, which either triggers agents or retrieves subgraphs. Novel queries are generated for out-of-distribution requests. The obtained subgraphs serve as additional context for the initial prompt such that the generated verbalization contains only graph-encoded information.
In proposing an XAI method for multi-agent systems integrated with GraphRAG, we aim to address a core contradiction, which we term the explanation paradox: All agents in our architecture are either intrinsically explainable or accompanied by XAI techniques. Via GraphRAG, a LLM exclusively accesses this precomputed information. Still, recipients of a LLM verbalization are not yet aware of which agent contributed to the answer. In particular, when prompting a LLM to combine explanations, the user cannot comprehend the relevance of each incorporated agent. The agent importance method proposed in the next section is designed to resolve this paradox.
Agent importance quantifies the influence of a serialized subgraph of an agent on the generated LLM verbalization for a given task in a GraphRAG scenario. The agent importance approach approximates feature attributions inspired by Shapley Additive Explanations (SHAP) [16]. It systematically occludes serialized subgraphs within a GraphRAG multi-agent system, assuming a fixed verbalization task. Given a task, we retrieve and serialize the subgraphs of all addressed agents to generate a baseline verbalization. Next, we occlude one subgraph at a time and regenerate verbalizations. The intuitive idea is: The 4https://platform.openai.com/docs/models/gpt-4o-mini, 28th May 2025. more similar the occluded output is to the baseline, the less the agent contributes to the verbalization. Attribution values are estimated by semantic similarity [17]. We compute the cosine similarity between the embeddings of the occluded and baseline verbalizations; agent importance is defined as the inverse of the calculated similarity.
Definition. Let be a subgraph serialization of a knowledge graph, = (1, . . . , , . . . , ) ∈ be an ordered sequence of serialized subgraphs, ∈ and ∈ denote prompts and verbalizations, and : × → be a LLM5, generating a verbalization given a prompt and a subgraph sequence. Let ℎ : × → R represent the cosine distance between subgraph sequence embeddings 6. Let ∖ := (1, . . . , −1 , , +1, . . . , ) with || = 0 denote the occlusion of in – practically, a substitution of with an empty sequence . Finally, let ,() := ,(, ∖ ) for all ∈ (1, . . . , ) with ,(, ∖ ) = 1 − ℎ(, ∖ ) measure the attribution of each subgraph in the subgraph sequence given a model and prompt . We assume that subgraph serializations are agent outcomes and thus can be mapped to their origin. Example. Suppose an AI literature search uses three agents: one for researchers, one for publications, and one for scientific areas, with results encoded as a knowledge graph, serialized as = ( 1, 2, 3). subgraph label
node 1 2 3 :Researcher :Publication :Area ’Ina Marie’ ’XAI in Multi-Agent Systems for Audit: Why Our Method is Important’ ’Financial Audits’ Suppose the following user prompt for a GraphRAG system, which accesses the (sequence of) subgraph serializations: =’Summarize the publications of Ina Marie in the field of financial audits’. context
∖1 ∖2 ∖3 ’Ina Marie has published several influential works, including XAI in Multi-Agent Systems for Audit: Why Our Method is Important, which highlights the importance of explainable AI in enhancing the eficiency and efectiveness of financial audits.’ ’The publication XAI in Multi-Agent Systems for Audit: Why Our Method is Important explores the role of explainable AI in improving the accuracy and eficiency of financial audits.’ ’Ina Marie has made significant contributions to the field of financial audits through her research on innovative approaches to auditing practices.’ ’Ina Marie’s work focuses on the applications of explainable AI in the auditing process.’ ,(, ∖ )
0.0 0.0482 The publication node subgraph frames the most important context for the verbalization.
The original SHAP framework is characterized by theoretical properties – local accuracy, faithfulness, missingness, and consistency – which have been formally established over the course of its development [18, 16, 19]. Furthermore, its computational feasibility has been systematically analyzed [20]. Despite proposing our method in a mathematically sound style, we leave a formal assessment for future work, yet highlight the importance of which. In contrast, the experiments of the subsequent section provide quantitative evidence through an ablation study that evaluates the importance of agent subgraph serializations in diverse tasks.
Suppose two single-agent – explanation interpretations – and three multi-agent tasks – explanation comparisons, either with or without narrative biases – in the domain of Section 2. We select the 100 companies with the highest relative return in 2016 and retrieve the subgraphs for the company and the corresponding sector explanation. We exploit the following evaluation strategy (Figure 4): (i) obtain occluded and baseline verbalizations, and (ii) estimate the agent importance (Definition 3). 5https://platform.openai.com/docs/models/gpt-4o-mini, 28th May 2025. 6https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2, 04th June 2025.
0.8 0.6
Figure 5 presents one subplot per task, with each containing a boxplot corresponding to an agent importance evaluation. Three observations can be drawn from the results: (i) Occluding agents presumed to be relevant increases their attribution scores. (ii) Narrative task context efects are only partially reflected, as the sector-level agent generally receives lower attribution. (iii) In multi-agent configurations, the ablation study (the additional occlusion of none or all subgraphs) demonstrates that agent importance scores are not strictly additive.
Within a narrow experimental scope, the research questions can be answered as follows: (R1) Agent importance eligibly quantifies the attribution of agent contexts retrieved from knowledge graphs. Context efects undermine desirable mathematical properties such as additivity. (R2) Agent importance tends to account for biases in the task specification. The experiments reveal narrative specificities of the domain such as an elevated importance of the term company compared to the term sector.
The integration of structured knowledge, XAI, and LLMs enables the automation of redundant tasks even in domains requiring substantial domain expertise, such as financial auditing. The use of multi-agent systems in combination with GraphRAG ofers a promising architecture – provided that each agent discloses its decision making. Determining the extent to which a LLM relies on the outputs of individual agents remains intractable. This challenge is encapsulated in what we term the explanation paradox: Even if every system component is individually explainable, the attribution of each component remains opaque. The agent importance method addresses this gap. It estimates the relative contribution of each agent by computing the inverse of the semantic similarity between the LLM’s baseline verbalization and the verbalization generated after occluding the respective agent.
Main Findings. Three findings emerge from our contribution: (i) Quantifying agent importance
addresses a critical gap on the way towards trustworthy multi-agent systems in high-stake domains; yet
is feasible as evidenced by our preliminary results. (ii) While our method is mathematically grounded, it
remains theoretically incomplete. Assumptions derived from supervised learning models, e.g., the ones
from SHAP [16], are not seamlessly transferable to generative models. (iii) Our findings hint at two key
challenges: (a) Narrative context efects introduced by task prompts may bias the relevance estimates.
And, (b) correlations between agent subgraphs might confound the measured importance values.
Related Results. Our main findings can be situated into an area of methods, which aims to overcome
semantic limitations [21] or hallucinations [22] in LLMs. What has started as a systematic combination
of supplementary information with LLMs [23], has transitioned into a structured representation of
domain knowledge with researchers questioning their attribution [
Ethical Considerations. There are no specific ethical concerns to declare. Our overarching goal is to develop trustworthy multi-agent systems. We believe transparency in the decision-making process is essential.
Declaration on Generative AI. During the preparation of this work, we used gpt-4o and Grammarly for spell-checking and grammar correction, and gpt-4o-mini for experimental purposes7.
Acknowledgments. This article is part of the project BayVFP Data Tales (# DIK-2407-00007 // DIK0660/01). 7gpt-4o: https://platform.openai.com/docs/models/gpt-4o, Grammarly: https://www.grammarly.com/, gpt-4o-mini: https: //platform.openai.com/docs/models/gpt-4o-mini, all 28th May 2025. 2025. arXiv:2506.00783. [16] S. M. Lundberg, S. Lee, A Unified Approach to Interpreting Model Predictions, in: I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 2017, pp. 4765–4774. URL: https: //proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html. [17] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi, BERTScore: Evaluating Text Generation with BERT, in: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020. URL: https://openreview.net/forum?id=SkeHuCVFDr. [18] H. P. Young, Monotonic solutions of cooperative games, International Journal of Game Theory 14 (1985) 65–72. doi:10.1007/BF01769885. [19] L. Heidrich, E. Slany, S. Scheele, U. Schmid, FairCaipi: A Combination of Explanatory Interactive and Fair Machine Learning for Human and Machine Bias Reduction, Machine Learning and Knowledge Extraction 5 (2023) 1519–1538. doi:10.3390/make5040076. [20] M. Arenas, P. Barcelo, L. Bertossi, M. Monet, On the Complexity of SHAP-Score-Based Explanations: Tractability via Knowledge Compilation and Non-Approximability Results, Journal of Machine Learning Research 24 (2023) 1–58. URL: http://jmlr.org/papers/v24/21-0389.html. [21] K. J. Hammond, D. B. Leake, Large Language Models Need Symbolic AI, in: A. S. d’Avila Garcez, T. R. Besold, M. Gori, E. Jiménez-Ruiz (Eds.), Proceedings of the 17th International Workshop on Neural-Symbolic Learning and Reasoning, La Certosa di Pontignano, Siena, Italy, July 35, 2023, volume 3432 of CEUR Workshop Proceedings, CEUR-WS.org, 2023, pp. 204–209. URL: https://ceur-ws.org/Vol-3432/paper17.pdf. [22] L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, T. Liu, A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023). doi:10.48550/ARXIV.2311.05232. [23] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. tau Yih, T. Rocktäschel, S. Riedel, D. Kiela, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2021. arXiv:2005.11401. [24] X. Yue, B. Wang, Z. Chen, K. Zhang, Y. Su, H. Sun, Automatic Evaluation of Attribution by Large Language Models, in: H. Bouamor, J. Pino, K. Bali (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics, Singapore, 2023, pp. 4615–4635. doi:10.18653/v1/2023.findings-emnlp.307. [25] A. Abolghasemi, L. Azzopardi, S. H. Hashemi, M. de Rijke, S. Verberne, Evaluation of Attribution
Bias in Retrieval-Augmented Large Language Models, 2024. arXiv:2410.12380. [26] J. Gao, X. Zou, Y. Ai, D. Li, Y. Niu, B. Qi, J. Liu, Graph Counselor: Adaptive Graph Exploration via Multi-Agent Synergy to Enhance LLM Reasoning, 2025. arXiv:2506.03939.