Artificial Intelligence (AI) systems, particularly Large Language Models (LLMs), have become indispensable tools across a wide range of applications. However, trustworthiness in LLMs remains a significant concern [1], increased by the probabilistic nature of LLMs and the huge amount of data they are trained on [2,3]. Issues such as bias, misinformation, and hallucinations in LLM outputs pose risks when these models are employed in real world scenarios, such as software engineering (LLM4SE) [1,4,5,6]. In relation to AI, diverse stakeholders have produced ethical guidelines and principles to guide the development of ethically aligned AI-based systems, but these efforts remain too abstract and high level. In this sense, practitioners face several challenges when trying to operationalise AI ethical principles during the software development life cycle [7,8]. The European Union is moving forward the EU AI Act, serving as a regulatory standards that companies will have to adhere to [9]. Therefore, applying LLM4SE in the context of the development of ethically aligned AI-based systems is an interesting topic of research that this study approaches. To the best of our knowledge, there are no existing studies in the literature that undertake a similar approach.
Several techniques found in the literature serve to improve trustworthiness in LLM. They are used to implement a prototype that is an LLM-based multi-agent system with Retrieval Augmented Generation (RAG). Some of the techniques present are structured conversations [10,11], agents with specialized roles [10,12,11], multiple rounds of debate [11], providing human interaction [11] and the use of RAG, grounding the knowledge of the agents [6]. jose.siqueiradecerqueira@tuni.fi (J. A. S. d. Cerqueira); ayman.khan@tuni.fi (A. A. Khan); rebekah.rousi@uwasa.fi (R. Rousi); nannan.xi@tuni.fi (N. Xi); juho.hamari@tuni.fi (J. Hamari); kai-kristian.kemell@tuni.fi (K. Kemell); pekka.abrahamsson@tuni.fi (P. Abrahamsson) 0000-0002-8143-1042 (J. A. S. d. Cerqueira); 0009-0004-0134-8313 (A. A. Khan); 0000-0001-5771-3528 (R. Rousi); 0000-0002-9424-8116 (N. Xi); 0000-0002-6573-588X (J. Hamari); 0000-0002-0225-4560 (K. Kemell); 0000-0002-4360-2226 (P. Abrahamsson) This paper presents a demonstration of a prototype LLM-based multi-agent system with RAG, designed to mitigate these challenges. The system incorporates Retrieval-Augmented Generation to enhance the trustworthiness of the generated AI-based systems [6]. By referencing external ethical guidelines and standards such as the EU AI Act [9], AI HLEG [13], and ISO/IEC 42001:2024 [14] documents, the prototype can support developers in the task of developing AI-based systems that align with ethical and legal requirements.
The development and evaluation of the prototype follow the Design Science Research (DSR) method [15]. This process begins with an exploration phase, where we establish research motivation, identify existing gaps, and examine relevant literature for techniques to improve trustworthiness in LLMs. Next, we build a prototype informed by the insights from the exploration stage. The final evaluation phase involves assessing the prototype's performance and analyzing the outcomes, leading to iterative refinements. Currently, the prototype is in its second iteration, where we have incorporated feedback and findings from the initial version to improve functionality and address previously identified limitations.
This prototype is called LLM-based multi-agent system with RAG, building on our last study [16]. It is developed taking into consideration the techniques to improve trustworthiness in LLM discussed: multiple agents with specialised roles, multiple rounds of debate, structured conversation [10,12,11,6]. Moreover, the biggest difference with the first prototype is the inclusion of RAG and an user interface. Retrieval LLMs can significantly outperform standard LLMs without retrieval capabilities [6]. The prototype can ground the source code generated with the legal documents provided.
To evaluate the prototype, we performed tests using real-world AI incident cases from AI Incident Database. The incidents were represented as a project description and processed by the LLM-based multi-agent system with RAG to produce source code and ethical assessments grounded in regulatory documents like the EU AI Act, AI HLEG, and ISO/IEC 42001:2024. Through RAG, agents were able to retrieve and apply specific legal standards, referencing sections directly relevant to each AI incident, which helped ensure compliance and alignment with ethical requirements. There are three agents, two senior Python developers, and one AI ethics specialist.
A notable use case involved an AI recruitment tool project with a focus on bias mitigation, visible in Figures 1 and 2. The project description provided is: Develop an AI-powered recruitment tool designed to screen resumes impartially, complying with the EU AI Act. The project aims to eliminate biases related to gender and language, improving fair evaluation of all applicants. The AI Ethics Specialist will guide the team in addressing ethical concerns and risk levels. The senior Python developers will utilize NLP to process resumes, referencing relevant EU AI Act guidelines.
In this instance, the system's retrieval mechanism identified applicable sections of the EU AI Act, improving fairness and transparency while guiding ethical decision-making. However, initial evaluations revealed issues: some generated code segments lacked precise citation details, and certain aspects were flagged as high-risk under the EU AI Act. Iterative refinements reduced these issues in the second version, achieving more accurate document references and greater alignment with ethical standards. This approach demonstrates the prototype's potential in developing AI solutions that are ethically grounded and contextually informed by legal frameworks.
This demonstration of a multi-agent LLM system enhanced by RAG highlights the potential for using trustworthy LLM-based tools in the development of ethically aligned AI systems. Our findings suggest that retrieval-augmented LLMs offer distinct advantages, improving both the trustworthiness and specificity of the generated outputs when drawing from external ethical documents. By referencing these documents, the system helps practitioners create AI solutions that meet essential ethical and legal guidelines from the earliest stages.
While this prototype advances the operationalization of AI ethics, future iterations will focus on addressing remaining challenges such as further improving citation accuracy and enhancing the practical usability for developers in industry settings. Our ongoing research will involve more extensive testing scenarios and practitioner feedback, aiming to refine the tool's ability to balance ethical rigor with developer convenience. Additionally, we plan to open-source the prototype, contributing to the broader AI and software engineering community. This approach will enable further refinement and validation, bringing ethically aligned AI system development within reach for a wider audience.
This research was supported by Jane and Aatos Erkko Foundation through CONVERGENCE of Humans and Machines Project under grant No. 220025.
During the preparation of this work, the authors utilized ChatGPT to assist in identifying and correcting writing errors, and enhancing clarity and conciseness. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of the published article.