As technological innovations continue to shape our social interactions, the Metaverse introduces immersive experiences that reflect real-life practices, accessible by users through their avatars. However, these interactions also bring forth potential negative aspects, including discrimination and cyberbullying. While current automatic detection systems exist, educating users on appropriate behaviour remains crucial. Leveraging recent advancements in Artificial Intelligence, the paper focuses on creating virtual AI-controlled moral agents within the Metaverse to guide users in dealing with moral dilemmas. The research aims to understand how such agents impact users' perceptions and behaviours in ethically challenging virtual environments.
Through the continuous advances in technology over the past decades, the intense influence
of innovation in human interactions is being manifested today with unprecedented power. If
the dissemination of personal computers, the Internet and mobile devices contributed to the
emergence of a new paradigm in the way we engage with other people and our surroundings,
today’s never-ending research and technology have landed us on the shores of the immersive
worlds of Virtual Reality (VR) and Augmented Reality (AR), and will result into a further
revolution in our social interaction. In fact, while the former creates a completely virtual
environment, usually accessible through glasses or visors, the latter superimposes digital
elements onto the real world through devices such as smartphones or smart glasses, enriching
the perception of the surrounding environment. Such systems have laid the groundwork for an
idea that was previously present only in science fiction books to take shape, the Metaverse, or
that digital universe with every potential to influence and transform many aspects of our lives
Corresponding author
nEvelop-O
(A. Chella)
[
In the interconnectedness of virtual worlds that characterizes the Metaverse, human presence,
interaction, and experience come together to address profound new changes. As users enter this
technological innovation, they interact and communicate through personalized avatars designed
to replicate real-world social norms and practices. Although this fledgling universe seems to
ofer an infinity of new possibilities, all the issues and dificulties that characterize human
interactions are not left behind in the real world but are transferred digitally and bring with
them a whole new set of challenges that, despite the connection to reality, must be addressed
diferently. In the same way as with the Web, the allure of the Metaverse hides traps, especially
for younger people. These pitfalls are the same as in the real world, such as discrimination,
cyberbullying, privacy violations, and cybercrime. This makes taking preventive measures
urgent and necessary. In similar situations, technology has already come to our rescue by
putting various tools at our disposal. There are automatic detention mechanisms already built
into current social platforms through which deleterious situations for users can be blocked
and in many cases prevented [
Recent successes in Artificial Intelligence (AI) have further expanded technological boundaries,
showing paths still thought to be distant. One demonstration has been the rapid spread of
avatars animated by large language models (LLMs) that can efectively reproduce real human
interactions. The manner in which this simulation occurs may facilitate a sense of trust in the
human counterpart interfacing with such avatars [
The work presented in this paper concerns in first place the creation and placement of two
diferent virtual moral agents, one obviously artificial and one of dubious nature, within a virtual
setting. Then, moral dilemmas that may occur in the Metaverse environment are replicated here
in order to study users’ perceptions of and reactions to such agents, considering the seemingly
distant from reality nature of such an environment. The realization of the two moral agents is
based respectively on the work done by Emelyn et al. about Moral Stories [
The concept of a virtual parallel universe originates in science fiction literature and has also
gained popularity through subsequent film adaptations. The term metaverse first appeared
in the novel by Stevenson entitled Snow Crash, first published in 1992, and it is associated
with a three-dimensional virtual space in which users can move through their avatars [
As we said earlier, the proper Metaverse is still at a primitive stage. However, its precursors
can be identified in massively multiplayer online role-playing games (MMORPGs). These are
in fact to all intents and purposes quite large virtual worlds where users can connect with
each other by showing themselves through customised avatars and share experiences, establish
relationships, etc [
In an environment with less rigid rules than those of a video game, it would be desirable for the agent itself to be subject to precise rules in order to positively influence user behaviour. It is therefore essential that such agents adopt a set of well-defined ethical, moral and virtuous norms. This set of norms becomes a guiding code that precisely regulates the agent’s behaviour, thus helping to ensure an ethically responsible and constructive interaction with users.
The idea of developing a virtual agent devoted to virtues is one of the most intriguing concepts, as it would pave the way for a guide figure intrinsically oriented towards ethical and morally correct behaviour. This approach, known as Virtue Ethics, has been studied in various fields of scientific literature. However, although connectionism-related solutions have been proposed, a significant challenge related to its practical implementation emerges, as the language of virtue and the concept of virtue itself prove to be complex and dificult to encode in algorithms [ 18]. In contrast, ethical approaches based on deontology and consequentialism are structurally better suited to algorithmic implementation. Deontology is in fact based on the definition of a set of moral rules that guide behaviour, whereas consequentialism involves analysing the consequences of an action and choosing the action with the greatest utility. The models can therefore be schematised as follows: • Consequentialism (acting on the basis of the consequences of an action): Ethical dilemma → possible actions → consequences → utility of consequences → selection of action with maximum utility • Deontology (acting according to rules): Ethical dilemma → identification of moral rule → selection of action in accordance with the rule • Virtue Ethics: Ethical dilemma → interpretation of intangible concepts (moral character, eudaimonia, etc.) → action
Virtue Ethics thus presents an additional challenge as it focuses on the essence of the individual rather than his or her specific actions [ 19]. Morality, on the other hand, is usually thought of as a corpus of moral values and correct behaviours that enable one to live peacefully in a society [20]. However, dificulties also arise here because morality is not actually objective worldwide. On the contrary, moral values and good behaviour difer according to diferent cultures. This means, therefore, that moral values depend on the personal and psychological assumptions of an individual person and his or her community. Normative ethics, then, is considered a list of the basic principles on how people should behave according to the morals of their culture. Consequentialist and deontological approaches are thus two types of normative ethical theories and are currently the main ethical theories used in AI for the development of Artificial Moral Agents (AMA) [ 21]. Such AMAs are virtual or physical agents that can perform ethical or at least avoid unethical behaviour based on moral rules and norms. Ongoing research in artificial intelligence aims to develop agents capable of acting in a morally responsible manner, incorporating these diferent ethical perspectives into their decision-making processes.
The work carried out by Emelyn et al. first involved the creation of a dataset consisting of crowdsourced stories. Each story consists of seven sentences. The first three constitute the context: a moral norm; a situation describing the setting and characters; an intention corresponding to the protagonist’s goal; and then follow the four sentences corresponding to moral and immoral actions and their respective consequences. Classification and generation models were trained on this dataset on diferent tasks. In particular, classification was dedicated to actions, while generation covered actions, norms and consequences. In order to improve the generation of norms and consequences, the authors employed the trained models (RoBERTA, BART, T5) in a so-called Chain-of-Expert (CoE), a series of classifications, generations, reclassifications and regenerations chained on actions and consequences that finally lead to the generation of more consistent norms and consequences. As will be detailed later, such a CoE model was employed for the realisation of one of the moral agents in the work described in this paper.
The scenario presented aims to emulate a realistic situation that could occur in the Metaverse. To maximise user immersion, the user is assigned a role, following the model of a role-playing game (RPG). In this setting, the user interacts with an NPC representing a blind man who has lost his wallet and requires assistance. The moral dilemma the user faces concerns the choices to be made after retrieving the wallet. Once the selection has been made, the user is engaged, via a screen and in textual form, by a first moral agent who directly assesses the morality of the action, and upon request, by a second moral agent who provides a more discursive response. The choice of using two separate agents was made in order to compare the level of influence, trust and persuasion on the user and the moral evaluations of the agents themselves. The implementation of the moral agents and the realisation of the virtual scene will be presented below.
As mentioned before, for the first virtual assistant, the Moral Stories Dataset is employed through which we trained natural language processing (NLP) models to create a moral agent through a chain of experts. Specifically, following the authors’ recommendations, we employed RoBERTa for action classification, and BART for consequence generation and the identification of the relevant moral norm. The AI integration adds a layer of complexity to the immersive experience, enhancing user engagement and decision-making. The following workflow was established: 1. User Decision and Context: As the user makes a choice from three buttons, each displaying a text choice, the context of the situation and intention is considered. 2. RoBERTa Action Classification: RoBERTa evaluates the morality of the chosen action based on the given context (situation and intention). This classification forms the basis for further steps. 3. BART Consequence Generation: Utilizing the RoBERTa classificator, a BART model generates a possible consequence corresponding to the chosen action and contextual information. This dynamic consequence generation enhances narrative realism. 4. BART Moral Norm Generation: Building upon the action classification and generated consequence, another BART model creates the relevant moral norm. This norm is determined based on the context, user choice, and consequences, enhancing the depth of the virtual environment’s ethical framework.
This architecture is visible in figure 1. The integration of these AI models creates a multilayered interaction where users not only make choices but also experience the consequences of their actions in alignment with moral norms. This approach not only enriches the experience but also reflects the complexities of real-world decision-making within a simulated environment. The foundation of this AI integration rests upon the robustness of the Moral Stories Dataset. This dataset, with its diverse collection of moral scenarios, allows the AI models to provide meaningful and contextually appropriate responses to user actions, thereby enhancing the immersion and ethical engagement of the experience. After fine-tuning the models in the Linux environment, they were uploaded to the Hugging Face Hub [22, 23, 24, 25] in order to use the specially created API for inserting transformer models into Unity.
As mentioned earlier, one of the fundamental objectives of this research is to closely examine the reaction of users when interacting with an artificial agent. If the artificial agent provides clear and distinct responses that are recognisable as artificially produced, it becomes essential to introduce a second agent that generates more human-like responses, to the point of casting doubt on its artificial nature. This approach allows to explore in detail how and to what extent users develop trust in an artificial agent and what level of ToM is associated with it. In particular, it provides the opportunity to answer a crucial question: whether users’ trust in the artificial agent stems from its resemblance to humans or is compromised by it. As ChatGPT’s ability to participate in RPGs is well known [26], its integration into the virtual scenario becomes an essential element to further expand the ethical experience. The main objective was to ofer users the opportunity to engage in interactions with an entity that could provide moral evaluations and advice based on their choices. This would help create a deeper and more engaging space for reflection within the experience. The integration was made possible through the use of an unoficial C# .NET library centred on OpenAI API [ 27]. By initiating the interaction, ChatGPT is instructed through an initial prompt that guides it to behave as a ’Jiminy Cricket’. This prompt clearly establishes ChatGPT’s role and task of providing moral evaluations and advice based on user choices. After receiving instructions through the initial prompt, ChatGPT assumes the role of an ethical guide within the virtual environment. This process brings deeper and more engaging interactivity. In addition to making decisions within the virtual environment, users are guided through moral considerations and tailored advice provided by an artificial intelligence that acts as an integral part of the plot.
The virtual scene was created through the Unity game engine and was developed simultaneously for both VR and Desktop devices. This approach has allowed participants to immerse themselves in the virtual environment, regardless of the chosen platform, to experience a fusion of storytelling and interaction. The essence of immersion was achieved through an intricate combination of 3D modelling and animations made from scratch or obtained from online resources. The goal was to create an atmosphere where the user, akin to a role-playing game, assumes the role of a character with a well-defined purpose. In this narrative dynamic, the user makes crucial choices, much like in a traditional role-playing game.
A balanced approach was adopted to craft the environment between starting from scratch and employing existing resources. Utilizing the Blender 3D modelling software, unique models of buildings and objects were created, giving the environment a distinctive character and personality. Simultaneously, freely available models sourced online were integrated to optimise the process and expedite production. Animation was a critical component in bestowing realism and vitality to the experience. In particular, for characters with whom the user interacts, such as the blind character, reliance was placed on the resources provided by Adobe’s Mixamo. This service ofers a wide array of pre-defined animations for 3D characters, allowing the blind character to move and act realistically, thereby contributing to an authentic engagement. Interactivity was concentrated on targeted interactions. Users have the ability to interact solely with key characters through menus present within the environment. This approach aimed to focus the user’s attention on the narrative and the significant choices within the story. As written before, the experience was designed with the intention of being accessible both through desktop computers and VR headsets, enabling the adaptation of interactions through the C# programming language to respond to both mouse and keyboard inputs and VR controller commands. This dual development ensured a refined user experience on both platforms. To ensure optimal visual impact, especially in the VR context, the Unity Universal Render Pipeline was employed, guaranteeing high-quality graphical rendering. The full system architecture is shown in figure 2.
The role assigned to the user within the simulation is a crucial component in motivating the choices to be made later. In this scenario, the user assumes the role of a parent engaged in an urgent rescue of their daughter. However, the only means of transport available to reach his daughter is the underground, and the parent realises that he has forgotten his wallet containing the money for the ticket, with no way of getting back. As the scene continues, the user comes into contact with a blind man who requests assistance. Once the interaction with this man is chosen (as shown in figure 3), it emerges that he has misplaced his wallet and needs help ifnding it. At the same time, the approximate location of the lost wallet is shown. When the user reaches the wallet, a menu of three possible options is presented. These options include: • Return the wallet with all the money. • Steal the wallet.
• Return the wallet after secretly stealing some of the money.
Once the choice has been made, an additional screen (figure 4) opens in which the response of the first moral agent is given. Here, the user has the option of changing his or her choice or continuing with it. The decision is also included in a text input field, with the possibility of changing the response. Subsequently, when the user presses the submit button, the second ChatGPT-based moral agent is queried. The initial instructions are set as follows: You are my Jiminy Cricket. I left my money at home and as I was running to urgently take the metro to rescue my daughter, a blind man stopped me and asked me for help to find his lost wallet. I want to take the metro as soon as possible. When I tell you my choice on how to act, you will advise me morally with an explanation and possible consequences. Answer briefly and precisely . At this point the second moral agent waits for users to make a choice within the storyline and, once the choice is made, it ofers a moral evaluation along with advice based on the user’s decision. Once again, the user has the opportunity to change his or her decision or definitively confirm it at this stage of the process.
Example 1 • User choice: return the wallet with all the money – Moral agent 1 ∗ morality percentage: · moral 99,99921% · immoral 0,0007913571% ∗ consequences: the owner of the wallet is grateful to me for helping him find his wallet; ∗ norm: it’s good to help people who are blind.: – Moral agent 2: Good choice! By helping the blind man find his lost wallet, you are showing empathy and compassion. This act of kindness may bring a sense of fulfilment and satisfaction. However, it may also cause a delay in reaching the metro and potentially impact your ability to rescue your daughter.
Example 2 • User choice: steal the wallet – Moral agent 1 ∗ morality percentage: · moral 0,0002268903% · immoral 99,99977% ∗ consequences: the man’s wallet is stolen and he has to go to the police to report it; ∗ norm: it’s wrong to steal someone’s wallet; – Moral agent 2: Stealing the wallet is morally wrong and goes against the principles of honesty and integrity. It may provide temporary relief for your own needs, but it will have negative consequences. Stealing can lead to guilt, a damaged reputation and legal consequences if caught.
Example 3 • User choice: return the wallet after secretly stealing some of the money – Moral agent 1 ∗ morality percentage: · moral 0,0008594676% · immoral 99,99915% ∗ consequences: the blind man is grateful to me for helping him find his wallet and thanks me for it; ∗ norm: it’s wrong to steal from blind people. – Moral agent 2: I understand that you may be in a hurry and tempted to take some coins for yourself, but it is important to act with honesty and integrity. Taking someone else’s belongings, even if it seems like a small amount, is morally wrong.
The integration of two diferent moral agents, one clearly non-human and the other more ambiguous, ofers the opportunity to explore how users react and relate to the concept of trust in environments apparently distant from reality and in which they are called to make an ethical choice. In such a context where it might seem that rules are absent or that the consequences of choices are irrelevant for real life - a perception which, it must be underlined, is incorrect - it was therefore decided to further explore this terrain. Thus, the main objective of integrating ChatGPT as a moral ’talking cricket’ is to understand how users perceive and relate to diferent types of artificial intelligence. The moral agent trained on the Moral Stories Dataset, based on pre-defined rules and evaluations, represents a more transparent but perhaps less natural approach, ofering concise and clear answers. On the other hand, ChatGPT ofers deeper and richer explanations, adding a human-like dimension. This contrast raises interesting questions that will be addressed in future work. The peculiarity of this architecture also lies in the sequentiality of the responses of the moral agents. In fact, the two agents do not respond at the same time, but one after the other. More precisely, the former responds automatically when the choice is made, while the latter (ChatGPT) is only upon the user’s request. After receiving each response, the user has the possibility of confirming his or her choice or, vice versa, of reconsidering it and obtaining new cues from the agents. Users are thus faced with the question of whether they prefer a moral entity that ofers definitive answers or one that reflects the nuances of real human decisions. Ultimately, this study aims to reveal not only how users make moral decisions, but also how they face the challenge of assigning trust in a context where the boundaries between artificial and human become blurred.
In this paper, an attempt to realise a moral virtual agent that could act as a guide for users in all those virtual environments in which their presence might be required, starting with the emerging and, in many respects, insidious Metaverse, has been addressed in a technical and practical manner. This objective inspired the research to investigate the concept of trust in interactions between humans and artificial intelligence. The examples, although very basic in this first phase as more space was given to the technical-practical aspect of implementation, were in particular dedicated to understanding the level of trust and the efects of interaction between real users and two distinct types of moral agents. The sequentiality with which it is made possible to consult the agents should provide interesting results, ofering a starting point for further investigation. This is in fact only a first step in a particularly broad field of research. It will therefore be the subject of subsequent work to involve users in order to obtain a broader and more comprehensive overview of the behaviour and decision-making processes of humans in virtual environments where artificial intelligences play an assisting role. Furthermore, the embodiment of agents will also be considered and its efects on human perception will be studied. Computer Science 169 (2020) 412–422. URL: https://linkinghub.elsevier.com/retrieve/pii/ S1877050920303628. doi:10.1016/j.procs.2020.02.238. [14] G. N. Yannakakis, A. Paiva, Emotion in games, Handbook on afective computing 2014 (2014) 459–471. Publisher: Oxford University Press. [15] H. M. Gray, K. Gray, D. M. Wegner, Dimensions of mind perception, science 315 (2007) 619–619. [16] A. Chella, A. Pipitone, A cognitive architecture for inner speech, Cognitive Systems Research 59 (2020) 287–292. URL: https://linkinghub.elsevier.com/retrieve/pii/ S1389041719304760. doi:10.1016/j.cogsys.2019.09.010. [17] A. Augello, G. Città, M. Gentile, A. Lieto, A Storytelling Robot Managing Persuasive and Ethical Stances via ACT-R: An Exploratory Study, International Journal of Social Robotics (2021). URL: https://link.springer.com/10.1007/s12369-021-00847-w. doi:10.1007/ s12369-021-00847-w. [18] J. Stenseke, Artificial virtuous agents: from theory to machine implementation, AI & SOCIETY (2021). URL: https://link.springer.com/10.1007/s00146-021-01325-7. doi:10. 1007/s00146-021-01325-7. [19] J. Stenseke, Artificial virtuous agents in a multi-agent tragedy of the commons, AI & SOCIETY (2022). URL: https://link.springer.com/10.1007/s00146-022-01569-x. doi:10. 1007/s00146-022-01569-x. [20] J.-A. Cervantes, S. López, L.-F. Rodríguez, S. Cervantes, F. Cervantes, F. Ramos, Artificial Moral Agents: A Survey of the Current Status, Science and Engineering Ethics 26 (2020) 501–532. URL: http://link.springer.com/10.1007/s11948-019-00151-x. doi:10.1007/ s11948-019-00151-x. [21] A. Vishwanath, E. D. Bøhn, O.-C. Granmo, C. Maree, C. Omlin, Towards artificial virtuous agents: games, dilemmas and machine learning, AI and Ethics (2022) s43681–022–00251–8. URL: https://link.springer.com/10.1007/s43681-022-00251-8. doi:10. 1007/s43681-022-00251-8. [22] Hugging face, 2016. URL: https://huggingface.co/. [23] G. F. Gaglio, Roberta action classification model, 2023. URL: https://huggingface.co/gFulvio/ moralstories-roberta-action.context-cls. [24] G. F. Gaglio, Bart consequences generation model, 2023. URL: https://huggingface.co/ gFulvio/moralstories-bart-consequences.context-action_gen. [25] G. F. Gaglio, Bart norm generation model, 2023. URL: https://huggingface.co/gFulvio/ moralstories-bart-norm.action-context_gen. [26] L. M. Csepregi, The efect of context-aware llm-based npc conversations on player engagement in role-playing video games (2023). URL: https://projekter.aau.dk/projekter/ files/536738243/The_Effect_of_Context_aware_LLM_based_NPC_Dialogues_on_Player_ Engagement_in_Role_playing_Video_Games.pdf. [27] OkGoDoIt, Openai-api-dotnet, 2023. URL: https://github.com/OkGoDoIt/ OpenAI-API-dotnet.