This paper introduces elements for the foundation of a notion of moral system of an agent society. The paper is specially concerned with elements for the design and analysis of moral systems of agent societies that are to be embedded in social contexts involving diverse human groups.
Moral systems embody norms and values about the conducts
(behaviors, interactions) that are possible in a society, as well
as any knowledge that may be available about those conducts,
norms and values [
In this paper, we introduce the core elements of a formal
foundation for moral systems of agent societies. In analogy
to H. Kelsen's theory of legal systems [
We use the term \moral knowledge" to denote knowledge that an agent has about another agent's morality. The set of moral features determined by such moral knowledge constitutes the moral model that the former (the moral modeler ) has about the latter (the one morally modeled ).
A moral model speci es the moral knowledge on the basis of which an agent ag1 analyzes both the conducts of some agent ag2 (possibly itself) and the moral assessments that ag2 does about the social conducts of any agent ag3. The core of the moral model that ag1 has of ag2 is the set of moral norms that ag1 believes that ag2 has adopted.
The moral knowledge embodied by a moral model is relativistic, for a variety of reasons. For instance, the moral knowledge embodied in a moral model depends on which are the agents (moral modeler and morally modeled) it concerns and on the means available for the moral modeler to gather information about the agent morally modeled.
Also, moral models are observational models, and the moral knowledge they embody can only be acquired in a piecewise way. In consequence, at each point in time, any moral model is tentative, regarding the information that the moral modeler could gather, up to that time.
Thus, the moral knowledge embodied in a moral model is always incomplete and, so, incapable to fully morally di erentiate that agent from others, morally similar agents.
In consequence, any moral modeling of an agent by another is, in fact, the moral modeling of a class of agents, always being more general than the modeling of one particular agent.
Any moral judgment of an individual agent is necessarily, then, a judgment based on a moral model of a class of agents, to which that agent is considered to belong, not about that individual agent, speci cally.
So, in principle, any such moral judgment is inevitably prejudicial, or stereotypical, in the sense that it is necessarily based on a prejudice about the individual agent being morally judged, namely, the prejudice that the individual fully ts the general moral features of the class of agents to which refers the moral model used to support the moral judgment.
By the same token, the moral judgment about an agent may be seamlessly extended, in an even more prejudicial way, to the totality of agents presumed to belong to the class of agents to which that agent is itself presumed to belong (that is, the class of agents referred to by the moral model).
One sees, then, that moral models have two important effects on the conducts of agents and groups of agents. They are a necessary means for the establishment of the indispensable minimum level of mutual moral understanding within any group of agents that constitutes itself as a social group.
They are also, however, a potential source of misconceptions of agents and groups of agents about each other. They are also, thus, a potential source of moral misunderstandings (more speci cally, moral con icts and their consequent moral contradictions) among those agents and groups of agents. 1.1
This paper aims to introduce conceptual elements necessary for a formal account of the structure and functioning of moral systems in agent societies, so that methods for the moral analysis and design of agent societies can be soundly established.
The paper concentrates on the basic components of such moral systems, namely, moral models, which are the structures that embody the moral knowledge that agents and social groups may have about each other.
In Sect. 2, we review J. Halpern and Y. Moses' way of formally accounting for knowledge that is about, and situated in, computational systems. We specialize their conception to knowledge about, and situated in, agent societies, and extend it to deal with the relativistic nature of such knowledge.
The result is the formal concept of knowledge that we use to account for the epistemic aspects of the notion of moral knowledge that we think is appropriate to agent societies.
In Sect. 3, we formally introduce the concepts of moral knowledge, moral model and moral judgments, as well as the concepts of morally assigned group identity, moral prejudice, and moral contradiction between social groups.
Finally, in Sect. 5, the paper introduces a notion of moral design of agent societies, built on the conceptual framework introduced previously, and brie y relates moral design to other parts of the organizational design of agent societies.
For completeness, we summarize now the notion of agent society adopted here. 1.2
The notion of agent society that we adopt here is the one
we have been using in our work (see, e.g., [
Formally, the organization of an agent society is a structure encompassing groups of agents (possibly singletons), together with the conducts that such groups of agents perform. The groups of agents constitute the organizational units of the society (independently of their being institutionalized or not). 2
We start with a general notion of knowledge, construed to
be both about an agent society, and situated in that agent
society. For that, we build on the general notion of knowledge
about a distributed computational system that is situated in
that system, which was introduced by Halpern and Moses [
Notice the crucial role that the concept of external observer plays in our overall conception. 2.1
A general characterization of knowledge in an agent society can be given as follows. Let:
G = fag1; : : : ; agng be a nite set, composed of n agents, generically ranged over by the variables agi and agj; P be a set of primitive propositions, generically ranged over by variables p and p0; 2 By a conduct of an agent or group of agents we understand either a behavior that that agent or group performs, when considered in isolation from other agents or groups, or the part of the interaction that an agent or group performs, when interacting with other agents or groups. ^ and : be propositional operators that (together with the operators _ and ), de ned from them) extend the set P to the set P of compound propositions, also generically ranged over by the variables p and p0.
We take Kag1 ; : : : ; Kagn to be epistemic operators, such that Kagi (p) means that p 2 P belongs to the knowledge of the agent agi, that is, that agent agi knows that p.
Three additional notions of knowledge are presented in [
EG(p), which means: p belongs to the knowledge of each of the agents of the group G; CG(p), which means: p belongs to the recursive notion of common knowledge of the agents of the group G, that is: each of the agents of the group G knows that p; each of the agents of the group G knows that each of the agents of the group G knows that p; etc.; IG(p), which means: p belongs to the implicit knowledge of the agents of the group G, that is, the union of the individual knowledges of the agents of the group G, so that an external observer that holds such union can deduce p if it reasons from that union, even if none of the agents can do that by reasoning from the common knowledge of G. This paper concentrates on propositions of the form Kagi (p). 2.2
With the notions of Kagi (p), EG(p), CG(p) and IG(p), Halpern
and colleagues [
We specialize their interpretation to agent societies in the following way. An agent society is characterized by a set of objective global states, de ned as SO = O T , where O is the set of all possible con gurations of the society 3, and T is a linear structure of discrete time instants, so that each global state of the society is a pair s = ( ; t) 2 SO.
The determination of such set of global states is objective in the sense that it is given by an external observer O that has access to all the details of the society, in a way that, from O's point of view, is taken to be complete. However, even though objective (external and complete), that characterization is still relativistic, precisely because it depends O's point of view, hence the index O in O and SO.
Regarding the individual agents, the approach assumes that - due to the locality of their particular points of view - each agent of the society partitions the set of global states SO (that O is capable of fully di erentiating) into equivalence classes. That is, each agent is lead to take as indistinguishable certain global states that can be objectively distinguished by O.
In precise terms: an agent is lead to take two objectively
di erent global states to be indistinguishable whenever the
agent's knowledge about the society is the same in the two
global states. That is, whenever the two states do not allow
the agent to elaborate di erent knowledges about the society.
3 See [
Formally, what is de ned is an epistemic structure MO = (SO; P ; vO; Kag1 ; : : : ; Kagn ) where:
SO = O T is the set of objective global states of the agent society, considered from the point of view of the external observer O; P is a set of propositions, with basic set P ; vO : SO P ! fT; F g is a truth assignment function that, to each global state s 2 So and each basic proposition p 2 P , assigns a truth value vO(s; p) 2 fT; F g, according with p being objectively true or false in the state s, from the point of view of O; each Kagi is an equivalence relation on SO, such that if (s; s0) 2 Kagi then agent agi can not distinguish between the global states s and s0, as O can; that is, given the knowledge that agent agi has about the society, the agent takes s and s0 to be indistinguishable.
We denote the fact that p 2 P is true in the global state s 2 SO by (MO; s) j= p.
With those de nitions, the semantics of the epistemic operators Kagi takes as its basis the objective truth of the primitive propositions in P , as given by the function vO.
Formally, we have:
1) (MO; s) j= p if and only if vO(s; p) = T ;
For any composed proposition p 2 P : 2) (MO; s) j= :p if and only if vO(s; p) = F ; 3) (MO; s) j= (p^p0) if and only if (MO; s) j= p and s j=MO p0; 4) (MO; s) j= Kagi (p) if and only if (MO; s0) j= p for each s0 2 SO such that (s; s0) 2 Kagi .
That is, an agent agi is objectively considered to know that p is true, in a given global state s, if and only if p is objectively true in s and p is objectively true in every state s0 that agi cannot distinguish from s.
Notice that the knowledge of an agent about p being true of a global state s, in which the agent nds itself, depends on p being objectively true in s, that is, being true from the point of view of the external observer O. That is, an agent is objectively considered to know something about its society if and only if the external observer O considers that it does.
Clearly, this possible world semantics makes use of an
observational notion of knowledge of an agent, di erent from
any intensional notion of knowledge, which takes as criterion
the occurrence of p in the knowledge base of the agent.
Accordingly, Halpern says that p is ascribed to the agent [
We call external relativity such condition that results from knowledge being assigned to agents on the basis of observations made by an external observer that also de nes the set of global states that should be taken into consideration. 2.3
We introduce now a crucial modi cation in the formal
characterization of knowledge just presented. Instead of having
an objective, external notion of truth, given by the function
vO : SO P ! fT; F g, determined by the external observer
of the society, we introduce a subjective, internal notion of
truth, given by a set of functions vagi : SO P ! fT; F g, one
per agent (see [
That is, we let each agent make use of vagi to decide, by itself, the truth of each proposition p 2 P , in each global state s 2 SO. At the same time, however, we keep the set of global states SO determined by the external observer O, so that a minimally objective connection is preserved in the account of the di erent truth functions of the agents.
What we obtain can be informally summarized as follows: an agent society is characterized by the set SO of its global states, as determined by the external observer O; each agent agi, according to the knowledge it has, establishes a relativistic equivalence relation KaRgi in the set of global states SO, so that if (s; s0) 2 KaRgi it happens that s and s0 are indistinguishable from agi's point of view; each agent agi, according to the knowledge it has, assigns to the primitive propositions of the set P , at each global state s, a truth value that is denoted by vagi (s; p) 2 fT; F g; the assignment of truth values to primitive propositions is extended to composed propositions in the natural way; the individual knowledge of each agent agi is characterized by the relativistic epistemic operator KaRgi ; whenever we want to refer to the objective knowledge of an agent agi (that is, knowledge that the agent can determine, if it uses the objective truth function vO), we make use of the objective epistemic operator that we have introduced above, denoted by Kagi .
The relativistic epistemic structure that characterizes the knowledge of the agents of the society is, then, given by MOR = (SO; P ; vO; Kag1 ; : : : ; Kagn ; vag1 ; : : : ; vagn ; KaRg1 ; : : : ; KaRgn ).
We denote by (MOR; s) j=agi p the fact that the proposition p is determined to be true in the state s, by the agent agi, in the context of the relativistic epistemic structure MOR.
Under these conditions, the semantics of the relativistic epistemic operator KaRgi , in a society that has MOR as its epistemic structure, is formally given by the following rules:
1) (MOR; s) j=agi p if and only if vagi (s; p) = T ;
For composed propositions p 2 P : 2) (MOR; s) j=agi :p if and only if vagi (s; p) = F ; 3) (MOR; s) j=agi (p ^ p0) if and only if (MOR; s) j=agi p and (MOR; s) j=agi p0; 4) (MOR; s) j=agi KaRgi (p) if and only if (MOR; s0) j=agi p for all (s; s0) 2 KaRgi ;
This allows us to establish another crucial point in our formal model, namely, the rule of internal relativity, according to which an agent agi is allowed to assign the knowledge of p to an agent agj, in accordance with agi's own knowledge.
Rule of Internal Assignment : In the global state s 2 SO, agent agi is allowed to assign the knowledge of p to an agent agj, denoted by (MOR; s) j=agi RaKgj (p), if and only if agi can verify that: 1. (MO; s) j= Kagj p, that is, it can be externally determined (i.e., from O's point of view) that agent agj knows p, in the global state s; 2. (MOR; s) j=agi RaKgi (p), that is, agi relativistically knows that p is true, in s.
Notice that the external assignment of the knowledge of p to agj, required by the rst condition, provides an objective point of comparison for di erent such assignments.
of Objective Epistemic Contradictions
The only way for an agent agi to argue that its relativistic (i.e., internal) truths are objective truths, is by the agent externalizing itself, that is, by agi considering itself to be in the role of O. In such situation, we say that agi has externalized and objecti ed its relativistic knowledge, and we denote by agiO that agi externalized itself, and by MaOgi its \objecti ed" subjective and relative epistemic structure.
By intending that MaOgi holds objectively, agi intends that (MOR; s) j=agi KaRgi (p) (i.e., that agi relativistically knows p in s) be equated both with (MaOgi ; s) j= p (i.e., that the externalized agent agiO objectively knows p in s) and with (MO; s) j= p (i.e., that p is objectively true in s).
Clearly, an externalized internal observer takes itself to be a superagent of the society, with the power to objectively determine what is true and what is false, in that society.
But, when two agents, agi and agj, externalize themselves, at the same time, an objective contradiction may be established between them, concerning what is objectively true and what is objectively false in the society.
For, in such situation, for some s 2 Sagi \ Sagj , the agent agi may consider it valid to equate (MOR; s) j=agi Kagi (p) with (MaOgi ; s) j= p and (MO; s) j= p while, at the same time, the agent agj may consider it valid to equate (MOR; s) j=agj R Kagj (:p) with (MagjO ; s) j=agj KagjO (:p) and (MO; s) j= :p. So that, jointly, the two agents claim both (MO; s) j= p and (MO; s) j= :p, which characterizes (from the point of view of O) the objective contradiction between them.
Moreover, under MaOgi and MaOgj , the agents may conclude that MaOgi j= Kagj (:p) and MaOgj j= Kagi (:p), each stating that the other is \objectively" wrong.
Such objective contradiction about a proposition p shows that (from the point of view of O) at least one of the agents involved in the contradiction is not assessing p objectively, that is, that either (MOR; s) j=agi KaRgi p or (MOR; s) j=agj KaRgj :p (or both) does not hold, so that either vagi or vagi (or both) is not in accordance with vO about s. 3 3.1
As indicated in the Introduction, moral knowledge refers both to the knowledge of moral norms of conducts that agents are supposed to follow and to the knowledge of facts involving conducts that agents have performed, are performing, or intend to perform. Moral knowledge also refers to the moral judgments that the agents make of their own conducts, or of the others, and to the moral norms with which agents perform those moral judgments.
We construe these four types of moral knowledge in terms of four basic types of moral propositions (each type admitting additional arguments and decorations): 1. moral norms: propositions of the forms prohib(Ag; Cnd ), oblig(Ag; Cnd ) and permit (Ag; Cnd ), meaning that agents of the class of agents Ag are (respectively) prohibited, obligated and permitted to perform conducts of the class of conducts Cnd ; 2. moral facts: propositions of the form prfrmt(agi; cnd ), meaning that, at the time t, agent agi performed (or is performing, or will perform) the conduct cnd ; 3. moral judgments: propositions of the form asgnt(agi; mfct ; mv ), meaning that, at time t, agent agi assigns (or is assigning, or will assign) the moral value mv 2 fprs; blmg (praise or blame) to the moral fact mfct ; 4. moral judgment rules: propositions of either forms: (a) If cmpl (cnd ; mnrm) and prfrmt(agj; cnd )
then allowed [asgnt0 (agi; prfmd t(agj; cnd ); prs)]. - meaning that if the conduct cnd complies4 with the moral norm mnrm and the agent agj performs that conduct at time t, then an agent agi is allowed to morally praise, at any time t0, the agent agj for performing cnd at the time t; (b) If :cmpl (cnd ; mnrm) and prfrmt(agj; cnd )
then allowed [asgnt0 (agi; pfrmd t(agj; cnd ); blm)]. - meaning that if the conduct cnd does not comply with the moral norm mnrm and the agent agj performs that conduct at time t then an agent agi is allowed to blame, at any time t0, the agent agj for performing cnd at the time t.
We remark that, among the conducts that agents may perform are moral judgments themselves, so that agents may be morally judged for performing moral judgments.
Also, we admit extensions of those forms (moral norms, facts, judgments and judgment rules), allowing for groups of agents substituting any of the agent arguments. For instance: If the collective conduct ccnd complies with the moral norm mnrm and the group of agents Ag performs that collective conduct at time t then an agent ag0 is allowed to praise, at any time t0, the group of agents Ag for performing ccnd at the time t. 3.2
We call moral model of a society any structure of the form MMdl = (RAgs; MNrms; MJRls; MFcts; MJdgms) where: RAg is a set of agents and groups of agents to which the model refers; MNrms is the set of moral rules which are valid in the model; MJRls is the set of moral judgment rules (see Sect. ??) that the agents and groups of agents in RAgs have adopted; MFcts is a set of moral facts involving an agent or a group of agents in RAgs; and MJdgms is a set of moral judgments, each with some agent or group of agents of RAgs assigning some moral value (praise or blame) to some moral fact. As mentioned above, we require MJdgms MFcts, so that moral judgments may be applied to moral judgments.
We let each agent ag (or group of agents Ag) develop its own moral model MMdl ag (or MMdl Ag ), referring such model to any set RAgsag (or RAgsAg ), of its own discretion. 4 We leave formally unde ned, here, the condition of a conduct complying with a moral norm.
Of course, regarding the epistemic structure MOR of the society, the knowledge embedded in a moral model is of the relativistic kind, both in what concerns the existence of agents and groups of agents (in RAgs) and moral norms (in MNrms), and in what concerns the occurrence of facts (in MFcts) and moral judgment rules (in MJRls).
For instance, an agent ag may have developed a moral model MMdl ag = (RAgsag ; MJRlsag ; MNrmsag ; MFctsag ; MJdgmsag ) embodying a relativistic moral knowledge such that, in s 2 SO, and from the point of view of the external observer O: (MOR; s) j=ag KaRg (fag1; Ag2g RAgsag ) - meaning that in the state s, from the point of view of ag, there are an agent ag1 and a group of agents Ag2 in the reference set RAgsag ; (MOR; s) j=ag KaRg (asgnt0 (ag3; prfmt(ag2; cnd 1); blm) 2 MAsgnsag ) - meaning that, in the state s, from the point of view of ag, it happened that, at time t0, agent ag3 blamed agent ag2 for having realized the conduct cnd 1 at time t; (MOR; s) j=ag KaRg (mrl 1 2 MRlsag ) - meaning that, in the state s, from the point of view of ag, there is a moral rule mrl 1 in the set MRlsag of moral rules that are applicable to the agents and groups of agents in the reference set RAgsag . 3.3
We call moral judgment any application of a moral judgment rule to the realization of a conduct by an agent or group of agents, the result of the moral judgment being the assignment of a moral value to the realization of that conduct.
Whenever an agent ag1 makes use of the moral judgment rule mjrl to perform, at time t0, a moral judgment of a conduct cnd realized by an agent ag2 at time t, the agent ag1 changes its current moral model MMdl ag1 , by including: the agent ag2 in the set Agsag1 , if it was not there already; the moral fact prfrmt(ag2; cnd ) in the set MFctsag1 , if it was not there already; the moral judgment asgnt0 (ag1; prfmt(ag2; cnd ); mv ) in the set MJdgmsag1 , where mv = blm if the judgment resulted in a blame, and mv = prs if it resulted in a praise. However, we require, for the agent ag1 to be able to perform such judgment, that the moral judgment rule mjrl already belonged to the set MJRlsag1 , at the time t0.
We say that there is a moral con ict between two moral rules, regarding a given conduct, if the rules are contradictory to each other, that is, if one permits or obliges the conduct while the other forbids it. 3.4
As mentioned above, moral prejudices arise from treating individual agents on the bases of judgments founded not on moral models of the individual agents themselves, but on moral models of the groups of agents to which those individual agents appear to belong (to the eyes of the moral modeler that performs the judgment).
Such transference of moral models of groups of agents to individual agents that seem to belong to them requires that groups of agents be morally modeled in terms of stereotypical conducts that their members appear to be used to perform (to the eyes of the moral modeler).
The set of stereotypical conducts that a moral modeler assigns to a group of agents constitutes a means to characterize the group, a way for the moral modeler to distinguish that group among other groups of agents, that is, an assigned group identity.
Moral prejudices arise, then, when an agent judges another agent on the basis of an identity assigned to a group to which the former considers the latter to belong.
To accommodate this notion of morally assigned group identity, we may extend the moral models with a component GIds, such that for each group of agents Ag in the reference set RAgs, one or more tuples of the form (Ag; id Ag ) may be present in GIds, where the group identity id Ag should be construed as a set of conducts considered by the moral modeler to be typical of the members of the group Ag.
With such addition, moral prejudices may be explained in terms of an operation of substitution of conducts, by which an individual agent is morally judged not by the particular conduct (with its precise characteristics, etc.) that it has performed, or intends to perform, but by a stereotypical conduct that is substituted for it, a conduct that is considered to be typical of the group of agents to which that agent is considered to belong.
On the other hand, we de ne a moral contradiction between two agents or groups of agents as a con ict between moral judgments made by such agents or groups of agents, on the basis of a moral con ict (objective or not) between them.
Since moral judgments are, in principle, relativistic judgments, moral contradictions can arise as objective issues, between given agents or groups of agents, only when their points of view are externalized and objecti ed : when they constitute their relative points of view as objective.
Only then one can characterize a moral contradiction arising from a moral contradiction as an objective moral contradiction. 4
Agent societies can operate in a stand alone fashion and, as any other type of isolated society, can develop its epistemic structure, and the moral system that it supports, in ways that are uncompromised by external conditions.
Whenever an agent society is embedded in a given human social context, however, its epistemic structure and the moral system that it supports necessarily have to take into account the points of view (both epistemic and moral) of the human agents and groups of human agents that constitute that human social context.
Moreover, when that agent society operates as an intermediary between di erent human groups, the agents and the groups of agents of the agent society necessarily have to take into account the possibility of the externalization of the relativistic points of view of the human agents and human groups, because those externalizations are the objective condition for the rise of moral contradictions among those human groups.
By moral design of an agent society, we mean the provision of architectural means to support the agents and groups of agents of the agent society in their handling of moral issues (specially moral contradictions and moral contradictions).
Similarly to the legal design of agent societies [
In particular, it belongs to the design of the normative
system [
As argued in several ways by several authors (see, e.g., [
In this paper, we have explored in a preliminary way a formalization of the notion of moral relativity in agent societies, taking a particular formalization of the notion of epistemic relativity as its foundation.
Formal moral concepts (of knowledge, model, judgment, prejudice, contradiction, contradiction, morally-based assignment of group identity, etc.) were introduced to capture moral issues that can arise in agent societies.
Also, the paper introduced the notion of moral design of agent society. Moral design should be a concern specially in regard to agent societies that are embedded in human social contexts that involve a variety of externalized and objecti ed moral models of individuals and social groups, and that are, thus, prone to produce objective moral contradictions and objective moral contradictions.
Although we have not touched the issue in the present
paper, it should be clear that the moral design of an agent
society should tackle also the de nition of the content of the moral
system of the society, and should proceed hand-in-hand with
the moral design of the agents themselves (see, e.g., [
Finally, it should also be clear that, when considering such
embedded agent societies, moral models (in the sense
introduced here) should be articulated with legal models (in the
sense proposed, e.g., in [