Moral Systems of Agent Societies: Some Elements for their Analysis and Design Antônio Carlos da Rocha Costa 1 Abstract. This paper introduces elements for the founda- In consequence, any moral modeling of an agent by another tion of a notion of moral system of an agent society. The is, in fact, the moral modeling of a class of agents, always paper is specially concerned with elements for the design and being more general than the modeling of one particular agent. analysis of moral systems of agent societies that are to be Any moral judgment of an individual agent is necessarily, embedded in social contexts involving diverse human groups. then, a judgment based on a moral model of a class of agents, to which that agent is considered to belong, not about that individual agent, specifically. 1 Introduction So, in principle, any such moral judgment is inevitably prej- Moral systems embody norms and values about the conducts udicial, or stereotypical, in the sense that it is necessarily (behaviors, interactions) that are possible in a society, as well based on a prejudice about the individual agent being morally as any knowledge that may be available about those conducts, judged, namely, the prejudice that the individual fully fits the norms and values [14]. general moral features of the class of agents to which refers In this paper, we introduce the core elements of a formal the moral model used to support the moral judgment. foundation for moral systems of agent societies. In analogy By the same token, the moral judgment about an agent to H. Kelsen’s theory of legal systems [13], the formal foun- may be seamlessly extended, in an even more prejudicial way, dation that we envisage concentrates on the principles of the to the totality of agents presumed to belong to the class of structure and operation of moral systems, not on the contents agents to which that agent is itself presumed to belong (that of their norms and values. is, the class of agents referred to by the moral model). We use the term “moral knowledge” to denote knowledge One sees, then, that moral models have two important ef- that an agent has about another agent’s morality. The set of fects on the conducts of agents and groups of agents. They moral features determined by such moral knowledge consti- are a necessary means for the establishment of the indispens- tutes the moral model that the former (the moral modeler ) able minimum level of mutual moral understanding within has about the latter (the one morally modeled ). any group of agents that constitutes itself as a social group. A moral model specifies the moral knowledge on the basis of They are also, however, a potential source of misconcep- which an agent ag 1 analyzes both the conducts of some agent tions of agents and groups of agents about each other. They ag 2 (possibly itself) and the moral assessments that ag 2 does are also, thus, a potential source of moral misunderstandings about the social conducts of any agent ag 3 . The core of the (more specifically, moral conflicts and their consequent moral moral model that ag 1 has of ag 2 is the set of moral norms contradictions) among those agents and groups of agents. that ag 1 believes that ag 2 has adopted. The moral knowledge embodied by a moral model is rela- 1.1 The Aims and Structure of the Paper tivistic, for a variety of reasons. For instance, the moral knowl- edge embodied in a moral model depends on which are the This paper aims to introduce conceptual elements necessary agents (moral modeler and morally modeled) it concerns and for a formal account of the structure and functioning of moral on the means available for the moral modeler to gather infor- systems in agent societies, so that methods for the moral anal- mation about the agent morally modeled. ysis and design of agent societies can be soundly established. Also, moral models are observational models, and the moral The paper concentrates on the basic components of such knowledge they embody can only be acquired in a piecewise moral systems, namely, moral models, which are the struc- way. In consequence, at each point in time, any moral model tures that embody the moral knowledge that agents and social is tentative, regarding the information that the moral modeler groups may have about each other. could gather, up to that time. In Sect. 2, we review J. Halpern and Y. Moses’ way of Thus, the moral knowledge embodied in a moral model is formally accounting for knowledge that is about, and situated always incomplete and, so, incapable to fully morally differ- in, computational systems. We specialize their conception to entiate that agent from others, morally similar agents. knowledge about, and situated in, agent societies, and extend 1 it to deal with the relativistic nature of such knowledge. Programa de Pós-Graduação em Informática na Educação da UFRGS. 90.040-060 Porto Alegre, Brazil. Programa de Pós- The result is the formal concept of knowledge that we use Graduação em Computação da FURG. 96.203-900 Rio Grande, to account for the epistemic aspects of the notion of moral Brazil. Email: ac.rocha.costa@gmail.com . knowledge that we think is appropriate to agent societies. In Sect. 3, we formally introduce the concepts of moral • ∧ and ¬ be propositional operators that (together with knowledge, moral model and moral judgments, as well as the the operators ∨ and ⇒, defined from them) extend the set concepts of morally assigned group identity, moral prejudice, P ∗ to the set P of compound propositions, also generically and moral contradiction between social groups. ranged over by the variables p and p0 . Finally, in Sect. 5, the paper introduces a notion of moral design of agent societies, built on the conceptual framework We take Kag 1 , . . . , Kag n to be epistemic operators, such that introduced previously, and briefly relates moral design to Kag i (p) means that p ∈ P belongs to the knowledge of the other parts of the organizational design of agent societies. agent ag i , that is, that agent ag i knows that p. For completeness, we summarize now the notion of agent Three additional notions of knowledge are presented in [10], society adopted here. besides this notion of individual knowledge Kag i (p). They refer to knowledge held by groups of agents: 1.2 Agent Society, Agent Conduct • EG (p), which means: p belongs to the knowledge of each of the agents of the group G; The notion of agent society that we adopt here is the one • CG (p), which means: p belongs to the recursive notion of we have been using in our work (see, e.g., [7]): we take an common knowledge of the agents of the group G, that is: agent society to be an open, organized, persistent and situated each of the agents of the group G knows that p; each of the multiagent system, where: agents of the group G knows that each of the agents of the • openness means that the agents of the society can freely group G knows that p; etc.; enter and leave it; • IG (p), which means: p belongs to the implicit knowledge of • organization means that the working of the society is based the agents of the group G, that is, the union of the indi- on an articulation of individual and collective conducts 2 , vidual knowledges of the agents of the group G, so that an the collective ones performed by groups of agents of various external observer that holds such union can deduce p if it kinds (institutionalized or not); reasons from that union, even if none of the agents can do • persistence means that the organization persists in time, that by reasoning from the common knowledge of G. independently of the agents that enter or leave the society; This paper concentrates on propositions of the form Kag i (p). • situatedness means that the society exists and operates in a definite physical environment, involving physical objects that the agents and groups of agents may make use of, in 2.2 External Relativity the performance of their individual and collective conducts. With the notions of Kag i (p), EG (p), CG (p) and IG (p), Halpern and colleagues [10, 11] proceed to analyze properties of com- Formally, the organization of an agent society is a structure munication and action coordination protocols in distributed encompassing groups of agents (possibly singletons), together systems. The basis of their approach is an interpretation, in with the conducts that such groups of agents perform. The terms of the set of the global states of a distributed com- groups of agents constitute the organizational units of the putational system, of the semantics of possible worlds that society (independently of their being institutionalized or not). Hintikka introduced in [12]. We specialize their interpretation to agent societies in the 2 Knowledge About an Agent Society that following way. An agent society is characterized by a set of is Situated in that Society objective global states, defined as SO = ΓO × T , where ΓO is We start with a general notion of knowledge, construed to the set of all possible configurations of the society 3 , and T is be both about an agent society, and situated in that agent a linear structure of discrete time instants, so that each global society. For that, we build on the general notion of knowledge state of the society is a pair s = (γ, t) ∈ SO . about a distributed computational system that is situated in The determination of such set of global states is objective in that system, which was introduced by Halpern and Moses [11]. the sense that it is given by an external observer O that has We take the presentation of that notion in [10] as our basis. access to all the details of the society, in a way that, from O’s Notice the crucial role that the concept of external observer point of view, is taken to be complete. However, even though plays in our overall conception. objective (external and complete), that characterization is still relativistic, precisely because it depends O’s point of view, hence the index O in ΓO and SO . 2.1 General Characterization Regarding the individual agents, the approach assumes that A general characterization of knowledge in an agent society - due to the locality of their particular points of view - each can be given as follows. Let: agent of the society partitions the set of global states SO (that O is capable of fully differentiating) into equivalence classes. • G = {ag 1 , . . . , ag n } be a finite set, composed of n agents, That is, each agent is lead to take as indistinguishable certain generically ranged over by the variables ag i and ag j ; global states that can be objectively distinguished by O. • P ∗ be a set of primitive propositions, generically ranged In precise terms: an agent is lead to take two objectively over by variables p and p0 ; different global states to be indistinguishable whenever the 2 By a conduct of an agent or group of agents we understand either agent’s knowledge about the society is the same in the two a behavior that that agent or group performs, when considered global states. That is, whenever the two states do not allow in isolation from other agents or groups, or the part of the inter- the agent to elaborate different knowledges about the society. action that an agent or group performs, when interacting with other agents or groups. 3 See [8] for the notion of configuration of agent society. 2 Formally, what is defined is an epistemic structure MO = That is, we let each agent make use of vag i to decide, by (SO , P ; vO , Kag 1 , . . . , Kag n ) where: itself, the truth of each proposition p ∈ P , in each global state s ∈ SO . At the same time, however, we keep the set of global • SO = ΓO ×T is the set of objective global states of the agent states SO determined by the external observer O, so that a society, considered from the point of view of the external minimally objective connection is preserved in the account of observer O; the different truth functions of the agents. • P is a set of propositions, with basic set P ∗ ; What we obtain can be informally summarized as follows: • vO : SO × P → {T, F } is a truth assignment function that, to each global state s ∈ So and each basic proposition p ∈ • an agent society is characterized by the set SO of its global P , assigns a truth value vO (s, p) ∈ {T, F }, according with states, as determined by the external observer O; p being objectively true or false in the state s, from the • each agent ag i , according to the knowledge it has, estab- R point of view of O; lishes a relativistic equivalence relation Kag i in the set of 0 R • each Kag i is an equivalence relation on SO , such that if global states SO , so that if (s, s ) ∈ Kag i it happens that s (s, s0 ) ∈ Kag i then agent ag i can not distinguish between and s0 are indistinguishable from ag i ’s point of view; the global states s and s0 , as O can; that is, given the • each agent ag i , according to the knowledge it has, assigns knowledge that agent ag i has about the society, the agent to the primitive propositions of the set P ∗ , at each global takes s and s0 to be indistinguishable. state s, a truth value that is denoted by vag i (s, p) ∈ {T, F }; • the assignment of truth values to primitive propositions is We denote the fact that p ∈ P is true in the global state extended to composed propositions in the natural way; s ∈ SO by (MO , s) |= p. • the individual knowledge of each agent ag i is characterized R With those definitions, the semantics of the epistemic op- by the relativistic epistemic operator Kag i ; erators Kag i takes as its basis the objective truth of the prim- • whenever we want to refer to the objective knowledge of an itive propositions in P , as given by the function vO . agent ag i (that is, knowledge that the agent can determine, Formally, we have: if it uses the objective truth function vO ), we make use of the objective epistemic operator that we have introduced • For any primitive proposition p ∈ P ∗ : above, denoted by Kag i . 1) (MO , s) |= p if and only if vO (s, p) = T ; The relativistic epistemic structure that characterizes the • For any composed proposition p ∈ P : R knowledge of the agents of the society is, then, given by MO = 2) (MO , s) |= ¬p if and only if vO (s, p) = F ; R R (SO , P ; vO , Kag 1 , . . . , Kag n ; vag 1 , . . . , vag n , Kag , . . . , K ). 3) (MO , s) |= (p∧p0 ) if and only if (MO , s) |= p and s |=MO p0 ; R 1 ag n 4) (MO , s) |= Kag i (p) if and only if (MO , s0 ) |= p for each We denote by (MO , s) |=ag i p the fact that the proposition s0 ∈ SO such that (s, s0 ) ∈ Kag i . p is determined to be true in the state s, by the agent ag i , in R the context of the relativistic epistemic structure MO . That is, an agent ag i is objectively considered to know that Under these conditions, the semantics of the relativistic R R p is true, in a given global state s, if and only if p is objectively epistemic operator Kag i , in a society that has MO as its epis- true in s and p is objectively true in every state s0 that ag i temic structure, is formally given by the following rules: cannot distinguish from s. • For primitive propositions p ∈ P ∗ : Notice that the knowledge of an agent about p being true 1) (MO R , s) |=ag i p if and only if vag i (s, p) = T ; of a global state s, in which the agent finds itself, depends • For composed propositions p ∈ P : on p being objectively true in s, that is, being true from the 2) (MO R , s) |=ag i ¬p if and only if vag i (s, p) = F ; point of view of the external observer O. That is, an agent is 3) (MO , s) |=ag i (p ∧ p0 ) if and only if (MO R R , s) |=ag i p and objectively considered to know something about its society if (MO R , s) |=ag i p0 ; and only if the external observer O considers that it does. 4) (MO R , s) |=ag i KagR (p) if and only if (MO R 0 , s ) |=ag i p for all i Clearly, this possible world semantics makes use of an ob- 0 R (s, s ) ∈ Kag i ; servational notion of knowledge of an agent, different from any intensional notion of knowledge, which takes as criterion This allows us to establish another crucial point in our for- the occurrence of p in the knowledge base of the agent. Ac- mal model, namely, the rule of internal relativity, according cordingly, Halpern says that p is ascribed to the agent [10]. to which an agent ag i is allowed to assign the knowledge of p We call external relativity such condition that results from to an agent ag j , in accordance with ag i ’s own knowledge. knowledge being assigned to agents on the basis of observa- • Rule of Internal Assignment: In the global state s ∈ SO , tions made by an external observer that also defines the set agent ag i is allowed to assign the knowledge of p to an of global states that should be taken into consideration. R agent ag j , denoted by (MO , s) |=ag i RK ag j (p), if and only if ag i can verify that: 2.3 Internal Relativity 1. (MO , s) |= Kag j p, that is, it can be externally determined (i.e., from O’s point of view) that agent ag j knows p, in We introduce now a crucial modification in the formal char- the global state s; acterization of knowledge just presented. Instead of having R an objective, external notion of truth, given by the function 2. (MO , s) |=ag i RK ag i (p), that is, ag i relativistically knows vO : SO × P → {T, F }, determined by the external observer that p is true, in s. of the society, we introduce a subjective, internal notion of Notice that the external assignment of the knowledge of p truth, given by a set of functions vag i : SO × P → {T, F }, one to ag j , required by the first condition, provides an objective per agent (see [9]). point of comparison for different such assignments. 3 2.4 The Externalization of Internally of the class of agents Ag are (respectively) prohibited, ob- Relativistic Knowledge, and the Rise ligated and permitted to perform conducts of the class of of Objective Epistemic Contradictions conducts Cnd ; Between Agents 2. moral facts: propositions of the form prfrm t (ag i , cnd ), meaning that, at the time t, agent ag i performed (or is The only way for an agent ag i to argue that its relativistic performing, or will perform) the conduct cnd ; (i.e., internal) truths are objective truths, is by the agent ex- 3. moral judgments: propositions of the form ternalizing itself, that is, by ag i considering itself to be in the asgn t (ag i , mfct, mv ), meaning that, at time t, agent role of O. In such situation, we say that ag i has externalized ag i assigns (or is assigning, or will assign) the moral value and objectified its relativistic knowledge, and we denote by mv ∈ {prs, blm} (praise or blame) to the moral fact mfct; ag O O i that ag i externalized itself, and by Mag i its “objectified” 4. moral judgment rules: propositions of either forms: subjective and relative epistemic structure. By intending that Mag O holds objectively, ag i intends that (a) If cmpl (cnd , mnrm) and prfrm t (ag j , cnd ) i R R 0 (MO , s) |=ag i Kag i (p) (i.e., that ag i relativistically knows p in then allowed [asgn t (ag i , prfmd t (ag j , cnd ), prs)]. O s) be equated both with (Mag , s) |= p (i.e., that the external- O i - meaning that if the conduct cnd complies4 with the ized agent ag i objectively knows p in s) and with (MO , s) |= p moral norm mnrm and the agent ag j performs that con- (i.e., that p is objectively true in s). duct at time t, then an agent ag i is allowed to morally Clearly, an externalized internal observer takes itself to be praise, at any time t0 , the agent ag j for performing cnd a superagent of the society, with the power to objectively de- at the time t; termine what is true and what is false, in that society. But, when two agents, ag i and ag j , externalize themselves, (b) If ¬cmpl (cnd , mnrm) and prfrm t (ag j , cnd ) at the same time, an objective contradiction may be estab- 0 then allowed [asgn t (ag i , pfrmd t (ag j , cnd ), blm)]. lished between them, concerning what is objectively true and what is objectively false in the society. - meaning that if the conduct cnd does not comply with For, in such situation, for some s ∈ Sag i ∩ Sag j , the agent the moral norm mnrm and the agent ag j performs that ag i may consider it valid to equate (MO R , s) |=ag i Kag i (p) conduct at time t then an agent ag i is allowed to blame, with (Mag O , s) |= p and (M O , s) |= p while, at the same time, at any time t0 , the agent ag j for performing cnd at the i the agent ag j may consider it valid to equate (MO R , s) |=ag j time t. R Kag j (¬p) with (Mag O , s) |=ag j Kag O (¬p) and (MO , s) |= ¬p. We remark that, among the conducts that agents may per- j j So that, jointly, the two agents claim both (MO , s) |= p and form are moral judgments themselves, so that agents may be (MO , s) |= ¬p, which characterizes (from the point of view of morally judged for performing moral judgments. O) the objective contradiction between them. Also, we admit extensions of those forms (moral norms, O O Moreover, under Mag i and Mag j , the agents may conclude facts, judgments and judgment rules), allowing for groups of O O that Mag i |= Kag j (¬p) and Mag j |= Kag i (¬p), each stating agents substituting any of the agent arguments. For instance: that the other is “objectively” wrong. • If the collective conduct ccnd complies with the moral norm Such objective contradiction about a proposition p shows mnrm and the group of agents Ag performs that collective that (from the point of view of O) at least one of the agents in- conduct at time t then an agent ag 0 is allowed to praise, at volved in the contradiction is not assessing p objectively, that R R R R any time t0 , the group of agents Ag for performing ccnd at is, that either (MO , s) |=ag i Kag p or (MO , s) |=ag j Kag ¬p i j the time t. (or both) does not hold, so that either vag i or vag i (or both) is not in accordance with vO about s. 3.2 Moral Model 3 Elements for Moral Systems of Agent We call moral model of a society any structure of the form Societies MMdl = (RAgs, MNrms, MJRls, MFcts, MJdgms) where: 3.1 Moral Knowledge RAg is a set of agents and groups of agents to which the model refers; MNrms is the set of moral rules which are valid As indicated in the Introduction, moral knowledge refers both in the model; MJRls is the set of moral judgment rules (see to the knowledge of moral norms of conducts that agents are Sect. ??) that the agents and groups of agents in RAgs have supposed to follow and to the knowledge of facts involving adopted; MFcts is a set of moral facts involving an agent or conducts that agents have performed, are performing, or in- a group of agents in RAgs; and MJdgms is a set of moral tend to perform. Moral knowledge also refers to the moral judgments, each with some agent or group of agents of RAgs judgments that the agents make of their own conducts, or of assigning some moral value (praise or blame) to some moral the others, and to the moral norms with which agents perform fact. As mentioned above, we require MJdgms ⊆ MFcts, so those moral judgments. that moral judgments may be applied to moral judgments. We construe these four types of moral knowledge in terms We let each agent ag (or group of agents Ag) develop its of four basic types of moral propositions (each type admitting own moral model MMdl ag (or MMdl Ag ), referring such model additional arguments and decorations): to any set RAgs ag (or RAgs Ag ), of its own discretion. 1. moral norms: propositions of the forms prohib(Ag, Cnd ), 4 We leave formally undefined, here, the condition of a conduct oblig(Ag, Cnd ) and permit(Ag, Cnd ), meaning that agents complying with a moral norm. 4 R Such transference of moral models of groups of agents to Of course, regarding the epistemic structure MO of the so- ciety, the knowledge embedded in a moral model is of the rel- individual agents that seem to belong to them requires that ativistic kind, both in what concerns the existence of agents groups of agents be morally modeled in terms of stereotypical and groups of agents (in RAgs) and moral norms (in MNrms), conducts that their members appear to be used to perform and in what concerns the occurrence of facts (in MFcts) and (to the eyes of the moral modeler). moral judgment rules (in MJRls). The set of stereotypical conducts that a moral modeler as- For instance, an agent ag may have developed a moral signs to a group of agents constitutes a means to characterize model MMdl ag = (RAgs ag , MJRls ag , MNrms ag , MFcts ag , the group, a way for the moral modeler to distinguish that MJdgms ag ) embodying a relativistic moral knowledge such group among other groups of agents, that is, an assigned group that, in s ∈ SO , and from the point of view of the external identity. observer O: Moral prejudices arise, then, when an agent judges another agent on the basis of an identity assigned to a group to which R R • (MO , s) |=ag Kag ({ag 1 , Ag 2 } ⊆ RAgs ag ) the former considers the latter to belong. - meaning that in the state s, from the point of view of ag, To accommodate this notion of morally assigned group there are an agent ag 1 and a group of agents Ag 2 in the identity, we may extend the moral models with a component reference set RAgs ag ; GIds, such that for each group of agents Ag in the reference 0 R R • (MO , s) |=ag Kag (asgn t (ag 3 , prfm t (ag 2 , cnd 1 ), blm) ∈ set RAgs, one or more tuples of the form (Ag, id Ag ) may be MAsgns ag ) present in GIds, where the group identity id Ag should be con- - meaning that, in the state s, from the point of view of ag, strued as a set of conducts considered by the moral modeler it happened that, at time t0 , agent ag 3 blamed agent ag 2 to be typical of the members of the group Ag. for having realized the conduct cnd 1 at time t; With such addition, moral prejudices may be explained in R R • (MO , s) |=ag Kag (mrl 1 ∈ MRls ag ) terms of an operation of substitution of conducts, by which - meaning that, in the state s, from the point of view of ag, an individual agent is morally judged not by the particular there is a moral rule mrl 1 in the set MRls ag of moral rules conduct (with its precise characteristics, etc.) that it has per- that are applicable to the agents and groups of agents in formed, or intends to perform, but by a stereotypical conduct the reference set RAgs ag . that is substituted for it, a conduct that is considered to be typical of the group of agents to which that agent is considered to belong. 3.3 Moral Judgments and Moral Conflicts On the other hand, we define a moral contradiction between We call moral judgment any application of a moral judgment two agents or groups of agents as a conflict between moral rule to the realization of a conduct by an agent or group of judgments made by such agents or groups of agents, on the agents, the result of the moral judgment being the assignment basis of a moral conflict (objective or not) between them. of a moral value to the realization of that conduct. Since moral judgments are, in principle, relativistic judg- Whenever an agent ag 1 makes use of the moral judgment ments, moral contradictions can arise as objective issues, be- rule mjrl to perform, at time t0 , a moral judgment of a conduct tween given agents or groups of agents, only when their points cnd realized by an agent ag 2 at time t, the agent ag 1 changes of view are externalized and objectified : when they constitute its current moral model MMdl ag 1 , by including: their relative points of view as objective. Only then one can characterize a moral contradiction aris- • the agent ag 2 in the set Ags ag 1 , if it was not there already; ing from a moral contradiction as an objective moral contra- • the moral fact prfrm t (ag 2 , cnd ) in the set MFcts ag 1 , if it diction. was not there already; 0 • the moral judgment asgn t (ag 1 , prfm t (ag 2 , cnd ), mv ) in the set MJdgms ag 1 , where mv = blm if the judgment resulted 4 The Embedding of Agent Societies in in a blame, and mv = prs if it resulted in a praise. Human Social Contexts However, we require, for the agent ag 1 to be able to perform Agent societies can operate in a stand alone fashion and, as such judgment, that the moral judgment rule mjrl already any other type of isolated society, can develop its epistemic belonged to the set MJRls ag 1 , at the time t0 . structure, and the moral system that it supports, in ways that We say that there is a moral conflict between two moral are uncompromised by external conditions. rules, regarding a given conduct, if the rules are contradictory Whenever an agent society is embedded in a given human to each other, that is, if one permits or obliges the conduct social context, however, its epistemic structure and the moral while the other forbids it. system that it supports necessarily have to take into account the points of view (both epistemic and moral) of the human agents and groups of human agents that constitute that hu- 3.4 Group Identity, Moral Prejudice, man social context. Moral Contradiction Moreover, when that agent society operates as an inter- As mentioned above, moral prejudices arise from treating in- mediary between different human groups, the agents and the dividual agents on the bases of judgments founded not on groups of agents of the agent society necessarily have to take moral models of the individual agents themselves, but on into account the possibility of the externalization of the rela- moral models of the groups of agents to which those individ- tivistic points of view of the human agents and human groups, ual agents appear to belong (to the eyes of the moral modeler because those externalizations are the objective condition for that performs the judgment). the rise of moral contradictions among those human groups. 5 Figure 1 illustrates the situation of a particular agent so- regard to agent societies that are embedded in human social ciety which embedded in a particular human social context, contexts that involve a variety of externalized and objectified with interactions between humans and agents, and some ac- moral models of individuals and social groups, and that are, cesses to the moral models that are taken to be common to thus, prone to produce objective moral contradictions and ob- all the agents of each society. jective moral contradictions. Although we have not touched the issue in the present pa- per, it should be clear that the moral design of an agent soci- ety should tackle also the definition of the content of the moral system of the society, and should proceed hand-in-hand with the moral design of the agents themselves (see, e.g., [4], for the latter issue). Finally, it should also be clear that, when considering such embedded agent societies, moral models (in the sense intro- duced here) should be articulated with legal models (in the sense proposed, e.g., in [6] and, more extensively, in [7]). REFERENCES [1] Peter L. Berger and Thomas Luckmann, The Social Construc- Figure 1. Agent society embeeded in a human social context. tion of Reality - A Treatise in the Sociology of Knowledge, Anchor Books, New York, 1966. [2] Guido Boella, Leendert van der Torre, and Harko Verha- gen, ‘Introduction to normative multiagent systems’, Compu- tational and Mathematical Organization Theory, 12, 71–79, (2006). [3] Karen Carley and Les Gasser, ‘Computational organization 5 The Notion of Moral Design of an Agent theory’, in Multiagent Systems: A Modern Approach to Dis- Society tributed Artificial Intelligence, ed., Gerhard Weiss, 299–330, MIT Press, Cambridge, (1999). By moral design of an agent society, we mean the provision [4] Helder Coelho and Antônio Carlos Rocha Costa, ‘On the in- of architectural means to support the agents and groups of telligence of moral agency’, in 14th Portuguese Conference agents of the agent society in their handling of moral issues on Artificial Intelligence - EPIA’2009/Social Simulation and (specially moral contradictions and moral contradictions). Modelling - SSM 2009, pp. 439–450. University of Aveiro, (2009). Similarly to the legal design of agent societies [7], the moral [5] Antônio Carlos Rocha Costa. The cultural level of agent so- design of agent societies belongs to the design of the culture cieties. Invited talk at WESAAC 2011 - 5o. Workshop-School of the agent society [5], and so belongs to various domains of Agent Systems, their Environments, and Applications. Cu- of its architectural design (organizational structure, symbolic ritiba, Brazil. Proceedings, 2011. (In Portuguese). environment, etc.). [6] Antônio Carlos Rocha Costa. On the legal aspects of agent societies. Open publication on www.ResearchGate.net - DOI: In particular, it belongs to the design of the normative sys- 10.13140/2.1.4345.7923, 2014. tem [2] of the agent society, as the moral system is a part of [7] Antônio Carlos Rocha Costa, ‘Situated legal systems and the normative system of the society. Also, it belongs to the de- their operational semantics’, Artificial Intelligence & Law, sign of the organizational intelligence and of the information 43(1), 43–102, (2015). [8] Antônio Carlos Rocha Costa and Graçaliz Pereira Dimuro, distribution constraints [3] of the society. ‘A minimal dynamical organization model’, in Hanbook of Multi-Agent Systems: Semantics and Dynamics of Organiza- 6 Conclusion tional Models, ed., V. Dignum, 419–445, IGI Global, Hershey, (2009). As argued in several ways by several authors (see, e.g., [1]), [9] Antônio Carlos da Rocha Costa, ‘Relativismo epistêmico em the social processes of knowledge construction are strongly sociedades de agentes: Uma modelagem semântica prelimi- nar’, in Anais do Workshop-Escola de InformáticaTeórica - conditioned by the social and historical contexts in which they WEIT 2011, pp. 122–133. UFPEL, (2012). (in Portuguese). occur, contexts that vary widely in time and space among dif- [10] Joseph Y. Halpern, ‘Using reasoning about knowledge to an- ferent societies, and even among different social groups within alyze distributed systems’, Annual Review of Computer Sci- a single society. So, any approach to the issue of the social ence, 2, 37–68, (1987). [11] Joseph Y. Halpern and Y. Moses, ‘Knowledge and common construction of moral knowledge has to deal with the issue of knowledge in a distributed environment’, in Proc. 4th ACM epistemic relativity. Symposium on Principles of Distributed Computing, pp. 50– In this paper, we have explored in a preliminary way a for- 61, (1984). malization of the notion of moral relativity in agent societies, [12] Jaakko Hintikka, Knowledge and Belief : An Introduction to taking a particular formalization of the notion of epistemic the Logic of the Two Notions, Cornell University Press, New York, 1962. relativity as its foundation. [13] Hans Kelsen, Pure Theory of Law, The Law Book Exchange, Formal moral concepts (of knowledge, model, judgment, New Jersey, 2009. prejudice, contradiction, contradiction, morally-based assign- [14] Émile Durkheim, ‘Introduction à la morale’, Revue ment of group identity, etc.) were introduced to capture moral Philosophique, 89, 81–97, (1920). issues that can arise in agent societies. Also, the paper introduced the notion of moral design of agent society. Moral design should be a concern specially in 6