This paper is a first step towards a formal model that is intended to be the basis of an artificial agent's reasoning that could be considered by a human as an ethical reasoning. This work is included in a larger project aiming at designing an authority-sharing manager between a robot and a human being when the human-robot system faces decision making involving ethical issues. Indeed the possible decisions in such a system will have to be considered in the light of arguments that may vary according to each agent's points of view. The formal model allows us to translate in a more rigourous way than in natural language what is meant by various ethical frameworks and paves the way for further implementation of an ”ethical reasoning” that could put forward arguments explaining one judgement or another. To this end the ethical frameworks models will be instantiated on some classical ethical dilemmas and then analyzed and compared to each other as far as their judgements on the dilemmas are concerned.
Let us consider two classical ethical dilemmas. How would you react?
A trolley that can no longer stop is hurtling towards five people working on the track. They will die hit by the trolley, unless you decide to move the switch to deviate the train to another track only one person is working on. What would you do? Sacrifice one person to save the other five, or let five people die? 2. The ”fatman” trolley
A trolley that can no longer stop is hurtling towards five people working on the track. This time you are on a bridge, a few meters before them, with a fat man. If you push this man on the track, he is fat enough to stop the trolley and save the five people, but he will die. Would you push the ”fatman” ?
There is no really ”right” answer to those dilemmas, nevertheless
ethics may be used to guide reasoning about them. Therefore we will
start by general definitions about ethics and related concepts.
Definition 1 (Ethics) Ricoeur [
As far as ethical dilemmas are concerned, one builds a decision on normative ethics.
are policies, ways of acting. Example: ”Thou shalt not lie”.
where it is impossible to make a decision without overriding one of our principles.
Note that the definition used (based on [
a decision through some norm established by a particular ethical
framework.[
way for dealing with situations involving ethical dilemmas thanks to principles, metrics, etc. For example utilitarianism focuses on the consequences of a decision, the best being the one which provides the most good or does the least harm.
We will consider that the agent is the entity that has to make a decision in an ethical dilemma.
In this paper, our aim is to formalize different kinds of judgements
according to various ethical frameworks, in order to provide an
artificial agent with the decision-making capability in front of an
ethical dilemma, together with the capability to explain its decision,
especially in a user/operator-robot interaction context [
The work of [
The work of [
Both approaches have given us ideas on how to model an ethical
judgement, starting from a world representation involving facts and
causality, so as about some modelling issues: how to determine a
moral action? how to define proportionality? As [
We will first propose some concepts to describe the world and the ethical dilemma itself. Then we will provide details about ethical frameworks, tools to formalize them and how they judge possible choices in the ethical dilemmas. Choice (or decision) is indeed the core of our model, since it is about determining what is ethically acceptable or not according to the ethical framework. We will show that although each ethical framework gives different judgements on the different ethical dilemmas, similarities can be highlighted. 2 2.1
The agent decides and acts in a complex world which changes. The ethical dilemma is studied from the agent’s viewpoint. For each ethical dilemma, the agent has to make a decision among all possible decisions. We will consider ”doing nothing” as a possible decision.
In the context of an ethical dilemma, the agent knows all the possible decisions and all the effects of a given decision.
Considerations as good/bad4 and positive/negative5 are defined as such from the agent’s viewpoint.
Moreover, as some dilemmas involve the human life question, we will make the simplifying assumption:
A human life is perfectly equal to another human life, whoever the human being is.
In the next sections we will define some concepts to represent the world and its evolution. Those concepts and their interactions are illustrated in figure 1. 2.2
We characterize the environment around the agent by world states. Definition 6 (World state - Set S) A world state is a vector of state components (see definition below). Let S be the set of world states. 4 A decision is good if it meets the moral values of the agent; a bad decision violates them. 5 A fact is positive if it is beneficial for the agent; it is negative if it is undesirable for the agent. 6 This model is not quite far from event calculus and situation calculus. As things currently stand, fluents are close to state components, and events and actions modify values of them through functions (such as Consequence in this paper). Definition 7 (State component / fact - Set F ) A state component, also named fact, is a variable that can be instantiated only with antagonist values. We consider antagonist values as two values regarding the same item, one being the negation of the other. An item can be an object (or several objects), a living being (or several living beings), or anything else which needs to be taken into account by the agent. Let F be the set of state components.
f5 = five people are alive f5 = five people are dead
Because two values of a fact concern the same item, f5 and f5 concern the same five people.
Depending on the context ” ” will not have exactly the same meaning. This notation allows us to consider antagonist values such as gain/loss, gain/no gain, loss/no loss, etc. Those values have to be defined for each fact.
Consequently an example of a world state is: s 2 S; s = [f1; f5]; f1; f5 2 F (1) 2.3
to do something, i.e. perform an action, or to do nothing and let the world evolve. Let D be the set of decisions.
When the agent makes a decision, this results in an event that modifies the world. Nevertheless an event can also occur as part of the natural evolution of the world, including the action of another agent. Consequently we will differentiate the event concept from the agent’s decision concept.
Definition 9 (Event - Set E ) An event is something that happens in the world that modifies the world, i.e. some states of the world. Let E be the set of events.
Let Event be the function computing the event linked to a decision: Event : D ! E (2)
The consequence of an event is the preservation or modification of state components. The resulting state is called effect.
Definition 10 (Effect) The effect of an event is a world state of the same dimension and composed of the same facts as the world state before the event; only the values of facts may change. Ef f ect 2 S. Let Consequence be the function to compute the effect from current state:
Consequence : E
S ! S
In the case of the crazy trolley dilemma, if the agent’s decision is to ”do nothing” (no action of the agent), the trolley will hit the five people (event) and they will be killed (effect). If the agent’s decision is to ”move the switch” (decision), the trolley will hit one person (event); and they will be killed (effect). 3 3.1
The agent will make a decision according to one or several ethical frameworks. Each ethical framework will issue a judgement on a decision, e.g. on the decision nature, the event consequence, etc. When several ethical frameworks are considered by the agent, their judgements may be confronted to compute the agent’s resulting decision, see figure 2: Indeed the judgement of an ethical framework determines whether a decision is acceptable, unacceptable or undetermined as regards this ethical frame. A decision is judged acceptable if it does not violate the principles of the ethical framework. A decision is judged unacceptable if it violates some principles of the ethical framework. If we cannot determine whether the decision violates principles or not, it is judged undetermined. Let V be the set V = facceptable(>); undetermined(?); unacceptable(?)g
J udgement : D
The literature highlights three major ethical frameworks [
By contrast, and according to [
This ethical framework focuses only on the consequences of an event. According to consequentialist ethics, the agent will try to have the best possible result (i.e. the best effect), disregarding the means (i.e. the event). The main issue with this framework is to be able to compare the effects of several events, i.e. to compare sets of facts. Consequently we will distinguish between positive facts and negative facts within an effect; we want to be able to compute preferences between effects, i.e. to compare set of positive (resp. negative) facts of an effect with set of positive (resp. negative) facts of another effect. 3.2.1
Positive/Negative facts Let P ositive and N egative the functions:
P ositive=N egative : S ! P (F ) returning the subset of facts estimated as positive (resp. negative) from an effect.
In this paper, we assume that for an effect s:
P ositive(s) \ N egative(s) = ; 3.2.2
Preference Let c be the preference relation on subsets of facts (P(F )). F1 c F2 means that subset F1 is preferred to subset F2 from the consequentialist viewpoint. Intuitively we will assume the following properties of c: if a subset of facts F1 is preferred to another subset F2, thus it is impossible to prefer F2 to F1.
F1 c F2 ! :(F2 c F1) if F1 is preferred to F2 and F2 is preferred to another subset of facts F3, then F1 is preferred to F3.
[(F1 c F2) ^ (F2 c F3)] !
F1 c F3 A subset of facts cannot be preferred to itself. (10) (11) (12) (13) (14) c Fi
tive).
c is a strict order (irreflexive, asymmetric and transi3.2.3 A decision d1 involving event e1 (Event(d1) = e1) is considered better by the consequentialist framework than decision d2 involving event e2 (Event(d2) = e2) iff for i 2 S: P ositive(Consequence(e1; i)) and N egative(Consequence(e1; i)) c P ositive(Consequence(e2; i)) (15) positive consequentialism (15), trying to have the ”better good” negative consequentialism (16), trying to have the ”lesser evil”
J udgementc(d1; i) = >; and J udgementc(d2; i) = ? If at least one property is not satisfied, there is no best solution:
J udgementc(d1; i) = J udgementc(d2; i) = ? In the case of a dilemma with more than two possible decisions, the best decision is the decision that is judged better than all the others. If such a decision does not exist, it is impossible to determine an acceptable solution with consequentialist ethics. Nevertheless if there is a decision d1 with another decision d2 better than d1, then d1 is judged unacceptable, as d1 cannot be the best. 3.3
This ethical framework focuses only on the nature of the decision, no matter the consequences. Indeed the agent wants to make a moral decision, which is close to abide by norms or to Kant’s theory. Therefore we have to define the nature of a decision. 3.3.1
Decision nature A decision may be good, neutral, bad or undetermined from the agent’s point of view. Let N be the set
N = fgood; neutral; bad; undeterminedg There is a partial order <d in N :
bad <d neutral <d good Meaning that a good nature is preferable to a neutral which is preferable to a bad. undetermined cannot be ordered, because it represents a lack of information.
We assume intuitively that:
bad <d good Likewise, we admit that good <d bad is false. We also define the following relations: (17) (18) (19) (20) (21) =d, for example good =d good d: a d b iff a <d b or a =d b. 3.3.2 DecisionN ature(d) =d undetermined ) J udgementd(d; i) = ? (24)
DecisionN ature(d) <d neutral ) J udgementd(d; i) = ? (25) 3.4
The Doctrine of Double Effect is considered here as an ethical
framework, as in other papers [
We already have the tools required for the first rule (see 3.3.1). The second rule involves something else as until now, the difference between causal deduction (e.g. if I unplug the computer, it turns off) and temporal deduction (e.g. if I erase a file on the boss’s computer, I will be fired) has not been considered. Only a function between an event and its effect has been defined and it does not any difference between an event preventing the occurrence of a fact which would happened as a natural evolution and an event inducing a fact by causality. As for the third rule, we need to define what proportional means. 3.4.1
Causality Let us consider two facts that are causally connected, what does it mean? This link is not always a logical implication. Indeed it could be an inference, but such an inference is not always direct or instant. That is why we will use a symbol of temporal modal logic: p ` F q (26) which means the occurrence of p induces the occurrence of q (in all possible futures): fact p is a way to obtain fact q.
Example: buy candy ` F possess candy Example: Sam wants a candy, if he steals it, he will feel guilty, which he considers acceptable and proportional to have a candy, but he will be punished too, which is too bad for a candy, not proportional from his point of view. Another solution is to buy candy. Of course, he will have no more money after that but, to have a candy, it is proportional, and even better, the seller will offer him a lollipop, which is proportional to have no more money too! The last solution is to kill the seller to take the candy. By doing that, he will have candy, but he will go to jail, which is not proportional, and he will never have candy again, which is not proportional either.
Positive facts : candy Negative facts : guilty; punished guilty .p candy (32) We want to know if fguilty; punishedg -p fcandyg. With the elitist proportional criterion, all facts of the set at the right of the symbol need to have (at least) a fact of the set at the left of the symbol proportional to themselves. Here this criterion is satisfied, candy is the only fact at the right of the symbol, and guilty at the left is proportional to candy (32). But, with the democratic proportional criterion, all facts of the set at the left of the symbol have to be proportional to (at least) one fact of the set at the right of the symbol. And, even if guilty is proportional to candy, punished is not proportional to any fact. Thus, the democratic proportional criterion is not satisfied.
Positive facts : candy; lollipop Negative facts : no more money
no more money .p candy no more money .p lollipop 1. It is proportional, in response to a cockroaches invasion, to set traps in a house. But it is not proportional to drop an A-bomb on the house to eliminate cockroaches.
Nevertheless proportionality is less obvious in other cases, for instance : 2. Someone will consider that it is proportional to give a certain amount of money for exchange of a thing or a service, while someone else will think that it is not (e.g. too expensive). 3. Even if it is ”easy” to compare the loss of one life to the loss of several lives, what about the comparison between the loss of one life and the safeguard of several lives?
In this paper, proportionality is implemented by relation .p between facts (F ). f1 .p f2 means that f1 is proportional to f2, i.e. f1 has an importance lower than or close to the importance of f2. Importance depends on the context and on the agent.
There is no fact closer of a fact than the fact itself. For example the most equivalent response to a slap is another slap. Thereby we will assume that a fact is proportional to itself.
8fi 2 F ! fi .p fi (28) .p is therefore reflexive.
Furthermore if f1 has an importance lower than or close to the importance of f2 (f1 .p f2), and the importance of f2 is lower than or close to the importance of f3 (f2 .p f3), thus the importance of f1 is necessary lower than or close to the importance of f3 (f1 .p f3). For example, if a murder is considered worse (i.e. more important) than a theft (thef t .p murder), and if a theft is considered worse than a lie (lie .p thef t), thus a murder is worse than a lie (lie .p murder).
8f1; f2; f3 2 F = (f1 .p f2 ^ f2 .p f3) ! f1 .p f3 (29) .p is transitive.
By contrast, f1 .p f2 does not mean that f2 .p f1. It is true only if the importances of both facts are close. For example it is proportional to hit someone who threatens me with a gun, but it is not proportional to threaten someone with a gun if they hit me. .p is neither symmetric nor asymmetric.
We extend the relation .p to a relation -p between sets of facts,
which means that the set of facts at the left of the symbol is
proportional to the set of facts at the right. Two criteria can be considered
to compute -p, they are inspired from [
tional to a set of facts G (F -p G) iff:
8f 2 F; 9g 2 G=f .p g which means that every single element of F needs to be proportional to an element of G.
Elitist proportional criterion : a set of facts F is proportional to a set of facts G (F -p G) iff:
8g 2 G; 9f 2 F =f .p g which means that every single element of G needs to have an element of F proportional to itself. (30) (31) (33) (34) (35) (36) We want to know if fno more moneyg -p fcandy; lollipopg. no more money is proportional to candy and lollipop (33,34) therefore both criteria are satisfied.
Positive facts : candy Negative facts : jail; no more candy f or ever We want to know if fjail; no more candy f or everg -p fcandyg. But in this case, there is no proportionality between negative and positive facts. Therefore no criterion is respected.
Therefore, it is possible to use the democratic proportional criterion or the elitist proportional criterion or both of them to determine whether a set of facts is proportional to another set of facts. 3.4.3
Judgement function Thanks to the previous tools, we can now assess whether a decision meets the DDE rules.
Let i be the initial state and d the decision: e s = =
Event(d) 1. Deontological rule: decision d has to be good or neutral according to deontological ethics. 2. Collateral damage rule: negative facts must be neither an end nor a mean (such as collateral damages). It can be expressed as: 8fn 2 N egative(s); @fp 2 P ositive(s); (fn ` F fp) (38) The ”evil wish” (negative fact(s) as a purpose) is not considered as we assume that the agent is not designed to make the evil. 3. Proportionality rule: the set of negative facts has to be proportional to the set of positive facts.
N egative(s) -p P ositive(s) (39) A decision d is acceptable for the DDE if it violates no rule, which means:
This section focuses on how our model can be instantiated on the ethical dilemmas that have been introduced at the beginning of the paper. For each dilemma the agent has to choose a decision. We will describe how consequentialist ethics, deontological ethics and the Doctrine of Double Effect assess the agent’s possible decisions.
meaning that it is better to lose one life than five lives (reverse numerical order 1 > 5).
Therefore
Facts can be compared with one another as they involve numbers of lives and deaths of people only.7 With consequentialist ethics we have
P ositive([f5; f1]) c P ositive([f5; f1]) N egative([f5; f1]) c N egative([f5; f1]) J udgementc(move the switch; i) = >
J udgementc(do nothing; i) = ?
Let us assess the nature of both possible decisions:
DecisionN ature(move the switch) = neutral
DecisionN ature(do nothing) = neutral No decision is unacceptable from the deontological viewpoint: 8d; DecisionN ature(d) > neutral
Consequently = = = = = = (53) (54) (55) (56) (57) (58) (59) (60) (61) (63) (64) (65) (66) P ositive([f5; f1]) N egative([f5; f1]) ff5g ff1g (47) (48) 2. do nothing: this decision is associated with the train hitting five people. The consequence is : five people dead, one person alive. do nothing: Event(do nothing) = train hits f ive people (49) Consequence(train hits f ive people; i)
P ositive([f5; f1]) N egative([f5; f1]) [f5; f1](50) ff1g ff5g (51) 1. Deontological rule: we have seen above that both decisions are neutral. Therefore both of them satisfy the first rule. 2. Collateral damage rule: move the switch:
N egative([f5; f1])
N egative([f5; f1]) ` = ` ff1g F fp ff5g F fp
Therefore both decisions respect the second rule. 7 For the sake of simplicity in this paper, we will consider that ff5g >c ff1g if f5 is preferred to f1 3. Proportionality rule: we will assume in this context that the death of one person is proportional to the safeguard of the lives of the five other people, and conversely that the death of five people is not proportional to safeguard one life: f1 .p f5 and :(f5 .p f1).
Both the democratic and the elitist proportional criteria (3.4.2) give the same results as sets of facts are composed of one fact. [N egative([f5; f1]) = ff1g] -p [P ositive([f5; f1]) = ff5g] (67) Move the switch is the only decision which respects the proportionality rule. Decision do nothing has same judgements as in the previous case. Let us study the judgements for decision push ”fatman”.
The result in terms of human lives is the same as in the first dilemma. Consequently we have exactly the same judgement.
J udgementc(push "f atman"; i) = >
Let us consider decision nature of push ”fatman” as bad.
DecisionN ature(push "f atman") = bad
J udgementd(push "f atman"; i) = ? 2. Collateral damage rule: push ”fatman”: and
N egative([f5; f at]) = ff atg
f at ` F f5 f5 2 P ositive([f5; f at]) It is because ”fatman” is pushed that the five people are alive. Therefore
J udgementdde(push "f atman"; i) = ? 3. Proportionality rule: if we assume that: Once the judgements are computed, we can analyse the similarities between ethical frameworks. Two frameworks are similar if they have common judgements values on the same decisions compared to the total number of decisions.
Figure 3 is based on three dilemmas (the crazy trolley, the ”fatman” trolley, and another one – UAV vs missile launcher – that is not described here).
We can notice that the consequentialist and deontological frameworks are quite different and that the DDE is close to the two others. This can be explained by the rules of the DDE, which allow this framework to be both deontological (deontological rule) and close to consequentialism (proportionality rule). 6
Because of their own natures, the three ethical frameworks that we have studied do not seem to be appropriate in all situations. For example we have seen that consequentialist ethics does not distinguish between crazy trolley and ”fatman” trolley dilemmas. Moreover the consequentialist preference relation between facts is a partial order, which means that it is not always possible to prefer some facts to others. Consequently judging a decision is sometimes impossible with consequentialist ethics. Furthermore consequentialist preference depends on the context: preferring to feel pain in order to stop the fall of a crystal glass with one’s foot does not mean that you prefer to cut your finger to get back a ring. As far as deontological ethics is concerned, judging the nature of some decisions can be tricky (see 3.3.1). Finally the Doctrine of Double Effect forbids the sacrifice of oneself. Nevertheless if a human life is threatened, shouldn’t the agent’s sacrifice be expected?
This leads us to the idea that one framework alone is not efficient enough to compute an ethical decision. It seems necessary to consider as much ethical frameworks as possible in order to obtain the widest possible view.
The limits of the model lie mainly in the different relations it contains. Indeed, we have not described how orders are assessed. Moreover it may be hardly possible to define an order (i.e. consequentialist preference) between two concepts. On the other hand the model is based on facts that are assumed to be certain, which is quite different in the real world where some effects are uncertain or unexpected. Furthermore, the vector representation raises a classical modelling problem: how to choose state components and their values? The solution we have implemented is to select only facts whose values change as a result of the agent’s decision. 7
The main challenge of our model is to formalize philosophical definitions described with natural language and to translate them in generic concepts that can be easy-to-understand by everyone. The interest of such a work is to get rid of ambiguities in a human/robot, and more broadly human/human, system dialog and to allow an artificial agent to compute ethical considerations by itself. This formalism raises many questions because of ethical concepts themselves (DDE’s proportionality, the good, the evil, etc.). Indeed ethics is not universal, that is why it is impossible to reason on fixed preferences and calculus. Many parameters such as context, agent’s values, agent’s priorities, etc. are involved. Some of those parameters can depend on ”social acceptance”. For example, estimating something negative or positive (or computing a decision nature) can be based on what society thinks about it, as on agent’s values.
Further work will focus on considering other frameworks such as virtue ethics on the one hand and a value system based on a partial order on values on the other hand. Furthermore game theory, voting systems or multicriteria approaches may be worth considering to compare ethical frameworks judgements.
We would like to thank ONERA for providing resources for this work, the EthicAA project team for discussions and advice, and reviewers who gave us relevant remarks.