=Paper= {{Paper |id=Vol-2640/paper_11 |storemode=property |title=Ethically Compliant Planning in Moral Autonomous Systems |pdfUrl=https://ceur-ws.org/Vol-2640/paper_11.pdf |volume=Vol-2640 |authors=Justin Svegliato,Samer Nashed,Shlomo Zilberstein |dblpUrl=https://dblp.org/rec/conf/ijcai/SvegliatoNZ20 }} ==Ethically Compliant Planning in Moral Autonomous Systems== https://ceur-ws.org/Vol-2640/paper_11.pdf

Ethically Compliant Planning in Moral Autonomous Systems
Justin Svegliato and Samer B. Nashed and Shlomo Zilberstein
College of Information and Computer Sciences, University of Massachusetts Amherst
{jsvegliato,snashed,shlomo}@cs.umass.edu

Abstract environments, autonomous systems may encounter unantici-
pated scenarios that lead to behavior that fails to reflect the
In many sequential decision-making problems, eth- intentions of developers or the values of stakeholders [Taylor
ical compliance is enforced by either myopic rule et al., 2016; Hadfield-Menell and Hadfield, 2019].
sets or provisional modifications to the objective Ideally, researchers and practitioners integrating ethical
function. The effect of these strategies is exceed- theories and decision processes could access methods that
ingly difficult to predict, often leading to inad- offer several desirable features. These features include sup-
vertent behavior that can jeopardize the values of port for interpretability and control over behavior with formal
stakeholders. We propose a novel approach for guarantees, nonmyopic reasoning, the acquisition of rules
ethically compliant planning, based on decoupling from a non-technical person, and the application of one or
ethical compliance from task completion within the more ethical theories simultaneously. Ethicists often describe
objective function, that produces optimal policies an ethical theory as a set of moral principles for evaluating
subject to the constraints of an ethical framework. if an action is required, permitted, or prohibited in a given
This paper introduces a formal definition of a moral scenario [Shafer-Landau, 2009]. Given this interpretation, an
autonomous system and its key properties. It also ethical theory can be operationalized in a decision process as
offers a range of ethical framework examples for constraints on the actions of the agent in specific states.
divine command theory, prima facie duties, and In this paper, we propose a novel approach for building
virtue ethics. Finally, it demonstrates the effective- moral autonomous systems that produce an optimal policy to
ness of our approach in a set of autonomous driving a decision-making problem subject to the constraints of an
simulations and a user study of MDP experts. ethical framework. The system models a task with a decision-
making model and models an ethical framework as a moral
principle and an ethical context. While we use MDPs for
1 Introduction the decision-making models in our experiments, our approach
Integrating decision making and ethics in autonomous sys- supports any decision process expressible as a mathematical
tems is challenging due to the diversity and complexity of program. The moral principle is an approximation of an in-
deployment domains and stakeholder value systems. For de- terpretation of an ethical theory that can be represented as a
cision making in the real world, Markov decision processes Boolean function that evaluates whether or not a policy vio-
(MDPs) are a common, general-purpose model because of lates a particular ethical theory. The ethical context contains
their support for long-term, nonmyopic reasoning in fully all of the information necessary to evaluate the moral prin-
observable, stochastic environments. However, MDPs pose ciple. Formally, this system is expressed as an optimization
two additional challenges when generating ethically compli- problem with a set of constraints representing the task and a
ant behavior. First, the complexity of these models often constraint that operationalizes the ethical framework. The so-
obfuscates the effect of the reward function on the behavior lution to the optimization problem is a policy that optimizes
of the agent. Seemingly innocuous adjustments may dras- completing the task while following the ethical framework.
tically change resulting behavior, leading to unpredictabil- We evaluate our approach in two experiments. First, in an
ity [Bostrom, 2016]. Second, and more fundamentally, using autonomous driving simulation, we confirm that our approach
the reward function to model both desirable and undesirable produces optimal behavior while complying with moral re-
behavior often involves incommensurable unit conversions. quirements. Second, in a user study, we find that MDP ex-
For example, an autonomous vehicle with a reward function perts who use our approach require less development time to
that encourages completing a route efficiently and discour- produce policies that have higher rates of ethical compliance
ages driving recklessly blends task completion and ethical compared to modifying the reward function directly.
compliance implicitly. The resulting policy may drive too Our main contributions in this paper are: (1) a formal def-
recklessly if offered enough time savings. Thus, in complex inition of a moral autonomous system and its key properties,
(2) a range of ethical framework examples for divine com-
Copyright c 2020 for this paper by its authors. Use permitted mand theory, prima facie duties, and virtue ethics, and (3)
under Creative Commons License Attribution 4.0 International (CC a set of autonomous driving simulations and a user study of
BY 4.0). MDP experts that shows the effectiveness of our approach.
2 Related Work We are aware of only one other approach that focuses
on proactive ethical governors of policies [Kasenberg and
Autonomous systems attempt to address a range of problems, Scheutz, 2018]. However, since it is specific to norms, it is
are deployed in diverse social contexts, and draw upon a het- unclear how it could support other forms of ethical reason-
erogeneous collection of algorithms. The potential harms ing, such as adherence to a moral principle like utilitarianism
of these systems can be mitigated through many strategies: or deontology. Moreover, both task completion and ethical
(1) abandonment of technologies that are likely to be abused behavior are defined in terms of real-valued norm weights,
when analyzed in a historical context [Browne, 2015], such the coupling of which elides guarantees of ethical behavior.
as facial recognition [Brey, 2004; Introna and Wood, 2004] In contrast, our approach can generate policies that follow
and surveillance of online activity [Burgers and Robinson, arbitrary ethical theories and avoid unpredictable trade-offs
2017; Zimmer, 2008], (2) legal or legislative intervention between task completion and ethical behavior.
that provides oversight and regulation in enough detail to
prevent or discourage malevolent or negligent use [Good- 3 Background
man and Flaxman, 2017; Desai and Kroll, 2017; Raymond
and Shackelford, 2013; Scherer, 2015], including meta- A Markov decision process (MDP) is a decision-making
regulation [Pasquale, 2017], and (3) algorithmic advances model for reasoning in fully observable, stochastic environ-
that improve accuracy and interpretability. Though these ments [Bellman, 1966]. An MDP can be described as a tuple
strategies will continue to play important roles in the future, hS, A, T, R, di, where S is a finite set of states, A is a finite set
our approach focuses on a fourth strategy that reduces the op- of actions, T : S × A × S → [0, 1] represents the probability
portunity for error during design and development. of reaching a state s0 ∈ S after performing an action a ∈ A in
Recently, various principles [Boden et al., 2017], guide- a state s ∈ S, R : S ×A×S → R represents the expected im-
lines [Robertson et al., 2019], and standards [Adamson et al., mediate reward of reaching a state s0 ∈ S after performing an
2019] have been proposed for the design and development of action a ∈ A in a state s ∈ S, and d : S → [0, 1] represents
autonomous systems. Although these are essential for pro- the probability of starting in a state s ∈ S. A solution to an
moting the values of stakeholders throughout the design pro- MDP is a policy π : S → A indicating that an action π(s) ∈
cess, these initiatives do not offer developers enough detail A should be performed in a state s ∈ S. A policy π induces
to operationalize ethical frameworks in autonomous systems. a value function V π : S → R representing the expected dis-
In fact, implicit ethical systems, which satisfy moral require- counted cumulative reward V π (s) ∈ R for each state s ∈ S
ments through careful design, may not always produce desir- given a discount factor 0 ≤ γ < 1. An optimal policy π ∗
able behavior [Moor, 2006]. Many autonomous systems must maximizes the expected discounted cumulative reward for ev-
therefore be capable of explicit moral reasoning [Dignum et ery state s ∈ S by satisfying the Bellman optimality equation
V ∗ (s) = maxa∈A s0 ∈S T (s, a, s0 )[R(s, a, s0 ) + γV ∗ (s0 )].
P
al., 2018; Bench-Capon and Modgil, 2017].
A common approach for finding an optimal policy ex-
Engineering efforts to develop explicit autonomous moral
presses the optimization problem as a linear program in either
agents take two forms [Allen et al., 2005]. Bottom-up ap-
the primal form or the dual form [Manne, 1960]. In this pa-
proaches generate ethical behavior naturally through learn-
per, we propose ethical frameworks that naturally map to the
ing or evolution [Anderson et al., 2017; Shaw et al., 2018].
dual form. The dual form maximizes a set of occupancy mea-
While this is theoretically compelling given the natural evolu-
sures µsa for the discounted number of times an action a ∈ A
tion of ethical ideas in human society, the instability and lack
is performed in a state s ∈ S subject to a set of constraints
of interpretability are major drawbacks. Instead, we choose
that maintain consistent and nonnegative occupancy.
a top-down approach in which prescriptive rules describing
moral behavior are provided to the agent. Many top-down XX X
approaches use logics, such as deontic logic [van der Torre, max µsa R(s, a, s0 )
µ
2003; Bringsjord et al., 2006], temporal logic [Wooldridge s∈S a∈A s0 ∈S
X 0 XX
and Van Der Hoek, 2005; Atkinson and Bench-Capon, 2006], s.t. µa0 = d(s0 ) + γ
s
T (s, a, s0 )µsa ∀s0
or Answer Set Programming [Berreby et al., 2015]. Some 0
a ∈A s∈S a∈A
even propose a form of metareasoning over logics [Bringsjord
µsa ≥ 0 ∀s, a
et al., 2011]. However, as systems become more complex
and operate in stochastic, partially observable environments,
norm specification represented by logics will become increas- 4 Moral Autonomous Systems
ingly challenging [Abel et al., 2016]. We propose a novel approach for building moral autonomous
A common approach for handling these environments em- systems that decouples ethical compliance from task comple-
ploys an ethical governor that reasons online about whether tion. The system completes a task by using a decision-making
an action is required, permitted, or prohibited [Arkin, 2008]. model and follows an ethical framework by adhering to a
Applications include eldercare [Shim et al., 2017] and phys- moral principle within an ethical context. We describe these
ical safety [Vanderelst and Winfield, 2018; Winfield et al., three components of a moral autonomous system below.
2014]. These methods use reactive ethical governors that only First, the system has a decision-making model that de-
consider a single action at a time as the situation presents the scribes the information needed to complete the task. For ex-
agent with the opportunity to act. In contrast, our approach is ample, a self-driving vehicle could have a decision-making
nonmyopic because it considers entire sequences of actions. model that includes a map of a city [Nashed et al., 2018;
Svegliato et al., 2019]. An engineer must select a represen- Moral
tation for the decision-making model that reflects the proper- Optimal Moral
Policies Amoral Policy
ties of the task. For many tasks, an MDP, a decision process Policies
Π ∗

that assumes full observability, can be used easily. However, Π

for more complex tasks with partial observability, start and
goal states, or multiple agents, it is possible to use a decision Immoral Optimal
Moral Moral Policy
process like a partially observable MDP, a stochastic shortest Policies
Policies ∗
path problem, or a decentralized MDP instead. In short, the Π¬
Π
decision-making model is an amoral, descriptive model for
completing the task but not following the ethical framework.
Next, the system has an ethical context that describes the Figure 1: A simple view of the goal of a moral autonomous system
information required to follow the ethical framework. For and a standard autonomous system in terms of the space of policies
instance, an autonomous vehicle could have an ethical con-
text that includes any details related to inconsiderate and haz-
However, the goal of a standard autonomous system has typi-
ardous driving that permit speeding on a highway in some
cally been to find an optimal amoral policy, π ∗ ∈ Π, that only
scenarios but never in a school zone or near a crosswalk [Van-
completes its task without following any ethical framework.
derelst and Winfield, 2018]. Similar to the decision-making
Figure 1 depicts the goal of a moral autonomous system
model, an ethicist must select a representation for the ethical
and a standard autonomous system. For a moral principle
context that informs the fundamental principles of the ethical
ρ, the space of policies Π is partitioned into a moral region
framework. While the ethical context can be represented as a
Πρ and an immoral region Π¬ρ . The moral region contains
tuple of different values, sets, and functions, the specification
the optimal moral policy πρ∗ ∈ Π of the moral autonomous
of the tuple depends on the ethical framework. In summary,
system while the immoral region contains the optimal amoral
the ethical context is a moral, descriptive model for following
policy π ∗ ∈ Π of the standard autonomous system. In gen-
the ethical framework but not completing the task.
eral, the optimal amoral policy π ∗ ∈ Π can be contained by
Finally, the system has a moral principle that evaluates the
either the moral region Πρ or the immoral region Π¬ρ .
morality of a policy of the decision-making model within the
A moral autonomous system may follow an ethical frame-
ethical context by considering the information that describes
work that adversely impacts completing its task. Engineers
how to complete the task and follow the ethical framework.
and ethicists can assess the cost of this impact by calculating
As an illustration, a moral principle could require a policy
the maximum difference across all states between the value
to maximize the overall well-being of the moral community
function of the optimal moral policy and the value function
in utilitarianism [Bentham, 1789; Mill, 1895] or universal-
of the optimal amoral policy. We define this cost below.
ize to the moral community without contradiction in Kantian-
ism [Kant and Schneewind, 2002]. Given a decision-making Definition 4. Given the optimal moral policy πρ∗ ∈ Π and the
model and an ethical context, a moral principle can be ex- optimal amoral policy π ∗ ∈ Π, the price of morality, ψ, can
∗ ∗
pressed as a function that maps a policy to its moral status. be represented by the expression ψ = kV πρ − V π k∞ .
Definition 1. A moral principle, ρ : Π → B, represents A moral autonomous system may even follow an ethi-
whether a policy π ∈ Π of a decision-making model D is cal framework that is mutually exclusive with completing its
moral or immoral within an ethical context E. task. In this situation, engineers and ethicists should recon-
By putting all of these attributes together, we provide a for- sider the moral implications of the system and could augment
mal description of a moral autonomous system as follows. the decision-making model or adjust the ethical context if
Definition 2. A moral autonomous system, hD, E, ρi, com- deemed safe. Naturally, depending on whether or not there
pletes a task by using a decision-making model D and fol- is a solution to the optimization problem, the system can be
lows an ethical framework by adhering to a moral principle considered either feasible or infeasible as follows.
ρ within an ethical context E. Definition 5. A moral autonomous system is realizable if and
A moral autonomous system has the goal of finding an op- only if there exists a policy π ∈ Π such that its moral princi-
timal policy that completes its task and follows its ethical ple ρ(π) is satisfied. Otherwise, the system is unrealizable.
framework. This can be expressed as an optimization prob- It is natural to find the optimal moral policy by solving the
lem solving for a policy in the space of policies that maxi- optimization problem of a moral autonomous system using
mizes the value of the policy subject to the constraint that the mathematical programming. This process involves four steps.
policy satisfies the moral principle. We define the goal of a First, the moral principle can be mapped to a moral constraint
moral autonomous system as follows. in terms of the occupancy measures of a policy. We show that
Definition 3. The goal of a moral autonomous system is to this mapping can always be performed as follows.
find an optimal moral policy, πρ∗ ∈ Π, by solving for a policy
π ∈ Π that maximizes a value function V π subject to a moral Theorem 1. A moral principle, ρ : Π → B, can be expressed
principle ρ(π) in the following optimization problem. as a moral constraint cρ (µ) in terms of the matrix of occu-
pancy measures µ for a given policy π ∈ Π.
maximize V π
π∈Π Proof (Sketch) 1. We start with a moral principle ρ(π) using
subject to ρ(π) a deterministic or stochastic policy π(s) or π(a|s). Recall
Moral Constraint Type Conjunctions Operations Computations
cρF (µ) = ∧s∈S,a∈A,f ∈F T (s, a, f )µsa = 0

Linear |S||A||F | 2 2|S||A||F |
P s P 0 P 0
cρ∆ (µ) = s∈S,a∈A µa s0 ∈S T (s, a, s ) δ∈∆ 0 φ(δ, s ) ≤ τ
s
Linear 1 3|S||A||S||∆| + 1 3|S||A||S||∆| + 1
cρM (µ) = ∧s∈S,a∈A µsa ≤ [α(s, a)]

Linear |S||A| 1 + 3L|M| |S||A| 1 + 3L|M|

Table 1: The moral constraints that have been derived from the moral principle of each ethical framework

that the discounted number of times that an action a ∈ A is Table 1 offers the moral constraints that have been derived
performed in a state s ∈ S is an occupancy measure µsa . Ob- from the moral principle of each ethical framework. For each
serve that the discountedP number of times that a state s ∈ S is moral constraint, there are several columns that describe its
s
s P a∈A µsa. A policy
visited is the expression π(s) or π(a|s) is computational tractability. The Type column lists whether
thus arg maxa∈A µa / a∈A µa or µa / a∈A µsa . There-
s the moral constraint is linear or nonlinear with respect to the
P
fore, by substitution, we end with a moral constraint cρ (µ). occupancy measures of a policy. The Conjunctions column
states the number of logical conjunctions that compose the
Second, the moral principle can be classified as either lin- moral constraint. The Operations column indicates an upper
ear or nonlinear depending on the form of its moral constraint. bound on the number of arithmetic, comparison, and logical
If the moral constraint is linear in the occupancy measures of operations that must be performed for each logical conjunc-
a policy, the moral principle is linear. Otherwise, the moral tion. The Computations column contains an upper bound on
principle is nonlinear. We formalize this property below. the number of computations that must be executed for the
Definition 6. A moral principle, ρ : Π → B, is linear if it can moral constraint to evaluate the moral status of a policy.
be expressed as a moral constraint cρ (µ) that is linear with We now present a set of simplified ethical frameworks.
respect to the matrix of occupancy measures µ for a given They are not definitive and do not capture all nuances of eth-
policy π ∈ Π. Otherwise, the moral principle is nonlinear. ical theories. Their purpose is to tractably operationalize an
Third, the optimization problem can be described as a ethical theory within a decision process. We encourage the
mathematical program. For task completion, following the development of more complex ethical frameworks that reflect
linear program of an MDP in the dual form, the program the depth of different ethical theories, including those below.
maximizes a set of occupancy measures µsa for the discounted
number of times an action a ∈ A is performed in a state s ∈ S 5.1 Divine Command Theory
subject to a set of constraints that maintain consistent and Divine command theory (DCT), a monistic, absolutist eth-
nonnegative occupancy. However, for ethical compliance, the ical theory, holds that the morality of an action is based
program has a moral constraint cρ (µ) derived from the moral on whether a divine entity commands or forbids that ac-
principle ρ(µ) given a matrix of occupancy measures µ. tion [Idziak, 1979; Quinn, 2013]. We consider a simplified
Fourth, the mathematical program can be solved to find ethical framework in which a moral autonomous system uses
the optimal moral policy. Given a linear moral principle, it a policy that selects actions that have a nil probability of tran-
can be solved using techniques designed for linear program- sitioning to any forbidden state [Mouaddib et al., 2015]
ming, such as the simplex method or the criss-cross algo-
rithm [Bertsimas and Tsitsiklis, 1997]. However, given a non- Definition 7. A DCT ethical context, EF , can represented by
linear moral principle, it can be solved using techniques de- a tuple, EF = hFi, where F is a set of forbidden states.
signed for nonlinear programming instead [Bertsekas, 1997]. Definition 8. A DCT moral principle, ρF , can be expressed
Note that, while we use the dual form of the linear program of as the following equation:
an MDP, this process can also be used with the primal form. ^
ρF (π) = T (s, π(s), f ) = 0 .
5 Ethical Frameworks s∈S,f ∈F

In this section, we offer a range of ethical framework exam-
ples that can be used to build a moral autonomous system. 5.2 Prima Facie Duties
Each ethical framework is influenced by an interpretation of Prima facie duties (PFD), a pluralistic, nonabsolutist ethi-
an ethical theory in moral philosophy [Shafer-Landau, 2009]. cal theory, holds that the morality of an action is based on
During the design of an ethical framework, ethicists and en- whether that action fulfills fundamental moral duties that can
gineers select a representation for the ethical context and the contradict each other [Ross, 1930; Morreau, 1996]. We con-
moral principle. This involves choosing the contextual details sider a simplified ethical framework in which a moral au-
of the ethical context and the logical structure of the moral tonomous system uses a policy that selects actions that do not
principle that most accurately describe the capabilities of the neglect duties of different penalties within some tolerance.
agent, the effect of its actions on its environment, and the
Definition 9. A PFD ethical context, E∆ , can be represented
moral implications of its behavior. In short, an ethical frame-
by a tuple, E∆ = h∆, φ, τ i, where
work, composed of an ethical context and a moral principle,
is an approximation of an interpretation of an ethical theory. • ∆ is a set of duties,
• φ : ∆ × S → R+ is a penalty function that represents or heavy with a probability Pr(Θ = θ). After the vehicle
the expected immediate penalty for neglecting a duty δ ∈ observes the pedestrian traffic θ ∈ Θ, the vehicle accelerates
∆ in a state s ∈ S, and to a speed σ ∈ Σ that reflects either a low, normal, or high
• τ ∈ R+ is a tolerance. speed under, at, or above the speed limit. To drive along the
road ω ∈ Ω from the current location λ ∈ Λ to the next
Definition 10. A PFD moral principle, ρ∆ , can be expressed location λ0 ∈ Λ, the vehicle cruises at the speed σ ∈ Σ. Note
as the following equation: that this is repeated until arriving at the goal location λg ∈ Λ.
X We represent the decision-making model of a navigation
ρ∆ (π) = d(s)J π (s) ≤ τ. task by an MDP D = hS, A, T, R, di. The set of states S =
s∈S
SΛ ∪ SΩ has a set of location states SΛ for being at a location
The expected cumulative penalty, J π : S → R, is below: λ ∈ Λ and a set of road states SΩ for being on a road ω ∈ Ω
X X of a type υ ∈ Υ with a pedestrian traffic θ ∈ Θ at a speed
J π (s) = T (s, π(s), s0 ) φ(δ, s0 ) + J π (s0 ) ,

σ ∈ Σ. The set of actions A = AΩ ∪ AΣ ∪ {⊗, } has a
s0 ∈S δ∈∆s0 set of turn actions AΩ for turning onto a road ω ∈ Ω, a set
of accelerate actions AΣ for accelerating to a speed σ ∈ Σ, a
where ∆s0 is the set of duties neglected in a state s0 ∈ S. stay action ⊗, and a cruise action . The transition function
T : S × A × S → [0, 1] reflects the dynamics of a turn action
5.3 Virtue Ethics a ∈ AΩ and a stay action ⊗ in a location state λ ∈ SΛ or an
Virtue ethics (VE), a monistic, absolutist ethical theory, holds accelerate action a ∈ AΣ and a cruise action in a road state
that the morality of an action is based on whether a virtuous s ∈ SΩ (with a self-loop for any invalid action a ∈ A). The
person who acts in character performs that action in a similar reward function R : S × A × S → R reflects the duration
situation [Anscombe, 1958; Hursthouse, 1999]. We consider of a turn action a ∈ AΩ from a location state SΛ to a road
a simplified ethical framework in which a moral autonomous state s ∈ SΩ , a stay action ⊗ at a location state λ ∈ SΛ , an
system uses a policy that selects actions that align with any accelerate action a ∈ AΣ at a road state s ∈ SΩ , and a cruise
moral trajectory performed by a moral exemplar. action from a road state s ∈ SΩ to a location state SΛ (with
Definition 11. A VE ethical context, EM , can represented by an infinite duration for any invalid action a ∈ A and a nil
a tuple, EM = hMi, where M is a set of moral trajectories. duration for a stay action ⊗ at a state s ∈ S that represents the
goal location λg ∈ Λ). The start state function d : S → [0, 1]
Definition 12. A VE moral principle, ρM , can be expressed
has unit probability at a state s ∈ S that represents the start
as the following equation:
location λ0 ∈ Λ and nil probability at every other state s ∈ S.
^
ρM (π) = α(s, π(s)).
6.2 Ethical Compliance
s∈S
The vehicle must follow one of the ethical frameworks. First,
The alignment function, α : S × A → B, is below: the vehicle can follow DCT with forbidden states comprised

α(s, a) = ∃m∈M,0≤i≤` s = m(si ) ∧ a = m(ai ) , of hazardous states H and inconsiderate states I. Hazardous
states H contain any road state at high speed while incon-
where m(si ) and m(ai ) are the ith state and the ith action of siderate states I contain any road state at normal speed with
a moral trajectory m = hs0 , a0 , s1 , a1 , . . . , s`−1 , a`−1 , s` i of heavy pedestrian traffic. With the DCT moral principle ρF ,
length ` ≤ L bounded by a maximum length L. we represent the DCT ethical context by a tuple, EF = hFi,
where F = H ∪ I is the set of forbidden states.
Next, the vehicle can follow PFD with duties comprised of
6 Autonomous Driving smooth operation δ1 and careful operation δ2 . Smooth oper-
We turn to an application of moral autonomy to autonomous ation δ1 is neglected in any road state at low speed with light
driving. A moral self-driving vehicle must complete a nav- pedestrian traffic while careful operation δ2 is neglected in
igation task by driving from an origin to a destination in a any road state at high speed or at normal speed with heavy
city. However, to follow an ethical framework, the moral self- pedestrian traffic. When smooth operation δ1 and careful op-
driving vehicle must adjust its route and speed depending on eration δ2 are neglected, they incur a low and high penalty
the type and pedestrian traffic of each road. We describe how that changes with any pedestrian traffic. Neglecting duties
to separate task completion and ethical compliance below. is permitted until a limit . With the PFD moral princi-
ple ρ∆ , we represent the PFD ethical context by a tuple,
6.1 Task Completion E∆ = h∆, φ, τ i, where ∆ = {δ1 , δ2 } is the set of duties,
The vehicle must complete a navigation task by driving from φ : ∆ × S → R+ is the penalty function that represents the
a start location λ0 ∈ Λ to a goal location λg ∈ Λ along a expected immediate penalty for neglecting smooth operation
set of roads Ω in a city with a set of locations Λ. At each δ1 ∈ ∆ and careful operation δ2 ∈ ∆ in a state s ∈ S with a
location λ ∈ Λ, the vehicle must turn onto a road ω ∈ Ω. pedestrian traffic θ ∈ Θ, and τ = is the tolerance.
Each road ω ∈ Ω is a type υ ∈ Υ that indicates either a city Finally, the vehicle can follow VE with moral trajectories
street, county road, or highway with a low, medium, or high comprised of cautious trajectories C and proactive trajectories
speed limit. Once the vehicle turns onto a road ω ∈ Ω, the P. Cautious trajectories C exemplify driving on any road state
vehicle observes the pedestrian traffic θ ∈ Θ as either light at normal speed with light pedestrian traffic or at low speed
Figure 2: A city with different places connected by city streets, county roads, and highways

with heavy pedestrian traffic while proactive trajectories P
exemplify avoiding any highway road states and a set of pop-
ulated location states. With the VE moral principle ρM , we
represent the VE ethical context by a tuple, EM = hMi,
where M = C ∪ P is the set of moral trajectories.

7 Experiments
We now demonstrate that the application of moral autonomy
to autonomous driving is effective in a set of simulations and
a user study. In the set of simulations, an amoral self-driving
vehicle and a moral self-driving vehicle that follows different
ethical frameworks both complete a set of navigation tasks.
Figure 3: An agent completes a task and follows an ethical frame-
Each navigation task can use a different start location λ0 ∈ work in a grid world with a blue amoral path and a green moral path.
Λ and goal location λg ∈ Λ based on the city in Figure 2. The
speed limits of city streets, county roads, and highways are
25, 45, and 75 mph. The probability Pr(Θ = θ) of observing traffic aside from driving on the first road at normal or high
light or heavy pedestrian traffic θ ∈ Θ is 0.8 and 0.2. A speed with some probability for light pedestrian traffic and
low, normal, and high speed is 10 mph under, at, and 10 mph at normal speed for heavy pedestrian traffic due to the toler-
above the speed limit. Turning onto a road ω ∈ Ω from a ance. For VE, the vehicle drives at low or normal speed based
location λ ∈ Λ requires 5 sec. Accelerating 10 mph requires on pedestrian traffic but drives a different route to avoid the
2 sec. Cruising requires a time equal to the distance of the highway road states and the set of populated location states.
road ω ∈ Ω divided by the speed σ ∈ Σ. Staying at a location In the user study, planning and robotics experts had to com-
λ ∈ Λ other than the goal location λg ∈ Λ requires 120 sec. plete two tasks in a randomized order. In both tasks, devel-
Each ethical framework can use different settings. For opers were given a complete decision-making model for nav-
DCT, the forbidden states F can be just hazardous states H or igating efficiently around the example city and had to enforce
both hazardous states H and inconsiderate states I. For PFD, the following moral requirements. The agent should drive at
the tolerance τ = can be the limit = 3, = 6, or = 9. high speed with light pedestrian traffic or at normal speed
For VE, the moral trajectories can be just cautious trajectories with heavy pedestrian traffic at most once in expectation but
C or both cautious trajectories C and proactive trajectories P should never drive at high speed with heavy pedestrian traf-
that avoid any highway road states and a set of populated lo- fic. In one task, developers were asked to achieve the desired
cation states that contains the School and College locations. behavior by modifying the existing decision-making model,
Table 2 highlights that the price of morality incurred by an MDP, by changing its reward function or transition func-
the behavior of the agent is appropriate given each ethical tion. In the other task, developers were asked to achieve the
framework. Naturally, the amoral self-driving vehicle does same desired behavior but by defining the ethical context for
not incur a price of morality. The moral self-driving vehicle, the prima facie duties ethical framework.
however, incurs a price of morality that increases with more Figure 4 illustrates that our method led to better policies
forbidden states for DCT, decreases with more tolerance for than the other method. In our method, all policies satisfy the
PFD, and increases with more moral trajectories for VE. requirements and optimize the navigation task with exactly
Figure 5 indicates that the behavior of the agent is correct one violation. However, in the other method, the majority of
given each ethical framework. The amoral self-driving vehi- policies fail to optimize the navigation task or even satisfy
cle drives the shortest route at high speed. The moral self- the requirements: aggressive policies in the upper right cor-
driving vehicle, however, differs for each ethical framework. ner are faster but immoral while conservative policies in the
For DCT, the vehicle drives the shortest route at low or normal lower left corner are slower but moral. It is also encourag-
speed based on pedestrian traffic. For PFD, the vehicle drives ing that our method (24 min) had a significantly lower mean
the shortest route at low or normal speed based on pedestrian development time than the other method (45 min).
Ethics Setting TASK 1 (%) TASK 2 (%) TASK 3 (%) Light Light Light

None — 0 0 0 (a) Gray Street Train Service Road Gas Sunrise Highway
Home Office
Station Station
H 14.55 15.33 20.12
DCT H∪I 21.13 22.35 27.92 Heavy Heavy Heavy

=3 16.07 16.52 24.30
PFD =6 11.96 11.80 21.37 Light Light Light
=9 7.91 7.15 18.87
(b) Home
Gray Street Train Service Road Gas Sunrise Highway
Office
C 21.13 22.35 27.92 Station Station
VE C∪P 40.89 94.43 30.28
Heavy Heavy Heavy

Table 2: The price of morality as a percentage of the value of the Light Light Light
optimal amoral policy for all vehicle options on each navigation task
(c) Home
Gray Street Train Service Road Gas Sunrise Highway
Office
Station Station
Violations

3 PFD Implementation
Heavy Heavy Heavy
MDP Modification
2
Light Light Light Light Light

1
(d) Home
Gray Train Merrick Pizza Pleasant Grocery State Town Oak
Office
Street Station Road Place Street Store Street Hall Road
0 Relative Efficiency (%)
20 15 10 5 0 5 10 15 20 Heavy Heavy Heavy Heavy Heavy

Figure 4: The results of the user study. For each exercise and loca-
tion in the city, there is a point that denotes the resulting policy. For Figure 5: The optimal policies for select vehicle options on a navi-
that policy, the horizontal axis is its time savings relative to the pol- gation task with (a) no ethical framework, (b) DCT with H ∪ I, (c)
icy from the opposing exercise while the vertical axis is its number PFD with = 9, and (d) VE with C ∪ P. A blue node denotes a
of violations. The moral and immoral regions are in green and red. location and a gray node denotes pedestrian traffic. With a thickness
for likelihood, a gray line denotes turning onto a road and an orange,
green, or purple line denotes cruising at high, normal, or low speed.
Our open source library, Morality.js, which is available on
the website https://www.moralityjs.com with the
customizable grid world environment dashboard seen in Fig- [Atkinson and Bench-Capon, 2006] Katie Atkinson and Trevor
ure 3, was used for all experiments [Svegliato et al., 2020]. Bench-Capon. Addressing moral problems through practical rea-
soning. In International Workshop on Deontic Logic and Artifi-
cial Normative Systems. Springer, 2006.
Acknowledgments
[Bellman, 1966] Richard Bellman. Dynamic programming. Sci-
This work was supported in part by an NSF Graduate Re- ence, 1966.
search Fellowship DGE-1451512 and the NSF grants IIS-
1724101 and IIS-1813490. [Bench-Capon and Modgil, 2017] Trevor Bench-Capon and Sanjay
Modgil. Norms and value based reasoning: justifying compliance
and violation. Artificial Intelligence and Law, 2017.
References
[Bentham, 1789] Jeremy Bentham. An introduction to the princi-
[Abel et al., 2016] David Abel, James MacGlashan, and Michael L ples of morals. London: Athlone, 1789.
Littman. Reinforcement learning as a framework for ethical de-
cisions. In AAAI Workshop on AI, Ethics, and Society, 2016. [Berreby et al., 2015] Fiona Berreby, Gauvain Bourgne, and Jean-
Gabriel Ganascia. Modelling moral reasoning and ethical respon-
[Adamson et al., 2019] Greg Adamson, John C Havens, and Raja
sibility with logic programming. In Logic for Programming, Ar-
Chatila. Designing a value-driven future for ethical autonomous tificial Intelligence, and Reasoning. Springer, 2015.
and intelligent systems. IEEE, 2019.
[Allen et al., 2005] Colin Allen, Iva Smit, and Wendell Wallach. [Bertsekas, 1997] Dimitri P Bertsekas. Nonlinear programming.
Artificial morality: Top-down, bottom-up, and hybrid ap- Journal of the Operational Research Society, 1997.
proaches. Ethics and Information Technology, 2005. [Bertsimas and Tsitsiklis, 1997] Dimitris Bertsimas and John N
[Anderson et al., 2017] Michael Anderson, Susan L Anderson, and Tsitsiklis. Introduction to linear optimization. Athena Scientific
Vincent Berenz. A value driven agent: An instantiation of a case- Belmont, MA, 1997.
supported principle-based behavior paradigm. In AAAI Workshop [Boden et al., 2017] Margaret Boden, Joanna Bryson, Darwin
on AI, Ethics, and Society, 2017. Caldwell, Kerstin Dautenhahn, Lilian Edwards, Sarah Kember,
[Anscombe, 1958] Gertrude Elizabeth Margaret Anscombe. Mod- Paul Newman, Vivienne Parry, Geoff Pegman, Tom Rodden,
ern moral philosophy. Philosophy, 1958. et al. Principles of robotics: Regulating robots in the real world.
Connection Science, 2017.
[Arkin, 2008] Ronald C Arkin. Governing lethal behavior: Embed-
ding ethics in a hybrid deliberative/reactive robot architecture. In [Bostrom, 2016] Nick Bostrom. Superintelligence: Paths, dangers,
3rd ACM/IEEE International Conference on Human Robot Inter- strategies. Science Fiction and Philosophy: From Time Travel to
action. ACM, 2008. Superintelligence, 2016.
[Brey, 2004] Philip Brey. Ethical aspects of facial recognition sys- identification of autonomous vehicles. In IEEE International
tems in public places. Journal of Information, Communication Conference on Robotics and Automation, 2018.
and Ethics in Society, 2004. [Pasquale, 2017] Frank Pasquale. Toward a fourth law of robotics:
[Bringsjord et al., 2006] Selmer Bringsjord, Konstantine Ark- Preserving attribution, responsibility, and explainability in an al-
oudas, and Paul Bello. Toward a general logicist methodology for gorithmic society. Ohio State Law Journal, 2017.
engineering ethically correct robots. Intelligent Systems, 2006.
[Quinn, 2013] Philip L Quinn. Divine command theory. The Black-
[Bringsjord et al., 2011] Selmer Bringsjord, Joshua Taylor, Bram well Guide to Ethical Theory, 2013.
Van Heuveln, Konstantine Arkoudas, Micah Clark, and Ralph
Wojtowicz. Piagetian roboethics via category theory: Moving [Raymond and Shackelford, 2013] Anjanette H Raymond and
beyond mere formal operations to engineer robots whose deci- Scott J Shackelford. Technology, ethics, and access to justice:
sions are guaranteed to be ethically correct. In Machine Ethics. should an algorithm be deciding your case. Michigan Journal
Cambridge University Press, 2011. International Law, 2013.
[Browne, 2015] Simone Browne. Dark matters: On the surveil- [Robertson et al., 2019] Lindsay J Robertson, Roba Abbas, Gursel
lance of blackness. Duke University Press, 2015. Alici, Albert Munoz, and Katina Michael. Engineering-based
design methodology for embedding ethics in robots. IEEE, 2019.
[Burgers and Robinson, 2017] Tobias Burgers and David RS
Robinson. Networked authoritarianism is on the rise. Sicherheit [Ross, 1930] William D Ross. The right and the good. Oxford
und Frieden, 2017. University Press, 1930.
[Desai and Kroll, 2017] Deven R Desai and Joshua A Kroll. Trust [Scherer, 2015] Matthew U Scherer. Regulating artificial intelli-
but verify: A guide to algorithms and the law. Harvard Journal gence systems: Risks, challenges, competencies, and strategies.
of Law & Technology, 2017. Harvard Journal of Law & Technology, 2015.
[Dignum et al., 2018] Virginia Dignum, Matteo Baldoni, Cristina [Shafer-Landau, 2009] Russ Shafer-Landau. The fundamentals of
Baroglio, Maurizio Caon, Raja Chatila, Louise Dennis, Gonzalo ethics. Oxford University Press, 2009.
Génova, Galit Haim, Malte S Kließ, Maite Lopez-Sanchez, et al. [Shaw et al., 2018] Nolan P Shaw, Andreas Stöckel, Ryan W Orr,
Ethics by design: necessity or curse? In AAAI/ACM Conference
Thomas F Lidbetter, and Robin Cohen. Towards provably moral
on AI, Ethics, and Society, 2018.
AI agents in bottom-up learning frameworks. In AAAI/ACM Con-
[Goodman and Flaxman, 2017] Bryce Goodman and Seth Flax- ference on AI, Ethics, and Society, 2018.
man. European union regulations on algorithmic decision-
[Shim et al., 2017] Jaeeun Shim, Ronald Arkin, and Michael Pet-
making and a “right to explanation”. AI Magazine, 2017.
tinatti. An intervening ethical governor for a robot mediator in
[Hadfield-Menell and Hadfield, 2019] Dylan Hadfield-Menell and patient-caregiver relationship. In IEEE International Conference
Gillian K Hadfield. Incomplete contracting and AI alignment. on Robotics and Automation, 2017.
In AAAI/ACM Conference on AI, Ethics, and Society, 2019.
[Svegliato et al., 2019] Justin Svegliato, Kyle Hollins Wray, Ste-
[Hursthouse, 1999] Rosalind Hursthouse. On virtue ethics. Oxford fan J Witwicki, Joydeep Biswas, and Shlomo Zilberstein. Belief
University Press, 1999. space metareasoning for exception recovery. In IEEE/RSJ Inter-
[Idziak, 1979] Janine Marie Idziak. Divine command morality. Ed- national Conference on Intelligent Robots and Systems, 2019.
win Mellen Press, 1979. [Svegliato et al., 2020] Justin Svegliato, Samer Nashed, and
[Introna and Wood, 2004] Lucas Introna and David Wood. Pictur- Shlomo Zilberstein. An integrated approach to moral au-
ing algorithmic surveillance: The politics of facial recognition tonomous systems. In 24th European Conference on Artificial
systems. Surveillance & Society, 2004. Intelligence, 2020.
[Kant and Schneewind, 2002] Immanuel Kant and Jerome B [Taylor et al., 2016] Jessica Taylor, Eliezer Yudkowsky, Patrick
Schneewind. Groundwork for the metaphysics of morals. Yale LaVictoire, and Andrew Critch. Alignment for advanced ma-
University Press, 2002. chine learning systems. Machine Intelligence Research Institute,
[Kasenberg and Scheutz, 2018] Daniel Kasenberg and Matthias 2016.
Scheutz. Norm conflict resolution in stochastic domains. In 32nd [van der Torre, 2003] Leendert van der Torre. Contextual deontic
AAAI Conference on Artificial Intelligence, 2018. logic: Normative agents, violations and independence. Annals of
[Manne, 1960] Alan S Manne. Linear programming and sequential Mathematics and Artificial Intelligence, 2003.
decisions. Management Science, 1960. [Vanderelst and Winfield, 2018] Dieter Vanderelst and Alan Win-
[Mill, 1895] John Stuart Mill. Utilitarianism. Longmans, Green field. An architecture for ethical robots inspired by the simulation
and Company, 1895. theory of cognition. Cognitive Systems Research, 2018.
[Moor, 2006] James H Moor. The nature, importance, and difficulty [Winfield et al., 2014] Alan FT Winfield, Christian Blum, and
of machine ethics. Intelligent Systems, 2006. Wenguo Liu. Towards an ethical robot: internal models, con-
sequences and ethical action selection. In Conference Towards
[Morreau, 1996] Michael Morreau. Prima facie and seeming duties.
Autonomous Robotic Systems. Springer, 2014.
Studia Logica, 1996.
[Wooldridge and Van Der Hoek, 2005] Michael Wooldridge and
[Mouaddib et al., 2015] Abdel-Illah Mouaddib, Laurent Jean-
Wiebe Van Der Hoek. On obligations and normative ability: An
pierre, and Shlomo Zilberstein. Handling advice in mdps for
analysis of the social contract. Journal of Applied Logic, 2005.
semi-autonomous systems. In ICAPS Workshop on Planning and
Robotics, Jerusalem, Israel, 2015. [Zimmer, 2008] Michael Zimmer. The gaze of the perfect search
[Nashed et al., 2018] Samer B Nashed, David M Ilstrup, and Joy- engine: Google as an infrastructure of dataveillance. In Web
deep Biswas. Localization under topological uncertainty for lane Search. Springer, 2008.