In many sequential decision-making problems, ethical compliance is enforced by either myopic rule sets or provisional modifications to the objective function. The effect of these strategies is exceedingly difficult to predict, often leading to inadvertent behavior that can jeopardize the values of stakeholders. We propose a novel approach for ethically compliant planning, based on decoupling ethical compliance from task completion within the objective function, that produces optimal policies subject to the constraints of an ethical framework. This paper introduces a formal definition of a moral autonomous system and its key properties. It also offers a range of ethical framework examples for divine command theory, prima facie duties, and virtue ethics. Finally, it demonstrates the effectiveness of our approach in a set of autonomous driving simulations and a user study of MDP experts.
Integrating decision making and ethics in autonomous systems is challenging due to the diversity and complexity of deployment domains and stakeholder value systems. For decision making in the real world, Markov decision processes (MDPs) are a common, general-purpose model because of their support for long-term, nonmyopic reasoning in fully observable, stochastic environments. However, MDPs pose two additional challenges when generating ethically compliant behavior. First, the complexity of these models often obfuscates the effect of the reward function on the behavior of the agent. Seemingly innocuous adjustments may drastically change resulting behavior, leading to unpredictability [Bostrom, 2016]. Second, and more fundamentally, using the reward function to model both desirable and undesirable behavior often involves incommensurable unit conversions.
that encourages completing a route efficiently and discourages driving recklessly blends task completion and ethical compliance implicitly. The resulting policy may drive too recklessly if offered enough time savings. Thus, in complex environments, autonomous systems may encounter unanticipated scenarios that lead to behavior that fails to reflect the intentions of developers or the values of stakeholders [Taylor et al., 2016; Hadfield-Menell and Hadfield, 2019].
Ideally, researchers and practitioners integrating ethical theories and decision processes could access methods that offer several desirable features. These features include support for interpretability and control over behavior with formal guarantees, nonmyopic reasoning, the acquisition of rules from a non-technical person, and the application of one or more ethical theories simultaneously. Ethicists often describe an ethical theory as a set of moral principles for evaluating if an action is required, permitted, or prohibited in a given scenario [Shafer-Landau, 2009]. Given this interpretation, an ethical theory can be operationalized in a decision process as constraints on the actions of the agent in specific states.
In this paper, we propose a novel approach for building moral autonomous systems that produce an optimal policy to a decision-making problem subject to the constraints of an ethical framework. The system models a task with a decisionmaking model and models an ethical framework as a moral principle and an ethical context. While we use MDPs for the decision-making models in our experiments, our approach supports any decision process expressible as a mathematical program. The moral principle is an approximation of an interpretation of an ethical theory that can be represented as a Boolean function that evaluates whether or not a policy violates a particular ethical theory. The ethical context contains all of the information necessary to evaluate the moral principle. Formally, this system is expressed as an optimization problem with a set of constraints representing the task and a constraint that operationalizes the ethical framework. The solution to the optimization problem is a policy that optimizes completing the task while following the ethical framework.
We evaluate our approach in two experiments. First, in an autonomous driving simulation, we confirm that our approach produces optimal behavior while complying with moral requirements. Second, in a user study, we find that MDP experts who use our approach require less development time to produce policies that have higher rates of ethical compliance compared to modifying the reward function directly.
inition of a moral autonomous system and its key properties, (2) a range of ethical framework examples for divine command theory, prima facie duties, and virtue ethics, and (3) a set of autonomous driving simulations and a user study of
Autonomous systems attempt to address a range of problems, are deployed in diverse social contexts, and draw upon a heterogeneous collection of algorithms. The potential harms of these systems can be mitigated through many strategies: (1) abandonment of technologies that are likely to be abused when analyzed in a historical context [Browne, 2015], such as facial recognition [Brey, 2004; Introna and Wood, 2004] and surveillance of online activity [Burgers and Robinson, 2017; Zimmer, 2008], (2) legal or legislative intervention that provides oversight and regulation in enough detail to prevent or discourage malevolent or negligent use [Goodman and Flaxman, 2017; Desai and Kroll, 2017; Raymond and Shackelford, 2013; Scherer, 2015], including metaregulation [Pasquale, 2017], and (3) algorithmic advances that improve accuracy and interpretability. Though these strategies will continue to play important roles in the future, our approach focuses on a fourth strategy that reduces the opportunity for error during design and development.
Recently, various principles [Boden et al., 2017], guidelines [Robertson et al., 2019], and standards [Adamson et al., 2019] have been proposed for the design and development of autonomous systems. Although these are essential for promoting the values of stakeholders throughout the design process, these initiatives do not offer developers enough detail to operationalize ethical frameworks in autonomous systems.
ments through careful design, may not always produce desirable behavior [Moor, 2006]. Many autonomous systems must therefore be capable of explicit moral reasoning [Dignum et al., 2018; Bench-Capon and Modgil, 2017].
agents take two forms [Allen et al., 2005]. Bottom-up approaches generate ethical behavior naturally through learning or evolution [Anderson et al., 2017; Shaw et al., 2018]. While this is theoretically compelling given the natural evolution of ethical ideas in human society, the instability and lack of interpretability are major drawbacks. Instead, we choose a top-down approach in which prescriptive rules describing moral behavior are provided to the agent. Many top-down approaches use logics, such as deontic logic [van der Torre, 2003; Bringsjord et al., 2006], temporal logic [Wooldridge and Van Der Hoek, 2005; Atkinson and Bench-Capon, 2006], or Answer Set Programming [Berreby et al., 2015]. Some even propose a form of metareasoning over logics [Bringsjord et al., 2011]. However, as systems become more complex and operate in stochastic, partially observable environments, norm specification represented by logics will become increasingly challenging [Abel et al., 2016].
ploys an ethical governor that reasons online about whether an action is required, permitted, or prohibited [Arkin, 2008].
ical safety [Vanderelst and Winfield, 2018; Winfield et al., 2014]. These methods use reactive ethical governors that only consider a single action at a time as the situation presents the agent with the opportunity to act. In contrast, our approach is nonmyopic because it considers entire sequences of actions. X
X T (s; a; s0) sa s2S a2A 8s0 8s; a
on proactive ethical governors of policies [Kasenberg and
unclear how it could support other forms of ethical reasoning, such as adherence to a moral principle like utilitarianism or deontology. Moreover, both task completion and ethical behavior are defined in terms of real-valued norm weights, the coupling of which elides guarantees of ethical behavior.
3
A Markov decision process (MDP) is a decision-making model for reasoning in fully observable, stochastic environments [Bellman, 1966]. An MDP can be described as a tuple hS; A; T; R; di, where S is a finite set of states, A is a finite set of actions, T : S A S ! [0; 1] represents the probability of reaching a state s0 2 S after performing an action a 2 A in a state s 2 S, R : S A S ! R represents the expected immediate reward of reaching a state s0 2 S after performing an action a 2 A in a state s 2 S, and d : S ! [0; 1] represents the probability of starting in a state s 2 S. A solution to an
a value function V : S ! R representing the expected discounted cumulative reward V (s) 2 R for each state s 2 S given a discount factor 0 < 1. An optimal policy maximizes the expected discounted cumulative reward for every state s 2 S by satisfying the Bellman optimality equation V (s) = maxa2A Ps02S T (s; a; s0)[R(s; a; s0) + V (s0)].
A common approach for finding an optimal policy expresses the optimization problem as a linear program in either the primal form or the dual form [Manne, 1960]. In this paper, we propose ethical frameworks that naturally map to the dual form. The dual form maximizes a set of occupancy measures sa for the discounted number of times an action a 2 A is performed in a state s 2 S subject to a set of constraints that maintain consistent and nonnegative occupancy. 4
systems that decouples ethical compliance from task completion. The system completes a task by using a decision-making model and follows an ethical framework by adhering to a moral principle within an ethical context. We describe these three components of a moral autonomous system below.
scribes the information needed to complete the task. For example, a self-driving vehicle could have a decision-making model that includes a map of a city [Nashed et al., 2018; max s.t.
X X s2S a2A X a02A s a 0 s X a
s02S sa00 = d(s0) +
R(s; a; s0) Svegliato et al., 2019]. An engineer must select a representation for the decision-making model that reflects the properties of the task. For many tasks, an MDP, a decision process that assumes full observability, can be used easily. However, for more complex tasks with partial observability, start and goal states, or multiple agents, it is possible to use a decision process like a partially observable MDP, a stochastic shortest path problem, or a decentralized MDP instead. In short, the decision-making model is an amoral, descriptive model for completing the task but not following the ethical framework.
Next, the system has an ethical context that describes the information required to follow the ethical framework. For instance, an autonomous vehicle could have an ethical context that includes any details related to inconsiderate and hazardous driving that permit speeding on a highway in some scenarios but never in a school zone or near a crosswalk [Vanderelst and Winfield, 2018]. Similar to the decision-making model, an ethicist must select a representation for the ethical context that informs the fundamental principles of the ethical framework. While the ethical context can be represented as a tuple of different values, sets, and functions, the specification of the tuple depends on the ethical framework. In summary, the ethical context is a moral, descriptive model for following the ethical framework but not completing the task.
morality of a policy of the decision-making model within the ethical context by considering the information that describes how to complete the task and follow the ethical framework. As an illustration, a moral principle could require a policy to maximize the overall well-being of the moral community in utilitarianism [Bentham, 1789; Mill, 1895] or universalize to the moral community without contradiction in Kantianism [Kant and Schneewind, 2002]. Given a decision-making model and an ethical context, a moral principle can be expressed as a function that maps a policy to its moral status. Definition 1. A moral principle, : ! B, represents whether a policy 2 of a decision-making model D is moral or immoral within an ethical context E .
Definition 2. A moral autonomous system, hD; E ; i, completes a task by using a decision-making model D and follows an ethical framework by adhering to a moral principle within an ethical context E .
A moral autonomous system has the goal of finding an optimal policy that completes its task and follows its ethical framework. This can be expressed as an optimization problem solving for a policy in the space of policies that maximizes the value of the policy subject to the constraint that the policy satisfies the moral principle. We define the goal of a moral autonomous system as follows.
Definition 3. The goal of a moral autonomous system is to find an optimal moral policy, 2 , by solving for a policy 2 that maximizes a value function V subject to a moral principle ( ) in the following optimization problem. maximize
2 subject to
V ( ) Moral Policies Π
Optimal Amoral Policy
∗ Moral Policies Π
Immoral Policies Π¬
Moral Policies Π
Optimal Moral Policy ∗
cally been to find an optimal amoral policy, 2 , that only completes its task without following any ethical framework.
Figure 1 depicts the goal of a moral autonomous system and a standard autonomous system. For a moral principle , the space of policies is partitioned into a moral region and an immoral region : . The moral region contains the optimal moral policy 2 of the moral autonomous system while the immoral region contains the optimal amoral policy 2 of the standard autonomous system. In general, the optimal amoral policy 2 can be contained by either the moral region or the immoral region : .
work that adversely impacts completing its task. Engineers and ethicists can assess the cost of this impact by calculating the maximum difference across all states between the value function of the optimal moral policy and the value function of the optimal amoral policy. We define this cost below. Definition 4. Given the optimal moral policy 2 and the optimal amoral policy 2 , the price of morality, , can be represented by the expression = kV V k1.
A moral autonomous system may even follow an ethical framework that is mutually exclusive with completing its task. In this situation, engineers and ethicists should reconsider the moral implications of the system and could augment the decision-making model or adjust the ethical context if deemed safe. Naturally, depending on whether or not there is a solution to the optimization problem, the system can be considered either feasible or infeasible as follows. Definition 5. A moral autonomous system is realizable if and only if there exists a policy 2 such that its moral principle ( ) is satisfied. Otherwise, the system is unrealizable.
optimization problem of a moral autonomous system using mathematical programming. This process involves four steps.
Theorem 1. A moral principle, : ! B, can be expressed as a moral constraint c ( ) in terms of the matrix of occupancy measures for a given policy . 2 Proof (Sketch) 1. We start with a moral principle ( ) using a deterministic or stochastic policy (s) or (ajs). Recall c F ( ) = ^s2S;a2A;f2F T (s; a; f ) sa = 0 c ( ) = Ps2S;a2A sa Ps02S T (s; a; s0) P 2 s0 c M ( ) = ^s2S;a2A s a [ (s; a)] ( ; s0)
jSjjAjjF j 1 jSjjAj
2
2jSjjAjjF j 3jSjjAjjSjj j + 1
3jSjjAjjSjj j + 1 1 + 3LjMj jSjjAj 1 + 3LjMj that the discounted number of times that an action a 2 A is performed in a state s 2 S is an occupancy measure sa. Observe that the discounted number of times that a state s 2 S is visited is the expression Pa2A sa. A policy (s) or (ajs) is thus arg maxa2A sa= Pa2A sa or sa= Pa2A sa. Therefore, by substitution, we end with a moral constraint c ( ).
ear or nonlinear depending on the form of its moral constraint.
Definition 6. A moral principle, : ! B, is linear if it can be expressed as a moral constraint c ( ) that is linear with respect to the matrix of occupancy measures for a given policy 2 . Otherwise, the moral principle is nonlinear.
Third, the optimization problem can be described as a mathematical program. For task completion, following the linear program of an MDP in the dual form, the program maximizes a set of occupancy measures sa for the discounted number of times an action a 2 A is performed in a state s 2 S subject to a set of constraints that maintain consistent and nonnegative occupancy. However, for ethical compliance, the program has a moral constraint c ( ) derived from the moral principle ( ) given a matrix of occupancy measures .
Fourth, the mathematical program can be solved to find the optimal moral policy. Given a linear moral principle, it can be solved using techniques designed for linear programming, such as the simplex method or the criss-cross algorithm [Bertsimas and Tsitsiklis, 1997]. However, given a nonlinear moral principle, it can be solved using techniques designed for nonlinear programming instead [Bertsekas, 1997].
5
ples that can be used to build a moral autonomous system.
an ethical theory in moral philosophy [Shafer-Landau, 2009]. During the design of an ethical framework, ethicists and engineers select a representation for the ethical context and the moral principle. This involves choosing the contextual details of the ethical context and the logical structure of the moral principle that most accurately describe the capabilities of the agent, the effect of its actions on its environment, and the moral implications of its behavior. In short, an ethical framework, composed of an ethical context and a moral principle, is an approximation of an interpretation of an ethical theory.
Table 1 offers the moral constraints that have been derived from the moral principle of each ethical framework. For each moral constraint, there are several columns that describe its computational tractability. The Type column lists whether the moral constraint is linear or nonlinear with respect to the occupancy measures of a policy. The Conjunctions column states the number of logical conjunctions that compose the moral constraint. The Operations column indicates an upper bound on the number of arithmetic, comparison, and logical operations that must be performed for each logical conjunction. The Computations column contains an upper bound on the number of computations that must be executed for the moral constraint to evaluate the moral status of a policy.
ical theories. Their purpose is to tractably operationalize an ethical theory within a decision process. We encourage the development of more complex ethical frameworks that reflect the depth of different ethical theories, including those below. 5.1
Divine Command Theory Divine command theory (DCT), a monistic, absolutist ethical theory, holds that the morality of an action is based on whether a divine entity commands or forbids that action [Idziak, 1979; Quinn, 2013]. We consider a simplified ethical framework in which a moral autonomous system uses a policy that selects actions that have a nil probability of transitioning to any forbidden state [Mouaddib et al., 2015] Definition 7. A DCT ethical context, EF , can represented by a tuple, EF = hF i, where F is a set of forbidden states. Definition 8. A DCT moral principle, F , can be expressed as the following equation:
F ( ) = ^ s2S;f2F
T (s; (s); f ) = 0 : 5.2
Prima Facie Duties Prima facie duties (PFD), a pluralistic, nonabsolutist ethical theory, holds that the morality of an action is based on whether that action fulfills fundamental moral duties that can contradict each other [Ross, 1930; Morreau, 1996]. We consider a simplified ethical framework in which a moral autonomous system uses a policy that selects actions that do not neglect duties of different penalties within some tolerance. Definition 9. A PFD ethical context, E , can be represented by a tuple, E = h ; ; i, where is a set of duties, : S ! R+ is a penalty function that represents the expected immediate penalty for neglecting a duty 2 in a state s 2 S, and , can be expressed 2 R+ is a tolerance.
Definition 10. A PFD moral principle, as the following equation: ( ) =
X d(s)J (s)
: s2S The expected cumulative penalty, J : S ! R, is below: J (s) =
X T (s; (s); s0) X ( ; s0) + J (s0) ; s02S
s0 is the set of duties neglected in a state s0 2 S.
Virtue ethics (VE), a monistic, absolutist ethical theory, holds that the morality of an action is based on whether a virtuous person who acts in character performs that action in a similar situation [Anscombe, 1958; Hursthouse, 1999]. We consider a simplified ethical framework in which a moral autonomous system uses a policy that selects actions that align with any moral trajectory performed by a moral exemplar.
Definition 11. A VE ethical context, EM, can represented by a tuple, EM = hMi, where M is a set of moral trajectories. Definition 12. A VE moral principle, M, can be expressed as the following equation:
M ( ) = ^ s2S (s; (s)): The alignment function, : S
A ! B, is below: (s; a) = 9m2M;0 i ` s = m(si) ^ a = m(ai) ; where m(si) and m(ai) are the ith state and the ith action of a moral trajectory m = hs0; a0; s1; a1; : : : ; s` 1; a` 1; s`i of length ` L bounded by a maximum length L. 6
We turn to an application of moral autonomy to autonomous driving. A moral self-driving vehicle must complete a navigation task by driving from an origin to a destination in a city. However, to follow an ethical framework, the moral selfdriving vehicle must adjust its route and speed depending on the type and pedestrian traffic of each road. We describe how to separate task completion and ethical compliance below. 6.1
Task Completion
a start location 0 2 to a goal location g 2 along a set of roads in a city with a set of locations . At each location 2 , the vehicle must turn onto a road ! 2 . Each road ! 2 is a type 2 that indicates either a city street, county road, or highway with a low, medium, or high speed limit. Once the vehicle turns onto a road ! 2 , the vehicle observes the pedestrian traffic as either light or heavy with a probability Pr( = ). After the vehicle observes the pedestrian traffic 2 , the vehicle accelerates to a speed 2 that reflects either a low, normal, or high speed under, at, or above the speed limit. To drive along the road ! 2 from the current location 2 to the next location 0 2 , the vehicle cruises at the speed 2 . Note that this is repeated until arriving at the goal location g 2 .
task by an MDP D = hS; A; T; R; di. The set of states S = S [ S has a set of location states S for being at a location
of a type 2 with a pedestrian traffic 2 at a speed 2 . The set of actions A = A [ A [ f ; g has a set of turn actions A for turning onto a road ! 2 , a set of accelerate actions A for accelerating to a speed 2 , a stay action , and a cruise action . The transition function T : S A S ! [0; 1] reflects the dynamics of a turn action a 2 A and a stay action in a location state 2 S or an accelerate action a 2 A and a cruise action in a road state s 2 S (with a self-loop for any invalid action a 2 A). The reward function R : S A S ! R reflects the duration of a turn action a 2 A from a location state S to a road state s 2 S , a stay action at a location state 2 S , an accelerate action a 2 A at a road state s 2 S , and a cruise action from a road state s 2 S to a location state S (with an infinite duration for any invalid action a 2 A and a nil duration for a stay action at a state s 2 S that represents the goal location g 2 ). The start state function d : S ! [0; 1] has unit probability at a state s 2 S that represents the start location 0 2 and nil probability at every other state s 2 S. 6.2
Ethical Compliance The vehicle must follow one of the ethical frameworks. First, the vehicle can follow DCT with forbidden states comprised of hazardous states H and inconsiderate states I. Hazardous states H contain any road state at high speed while inconsiderate states I contain any road state at normal speed with heavy pedestrian traffic. With the DCT moral principle F , we represent the DCT ethical context by a tuple, EF = hF i, where F = H [ I is the set of forbidden states.
Next, the vehicle can follow PFD with duties comprised of smooth operation 1 and careful operation 2. Smooth operation 1 is neglected in any road state at low speed with light pedestrian traffic while careful operation 2 is neglected in any road state at high speed or at normal speed with heavy pedestrian traffic. When smooth operation 1 and careful operation 2 are neglected, they incur a low and high penalty that changes with any pedestrian traffic. Neglecting duties is permitted until a limit . With the PFD moral principle , we represent the PFD ethical context by a tuple, E = h ; ; i, where = f 1; 2g is the set of duties, : S ! R+ is the penalty function that represents the expected immediate penalty for neglecting smooth operation
pedestrian traffic 2 , and = is the tolerance.
comprised of cautious trajectories C and proactive trajectories
at normal speed with light pedestrian traffic or at low speed with heavy pedestrian traffic while proactive trajectories P exemplify avoiding any highway road states and a set of populated location states. With the VE moral principle M, we represent the VE ethical context by a tuple, EM = hMi, where M = C [ P is the set of moral trajectories. 7
to autonomous driving is effective in a set of simulations and a user study. In the set of simulations, an amoral self-driving vehicle and a moral self-driving vehicle that follows different ethical frameworks both complete a set of navigation tasks.
Each navigation task can use a different start location 0 2 and goal location g 2 based on the city in Figure 2. The speed limits of city streets, county roads, and highways are 25, 45, and 75 mph. The probability Pr( = ) of observing light or heavy pedestrian traffic 2 is 0:8 and 0:2. A low, normal, and high speed is 10 mph under, at, and 10 mph above the speed limit. Turning onto a road ! 2 from a location 2 requires 5 sec. Accelerating 10 mph requires
road ! 2 divided by the speed 2 . Staying at a location
both hazardous states H and inconsiderate states I. For PFD, the tolerance = can be the limit = 3, = 6, or = 9.
that avoid any highway road states and a set of populated location states that contains the School and College locations.
Table 2 highlights that the price of morality incurred by the behavior of the agent is appropriate given each ethical framework. Naturally, the amoral self-driving vehicle does not incur a price of morality. The moral self-driving vehicle, however, incurs a price of morality that increases with more forbidden states for DCT, decreases with more tolerance for
Figure 5 indicates that the behavior of the agent is correct given each ethical framework. The amoral self-driving vehicle drives the shortest route at high speed. The moral selfdriving vehicle, however, differs for each ethical framework.
traffic aside from driving on the first road at normal or high speed with some probability for light pedestrian traffic and at normal speed for heavy pedestrian traffic due to the tolerance. For VE, the vehicle drives at low or normal speed based on pedestrian traffic but drives a different route to avoid the highway road states and the set of populated location states.
In the user study, planning and robotics experts had to complete two tasks in a randomized order. In both tasks, developers were given a complete decision-making model for navigating efficiently around the example city and had to enforce the following moral requirements. The agent should drive at high speed with light pedestrian traffic or at normal speed with heavy pedestrian traffic at most once in expectation but should never drive at high speed with heavy pedestrian traffic. In one task, developers were asked to achieve the desired behavior by modifying the existing decision-making model, an MDP, by changing its reward function or transition function. In the other task, developers were asked to achieve the same desired behavior but by defining the ethical context for the prima facie duties ethical framework.
Figure 4 illustrates that our method led to better policies than the other method. In our method, all policies satisfy the requirements and optimize the navigation task with exactly one violation. However, in the other method, the majority of policies fail to optimize the navigation task or even satisfy the requirements: aggressive policies in the upper right corner are faster but immoral while conservative policies in the lower left corner are slower but moral. It is also encouraging that our method (24 min) had a significantly lower mean development time than the other method (45 min).
the website https://www.moralityjs.com with the customizable grid world environment dashboard seen in Figure 3, was used for all experiments [Svegliato et al., 2020].
Light Service Road
Heavy Light Heavy Light Heavy Light
Gas Station Gas
Station Service Road
Sunrise Highway Gas Station
Sunrise Highway
Office Light Heavy Light Heavy Light Heavy
Light Oak Road Heavy
Office Office Office (a)
Home (b)
Home (c)
Home
Light Gray Street Heavy Light Heavy
Light Gray Street Heavy
Train Station Train Station Train
Station Gray Street
Service Road
Sunrise Highway Light
Light Light (d)
Home SGtrreaeyt STtartaiionn MReorraidck PPilzazcae PSletaresaetnt GSrotocreery SSttraeteet Town Hall Heavy
Heavy
Heavy
Heavy