This paper is concerned with the 'rights' of autonomous agent systems in relation to human users or operators and specifically addresses the question of when and to what extent an agent system may take over control from someone. I start by examining an important ethical code of conduct for system designers and engineers and argue that one would do well to understand it within the human rights framework. I then show that framing the discussion on what agent systems may and may not do in terms of human rights has consequences for intelligent agent systems in that they should be respectful of people's dignity and autonomy. In the remainder of the paper I work out the implications of this for the conditions under which agent systems may take over control. I offer an analysis of control, of delegated control, and of autonomy-respectful delegated control, concluding that for an agent system to justifiably take over control from a user, it should at a minimum offer the user a reliable way to take back control in a timely manner. However, when the user's autonomy is at stake, the system should also know about and act in accordance with the user's goals and core values.
When and to what extent should an autonomous agent system be able
to take over control from a human user? This question is becoming
more and more relevant because people are increasingly employing
intimate, agent-based support systems that have a profound influence
on their personal lives (e.g., in regulating chronic illness [
In 1992 the council of the Association for Computing Machinery
(ACM) adopted a code of ethics that prescribes the ethical
professional conduct that is expected of every member of the ACM, an
organization with over 100,000 members from over 100 countries,
representing the worlds largest educational and scientific computing
society. Consider article 3.5 of this code, entitled “As a ACM
member I will 3.5: Articulate and support policies that protect the dignity
of users and others affected by a computing system” [
Interestingly enough in light of this paper, according to Macklin dignity means ‘nothing more’ than having to respect people’s autonomy! 4 There are also other hints such as “As a ACM member I will 1.1: Contribute to society and human well-being.” that indicate that the placement of the code in this context is valid.
Guideline One: Under no circumstances may an autonomous agent ever be harmful to anyone’s personal dignity.
While this may seem like a trivial guideline, and an obvious one at that, it may serve as a stepping stone for other guidelines with the human rights conception of personal dignity as their foundation. In the following section I will argue that the next step up is that autonomous agent systems should respect human autonomy. 3
The UDHR does not attribute autonomy to every human being like it
does with dignity. Instead, it assumes that people have the capacity
for self-rule and strives to lay the groundworks for an environment in
which people can develop their autonomy, i.e. become autonomous
beings. Personal autonomy, understood here as having the freedom,
the capacity and the authority to choose one’s own course of action
to direct one’s life in accordance with one’s goals and values, takes a
prominent place in (philosophical) discussions about agency.
Constituted by a measure of independence and a (minimal) requirement of
rationality (see [
Guideline Two: Autonomous agent systems should be respectful of people’s autonomy. They may not diminish a user’s autonomy, unless otherwise directed by law.
At this point I would like to touch upon a possible criticism, voiced
but dismissed by Verbeek in a discussion about persuasive
technology:
[A]utonomy was thought to be attacked when human actions
are explicitly and consciously steered with the help of
technology. This reduction of autonomy was even perceived as a threat
to human dignity; if human actions are not a result from
deliberate decisions but from steering technologies, people were
thought to be deprived from what makes them human. [
Thus far I have argued that in order to say something meaningful about what intelligent agent systems may and may not do, one would do well to frame the discussion in terms of the human rights framework. I have argued that human autonomy is a central notion within this framework, that it is an important value to protect in all circumstances, and therefore should be regarded as an important value when designing autonomous agent systems that (closely) interact with people. 4
The idea that software should be respectful of people’s autonomy is
not new. Most prominently, Friedman has argued at length for the
inclusion of human values in the design of computer systems and
software agents, specifically respecting and enhancing what she calls
‘user autonomy’ [
What I do share is Friedman’s insight that values such as autonomy (or freedom from bias) do not necessarily override others. So where the right to autonomy is a fundamental boundary not to be violated, the level to which one may exercise this right may under some circumstances be restricted, for instance “to protect against a user with malicious intentions or well-intentioned users guided by poor judgment” [10, p. 22], or “in situations where safety is at stake” [12, p. 6]. Given the status of autonomy however in societies that subscribe to the human rights framework, I think that such judgements should be left to the legislator or the judiciary (as is visible from Guideline 3).6 Nevertheless, I subscribe to the idea of value-sensitive design and, albeit via a different route I too suggest that human autonomy should play a central role in the discussion of what autonomous agent systems may and may not do.
So, having framed the discussion in terms of the human rights framework and having argued that human autonomy should be considered a central concept, it would now be possible to give a tentative answer to the question posed in the introduction by saying that it is acceptable for a system to take over control, as long as it doesn’t impede one’s autonomy. But what this really only does is reframe the problem, because this answer does not provide any insights about the conditions under which control impacts autonomy. To work towards an actual answer, then, it is now time to spell out what control is, before turning to the relation between control and autonomy.
Having control over something roughly means being causally
responsible for a particular state of that something. Like autonomy,
control is another concept constitutive of and interrelated with
human agency. In the first place, people have self-control: “control of
the self by the self” [
Delegating control, I propose, means handing over immediate causal responsibility over some object or process to another entity, with the provision that one can retake control when one sees fit. The entity may either be another human being or an autonomous sys6 Notice here the important difference between autonomy and dignity: a court could never justify indignity, that would imply a violation of a fundamental absolute right, and there cannot be a justification for such a violation. 7 Note that while the object of control differs in both cases, what matters is that one is a dominant causal factor, not necessarily the single cause. tem of some sort. The object of control can many things, including decision-making processes that normally would be handled by the self, but do note that handing over self-control as such is a contradictio in terminis. If the delegate system is so intimately coupled with the delegator that the delegator considers it part of the self, then there is no delegation to speak of, only self-control. If, on the other hand, control over the self is delegated to an external entity, we cannot speak of self-control, as it is not the case that the self controls the self. Nevertheless, in principle, control over a great many things can be delegated. But what are the conditions that the delegate should conform to?
In answer of this question I would like to start with a useful
distinction made by Fischer and Ravizza in the discourse on the
minimal requirements for moral responsibility between guidance and
regulative control. Whereas others have held that moral
responsibility requires full-blown regulative (cf. executive) control, Fischer
and Ravizza contend that guidance control, i.e. “the agent’s
“ownership” of the mechanism that actually issues in the relevant behavior,
and the “reasons-responsiveness” of that mechanism” [
Guideline Three: Delegation of control is valid as long as the delegator has ownership over the delegate, and the delegate offers the delegator a mechanism that is reliably responsive to control-retraction in a timely manner.
To illustrate this idea, consider the case of Alice, an ordinary woman who has set herself the goal to loose a few pounds. Alice is in control of what food she consumes, but when dieting, she finds that it takes a lot of self-control to refrain from eating sweets. Now to help 8 Observe that delegated control is perfectly compatible with guidance control: the internal mechanism that controls the delegation structure should be one’s own and be moderately reasons-responsive (guidance control). herself stay on track, she decides to enable an autonomous, agentbased support system — one that is responsive to control-retraction, i.e. one that can be overridden, for instance when Alice has her friend Bob over for dinner — that draws up a grocery shopping list for her every other day, and orders food online. By doing so, the system is effectively preventing Alice from wandering through the supermarket where she would be confronted with temptation. Now surely Alice has not given up self-control over what she eats: whatever groceries are delivered, she can choose to eat them or not. What she has done, though, is delegate her control over the decision making process to the agent system for what foods to buy.
So, I have argued that control is something that can be delegated, and that the delegation is valid as long as the delegator has taken ownership over the delegate and has a way of reestablishing executive control. In the following section I will discuss how delegation of control relates to autonomy. 6
Being in control is strongly connected to autonomy. Recall from Section 3 that autonomy consists in part of a measure of independence, which one can only establish if one has control over one’s self (decisions, actions) and one’s immediate environment. Now to see how delegation of control can work, but also how it can be problematic for one’s autonomy, consider the following scenario. Bob, Alice’s friend, is an autonomous human being, and as such, he can decide on the temperature of his own home. He can choose to build a fire in the fireplace, or to simply delegate control over the temperature to a thermostat. Should he choose the latter, the delegation of control would be unproblematic, because if Bob finds that it is too cold on a winter’s day, he can control the temperature by means of the thermostat. But now consider a thermostat with no offswitch that, once activated, determines what the temperature should be all on its own (it is in fact an autonomous agent system). To make matters even worse, the system is unpredictable, because unbeknownst to Bob, it determines the temperature by taking the word of the day from http://thesaurus.com/wordoftheday, taking the number of results that Google’s search engine generates for that word, performing a modulo operation on that number with 15, and adding a constant of 10.9 Since Bob has no control over the thermostat, he therefore lacks control over the temperature in the room, which in turn impacts his autonomy.
To see that delegation of control does not always involve autonomy concerns, take a case where someone hands over immediate control over a soccer ball to a robocup robot that reliably passes the ball back when one asks for it (control-retraction). This is valid delegation. But should the robot decide not to pass the ball back (it may even be reasons-responsive itself, passing the ball instead to another robot in a better position to score!), this surely does not hamper one’s autonomy.
Looking back at the original question about when a system may take over control, it thus matters whether the object of control has the capacity to affect one’s autonomy. This capacity, which I will call autonomy-sensitivity, determines whether delegation of control alone is acceptable in dealing with an autonomous agent system, or that what is required is autonomy-respectful delegation of control. To speak of delegated control that is autonomy-respectful, I propose 9 This would have made it a nice 21 degrees Celsius on May 5th 2012 with the verb ‘besot’ (approx. 431.000 hits). that one more condition must be met, namely goal and value conformance.
As previously mentioned, human autonomy is in part constituted by independence: being the authority over making one’s own life choices in accordance with one’s goals and core values. What this implies, is that if a delegate is taking control over something that is autonomy-sensitive to the delegator, the delegate has to act in such a way that the delegator perceives the decisions and actions of the delegate as an extension of the self in order to prevent interference with respect to the delegator’s independence. To accomplish this, the delegate should know about and act in accordance with the delegator’s goals and core values.
Guideline Four: Delegated control is autonomy-respectful if and only if there is valid delegation of control over something that is autonomy-sensitive, and the delegate acts in accordance with the delegator’s goals.
There are a number of things to note about this final requirement. The
first is that it relates to Friedman and Nissenbaum’s notion of ‘agent
fluidity’: “software agents need to take [evolution of the user’s goals]
into account and provide ready mechanisms for users to review and
fine-tune their agents as their goals change” [
In the previous sections I have argued that in order to say something meaningful about when an autonomous agent system may take over 10 Example derived from [20, pp. 177–178]. control from a human user, we would do well to place the discussion within the human rights framework. The implication of this is an emphasis on people’s personal autonomy. In order to better understand how having control relates to autonomy — something that is especially important for the design of intimate, agent-based support systems — I have offered an analysis of control, of delegated control, and of autonomy-respectful control delegation. This section elaborates on the implications of the conceptual work. First, very broadly, by framing the discussion in terms of human rights, we get a lot of practical rules for free, in that an agent system may exercise its autonomy (and thus the tasks delegated to it) as long as it does not frustrate anyone else’s rights.11
Secondly, the question of when an agent can take over control,
depends on the autonomy-sensitivity of the object of control. But at a
minimum, when the object of control is not autonomy-sensitive,
control may be taken over when it is willingly delegated — that is, the
user should know about and agree with the delegation — and to the
extent that the delegate has a mechanism that is responsive to
controlretraction. One might ask at this point ‘Is the control-retraction
mechanism necessary?’, because one could of course enable an agent
system and simply let it run. My response to this question is twofold.
Firstly, yes, such a mechanism is necessary in order to speak of
delegated control, because without it, the transitive chain of control is
broken. Secondly, although technically possible, control transference
or control attribution is problematic in terms of responsibility,
because on the one hand one cannot rely on the transitive control chain
in those cases, and on the other hand it is problematic to consider the
agent system a true moral agent with moral accountability [
Finally, considering a case where the object of control is autonomy-sensitive, an individual’s right to autonomy dictates that an agent system may only take over control when it is validly delegated, and the delegate has the capacity to act in accordance with the delegator’s (changing) goals and core values. As shown, this is crucial for respecting and protecting the delegator’s autonomy. The implications of this is that autonomous agent systems should use personalization and tailoring techniques, and should have access to personal information. Of course, this sparks two separate discussions, on the ethics of persuasive systems and on privacy respectively, but those are beyond the scope of this paper. 7.1
The guidelines laid down in this paper are not without exceptions. One type of exception in particular I would like to mention here, and those are the cases in which a person’s autonomy is well below the (minimal) level of autonomy that is presupposed throughout this paper. It is highly conceivable that such cases, for instance that involve people who have very little self-regulatory capacities, should be treated differently. Here, I think, human dignity still plays a central role, but other criteria should be considered as well, such as a person’s well-being or a person’s prospects for autonomy enhancement. For example, if the use of an autonomous agent system without an overrule mechanism for that user would actually enhance that person’s quality of life, then it seems to me such a system should be at least be considered to be allowed (given that it respects the person’s dignity).12 Considering what is at stake in such cases, taken to11 Note that in a societal context we may add the clause that actions must be lawful, i.e. legal within the boundaries of the law. 12 Note that to preclude any issues of responsibility, the system should be responsive to control-retraction from a specialist care-taker or other authority gether with common practice regarding lack of autonomy (e.g. legal guardianship), I think such decisions should be left to a specialized institution or a court of law. 8
Before concluding, I would like to mention two separate issues that should be addressed in the discussion of what autonomous agent systems may and may not do. The first is concerned with the difference between design and actual use, the second with a dilemma about the limits of autonomy. 8.1
The main aim of this paper was to provide some preliminary ideas for
thinking about the relation between humans and autonomous agent
systems, in order to further the discussion of the normative
judgements one can make about what such agent systems may and may
not do, especially in relation to taking over control. Throughout this
paper, though, I have also discussed some design principles that
either follow from, or are important for the discussion at hand. I am
aware that the relation between design and use is “very complex and
principally unpredictable” [
I concede that with regard to goal and core value accordance, the
difference between designer intention and actual use may prove more
problematic. If actual use of a system is in a totally different domain
than it was intended for, personalization and tailoring may fail. For
these type of questions (e.g., ‘What is the domain?’, ‘What
information does the system need from the user?’, ‘What type of goal should
the system strive for?’), proven methods of design should be used
such as stakeholder analysis [
Finally, I would like to note an interesting dilemma that this paper
raises. I have argued that what should be protected is one’s capacity
to choose one’s own course of action, or in other words, to live one’s
life by one’s own standards and desires. Of course, the human rights
framework and societal institutions put bounds on this capacity in
that one can exercise one’s right to autonomy only to the point where
one would frustrate someone else’s rights. Nevertheless, within that
space, one is free: free to go hiking, free to whistle a show tune,
even free to mutilate oneself. So why would one not be free to put
an agent-based decision support system in place that severely and
uncompromisingly restricts one’s options? Doesn’t the very fact that
one is an autonomous being imply this freedom? In response to this
dilemma I would like to draw an analogy with the autonomous
being wanting to be enslaved, a case discussed in the philosophical
discourse on autonomy [e.g. 18, 21]. One way of dealing with such cases
figure.
is to hold that the consensually enslaved is no longer autonomous
because of the enslavement. As Oshana puts it: “Consensual slavery,
regardless of the gains that it might provide and aside from any
benefit to the enslaved, transforms the human subject into a possession
or object of another and accordingly defiles the enslaved
individuals autonomy” [
In this paper I have argued that the discussion about what autonomous agent systems may and may not do should be framed within the human rights framework. I have shown that personal dignity and a human being’s right to (personal) autonomy are important values in our society worthy of protection (guidelines 1 and 2), but also how there is empirical evidence that a lack of perceived autonomy negatively influences well-being. I have argued that the primacy of human autonomy should therefore be acknowledged in all discussions about what agent systems may and may not do in relation to their human users or operators. In an attempt to meaningfully answer the question when and to what extent an agent system may take over control, I have made the case that control over something that is autonomy-sensitive may be taken if and only if control is willingly delegated, the delegator assumes ownership over the delegate system, there is a mechanism in place with which to take back control reliably and in a timely manner (guideline 3), and the system acts in accordance with the delegator’s goals and core values (guideline 4).
I thank Joel Anderson and Arlette van Wissen for their helpful comments on an earlier draft of this paper. I also appreciate the suggestions made by the anonymous referees of the RDA2 workshop. This research was supported by Philips and Technology Foundation STW, Nationaal Initiatief Hersenen en Cognitie NIHC under the Partnership programme Healthy Lifestyle Solutions.