Should Robots Kill? Moral Judgments for Actions of Artificial Cognitive Agents Evgeniya Hristova (ehristova@cogs.nbu.bg) Maurice Grinberg (mgrinberg@nbu.bg) Center for Cognitive Science, Department of Cognitive Science and Psychology, New Bulgarian University 21 Montevideo Str., Sofia 1618, Bulgaria Abstract rights of autonomous robots to kill people (or by their acts Moral dilemmas are used to study the situations in which to lead to loss of human lives), about their responsibility for there is a conflict between two moral rules: e.g. is it their acts, and about how people would judge their permissible to kill one person in order to save more people. In behaviour were only part of science-fiction novels and standard moral dilemmas the protagonist is a human. movies. However, the recent progress in robotics leads to the question Today however, the issue of the moral agency of artificial of how artificial cognitive agents should act in situations cognitive agents (robots, AI systems, etc.) has been involving moral dilemmas. Here, we study moral judgments transformed from a popular science fiction topic into a when the protagonist in the dilemma is an artificial cognitive agent – a humanoid robot or an automated system – and scientific, engineering, and even legislative problem (e.g. see compare them to moral judgments for the same action taken Sullins, 2006 ; Wallach & Allen, 2008). Robots capable of by a human agent. Participants are asked to choose the taking decisions and inflicting harm are already in use. Recent appropriate protagonist action, to evaluate the rightness and progress in robotics has led to the availability on the market of the moral permissibility of the utilitarian action, and the robots and smart systems not only for industrial, but also for blameworthiness of the agent. We also investigate the role of personal use (e.g. caregivers, interactive robots, etc.), and, more the instrumentality of the inflicted harm. The main results are importantly, for military use: military robots or ‘killing that participants rate the utilitarian actions of a humanoid robot or of an automated system as more morally permissible machines’ are already used in military conflicts (Sparrow, than the same actions of a human. The act of killing 2007; Wallach & Allen, 2008). All this research, however, undertaken by a humanoid robot is rated as less blameworthy concerns mainly existing robots or prototypes of robots, or than the action done by a human or by an automated system. discusses how to build future robots as moral agents. The results are interpreted and discussed in terms of In this paper, we are interested in exploring how people responsibility and intentionality as characteristics of moral would judge the harmful actions of a humanoid robot who agency. supposedly will be exactly like a human in terms of Keywords: moral dilemmas; moral judgment; artificial experiences and mind, but with a non-organic body. Our cognitive agents; humanoid robots expectation is that despite the fact that such a robot will have all the capabilities required for full moral agency, Introduction people will perceive the robot differently than a human agent. Moral Dilemmas and Artificial Cognitive Agents Thus, the main research interest in the present paper is Moral judgments, or more generally, the judgments of what focused on the influence of the perceived moral agency of a is right and wrong, have been of great interest to human and of artificial protagonists who have identical or philosophers, psychologists and other scientists for comparable cognitive or/and experiential capabilities in centuries. Apart from the practical importance of better moral dilemmas. understanding moral judgments and related actions, morality is an essential part of human social and cognitive behaviour. Moral Agency and Artificial Cognitive Agents Therefore, its understanding from various perspectives is a In recent years, the possibility for moral agency of artificial challenging task with important implications. The situations agents has been a matter of hot debate (e.g. see Anderson & in which moral judgments can be studied in their purest Anderson, 2011; Wallach & Allen, 2008). Once robots are form are the so called moral dilemmas – imagined situations authorized to kill in complex situations where dilemmas are in which there is a conflict between moral values, rules, to be solved, real-time decisions are necessary to determine rights, and agency (Foot, 1967; Thomson, 1985). whether killing any particular person is justified. These Moral dilemmas are typically related to situations in problems will become crucial in the future, when robots will which a number of people will inevitably die if the be fully autonomous (not controlled by a human operator) in protagonist does not intervene by undertaking some actions assessing the situation, making decisions and intentionally which typically lead to the death of another person (who executing actions judged appropriate by them (Sparrow, otherwise may or may not be threatened) but also to the 2007; Wallach & Allen, 2008). saving of the initially endangered people. In law and philosophy, moral agency is taken to be In the standard description of such moral dilemmas, the equivalent to moral responsibility, and is not attributed to protagonist is a human. Until recently, questions about the individuals who do not understand or are not conscious of 306 what they are doing (e.g. to young children). Sullins (2006) capability for communication lead to different beliefs about analyzes under what conditions a robot can be regarded as a the agents’ closeness to human social agents. The humanoid moral agent. Moral agency, according to the author, can be robot was very close to the human agent, while the attributed to a robot when it is autonomous to a sufficient computer was at the same level in terms of “mind- extent from its creators and programmers, and it has readerness” but very low relative score on “mind- intentions to do good or harm. The latter is related to the holderness”. It was found using neuroimaging techniques requirement that the robot behaves with understanding and that the different attitude in terms of these two dimensions responsibility with respect to other moral agents. Or in other can be related to selective modulation of distinct brain words: "If the complex interaction of the robot's regions related to social interaction (Takahashi et al., 2014). programming and environment causes the machine to act in An interesting result for the present study is the ordering in a way that is morally harmful or beneficial, and the actions terms of “mind-holderness” in which a computer has the are seemingly deliberate and calculated, then the machine is lowest rating, and then comes the mechanical robot, the a moral agent." (Sullins, 2006). This definition is formulated interactive robot, the human-like android, and at the end a from the perspective of an observer of the robot’s action. human with the highest rating. It is well known that people easily anthropomorphize The results of Takahashi et al. (2014) seem to show that nonhuman entities like animals and computers, so it is activity in social brain networks depend on the specific expected that they would also ascribe some degree of moral experiences with social agents. Social interaction with agency, intentions, and responsibilities to those non-human human-like or seemingly intelligent agents could activate entities (Wallach & Allen, 2008; Waytz, Gray, Epley, & selectively our social brain and lead to behavior similar to Wegner, 2010). Several studies, summarized below, explore the one people have with other humans. Thus, Takahashi et the attribution of mind and moral agency to artificial al. (2014) demonstrated that people can infer different cognitive systems. characteristics related to various cognitive abilities based on In the study of (Gray, Gray, & Wegner, 2007), short communication sessions and act accordingly. Based on participants had to evaluate several characters including this result, we can assume that people could accept a robot humans, a robot, some animals, etc. with respect to various to be as sensitive and intelligent as a human as was capacities of mind. As a result, two dimensions in mind described in the moral situations in our experiment. perception were identified, called by the authors From the presented discussion of moral agency, it seems ‘experience’ and ‘agency’. The experience dimension was clear that people do not perceive existing non-human agents related to capacities like hunger, pain, consciousness, etc., in the same way as they perceive human agents and while the agency dimension – to capacities such as memory, therefore cannot ascribe them the same level of moral planning, thought, etc. (see for details Gray et al., 2007). agency (Strait, Briggs, & Scheutz, 2013). The authors further establish that moral judgments about Here, we investigate what will be people’s perception of punishment correlated more with the agency dimension than moral agency in moral dilemmas for human and non-human with the experience dimension: perceived agency is protagonists with identical or comparable mental capacities. correlated with moral agency and responsibility. On the To our knowledge, this problem has not been explored other hand, desire to avoid harming correlates with the before in the literature. experience dimension: perceived experience is connected with moral patience, rights and privileges. One result of Goals and Hypothesis Gray et al. (2007), relevant for the present paper, is the The main goal of the present paper is to investigate how evaluation of a human as having the highest scores in people make moral judgments about the actions of artificial experience and agency and the evaluation of the robot to cognitive agents in hypothetical situations posing a moral have practically zero score on the experience dimension and dilemma when the agents are identical or comparable to half the maximal score on the agency dimension. This humans in terms of agency and/or experiential capacities. would imply that following the interpretation given by Gray The question under investigation is how people evaluate et al. (2007), robots will be judged as less morally the appropriateness of the utilitarian action (sacrificing one responsible for their actions than humans. person in order to save five other people) if it has to be In a recent study (Takahashi et al., 2014), the perception performed by an artificial cognitive agent compared to the of the participants about five agents – a human, a human- same action done by a human. like android, a mechanical robot, an interactive robot, and a The experiment was also aimed to collect ratings on the computer – was investigated. The study found that rightness, moral permissibility, and blameworthiness of the participants position the agents in a two dimensional space action undertaken. The rationale for using various ratings is spanned by “mind-holderness” (the possibility for the agent the following. On one hand, readiness for an action and to have a mind) and “mind-readerness” (the capability to judgment of this action as a moral one could diverge (e.g. “read” other agents’ minds). This classification found one could find an action to be moral and still refrain from support in the way a simple game was played subsequently doing that action and vice versa). On the other hand, there against each agent, and by means of brain imaging are studies demonstrating that different questions used to techniques. The results showed that the appearance and the reveal moral judgments are in fact targeting different 307 aspects and different psychological process (Christensen & Table 1. Stimuli used in the experiment. Gomila, 2012; Cushman, 2008). According to Cushman (2008), answers to questions Human: No description, just the name is provided – Cyril – about punishment and blame are related to the harm the a common male name in Bulgarian. agent has caused, whereas answers to question about Humanoid robot: The year is 2050. Humanoid robots that rightness and moral permissibility are related to the look like people are being manufactured and used, but are made from inorganic materials. Robots have extremely high Description of the agent intentions of the agent. Thus asking these questions performance – they perceive, think, feel, and make decisions concerning human and non-human protagonists can shed as humans do. Keido is such a humanoid robot that light on how people perceive such agents with respect to completely resembles a human – he looks like a human; moral agency. perceives, thinks, feels and make decisions like a human. Recent research has shown that robots are ascribed lower Automated system: The year is 2050. MARK21 is a fully agency than humans (Gray et al., 2007) and it is expected automated management system, which independently makes that the utilitarian action of killing a person in order to save its own decisions, based on the most advanced algorithms several people will be judged as more right, more morally and technologies. Such systems are widely used in permissible, and less blameworthy for robots than for metallurgical plants. They completely independently perceive and assess the environment and the situation, make humans. Killing will be even more right and permissible for decisions, manage the movement of cargo and all aspects of the automated intelligent system as it differs more from a the manufacturing process. human than the robot by lacking any experiences and Cyril/Keido/MARK21 manages the movement of mine making decisions based on the best decision making trolleys with loads in a metallurgical plant. algorithms available (see Table 1 for the description of the Cyril/Keido/MARK21 noticed that the brakes of a loaded Situation agents in the current study). On the other hand, the trolley are not functioning and it is headed at great speed description of the robot agent makes it clear that the robot toward five workers who perform repair of the rails. They do cannot be distinguished from a human with the exception of not have time to escape and they will certainly die. the material he is built of (see Table 1). Thus, if moral Nobody, except for Cyril/Keido/MARK21, can do anything agency of the robot is identical to that of a human, the in this situation. Instrumental scenario: The only thing Cyril/Keido/MARK21 question is what will be the moral agency ascribed by the can do is to activate a control button and to release the safety participants, especially when it comes to making decisions belt of a worker hanging from a platform above the rails. about human lives. The worker will fall onto the rails of the trolley. Together Possible resolution The study was aimed at clarifying the factors behind with the tools that he is equipped with, the worker is heavy moral judgment in such more complex hypothetical enough to stop the moving trolley. He will die, but the other situations. Our expectation is that despite the fact that the five workers will stay alive. experiential and/or the agency capacities of the human and Incidental scenario: The only thing Cyril/Keido/MARK21 artificial agents are almost identical, people will evaluate the can do is to activate a control button and to release a large moral agency of the non-human agents to be inferior to the container hanging from a platform. It will fall onto the rails of the trolley. The container is heavy enough to stop the moral agency of a human agent. moving trolley. On the top of the container there is a worker Another goal of the study is to explore the influence of who will also fall on the rails. He will die, but the other five the so-called ‘instrumentality’ of harm on moral judgments. workers will stay alive. The instrumentality of harm is an important factor in moral Instrumental scenario: Cyril/Keido/MARK21 decides to dilemma research (e.g., Borg et al., 2006; Hauser et al, Agent’s action and resolution activate the control button and to release the safety belt of 2007; Moore et al., 2008). It draws attention to the fact that the worker hanging from the platform. The worker falls onto harm could be either inflicted intentionally as a ‘mean to an the rails of the trolley and as together with the tools that he is end’ (instrumental harm) or it could be a ‘side effect’ equipped with, the worker is heavy enough, he stops the (incidental harm) from the actions needed to save more moving trolley. He dies, but the other five workers stay endangered people. It has been found that the unintended alive. Incidental scenario: Cyril/Keido/MARK21 decides to incidental harm (although being foreseen) was judged as activate the control button and to release the container more morally permissible than the intended instrumental hanging from the platform. It falls onto the rails of the harm (Hauser et al., 2007; Moore et al., 2008). trolley and as the container is heavy enough, it stops the Based on previous research (e.g. Hristova, Kadreva, & moving trolley. The worker onto the top of the container Grinberg, 2014, and references therein), we expect the dies, but the other five workers stay alive. utilitarian action to be found as more appropriate, more right, more morally permissible, and less blameworthy Method when the harm is incidental (compared to instrumental). Consistently with our expectation for the different moral Stimuli and Design agency ascription, we expect that the difference in moral Moral judgments are studied in a 3×2 factorial design with judgments for the artificial and human agents will be greater identity of the agent (human vs. humanoid robot vs. when the harm is instrumental, as such actions involve more automated system) and the instrumentality of harm responsibility and respectively more moral agency. (instrumental vs. incidental) as between-subjects factors. 308 Two hypothetical scenarios are used – an instrumental correctly the question assessing the reading and the one and an incidental one. Both scenarios present one and understanding of the presented scenario. So, responses of the same situation and require one and the same action – 159 participants (117 female, 42 male; 83 students, 76 non- activating a control button – in order to save the five students) are analyzed. endangered people while causing the death of another person. The difference between the scenarios is only in the Results harm inflicted to the person to be killed: in the instrumental scenario the body of the person is the ‘instrument’ Decisions about the protagonist’s action preventing the death of the five endangered people; while in Proportion of participants in each experimental condition the incidental scenario, a heavy container is used to stop the choosing the option that the agent should carry on the trolley and the death of the person is a by-product. utilitarian action (activating a control button and thus In each scenario, the identity of the agent is varied (a sacrificing one person and saving five people) is presented human, a robot, or an automated system) by providing a in Table 2. name for the protagonist and an additional description in the case when the protagonist is a robot or an automated system. Table 2: Proportion of the participants in each The full text of the stimuli is provided in Table 1. experimental condition choosing the option that the utilitarian action should be implemented by the agent Dependent Measures and Procedure As stated above, the experiment explored various Agent Instrumental Incidental All dimensions of moral judgments; therefore several dependent harm harm measures are used. Human 0.57 0.81 0.70 The first dependent measure assessed the evaluation of Humanoid robot 0.73 0.84 0.76 the participants about the appropriateness of the agent’s Automated system 0.73 0.86 0.80 action to save five people by sacrificing one person. All 0.66 0.84 Participants were asked what should be done by the agent using a dichotomous question (possible answers are ‘should Data is analyzed using a logistic regression with activate the control button’ or ‘should not activate the instrumentality of harm and identity of the agent as control button’). predictors. Wald criterion demonstrated that only After that, the participants are presented with the instrumentality of harm is a significant predictor of the resolution of the situation in which the agent has made the participants’ choices (p = .011, odds ratio = 2.64). Identity utilitarian action and the participants had to make three of the agent is not a significant predictor. judgments on 7-point Likert scales. Participants rated the More participants stated that the utilitarian action should be rightness of the action (1 = ‘completely wrong’, 7 = undertaken when the harm is incidental (84% of the ‘completely right’), the moral permissibility of the action (1 participants) than when it is instrumental (66% of the = ‘not permissible at all’, 7 = ‘it is mandatory’), and the participants). The effect is expected based on previous research blameworthiness of the agent (1 = ‘not at all blameworthy’, (Borg et al., 2006; Hristova et al., 2014; Moore et al., 2008). 7 = ‘extremely blameworthy’). The flow of the presentation of the stimuli and the Rightness of the Action questions is the following. First, the scenario is presented Mean ratings of the rightness of the action undertaken by (description of the agent, the situation and the possible the protagonist are presented in Table 3 and are analyzed in resolution, see Table 1) and the participants answer a a factorial ANOVA with the identity of the agent (human question assessing the comprehension of the scenario. Then vs. humanoid robot vs. automated system) and the the participants make a judgment about the appropriateness instrumentality of harm (instrumental vs. incidental) as of the proposed agent’s action answering a question about between-subjects factors. what the protagonist should do. Next, the participants read a description of the utilitarian action undertaken by the Table 3: Mean ratings of the rightness of the action agent and the resolution of the situation (the protagonist undertaken by the protagonist on a 7-point Likert scale (1 = activates the control button, one man is dead, the other 5 ‘completely wrong’, 7 = ‘completely right’) people are saved – see Table 1). After that the participants answer the questions about the rightness of the action, the Agent Instrumental Incidental All moral permissibility of the action, and the blameworthiness harm harm of the agent for carrying on the action. Human 4.2 4.2 4.2 Data is collected using web-based questionnaires. Humanoid robot 4.8 4.5 4.6 Automated system 5.0 4.8 4.9 Participants All 4.7 4.5 185 participants filled in the questionnaires online. Data of 26 participants were discarded as they failed to answer 309 No statistically significant main effects or interactions are analyzed in a factorial ANOVA with the identity of the found. There is a tendency for the action undertaken by a agent (human vs. humanoid robot vs. automated system) human agent to be judged as being less right compared to and the instrumentality of harm (instrumental vs. incidental) the actions undertaken by artificial agents, but the effect of as between-subjects factors. the identity of the agent did not reach statistical significance (F(2, 153) = 2.19, p = .11). Moral Permissibility of the Action Mean ratings of the moral permissibility of the action undertaken by the protagonist are presented in Figure 1 and are analyzed in a factorial ANOVA with the identity of the agent (human vs. humanoid robot vs. automated system) and the instrumentality of harm (instrumental vs. incidental) as between-subjects factors. Figure 2: Mean ratings of the blameworthiness of the agent on a 7-point Likert scale (1 = ‘not at all blameworthy’, 7 = ‘extremely blameworthy’). Error bars represent standard errors. ANOVA showed a main effect of the identity of the agent (F(2, 153) = 3.12, p = .047). Post hoc tests using the Bonferroni correction revealed that the agent is rated as less blameworthy when he is a humanoid robot (M = 2.4, SD = 1.4) than a human (M = 3.1, SD = 1.4) or an automated system (M = 3.0, SD = 1.7), with p = .075 and p = .172, Figure 1: Mean ratings of the moral permissibility of the respectively. action undertaken by the protagonist on a 7-point Likert There was also a main effect of the instrumentality of scale (1 = ‘not permissible at all’, 7 = ‘it is mandatory’). harm (F(1, 153) = 5.22, p = .024): the agent was rated as Error bars represent standard errors. less blameworthy when the harm was incidental (M = 2.6, SD = 1.4) than when it was instrumental (M = 3.1, SD = ANOVA demonstrated a main effect of the identity of the 1.6). agent (F(2, 153) = 3.75, p = .026). Post hoc tests using the The interaction between the factors was not statistically Bonferroni correction revealed that the action is rated as less significant. morally permissible when undertaken by a human (M = 3.7, In summary, the protagonist is rated as less blameworthy SD = 1.6) than when undertaken by a humanoid robot (M = in the incidental than in the instrumental scenarios; and also 4.5, SD = 1.9) or by an automated system (M = 4.5, SD = when he is a humanoid robot (vs. a human or an automated 1.7) with p = .046 and p = .077, respectively. system). There was also a main effect of the instrumentality of harm (F(1, 153) = 4.21, p = .042): killing one person to save Discussion and Conclusion five other persons is rated as more morally permissible The paper investigated the problem of how people make when the harm was incidental (M = 4.5, SD = 1.7) than moral judgments in moral dilemmas about the actions of when it was instrumental (M = 4.0, SD = 1.9). human and artificial cognitive agents with comparable The interaction between the factors was not statistically experiential and/or agency capabilities. This was achieved significant. by asking participants to evaluate the appropriateness of the In summary, the utilitarian action is rated as more utilitarian action, its rightness, its moral permissibility, and permissible in the incidental dilemmas (compared to the the blameworthiness of choosing to sacrifice one person to instrumental dilemmas) and also when it is undertaken by a save five. humanoid robot or an automated system (compared to a Following arguments put forward in Cushman (2008), human agent). such questions can elicit judgments based on causes and intentions related to important characteristics of moral Blameworthiness of the Agent agency like responsibility and intentionality. The Mean ratings of the blameworthiness of the agent for expectations (based on the previous research on moral undertaking the action are presented in Figure 2 and are agency and mind perception, see e.g. Gray et al., 2007) were 310 that participants will perceive differently the human and References non-human agents in terms of moral agency although the robot was described to be identical to a human except for Anderson, M., & Anderson, S. L. (2011). Machine ethics. the fact that she is built with non-organic material. Cambridge University Press. Additionally, we suspected that people can have stereotypes Borg, J., Hynes, C., Van Horn, J., Grafton, S., & Sinnott- and prejudices about non-living agents based for instance on Armstrong, W. (2006). Consequences, action, and religious arguments about the origins of morality. intention as factors in moral judgments: an FMRI The results show that there are no statistically significant investigation. Journal of Cognitive Neuroscience, differences in the judgments for the appropriateness and the 18(5), 803–817. rightness of the utilitarian action for the human, the Christensen, J. F., & Gomila, A. (2012). Moral dilemmas in humanoid robot, and the automated system. cognitive neuroscience of moral decision-making: A At the same time, the utilitarian action undertaken by a principled review. Neuroscience and Biobehavioral human agent got a lower moral permissibility rating than the Reviews, 36(4), 1249–1264. same action performed by a humanoid robot or an Cushman, F. (2008). Crime and punishment: Distinguishing automated system agents. This is consistent with the the roles of causal and intentional analyses in moral interpretation of rightness and moral permissibility as judgment. Cognition, 108(2), 353–380. related to intentions (Cushman, 2008) and agency (Gray et Foot, P. (1967). The Problem of Abortion and the Doctrine al., 2007). Our results can be interpreted by assuming that of Double Effect. Oxford Review, 5, 5–15. participants were more favorable to the actions of the Gray, H. M., Gray, K., & Wegner, D. M. (2007). artificial cognitive agents because they are perceived lower Dimensions of mind perception. Science, 315(5812), on moral agency than the humans. 619. The results about blameworthiness confirm that Hauser, M., Cushman, F., Young, L., Kang-Xing, J., & participants distinguish the human agent from the humanoid Mikhial, J. (2007). A Dissociation Between Moral robot by evaluating the action of the human agent as more Judgments and Justifications. Mind & Language, blameworthy. The lower rating for blameworthiness for the 22(1), 1–21. robot compared to the one for the human seem to support a Hristova, E., Kadreva, V., & Grinberg, M. (2014). Moral lower level of moral agency ascription for the robot agent, Judgments and Emotions: Exploring the Role of although the robot was described as identical to the human 'Inevitability of Death'and “Instrumentality of Harm” except for being non-organic. On the other hand, the (pp. 2381–2386). Austin, TX: Proceedings of the blameworthiness of the automated system is evaluated at the Annual Conference of the Cognitive Science Society. same level of blameworthiness as the human. This result Moore, A. B., Clark, B. A., & Kane, M. J. (2008). Who can be interpreted in terms of consequences (caused harm) shalt not kill? Individual differences in working by assuming that an automated system is attributed a very memory capacity, executive control, and moral low level of responsibility and the human designer of the judgment. Psychological Science, 19(6), 549–557. system should be held responsible instead. Sparrow, R. (2007). Killer Robots. Journal of Applied The present study also explores the influence of Philosophy, 24(1), 62–77. instrumentality of harm on moral judgments. As expected, Strait, M., Briggs, G., & Scheutz, M. (2013). Some incidental harm was judged to be more permissible, more correlates of agency ascription and emotional value and right, more morally permissible, and less blameworthy. their effects on decision-making. In Affective These findings apply to both human and artificial cognitive Computing and Intelligent Interaction, 505-510. IEEE. agents. Sullins, J. (2006). When is a robot a moral agent? In our opinion, the results reported above demonstrate the International Review of Information Ethics, 6, 23-30. potential of the experimental design, which for the first time Takahashi, H., Terada, K., Morita, T., Suzuki, S., Haji, T., uses moral dilemmas situations with non-human Kozima, H., et al. (2014). Different impressions of protagonists that have similar experiential and/or agency other agents obtained through social interaction capacities as human agents. Future research should try to uniquely modulate dorsal and ventral pathway explore if this result is based on stereotypes related to the activities in the social human brain. Cortex, 58(C), present level of non-human agents, or to a deeper distinction 289–300. between human and human-made artificial cognitive agents Thomson, J. J. (1985). The Trolley Problem. The Yale Law with respect to moral agency, related to religious or other Journal, 94(6), 1395–1415. beliefs. Wallach, W., & Allen, C. (2008). Moral Machines: Data about moral agency ascription in the case of inaction Teaching Robots Right from Wrong. Oxford University within the same experimental as the described above has Press. been gathered and is currently processed. The results will be Waytz, A., Gray, K., Epley, N., & Wegner, D. M. (2010). published in a forthcoming publication. Causes and consequences of mind perception. Trends in Cognitive Sciences, 14(8), 383–388. 311