<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Learning Answer Set Programming Rules For Ethical Machines</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Abeer Dyoub</string-name>
          <email>Abeer.Dyoub@graduate.univaq.it</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Stefania Costantini</string-name>
          <email>Stefania.Costantini@univaq.it</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Francesca A. Lisi</string-name>
          <email>FrancescaAlessandra.Lisi@uniba.it</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Dipartimento di Informatica &amp; Centro Interdipartimentale di Logica e Applicazioni (CILA) Universita degli Studi di Bari \Aldo Moro"</institution>
          ,
          <country country="IT">Italy</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Dipartimento di Ingegneria e Scienze dell'Informazione e Matematica Universita degli Studi dell'Aquila</institution>
          ,
          <country country="IT">Italy</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>Codes of ethics are abstract rules. These rules are often quite di cult to apply. Abstract principles such as these contain open textured terms that cover a wide range of speci c situations. These codes are subject to interpretations and might have di erent meanings in di erent contexts. There is an implementation problem from the computational point of view with most of these codes, they lack clear procedures for implementation. In this work we present a new approach based on Answer Set Programming and Inductive logic Programming for monitoring the employees behavior w.r.t. ethical violations of their company's codes of ethics. After brie y reviewing the domain, we introduce our proposed approach, followed by a discussion, then we conclude highlighting possible future directions and potential developments.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>Motivation and Background Machine Ethics is an emerging interdisciplinary
eld which draws heavily from philosophy and psychology [23]. The Machine
Ethics eld is concerned with the moral behavior of arti cial intelligent agents.
Nowadays, with the growing power and increasing autonomy of arti cial
intelligent agents, which are used in our everyday life performing tasks on our behalf, it
has become imperative to equip these agents with capabilities of ethical
reasoning. Robots in elder care, robot nannies, virtual companions, chatbots, robotic
weapons systems, autonomous cars, etc. are examples of some of the arti cial
intelligent systems undergoing research and development. These kinds of systems
usually need to engage in complex interactions with humans. For this reason,
taking into consideration ethical aspects during the design of such machines has
become a pressing concern.</p>
      <p>The problem of adopting ethical approach to AI has been attracting a lot
of attention in the last few years. Lately, the European Commission has
published a 'Draft Ethics Guidelines for Trustworthy AI' [18]. In this document,
the European Commission's High-Level Expert Group on Arti cial Intelligence
speci es the requirements of trustworthy AI, and the technical and non technical
methods to ensure the implementation of these requirements into the AI system.
There is a the world wide urge that ethics should be embedded in the design of
intelligent autonomous systems and technologies (IEEE global initiative 'Ethics
in Action' 3). The tech giant 'Google', after a protest from company employees
over ethical concerns, ended its involvement in an American Pentagon Project
on autonomous weapons 4. Because of the controversy over its Pentagon work,
Google laid down a set of AI principles 5 meant as a guide for future projects.
However, the new principles are open to interpretations.</p>
      <p>Moral thinking pervades everyday decision making, though understanding the
nature of morality and the psychological underpinnings of moral judgment and
decision making have been always a big concern for researchers. Moral judgment
and decision making often concern actions that entail some harm especially
loss of life or other physical harm, loss of rightful property, loss of privacy, or
other threats to autonomy. Moral decision-making and judgment is a
complicated process involving many aspects: it is considered as a mixture of reasoning
and emotions. In addition moral decision making is highly exible, contextual
and culturally diverse. Since the beginning of this century there were several
attempts for implementing ethical decision making into intelligent autonomous
agents using di erent approaches. But, no fully descriptive and widely accepted
model of moral judgment and decision-making exists. None of the developed
solutions seems to be fully convincing for providing a trusted moral behavior.
In addition, all the existing research in machine ethics try to satisfy certain
aspects of ethical decision making but fail to satisfy others. Approaches to machine
ethics are classi ed into: top-down approaches, which try to implement speci c
normative theory of ethics into the autonomous agent so that to ensure that
the agent acts in accordance with the principles of this theory; the bottom-up
approaches are developmental or learning approaches, in which ethical mental
models emerge via the activity of individuals rather than in terms of normative
theories of ethics. In other words, generalism versus particularism, principles
versus case based reasoning. Some researchers argue that morality can only be
grounded in particular cases while others defend the existence of general
principles related to ethical rules. Both approaches to morality have advantages and
disadvantages. We need to adopt a hybrid strategy that allows both top down
design and bottom up learning via context-sensitive adaptation of models of
ethical behavior.</p>
      <p>
        Contribution Ethics in customer dealings present the company in a good light,
and customers will trust the company in the future. Ethics improves the quality
of service and fosters positive relationships. Many top leading companies have a
booklet called "code of conduct and ethics" and new employees are made to sign
it. However, enforcing codes of conduct and ethics is not an easy task. These
3 https://ethicsinaction.ieee.org
4 https://www.nytimes.com/2019/03/01/business/ethics-arti cial-intelligence.html
5 https://www.blog.google/technology/ai/ai-principles/
codes being mostly abstract and general rules e.g. con dentiality, accountability,
honesty, inclusiveness, empathy, delity, etc., they are quite di cult to apply.
Moreover, abstract principles such as these contain open textured terms ([
        <xref ref-type="bibr" rid="ref14">14</xref>
        ])
that cover a wide range of speci c situations. They are subject to
interpretations and may have di erent meanings in di erent contexts. Thus, there is an
implementation problem from the computational point of view. It is di cult to
use deductive logic to address such a problem ([36], [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ]). It is impossible for
experts to de ne intermediate rules to cover all possible situations. Codes of
ethics in their abstract form are very di cult to apply in real situations [19].
All the above mentioned reasons make learning from cases and generalization
crucial for judgment of future cases and violations.
      </p>
      <p>In this work and with the future perspective of ethical chatbots in customer
service, we propose an approach to address the problem of evaluating the
ethical behavior of customer service employees for violations of the codes of ethics
and conduct of their company. Our approach is based on Answer Set
Programming (ASP) and Inductive Logic Programming (ILP). We use ASP for ethical
knowledge representation and reasoning. The ASP rules needed for reasoning
are learned using ILP. ASP, the non-monotonic reasoning paradigm, was chosen
because it is common to say that ethical rules are default rules, which means
that they tolerate exceptions. This in fact nominates non-monotonic logics which
simulate common sense reasoning to be used to formalize di erent ethical
conceptions. In addition, there are the many advantages of ASP including it is
expressiveness, exibility, extensibility, ease of maintenance, readability of its
code and the performance of the available 'solvers', etc. which gained ASP an
important role in the eld of Arti cial Intelligence. ILP was chosen as a machine
learning approach because ILP as a logic-based machine learning approach
supports two very important and desired aspects of machine ethics implementation
into arti cial agents viz. explainability and accountability [18], ILP is known for
its explanatory power which is compelling: as an action is chosen by the system,
clauses of the principle that were instrumental in its selection can be determined
and used to formulate an explanation of why a particular action was chosen
over others; moreover, ILP also seems better suited than statistical methods to
domains in which training examples are scarce as in the case of ethical domain.
Many research works have suggested the use of ASP and ILP, but separately,
for programming ethical agents which we review in Section ve. We think that
an approach combining both programming languages would have a great
potential for programming ethical agents. Finally we would like to mention that our
approach can be applied to generate detailed codes of ethics for any domain.
Structure The Paper is organized as follows: in Sections two, we brie y
introduce both ASP and ILP as the logic programming techniques used in this work.
In Section three we present our approach with examples. Then Section four we
review the research done for modeling ethical agents using ASP and ILP. Then
we conclude with future directions in Section ve.</p>
    </sec>
    <sec id="sec-2">
      <title>Background</title>
      <sec id="sec-2-1">
        <title>ASP Formalism</title>
        <p>ASP is a logic programming paradigm under answer set (or "stable model")
semantics [16], which applies ideas of autoepistemic logic and default logic. ASP
features a highly declarative and expressive programming language, oriented
towards di cult search problems. It has been used in a wide variety of applications
in di erent areas like problem solving, con guration, information integration,
security analysis, agent systems, semantic web, and planning. ASP has emerged
from interaction between two lines of research: rst on the semantics of
negation in logic programming, and second on applications of satis ability solvers to
search problems [21]. In ASP, search problems are reduced to computing answer
sets, and an answer set solver (i.e., a program for generating stable models) is
used to nd solutions. The expressiveness of ASP, the readability of its code and
the performance of the available "solvers" gained ASP an important role in the
eld of arti cial intelligence.</p>
        <p>An answer set Program is a collection of rules of the form,</p>
        <p>H</p>
        <p>
          A1; : : : ; Am; not Am+1; : : : ; not An
were each of Ai's is a literal in the sense of classical logic. Intuitively the above
rule means that if A1; : : : ; Am are true and if Am+1; : : : ; An can be safely assumed
to be false then H must be true. The left-hand side and right-hand side of rules
are called head and body, respectively. A rule with empty body (n = 0) is called
a unit rule, or fact. A rule with empty head is a constraint, and states that
literals of the body cannot be simultaneously true in any answer set. Unlike
other semantics, a program may have several answer sets or may have no answer
set, each answer set is seen as a solution of given problem, encoded as an ASP
program (or, better, the solution is extracted from an answer set by ignoring
irrelevant details and possibly re-organizing the presentation). So, di erently
from traditional logic programming, the solutions of a problem are not obtained
through substitutions of variables values in answer to a query. Rather, a program
describes a problem, of which its answer sets represent the possible solutions.
For more information about ASP and its applications the reader can refer, among
many, [
          <xref ref-type="bibr" rid="ref15">15</xref>
          ], [
          <xref ref-type="bibr" rid="ref12">12</xref>
          ] and the references therein eld of arti cial intelligence.
2.2
        </p>
      </sec>
      <sec id="sec-2-2">
        <title>ILP Approach</title>
        <p>ILP [24] is a branch of arti cial intelligence (AI) which investigates the
inductive construction of logical theories from examples and background knowledge.
It is the intersection between logic programming and machine learning. From
computational logic, inductive logic programming inherits its representational
formalism, its semantical orientation, and various well-established techniques.</p>
        <p>In the general settings, we assume a set of Examples E, positive E+ and
negative E , and some background knowledge B. An ILP algorithm nds the
hypothesis H such that B S H j= E+ and B S H 6j= E . The possible hypothesis
space is often restricted with a language bias that is speci ed by a series of mode
declarations M [25]. A mode declaration is either a head declaration modeh(r,
s) or a body declaration modeb(r, s), where s is a ground literal, this scheme
serves as a template for literals in the head or body of a hypothesis clause, where
r is an integer, the recall, which limits how often the scheme can be used. An
asterisk denotes an arbitrary recall. A scheme can contain special placemaker
terms of the form ] type, +type and -type, which stand, respectively, for ground
terms, input terms and output terms of a predicate type. Each set M of mode
declarations is associated with a set of clauses L(M ), called the language of M,
such that C = a l1; : : : ; ln 2 L(M ) i the head atom a (resp. each body literal
li ) is obtained from some head (resp. body) declaration in M by replacing all
] placemakers with ground terms and all + (resp. -) placemakers with input
(resp. output) variables. Finally, it is important to mention that ILP has found
applications in many areas. For more information on ILP and applications, refer,
among many to [26].</p>
        <p>
          ILP has received a growing interest over the last two decades. ILP has many
advantages over statistical machine learning approaches: the learned hypotheses
can be easily expressed in plain English and explained to a human user, and it is
possible to reason with learned knowledge. Most of the work on ILP frameworks
has focused on learning de nite logic programs (e.g. among many, [24], [35])
and normal logic programs (e.g. [
          <xref ref-type="bibr" rid="ref11">11</xref>
          ]). In the recent years, several new learning
frameworks and algorithms have been introduced for learning under the answer
set semantics. In fact generalizing ILP to learn ASP makes ILP more powerful.
Among many, refer to [32]. [22], [30], [34], and [20].
3
        </p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>Our Approach: An Application</title>
      <p>Codes of ethics in domains such as customer service are mostly abstract general
codes, which make them quite di cult to apply. Examples, con dentiality,
accountability, honesty, delity, etc. They are subject to interpretations and may
have di erent meanings in di erent contexts. Therefore it is quite di cult if not
impossible to de ne codes in a manner that they maybe applied deductively.
There are no intermediate rules that elaborate the abstract rules or explain how
they apply in concrete circumstances. Consider for example the following codes
of ethics taken from a customer service code of ethics and conduct document of
some company:
Con dentiality: The identity of the customer and the information provided will
be shared only on a \need-to-know" basis with those responsible for addressing
and resolving the concern.</p>
      <p>Accuracy: We shall do all it can to collect, rely and process customer requests
and complaints accurately. We shall ensure all correspondence is easy to
understand, professional and accurate.</p>
      <p>Accountability: Our employees are committed to own a service request or a
complaint received and they are responsible for nding answers and getting the issue
resolved. If the employee cannot solve the problem himself, he is expected to nd
someone who can and follow up until the issue is resolved.</p>
      <p>Abstract principles such as these seems reasonable and appropriate, but in fact it
is very hard to apply them in real-world situations [19] (e.g. how can we precisely
de ne "We shall ensure all correspondence is easy to understand, professional
and accurate."? or "shall do all it can to collect, rely and process customer
request and complaint accurately."?). It is not possible for experts to de ne
intermediate rules to cover all possible situations to which a particular code
applies. In addition, there are many situations in which obligations might con ict.
An important question to ask here is how can the company's managers evaluate
the ethical behavior of employees in such setting. To achieve this end, and help
managers to have detailed rules in place for monitoring the behavior of their
employees at customer service for violations of the company's ethical codes, we
propose an approach for generating these detailed rules of evaluation from
interactions with customers. So, the new codes of ethics to be used for ethical
evaluation are a combination of the existing clear codes (those that give a clear
evaluation procedure that can be deductively encoded using ASP) and the newly
generated ones. The approach uses ASP Language as the knowledge
representation and reasoning language. ASP is used to represent the domain knowledge,
the ontology of the domain, and scenarios information. Rules required for ethical
reasoning and evaluation of the agent behavior in a certain scenario are learned
using XHAIL [30], which is a Non-monotonic ILP algorithm. The inputs to the
system are a series of scenarios(cases) in the form of requests and answers, along
with the ethical evaluation of the response considering each particular situation.
The system remembers the facts about the narratives and the annotations given
to it by the user, and learns to form rules and relations that are consistent with
the evaluation given by the user of the responses to the given requests.
To illustrate our approach, let us consider the following scenario: a customer
contacting the customer service asking for a particular product of the company, and
the employee talking about the product characteristics and trying to convince
the customer to buy the product. (S)he started saying that the product is
environmentally friendly (which is irrelevant in this case), and this is an advantage
of their product over the same products of other companies. The question: is it
ethical for the employee to say that? The answer is no, it is unethical to make
use of irrelevant but sensitive slogans like environmentally friendly" to attract
and provoke the customers to buy a certain product or service. This would be a
violation of 'Honesty'.</p>
      <p>We can form an ILP task ILP (B; E = fE+; E g; M ) for our example, where B
is the background knowledge:
8&gt;ask(customer; inf oabout(productx)):
&gt;
&gt;&gt;&gt;answer(environmentallyF riendly):
&gt;
&gt;
&gt;&gt;&gt;sensitiveSlogan(environmentallyF riendly):
&gt;
&gt;
&gt;&gt;&gt;not relevant(environmentallyF riendly):
&gt;
&gt;
B = &lt;&gt;answer(xxx): sensitiveSlogan(xxx): not relevant(xxx):
&gt;answer(yyy): sensitiveSlogan(yyy): not relevant(yyy):
&gt;
&gt;&gt;&gt;answer(zzz): not sensitiveSlogan(zzz): relevant(zzz):
&gt;
&gt;
&gt;&gt;&gt;answer(eee): not sensitiveSlogan(eee): relevant(eee):
&gt;
&gt;
&gt;&gt;&gt;&gt;not relevant(X) : not relevant(X); answer(X):
&gt;
&gt;:not sensitiveSlogan(X) : not sensitiveSlogan(X); answer(X):</p>
      <sec id="sec-3-1">
        <title>E are the positive and negative examples:</title>
        <p>8&gt;example unethical(environmentallyF riendly):
E+ = &lt;example unethical(xxx):
&gt;:example unethical(yyy):</p>
        <p>E =
(example notunethical(zzz):</p>
        <p>example notunethical(eee):</p>
      </sec>
      <sec id="sec-3-2">
        <title>M is The mode declarations:</title>
        <p>8&gt;modeh unethical(+answer):
&gt;
&gt;&gt;&gt;&gt;modeb sensitiveSlogan(+answer):
M = &lt;modeb notsensetiveSlogan(+answer):
&gt;&gt;&gt;&gt;modeb notrelevant(+answer):
&gt;
&gt;:modeb relevant(+answer):
In the running example, E contains three positive examples and two negative
examples which must all be explained. XHAIL derives the hypothesis in three
steps process:
Step 1: The Abductive Phase: the head atoms of each Kernel Set are computed.
The set of abducibles (ground atoms) is = Sin=1 i such that B S j= E
where each i is a ground instance of the modeh(d) declaration atom. This is
a straight-forward abductive task. For our example there is only one modeh
declaration. Then the set contain ground instances of this atom in the single
modeh declaration. So the set of abducibles for our example would be:
8&gt;unethical(environmentallyF riendly):
= &lt;unethical(xxx):</p>
        <p>&gt;:unethical(yyy):
Step 2: The Deductive Phase: This step computes the body literals of a Kernel
Set. i.e., the clause i i1 : : : imi for each i 2 is computed, where B S j=
ij; 81 i n; 1 j mi and each clause i i1 : : : imi is a ground instance
of a rule in L(M ) (the language of M, where M is the mode declarations). To do
this, each head atom is saturated with body literals using a nonmonotonic
generalization of the Progol level saturation method ([25]).In our example, contains
three atoms where each one leads to a clause ki, so, we will have K1; K2; K3.
The rst atom 1 = unethical(environmentallyF riendly) is initialized to the
head of the clause K1. The body of K1 is saturated by adding all possible ground
instances of the literals in modeb(s) declarations that satisfy the constraints
mentioned earlier. There are ten ground instances of the literals in the modeb(d)
declarations, but only two of them, i.e. sensitiveSlogan(environmentallyF riendly)
and not relevant(environmentallyF riendly) can be added to the body of K1.
At the end of the deductive phase we will have the set of ground clauses K:
8&gt;K1 = unethical(environmentallyF riendly)
&gt;
&gt;&gt;&gt;&gt; sensitiveSlogan(environmentallyF riendly);
K = &lt; not relevant(environmentallyF riendly):
&gt;&gt;&gt;&gt;K2 = unethical(xxx) sensitiveSlogan(xxx); not relevant(xxx):
&gt;:&gt;K3 = unethical(yyy) sensitiveSlogan(yyy); not relevant(yyy):
and the set of their "variablized" version that is obtained by replacing all input
and output terms by variables:</p>
        <p>8&gt;unethical(V )
Kv = &lt;unethical(V )
&gt;:unethical(V )
sensitiveSlogan(V ); not relevant(V ):
sensitiveSlogan(V ); not relevant(V ):
sensitiveSlogan(V ); not relevant(V ):
Step 3: The Inductive Phase: By construction, the Kernel Set covers the
provided examples. In this phase XHAIL computes a compressive theory H =
Sin=0 1 i di1; : : : ; dim0i that subsumes K and entails E w.r.t. B. This is done
through actual search for hypothesis which is biased by minimality i.e.
preference towards hypothesis with fewer literals. Thus a hypothesis is constructed by
deleting from Kv as many literals (and clauses) as possible while ensuring correct
coverage of the examples. This is done by subjecting Kv to syntactic
transformation of its clauses which involves two new predicates try=3 and use=2. This
syntactic transformation results in the following defeasible program:
8&gt;unethical(V ) use(1; 0); try(1; 1; vars(V )); try(1; 2; vars(V )):
&gt;
&gt;&gt;&gt;&gt;try(1; 1; vars(V )) use(1; 1); sensitiveSlogan(V ):
UKv = &lt;try(1; 1; vars(V )) not use(1; 1):
&gt;&gt;&gt;&gt;try(1; 2; vars(V )) use(1; 2); not relevant(V ):
&gt;&gt;:try(1; 2; vars(V )) not use(1; 2):
literals and clauses necessary to cover the examples are selected from UKv by
means of abducing a set of use=2 atoms as explanation for the examples from
the ALP (Abductive Logic Programming) task ALP (B [ UKv ; fuse=2g; E).</p>
        <p>2 = fuse(1; 0); use(1; 1); use(1; 2)g is a minimal explanation for this ALP task.
use(1; 0) is the head atom of one of the Kv clauses (which are identical in this
example), use(1; 1) and use(1; 2) correspond to the body literals. The output
hypothesis is constructed by these literals. The three clauses in Kv produce
identical transformations resulting in the same nal hypothesis:
H = nunethical(V )</p>
        <p>sensitiveSlogan(V ); not relevant(V ); answer(V ):
XHAIL did learn this rule in a total time of 1.671 seconds on AMD Athlon(tm)
II Dual-Core M300x2 laptop PC running Ubuntu 14.04 with 3.6G Ram: loading
time : 0.987s, abduction : 0.221s, deduction : 0.031s, induction : 0.055s</p>
        <p>Let us now consider our agent having three cases together, the above
mentioned case and the following two cases(scenarios) along with a set of examples
for each case.
case1: an employee give information about client1 to client2 without checking or
being sure that client2 is authorized to be given such information. This behavior
is unethical because it violates 'Con dentiality' which is very critical especially
when dealing with sensitive products and services like services or products
provided to patients with critical medical conditions.
case2: a customer contacting customer service asking to buy a certain product x.
In this context the customer asks about a similar product of another competitor
company which is slightly cheaper. Then the employee, in order to convince the
customer to buy their product and not think about the other company product,
said that the other company uses substandard materials in their production.
The question: is this an ethical answer from the employee to say that the other
company uses substandard materials, supposing that it is true? The answer: no.
In general, it is true that the employee should be truthful with the customer,
but in this context, the answer is not ethical because it is not ethical and not
professional to talk bad about other competitor companies.</p>
        <p>From these three cases our agent learned the following three rules for
evaluating the employees ethical behavior (for the lack of space we omitted the
details):</p>
        <p>sensitiveSlogan(V ); not relevant(V ); answer(V ):
8&gt;unethical(V )
&gt;
&gt;&gt;&gt;unethical(giveinf o(V 1; V 2))
&gt;
&lt;
&gt;&gt;&gt;unethical(tell(V 2; inf oabout(V 2)))
&gt;
&gt;
&gt;
:</p>
        <p>not authorized(tell(V 1; inf oabout(V 2))); client(V 1); client(V 2):
H =</p>
        <p>context(competitor(V 2)); badinf o(V 1); inf o(V 1); company(V 2):
The above three hypotheses were learned by our agent in a total time of 9.391
seconds: loading time : 0.271s, abduction : 0.124s, deduction : 0.091s, induction
: 8.809s . In addition, supposing that our agent already have the following rule
as a background knowledge in his knowledge base:
rule1 = nunethical(V )</p>
        <p>not correct(V ); answer(V ):
which says that it is unethical to give incorrect information to the customers. So
now our agent have four rules for ethical evaluation (the one that she already
have plus the three learned ones).
4</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>Related Work</title>
      <p>Engineering machine ethics (building practical ethical machines) is not just
about traditional engineering, we need to nd out how to practically build
machines that are ethically constrained and also reason about ethics, which of course
involve philosophical aspects. Even though it is more computational by nature.
Below we review research works which used ASP for modeling ethical agents and
then those that use ILP.
4.1</p>
      <sec id="sec-4-1">
        <title>Non-monotonic Logic and Ethical Reasoning</title>
        <p>
          Ethical reasoning is a form of common sense reasoning. Thus, it seems
appropriate to use non-monotonic logics which simulates common sense reasoning to
formalize di erent ethical conceptions. Moreover, logical representations help to
make ideas clear and highlight di erences between di erent ethical systems.
Ethical rules usually dictate the ethical behavior, i.e. help us to decide what to do
and what not to do. Thus, to achieve this ethical behavior, it is required to de ne
a decision making procedure. ASP as a purely declarative nonmonotonic logic
paradigm has been nominated as a modern logic-based AI technique to model
ethical reasoning systems. Using the nonmonotonic logic of ASP o ers a more
feasible approach than the deontic logics approaches (like [
          <xref ref-type="bibr" rid="ref9">9</xref>
          ] and [28]), since it
can address not only the consequentialist ethical systems but also deontic ones
as it can represent (limited forms of) modal logic and deontic logics. In addition,
the existence of solvers to derive consequences of di erent ethical principles
automatically, can help in precise comparison of ethical theories, and makes it easy
to validate our models in di erent situations.
        </p>
        <p>
          Using nonmonotonic logic is appropriate to address the opposition between
generalism and particularism by capturing justi ed exceptions in general ethics
rules. This opposition corresponds to the old opposition between written laws
and the cases on which the laws are based. General rules that they may be
correct in theory, but not applicable to all particular cases. In [
          <xref ref-type="bibr" rid="ref13">13</xref>
          ], the authors
formalized three ethical conceptions (the Aristotelian rules, Kantian categorical
imperative, and Constant's objection) using nonmonotonic logic, particularly
ASP. Each model is illustrated using the classical dilemma of lying [
          <xref ref-type="bibr" rid="ref13">13</xref>
          ]. In the
case of lying, default rules with justi ed exceptions could be used to satisfy a
general rule that prohibit lying, while simultaneously recommending telling a lie
in given particular situations where the truth would violate other rules of duty.
        </p>
        <p>
          [
          <xref ref-type="bibr" rid="ref8">8</xref>
          ] proposes an ethical modular architecture that allows for systematic and
adaptable representation of ethical principles. This work is implemented in ASP.
In their framework, the authors model the knowledge of the world in separated
models from those used for ethical reasoning and judgment. Many theories of the
right were modeled in this paper. However, as mentioned before the framework
presented in this paper assesses the permissibility of an action or a set of actions
using di erent theories of Good and Right separately, i.e. it only permits to
judge an option with respect to a single ethical principle. It doesn't handle
the con icting decisions given by di erent theories, i.e. doesn't provide a nal
decision for the agent about what it should do as a result.
        </p>
        <p>
          In the context of logic-based ethics, Pereira and Saptawijaya have proposed
the use of di erent logic-based features for representing diverse issues of moral
facets such as moral permissibility, doctrines of Double E ect and Triple E ect,
the Dual-process Model, counterfactual thinking in moral reasoning. [27]. Their
formalization embeds the moral requirements directly into the model of a
situation. By indicating, e.g., whether a killing is intentional or not, the program is
told whether the outcome of the action ts with the ethical rules in place. This
approach fails in representing the actual reasoning that underlies moral decision
making, namely, what constitutes intentionality. Furthermore, because they
automatically specify the ethical character of the situation outcome, one needs to
write di erent programs for each case. This is redundant and can lead to
inconsistencies. Their approach was also criticized by [
          <xref ref-type="bibr" rid="ref7">7</xref>
          ], for there is no account for
causality and ethical responsibility because action and its consequences are not
dynamically linked; the relationship between them is stated by the
programmer rather than inferred. In addition, it fails to provide a general framework
to model morality computationally because the model cannot logically confront
ethical theories making their assumptions explicit and cannot enable us to
explore and generate new ethical dilemmas for further testing.
        </p>
        <p>
          In [
          <xref ref-type="bibr" rid="ref10">10</xref>
          ], the authors introduce a generic judgment model that an agent can use
in order to judge the ethical dimensions of both its own behavior and the other
agents' behaviors. This model is based on a rationalist and explicit approach
that distinguish theory of good and theory of right. A proof of concept was
implemented in ASP. However, the model is still based on a qualitative approach.
Whereas we can de ne several moral valuations, there is neither a degree of
desires, nor a degree of capability, nor a degree of rightfulness. Moreover, ethical
principles need to be more precisely de ned to capture various sets of theories
suggested by philosophers.
        </p>
        <p>
          In [33], Sergot provides an alternative representation to the argumentative
representation of a moral dilemma case concerning a group of diabetic persons,
presented in [
          <xref ref-type="bibr" rid="ref5">5</xref>
          ], where the authors used value-based argumentation to solve
this dilemma. According to Sergot, the argumentation framework representation
doesn't work well and doesn't scale. Sergot proposal for handling this kind of
dilemmas is based on Defeasible Conditional Imperatives [17]. The proposed
solution was implemented in ASP.
4.2
        </p>
      </sec>
      <sec id="sec-4-2">
        <title>ILP and Machine Ethics</title>
        <p>As mentioned above, ethics is more complicated than following a single absolute
ethical principle. Thus, according to Ross ([31]), any single-principled ethical
theory like Act Utilitarianism is sentenced to fail. Ross suggested that ethical
decision making involves considering several Prima Facie duties (duties that in
general we should try to follow, where on some occasions the strongest duty can
override others). Ross' Theory of Prima Facie Duties seems to more completely
account for the di erent types of ethical obligations that most of us recognize,
than any single-principled ethical theory. However, Ross' Theory gives us no
decision procedure for determining which duty becomes the strongest one, when
several duties pull in di erent directions as often happens in an ethical dilemma.
Rawls' "Re ective Equilibrium" approach [29] was suggested later for re
ecting on duties wherever necessary, in order to achieve an acceptable coherence
amongst them, the so-called "equilibrium" which serves as decision procedure,
lacking in Ross' theory. ILP was used to handle the non-classical relationships
that might exist between di erent duties.</p>
        <p>
          In [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ], authors created a system called W.D. Their system follow the theory
of prima facie duties of Ross [31]. In W.D., the strength of each duty is measured
by assigning it a weight, capturing the view that a duty may take precedence
over another. W.D. computes, for each possible action, the weighted sum of
duty satisfaction, and returns the greatest sum as the right action. In order to
improve the decision, in the sense of conforming to a consensus of correct ethical
behavior, the weight of a duty is allowed to be adjusted through a supervised
learning, by acquiring suggested action from the user. This weight adjustment
to re ne moral decisions is inspired by the re ective equilibrium of Rawls [29].
W.D. uses inductive logic programming [24] to achieve this end. W.D. uses ILP
to learn the relation supersedes(A1,A2) which states that action A1 is preferred
over action A2 in an ethical dilemma involving these choices.
        </p>
        <p>
          MedEthEx [
          <xref ref-type="bibr" rid="ref4">4</xref>
          ], and EthEl [
          <xref ref-type="bibr" rid="ref1">1</xref>
          ] are two systems based on a more speci c theory
of prima facie duties viz., the principle of Biomedical ethics of Beauchamp and
Childress [
          <xref ref-type="bibr" rid="ref6">6</xref>
          ]. Moreover the two systems are implemented using ILP [24]. ILP is
used in both MedEthEx, and EthEl to learn the relation supersedes(A1, A2),
i.e., whether action A1 supersedes (i.e., is ethically preferable to) action A2. The
training (positive) examples comprise cases, where each case is associated with an
estimate satisfaction/violation value of each duty for each possible action (scaled
from -2 to 2) and the ethically preferred action for the case. The considered cases
are a variety of the following type of ethical dilemma: "A healthcare professional
has recommended a particular treatment for her competent adult patient, but
the patient has rejected it. Should the healthcare professional try to change the
patient's mind or accept the patient's decision as nal?" EthEl is applied to
the domain of eldercare with the main purpose to remind a patient to take her
medication, taking ethical duties into consideration.
        </p>
        <p>
          GenEth (General Ethical Dilemma Analyzer)[
          <xref ref-type="bibr" rid="ref2">2</xref>
          ] is another System that uses
ILP as a machine learning technique to discern ethical principles that resolve
ethical dilemmas due to con icting obligations and duties. GenEth has been
used to codify principles in a number of domains pertinent to the behavior of
autonomous systems.
        </p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>Conclusions and Future Directions</title>
      <p>In this paper we reviewed approaches to modeling ethics using ASP (rule-based
approaches) and ILP (case-based learning approaches). Then we presented an
approach that makes use of ASP for ethical knowledge representation and
reasoning, and uses inductive logic programming for learning ASP rules needed for
ethical reasoning. Combining ASP with ILP for modeling ethical agents provides
many advantages: increases the reasoning capability of our agent; promotes the
adoption of hybrid strategy that allow both topdown design and bottom up
learning via context sensitive adaptation of models of ethical behavior; allows
the generation of rules with valuable expressive and explanatory power which
equips our agent with the capacity to give an ethical evaluation and explain
the reasons behind this evaluation. In other words, our method supports
transparency and accountability of such models, which facilitates instilling con dence
and trust in our agent. Furthermore, in our opinion and for the sake of
transparency, evaluating the ethical behavior of others should be guided by explicit
ethical rules determined by competent judges or ethicists or through consensus
of ethicists. Our approach provides support for developing these ethical rules.</p>
      <p>Computational techniques, such as neural networks, can be viewed as
making use of a form of inductive inference. However, ILP algorithms, unlike neural
networks, output rules which are easily understood by people. Statistical
machine learning methods produce models that are not comprehensible for humans
because they are algebraic solutions to optimization problems such as risk
minimization or data likelihood maximization. These methods do not produce any
intuitive description of the learned model. Lack of intuitive descriptions makes
it hard for users to understand and verify the underlying rules that govern the
model. Also, these methods cannot produce a justi cation for a prediction they
compute for a new data sample. Furthermore, if prior knowledge (background
knowledge) is extended in these methods, then the entire model needs to be
re-learned. Finally, no distinction is made between exceptions and noisy data
in these methods. This makes ILP particularly appropriate for scienti c theory
formation tasks in which the comprehensibility of the generated knowledge is
essential. Moreover, in an ill-de ned domain like the ethics domain, it is
infeasible to de ne abstract codes in precise and complete enough terms to be able
to use deductive problem solvers to apply them correctly. A combination of
deductive (rule-based) and inductive (case-based learning) is needed. The use of
ASP allows us to encode the domain information plus the nonmonotonic domain
rules that are already available and can help our agent in the evaluation process.
However, in the many other cases where we don't have clear intermediate rules
for evaluation, learning is needed to learn these rules and then add them to our
knowledge base to be used for future evaluations.</p>
      <p>With respect to the approach of the works mentioned in the previous section,
the authors used ILP to learn rules to help decide between two or more
available actions based on a set of involved ethical duties. So their approach can be
applied to choose the most ethical action when we have speci c clear ethical
duties involved and to do so we need to assign weights of importance(priority)
to these duties for each available action, then the system computes the weighted
sum for each action, and the one with highest weighted sum is the best action
to do. In this approach it is not really clear the basis of assigning weights to
duties(we doubt whether we can really quantify the importance of ethical duties
on a grade from 2 to -2 as was done in these works). On the other hand, in our
approach we use ILP to generate rules for ethical evaluation of actions(in the
case of the application we are handling in this paper, actions are the responses
to requests from customers) based on di erent facts extracted from cases. In
other words ILP is used to learn the relation between the evaluation of an
action to be ethical or unethical and the related facts in the case scenario. To this
end, di erent facts are extracted from the case scenario and our system try to
nd the relation between these facts and the conclusion (ethical or un ethical or
probably unknown).our approach can be used to generate ethical rules to follow
when there is no ethical rules available in place for evaluation, by considering the
involved facts and possibly involving counterfactual reasoning in the evaluation.
We think that our approach is more general and can be used to generate ethical
rules for any domain (and/or elaborate existing ones).</p>
      <p>As a matter of fact XHAIL provides an appropriate framework for learning
ethical rules for customer service. However XHAIL has the following limitations: the
problem of scalability: The computation of a hypothesis H depends on the
Kernel Set (K ) generation, particularly on the choice of (the set of heads of K's
clauses), which is a set of instances of head mode declaration atoms derived from
B; the Kernel Set is generated from all positive examples at once, then XHAIL
performs a search in the space of theories that subsume it, in order to arrive to
a "good" hypothesis. Thus, when the size of the examples is small, XHAIL
performs well. But with the increasing size of examples space, XHAIL scales poorly,
two reasons are behind this. First: the increased computational complexity of
abduction, which lies at the core of its functionality; second: the combinatorial
complexity of learning whole theories which may result in an intractable search
space. Furthermore, every time we want to add new cases, XHAIL need to
relearn the new hypothesis from the whole set of examples (old ones plus the new
added ones). Therefore, to cope with large volumes of sequential data and also
to cope with ethics change over time, we need an incremental learning technique
that is able to revise the old learned hypothesis when a new set of examples
arrive. In fact we are working now to improve the ethical evaluation
capabilities of our agent by using an incremental learning algorithm like ILED ([20])
to overcome the limitations mentioned above. So our agent can learn
incrementally from the interactions with customers to give more accurate evaluations to
customer service employees ethical behavior. Furthermore, we would like to test
our agent in a real chat scenario. Finally, as another future direction we would
like to investigate the possibility of judging the ethical behavior from a series of
related chat sessions.
16. Gelfond, M., Lifschitz, V.: The stable model semantics for logic programming. In:
Kowalski, R., Bowen, K. (eds.) Proc. of the 5th Intl. Conf. and Symposium on
Logic Programming. pp. 1070{1080. MIT Press (1988)
17. Hansen, J.: Prioritized conditional imperatives: problems and a new proposal.
Autonomous Agents and Multi-Agent Systems 17(1), 11{35 (2008)
18. High-Level Expert Group on Arti cial Intelligence: Draft ethics guidelines for
trustworthy AI. European Commission, Brussel (2018),
https://ec.europa.eu/digitalsingle-market/en/news/draft-ethics-guidelines-trustworthy-ai
19. Jonsen, A.R., Toulmin, S.E.: The abuse of casuistry: A history of moral reasoning.</p>
      <p>Berkeley: Univ of California Press (1988)
20. Katzouris, N., Artikis, A., Paliouras, G.: Incremental learning of event de nitions
with inductive logic programming. Machine Learning 100(2-3), 555{585 (2015),
https://doi.org/10.1007/s10994-015-5512-1
21. Kautz, H.A., Selman, B., et al.: Planning as satis ability. In: ECAI. vol. 92, pp.</p>
      <p>359{363. Citeseer (1992)
22. Law, M., Russo, A., Broda, K.: Iterative learning of answer set programs from
context dependent examples. TPLP 16(5-6), 834{848 (2016)
23. Moor, J.H.: The nature, importance, and di culty of machine ethics. IEEE
intelligent systems 21(4), 18{21 (2006)
24. Muggleton, S.: Inductive logic programming. New generation computing 8(4), 295{
318 (1991)
25. Muggleton, S.: Inverse entailment and progol. New Generation Comput. 13(3&amp;4),
245{286 (1995), https://doi.org/10.1007/BF03037227
26. Muggleton, S., Raedt, L.D.: Inductive logic programming: Theory and
methods. J. Log. Program. 19/20, 629{679 (1994), https://doi.org/10.1016/
0743-1066(94)90035-3
27. Pereira, L.M., Saptawijaya, A.: Programming Machine Ethics, Studies in Applied
Philosophy, Epistemology and Rational Ethics, vol. 26. Springer (2016), https:
//doi.org/10.1007/978-3-319-29354-7
28. Powers, T.M.: Prospects for a kantian machine. IEEE Intelligent Systems 21(4),
46{51 (2006)
29. Rawls, J.: A theory of justice. Harvard University Press, Cambridge (1971)
30. Ray, O.: Nonmonotonic abductive inductive learning. J. Applied Logic 7(3), 329{
340 (2009), https://doi.org/10.1016/j.jal.2008.10.007
31. Ross, W.D.: The Right and the Good. Oxford University Press, Oxford (1930)
32. Sakama, C., Inoue, K.: Brave induction: a logical framework for learning from
incomplete information. Machine Learning 76(1), 3{35 (2009)
33. Sergot, M.: Prioritised Defeasible Imperatives. Dagstuhl Seminar 16222
Engineering Moral Agents { from Human Morality to Arti cial Morality
(2016), https://materials.dagstuhl.de/files/16/16222/16222.MarekSergot.</p>
      <p>Slides.pdf, schloss Dagstuhl-Leibniz-Zentrum fuer Informatik
34. Shakerin, F., Salazar, E., Gupta, G.: A new algorithm to automate inductive
learning of default theories. TPLP 17(5-6), 1010{1026 (2017)
35. Srinivasan, A.: The Aleph Manual (version 4). Machine
Learning Group, Oxford University Computing Lab (2003),
https://www.cs.ox.ac.uk/activities/machlearn/Aleph/aleph.html
36. Toulmin, S.E.: The uses of argument. Cambridge university press (2003)</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Anderson</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Anderson</surname>
            ,
            <given-names>S.L.</given-names>
          </string-name>
          :
          <article-title>ETHEL: toward a principled ethical eldercare system</article-title>
          . In:
          <article-title>AI in Eldercare: New Solutions to Old Problems, Papers from the 2008 AAAI Fall Symposium</article-title>
          , Arlington, Virginia, USA, November 7-
          <issue>9</issue>
          ,
          <year>2008</year>
          .
          <source>AAAI Technical Report</source>
          , vol.
          <source>FS-08-02</source>
          , pp.
          <volume>4</volume>
          {
          <fpage>11</fpage>
          .
          <string-name>
            <surname>AAAI</surname>
          </string-name>
          (
          <year>2008</year>
          ), http://www.aaai.org/ Library/Symposia/Fall/fs08-
          <fpage>02</fpage>
          .php
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Anderson</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Anderson</surname>
            ,
            <given-names>S.L.</given-names>
          </string-name>
          :
          <article-title>Geneth: A general ethical dilemma analyzer</article-title>
          .
          <source>In: Proceedings of the Twenty-Eighth AAAI Conference on Arti cial Intelligence, July 27 -31</source>
          ,
          <year>2014</year>
          ,
          <string-name>
            <given-names>Quebec</given-names>
            <surname>City</surname>
          </string-name>
          , Quebec, Canada. pp.
          <volume>253</volume>
          {
          <fpage>261</fpage>
          . AAAI Press (
          <year>2014</year>
          ), http://www.aaai.org/Library/AAAI/aaai14contents.php
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Anderson</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Anderson</surname>
            ,
            <given-names>S.L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Armen</surname>
            ,
            <given-names>C.:</given-names>
          </string-name>
          <article-title>Towards machine ethics</article-title>
          .
          <source>In: AAAI-04 workshop on agent organizations: theory and practice</source>
          , San Jose, CA (
          <year>2004</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Anderson</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Anderson</surname>
            ,
            <given-names>S.L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Armen</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          : Medethex:
          <article-title>Toward a medical ethics advisor</article-title>
          . In: Caring Machines:
          <article-title>AI in Eldercare, Papers from the 2005 AAAI Fall Symposium</article-title>
          , Arlington, Virginia, USA, November 4-
          <issue>6</issue>
          ,
          <year>2005</year>
          .
          <source>AAAI Technical Report</source>
          , vol.
          <source>FS-05-02</source>
          , pp.
          <volume>9</volume>
          {
          <fpage>16</fpage>
          . AAAI Press (
          <year>2005</year>
          ), https://www.aaai.org/Library/ Symposia/Fall/fs05-
          <fpage>02</fpage>
          .php
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Atkinson</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bench-Capon</surname>
            ,
            <given-names>T.J.M.:</given-names>
          </string-name>
          <article-title>Addressing moral problems through practical reasoning</article-title>
          .
          <source>In: Deontic Logic and Arti cial Normative Systems</source>
          , 8th International Workshop on Deontic Logic in Computer Science, DEON
          <year>2006</year>
          , Utrecht,
          <source>The Netherlands, July 12-14</source>
          ,
          <year>2006</year>
          ,
          <source>Proceedings. Lecture Notes in Computer Science</source>
          , vol.
          <volume>4048</volume>
          , pp.
          <volume>8</volume>
          {
          <fpage>23</fpage>
          . Springer (
          <year>2006</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Beauchamp</surname>
            ,
            <given-names>T.L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Childless</surname>
            ,
            <given-names>J.F.</given-names>
          </string-name>
          :
          <article-title>Principles of biomedical ethics</article-title>
          .
          <source>International Clinical Psychopharmacology</source>
          <volume>6</volume>
          (
          <issue>2</issue>
          ),
          <volume>129</volume>
          {
          <fpage>130</fpage>
          (
          <year>1991</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Berreby</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bourgne</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ganascia</surname>
            ,
            <given-names>J.G.</given-names>
          </string-name>
          :
          <article-title>Modelling moral reasoning and ethical responsibility with logic programming</article-title>
          .
          <source>In: Logic for Programming</source>
          ,
          <source>Arti cial Intelligence, and Reasoning</source>
          . pp.
          <volume>532</volume>
          {
          <fpage>548</fpage>
          . Springer (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Berreby</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bourgne</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ganascia</surname>
            ,
            <given-names>J.:</given-names>
          </string-name>
          <article-title>A declarative modular framework for representing and applying ethical principles</article-title>
          .
          <source>In: AAMAS</source>
          . pp.
          <volume>96</volume>
          {
          <fpage>104</fpage>
          .
          <string-name>
            <surname>ACM</surname>
          </string-name>
          (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Bringsjord</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Arkoudas</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bello</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          :
          <article-title>Toward a general logicist methodology for engineering ethically correct robots</article-title>
          .
          <source>IEEE Intelligent Systems</source>
          <volume>21</volume>
          (
          <issue>4</issue>
          ),
          <volume>38</volume>
          {
          <fpage>44</fpage>
          (
          <year>2006</year>
          ), https://doi.org/10.1109/MIS.
          <year>2006</year>
          .82
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Cointe</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bonnet</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Boissier</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          :
          <article-title>Ethical judgment of agents' behaviors in multi-agent systems</article-title>
          .
          <source>In: Proceedings of the 2016 International Conference on Autonomous Agents &amp; Multiagent Systems</source>
          , Singapore, May 9-
          <issue>13</issue>
          ,
          <year>2016</year>
          . pp.
          <volume>1106</volume>
          {
          <fpage>1114</fpage>
          .
          <string-name>
            <surname>ACM</surname>
          </string-name>
          (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Corapi</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Russo</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lupu</surname>
          </string-name>
          , E.:
          <article-title>Inductive logic programming as abductive search</article-title>
          .
          <source>In: Technical Communications of the 26th International Conference on Logic Programming</source>
          ,
          <source>ICLP 2010, July 16-19</source>
          ,
          <year>2010</year>
          , Edinburgh, Scotland, UK.
          <source>LIPIcs</source>
          , vol.
          <volume>7</volume>
          , pp.
          <volume>54</volume>
          {
          <fpage>63</fpage>
          .
          <string-name>
            <surname>Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik</surname>
          </string-name>
          (
          <year>2010</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Dyoub</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Costantini</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gasperis</surname>
          </string-name>
          , G.D.:
          <article-title>Answer set programming and agents</article-title>
          .
          <source>Knowledge Eng. Review</source>
          <volume>33</volume>
          ,
          <issue>e19</issue>
          (
          <year>2018</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Ganascia</surname>
            ,
            <given-names>J.G.</given-names>
          </string-name>
          :
          <article-title>Modelling ethical rules of lying with answer set programming</article-title>
          .
          <source>Ethics and information technology 9</source>
          (
          <issue>1</issue>
          ),
          <volume>39</volume>
          {
          <fpage>47</fpage>
          (
          <year>2007</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Gardner</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          v.d.L.:
          <article-title>An arti cial intelligence approach to legal reasoning</article-title>
          . MIT Press (
          <year>1987</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Gelfond</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          :
          <article-title>Answer sets</article-title>
          .
          <source>In: Handbook of Knowledge Representation. Chapter 7, Foundations of Arti cial Intelligence</source>
          , vol.
          <volume>3</volume>
          , pp.
          <volume>285</volume>
          {
          <fpage>316</fpage>
          .
          <string-name>
            <surname>Elsevier</surname>
          </string-name>
          (
          <year>2007</year>
          )
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>