<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Reasoning with Artificial Mental States</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>An Algebraic Approach</string-name>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Nourhan Ehab</string-name>
          <email>nourhan.ehab@guc.edu.eg</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Haythem O. Ismail</string-name>
          <email>haythem.ismail@guc.edu.eg</email>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Cairo University, Egypt Department of Engineering Mathematics</institution>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>German University in Cairo, Egypt Department of Computer Science and Engineering</institution>
        </aff>
      </contrib-group>
      <fpage>27</fpage>
      <lpage>40</lpage>
      <abstract>
        <p>Modelling the human mind, with its astounding complexity, has always been a long-sought goal of AI research. One of the most successful approaches to attain this goal is to ascribe human-like mental states to artificial agents. A mental state is based on a set of mental attitudes such as beliefs, desires, intentions, promises, obligations...etc. While there are several accounts in the literature for endowing artificial agents with mental attitudes, such approaches predominantly focus on investigating each attitude separately or on studying the interaction of a handful of particular attitudes, notably beliefs, desires, and intentions. Since human epistemic and practical reasoning processes are typically more complex, involving a myriad of attitudes, accounting for the interaction among generic mental attitudes is called for. To this end, we present an algebraic framework for modeling the interaction among generic mental attitudes. The framework is used to provide formal semantics for a logical language which may be used by a logic-based agent to reason with arbitrary mental attitudes.</p>
      </abstract>
      <kwd-group>
        <kwd>Agents Architectures</kwd>
        <kwd>Mental States</kwd>
        <kwd>Algebraic Semantics</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>A hallmark of human intelligence is the ability to reason with a wide diversity of mental
attitudes including beliefs, intentions, desires, promises, obligations...etc. which
constitute our collective mental state. We are confronted everyday with situations that require
us to deliberate given our current mental state, and we usually do so with ease. To
demonstrate the variety of mental attitudes we deal with, even in the simplest of
situations, consider the following example.</p>
      <p>Example 1. Ted promised his best friend Marshall to go on a hunting trip with him
during the weekend. Since Ted is a man of his word, he feels obliged to intend his
promises and indeed intends what he is obliged to intend. If Ted intends to go to the
trip, he must rent a car. At the same time, Ted has been procrastinating working on a
long overdue report for weeks. He believes that if he does not work on the report this
weekend, his boss will be really mad and will fire him. Ted fears being fired as he really
likes his job. He regrets that he did not work on the report the previous weeks which
Copyright c 2020 for this paper by its authors. Use permitted under
Creative Commons License Attribution 4.0 International (CC BY 4.0).
makes him feel obliged to work on the report during this weekend. Much to Ted’s relief,
he believes that he can go to the trip and dedicate some time to work on the report there
if the trip location has internet connectivity. However, Ted doubts that there is internet
connectivity at the trip location which makes him fear that he will not be able to work
on the report after all. Since Ted is paranoid, he believes what he fears.</p>
      <p>
        In this example, Ted is reasoning with a mental state comprised of his promises,
beliefs, obligations, intentions, fears, regrets, and doubts. In the modern world we often
talk about machines as if they exhibit human-like mental attitudes as the
aforementioned. Our daily lives typically involve numerous references to machines knowing,
believing, desiring, intending, liking or disliking, understanding, owing, having duties
and rights, or deserving rewards and punishments [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ]. For this reason, a commonly
investigated approach to achieving general AI is to ascribe mental attitudes to artificial
agents as first suggested by McCarthy [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ] and Newell [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ]. Supporters of this line of
research argue that mental-level modelling of artificial agents offers several advantages
on both the theoretical and practical levels [
        <xref ref-type="bibr" rid="ref27">27</xref>
        ].
      </p>
      <p>
        From a theoretical perspective, the abstract nature of mental models proved to be
very useful in analysing and comparing different agent architectures. An example of
this is Levesque’s Computers as Believers paradigm which offered a uniform basis for
analysing general knowledge representation schemes [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ]. On the other hand, from
a practical perspective, mental models offer a convenient abstraction based on
wellunderstood attitudes while hiding low-level implementation-specific details [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. This is
a very useful feature in developing coopeartive multiagent systems as abstract explicit
representations of each of the agents’ mental attitudes enable more coherent
interactions between them [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ]. Moreover, endowing artificial agents with mental attitudes
can facilitate the design of autonomous planning agents. For such agents, explicit
representations of beliefs, desires, and obligations, for example, can drive the agents to take
actions compatible with their beliefs to achieve their desires while respecting their
obligations. Another practical realization of ascribing mental attitudes to artificial agents in
a computational setting is the Agent Oriented Programming (AOP) paradigm. In AOP,
the different modules of a program are viewed as agents possessing mental attitudes
such as beliefs, decisions, capabilities, and obligations [
        <xref ref-type="bibr" rid="ref26">26</xref>
        ].
      </p>
      <p>
        Perhaps the most renowned approach to designing agents with mental attitudes is
the BDI agent architecture [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ] and its extensions to include obligations [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]. These
approaches exclusively focus on investigating the interactions between beliefs, desires,
obligations, and intentions. For this reason, the BDI architecture fails to provide a good
mental model for the human epistemic and practical reasoning processes as they
typically involve a myriad of other mental attitudes such as plans, goals, and fears (just
to name a few). This makes the BDI architecture not suitable for modelling
humancentered trust-worthy agents which recently attracted a lot of research interest [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. Even
if we restrict ourselves to modelling rational agents, the BDI architecture still, in our
opinion, falls short. Archetypal rational behaviour, for instance, is to form intentions to
avoid one’s fears or to mitigate ones’s doubts which can not be represented (in a straight
forward way) within the BDI/BOID frameworks.
      </p>
      <p>
        To address this shortcoming, we propose in this paper a general algebraic framework
capable of representing a first-person perspective of artificial agents possessing any
set of mental attitudes while capturing their interactions. We follow [
        <xref ref-type="bibr" rid="ref27">27</xref>
        ] and define
a mental state as a set of mental attitudes. To the best of our knowledge, there does
not exist in the literature a framework that allows for the reasoning with an arbitrary
set of attitudes like our framework does offering a more refined mental model for a
human-centered logical agent. To this end, we present a logic we refer to as LogAM
(“ Log” stands for logic, “ A” for algebraic, and “ M” for mental states) with precise
semantics where mental states can be represented and the reasoning with the different
mental attitudes of the agent can be captured. In defining the semantics ofLogAM, we
depart from the mainstream modal approaches to representing mental attitudes and take
the algebraic approach instead.
      </p>
      <p>
        As a starting step, we assume in LogAM that the mental state is comprised of binary
mental That is, the mental state can include information about the agent’s beliefs (for
example) but will not include information about the degrees of such beliefs. Further,
we define amonotonic consequence relation for each mental attitude in the mental state
based on pure algebraic notions. We already developed an extension of LogAM to
accommodate non-monotonic reasoning with graded mental states. We will informally
outline this extension in Section 5, but we reserve the formal presentation to a longer
version of the paper. An interesting special case of this graded extension for practical
reasoning with graded beliefs and motivations is presented in [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ].
      </p>
      <p>The rest of the paper is structured as follows. We start by motivating our choice to
pursue the algebraic route in defining the semantics ofLogAM in Section 2. We review
in Section 3 foundational concepts of Boolean algebra on which LogAM is based. We
also generalize the classical notion of filters in Boolean algebra into what we will refer
to as multifilters providing a generalized algebraic treatment of reasoning with mental
states. Next, in Section 4, we present the syntax and semantics of LogAM. In Section
5, we informally describe the graded non-monotonic extension of LogAM. Finally, in
Section 6, we present some concluding remarks.
2</p>
    </sec>
    <sec id="sec-2">
      <title>Why the Algebraic Approach?</title>
      <p>
        Before we delve into the technical details of LogAM, it is perhaps apt to ponder the
merits of choosing to pursue the algebraic route. LogAM is the most recent addition to a
growing family of algebraic logics [
        <xref ref-type="bibr" rid="ref11 ref12 ref13 ref7">11,12,13,7</xref>
        ]. As such, it is essential for a treatment
of reasoning with mental states within the algebraic framework. Hence, independent
motivations for the algebraic approach are also motivations for LogAM. We will briefly
present the motivations for the algebraic approach in this section.
      </p>
      <p>
        The algebraic approach is based on an ontological commitment to propositions as
first-class individuals in the universe of discourse; this leads to a language with no
sentences, but with some of the terms taken to denote propositions. What does this buy
us? Take LogAB [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ] for example. As an algebraic language for reasoning about
belief, it strikes a middle ground between two major approaches to doxastic logic: the
dominant, modal approach [28, for example] and the (now relatively out of fashion)
first-order syntactical approach [22, for instance]. This allows LogAB, on one hand,
to avoid problems of logical omniscience, which mar the classical modal approach,
while, on the the other hand, staying immune to paradoxes of self-reference plaguing
the syntactical approach. LogAG [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ] is an algebraic logic for non-monotonic reasoning
about graded beliefs. It is demonstrably useful for modelling resource-bounded
reasoning; simulating inconclusive reasoning with circular, liar-like sentences; and reasoning
about information arriving over a chain of sources each with a different degree of trust.
As proven in [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ], LogAG can capture a wide array of non-monotonic reasoning
formalisms such as possibilistic logic, circumscription, default logic, autoepistemic logic,
and the principle of negation as failure. As such, LogAG can be considered an
algebraic unifying framework for non-monotonicity. LogACn [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ], which is an algebraic
logic for reasoning about preference, desire, and obligation, avoids the so-called
paradoxes of deontic logic [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ] by, again, abandoning classical possible-worlds semantics.
In [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ], the algebraic approach is adopted for the representation of temporal phenomena
using the language LogAS. In classical first-order approaches to temporal logic [17,1,
for example], tersely axiomatizing temporal properties often calls for the introduction
of reified fluents into the ontology. In these approaches, reference to composite fluents
(conjunctions thereof, for example) either is forbidden (as, for example, in the situation
calculus [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ]) or results in duplicating the logical connectives for statements and
fluentdenoting terms (as, for example, in [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ].) In LogAS, reference to composite fluents is
straightforward, with a single set of proposition-based logical connectives. These
different motivations for the algebraic approach suggest that it is only natural to consider
a language like LogAM if one is to model reasoning with mental states the algebraic
way to gain its several indispensable advantages.
3
      </p>
    </sec>
    <sec id="sec-3">
      <title>Boolean Algebras and Multifilters</title>
      <p>In this section, we start by reviewing the algebraic concepts of Boolean algebras and
filters underlying classical logic, then we extend the notion of filters to accommodate a
logic of mental states where a mental state is a set of mental attitudes.</p>
      <p>
        Definition 1. A Boolean algebra is a sextuple A = hP, +, ·, −, ⊥, &gt;i where P is a
non-empty set with {⊥, &gt;} ⊆ P. A is closed under the two binary operators + and ·
and the unary operator − observing the following properties [
        <xref ref-type="bibr" rid="ref24">24</xref>
        ].
      </p>
      <p>B1.1: a + b = b + a
B1.2: a · b = b · a
B2.1: a + (b + c) = (a + b) + c
B2.2: a · (b · c) = (a · b) · c
B3.1: a + (a · b) = a (Absorption)
B3.2: a · (a + b) = a
B4.1: a · (b + c) = (a · b) + (a · c) (Distribution)
B4.2: a + (b · c) = (a + b) · (a + c)
B5.1: a + −a = &gt;
B5.2: a · −a = ⊥
(Associativity)
(Complements)
(Commutativity)</p>
      <p>
        For the purposes of this paper, we take the elements of P to be propositions and the
operators +, ·, and − to be disjunction, conjunction, and negation, respectively.
The following definition offilters is an essential notion of Boolean algebras to
represent an algebraic counterpart to logical consequence [
        <xref ref-type="bibr" rid="ref24">24</xref>
        ]. Filters are defined in pure
algebraic terms, without alluding to the notion of truth, by utilizing the natural lattice
order ≤ on the the algebra: for p1, p2 ∈ P, p1 ≤ p2 =def p1 · p2 = p1. Henceforth, A
is a Boolean algebra hP, +, ·, −, ⊥, &gt;i.
      </p>
      <sec id="sec-3-1">
        <title>Definition 2. A filter of A is a subset F of P where 1. &gt; ∈ F ; 2. If a, b ∈ F , then a · b ∈ F ; and 3. If a ∈ F and a ≤ b, then b ∈ F .</title>
        <p>The filter generated by Q ⊆ P is the smallest filter F (Q) of which Q is a subset.</p>
        <p>The just presented definition of a filter is only suitable for modelling reasoning
with a single set of propositions Q. With the purpose of modelling the reasoning with
mental states defined as a tuple of sets of propositions where each set represents a
separate mental attitude of the agent, we extend the notion of filters to what we refer to
as multifilters . For this reason, we extend classical filters that rely on the natural order
≤ on the Boolean algebra to what will refer to as multifilters that rely on an order on
tuples of propositions where each proposition belongs to a mental attitude. (Recall that
≤ is the classical lattice order.)
Definition 3. Let k be a positive integer. A k partial-order on A is a partial order
k on Pk such that, (a1, . . . , ak) k (b1, . . . , bk) and bi = ⊥, for some 1 ≤ i ≤
k, only if aj = ⊥, for some 1 ≤ j ≤ k. Further, we say that k is classical in i
just in case, (i) if (a1, ..., ak) k (b1, ..., bk) then ai ≤ bi and (ii) if a ≤ b then
({&gt;}i−1 × {a} × {&gt;}k−i) × (Pi−1 × {b} × Pk−i) ⊆ k.</p>
        <p>In the sequel, we will drop the subscript k in k whenever there is no resulting
ambiguity. We now define multifilters based on ak partial-order .</p>
        <p>Definition 4. Let be a k partial order on A and S ⊆ {1, ..., k}. A -multifilter of A
with respect to S is a tuple F (S) = hF1, F2, ...., Fki of subsets of P such that
1. &gt; ∈ Fi, for all i such that 1 ≤ i ≤ k;
2. for all i, if i ∈ S, a ∈ Fi, and b ∈ Fi, then a.b ∈ Fi; and
3. if (a1, ..., ak) (b1, ..., bk) and (a1, ..., ak) ∈×ik=1 Fi, then (b1, ..., bk) ∈×ik=1 Fi.</p>
        <p>We can observe at this point that the three conditions on multifilters are just
generalizations of the three conditions on filters. The second condition though need not apply
to all the sets F1, ..., Fk. The index set S specifies the sets which are closed under the
meet operation “ ·” and hence observe the second condition. This is necessary as some
mental attitudes need not be closed under “ ·”.</p>
        <p>We next define how multifilters can be generated by a tuple of sets of propositions.
The intuition is that each set of propositions represents a mental attitude and the tuple
of sets represents the collective mental state. In this way, multifilters generalize filters
to accomodate reasoning with multiple mental attitudes where some attitudes need not
be closed under “ ·”.
Definition 5. Let Q1, ..., Qk ⊆ P, be a k partial order on A, and S ⊆ {1, ..., k}. The
-multifilter generated by hQ1, ..., Qki with respect to S, denoted F (hQ1, ..., Qki, S),
is a -multifilter hQ01, ..., Q0ki with respect to S where Q0i is the smallest set containing
Qi, for all 1 ≤ i ≤ k.</p>
        <p>Having defined multifilters, we are now ready to present an important result. The
following theorem states that, under certain conditions, multifilters can be reduced to
classical filters applied to the different sets of propositions representing the mental state.
Theorem 1. Let Q1, ..., Qk ⊆ P, S ⊆ {1, ..., k}, and be a k partial order on A
which is classical in i for some i ∈ S. If F (hQ1, ..., Qki, S) = hQ01, ..., Q0ki, then
Q0i = F (Qi).
4</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>LogAM Languages</title>
      <p>
        In this section, we present the syntax and semantics of LogAM in addition to defining
a logical consequence relation for each mental attitude in the mental state. Utilizing the
multifilters presented in Section 3, we show that our logical consequence relations have
the distinctive properties of classical Tarskian logical consequence. The proofs of the
theorems presented in this section are omitted for space limitations but can be found in
[
        <xref ref-type="bibr" rid="ref8">8</xref>
        ].
LogAM consists of terms constructed algebraically from function symbols. There are
no sentences; instead, we use terms of a distinguished syntactic type to denote
propositions. Propositions are included as first-class individuals in theLogAM ontology and
are structured in a Boolean algebra. Though non-standard, the inclusion of propositions
in the ontology has been suggested by several authors [
        <xref ref-type="bibr" rid="ref2 ref21 ref25 ref5">5,2,21,25</xref>
        ].
      </p>
      <p>A LogAM language is a many-sorted language composed of a set of terms
partitioned into two base sorts: σP is a set of terms denoting propositions and σI is a set
of terms denoting anything else. A LogAM alphabet Ω includes a non-empty,
countable set of constant and function symbols each having a syntactic sort from the set
σ = {σP , σI } ∪ {τ1 −→ τ2 | τ1 ∈ {σP , σI } and τ2 ∈ σ} of syntactic sorts.
Intuitively, τ1 −→ τ2 is the syntactic sort of function symbols that take a single argument
of sort σP or σI and produce a functional term of sort τ2. Given the restriction of the
first argument of function symbols to base sorts, LogAM is, in a sense, a first-order
language.</p>
      <p>An alphabet Ω includes a countably infinite set of variables of the two base sorts;
a set of syncategorematic symbols including the comma, various matching pairs of
brackets and parentheses, the symbol ∀, and a set of logical symbols defined as the
union of the following sets:
– {¬} ⊆ σP −→ σP .
– {∧, ∨} ⊆ σP −→ σP −→ σP
– {Ai}ik=1 ⊆ σP −→ σP .
The symbols ¬, ∧, ∨ denote negation, conjunction, and disjunction respectively.
Ai(t) denotes that the agent has the attitude Ai towards the propositional term t. Terms
involving ⇒ (material implication), ⇔ (logical equivalence), and ∃ are abbreviations
defined in the standard way. In the following, we defineLogAM languages.
Definition 6. A LogAM language L is the smallest set of terms formed according to
the following rules, where t and ti (i ∈ N) are terms in L.</p>
      <p>– All variables and constants in the alphabet Ω are in L.
– f (t1, . . . , tm) ∈ L, where f ∈ Ω is of type τ1 −→ . . . −→ τm −→ τ (m &gt; 0) and
ti is of type τi.
– ¬t ∈ L, where t ∈ σP .
– (t1 ⊗ t2) ∈ L, where ⊗ ∈ {∧, ∨} and t1, t2 ∈ σP .
– ∀x(t) ∈ L, where x is a variable in Ω and t ∈ σP .
– Ai(t) ∈ L, where t ∈ σP .</p>
      <p>We are now ready to defineLogAM theories based on the previously definedLogAM
languages.</p>
      <sec id="sec-4-1">
        <title>Definition 7. A LogAM theory T is a triple hA, R, Si where: – A = (A1, ..., Ak) is a k-tuple where A1, ..., Ak ⊆ σP ; and – R is a set of bridge rules each of the form A1, ..., Ak 7−→</title>
        <p>A1, ..., Ak, A01, ..., A0k ⊆ σP .
– S ⊆ {1, ..., k}.</p>
        <p>A01, ..., A0k where</p>
        <p>
          The tuple A represents the mental state of the agent. Each set A1, ..., Ak represents
a separate mental attitude of the agent. If a propositional term t ∈ Ai, then the agent
has the attitude Ai towards t. It is worth pointing out here the utility of the terms of the
form Ai(t). The membership of Ai(t) in Aj for all 1 ≤ i, j ≤ k means that the agent
has the attitude Aj towards Ai(t). For example, if A3(φ) ∈ A2, A3(φ) represents that
the agent intends φ and A2 represents the agent’s beliefs, then this means that the agent
believes that it intends φ. This is very useful in representing higher-order motivations
first suggested in [
          <xref ref-type="bibr" rid="ref10">10</xref>
          ]. The bridge rules serve to “bridge” propositions across the
different mental attitudes. A bridge rule A1, ..., Ak 7−→ A01, ..., A0k means that if Ai is a
subset of the current i-th mental attitude, then A0i should be added to the current i-th
mental attitude. The bridge rules facilitate the representation of the interaction between
the mental attitudes. The set S specifies the sets inA whose denotations are closed
under the meet operation and, hence, observe the second condition in the definition of
multifilters. This facilitates having some attitudes in the mental state that are not closed
under meet/conjunction. An example of such attitudes are desires. If one desires to go to
the beach and desires to work on the report, one might not desire to go to the beach and
work on the report. We now go back to Example 1 showing a corresponding encoding
of it as a LogAM theory.
        </p>
        <p>Example 2. Let “r” denote working on the report, “t” denote going to the trip, “m”
denote the boss’s getting mad, “f ” denote Ted getting fired, “c” denote Ted renting
a car, “l” denote Ted liking his job, and “i” denote internet connectivity at the trip
location. A possible LogAM theory representing Example 1 is
T = h(A1, A2, A3, A4, A5, A6), R, Si where:
– A1 represents Ted’s promises (P ). A1 = {t}.
– A2 represents Ted’s beliefs (B). A2 = {¬r ⇒ m, m ⇒ f, l, i ⇒ r ∧ t}.
– A3 represents Ted’s intentions (I). A3 = {}.
– A4 represents Ted’s fears (F ). A4 = {}.
– A5 represents Ted’s regrets (R). A5 = {¬r}.
– A6 represents Ted’s doubts (D). A6 = {¬i}.
– A7 represents Ted’s obligations (O). A7 = {}.
– R is the set of instances of the following rule schema where φ is a variable. In what
follows, we eliminate the empty sets in the bridge rules and add to each set the first
letter of the attitude it is representing. For example, the rule {φ},{},{},{},{},{},{}
7−→ {}, {}, {},{},{},{},{φ} will be written as P = {φ} 7−→ O = {φ}.
r1. P = {φ} 7−→ O = {I(φ)}
r2. O = {I(φ)} 7−→ I = {φ}
r3. I = {t} 7−→ I = {c}
r4. B = {l} 7−→ F = {f }
r5. R = {¬r} 7−→ O = {r}
r6. D = {¬i} 7−→ F = {¬r}
r7. F = {φ} 7−→ B = {φ}
In the above rules, we use I(φ) as a mnemonic equivalent to A3(φ) to denote that
Ted intends φ. To illustrate how the bridge rules can be read, r1 represents that if
Ted promised to φ, then he is obliged to intend φ and r2 represents that if Ted is
obliged to intend any φ, then he intends φ. The rest of the rules can be read in a
similar way.
– S = {2}. This means that only the set of beliefs is closed under the meet/conjunction
operation.</p>
        <p>The representation of a first-person variant of a BDI agent as a LogAM theory
should now be straightfoward. The corresponding LogAM theory will contain a mental
state A composed of three sets of attitudes representing the agent’s beliefs, desires, and
intentions. The bridge rules can be used to represent the axioms of BDI logics governing
the interactions between the three mental attitudes.
4.2</p>
        <sec id="sec-4-1-1">
          <title>From Syntax to Semantics</title>
          <p>In this section, we present semantics for the syntax of LogAM in addition to defining
an interpretation function. We start by presenting a key element in the semantics of
LogAM which is the notion of a LogAM structure.</p>
          <p>Definition 8. A LogAM structure is a triple Sk = hD, A, Aki, where
– D is the domain of discourse containing a distinguished non-empty countable set
of propositions P.
– A = hP, +, ·, −, ⊥, &gt;i is a complete, non-degenerate Boolean algebra.
– Ak = {ai | 1 ≤ i ≤ k} where ai : P −→ P, 1 ≤ i ≤ k, is a function modeling an
mental attitude.
A valuation V of a LogAM language is a triple hSk, Vf , Vxi, where Sk is a LogAM
structure, Vf is a function that assigns to each function symbol an appropriate function
on D, and Vx is a function mapping each variable to a corresponding element of the
appropriate block of D. An interpretation of LogAM terms is given by a function [[·]]V .
Definition 9. Let L be a LogAM language and let V be a valuation of L. An
interpretation of the terms of L is given by a function [[·]]V :
– [[x]]V = Vx(x), for a variable x
– [[c]]V = Vf (c), for a constant c
– [[f (t1, . . . , tn)]]V = Vf (f )([[t1]]V , . . . , [[tm]]V ), for an m-adic (m ≥ 1) function
symbol f
– [[(t1 ∧ t2)]]V = [[t1]]V · [[t2]]V
– [[(t1 ∨ t2)]]V = [[t1]]V + [[t2]]V
– [[¬t]]V = −[[t]]V
– [[∀x(t)]]V = Y [[t]]V[a/x]</p>
          <p>a∈D
– [[Ai(t1)]]V = ai([[t1]]V )</p>
          <p>In the rest of the paper, for any Γ ⊆ σp, we will use [[Γ ]]V to denote Qp∈Γ [[p]]V for
notational convenience.
4.3</p>
        </sec>
        <sec id="sec-4-1-2">
          <title>Logical Consequence</title>
          <p>Having defined the syntax and semantics ofLogAM. What remains for us is to define
logical consequence. Since we are taking the algebraic route, we employ our notion of
multifilters from Section 3 to define a consequence relation for each mental attitude in
a LogAM theory.</p>
          <p>In Section 3, we defined multifilters based on an arbitrary partial order . We start
by defining how to construct such an order for the tuples of propositions in P. The
intuition is that the order is induced by the bridge rules in a LogAM theory in addition
to the natural order ≤ among the attitudes that observe the second condition of the
definition of multifilters (closure under the meet/conjunction operation).
Definition 10. Let T = hA, R, Si be a LogAM theory and V a valuation. A TV
induced order, denoted TV , is a partial order over Pk with the following properties.
1. If i ∈ S and a ≤ b, then ({&gt;}i−1 ×{a}×{&gt;}k−i)
2. If (A1, . . . , Ak 7−→ A01, . . . , A0k) ∈ R, then
([[A1]]V , . . . , [[Ak]]V ) TV ([[A01]]V , . . . , [[A0k]]V ).</p>
          <p>TV ({&gt;}i−1 ×{b}×{&gt;}k−i).</p>
          <p>At this point we observe that if the bridge rules in a LogAM theory T observe some
restrictions, then the order induced by the theory is classical (recall what it means for
an order to be classical according to Definition 3).</p>
          <p>Observation 1. Let T = h(A1, . . . , Ak), R, Si is a LogAM theory, V a valuation, and</p>
          <p>TV be a k partial-order on A. For every (A1, ..., Ak 7−→ A01, ..., A0k) ∈ R, and for
every i, j such that 1 ≤ i, j ≤ k and j 6= i, TV is classical in i if and only if A0i 6= {}
just in case Aj = A0j = {} and [[Ai]]V ≤ [[A0i]]V .
We next utilise a multifilter based on a TV -induced order to define an extended
logical consequence relation for each mental attitude.</p>
          <p>Definition 11. Let T = h(A1, ..., Ak), R, Si a LogAM theory and TV be a TV
induced order. For every φ ∈ σP , φ is an Ai consequence of T for 1 ≤ i ≤ k, denoted
T |=Ai φ, if, for every valuation V, [[φ]]V ∈ Fi where
hF1, ..., Fki = F TV (h[[A1]]V , ..., [[Ak]]V i, S).</p>
          <p>We now inspect the properties of our extended consequence relations. The following
theorem states that each |=Ai is monotonic and has the distinctive properties of classical
Tarskian logical consequence. Further, if i ∈ S, then |=Ai observes a variant of the
deduction theorem.</p>
          <p>Theorem 2. Let T = h(A1, ..., Ak), R, Si and T0 = h(A01, ..., A0k), R0, S0i be LogAM
theories with S = S0
1. If φ ∈ Ai for some Ai ∈ A, then T |=Ai φ.
2. If T |=Ai φ, Aj ⊆ A0j for all 1 ≤ j ≤ k, and R0 ⊆ R, then T0 |=Ai φ.
3. Let A0i = Ai ∪ {ψ} for some i such that 1 ≤ i ≤ k, A0j = Aj for all j 6= i, and</p>
          <p>R0 = R. If T |=Ai ψ and T0 |=Ai φ, then T |=Ai φ.
4. Let A0i = Ai ∪{φ} for some i such that 1 ≤ i ≤ k. If i ∈ S, R0 = R, and T0 |=Ai ψ,
then T |=Ai φ ⇒ ψ.</p>
          <p>In the remainder of this section, we go back to our running example showing the
consequences of the LogAM theory of Example 2. In what follows, let A0i = {φ | T |=Ai
φ} for 1 ≤ i ≤ k.</p>
          <p>Example 3. Recall the LogAM theory T in Example 2. In the following, we
demonstrate the effect of the application of the bridge rules.
1. Initially, the applicable bridge rules are r1, r4, r5, and r6. This causes I(t) to be
an obligation consequence of T, f a fear consequence of T, r an obligation
consequence of T, and ¬r a fear consequence of T.
2. Once I(t) becomes an obligation consequence and f and ¬r become fear
consequences, r2 and r7 become applicable. This causes t to be an intention consequence
of T, and ¬r and f to be belief consequences of T. Adding ¬r to the belief
consequences adds m to the belief consequences as well due to the belief ¬r ⇒ m.
3. Finally r3 becomes applicable. This causes c to become an intention consequence
of T.</p>
          <p>The following are the final consequences ofT.
– Promises: A01 = {t}.
– Beliefs: A02 = {¬r ⇒ m, m ⇒ f, l, i ⇒ r ∧ t, ¬r, f, m}
– Intentions: A03 = {t, c}
– Fears: A04 = {f, ¬r}
– Regrets: A05 = {¬r}
– Doubts: A06 = {¬i}
– Obligations: A07 = {I(t), r}</p>
        </sec>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>5 Incorporating Non-Monotonicity in LogAM</title>
      <p>
        According to Theorem 2, the consequence relations for the different mental attitudes of
LogAM have a monotonic nature. This means that newly acquired propositions in the
different mental attitudes can never invalidate previous propositions. Moreover, the
consistency of the mental attitudes of the agent is not guaranteed. These are inconvenient
assumptions for some mental attitudes. For example, a natural consequence of typical
incomplete knowledge about the world is that newly acquired beliefs can invalidate
previous beliefs. Consequently, some intentions might be dropped as their supporting
beliefs are not believed anymore. Furthermore, it would also make sense that the beliefs
and intentions (for instance) of the agents are always collectively consistent if we are to
model a rational agent. For these reasons, in this section we informally describe a
nonmonotonic extension of LogAM where the consistency of selected mental attitudes is
preserved. The extension we are proposing is a generalization of a framework we
developed in [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ] for non-monotonic practical reasoning with beliefs and motivations.
      </p>
      <p>Towards incorporating non-monotonicity in LogAM, the first thing we do is that we
associate grades with the different mental attitudes. The grades are reified objects with
some total order on them and are taken to represent measures of trust or preference.
Moreover, we define the agent’s character as a total order over the mental attitudes
that we wish to maintain their collective consistency. Whenever inconsistencies arise,
the agent character and the grades of the contradictory propositions are utilised to
resolve them. The agent character orders the attitudes from the least preffered to the most
preferred. The agent always prefers to give up propositions from the least preferred
attitude. Similarly, the least preferred proposition is the proposition with the least grade.
We also enforce that the consequents of the bridge rules are only graded propositions
to make sure that any newly added proposition has a grade that can be inspected if this
newly added proposition causes a contradiction. To illustrate this, we go back to the
LogAM theory in Example 2. We first show the modified theory after we add grades to
the different attitudes and modify the consequents of the bridge rules.</p>
      <p>Example 4. In what follows, we use P for A1, B for A2, I for A3, F for A4, R for A5,
D for A6, and O for A7 for readability. A possible graded LogAM theory representing
Example 1 is
T = h(A1, A2, A3, A4, A5, A6), R, Si where:
– A1 represents Ted’s promises. A1 = {P(t, 3)}.
– A2 represents Ted’s beliefs. A2 = {B(¬r ⇒ m, 1), B(m ⇒ f, 5), B(l, 10),</p>
      <p>B(i ⇒ r ∧ t, 4)}.
– A3 represents Ted’s intentions. A3 = {}.
– A4 represents Ted’s fears. A4 = {}.
– A5 represents Ted’s regrets. A5 = {R(¬r, 6)}.
– A6 represents Ted’s doubts. A6 = {D(¬i, 3)}.
– A7 represents Ted’s obligations. A7 = {}.
– R is the set of instances of the following rule schema where φ and g are variables.</p>
      <p>These are a modified version of the rules in Example 2 to add grades to the
consequences of the bridge rules.</p>
      <p>r1. P = {P(φ, g)} 7−→ O = {I(φ, g)}
r2. O = {I(φ, g)} 7−→ I = {I(φ, g)}
r3. I = {I(t, g)} 7−→ I = {I(c, g)}
r4. B = {B(l, g)} 7−→ F = {F(f, g)}
r5. R = {R(¬r, g)} 7−→ O = {O(r, g)}
r6. D = {D(¬i, g)} 7−→ F = {F(¬r, g)}
r7. F = {F(φ, g)} 7−→ B = {B(φ, g)}
– S = {2}.</p>
      <p>Now consider the above graded LogAM theory and suppose that we only care that
Ted’s collective beliefs, promises, obligations, and intentions are consistent. Given that,
for instance, Ted believes B(l, 10) and does not believe ¬l, it would make sense for
him to accept l despite his uncertainty about it. Similarly, it would make sense for Ted
to add t to his promises if they do not conflict with other beliefs, promises, obligations,
or intentions. However, if we only use multifilters, we will never be able to reason with
those nested graded attitudes as they are not themselves in the agent’s theory but only
grading propositions thereof. For this reason, we extend our notion of multifilters into
a more liberal notion of graded multifilters to enable the agent to conclude, in addition
to the consequences of the initial theory, attitudes graded by the initial attitudes (like
l and t). Should this lead to contradictions, the agent’s character and the grades of the
contradictory propositions are used to resolve them. In what follows, we show how
graded multifilters are used to get the set of consequences for each mental attitude.
Example 5. We first apply the bridge rules toT just like we did in Example 3. We get
the following updated mental state.</p>
      <p>– Promises: A01 = A1.
– Beliefs: A02 = A2 ∪ {B(f, 10), B(¬r, 3)}.
– Intentions: A03 = {I(t, 3), I(c, 3)}.
– Fears: A04 = {F(f, 10), F(¬r, 3)}.
– Regrets: A05 = A5
– Doubts: A06 = A6
– Obligations: A07 = {I(t, 3), O(r, 6)}.</p>
      <p>Next, we admit the graded attitudes in the initial theory. The following becomes the
updated mental state of the agent.</p>
      <p>– Promises: A010 = A01 ∪ {t}.
– Beliefs: A020 = A02 ∪ {¬r ⇒ m, m ⇒ f, l, i ⇒ r ∧ t, f, ¬r, m}.
– Intentions: A030 = {t, c}.
– Fears: A040 = {f, ¬r}.
– Regrets: A050 = A05 ∪ {¬r}.
– Doubts: A060 = A06 ∪ {¬i}.</p>
      <p>r .
– Obligations: A070 = { }</p>
      <p>Note that we add m to the agent’s beliefs not because it was extracted out of a
graded belief, but because it follows from ¬r ⇒ m and ¬r that was just extracted out
of B(¬r, 3). At this point, Ted’s beliefs and obligations are contradictory as his beliefs
include ¬r and his obligations include r. This is where the agent’s character come into
play. If Ted’s character prefers to give up his beliefs, then ¬r will be retracted from his
beliefs. Otherwise, r will be given up as an obligation.</p>
      <p>Now suppose that Ted acquires the new belief that he will not be firedB(¬f, 15).
We extract ¬f and add it to Ted’s beliefs. Once we do this, Ted’s set of beliefs becomes
contradictory as it contains ¬f and f . Since the contradiction is now within the same
attitude, we allude to the grades of the contradictory propositions. Since ¬f has the
grade of 15 and f has the grade of 10, then f will be kicked out of Ted’s beliefs resolving
the contradiction.</p>
      <p>In general, to resolve inconsistencies among the attitudes we select to be
consistent, we always remove the propositions with the lowest grade in the least preferred
attitude according to the agent character. Next, any propositions supported only by the
removed propositions are removed as well. If two contradictory propositions have the
same grade, they both go away.
6</p>
    </sec>
    <sec id="sec-6">
      <title>Conclusion</title>
      <p>In this paper, we presented a general algebraic framework for reasoning with mental
states. We also provided semantics for an algebraic logic, LogAM, where any set of
mental attitudes can be represented. We defined a monotonic consequence relation for
each mental attitude and showed that the consequence relations observe the distinctive
properties of Tarskian logical consequence. Moreover, we informally described how
LogAM can be extended to handle non-monotonic reasoning with graded mental
attitudes. We are currently working on a proof theory for the non-monotonic version of
LogAM. Reasons for the different mental attitudes are to be computed in the same way
reason-maintenance systems compute supports for beliefs. Hence, the end result will
be a versatile framework for reasoning with graded mental attitudes giving rise to an
explainable AI system.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <given-names>James</given-names>
            <surname>Allen</surname>
          </string-name>
          .
          <article-title>Towards a general theory of action and time</article-title>
          .
          <source>Artificial Intelligence</source>
          ,
          <volume>23</volume>
          :
          <fpage>123</fpage>
          -
          <lpage>154</lpage>
          ,
          <year>1984</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <given-names>George</given-names>
            <surname>Bealer</surname>
          </string-name>
          .
          <article-title>Theories of properties, relations, and propositions</article-title>
          .
          <source>The Journal of Philosophy</source>
          ,
          <volume>76</volume>
          (
          <issue>11</issue>
          ):
          <fpage>634</fpage>
          -
          <lpage>648</lpage>
          ,
          <year>1979</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Ronen</surname>
            <given-names>I. Brafman</given-names>
          </string-name>
          and
          <string-name>
            <given-names>Moshe</given-names>
            <surname>Tennenholtz</surname>
          </string-name>
          .
          <article-title>Modeling agents as qualitative decision makers</article-title>
          .
          <source>Artificial Intelligence</source>
          ,
          <volume>94</volume>
          (
          <issue>1-2</issue>
          ):
          <fpage>217</fpage>
          -
          <lpage>268</lpage>
          ,
          <year>1997</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <given-names>Jan</given-names>
            <surname>Broersen</surname>
          </string-name>
          , Mehdi Dastani, Joris Hulstijn, Zisheng Huang, and
          <article-title>Leendert van der Torre. The BOID architecture: conflicts between beliefs, obligations, intentions and desires</article-title>
          .
          <source>In Proceedings of the fifth international conference on Autonomous agents</source>
          , pages
          <fpage>9</fpage>
          -
          <lpage>16</lpage>
          ,
          <year>2001</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <given-names>Alonzo</given-names>
            <surname>Church</surname>
          </string-name>
          .
          <article-title>On carnap's analysis of statements of assertion and belief</article-title>
          .
          <source>Analysis</source>
          ,
          <volume>10</volume>
          (
          <issue>5</issue>
          ):
          <fpage>97</fpage>
          -
          <lpage>99</lpage>
          ,
          <year>1950</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <given-names>Nourhan</given-names>
            <surname>Ehab</surname>
          </string-name>
          and
          <string-name>
            <given-names>Haythem O.</given-names>
            <surname>Ismail</surname>
          </string-name>
          .
          <article-title>Algebraic foundations for non-monotonic practical reasoning</article-title>
          . In Ivan Jose´ Varzinczak and Mar´ıa Vanina Mart ´ınez, editors,
          <source>Proceedings of the 18th International Workshop on Non-Monotonic Reasoning (NMR2020)</source>
          ,
          <year>2020</year>
          . To appear.
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <given-names>Nourhan</given-names>
            <surname>Ehab</surname>
          </string-name>
          and
          <string-name>
            <surname>Haythem O. Ismail.</surname>
          </string-name>
          <article-title>LogAG: An algebraic non-monotonic logic for reasoning with graded propositions</article-title>
          .
          <source>Annals of Mathematics and Artificial Intelligence</source>
          ,
          <year>2020</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <given-names>Nourhan</given-names>
            <surname>Ehab</surname>
          </string-name>
          and
          <string-name>
            <given-names>Haythem O.</given-names>
            <surname>Ismail</surname>
          </string-name>
          .
          <article-title>Reasoning with artificial mental states: An algebraic approach</article-title>
          .
          <source>Technical report</source>
          , German University in Cairo,
          <year>2020</year>
          . https://met.guc.edu.eg/ Repository/Faculty/Publications/950/FCR2020-Appendix.pdf.
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Kenneth</surname>
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Ford</surname>
            ,
            <given-names>Patrick J.</given-names>
          </string-name>
          <string-name>
            <surname>Hayes</surname>
            , Clark Glymour, and
            <given-names>James</given-names>
          </string-name>
          <string-name>
            <surname>Allen</surname>
          </string-name>
          .
          <article-title>Cognitive orthoses: toward human-centered AI</article-title>
          .
          <source>AI Magazine</source>
          ,
          <volume>36</volume>
          (
          <issue>4</issue>
          ):
          <fpage>5</fpage>
          -
          <lpage>8</lpage>
          ,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Harry</surname>
            <given-names>G.</given-names>
          </string-name>
          <string-name>
            <surname>Frankfurt</surname>
          </string-name>
          .
          <article-title>Freedom of the will and the concept of a person</article-title>
          .
          <source>In What is a person?</source>
          , pages
          <fpage>127</fpage>
          -
          <lpage>144</lpage>
          . Springer,
          <year>1988</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Haythem</surname>
            <given-names>O. Ismail.</given-names>
          </string-name>
          <article-title>LogAB: A first-order, non-paradoxical, algebraic logic of belief</article-title>
          .
          <source>Logic Journal of the IGPL</source>
          ,
          <volume>20</volume>
          (
          <issue>5</issue>
          ):
          <fpage>774</fpage>
          -
          <lpage>795</lpage>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Haythem</surname>
            <given-names>O. Ismail.</given-names>
          </string-name>
          <article-title>Stability in a commonsense ontology of states</article-title>
          .
          <source>Proceedings of the Eleventh International Symposium on Logical Formalization of Commonsense sense Reasoning (COMMONSENSE</source>
          <year>2013</year>
          ),
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Haythem</surname>
            <given-names>O. Ismail.</given-names>
          </string-name>
          <article-title>The good, the bad, and the rational: Aspects of character in logical agents</article-title>
          .
          <source>In Alia ElBolock</source>
          , Yomna Abdelrahman, and Slim Abdennadher, editors,
          <source>Character Computing</source>
          . Springer,
          <year>2020</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Hector</surname>
            <given-names>J.</given-names>
          </string-name>
          <string-name>
            <surname>Levesque</surname>
          </string-name>
          .
          <article-title>Making believers out of computers</article-title>
          .
          <source>Artificial Intelligence</source>
          ,
          <volume>30</volume>
          (
          <issue>1</issue>
          ):
          <fpage>81</fpage>
          -
          <lpage>108</lpage>
          ,
          <year>1986</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>John McCarthy</surname>
          </string-name>
          .
          <article-title>Ascribing mental qualities to machines</article-title>
          .
          <source>Philosophical Perspectives in Artificial Intelligence</source>
          , Humanities Press , pages
          <fpage>161</fpage>
          -
          <lpage>195</lpage>
          ,
          <year>1979</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>John McCarthy</surname>
          </string-name>
          .
          <article-title>The little thoughts of thinking machines</article-title>
          .
          <source>Psychology Today</source>
          ,
          <volume>17</volume>
          (
          <issue>12</issue>
          ):
          <fpage>46</fpage>
          -
          <lpage>49</lpage>
          ,
          <year>1983</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>John McCarthy</surname>
            and
            <given-names>Patrick</given-names>
          </string-name>
          <string-name>
            <surname>Hayes</surname>
          </string-name>
          .
          <article-title>Some philosophical problems from the standpoint of artificial intelligence</article-title>
          . In D. Meltzer and D. Michie, editors,
          <source>Machine Intelligence</source>
          , volume
          <volume>4</volume>
          , pages
          <fpage>463</fpage>
          -
          <lpage>502</lpage>
          . Edinburgh University Press, Edinburgh, Scotland,
          <year>1969</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <given-names>Paul</given-names>
            <surname>McNamara</surname>
          </string-name>
          .
          <article-title>Deontic logic</article-title>
          . In Edward N. Zalta, editor,
          <source>The Stanford Encyclopedia of Philosophy</source>
          . Metaphysics Research Lab, Stanford University, fall
          <year>2018</year>
          edition,
          <year>2018</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>Allen</surname>
          </string-name>
          Newell et al.
          <article-title>The knowledge level</article-title>
          .
          <source>Artificial intelligence</source>
          ,
          <volume>18</volume>
          (
          <issue>1</issue>
          ):
          <fpage>87</fpage>
          -
          <lpage>127</lpage>
          ,
          <year>1982</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>Pietro</surname>
            <given-names>Panzarasa</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Nicholas R. Jennings</surname>
            , and
            <given-names>Timothy J.</given-names>
          </string-name>
          <string-name>
            <surname>Norman</surname>
          </string-name>
          .
          <article-title>Formalizing collaborative decision-making and practical reasoning in multi-agent systems</article-title>
          .
          <source>Journal of logic and computation</source>
          ,
          <volume>12</volume>
          (
          <issue>1</issue>
          ):
          <fpage>55</fpage>
          -
          <lpage>117</lpage>
          ,
          <year>2002</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          21.
          <string-name>
            <given-names>Terence</given-names>
            <surname>Parsons</surname>
          </string-name>
          .
          <article-title>On denoting propositions and facts</article-title>
          .
          <source>Philosophical Perspectives</source>
          ,
          <volume>7</volume>
          :
          <fpage>441</fpage>
          -
          <lpage>460</lpage>
          ,
          <year>1993</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          22.
          <string-name>
            <given-names>Donald</given-names>
            <surname>Perlis</surname>
          </string-name>
          .
          <article-title>Languages with self-reference II: Knowledge, belief, and modality</article-title>
          .
          <source>Artificial Intelligence</source>
          ,
          <volume>34</volume>
          (
          <issue>2</issue>
          ):
          <fpage>179</fpage>
          -
          <lpage>212</lpage>
          ,
          <year>1988</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          23.
          <string-name>
            <surname>Anand</surname>
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Rao</surname>
          </string-name>
          and Michael P. Georgeff. BDI agents:
          <article-title>From theory to practice</article-title>
          .
          <source>In ICMAS</source>
          , volume
          <volume>95</volume>
          , pages
          <fpage>312</fpage>
          -
          <lpage>319</lpage>
          ,
          <year>1995</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          24.
          <string-name>
            <given-names>H.P.</given-names>
            <surname>Sankappanavar</surname>
          </string-name>
          and
          <string-name>
            <given-names>Stanley</given-names>
            <surname>Burris</surname>
          </string-name>
          .
          <article-title>A course in universal algebra</article-title>
          . Graduate Texts Math,
          <volume>78</volume>
          ,
          <year>1981</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          25.
          <string-name>
            <surname>Stuart</surname>
            <given-names>C.</given-names>
          </string-name>
          <string-name>
            <surname>Shapiro</surname>
          </string-name>
          .
          <article-title>Belief spaces as sets of propositions</article-title>
          .
          <source>Journal of Experimental &amp; Theoretical Artificial Intelligence</source>
          ,
          <volume>5</volume>
          (
          <issue>2</issue>
          -3):
          <fpage>225</fpage>
          -
          <lpage>235</lpage>
          ,
          <year>1993</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          26.
          <string-name>
            <given-names>Yoav</given-names>
            <surname>Shoham</surname>
          </string-name>
          .
          <article-title>An overview of agent-oriented programming</article-title>
          .
          <source>Software agents</source>
          ,
          <volume>4</volume>
          :
          <fpage>271</fpage>
          -
          <lpage>290</lpage>
          ,
          <year>1997</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          27.
          <string-name>
            <given-names>Yoav</given-names>
            <surname>Shoham</surname>
          </string-name>
          and
          <string-name>
            <given-names>Steve B.</given-names>
            <surname>Cousins</surname>
          </string-name>
          .
          <article-title>Logics of mental attitudes in AI</article-title>
          .
          <source>In Foundations of Knowledge Representation and Reasoning</source>
          , pages
          <fpage>296</fpage>
          -
          <lpage>309</lpage>
          . Springer,
          <year>1994</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          28. Hans van Ditmarsch,
          <string-name>
            <surname>Joseph Halpern</surname>
            , and
            <given-names>Barteld</given-names>
          </string-name>
          <string-name>
            <surname>Kooi</surname>
          </string-name>
          .
          <article-title>An introduction to logics of knowledge and belief</article-title>
          . In Hans van Ditmarsch,
          <string-name>
            <surname>Joseph Halpern</surname>
          </string-name>
          , Wiebe van der Hoek, and Barteld Kooi, editors,
          <source>Handbook of Epistemic Logic</source>
          , pages
          <fpage>1</fpage>
          -
          <lpage>51</lpage>
          . College Publications,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>