<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Modelling Suspicion as a Game Mechanism for Designing a Computer-Played Investigation Character</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Nahum Alvarez</string-name>
          <email>nahum.alvarezayerza@gameloft.com</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Federico Peinado</string-name>
          <email>email@federicopeinado.com</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Departamento de Ingeniería del Software e Inteligencia Artificial, Facultad de Informática, Universidad Complutense de Madrid 28040</institution>
          <addr-line>Madrid</addr-line>
          ,
          <country country="ES">Spain</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Production Department</institution>
          ,
          <addr-line>Gameloft Tokyo 151-0061 Tokyo</addr-line>
          ,
          <country country="JP">Japan</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>Nowadays creating believable characters is a top trend in the video game industry. Recent projects present new design concepts and improvements in artificial intelligence that are oriented to this goal. The expected behaviour of computer-played characters becomes increasingly demanding: proactivity, autonomous decision-making, social interaction, natural communication, reasoning and knowledge management, etc. Our research project explores the possibility for one of these characters to investigate, solving an enigma while dealing with incomplete information and the lies of other characters. In this paper we propose how to manage trust issues when modelling suspicion in the artificial mind of a Columbo-like detective, considering that our desired gameplay is a murder mystery game in which the player plays the role of the culprit.</p>
      </abstract>
      <kwd-group>
        <kwd />
        <kwd>Interactive Digital Storytelling</kwd>
        <kwd>Video Game Design</kwd>
        <kwd>Artificial Intelligence</kwd>
        <kwd>Believable Characters</kwd>
        <kwd>Epistemic Modal Logic</kwd>
        <kwd>Trust Systems</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1 Introduction</title>
      <p>Computer simulations allow a high grade of realism; however, computer-controlled
characters in virtual worlds still feel like mere automatons: the user gives an input and
receives a response. Advances in intelligent behaviour are changing the user
experience, improving these characters’ performance and expanding the scope of their
potential uses. One aspect that refrain us from believing such characters are “alive” is
that they seem completely “naive”: they will believe without doubts all the
information received from other sources. Of course, in interactive storytelling this issue is
mainly confronted with a script that the character will follow: if it has to suspect, it
will suspect, if it has to lie, it will lie, etc. but there is no real intelligence behind those
decisions. A system capable of managing suspicion would improve greatly the degree
of realism of the characters, but such feature has not been sufficiently explored.</p>
      <p>In this work we focus in the question of trust, intrinsic to human nature, and
introduce a model for a computer-played character that suspect about others and try to
detect when they are covering their actions or telling lies. In this model, the
autonomous “investigator” works with incomplete information, managing uncertain
knowledge and potentially false pieces of evidence. We also propose a videogame
application designed to test our model. The application consists of an interactive
version of the popular TV series from the NBC: Columbo, where the player controls the
murderer instead of the main character of the story, Lieutenant Columbo, who is the
homicide detective. The culprit is obvious to the player since the very beginning of the
game, and the goal of this character is to deceive the detective for not getting caught
(as in the original episodes, the format of the typical “whodunit” mystery is reversed
to the “howcatchthem” paradigm or “inverted detective story”).</p>
      <p>In order to frame our proposal, in Section 2 we review other models of deception
and trust. Section 3 presents our model of suspicion for computer-controlled
characters and Section 4 describes an example scenario of how our model would work using
a murder mystery game as an application. Finally, in Section 5 we present and discuss
our conclusions, foreseen the next steps of this project.</p>
    </sec>
    <sec id="sec-2">
      <title>2 Related Work</title>
      <p>
        Modelling deception has been object of research from long ago. Jameson presented
IMP in 1983 [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ], a system that simulates a real estate agent who tries to deceive the
user. Nissan and Rousseau described how to model the mental state of agents in a
more playable scenario: a crime investigation interactive fiction [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ]. In later works,
Nissan showed a model for representing complex stories involving deceptions and
suspicion [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ]. This proposal is a theoretical model, but it established a basis to
construct working systems using trust and partial knowledge. Such formal models have
been presented since then, showing an integrated frame for uncertainty and deception
in game theory [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ].
      </p>
      <p>
        Sometimes the character may work not only with incomplete information, but also
with information that is considered as “uncertain” (depending on the sincerity of the
characters that reveal it). In order to manage these features, intelligent characters have
to decide which information to trust. Research on trust and deception itself is a broad
concept with multiple aspects that have been explored in the literature, such as
selfdeception and how it affects decision-making [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ], deceptive strategies in auction
style interactions [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] or deceptive information in decision-making processes for games
[
        <xref ref-type="bibr" rid="ref14">14</xref>
        ]. However, these theoretic approaches were mostly oriented to a very limited type
of interactions.
      </p>
      <p>
        Certainly, trust is a crucial aspect to deal when requesting services or information
from third parties [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ]. It has been extensively treated in literature but still it does not
have a standard formal definition [
        <xref ref-type="bibr" rid="ref22">22</xref>
        ]. In a general classification proposed in [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ],
four general types of trust are defined: basic trust, a priori generalized trust, inner
dialogicality and context-specific trust. The last type is defined for applications that
require trust management, which is the case in a murder mystery game; for instance,
we trust a doctor about health issues when he is treating us, but not about the death of
his wife when he is the main suspect in her murder. This type of trust contains
contextspecific rules, so we can decide if trusting or not using different methods. For
example, Griffin and Moore [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ] present a model that assigns different truth values to the
concepts managed by autonomous agents, allowing them to choose if they trust their
sources or not. These agents simulate how the situation would evolve if they take
certain decisions, trying to maximize their outcome in a minimax fashion. The system
of Wang et al. [
        <xref ref-type="bibr" rid="ref24">24</xref>
        ] uses fuzzy values, implying the information of a source is not
definitely true or false, but it has some grade of truth, and we can also find systems
using a vector of parameters for different aspects of trust [
        <xref ref-type="bibr" rid="ref25">25</xref>
        ]. Another model of trust
and commitment [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ] proposes that positive and negatives experiences impact in the
trustee, treating all of them equally (although probably in real life negative ones have
a greater impact).
      </p>
      <p>
        Trusting a source of information does not only rely in judging over the data
received from that source, but we also have to take in account all the related sources.
Moreover, we also should take in account third parties’ information: for example, in
[
        <xref ref-type="bibr" rid="ref3">3</xref>
        ] trust is built without direct interaction between the parts using a set of relationships
and trust parameters in their multi-agent model, allowing to build trust on the
information of a third party agent. This transitivity is well defined in the theory of Human
Plausible Reasoning [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ], a frame designed to simulate the way a human reason about
truth values. This frame establishes an ontology-based model that uses a fuzzy
parameter for measuring the degree of trust of a statement or a source. This has been used as
a basis for developing complex trust systems, showing good results.
      </p>
      <p>
        ScubAA [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ] is another example consisting of a generic framework for managing
trust in multi-agent systems. A central trust manager is in charge of sending the user’s
request to the most trusted agent for the needed task. The trust degree of each agent is
calculated using not only the information it has but also the information of the agents
related to them, so an agent can evaluate agents directly unknown for it. This
transitivity is useful for building a trust network where information of third parties is
considered, affecting the trust of the agents, even if they are not present or currently known.
We also use this approach in our proposal, but in our system, instead of designing a
centralized system, each agent can manage its own trust network. Also, it is advisable
to take into account the trust history of the other parties [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ], in order to trust in
sources that were “trustful” in the past for us or for our partners [
        <xref ref-type="bibr" rid="ref26">26</xref>
        ]. Anyway, agents
have to take this information carefully, because they only receive partial information
from others. This also presents an interesting feature and a powerful possibility: if a
malicious agent manages to tarnish your trust level for other parties, they will not trust
you in future interactions.
      </p>
      <p>In our model, allowing the character to build its own trust values, makes the
information they hold to be incomplete and different for each character. This feature is
beneficial from our point of view, because it allows introducing deceptive information
in the system without being “caught”, testing the investigator’s ability to find the truth.</p>
      <p>
        Finally, in order to generate trust in other parties or deceive them with our lies, we
have to relay in argumentation. In literature, different analysis about epistemology
and argumentation has been examined and described thoroughly, establishing
directions that allow modelling critical reasoning [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]. In short, we can trust a source only
after building argumentation backing it [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ]. Especially interesting for our goal is how
to build critical questions in order to attack other’s argumentation, a process that will
help us to discover the “holes” in an agent’s argumentation, finding not only lies but
also important information previously unknown [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ].
      </p>
    </sec>
    <sec id="sec-3">
      <title>3 A Model of Suspicion</title>
      <p>Our knowledge model is designed to allow computer-controlled characters to decide if
they should believe (trust in) the information received from a third party. The
decision-making or judgement about a concrete piece of information will be based on the
previous data they have about it, and about the source from they received that
information. This paper is focused on introducing the model, so we have deliberately
postponed the analysis of the more technical aspects (i.e. the reasoning engine) for further
research. In order to test our model, the characters will operate in a detective scenario,
a great domain example where trust and deception are key features to have into
account. In this scenario, the user plays the role of a murderer who has to mislead an
important non-player character: the detective. The player’s goal will be to prevent the
detective to discover the murderer’s identity (his/her own identity). The scenario
contains also other suspicious characters, each one with its own information about the
incident. The detective will obtain new information (from now on, Facts) about the
scenario, talking with other characters and finding evidences, storing all this
information in a knowledge base.</p>
      <p>In order to decide what information should trust and how, the first challenge the
detective confronts is determining the truth value of Facts. As we saw in the previous
section, other models are based on lists of trust parameters, fuzzy truth values, or
statistical probabilities. We decided to use a discrete system using a ternary value
(“true”, “false” and “unknown”) for each Fact. The first time the detective gets
informed about a Fact, he will create an entry in his knowledge base about it, and he will
update its value depending of the additional information he finds about it. Not
knowing certain information is not the same than knowing it is true or false (i.e. open world
assumption), so we use the value “unknown” in order to mark those doubtful Facts and
subsequently trying to discover their real value with abductive reasoning .</p>
      <p>The next question is how to “quantify” the truth value of a Fact. We only have
found simple methods in the literature for this decision-making process: usually it is
enough to use a fixed probability number or comparing the number of agents
supporting a fact with the number of agents denying it. However, if the system is that simple
we may lose certain desirable characteristics as taking into account evidences that
support the truth or falsehood about a fact, or remembering if a character previously
lied to us before evaluating her statements, especially if those statements are related
with the one about she lied.</p>
      <p>
        Considering these requirements, our detective will have a list of truth values about
a Fact where they will store each evidence they obtain about it. Having a bigger
number of supporting evidence for a Fact (“true” values) will make him to believe the
statement, and on the other hand, having more contradictory evidence (“false” values)
will lead the agent to consider the statement as a lie. This is similar to the system used
in [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ] where the positive evidences are cancelled by negative ones. Whenever we
confirm or discard a fact from a source, we will record whether he lied to us or not. If
we receive information from one source, the truth value of that information would be
set initially to “true”, assuming the source did not lie before (so we trust in him). We
would set the information as “false” if the source lied about a related topic, or
“unknown” if we don’t have previous information about the source, or if the source lied
but about another topic.
      </p>
      <p>Once we have a model describing how the information will be stored and judged,
we need a reasoning method in order to figure out the truth value of unknown Facts.
Although it is not the main focus of this paper to present a reasoning implementation,
it is necessary to describe such method in order to show how to deal with its
characteristic features: explicitly unknown values and non-trustable sources. In our model, the
automatic reasoner has the goal to clear “unknown” values by giving them “false” or
“true” values by identifying if assigning those values generates a contradiction or a
possible outcome. We propose using two joint techniques that resemble a detective
work to achieve our goal.</p>
      <p>The first one would be using an “established knowledge” database. Facts we will
receive are checked using a common sense’s rules database: if some Fact contradicts a
rule in that database we will know that the source is lying; for example, stating that a
character was outside of a building while it was raining, but seeing that his clothes are
dry. Also, we can do this Fact checking with our first person evidences: if we know a
Fact for sure, any information that contradicts it would be false.</p>
      <p>The second technique consists in trying to clear the “unknown” actively. If we want
to know who the murderer is, but we have limited information including false
evidence from the culprit, it’s very likely that once we obtain enough information from
other characters we will have some “relative contradictions”, represented by having
“true” and “false” values in the list of truth values obtained from different sources
about the same fact. Upon finding a contradiction, the character will run two
simulations, respectively binding the truth value for that fact to “true” and “false”, and
propagating further consequences. If the resulting knowledge base of the simulation has a
contradiction with another Fact we know for sure is true (values in our “established
knowledge” database), we can discard that value.</p>
      <p>
        We are currently analysing reasoning tools in order to apply the most suitable one
to work with our suspicion model. In order to model sophisticated scenarios, the
reasoner should have planning capabilities, and be able to work with hypothetic or
potential facts. For example, since our knowledge structure works with undetermined truth
values, a model based in Intuitionistic Logic would fit well. Using this technique, a
suitable framework would be the Hypothetical Logic of Proof [
        <xref ref-type="bibr" rid="ref21">21</xref>
        ]. This work is
derived from Natural Deduction [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ], which is designed to be similar to intuitive,
informal reasoning, like the one we can see in detective’s stories. Next steps in this
research will explore the different techniques that work well with our model, presenting
a working reasoning prototype. This prototype will consist on as a detective game like
the one we have mentioned, which is detailed in the next section.
In order to test our model we propose a game around the concept of trust and
deception. Previous researchers have also used games as well for simulating trust relations,
like [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ] where an investment game (a modification of Berg’s game [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]) is used for
analysing how users react when trust and deception is introduced to the game
(showing that users trust more in a system with advisor, even if he can deceive them).
      </p>
      <p>Our scenario will be an inverted detective story based in the Columbo TV episodes,
where the player has the role of the culprit in a murder case, and his goal is to deceive
a computer-controlled detective (a sort of virtual Columbo) and lead him to arrest
another suspect instead of the player.</p>
      <p>The simulation will play as a series of rounds in which the detective will question
the player and the other suspects. Obviously, the player knows the details of the
murder and he will have the opportunity to hear what other characters say in order to gain
additional information from them to be used for deceiving the detective. The player
can, for example, create an alibi for himself (“I was with that person at the moment of
the crime”), or blame a third party (“I saw that person entering in the victim room just
before the crime”). If the player manages to plant enough false evidence, the detective
will arrest the wrong character and the player will win the game. However, if the
detective find contradictions in the player’s alibi or manages to extract the true testimony
from the other suspects, he will soon discover the player as the real culprit, making
him lose the game.</p>
      <p>In order to illustrate these ideas, the next figures show how the knowledge of the
detective evolves during a game session. The example is inspired in the first episode
of the original Columbo series, “Murder by the Book” in which Jim Ferris, a famous
writer, is killed by his colleague, Ken Franklin. As part of his alibi, Ken tries to
incriminate the Mafia, dropping a false document in the crime scene and lying about the
next book Jim was working on. The player plays the role of Ken, explaining his alibi
to the detective, who will try to find something that contradicts Ken’s suggestions. The
next paragraphs explain just a small part of the complete plot of the episode, enough
to illustrate the basic ideas behind our model.</p>
      <p>Firstly, the crime is introduced (Fig. 1) with a clear fact (represented by a white
box), and some basic deductions (black arrows) performed by the automatic reasoner.
This puts the detective in his way to find the murderer, that could be someone close to
the victim, or not (red arrows with a white head represents disjoint Facts).</p>
      <p>Fig. 1. The crime is introduced and basic deductions are performed
Then, the interrogation starts (Fig. 2) and the detective asks Joanna, Jim’s wife, about
the enemies of his husband. If Joanna tells the truth, which initially is “believable”
(discontinuous line) because there are no contradictions, Jim has no enemies. That
means the murderer could be Joanna herself or Ken, his closest friend.
Considering the complete case, there are still many unknown Facts, so the
interrogation continues (Fig. 3), this time asking Ken about his friend. As the real culprit, the
player may try to deceive the detective lying about the existence of a secret project:
Jim was writing a book about the Mafia. He could probably reinforce the idea with
more Facts, explicitly lying about Jim and his “enemies”. This contradiction makes
“Jim has enemies” an unknown Fact that the detective will investigate.
Later on the game session, a false document is found in the crime scene (Fig. 4). As
the person who put that document there, Ken has been waiting for this moment and its
“dramatic effect”. A list of names of dangerous Mafia members is found in the desk of
the victim. The reasoner of the computer-controlled detective finds plausible that the
writer created that list, so it seems evident that Jim was really working on that book.</p>
      <p>But at the end of the game session, looking for more clues to solve other parts of
the mystery, more evidence appears: the detective notices that the document has been
folded as if someone stored it in his pocket (Fig. 5). It does not make sense that Jim
created that list and folded it before putting on his desk, so other person should have
done it, possibly trying to incriminate the Mafia instead of the real murderer. So Ken’
story about the Mafia was a lie, and probably previous Facts coming from him should
be considered “false”. Now the detective believes Ken is a liar and one of the main
suspects: a good approach to the solution of the whole case.</p>
    </sec>
    <sec id="sec-4">
      <title>5 Conclusions</title>
      <p>In this paper we surveyed previous works on computational trust and deception and
proposed a model for computer-controlled characters that suspects from others, not
trusting any information received from third parties. Previous work on trust models
relies on sources that generally do not lie, but in our case that assumption cannot be
accepted. Our model is designed for situations with incomplete information and a
constant need of judge what is likely to be true or false. Deciding who to trust is a key
process in real life that can support and enrich a very wide range of games.</p>
      <p>We used a ternary depiction for the truth value of facts and an automatic reasoner
for inferring potential outcomes starting from the known facts. The model manages a
list of potential truth values for each Fact, even coming from information given by a
third party, selecting the value that appears the most from trustful sources. It also
marks the other parties as trustful or not whenever it discards contradictory facts or
accepts valid ones coming from them. If the model cannot decide the truth value of
something (hence, it has the same number of supporting and refuting evidence), it will
mark it as “unknown”. We described superficially how a reasoning engine could work
using this model, trying to clear “unknown” values by reasoning over the potential
outcomes that any of the truth values could generate. Such engine needs to be
explored more in depth after a comparison of potential techniques, and will be the focus
of our next steps. We also designed a game mechanism that is illustrated with an
example scenario based in the first episode of the Columbo TV series for developing our
model. In this gameplay, the player takes the role of the culprit in a murder case and a
computer-controlled detective tries to discover who did it by looking for evidence and
questioning suspects, including the player.</p>
      <p>As the next steps of our research, we are currently working on a computational
model that implements this example scenario as a simple text-based dialogue,
allowing the user to play the culprit role with the computer acting as the detective. Also, we
plan to test the scenario, containing the complete murder mystery, with real users
taking the roles of the culprit and the detective in a “Wizard of Oz experiment” where
we can analyse how they act and what kind of questions are asked in order to evade
suspicion or discovering lies. With the results of this experiment, we will establish a
behavioural baseline for comparing our model with the one someone in the role of a
detective would have. Then we will re-enact the experiment, but this time using our
application where the users only play the culprit role, and we will compare the
behaviours of the human detective and the computational one.</p>
      <p>Further research opens interesting paths: we want to analyse the nature of deception
as well, so we want to expand our model allowing the rest of the non-player characters
to decide when they should lie in order to cover their alibi, or hide private
information. We think that creating a working model for trust and deception would be a
useful contribution to interactive storytelling and video games, and to the Artificial
Intelligence community as well, because it is a feature not fully explored yet.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Abedinzadeh</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sadahoui</surname>
            <given-names>S.:</given-names>
          </string-name>
          <article-title>A trust-based service suggestion system using human plausible reasoning</article-title>
          .
          <source>Applied Intelligence</source>
          <volume>41</volume>
          ,
          <issue>1</issue>
          (
          <year>2014</year>
          ):
          <fpage>55</fpage>
          -
          <lpage>75</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Amgoud</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cayrol</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>A reasoning model based on the production of acceptable arguments</article-title>
          .
          <source>Annals of Mathematics and Artificial Intelligence</source>
          <volume>34</volume>
          ,
          <fpage>1</fpage>
          -
          <lpage>3</lpage>
          (
          <year>2002</year>
          ):
          <fpage>197</fpage>
          -
          <lpage>215</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Barber</surname>
            ,
            <given-names>K.S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fullan</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kim</surname>
          </string-name>
          , J.:
          <article-title>Challenges for trust, fraud and deception research in multiagent systems</article-title>
          . Trust, Reputation, and
          <source>Security: Theories and Practice</source>
          . Springer Berlin Heidelberg (
          <year>2003</year>
          ):
          <fpage>8</fpage>
          -
          <lpage>14</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Berg</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dickhaut</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Mccabe</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          :
          <article-title>Trust, reciprocity, and social history</article-title>
          .
          <source>Games and Economic Behavior</source>
          <volume>10</volume>
          ,
          <issue>1</issue>
          (
          <year>1995</year>
          ):
          <fpage>122</fpage>
          -
          <lpage>142</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Broin</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          <string-name>
            <surname>Ó.</surname>
          </string-name>
          ,
          <string-name>
            <surname>O'Riordan</surname>
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>An evolutionary approach to deception in multi-agent systems</article-title>
          .
          <source>Artificial Intelligence Review</source>
          <volume>27</volume>
          ,
          <issue>4</issue>
          (
          <year>2007</year>
          ):
          <fpage>257</fpage>
          -
          <lpage>271</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Buntain</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Azaria</surname>
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kraus</surname>
            <given-names>S.:</given-names>
          </string-name>
          <article-title>Leveraging fee-based, imperfect advisors in human-agent games of trust</article-title>
          .
          <source>AAAI Conference on Artificial Intelligence</source>
          (
          <year>2014</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7. Collins,
          <string-name>
            <surname>A.</surname>
          </string-name>
          , Michalski, R.:
          <article-title>The logic of plausible reasoning: A core theory</article-title>
          .
          <source>Cognitive Science 13</source>
          ,
          <issue>1</issue>
          (
          <year>1989</year>
          ):
          <fpage>1</fpage>
          -
          <lpage>49</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Griffin</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Moore</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          :
          <article-title>A framework for modeling decision making and deception with semantic information</article-title>
          .
          <source>Security and Privacy Workshops</source>
          , IEEE Symposium (
          <year>2012</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Godden</surname>
            ,
            <given-names>D.J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Walton</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>Advances in the theory of argumentation schemes and critical questions</article-title>
          .
          <source>Informal Logic</source>
          <volume>27</volume>
          ,
          <issue>3</issue>
          (
          <year>2007</year>
          ):
          <fpage>267</fpage>
          -
          <lpage>292</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Ito</surname>
            ,
            <given-names>J. Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pynadath</surname>
            ,
            <given-names>D. V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Marsella</surname>
          </string-name>
          , S. C.
          <article-title>Modeling self-deception within a decisiontheoretic framework</article-title>
          .
          <source>Autonomous Agents and Multi-Agent Systems 20</source>
          ,
          <issue>1</issue>
          (
          <year>2010</year>
          ):
          <fpage>3</fpage>
          -
          <lpage>13</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Jameson</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>Impression monitoring in evaluation-oriented dialog: The role of the listener's assumed expectations and values in the generation of informative statements</article-title>
          .
          <source>International Joint Conference on Artificial intelligence 2</source>
          (
          <year>1983</year>
          ):
          <fpage>616</fpage>
          -
          <lpage>620</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Kalia</surname>
            ,
            <given-names>A.K.</given-names>
          </string-name>
          :
          <article-title>The semantic interpretation of trust in multiagent interactions</article-title>
          .
          <source>AAAI Conference on Artificial Intelligence</source>
          (
          <year>2014</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Lewicki</surname>
            ,
            <given-names>R.J.</given-names>
          </string-name>
          and
          <string-name>
            <surname>Bunker</surname>
            ,
            <given-names>B.B.</given-names>
          </string-name>
          :
          <article-title>Trust in relationships: A model of development and decline</article-title>
          . Jossey-Bass (
          <year>1995</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Li</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cruz</surname>
            ,
            <given-names>J.B.</given-names>
          </string-name>
          :
          <article-title>Information, decision-making and deception in games</article-title>
          .
          <source>Decision Support Systems</source>
          <volume>47</volume>
          ,
          <issue>4</issue>
          (
          <year>2009</year>
          ):
          <fpage>518</fpage>
          -
          <lpage>527</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Li</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wang</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          :
          <article-title>The roadmap of trust and trust evaluation in web applications and web services</article-title>
          .
          <source>Advanced Web Services</source>
          . Springer New York (
          <year>2014</year>
          ):
          <fpage>75</fpage>
          -
          <lpage>99</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Ma</surname>
            ,
            <given-names>Z.S.</given-names>
          </string-name>
          :
          <article-title>Towards an extended evolutionary game theory with survival analysis and agreement algorithms for modeling uncertainty, vulnerability, and deception</article-title>
          .
          <source>Artificial Intelligence and Computational Intelligence</source>
          . Springer Berlin Heidelberg (
          <year>2009</year>
          ):
          <fpage>608</fpage>
          -
          <lpage>618</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Marková</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gillespie</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>Trust and distrust: Sociocultural perspectives</article-title>
          .
          <source>Information Age Publishing</source>
          , Inc. (
          <year>2008</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>Nissan</surname>
          </string-name>
          , E.:
          <article-title>Epistemic formulae, argument structures, and a narrative on identity and deception: a formal representation from the AJIT subproject within AURANGZEB</article-title>
          .
          <source>Annals of Mathematics and Artificial Intelligence</source>
          <volume>54</volume>
          ,
          <issue>4</issue>
          (
          <year>2008</year>
          ):
          <fpage>293</fpage>
          -
          <lpage>362</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>Nissan</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rousseau</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>Towards AI formalisms for legal evidence</article-title>
          .
          <source>Foundations of Intelligent Systems</source>
          . Springer Berlin Heidelberg (
          <year>1997</year>
          ):
          <fpage>328</fpage>
          -
          <lpage>337</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>Prawitz</surname>
            ,
            <given-names>D. Natural</given-names>
          </string-name>
          <string-name>
            <surname>Deduction</surname>
          </string-name>
          . A
          <string-name>
            <surname>Proof-Theoretical Study</surname>
          </string-name>
          .
          <source>Almqvist &amp; Wiksell</source>
          , Stockholm (
          <year>1965</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          21.
          <string-name>
            <surname>Steren</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bonelli</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          <article-title>Intuitionistic hypothetical logic of proofs</article-title>
          .
          <source>Electronic Notes in Theoretical Computer Science</source>
          ,
          <volume>300</volume>
          (
          <year>2014</year>
          ):
          <fpage>89</fpage>
          -
          <lpage>103</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          22.
          <string-name>
            <surname>Walterbusch</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <given-names>Graüler. M.</given-names>
            ,
            <surname>Teuteberg</surname>
          </string-name>
          ,
          <string-name>
            <surname>F.</surname>
          </string-name>
          :
          <article-title>How trust is defined: A qualitative and quantitative analysis of scientific literature (</article-title>
          <year>2014</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          23.
          <string-name>
            <surname>Walton</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          :
          <article-title>The three bases for the enthymeme: A dialogical theory</article-title>
          .
          <source>Journal of Applied Logic</source>
          <volume>6</volume>
          ,
          <issue>3</issue>
          (
          <year>2008</year>
          ):
          <fpage>361</fpage>
          -
          <lpage>379</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          24.
          <string-name>
            <surname>Wang</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lin</surname>
            ,
            <given-names>K.J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wong</surname>
            ,
            <given-names>D.S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Varadharajan</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          :
          <article-title>Trust management towards serviceoriented applications</article-title>
          .
          <source>Service Oriented Computing and Applications</source>
          <volume>3</volume>
          ,
          <issue>2</issue>
          (
          <year>2009</year>
          ):
          <fpage>129</fpage>
          -
          <lpage>146</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          25.
          <string-name>
            <surname>Xiong</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Liu</surname>
          </string-name>
          , L.:
          <article-title>PeerTrust: Supporting reputation-based trust for peer-to-peer electronic communities. Knowledge and Data Engineering</article-title>
          ,
          <source>IEEE Transactions 16</source>
          ,
          <issue>7</issue>
          (
          <year>2004</year>
          ):
          <fpage>843</fpage>
          -
          <lpage>857</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          26.
          <string-name>
            <surname>Zhou</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          <string-name>
            <surname>Hwang</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          :
          <article-title>PowerTrust: A robust and scalable reputation system for trusted peerto-peer computing. Parallel and Distributed Systems</article-title>
          ,
          <source>IEEE Transactions 18</source>
          ,
          <issue>4</issue>
          (
          <year>2007</year>
          ):
          <fpage>460</fpage>
          -
          <lpage>473</lpage>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>