<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>To build conscious machines, focus on general intelligence: a framework for the assessment of consciousness in biological and artificial systems</article-title>
      </title-group>
      <contrib-group>
        <aff id="aff0">
          <label>0</label>
          <institution>Leverhulme Centre for the Future of Intelligence, University of Cambridge</institution>
        </aff>
      </contrib-group>
      <fpage>0000</fpage>
      <lpage>0002</lpage>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>Introduction</title>
      <p>
        Consciousness presents us with a number of different explanatory challenges. The most
fundamental, sometimes called the ‘Hard Problem’, is focused on how subjective states
can arise from objective physical systems [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. Another important question concerns the
cognitive mechanisms that distinguish conscious from unconscious states [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]. A third
debate, and the one that will be the focus of the present enquiry, concerns how we can
determine whether a given biological or artificial agent is conscious at all. Of the three
questions, the latter has particular practical and ethical significance: our treatment of
animals depends in part on whether we regard them as having a capacity for conscious
experience [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. Likewise, while few would endorse the idea that current artificial
systems are conscious, as their capacities improve and come to more closely resemble
those of animals and humans, ethical and legal questions concerning machine
consciousness will likely loom large.
      </p>
      <p>In this paper, I will argue that a useful framework for the assessment of
consciousness in both animals and machines can come from the notion of general intelligence. I
begin in Section 2 by noting the important connections between our concepts of
intelligence and consciousness. In Section 3, I argue that it is general rather than specialised
intelligence that carries the greatest weight in our assessments of consciousness, and
offer a preliminary framework for the assessment of general intelligence that appeals
to three features, namely robustness, flexibility, and system-wide integration. In
Section 4, I argue that current artificial systems, unlike many non-human animals, currently
fail to exhibit these features of general intelligence to any significant degree. As a result,
I suggest that we have some reason to think artificial consciousness remains a distant
goal. Finally, in Section 5, I briefly survey some challenges faced by a framework that
takes general intelligence to be our best evidence of consciousness.</p>
      <p>
        Before proceeding, it will be helpful to provide a brief gloss on the notions of
consciousness and intelligence. In short, I use the term consciousness to refer to a capacity
for subjective experience. For conscious creatures, there is something it’s like to have
experiences in the sense of Nagel [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]: colours may look a certain way and pains feel a
certain way to them. I will use the term intelligence broadly to refer to the capacity of
a system to use information processing to achieve its goals in an efficient and effective
manner. Note that I regard consciousness as a pretheoretical concept whose reference
we grasp first-hand. By contrast, intelligence is a theoretical concept open to revision.
2
      </p>
    </sec>
    <sec id="sec-2">
      <title>Consciousness and intelligence</title>
      <p>
        We have all intuitions about which animals are conscious. Most of us, I suspect, would
regard it as beyond serious question that dolphins, chimpanzees, and dogs undergo
subjective experiences. Likewise, relatively few take seriously the idea that extremely
simple systems such as a thermostats, bacteria, or plants are conscious (however, see [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ]
and [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]). Between these two extremes, however, there is considerable disagreement.
For example, there is considerable controversy regarding consciousness in fish [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ],
cephalopods [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ], and insects [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ]. This lack of agreement in intuitions is reflected by the
disparate nature of legal protections for different species in different jurisdictions. Thus
while the British Animals (Scientific Procedures) Act of 1986 extends protections to
all vertebrates as well as octopuses, the corresponding American legislation (7 U.S.C.
§ 2131-2156) makes provision only for warm-blooded animals. Similar controversies
also arise for humans in relation to patients in comas and persistent vegetative states,
as well as foetuses, and are likely in time to arise for machine intelligences.
      </p>
      <p>A tempting response to these conflicting intuitions may be to disregard their value
entirely, and adopt a purely scientific criterion of consciousness. I would suggest,
however, that such a conclusion would be misguided. Consciousness is a pretheoretical
concept with deep connections to our ethical practises, and cannot simply be
operationalised in the interests of scientific expediency. While we might simply stipulate, for
example, that we will define consciousness as sensitivity to external stimuli or a
capacity for higher-order cognition, these definitions will not serve the purposes that are
required of a theory of consciousness. Faced with questions like whether fish feel pain,
or whether patients in persistent vegetative states are having experiences, we wish to
know whether they really have subjective experiences, not just whether they are
sensitive to external stimuli or have intact metacognitive capacities.</p>
      <p>Any theory of consciousness that will serve the purposes to which we wish to put it,
then, must reflect and engage with our standing folk psychological concept of
consciousness. This is not to say, of course, that our current attitudes are immune to
revision. An important part of our task in giving a theory of consciousness is identifying
ungrounded biases and assumptions that regulate our intuitions about which animals
are conscious, such as our tendency to attribute it to charismatic megafauna. Moreover,
after identifying and regimenting our core pretheoretical commitments concerning the
nature of consciousness, we may find that far more (or far fewer) systems satisfy them
than we had previously assumed.</p>
      <p>
        In light of this, I would suggest that an important starting point in reflecting on
consciousness comes from the powerful pretheoretical connection between consciousness
and intelligence. As noted above, we unhesitatingly assign consciousness to creatures
like chimpanzees, dolphins, and dogs. It is surely no coincidence that these animals are
all highly intelligent. Likewise, systems that strike us as very poor consciousness
candidates (in the sense of Birch, [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ]) tend to be extremely cognitively simple, capable of
little intelligent behaviour. A somewhat similar story can be told for systems
intermediate between these two poles, as shown below (Fig. 1).
      </p>
      <p>
        There are grounds for thinking that this apparent connection between intelligence
and our assessment of consciousness is not an idle correlation. When new evidence
emerges of intelligent behaviour in a species, it is likely to increase our confidence that
the species is conscious, and in turn be reflected in legal protections; the decisions to
afford rights to many cephalopods in European Directive 2010/63/EU (5), for example,
was prompted by evidence that they possess higher-brain areas and sophisticated
behavioural responses to pain [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ]. Similarly, the growing scientific consensus that some
patients in persistent vegetative states are conscious followed the discovery that some
PVS patients retained an ability to accurately answer yes or no questions in a brain
scanning paradigm [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ].
      </p>
      <p>There are of course many tricky conceptual issues to wrangle with in relating
consciousness to intelligence, some of which will occupy much of the rest of this paper.
One important issue worth immediately flagging concerns whether we should regard
consciousness as a discrete phenomenon, or might be able talk of degrees of
consciousness in intelligent systems. This is a point of considerable philosophical and scientific
controversy, however, and will be set aside in what follows.
3</p>
    </sec>
    <sec id="sec-3">
      <title>Distinguishing general and specialised intelligence</title>
      <p>Despite the connection between judgments of intelligence and consciousness, few
would consider existing machine intelligences to be strong consciousness candidates,
despite their impressive performance on a number of seemingly demanding tasks. There
are a number of reasons both good and bad why this might be the case. However, as I
will now argue, one well-founded motivation may come from the difference between
specialised and general intelligence. In short, I suggest that insofar as there is an
important connection between consciousness and intelligence, it is general intelligence
that matters.</p>
      <p>
        First, however, it is necessary to at least outline what I take the terms to mean.
Specialised intelligence is relatively easy to grasp, and can be spelled out in terms of the
definition of intelligence given earlier, namely the ability to use information processing
effectively and efficiently in the pursuit of some narrowly specified goal. General
intelligence is more difficult. The notion of generality in artificial intelligence was
discussed in an important paper by McCarthy [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ], and can broadly be understood as the
ability to use “the same goal-seeking mechanism for all kinds of problems, changing
only the particular productions.” As humans, we like to think of ourselves as having a
high degree of general intelligence, as reflected in our complex societies and elaborate
cultural products. However, on McCarthy’s definition, it seems like that many different
animals would also qualify as having high general intelligence: via well-integrated
systems of perception, memory, and learning, they overcome numerous different tasks,
ranging from long-distance navigation, predation and predator avoidance, mating, and
the satisfaction of numerous competing physiological needs.
      </p>
      <p>What is needed, however, is a more systematic framework for assessing general
intelligence in different systems. With this in mind, I would suggest that three features
are commonly found in systems that we would regard as generally intelligent. In short,
these are robustness, flexibility, and whole-system integration, to be spelled out as
follows.</p>
      <p>Robustness: the ability to achieve tasks despite interference. Systems with a high
general intelligence are typically robust (resisting failure) and resilient (recovering from
failure).</p>
      <p>Flexibility: the ability to transfer knowledge across tasks. Generally intelligent
systems can readily apply existing skills/information to new domains.</p>
      <p>Whole-system integration: the ability of a creature to integrate and effectively
counterbalance inputs from different systems, including perception, memory, and
drives.</p>
      <p>As an inventory of features of generally intelligent systems, this list is highly
preliminary, and may prove to be incomplete or have unnecessary components. Note, for
example, that there are clear connections between robustness and flexibility: a system
is likely to be more robust in dealing with unexpected impediments to its goals if it also
displays a high degree of flexibility. Likewise, a system is likely to be more flexible if
it can integrate all of its sources of information for the intelligent production of
behaviour.</p>
      <p>Nonetheless, I would suggest that this initial framing of the features of general
intelligence has considerable value for the assessment of general intelligence in different
systems. More importantly for present purposes, however, all of these features have
both pretheoretical and scientific appeal as parameters in the assessment of which
creatures are conscious.</p>
      <p>
        To illustrate this point, imagine that we have identified some seemingly very
intelligent behaviour in a species not previously considered a good consciousness candidate;
suppose, say, that we found evidence of complex tool use in bivalve molluscs. Prima
facie, this would make the creature a stronger consciousness candidate. However, if it
were found that the behaviour failed to satisfy one of the dimensions of general
intelligence discussed above, this impression would be undermined. Thus if we discovered
that the behaviour was non-robust and failed outside of extremely specific conditions,
we might naturally assume its occurrence relied on simple hard-wired mechanisms
rather than being a marker of sophisticated cognition; the contested case of the Sphex
wasp might serve as an example of such [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ]. We might draw a similar conclusion if
the behaviour proved to be wholly non-flexible, and could be applied to only one very
narrow purpose; in such a case, the behaviour might best be explained via a single
evolutionarily honed instinct. Consider, for example, the complex but highly stereotyped
nest-building performed by many insects. Finally, if neuroscientific enquiry revealed
the behaviour to be accomplished not via the central nervous system, but some wholly
non-integrated neural module, we would be much less likely to think it good evidence
for the creature’s being conscious.
      </p>
      <p>
        The above examples should serve to provide some initial motivation for the claim
that general intelligence understood as the possession of robust, flexible, and integrated
behavioural capacities provides evidence of consciousness. However, I would also note
that it provides a good fit for many other approaches to consciousness. As noted above,
evidence for consciousness in persistent vegetative state patients comes in large part
from their ability to perform a highly flexible task, namely accurately answering a range
of yes or no questions about different personal and factual matters. Likewise, tests of
machine intelligence and consciousness such as the Turing Test and the Winograd
schema measure an artificial system’s capacity to engage in flexible and robust forms
of verbal reasoning, and objections to the value of such tests such as Searle’s Chinese
Room sometimes rely on showing how the task can be performed via ‘dumb’
non-integrated processes [
        <xref ref-type="bibr" rid="ref15 ref16">15, 16</xref>
        ]. Finally, note that the proposed schema for assessing general
intelligence exhibits some promising connections with contemporary scientific
approaches to consciousness. An emphasis on cross system integration, for example, is
common to many leading theories such as Integrated Information Theory and Global
Workspace Theory, and there is empirical reason to think that many highly flexible
forms of behaviour such as memory-trace conditioning and unconscious two-step
arithmetic can be performed only under conscious conditions [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ].
      </p>
      <p>Note that I am not proposing that consciousness be identified with general
intelligence, nor suggesting general intelligence as the mechanism by which consciousness
arises. The former goal involves conceptual difficulties best left to metaphysicians,
while I regard questions about the mechanisms of consciousness as complementary to
the current proposal. Instead, I am suggesting that general intelligence – understood as
a capacity for robust, flexible, integrated cognitive performance – constitutes an
important (if not our best) source of evidence of consciousness.
4</p>
    </sec>
    <sec id="sec-4">
      <title>General intelligence in biological and artificial systems</title>
      <p>Assuming, then, that general intelligence is a good marker for consciousness, then, we
might ask how it applies to different cognitive systems. As suggested earlier, many
animals do well by this metric. Most animal behaviour is frequently highly robust, with
feeding, mating, and thriving being accomplished in a wide range of varied
environments and climatic conditions. While it is easy to find video examples of animals
making foolish mistakes or falling over themselves, these occurrences are rare enough that
they amuse and surprise us when they do occur. There are of course strong evolutionary
reasons why we should expect animals to have robust capacities. Nonetheless, that does
not mean that this achievement is easily won in cognitive terms.</p>
      <p>
        Similarly, animal behaviour is often highly flexible. The ability of even simple
creatures such as bees to engage in novel social learning and concept acquisition [
        <xref ref-type="bibr" rid="ref18 ref19">18, 19</xref>
        ]
in conditions significantly removed from their evolutionary environments is striking.
Among more intelligent creatures such as crows and scrubjays, examples abound of
sophisticated and adaptive causal reasoning [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ] and clever caching behaviours that are
sensitive to a wide range of environmental factors and physiological needs [
        <xref ref-type="bibr" rid="ref21">21</xref>
        ].
      </p>
      <p>
        Finally, there are many examples of highly-developed integration of different
systems within animals, ranging from simple phenomena such as the use of motor and
vestibular cues to distinguish endogenously- from exogenously-generated changes in
sensory input, to the ability of fish, rats, and some crustaceans to engage in
‘motivational tradeoff’, the rapid online adjustment of behaviour to accommodate different
desires and aversions [
        <xref ref-type="bibr" rid="ref22">22</xref>
        ].
      </p>
      <p>
        This is a highly condensed review of just some of the ways in which animals display
impressive forms of general intelligence, but it at least provides a useful point of
comparison for examining the state of general intelligence current artificial systems. As
anyone familiar with the current capacities of artificial systems can attest, these are
fairly dismal. In most domains, AI behaviour is non-robust: machine intelligences
struggle with tasks outside of highly-regulated training environments, and are
vulnerable to a pile up of small errors. The problem of adversarial examples constitutes a vivid
case of this: machine vision systems remain vulnerable to making spectacular errors
when fooled by clever perturbations of input data (see Fig. 2, below) [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ].
      </p>
      <p>Filter
‘Ostrich’</p>
      <p>
        Similarly, most current AIs are highly inflexible. While important progress has been
made on transfer learning tasks, even the best current systems gain only minor
performance improvements when leveraging prior knowledge to variant tasks [
        <xref ref-type="bibr" rid="ref25">25</xref>
        ].
Somewhat more progress has been made on minimising the impact of catastrophic forgetting
in machine intelligences, but again, artificial systems exhibit strikingly limited capacity
in comparison to non-human animals to engage in fluid task switching without
considerable loss of prior knowledge. Finally, most machine intelligence systems rely on
extended exposure to large training sets, again a dramatic contrast with biological
intelligence (consider that a newborn fawn learns to stand up after 10 minutes and walk
smoothly in just 7 hours).
      </p>
      <p>Most AIs similarly fail to satisfy the integration component of general intelligence,
for the simple reason they are wholly specialised machines. Even in systems that
perform multiple functions, this is typically done via highly modular design in which there
is little true integration of processing across subsystems. Integration of the kind found
in animals arguably requires a centralised capacity for modelling the world and one’s
action space within it in rich detail, something AI has yet to come close to achieving.
5</p>
    </sec>
    <sec id="sec-5">
      <title>Objections</title>
      <p>
        Before concluding, it is worth briefly mentioning three important objections to this
approach. First, one may question whether general intelligence is really a robust scientific
concept [
        <xref ref-type="bibr" rid="ref26">26</xref>
        ]. If what we call general intelligence cannot really be quantified even in a
multidimensional framework, but is best considered a loose ‘bag of tricks’, then this
might certainly limit the utility of the approach defended her. However, this is a matter
of outstanding scientific debate, and I would suggest that general intelligence may serve
as a useful heuristic for assessments of consciousness even if it fails to track any unified
set of cognitive mechanisms.
      </p>
      <p>
        Second, it might be objected that there is little point developing measures of
consciousness that are not explicitly related to specific psychological mechanisms such as
attention, metacognition, or working memory. My response to this claim is that I regard
the current approach as complementary to attempts to identify fine-grained cognitive
structure of consciousness. It is also compatible with my approach that, as many have
argued, there is no such structure to be found [
        <xref ref-type="bibr" rid="ref27">27</xref>
        ].
      </p>
      <p>Third, a key worry for the tripartite schema for assessing general intelligence and
consciousness given above is that it will be of practical value only insofar as we can
develop principled ways of measuring robustness, flexibility, and whole-system
integration. This may be an extremely challenging task. For example, while we might
regard human arithmetical capabilities as highly flexible, few of us could compute square
roots of large numbers in our heads, while this task is trivial for many machines. How
to assess and assign weights both to the overall flexibility of a system and toits
flexibility in different domains, then, remains an important outstanding challenge.
6</p>
    </sec>
    <sec id="sec-6">
      <title>Conclusion</title>
      <p>My main claims in this paper have been threefold. First, I have argued that there are
important connections between the notions of consciousness and intelligence, with
general intelligence in particular having an important evidential role for our assessments
of consciousness. Second, I have suggested that general intelligence can be helpfully
conceptualised as spanning three dimensions, encompassing robustness, generality, and
whole system integration. Third, I have claimed that while many animals perform well
by these metrics, current artificial systems perform extremely poorly, and as such, it is
unlikely that near-future artificial intelligences will be conscious. This may seem to
some a pessimistic conclusion. However, if my arguments in this paper are well
founded, I would suggest it provides important guidance for those interested in building
conscious machines, and clear criteria to aim for: if you want a conscious machine,
focus on building one that is as smart as a crow.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Chalmers</surname>
            ,
            <given-names>David J.</given-names>
          </string-name>
          (
          <year>1995</year>
          ).
          <article-title>Facing up to the problem of consciousness</article-title>
          .
          <source>Journal of Consciousness Studies</source>
          <volume>2</volume>
          (
          <issue>3</issue>
          ):
          <fpage>200</fpage>
          -
          <lpage>19</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Block</surname>
          </string-name>
          ,
          <string-name>
            <surname>Ned</surname>
          </string-name>
          (
          <year>2009</year>
          ).
          <article-title>Comparing the major theories of consciousness</article-title>
          . In Michael Gazzaniga (ed.),
          <source>The Cognitive Neurosciences IV</source>
          . pp.
          <fpage>1111</fpage>
          -
          <lpage>1123</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Singer</surname>
          </string-name>
          ,
          <string-name>
            <surname>Peter</surname>
          </string-name>
          (
          <year>1989</year>
          ).
          <article-title>All Animals Are Equal</article-title>
          . In Tom Regan &amp; Peter Singer (eds.),
          <source>Animal Rights and Human Obligations</source>
          . Oxford University Press. pp.
          <fpage>215</fpage>
          --
          <lpage>226</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Nagel</surname>
          </string-name>
          , Thomas (
          <year>1974</year>
          ).
          <article-title>What is it like to be a bat? Philosophical Review 83 (October</article-title>
          ):
          <fpage>435</fpage>
          -
          <lpage>50</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Chalmers</surname>
            ,
            <given-names>David J.</given-names>
          </string-name>
          (
          <year>1996</year>
          ).
          <article-title>The Conscious Mind: In Search of a Fundamental Theory</article-title>
          . Oxford University Press.
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Tononi</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          , &amp;
          <string-name>
            <surname>Koch</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          (
          <year>2015</year>
          ).
          <article-title>Consciousness: Here, there</article-title>
          and everywhere?
          <source>Philosophical Transactions of the Royal Society B: Biological Sciences</source>
          ,
          <volume>370</volume>
          (
          <issue>1668</issue>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Key</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          (
          <year>2014</year>
          ).
          <article-title>Fish do not feel pain and its implications for understanding phenomenal consciousness</article-title>
          .
          <source>Biology &amp; Philosophy March</source>
          <year>2015</year>
          , Volume
          <volume>30</volume>
          , Issue 2, pp
          <fpage>149</fpage>
          -
          <lpage>165</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Godfrey-Smith</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          (
          <year>2017</year>
          ).
          <article-title>Other Minds: The Octopus and the Evolution of Intelligent Life</article-title>
          . William Collins.
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Barron</surname>
            ,
            <given-names>A. B.</given-names>
          </string-name>
          , &amp;
          <string-name>
            <surname>Klein</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          (
          <year>2016</year>
          ).
          <article-title>What insects can tell us about the origins of consciousness</article-title>
          .
          <source>Proceedings of the National Academy of Sciences Proc Natl Acad Sci USA</source>
          ,
          <volume>113</volume>
          (
          <issue>18</issue>
          ),
          <fpage>4900</fpage>
          -
          <lpage>4908</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Birch</surname>
          </string-name>
          ,
          <string-name>
            <surname>Jonathan</surname>
          </string-name>
          (
          <year>2017</year>
          ).
          <article-title>Animal Sentience and the Precautionary Principle</article-title>
          .
          <source>Animal Sentience</source>
          <volume>2</volume>
          :
          <issue>16</issue>
          (
          <issue>1</issue>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Fiorito</surname>
            <given-names>G</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Affuso</surname>
            <given-names>A</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Basil</surname>
            <given-names>J</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cole</surname>
            <given-names>A</given-names>
          </string-name>
          ,
          <string-name>
            <surname>de Girolamo</surname>
            <given-names>P</given-names>
          </string-name>
          ,
          <string-name>
            <surname>D'Angelo</surname>
            <given-names>L</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dickel</surname>
            <given-names>L</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gestal</surname>
            <given-names>C</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Grasso</surname>
            <given-names>F</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kuba</surname>
            <given-names>M</given-names>
          </string-name>
          ,
          <article-title>Mark F. Guidelines for the Care and Welfare of Cephalopods in Research-A consensus based on an initiative by CephRes, FELASA</article-title>
          and the Boyd Group. Laboratory animals.
          <year>2015</year>
          ;
          <volume>49</volume>
          (
          <issue>2 suppl)</issue>
          :
          <fpage>1</fpage>
          -
          <lpage>90</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Monti</surname>
            ,
            <given-names>M.M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Owen</surname>
            ,
            <given-names>A.M.</given-names>
          </string-name>
          (
          <year>2010</year>
          ).
          <article-title>The behavior in the brain: using functional neuroimaging to assess residual cognition and awareness after severe brain injury</article-title>
          ,
          <source>Journal of Psychophysiology</source>
          ,
          <volume>24</volume>
          (
          <issue>2</issue>
          ):
          <fpage>76</fpage>
          -
          <lpage>82</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>McCarthy</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <year>1990</year>
          .
          <article-title>"Generality in artificial intelligence"</article-title>
          . In
          <string-name>
            <surname>Lifschitz</surname>
          </string-name>
          , V., ed.,
          <source>Formalizing Common Sense. Ablex</source>
          .
          <volume>226</volume>
          -
          <fpage>236</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Dennett</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          (
          <year>1973</year>
          ).
          <article-title>Mechanism and responsibility</article-title>
          . In T. Honderich (Ed.),
          <source>Essays on freedom of action. London: Routledge.</source>
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Searle</surname>
            ,
            <given-names>J.R.</given-names>
          </string-name>
          (
          <year>1980</year>
          ).
          <article-title>Minds, brains and programs</article-title>
          .
          <source>Behavioral and Brain Sciences</source>
          <volume>3</volume>
          (
          <issue>3</issue>
          ):
          <fpage>417</fpage>
          -
          <lpage>57</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Sharma</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ha Vo</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Aditya</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Baral</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          "
          <volume>015</volume>
          ).
          <article-title>Towards addressing the winograd schema challenge-building and using a semantic parser and a knowledge hunting module</article-title>
          .
          <source>In IJCAI</source>
          , pages
          <fpage>1319</fpage>
          -
          <lpage>1325</lpage>
          ,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Dehaene</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          (
          <year>2014</year>
          ).
          <article-title>Consciousness and the Brain: Deciphering How the Brain Codes Our Thoughts</article-title>
          . Viking Press,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>Alem</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Perry</surname>
            ,
            <given-names>C.J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zhu</surname>
            ,
            <given-names>X.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Loukola</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ingraham</surname>
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Søvik</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Chittka</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          <article-title>Associative mechanisms allow for social learning and cultural transmission of string pulling in an insect</article-title>
          .
          <source>2016; PLoS Biol</source>
          <volume>14</volume>
          (
          <issue>10</issue>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>Avargues-Weber</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dyer</surname>
            ,
            <given-names>A. G.</given-names>
          </string-name>
          , &amp;
          <string-name>
            <surname>Giurfa</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          (
          <year>2010</year>
          ).
          <article-title>Conceptualization of above and below relationships by an insect</article-title>
          .
          <source>Proceedings of the Royal Society B: Biological Sciences</source>
          ,
          <volume>278</volume>
          (
          <issue>1707</issue>
          ),
          <fpage>898</fpage>
          -
          <lpage>905</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>Jelbert</surname>
            <given-names>SA</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Taylor</surname>
            <given-names>AH</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cheke</surname>
            <given-names>LG</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Clayton</surname>
            <given-names>NS</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gray</surname>
            <given-names>RD</given-names>
          </string-name>
          (
          <year>2014</year>
          ).
          <article-title>"Using the Aesop's Fable Paradigm to Investigate Causal Understanding of Water Displacement by New Caledonian Crows"</article-title>
          .
          <source>PLoS ONE. 9: e92895. doi:10.1371/journal.pone.0092895</source>
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          21.
          <string-name>
            <surname>Clayton N.S</surname>
          </string-name>
          ,
          <string-name>
            <surname>Bussey</surname>
            <given-names>T.J</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dickinson</surname>
            <given-names>A</given-names>
          </string-name>
          .
          <article-title>Can animals recall the past and plan for the future</article-title>
          ?
          <source>Nat. Rev. Neurosci. 2003a;</source>
          <volume>4</volume>
          :
          <fpage>685</fpage>
          -
          <lpage>691</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>22. Motivational Tradeoff</mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          23.
          <string-name>
            <surname>Kurakin</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Goodfellow</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Bengio</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <article-title>Adversarial examples in the physical world</article-title>
          .
          <source>arXiv preprint arXiv:1607.02533</source>
          (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          24.
          <string-name>
            <surname>Scharr</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          (
          <year>2018</year>
          ).
          <article-title>Army of None: Autonomous Weapons and the Future of War</article-title>
          . Norton.
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          25.
          <string-name>
            <surname>Lake</surname>
            ,
            <given-names>B.M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ullman</surname>
          </string-name>
          , T.D.,
          <string-name>
            <surname>Tenenbaum</surname>
            ,
            <given-names>J.B.</given-names>
          </string-name>
          , &amp;
          <string-name>
            <surname>Gershman</surname>
            ,
            <given-names>S.J.</given-names>
          </string-name>
          (
          <year>2016</year>
          ).
          <article-title>Building Machines That Learn and Think like People.” Behavioral and Brain Sciences</article-title>
          , vol.
          <volume>40</volume>
          ,
          <year>2016</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          26.
          <string-name>
            <surname>Serpico</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          (
          <year>2018</year>
          ).
          <article-title>What kind of kind is intelligence</article-title>
          ? Philosophical Psychology,
          <volume>31</volume>
          (
          <issue>2</issue>
          ),
          <fpage>232</fpage>
          -
          <lpage>252</lpage>
          . doi:
          <volume>10</volume>
          .1080/09515089.
          <year>2017</year>
          .1401706
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          27. Dennett, Daniel C. (
          <year>1991</year>
          ). Consciousness Explained. Penguin Books.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>