<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta>
      <journal-title-group>
        <journal-title>AT</journal-title>
      </journal-title-group>
    </journal-meta>
    <article-meta>
      <title-group>
        <article-title>The Design of Intelligent Socio-Technical Systems*</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Andrew J I Jones</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Alexander Artikis</string-name>
          <xref ref-type="aff" rid="aff2">2</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Jeremy Pitt</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Department of Electrical &amp; Electronic Engineering, Imperial College London</institution>
          ,
          <country country="UK">UK</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Department of Informatics, King's College London</institution>
          ,
          <country country="UK">UK</country>
        </aff>
        <aff id="aff2">
          <label>2</label>
          <institution>National Centre for Scientific Research 'Demokritos'</institution>
          ,
          <addr-line>Athens</addr-line>
          ,
          <country country="GR">Greece</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2012</year>
      </pub-date>
      <volume>15</volume>
      <fpage>15</fpage>
      <lpage>16</lpage>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>Introduction</title>
      <p>………there is no such thing as philosophy-free science; there is only science whose
philosophical baggage is taken on board without examination. [5, p. 21]</p>
      <p>To a significant extent, research in Computer Science that aims to develop
sociotechnical systems has to address issues pertaining to the interpretation of social and
organizational concepts. The components of socio-technical systems, be they artefacts
or humans, carry out their work by interacting with each other against a social,
organizational or legal background. The field of Autonomous Agents and Multi-agent
Systems has for some time represented an obvious example of this work, but the
important part played by social concepts extends into other parts of Computer Science
too. Consider – to mention just three further domains – Computer Security, where the
notions of trust, reputation and role have figured prominently; E-commerce, where
the representation, formation and fulfillment of contracts is fundamental; and
Egovernment, where representing and reasoning about policies and norms are essential.</p>
      <p>In Biology and Social Science, in Jurisprudence, and in Analytical Philosophy,
among other disciplines, we find examples of conceptual models designed to enhance
our understanding of the nature of organized interaction. In writing this paper, our
initial question was this: in their construction of so-called computational models of
social concepts, such as those mentioned in the previous paragraph, have computer
scientists been sufficiently informed by conceptual models of social phenomena, the
construction of which was not motivated by computational considerations, but aimed
primarily to reveal, in a systematic fashion, the structure and interconnections of the
concepts themselves ? Through its attempt to answer that question, the principal
contribution of this paper is a proposed approach to the engineering of socio-technical
systems that respects the interdisciplinary nature of the task, in regard to both its
theoretical and practical dimensions.</p>
      <p>In the full paper, of which this document is an extended abstract, we proceed in
Section 2 by giving some examples of work that would justify a negative answer to
our initial question, and we explain their shortcomings. Against that background,
Section 3 describes an approach to the engineering of socio-technical systems in
which rich, conceptual-analytical models and computational frameworks are
combined, providing a basis for principled operationalisation, observing that similar
methodological concerns have arisen in the field of biologically-inspired computing.
We describe the approach in terms of a sequence of steps and, accordingly, in Section
4 we formulate and illustrate adequacy criteria that, ideally, the key steps should
satisfy. In the concluding section we suggest, in particular, that if our general
methodological proposals were to be adopted, they should have significant consequences for the
ways in which researchers are trained, not least in the area of Autonomous Agents and
Multi-agent Systems.
2</p>
    </sec>
    <sec id="sec-2">
      <title>Motivating Examples</title>
      <p>In this section we consider three examples of work on the engineering of
sociotechnical systems in which social concepts – specifically trust, role and normative
power – have figured prominently.
2.1</p>
      <sec id="sec-2-1">
        <title>Normative Power</title>
        <p>
          Any reasonably comprehensive model, formal or informal, of norm-governed
multiagent systems must be able to accommodate norms pertaining to institutionalized
normative power, in addition to those that express obligations and permissions. It is a
commonplace feature of organizations that particular agents, individually or
collectively, are empowered to carry out actions, the consequences of which have a
significant bearing on the way the organization is governed or administered. For instance,
some public officials/bodies will be empowered to create a state of marriage between
two individuals, or to validate wills, or to appoint some other persons to particular
roles (including roles that themselves involve the possession of powers), or to create
or modify laws and regulations. Powers of this sort are types of rights, or entitlements,
that some agents have, and others lack. There is a substantial body of literature,
stemming from Hohfeld [
          <xref ref-type="bibr" rid="ref8">8</xref>
          ], that focuses on the systematic characterization of types
of rights-relations, including in some cases formal analyses of these relations
expressed in terms of a small set of basic operators drawn from modal logic
[
          <xref ref-type="bibr" rid="ref10 ref11 ref12 ref15 ref20 ref9">11,12,20,15,9,10</xref>
          ].
        </p>
        <p>
          Oren et al. [
          <xref ref-type="bibr" rid="ref17">17</xref>
          ] present a model of what they call ‘normative power’, which they
associate with the power to create and/or modify norms. While they refer to the
Hohfeldian tradition, they make no use of the analyses offered therein, preferring
instead to characterize normative power by means of a first-order logic tuple, the key
element of which is called ‘mandators’. “Mandators is a set of predicates identifying
agents” [17, p. 817], and “A mandator of the form professor(x) means that any agent
in the professor role is able to exercise the power”, for instance the power to place a
student under an obligation to write a conference paper [17, p. 819]. Note that the
interpretation of what it means for an agent to be able to exercise a normative power
is not here explicated; rather, it remains implicit in the natural-language reading the
authors assign to the ‘mandator’ predicate. This attempt at modelling jumps straight
from an informal description of the concept of normative power to a first-order logic
representation – a transition that is presumably motivated primarily by considerations
of computational tractability.
        </p>
        <p>The practice of giving a rather simple, but computationally convenient,
representation of complex social concepts is quite widespread in Computer Science – the areas
discussed in sections 2.2 and 2.3, to follow, provide further examples of it. But it is a
problematic practice because it provides no clear picture of the nature of the
simplifications made, and thus also no proper framework for assessing whether a system
implemented on the basis of such a computational model behaves in a way that
adequately reflects the properties of the social concept itself.
2.2</p>
      </sec>
      <sec id="sec-2-2">
        <title>Role-based Access Control (summary only)</title>
        <p>
          The NIST model for role-based access control (RBAC) [
          <xref ref-type="bibr" rid="ref22">22</xref>
          ] formed the basis for the
ANSI RBAC standard.
        </p>
        <p>In their introductory section, the authors maintain that “….the basic role concept is
simple: establish permissions based on the functional roles in the enterprise, and then
appropriately assign users to a role or set of roles” [22, p.47]. But very soon thereafter
they allude to a structure that is considerably more complex: “Roles could represent
the tasks, responsibilities and qualifications associated with an enterprise”. It is
revealing that the latter description of roles is by no means confined to mere
permissions, since it appears that some key aspects of the overall NIST model are motivated
by the largely unexplicated assumption that agents get assigned to particular roles in
virtue of their qualifications, and that – as role-holders – they also acquire obligations
associated with the organizational tasks for which they are deemed to be responsible.
(Note also the remark: “A role is a job function or job title within the organization
with some associated semantics regarding the authority and responsibility conferred
on a member of a role” [22, p. 51].)</p>
        <p>We note the apparent lack of clarity and uniformity in these informal descriptions
of the role concept. Then, with reference to some of the issues that have arisen in the
further development of the RBAC approach, we argue that considerable advantage
could have been gained had those developments been informed and directed, from the
outset, by a comprehensive, precise model of the role concept itself.
2.3</p>
      </sec>
      <sec id="sec-2-3">
        <title>Trust (summary only)</title>
        <p>
          A third source of motivating examples is provided by the literature on the design of
socio-technical systems addressing issues of trust in agent interaction. A useful survey
of that literature has recently appeared [
          <xref ref-type="bibr" rid="ref19">19</xref>
          ], in which the authors present a
classification of a range of models in terms of several dimensions.
        </p>
        <p>We discuss aspects of their classification, and note with interest that a principal
conclusion they draw very strongly suggests the need for a methodology that brings
together both conceptual modelling and a computational framework informed by it.
Just one of the eighteen approaches considered in the survey achieves this synthesis,
according to the authors; concerning that one model they say, in their concluding
remarks, that it “…..summarizes one of the most prominent future research lines in
trust and reputation models: implementable cognitive models” [19, Section 5]. In our
view, the key point about those cognitive models is that they are conceptual models,
designed primarily to clarify the trust concept itself; non-cognitive analyses of trust
might also be possible, but the essential methodological requirement emerging from
the survey pertains to the need to integrate the conceptual and computational aspects.
3</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>Towards a Method for Designing Intelligent Socio-Technical</title>
    </sec>
    <sec id="sec-4">
      <title>Systems</title>
      <p>This section outlines the structure of an approach to engineering intelligent
sociotechnical systems in which an abstract analysis of social concepts informs the
development of a computational framework, providing a suitable platform for system
implementation.</p>
      <p>
        We are here, in part, building on the synthetic method underlying some research in
artificial societies and artificial life [
        <xref ref-type="bibr" rid="ref25">25</xref>
        ]. The main steps of the synthetic method
involve generalizing from some observations of phenomena to produce a theory, on the
basis of which an artificial system can be constructed and then used to test predictions
deriving from the theory. The outcome of applying the synthetic method is to
engineer an artificial system, with the resulting animation, experiments or performance
serving to support or refute the theory. Several other attempts to apply ideas from the
social sciences to the design of computational systems (see, e.g., [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]) have followed a
similar pattern. Furthermore, researchers in biologically-inspired computing, notably
those concerned with artificial immune systems [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ], have developed a comparable
approach.
3.1
      </p>
      <sec id="sec-4-1">
        <title>Structure of the method (summary only)</title>
        <p>The root of the concerns we highlighted in Section 2 may be expressed in the
following way: we fully accept that, in the design of socio-technical systems, the need for
computational tractability makes it probable that there will have to be some degree of
simplification of the principal social concepts involved; but in the interests of good
scientific practice – and thus, also, good engineering practice – it is essential to
achieve as clear a picture as possible of just what it is that is being simplified. We
need first to have a clear characterization of the phenomena, before we set about
simplifying them. Any computationally motivated simplifications should be carried out
against the background of, and should be properly informed by, precise models of the
social concepts themselves. And, crucially, the construction of those conceptual
models should not itself be constrained by considerations of computational tractability.</p>
        <p>We present our proposals in terms of different steps pertaining to the description
and analysis of the members of a set S of observed social phenomena, as illustrated in
Figure 1. The principal steps are theory construction, formal characterization, and
principled operationalisation.</p>
        <sec id="sec-4-1-1">
          <title>Step2</title>
          <p>Formal
Characterization</p>
        </sec>
      </sec>
      <sec id="sec-4-2">
        <title>Calculus1</title>
        <p>.</p>
        <sec id="sec-4-2-1">
          <title>Step3</title>
          <p>Principled
Operationalisation</p>
        </sec>
      </sec>
      <sec id="sec-4-3">
        <title>Pre-Formal</title>
      </sec>
      <sec id="sec-4-4">
        <title>Representation</title>
      </sec>
      <sec id="sec-4-5">
        <title>Step1</title>
        <p>Theory
Construction</p>
      </sec>
      <sec id="sec-4-6">
        <title>Observed Social</title>
      </sec>
      <sec id="sec-4-7">
        <title>Phenomena S . . .</title>
        <p>Step1 representations of the members of S are characterized by the natural-language
terms that are used to denote the social phenomena concerned – terms such as
empowerment, role, trust, and so on.</p>
        <p>The process of formal characterization is the process leading from Step1
representations to Step2 representations. A Step2 representation must be expressed in a formal
language or ‘calculus’ of some kind, where by ‘calculus’ we mean any system of
calculation or computation based on the manipulation of symbolic representations.</p>
        <p>However, there are Step2 representations of various sorts, which for our purposes
are appropriately divided into two sub-steps, or phases. Step2-Phase1 representations
define a conceptual framework for the phenomena in S, in which conceptual analyses
are expressed in terms of, for instance, a formal-logical language; the key point about
Step2-Phase1 representations is that they aim to provide an analysis of conceptual
structure, identifying the fundamental elements of which complex concepts are
composed, and articulating the principles governing their composition and inter-relations.
1 The diagram is simplified in that it depicts the method as uni-directional. However, there are
important aspects of two-way interplay between the key steps. We describe some of these in
Section 3.3 of the full paper.</p>
        <p>Crucially, Step2-Phase1 representations are constrained primarily by considerations
of expressive capacity, not those of computational tractability.</p>
        <p>By contrast, it is at the Step2-Phase2 stage that issues of computational tractability
begin to come into play. A Step2-Phase2 computational framework models the
conceptual framework of Step2-Phase1 in terms of a language, or languages, that are
themselves amenable to the development of software implementations; the key points
to note about Step2-Phase2 computational frameworks are that the principles
governing their composition are informed and guided by the conceptual characterizations of
Step2-Phase1, but that they may well involve some degree of simplification, or
approximation. Crucially, however, on this approach the designer of a computational
framework will have a very clear picture, from Step2-Phase1, of the nature of the
simplifications or approximations that may have been made.</p>
        <p>Step2-Phase1 representations are essentially theory-facing, whereas Step2-Phase2
representations are essentially implementation-facing. The recommendation to adopt
two Step2 phases is motivated by the need to guard against trying to force subtle
societal concepts into the straitjacket of some particular computationally tractable
language.</p>
        <p>One further observation should be made about the relationship between the two
phases of Step2. We have emphasized that some of the conceptual detail that is
captured in the Step2-Phase1 model might be omitted from the Step2-Phase2
computational framework; but we should also point out that there may well be abstractions
that can be tolerated at the Step2-Phase1 level that cannot be ignored in an
implementation-facing framework, for example the representation of time, and of the means by
which a particular state of affairs is to be brought about.</p>
        <p>Step 3 representations are exemplified not so much by formalisms but by tools that
are employed in moving from the computational framework to a model of the
artificial system, with algorithmic intelligence of the agents embedded in identifiable
system processes. This is the transition that we call principled operationalisation.
Operationalisation may well be selective, vis-à-vis the computational framework; but it is
principled operationalisation in that it is conducted in the full knowledge of which
selections have been made, and why.
3.2
3.2.1</p>
      </sec>
      <sec id="sec-4-8">
        <title>Step2 Exemplified (Summary only)</title>
        <sec id="sec-4-8-1">
          <title>Step2-Phase1: Formal Characterization using Modal Logic</title>
          <p>The focus is on some formal-logical tools, drawn from modal logic, that have been
used in the analysis of the group of social concepts that were discussed in Section 2.
(This choice is of course not intended to suggest that modal logic is the only tool
suited to the formal analysis of social concepts.)
3.2.2</p>
        </sec>
        <sec id="sec-4-8-2">
          <title>Step2-Phase2: Formal Characterization with Action Languages</title>
          <p>
            Suitable examples come from research in AI on action languages: the Situation
Calculus [
            <xref ref-type="bibr" rid="ref14 ref16 ref18 ref21">18,16,21,14</xref>
            ]; the Event Calculus ([
            <xref ref-type="bibr" rid="ref13">13</xref>
            ]); C+, an action language with transition
system semantics [
            <xref ref-type="bibr" rid="ref1 ref7">7,1</xref>
            ] - Artikis et al. [
            <xref ref-type="bibr" rid="ref3">3</xref>
            ] have used this language to develop
executable MAS specifications in terms of institutionalized power, permission and sanction.
Other relevant references here include [
            <xref ref-type="bibr" rid="ref4">4</xref>
            ], [
            <xref ref-type="bibr" rid="ref23">23</xref>
            ] and [
            <xref ref-type="bibr" rid="ref24">24</xref>
            ]. In general, Sergot’s
framework was informed by abstract conceptual models of normative systems.
3.3
          </p>
        </sec>
      </sec>
      <sec id="sec-4-9">
        <title>A Note on the Interplay between Steps (Summary only)</title>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>Adequacy Criteria for Step2 (Summary only)</title>
      <sec id="sec-5-1">
        <title>Adequacy Criteria for Step2-Phase1</title>
        <p>The principal criterion pertains to expressive capacity. This sub-section discusses and
illustrates various aspects of this requirement.
4.2</p>
      </sec>
      <sec id="sec-5-2">
        <title>Adequacy Criteria for Step2-Phase2</title>
        <p>The key criteria discussed and illustrated in this sub-section are: a formal semantics; a
declarative semantics; expressive capacity; support for computational tasks; efficient
execution.</p>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Akman</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Erdogan</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lee</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lifschitz</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Turner</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          (
          <year>2004</year>
          ),
          <article-title>“Representing the Zoo World and the Traffic World in the language of the Causal Calculator,”</article-title>
          <source>Artificial Intelligence</source>
          ,
          <volume>153</volume>
          (
          <issue>1-2</issue>
          ),
          <fpage>105</fpage>
          -
          <lpage>140</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Andrews</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Polack</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sampson</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Stepney</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          , and Timmis. (
          <year>2010</year>
          ), “
          <article-title>The CoSMoS Process, Version 0.1: A Process for the Modelling</article-title>
          and
          <source>Simulation of Complex Systems,” echnical Report YS-2010-453</source>
          , University of York.
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Artikis</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sergot</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Pitt</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          (
          <year>2007</year>
          ), “
          <article-title>Executable Specification of a Formal Argumentation Protocol,”</article-title>
          <source>Artificial Intelligence</source>
          ,
          <volume>171</volume>
          (
          <fpage>10</fpage>
          -
          <lpage>15</lpage>
          ),
          <fpage>776</fpage>
          -
          <lpage>804</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Craven</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Sergot</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          (
          <year>2008</year>
          ), “
          <article-title>Agent strands in the action language nC+</article-title>
          ,
          <source>” Journal of Applied Logic</source>
          ,
          <volume>6</volume>
          (
          <issue>2</issue>
          ),
          <fpage>172</fpage>
          -
          <lpage>191</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Dennett</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          (
          <year>1995</year>
          ),
          <article-title>Darwin's Dangerous Idea: Evolution and the Meanings of Life</article-title>
          , London: Penguin Books.
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Edmonds</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gilbert</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gustafson</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hales</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Krasnogor</surname>
          </string-name>
          , N. (eds.),
          <source>Socially Inspired Computing. Proceedings of the Joint Symposium on Socially Inspired Computing</source>
          ,
          <string-name>
            <surname>AISB</surname>
          </string-name>
          (
          <year>2005</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Giunchiglia</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lee</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lifschitz</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>McCain</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Turner</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          (
          <year>2004</year>
          ), “Nonmonotonic causal theories,
          <source>” Artificial Intelligence</source>
          ,
          <volume>153</volume>
          (
          <issue>1-2</issue>
          ),
          <fpage>49</fpage>
          -
          <lpage>104</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Hohfeld</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          (
          <year>1913</year>
          ),
          <article-title>“Some Fundamental Legal Conceptions as Applied in Judicial Reasoning</article-title>
          ,” Yale
          <source>Law Journal</source>
          ,
          <volume>23</volume>
          (
          <issue>16</issue>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Jones</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Sergot</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          (
          <year>1993</year>
          ), “
          <article-title>On the Characterization of Law and Computer Systems: The Normative Systems Perspective</article-title>
          ,” in Deontic Logic in Computer Science eds.
          <source>J</source>
          .-J. Meyer and
          <string-name>
            <given-names>R.</given-names>
            <surname>Wieringa</surname>
          </string-name>
          , Chichester: John Wiley and Sons.
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Jones</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Sergot</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          (
          <year>1996</year>
          ),
          <article-title>“A formal characterization of institutionalised power</article-title>
          ,
          <source>” Journal of the IGPL</source>
          ,
          <volume>4</volume>
          (
          <issue>3</issue>
          ),
          <fpage>429</fpage>
          -
          <lpage>445</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Kanger</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          (
          <year>1957</year>
          ),
          <article-title>New Foundations for Ethical Theory</article-title>
          , University of Stockholm, Department of Philosophy. Also in R. Hilpinen (ed.),
          <source>Deontic Logic: Introductory and Systematic Readings</source>
          , Dordrecht: Reidel,
          <year>1971</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Kanger</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Kanger</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          (
          <year>1966</year>
          ), “Rights and Parliamentarism,” Theoria 32,
          <fpage>85</fpage>
          -
          <lpage>115</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Kowalski</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Sergot</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          (
          <year>1986</year>
          ),
          <article-title>“A Logic-Based Calculus of Events</article-title>
          ,” New Generation Computing,
          <volume>4</volume>
          (
          <issue>1</issue>
          ),
          <fpage>67</fpage>
          -
          <lpage>96</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Levesque</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pirri</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Reiter</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>1998</year>
          ), “
          <article-title>Foundations for the Situation Calculus,” Linköping Electronic Articles in Computer</article-title>
          and Information Science,
          <volume>3</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Lindahl</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          (
          <year>1977</year>
          ),
          <article-title>Position and Change: A Study in Law and Logic</article-title>
          , Dordrecht: reidel.
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>McCarthy</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          (
          <year>1963</year>
          ),
          <article-title>“A Basis for a Mathematical Theory of Computation,” in Computer Programming</article-title>
          and
          <string-name>
            <given-names>Formal</given-names>
            <surname>Systems</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Braffort</surname>
          </string-name>
          and
          <string-name>
            <given-names>D.</given-names>
            <surname>Hirschberg</surname>
          </string-name>
          , (eds.), Amsterdam: North-Holland,
          <fpage>33</fpage>
          -
          <lpage>70</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Oren</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Luck</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Miles</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          (
          <year>2010</year>
          ),
          <article-title>“A Model of Normative Power,”</article-title>
          <source>in Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems (AAMAS)</source>
          ,
          <fpage>815</fpage>
          -
          <lpage>822</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>Pinto</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Reiter</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>1993</year>
          ), “
          <article-title>Temporal Reasoning in Logic Programming: a case for the Situation Calculus</article-title>
          ,”
          <source>in Proceedings of Conference on Logic Programming</source>
          ,
          <string-name>
            <surname>D. Warren</surname>
          </string-name>
          , (ed.), Cambridge, Mass.: MIT Press,
          <fpage>203</fpage>
          -
          <lpage>221</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>Pinyol</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Sabater-Mir</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          (
          <year>2011</year>
          ), “
          <article-title>Computational trust and reputation models for open multi-agent systems: a review</article-title>
          ,
          <source>” Artificial Intelligence Review</source>
          (Springer Online-First).
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>Pörn</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          (
          <year>1970</year>
          ),
          <source>The Logic of Power</source>
          , Oxford: Blackwell.
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          21.
          <string-name>
            <surname>Reiter</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>1993</year>
          ), “
          <article-title>Proving Properties of States in the Situation Calculus,”</article-title>
          <source>Artificial Intelligence</source>
          ,
          <volume>64</volume>
          ,
          <fpage>337</fpage>
          -
          <lpage>351</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          22.
          <string-name>
            <surname>Sandhu</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Ferraiolo</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Kuhn</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>2000</year>
          ), “
          <article-title>The NIST Model for Role-Based Action Control: Toward a Unified Standard,”</article-title>
          <source>in the 5th ACM Workshop on Role-Based Access Control</source>
          , RAC '
          <volume>00</volume>
          ,
          <fpage>47</fpage>
          -
          <lpage>63</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          23.
          <string-name>
            <surname>Sergot</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          (
          <year>2008</year>
          ), “
          <article-title>Action and Agency in Norm-Governed Multi-Agent Systems,” in Proceedings of ESAW VIII, A</article-title>
          . Artikis,
          <string-name>
            <given-names>G. O</given-names>
            <surname>'Hare</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Stathis</surname>
          </string-name>
          and G. Vouros, (eds.),
          <source>LNAI 4995</source>
          , Springer,
          <fpage>1</fpage>
          -
          <lpage>54</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          24.
          <string-name>
            <surname>Sergot</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Craven</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>2006</year>
          ), “
          <article-title>The Deontic Component of Action Language nC+,” in Deontic Logic in Computer Science</article-title>
          (DEON'06),
          <string-name>
            <given-names>L.</given-names>
            <surname>Goble</surname>
          </string-name>
          and J.-J. Meyer, (eds.),
          <source>LNAI 4048</source>
          , Springer,
          <fpage>222</fpage>
          -
          <lpage>237</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          25.
          <string-name>
            <surname>Steels</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          , and
          <string-name>
            <surname>Brooks</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          (
          <year>1994</year>
          ),
          <source>The Artificial Life Route to Artificial Intelligence: Building Situated Embodied Agents</source>
          , New Haven: Lawrence Erlbaum Ass.
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>