<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>A Roadmap for Self-Evolving Communities</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Nardine Osman</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Carles Sierra</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Arti cial Intelligence Research Institute (IIIA-CSIC)</institution>
          ,
          <addr-line>Bellaterra, Barcelona</addr-line>
          ,
          <country country="ES">Spain</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>Self-organisation and self-evolution is evident in physics, chemistry, biology, and human societies. Despite the existing literature on the topic, we believe self-organisation and self-evolution is still missing from the IT tools (whether online or o ine) we are building and using. In the last decade, human interactions have been moving more and more towards social media. The time we spend interacting with others in virtual communities and networks is tremendous. Yet, the tools supporting those interactions remain rigid. This position paper argues the need for self-evolving software-enabled communities, and proposes a roadmap for achieving this required self-evolution. The proposal is based on building normative-based communities, where community interactions are regulated by norms and community members are free to discuss and modify their community's norms. The evolution of communities is then dictated by the evolution of its norms.</p>
      </abstract>
      <kwd-group>
        <kwd>norms</kwd>
        <kwd>interaction protocols</kwd>
        <kwd>self-evolution</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>The concept of social self-organisation has deep roots in political theory. In
the Leviathan, Hobbes expresses the idea that the social structure, the State
or res publica, is the result of a social contract or pact. The social contract is
an agreement among free individuals in order to surrender power to a central
authority in order to get a guarantee of peace. In essence, a social contract is
the implicit acceptance of self-limiting individual freedom in order to guarantee
self-preservation and a better life. The concept of social contract implies that
individuals re ect on the community governance in order to decide on the means
to reach commonwealth.</p>
      <p>This capacity of individuals to re ect and decide on community governance
is missing in the IT tools we are currently building. How individuals interact
is decided by groups of engineers that design tools, determine the individuals'
privacy levels, hinder some from performing certain tasks, and all this without
the individuals having a say. We argue that it is time for us to take control of
the evolution of our own virtual communities, which we refer to as
softwareenabled communities | examples of such communities are those based on social
networks or any other community whose management relies on software, such as
customer relationship management systems (CRMs) or conference management
systems. To address this issue, we propose normative-based communities, where
community interactions are regulated by norms and community members are
free to discuss and modify their community's norms (or social contracts), while
respecting their free will. The roadmap for self-evolution is then dictaded by
the lifecycle of norms, and the research required for supporting this lifecycle is
varied and multidisciplinary | from social sciences and legal studies, to learning
mechanisms, agreement technologies, natural language processing, formal logics,
norm enforcement and regimentation, human computer interaction, and other
software engineering techniques.</p>
      <p>
        It is true that the notion of self-organisation is not new. Self-organisation,
in general, refers to how a system reorganises to adapt to changes to its goals
or environment, where the reorganisation is driven from within the system as
opposed to some external control. In many cases, self-organisation is associated
with emergence, where a structure appears at a higher level without it being
represented at a lower level. Self-organisation has been studied in a number
of areas, such as biology, chemistry, geology, sociology, as well as information
technology [
        <xref ref-type="bibr" rid="ref26">26</xref>
        ]. In AI, self-organisation has mostly been inspired by naturally
existing self-organisation models, although some new mechanisms have been
designed speci cally for software applications [
        <xref ref-type="bibr" rid="ref27">27</xref>
        ]. For example, self-organisation
may be found in bio-inspired robot teams, where insect-based technology is used
to aid robots self-organise to accomplish a task [
        <xref ref-type="bibr" rid="ref44">44</xref>
        ]. Another example is
intrusion detection in network security, which is inspired from the natural immune
system [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ]. Swarm intelligence may be used to mimic insect foraging behaviour
on the coordination and control of data network tra c [
        <xref ref-type="bibr" rid="ref48">48</xref>
        ]. With the notions of
self-adaptability and long-lasting evolvability in mind, it has also been proposed
that natural ecosystems can inspire building ecosystems of services [
        <xref ref-type="bibr" rid="ref47">47</xref>
        ].
      </p>
      <p>In this paper, however, we do not propose to follow traditional self-organisation
AI techniques, but to open a new direction in AI research with a novel method for
self-evolving software-enables communities. The proposed research is original in
the sense that we do not simply talk about self-organising software that usually
imitates existing natural systems, but we study how the human users' evolving
needs consciously direct evolution. We essentially propose to provide the human
users with learning mechanisms that help them learn the best evolution path,
along with agreement technologies that help them discuss, argue, and agree on
their preferred path of evolution. The system then takes care of the
formalisation, operationalisation, and enforcement of the agreed upon norms. This is a
fundamentally di erent approach from existing self-organising systems.
2</p>
    </sec>
    <sec id="sec-2">
      <title>Roadmap for Self-Evolving Communities</title>
      <p>In this paper, we argue that it is time for us to take control of the evolution
of our own virtual communities. To address this issue, we propose
normativebased communities, where community interactions are regulated by norms and
community members are free to discuss and modify their community's norms.</p>
      <p>
        Norms are the rules that govern behaviour in groups and societies [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]. They
motivate and in uence individual actions by dictating what values, beliefs,
attitudes, and behaviours are deemed appropriate or not. Social norms have been
extensively studied by anthropologists, sociologists, philosophers, and economists
in the hope of understanding how they motivate individual actions, in uence
market behaviour, and so on. In multiagent systems, the study of norms gained
tremendous attention due to the critical issue of coordinating agent behaviour
and actions. Although, unlike other social sciences, the distinction between social
and legal norms has not been concrete in the eld of computer science.
      </p>
      <p>We adopt the idea of using norms to control, or mediate, community
behaviour. This is motivated by the fact that software is usually engineered based
on some notion of norms in mind; furthermore, the use of norms allows human
users to discuss their interactions without the need for any technical knowledge
about the software actually mediating their interactions. We note that by
specifying a community through its norms, the evolution of the community becomes
dictated by the evolution of its norms. As such, the evolution of software-enabled
communities may be depicted by the lifecycle of norms, as illustrated by Figure 1.
! coBmimrthunoitfyNmoermmbse:rs
collectively decide
on their norms</p>
      <p>Norms
Community
Interactions</p>
      <p>Automated
" Formalisation of Norms:
norms are translated from natural
language to a formal specification
$ EnforcAeumteonmtaotefdNorms:
software ensures community
interactions follow the norms</p>
      <p>Formalised</p>
      <p>Norms
Software
# Operationalisation of Norms:</p>
      <p>Automated
software is modified
to incorporate the norms</p>
      <p>We divide the lifecycle of norms in a software-enabled community into four
di erent stages. In the rst stage, community members discuss and agree on their
community's norms. If community members do agree on a new set of norms, then
the norms' lifecycle enters its second stage, where the agreed upon norms are
automatically translated into some formal speci cation. Given the formal
speci cation of norms, the lifecycle then enters the third stage, where the
softwareenabled community interactions are modi ed to operationalise the new norms.
Finally, as the software runs to mediate community interactions, norms are
enforced by ensuring that community interactions follow the agreed upon norms.
The details of these four stages are presented next.</p>
      <sec id="sec-2-1">
        <title>Birth of Norms</title>
        <p>In this stage, community members propose norm modi cations (including the
abandoning and adoption of norms), discuss the di erent proposals amongst
themselves, and collectively agree on their community's new (or modi ed) set of
norms.</p>
        <p>An interdisciplinary research is highly recommended at this stage. For
example, social sciences can help understand the emergence of norms, their social
function, their impact on behviour, as well as understand the possible deviance
from norms. Legal studies can help understand the legal aspects of norms,
such as the enforcement of norms, the e ectivity and validity of norms, and
the resolve of con icting norms. Learning mechanisms can help learn from
a community's past experience, or a similar community's past experience, and
suggest norm modi cations accordingly. Finally, to aid the collective agreement
on community norms, agreement technologies will be needed.</p>
        <p>
          Concerning learning mechanisms, learning from data has been one of the main
sub elds of arti cial intelligence [
          <xref ref-type="bibr" rid="ref2">2</xref>
          ], yet the literature lacks the in-depth study of
learning the best norms for multiagent interactions. In multiagent systems, some
work has been carried out on norm synthesis [
          <xref ref-type="bibr" rid="ref40">40</xref>
          ]. Most norm synthesis (online)
approaches, such as [
          <xref ref-type="bibr" rid="ref23 ref37 ref38 ref46">38, 46, 37, 23</xref>
          ], are based on norm emergence techniques that
require agents to collaborate to synthesise their own norms. [
          <xref ref-type="bibr" rid="ref29">29</xref>
          ] does not require
the active participation of agents in norm synthesis as norms are synthesised
from the observation of agents' interactions. The norm synthesis of [
          <xref ref-type="bibr" rid="ref30">30</xref>
          ] is based
on the capability of generalising norms, which allows characterising necessary
conditions for coordination, avoiding over-regulation.
        </p>
        <p>Concerning agreement technologies, we note that this eld is one of the
vibrant and active elds in multiagent systems. Agreement technologies aim at
helping individuals collaboratively reach a decision, which is crucial when
individuals are intelligent and autonomous, having their own goals and agendas to
ful l. The eld is based on a number of models and mechanisms, such as
argumentation and negotiation mechanisms, social choice theory and voting
mechanisms, and trust and reputation models.</p>
        <p>
          Argumentation theory uses logical reasoning to illustrate how conclusions
may be drawn in cases of uncertainty and con ict. Argumentation frameworks
in multiagent systems [
          <xref ref-type="bibr" rid="ref32">32</xref>
          ] have mostly been built on Dung's theory of abstract
argumentation [
          <xref ref-type="bibr" rid="ref17">17</xref>
          ], where a directed graph is used to represent arguments (the
nodes of the graph) and the attack relations between them (the directed links
of the graph). A `calculus of opposition' is then applied to help determine the
winning argument. Dung's framework has been extended in a number of
directions, such as adding preferences [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ], allowing attacks on attacks [
          <xref ref-type="bibr" rid="ref7">7</xref>
          ], and adding
weights to attacks [
          <xref ref-type="bibr" rid="ref18">18</xref>
          ]. One interesting extension to Dungs framework is one
that incorporates social voting and is used in social networks [
          <xref ref-type="bibr" rid="ref11 ref25">25, 11</xref>
          ].
        </p>
        <p>
          Social choice theory is concerned with the representation and aggregation of
individual preferences. Voting mechanisms and ranking systems are considered
to fall under the social choice category. Social choice theory studies a variety of
aspects, such as the manipulation of a voting rule [
          <xref ref-type="bibr" rid="ref10">10</xref>
          ], the use of combinatorial
domains where voters may specify a list of preferences [
          <xref ref-type="bibr" rid="ref9">9</xref>
          ], and the e ciency and
fairness of voting rules [
          <xref ref-type="bibr" rid="ref19">19</xref>
          ].
        </p>
        <p>
          A common issue that arises in building open distributed systems is the issue
of trust and reputation. Trust has become an increasingly important concept in
computer science [
          <xref ref-type="bibr" rid="ref22 ref5">5, 22</xref>
          ]. For example, there have been studies on the
development of trust in e-commerce [
          <xref ref-type="bibr" rid="ref34">34</xref>
          ], on mechanisms to determine which sources to
trust when faced with multiple con icting information sources [
          <xref ref-type="bibr" rid="ref54">54</xref>
          ], and
mechanisms for identifying which individuals to trust based on their past activity
[
          <xref ref-type="bibr" rid="ref1">1</xref>
          ]. Trust is an especially important issue from the perspective of autonomous
agents and multiagent systems [
          <xref ref-type="bibr" rid="ref42">42</xref>
          ]. In multiagent interactions, agents will have
to reason about how much should they trust other entities (in various contexts).
For reviews on trust in multiagent systems, we refer the reader to [
          <xref ref-type="bibr" rid="ref33 ref36">33, 36</xref>
          ].
        </p>
        <p>
          Roadmap: In the context of self-evolving communities, research in the
information and computer Science eld should focus on two main issues. (1) We
should aim at building a system that is capable of automatically suggesting norms
that can help improve the quality of interactions in a community, based on
learning from similar past experiences. Improving the quality of interactions may refer
to a number of issues, such as improving the individual user satisfaction, or
improving the e ciency of the interactions by helping community members achieve
their goals faster and easier. We note that learning mechanisms may also be
combined with other techniques, such as analogical reasoning [
          <xref ref-type="bibr" rid="ref45">45</xref>
          ] or coherence
theory [
          <xref ref-type="bibr" rid="ref43">43</xref>
          ]. (2) We should investigate the combination of di erent agreement
technologies to help community members agree on how should their community
evolve. For example, it would be interesting to study the combination of trust
and argumentation mechanisms, for instance, by modifying the reasoning
process to consider reputation/trust based measures of the strengths of arguments.
It would also be interesting to study voting algorithms mediated by trust, with
votes potentially weighted by the trustworthiness or degree of involvement in the
community. The outcome of this research work would be to aid community
members in agreeing on their community norms.
2.2
        </p>
      </sec>
      <sec id="sec-2-2">
        <title>Automated Formalisation of Norms</title>
        <p>Norms agreed upon by community members need to be formalised in a language
that can be understood by the system mediating community interactions. We
expect community members to be non-technical people who discuss and specify
norms in natural language. An automatic translator is then needed to translate
the norms from their natural language speci cation to some formal speci cation
that is comprehensible by the system.</p>
        <p>The research that should support this stage should focus on natural
language processing, which will help perform the automatic translation, and
formal logics, which will de ne the formal language used to specify norms.</p>
        <p>
          Concerning natural language processing, machine translation [
          <xref ref-type="bibr" rid="ref49">49</xref>
          ] is a
subeld that translates one natural language into another. This sub- eld has been
well studied and it has been even used in commercial applications, such as Google
Translate or Yahoo's Babel Fish. The translation from a natural language to
some logic, on the other hand, has attracted much less attention in the natural
language processing eld. Nevertheless, important research has been carried in
that direction. [
          <xref ref-type="bibr" rid="ref50 ref52">50, 52</xref>
          ] illustrates and discusses the use of existing state-of-the-art
in the automatic translation of regulatory rules in natural language into a
machine readable formal representation. [
          <xref ref-type="bibr" rid="ref39">39</xref>
          ] translates a complete set of pediatric
guideline recommendations into a controlled language (Attempto Controlled
English { ACE). [
          <xref ref-type="bibr" rid="ref51">51</xref>
          ] adopts and applies a controlled natural language to constrain
the domain of discourse in a sample discussion from an on-line discussion forums
for e-government policy-making. The controlled natural language helps eliminate
ambiguity and unclarity, and allows a logical representation of statements. Each
of the policy statements is then automatically translated into rst-order logic.
[
          <xref ref-type="bibr" rid="ref53">53</xref>
          ] presents a linguistically-oriented, rule-based approach, for extracting
conditional and deontic rules from regulations speci ed in natural language. Finally,
[
          <xref ref-type="bibr" rid="ref4 ref6">4, 6</xref>
          ] present approaches for the logical representation of regulations.
        </p>
        <p>Concerning the formal speci cation of norms, existing approaches may be
divided into two main categories: declarative approaches and procedural ones. We
note that in this paper, we refer to the declarative approaches as norms and the
procedural ones as interaction models. Declarative formalisms are concerned with
the expressiveness of norms, the formal semantics, and how to resolve con icts
arising from an inconsistent set of norms. Declarative approaches are usually
based on deontic logic, which is the logic of duties. They deal with concepts like
permissions, prohibitions, and obligations, which help specify who can do what,
under what conditions, and so on.</p>
        <p>
          Deontic-based policy languages have been used widely in hardware systems
and networks for security reasons, trust negotiation, access control,
authorisation, and so on. [
          <xref ref-type="bibr" rid="ref41">41</xref>
          ] de nes policies to be \one aspect of information which in
uences the behaviour of objects within the system". [
          <xref ref-type="bibr" rid="ref13">13</xref>
          ] categorises policies into
two types. The rst type covers obligation policies for managing actions, which
are usually event triggered condition-action rules. The basic concept is that
speci c events trigger speci c actions, and the actions may only be executed if a
prede ned set of conditions is satis ed. The second type covers authorisation
policies, which are usually used for access control.
        </p>
        <p>
          In multiagent systems, several deontic based formal logics have been proposed
for de ning a normative speci cation of agents interaction, such as [
          <xref ref-type="bibr" rid="ref31">31</xref>
          ] and [
          <xref ref-type="bibr" rid="ref14">14</xref>
          ].
In [
          <xref ref-type="bibr" rid="ref31">31</xref>
          ], it is proposed that a community should be de ned whenever a group
of agents have some common goals and they need to act as a whole within the
society in order to ful l those goals. They propose to de ne the roles within the
community, the relationships among them, which actions each role can do, and
how the obligations are distributed among the roles. Each role has associated
deontic notions that describe the role obligations, permissions and prohibitions.
[
          <xref ref-type="bibr" rid="ref14">14</xref>
          ] extends the BDI model of agents to include goals, obligations, and norms;
the proposal is based on providing a formal de nitions of norms by means of
some variation of deontic logic that includes conditional and temporal aspects.
        </p>
        <p>Roadmap: In the context of self-evolving communities, research should focus
on two main issues. (1) We should aim at building an automatic translator that
can translate norms speci ed in a natural language into some formal logic. (2)
We should select, or design, a logic that ful lls the following three requirements:
i) it should be expressive enough to capture the community members' needs; ii)
it should be compatible with the natural language processing technique that needs
to translate norms speci ed in natural language into the logic; and iii) it should
be compatible with the software system that needs to operationalise those norms.
The outcome of this research work would be to have the agreed upon norms
translated into some formal logic that is ready to be operationalised by the system.
2.3</p>
      </sec>
      <sec id="sec-2-3">
        <title>Automated Operationalisation of Norms</title>
        <p>Given a formal speci cation of norms, the ultimate goal is to enforce those norms.
One approach to achieve this is through what is known as norm
regimentation. In other words, the system mediating community interactions needs to be
modi ed in order to operationalise those norms.</p>
        <p>
          The literature provides a variety of solutions that deal with specifying and
regulating interactions in multiagent systems based on the concept of following
social norms [
          <xref ref-type="bibr" rid="ref40">40</xref>
          ], such as having contracts and commitments [
          <xref ref-type="bibr" rid="ref15">15</xref>
          ], organisational
approaches [
          <xref ref-type="bibr" rid="ref24">24</xref>
          ], electronic institutions [
          <xref ref-type="bibr" rid="ref16">16</xref>
          ], distributed dialogues [
          <xref ref-type="bibr" rid="ref35">35</xref>
          ], and so on.
        </p>
        <p>
          Two speci cally interesting approaches are electronic institutions [
          <xref ref-type="bibr" rid="ref16">16</xref>
          ] and
the lightweight coordination calculus [
          <xref ref-type="bibr" rid="ref35">35</xref>
          ]. In [
          <xref ref-type="bibr" rid="ref16">16</xref>
          ], it is argued that open
multiagent systems can be e ectively designed and implemented as agent mediated
electronic institutions where heterogeneous (human and software) agents can
participate, playing di erent roles and interacting by means of speech acts. An
institution is de ned by a set of roles that agents participating in the
institution will play, a common language to allow heterogenous agents to exchange
knowledge, the valid interactions that agents may have structured in
conversations, and the consequences of agents' actions within an institution, captured by
obligations that agents acquire and ful l.
        </p>
        <p>
          The lightweight coordination calculus (LCC) [
          <xref ref-type="bibr" rid="ref35">35</xref>
          ] is a process calculus, based
on logic programming, that provides means of achieving coordination in
distributed systems by enforcing social norms. The process calculus speci es what
actions agents can perform, when they can perform such actions, under what
conditions these actions may be carried out, and so on. However, unlike
electronic institutions, there are no `governors' that ensure that agents abide by
norms. Of course, like all the approaches above, these rules are associated with
roles rather than physical agents; and agents can play more than one role in more
than one interaction. This provides an abstraction for the interaction model from
the individual agents that might engage in such an interaction.
        </p>
        <p>
          In the context of self-evolution, it is important to build a system whose
interaction models can operationalise norms requested by its uers. We note,
however, that not all norms are capable of being operationalised. For example, in
an e-commerece context, the system can operationalise the norm that states that
a buyer cannot rate the seller more than once, as the system can actually prevent
the buyer from doing so. However, the system cannot operationalise a norm
that states that only people with su cient credit can bid, as credit is private
information that cannot be accessed and assessed by the system. To deal with
norms that cannot be operationalised, alternative norm enforcement methods
will need to be applied, such as applying sanctions (punishments and rewards).12
In this case, the system will need to ensure the enforcement of un-operationalised
norms by checking the members' abidance to those norms and enforcing the
appropriate sanctions accordingly. To achieve this, it is important to design a
lightweight computational norm enforcement model that allows for the detection
of norm violations and the application of remedial actions. This work may build
upon existing work, as the literature is rich with norm engines that may be used
for resolving con icting norms and applying appropriate sanctions [
          <xref ref-type="bibr" rid="ref12 ref21 ref28">21, 28, 12</xref>
          ].
        </p>
        <p>Roadmap: In summary, in the context of self-evolution, research should
focus on two main issues: (1) building a mechanism that allows the system
to operationalise the norms agreed upon by the community members; and (2)
building a norm enforcement model that can check members' abidance to
unoperationalised norms and enforce appropriate sanctions accordingly. The
outcome of this research work would be to have a system that is capable of enforcing
the norms agreed upon by the community members.
2.4</p>
      </sec>
      <sec id="sec-2-4">
        <title>Automated Enforcement of Norms</title>
        <p>Finally, running the software that mediates community interactions will allow
members of the community to interact, while ensuring the enforcement of the
norms. The research that will support this stage will need to focus on human
computer interaction techniques that are required to ensure an intuitive,
user friendly interaction for non-technical community members. Research will
also need to focus on software engineering techniques to ensure e cient
community interactions. However, the context of self-evolution imposes an
additional and critical challenge: how can the system be modi ed at runtime when
di erent members may still be interacting together? Indeed, it is very
unrealistic to assume that all members will need to complete their current interactions
successfully and pause any future interactions for the system to get modi ed
(as future interactions will need to follow the new norms). As such, research
will need to address the issue of seamlessly adapting the system at runtime, and
it may take inspiration from existing software compatibility models that allow
di erent versions of an application to continue to communicate and collaborate.</p>
        <p>Roadmap: In summary, in the context of self-evolution, research should
focus on two main issues: (1) building a user friendly interface that would allow
1 The literature usually refers to the operationalisation of norms as `norm
regimentation', and alternative norm enforcement methods (such as the application of
sanctions) as `norm enforcement'. In this paper, however, we overload the word `enforce'
as we use it to describe both approaches.
2 It may be argued that punishment and reward is not always the right approach for
motivating the abidance to norms. Furthermore, what may be considered a
punishment for one may be viewed as a reward for another. In this paper, although we
talk about punishments and rewards, we concur that sanctions may be labeled more
generally as the post-conditions of abiding with or breaking a norm.
community member to interact, following the norms, without any knowledge of
their interactions and their complexity, and in cases when users need to discuss
norms, norms and norm violations will need to be visualised in a user friendly
and comprehensible manner; and (2) engineering the system in such a way that
ensures e cient interactions and permits evolution at runtime in a seamless
manner that does not interrupt community interactions. The outcome of this
research work would be to have the system run and manage evolving
normativebased community interactions in an e cient, userfriendly, and seamless manner.
3</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>Conclusion</title>
      <p>We argue that just like human communities, e-communities (or virtual
communities) also need to self-evolve. Instead of creating numerous rigid systems, which is
very common in online social systems, what we should aim at instead is providing
tools for creating self-evolving systems that adapt to the community's needs. We
believe di erent communities should be governed by di erent rules. These rules
should be an ever evolving set resulting from the aspirations of its members, and
not a rigid set de ned by a corporation (or a system designer) that does not take
the community members' interests at heart. Furthermore, for the community's
rules to be e ective, they need to be tailored to the speci c community, such as
considering the character traits of the community's members.</p>
      <p>We believe that the notion of self-evolving communities will revolutionise the
way how software is build, as well as how we interact with software. We say
that in the context of social interactions, software does not need to be rigid, nor
customised for the individual, but adaptable to the group. Software engineers will
need to design intelligent software that is capable of evolving according to the
needs of the community as a whole, and users (or community members) should
become more active in shaping their community by discussing and agreeing on
their community rules.</p>
      <p>In summary, the proposal is based on the traditional notion of mediating
social interactions via norms. The roadmap for self-evolution is then dictaded by
the lifecycle of norms, which we divide into four di erent stages (see Figure 1):
(1) the birth of norms, (2) the automated formalisation of norms, (3) the
automated operationalisation of norms, and (4) the automated enforcement of norms.
Varied, and even interdisciplinary, research will be required for supporting these
four stages, from social sciences and legal studies, to learning mechanisms,
agreement technologies, natural language processing, formal logics, norm enforcement
and regimentation, human computer interaction, and other software engineering
techniques.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Adler</surname>
          </string-name>
          , B.T.,
          <string-name>
            <surname>de Alfaro</surname>
          </string-name>
          , L.:
          <article-title>A content-driven reputation system for the Wikipedia</article-title>
          .
          <source>In: Proceedings of the 16th International World Wide Web Conference</source>
          . Ban , Alberta, Canada (May
          <year>2007</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Alpaydin</surname>
          </string-name>
          , E.:
          <article-title>Introduction to machine learning</article-title>
          . MIT press (
          <year>2004</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Amgoud</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cayrol</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>A reasoning model based on the production of acceptable arguments</article-title>
          .
          <source>Annals of Mathematics and Arti cial Intelligence</source>
          <volume>34</volume>
          (
          <issue>1-3</issue>
          ),
          <volume>197</volume>
          {215 (Mar
          <year>2002</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Antoniou</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Antoniou</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Billington</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Governatori</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Maher</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          :
          <article-title>On the modelling and analysis of regulations</article-title>
          .
          <source>In: In Proceedings of the Australian Conference on Information Systems</source>
          (
          <year>1999</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Artz</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gil</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          :
          <article-title>A survey of trust in computer science and the semantic web</article-title>
          .
          <source>J. of Web Semantics</source>
          <volume>5</volume>
          (
          <issue>2</issue>
          ),
          <volume>58</volume>
          {
          <fpage>71</fpage>
          (
          <year>2007</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Athan</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Boley</surname>
            ,
            <given-names>H.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Governatori</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Palmirani</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Paschke</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wyner</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>OASIS LegalRuleML</article-title>
          . In: Francesconi,
          <string-name>
            <given-names>E.</given-names>
            ,
            <surname>Verheij</surname>
          </string-name>
          ,
          <string-name>
            <surname>B</surname>
          </string-name>
          . (eds.)
          <source>International Conference on Arti cial Intelligence and Law</source>
          ,
          <source>ICAIL '13</source>
          . pp.
          <volume>3</volume>
          {
          <fpage>12</fpage>
          .
          <string-name>
            <surname>ACM</surname>
          </string-name>
          (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Baroni</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cerutti</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Giacomin</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Guida</surname>
          </string-name>
          , G.:
          <article-title>AFRA: Argumentation framework with recursive attacks</article-title>
          .
          <source>International Journal of Approximate Reasoning</source>
          <volume>52</volume>
          (
          <issue>1</issue>
          ),
          <volume>19</volume>
          {37 (Jan
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Bicchieri</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Muldoon</surname>
          </string-name>
          , R.:
          <article-title>Social norms</article-title>
          . In: Zalta,
          <string-name>
            <surname>E.N.</surname>
          </string-name>
          <article-title>(ed.) The Stanford Encyclopedia of Philosophy</article-title>
          . The Metaphysics Research Lab, Stanford University, Stanford, CA, USA, spring
          <year>2011</year>
          edn. (
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Brams</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kilgour</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zwicker</surname>
            ,
            <given-names>W.:</given-names>
          </string-name>
          <article-title>The paradox of multiple elections</article-title>
          .
          <source>Social Choice &amp; Welfare</source>
          <volume>15</volume>
          (
          <issue>2</issue>
          ),
          <volume>211</volume>
          {
          <fpage>236</fpage>
          (
          <year>1998</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Conitzer</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lang</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sandholm</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          :
          <article-title>How many candidates are needed to make elections hard to manipulate?</article-title>
          <source>In: Proceedings of the 9th Conference on Theoretical Aspects of Rationality and Knowledge</source>
          . pp.
          <volume>201</volume>
          {
          <fpage>214</fpage>
          .
          <string-name>
            <surname>ACM</surname>
          </string-name>
          (
          <year>2003</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Correia</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cruz</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Leite</surname>
          </string-name>
          , J.:
          <article-title>On the e cient implementation of social abstract argumentation</article-title>
          . In: Schaub,
          <string-name>
            <given-names>T.</given-names>
            ,
            <surname>Friedrich</surname>
          </string-name>
          ,
          <string-name>
            <surname>G.</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O</given-names>
            <surname>'Sullivan</surname>
          </string-name>
          ,
          <string-name>
            <surname>B.</surname>
          </string-name>
          <source>(eds.) 21st European Conference on Arti cial Intelligence</source>
          ,
          <string-name>
            <surname>ECAI</surname>
          </string-name>
          <year>2014</year>
          .
          <article-title>Frontiers in Arti cial Intelligence and Applications</article-title>
          , vol.
          <volume>263</volume>
          , pp.
          <volume>225</volume>
          {
          <fpage>230</fpage>
          . IOS Press (
          <year>2014</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Criado</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Argente</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Noriega</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Botti</surname>
            ,
            <given-names>V.J.:</given-names>
          </string-name>
          <article-title>MaNEA: A distributed architecture for enforcing norms in open MAS</article-title>
          .
          <source>Eng. Appl. of AI</source>
          <volume>26</volume>
          (
          <issue>1</issue>
          ),
          <volume>76</volume>
          {
          <fpage>95</fpage>
          (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Damianou</surname>
            ,
            <given-names>N.C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bandara</surname>
            ,
            <given-names>A.K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sloman</surname>
            ,
            <given-names>M.S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lupu</surname>
            ,
            <given-names>E.C.</given-names>
          </string-name>
          :
          <article-title>A survey of policy speci cation approaches</article-title>
          .
          <source>Tech. rep.</source>
          , Department of Computing,
          <source>Imperial College of Science Technology and Medicine</source>
          , London, UK (
          <year>2002</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Dignum</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kinny</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sonenberg</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          :
          <article-title>From desires, obligations and norms to goals</article-title>
          .
          <source>Cogn. Sci. Quarterly</source>
          <volume>2</volume>
          ,
          <issue>407</issue>
          {
          <fpage>430</fpage>
          (
          <year>2002</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Dignum</surname>
            , V., Meyer,
            <given-names>J.J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Weigand</surname>
          </string-name>
          , H.:
          <article-title>Towards an organizational model for agent societies using contracts</article-title>
          .
          <source>In: Proceedings of AAMAS 2002</source>
          . pp.
          <volume>694</volume>
          {
          <fpage>695</fpage>
          .
          <string-name>
            <surname>ACM</surname>
          </string-name>
          , New York, NY, USA (
          <year>2002</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>d'Inverno</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Luck</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Noriega</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rodriguez-Aguilar</surname>
            ,
            <given-names>J.A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sierra</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>Communicating open systems</article-title>
          .
          <source>Arti cial Intelligence</source>
          <volume>186</volume>
          ,
          <fpage>38</fpage>
          {94 (Jul
          <year>2012</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Dung</surname>
            ,
            <given-names>P.M.</given-names>
          </string-name>
          :
          <article-title>On the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games</article-title>
          .
          <source>Arti cial Intelligence</source>
          <volume>77</volume>
          (
          <issue>2</issue>
          ),
          <volume>321</volume>
          {357 (Sep
          <year>1995</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>Dunne</surname>
            ,
            <given-names>P.E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hunter</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>McBurney</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Parsons</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wooldridge</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          :
          <article-title>Weighted argument systems: Basic de nitions, algorithms, and complexity results</article-title>
          .
          <source>Arti cial Intelligence</source>
          <volume>175</volume>
          (
          <issue>2</issue>
          ),
          <volume>457</volume>
          {486 (Feb
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>Endriss</surname>
            ,
            <given-names>U.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Maudet</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sadri</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Toni</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          :
          <article-title>Negotiating socially optimal allocations of resources</article-title>
          .
          <source>Journal of Arti cial Intelligence Research</source>
          <volume>25</volume>
          (
          <issue>1</issue>
          ),
          <volume>315</volume>
          {348 (Mar
          <year>2006</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>Foukia</surname>
          </string-name>
          , N.:
          <article-title>IDReAM: Intrusion detection and response executed with agent mobility architecture and implementation</article-title>
          .
          <source>In: Proceedings of AAMAS 2005</source>
          . pp.
          <volume>264</volume>
          {
          <fpage>270</fpage>
          .
          <string-name>
            <surname>ACM</surname>
          </string-name>
          (
          <year>2005</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          21.
          <string-name>
            <surname>Gaertner</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Garcia-Camino</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Noriega</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rodriguez-Aguilar</surname>
            ,
            <given-names>J.A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Vasconcelos</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          :
          <article-title>Distributed norm management in regulated multiagent systems</article-title>
          .
          <source>In: Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems</source>
          . pp.
          <volume>90</volume>
          :
          <issue>1</issue>
          {
          <issue>90</issue>
          :
          <article-title>8</article-title>
          . AAMAS '07,
          <string-name>
            <surname>ACM</surname>
          </string-name>
          , New York, NY, USA (
          <year>2007</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          22.
          <string-name>
            <surname>Grandison</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sloman</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>A survey of trust in internet applications</article-title>
          .
          <source>IEEE Commun. Surveys Tuts</source>
          .
          <volume>4</volume>
          (
          <issue>4</issue>
          ),
          <volume>2</volume>
          {
          <fpage>16</fpage>
          (
          <year>2000</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          23. Gri ths, N.,
          <string-name>
            <surname>Luck</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          :
          <article-title>Norm Emergence in Tag-Based Cooperation</article-title>
          .
          <source>In: Proceedings of COIN</source>
          (
          <year>2010</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          24.
          <string-name>
            <surname>Horling</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lesser</surname>
          </string-name>
          , V.:
          <article-title>A survey of multi-agent organizational paradigms</article-title>
          .
          <source>Knowl. Eng. Rev</source>
          .
          <volume>19</volume>
          ,
          <issue>281</issue>
          {
          <fpage>316</fpage>
          (
          <year>2004</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          25.
          <string-name>
            <surname>Leite</surname>
            ,
            <given-names>J.a.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Martins</surname>
            ,
            <given-names>J.a.</given-names>
          </string-name>
          :
          <article-title>Social abstract argumentation</article-title>
          .
          <source>In: Proceedings of the Twenty-Second International Joint Conference on Arti cial Intelligence</source>
          . pp.
          <volume>2287</volume>
          {
          <fpage>2292</fpage>
          . IJCAI'11, AAAI Press (
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          26.
          <string-name>
            <given-names>Marzo</given-names>
            <surname>Serugendo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G.</given-names>
            ,
            <surname>Gleizes</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.P.</given-names>
            ,
            <surname>Karageorgos</surname>
          </string-name>
          ,
          <string-name>
            <surname>A.</surname>
          </string-name>
          :
          <article-title>Self-organising systems</article-title>
          . In: Di Marzo Serugendo,
          <string-name>
            <given-names>G.</given-names>
            ,
            <surname>Gleizes</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.P.</given-names>
            ,
            <surname>Karageorgos</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A</given-names>
            . (eds.)
            <surname>Self-organising Software</surname>
          </string-name>
          , pp.
          <volume>7</volume>
          {
          <fpage>32</fpage>
          . Natural Computing Series, Springer (
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          27.
          <string-name>
            <given-names>Marzo</given-names>
            <surname>Serugendo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G.</given-names>
            ,
            <surname>Karageorgos</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            ,
            <surname>Rana</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            ,
            <surname>Zambonelli</surname>
          </string-name>
          ,
          <string-name>
            <surname>F.</surname>
          </string-name>
          :
          <source>Engineering SelfOrganising Systems, Lecture Notes in Computer Science</source>
          , vol.
          <volume>2977</volume>
          . Springer Berlin Heidelberg (
          <year>2004</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          28.
          <string-name>
            <surname>Modgil</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Faci</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Meneguzzi</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Oren</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Miles</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Luck</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>A framework for monitoring agent-based normative systems</article-title>
          .
          <source>In: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 1</source>
          . pp.
          <volume>153</volume>
          {
          <fpage>160</fpage>
          . AAMAS '09,
          <string-name>
            <surname>International</surname>
            <given-names>Foundation</given-names>
          </string-name>
          <source>for Autonomous Agents and Multiagent Systems</source>
          , Richland,
          <string-name>
            <surname>SC</surname>
          </string-name>
          (
          <year>2009</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          29.
          <string-name>
            <surname>Morales</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lopez-Sanchez</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Esteva</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          :
          <article-title>Using Experience to Generate New Regulations</article-title>
          . In: IJCAI. pp.
          <volume>307</volume>
          {
          <issue>312</issue>
          (
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref30">
        <mixed-citation>
          30.
          <string-name>
            <surname>Morales</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lopez-Sanchez</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rodriguez-Aguilar</surname>
            ,
            <given-names>J.A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Wooldridge</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Vasconcelos</surname>
          </string-name>
          , W.:
          <article-title>Automated synthesis of normative systems</article-title>
          .
          <source>In: AAMAS 2013</source>
          . pp.
          <volume>483</volume>
          {
          <issue>490</issue>
          (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref31">
        <mixed-citation>
          31.
          <string-name>
            <surname>Pacheco</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Carmo</surname>
            ,
            <given-names>J.:</given-names>
          </string-name>
          <article-title>A role based model for the normative speci cation of organized collective agency and agents interaction</article-title>
          .
          <source>Autonomous Agents and MultiAgent Systems</source>
          <volume>6</volume>
          ,
          <fpage>145</fpage>
          {184 (March
          <year>2003</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref32">
        <mixed-citation>
          32.
          <string-name>
            <surname>Rahwan</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Simari</surname>
            ,
            <given-names>G.R.</given-names>
          </string-name>
          :
          <source>Argumentation in Arti cial Intelligence</source>
          . Springer, 1st edn. (
          <year>2009</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref33">
        <mixed-citation>
          33.
          <string-name>
            <surname>Ramchurn</surname>
            ,
            <given-names>S.D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Huynh</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Jennings</surname>
            ,
            <given-names>N.R.:</given-names>
          </string-name>
          <article-title>Trust in multi-agent systems</article-title>
          .
          <source>Knowledge Engineering Review</source>
          <volume>19</volume>
          (
          <issue>1</issue>
          ),
          <volume>1</volume>
          {
          <fpage>25</fpage>
          (
          <year>2004</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref34">
        <mixed-citation>
          34.
          <string-name>
            <surname>Resnick</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zeckhauser</surname>
          </string-name>
          , R.:
          <article-title>Trust among strangers in Internet transactions: Empirical analysis of eBay's reputation system</article-title>
          . In: Baye,
          <string-name>
            <surname>M.R</surname>
          </string-name>
          . (ed.)
          <article-title>The Economics of the Internet and</article-title>
          E-Commerce, pp.
          <volume>127</volume>
          {
          <fpage>157</fpage>
          .
          <string-name>
            <surname>Elsevier Science</surname>
          </string-name>
          (
          <year>2002</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref35">
        <mixed-citation>
          35.
          <string-name>
            <surname>Robertson</surname>
            ,
            <given-names>D.:</given-names>
          </string-name>
          <article-title>A lightweight method for coordination of agent oriented web services</article-title>
          .
          <source>In: Proceedings of AAAI Spring Symposium on Semantic Web Services</source>
          .
          <source>Stanford</source>
          (
          <year>2004</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref36">
        <mixed-citation>
          36.
          <string-name>
            <surname>Sabater</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sierra</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>Review on computational trust and reputation models</article-title>
          .
          <source>AI</source>
          Review
          <volume>23</volume>
          (
          <issue>1</issue>
          ),
          <volume>33</volume>
          {60 (
          <year>September 2005</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref37">
        <mixed-citation>
          37.
          <string-name>
            <surname>Sen</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sen</surname>
            ,
            <given-names>S.:</given-names>
          </string-name>
          <article-title>E ects of social network topology and options on norm emergence</article-title>
          .
          <source>In: Proceedings of COIN</source>
          . pp.
          <volume>211</volume>
          {
          <issue>222</issue>
          (
          <year>2010</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref38">
        <mixed-citation>
          38.
          <string-name>
            <surname>Sen</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Airiau</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          :
          <article-title>Emergence of norms through social learning</article-title>
          .
          <source>In: IJCAI</source>
          . pp.
          <volume>1507</volume>
          {
          <issue>1512</issue>
          (
          <year>2007</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref39">
        <mixed-citation>
          39.
          <string-name>
            <surname>Shi</surname>
            <given-names>man</given-names>
          </string-name>
          , R.N.,
          <string-name>
            <surname>Michel</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Krauthammer</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fuchs</surname>
            ,
            <given-names>N.E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kaljurand</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kuhn</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          :
          <article-title>Writing clinical practice guidelines in controlled natural language</article-title>
          .
          <source>In: Proceedings of the 2009 Conference on Controlled Natural Language</source>
          . pp.
          <volume>265</volume>
          {
          <fpage>280</fpage>
          . CNL'
          <volume>09</volume>
          , Springer-Verlag, Berlin, Heidelberg (
          <year>2010</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref40">
        <mixed-citation>
          40.
          <string-name>
            <surname>Shoham</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tennenholtz</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          :
          <article-title>On social laws for arti cial agent societies: o -line design</article-title>
          .
          <source>Artif</source>
          . Intell.
          <volume>73</volume>
          ,
          <issue>231</issue>
          {
          <fpage>252</fpage>
          (
          <year>1995</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref41">
        <mixed-citation>
          41.
          <string-name>
            <surname>Sloman</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          :
          <article-title>Policy driven management for distributed systems</article-title>
          .
          <source>Journal of Network and Systems Management</source>
          <volume>2</volume>
          ,
          <issue>333</issue>
          {
          <fpage>360</fpage>
          (
          <year>1994</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref42">
        <mixed-citation>
          42.
          <string-name>
            <surname>Teacy</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Chalkiadakis</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Rogers</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Jennings</surname>
            ,
            <given-names>N.R.</given-names>
          </string-name>
          :
          <article-title>Sequential decision making with untrustworthy service providers</article-title>
          .
          <source>In: Proceedings of the 7th International Conference on Autonomous Agents and Multiagent Systems</source>
          (
          <year>2008</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref43">
        <mixed-citation>
          43.
          <string-name>
            <surname>Thagard</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          :
          <article-title>Coherence in thought and action</article-title>
          . MIT Press (
          <year>2002</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref44">
        <mixed-citation>
          44.
          <string-name>
            <surname>Thakoor</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Morookian</surname>
            ,
            <given-names>J.M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Chahl</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hine</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zornetzer</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          : BEES:
          <article-title>Exploring mars with bioinspired technologies</article-title>
          .
          <source>Computer</source>
          <volume>37</volume>
          (
          <issue>9</issue>
          ),
          <volume>38</volume>
          {
          <fpage>47</fpage>
          (
          <year>2004</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref45">
        <mixed-citation>
          45.
          <string-name>
            <surname>Veloso</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <article-title>M.: Planning and learning by analogical reasoning</article-title>
          , vol.
          <volume>886</volume>
          . Springer (
          <year>1994</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref46">
        <mixed-citation>
          46.
          <string-name>
            <surname>Villatoro</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sabater-Mir</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sen</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          :
          <article-title>Social instruments for robust convention emergence</article-title>
          .
          <source>In: IJCAI</source>
          . pp.
          <volume>420</volume>
          {
          <issue>425</issue>
          (
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref47">
        <mixed-citation>
          47.
          <string-name>
            <surname>Viroli</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zambonelli</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          :
          <article-title>A biochemical approach to adaptive service ecosystems</article-title>
          .
          <source>Inf. Sci</source>
          .
          <volume>180</volume>
          (
          <issue>10</issue>
          ),
          <year>1876</year>
          {
          <year>1892</year>
          (
          <year>2010</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref48">
        <mixed-citation>
          48.
          <string-name>
            <surname>Wedde</surname>
            ,
            <given-names>H.F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Farooq</surname>
            ,
            <given-names>M.:</given-names>
          </string-name>
          <article-title>A comprehensive review of nature inspired routing algorithms for xed telecommunication networks</article-title>
          .
          <source>Journal of Systems Architecture</source>
          <volume>52</volume>
          (
          <issue>8</issue>
          ),
          <volume>461</volume>
          {484 (Aug
          <year>2006</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref49">
        <mixed-citation>
          49.
          <string-name>
            <surname>Wilks</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          :
          <article-title>An arti cial intelligence approach to machine translation</article-title>
          . Springer (
          <year>2009</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref50">
        <mixed-citation>
          50.
          <string-name>
            <surname>Wyner</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bos</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Basile</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Quaresma</surname>
            ,
            <given-names>P.:</given-names>
          </string-name>
          <article-title>An empirical approach to the semantic representation of laws</article-title>
          . In: Schafer, B. (ed.)
          <source>Legal Knowledge and Information Systems - JURIX</source>
          <year>2012</year>
          :
          <article-title>The Twenty-Fifth Annual Conference</article-title>
          .
          <source>Frontiers in Arti - cial Intelligence and Applications</source>
          , vol.
          <volume>250</volume>
          , pp.
          <volume>177</volume>
          {
          <fpage>180</fpage>
          . IOS Press (
          <year>2012</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref51">
        <mixed-citation>
          51.
          <string-name>
            <surname>Wyner</surname>
            , A., van Engers,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bahreini</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          :
          <article-title>From policy-making statements to rstorder logic</article-title>
          .
          <source>In: Proceedings of the First International Conference on Electronic Government and the Information Systems Perspective</source>
          . pp.
          <volume>47</volume>
          {
          <fpage>61</fpage>
          . EGOVIS'
          <volume>10</volume>
          , Springer-Verlag, Berlin, Heidelberg (
          <year>2010</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref52">
        <mixed-citation>
          52.
          <string-name>
            <surname>Wyner</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Governatori</surname>
          </string-name>
          , G.:
          <article-title>A study on translating regulatory rules from natural language to defeasible logic</article-title>
          .
          <source>In: RuleML 2013</source>
          . pp.
          <volume>16</volume>
          .
          <issue>1</issue>
          {
          <issue>16</issue>
          .8.
          <string-name>
            <surname>Seatle</surname>
          </string-name>
          , USA (
          <year>July 2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref53">
        <mixed-citation>
          53.
          <string-name>
            <surname>Wyner</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Peters</surname>
            ,
            <given-names>W.</given-names>
          </string-name>
          :
          <article-title>On rule extraction from regulations</article-title>
          . In: Atkinson,
          <string-name>
            <surname>K</surname>
          </string-name>
          . (ed.)
          <source>Legal Knowledge and Information Systems - JURIX</source>
          <year>2011</year>
          :
          <article-title>The Twenty-Fourth Annual Conference</article-title>
          .
          <source>Frontiers in Arti cial Intelligence and Applications</source>
          , vol.
          <volume>235</volume>
          , pp.
          <volume>113</volume>
          {
          <fpage>122</fpage>
          . IOS Press (
          <year>2011</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref54">
        <mixed-citation>
          54.
          <string-name>
            <surname>Yin</surname>
            ,
            <given-names>X.</given-names>
          </string-name>
          , Han,
          <string-name>
            <given-names>J.</given-names>
            ,
            <surname>Yu</surname>
          </string-name>
          ,
          <string-name>
            <surname>P.S.:</surname>
          </string-name>
          <article-title>Truth discovery with multiple con icting information providers on the Web</article-title>
          .
          <source>In: Proceedings of the Conference on Knowledge and Data Discovery</source>
          (
          <year>2007</year>
          )
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>