<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta>
      <journal-title-group>
        <journal-title>Workshop “From Objects to Agents”, September</journal-title>
      </journal-title-group>
    </journal-meta>
    <article-meta>
      <title-group>
        <article-title>Computable Law as Argumentation-based MAS</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Roberta Calegari</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Andrea Omicini</string-name>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Giovanni Sartor</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Alma AI - Alma Mater Research Institute for Human-Centered Artificial Intelligence , Alma Mater Studiorum-Università di Bologna</institution>
          ,
          <country country="IT">Italy</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2020</year>
      </pub-date>
      <volume>1</volume>
      <fpage>4</fpage>
      <lpage>16</lpage>
      <abstract>
        <p>In this paper we sketch a vision of computable law as argumentation-based MAS, i.e., human-centred intelligent systems densely populated by agents (software or human) capable of understanding, arguing, and reporting, via factual assertions and arguments, about what is happening and what they can make possibly happen. A multi-agent system based on argumentation, dialogue, and conversation is, in this vision, the basis for making the law computable: through argumentation, dialogue, and adherence to social judgment, the behaviour of the intelligent system can be reached, shaped, and controlled with respect to the law. In such a scenario, computable law - and related intelligent behaviour - is likely to become associated with the capability of arguing about state and situation, by reaching a consensus on what is happening around and what is needed, and by triggering and orchestrating proper decentralised semantic conversations to decide how to collectively act in order to reach a future desirable state. Interpretability and explainability become important features for that sort of systems, based on the integration of logic-based and sub-symbolic techniques. Within this novel setting, MAS methodologies and technologies become the starting point to achieve computable law, even if they need to be adapted and extended for dealing with new challenges. Accordingly, in this paper we discuss how this novel vision can build upon some readily-available technologies, and the research challenges it poses. We analyse a number of approaches and technologies that should be involved in the engineering of systems and services, and become core expertise for distributed systems engineers. Among the others, these include knowledge representation, machine learning, and logic argumentation.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;computable law</kwd>
        <kwd>multi-agent system</kwd>
        <kwd>argumentation</kwd>
        <kwd>logic</kwd>
        <kwd>hybrid approaches</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        The research field of computable law studies the engineering of the law – i.e., the design of
appropriate formal models and the development of the corresponding technology – to allow
norms, terms and conditions to be represented in a machine-understandable way [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. The aim
is to enable machine and software agents to process, and reason about, legal abstractions with a
certain degree of accuracy, and to take autonomous decisions based on this. In the context of
computable law, two key factors have to be considered. The first is that a single or unique way
of modelling legal knowledge cannot be taken for granted—namely, there are multiple ways of
identifying and circumscribing the “law” to be modelled, and multiple ways of representing
legal contents into automatically processable information structures. Therefore, the computable
law model should be able to deal with heterogeneous “legal ontologies”.
      </p>
      <p>The second factor is that nowadays application scenarios for computable law are strongly
characterised by the same “symbolic vs sub-symbolic” dichotomy that also informs the most
promising techniques in artificial intelligence today. Indeed, if, on the one hand, legal intelligence
has historically been bound to logic and argumentation (i.e., symbolic approaches), on the other
hand, the applications of machine learning algorithms (i.e., sub-symbolic approaches) are
increasing today, especially in the field of data analysis and predictive justice. The use of the
latter techniques raises ethical and fairness issues linked to the lack of transparency and the
possibility of hidden biases. Therefore, models aimed at making the law computable must
somehow consider that dichotomy and should try a reconciliation – possibly via a blended
integration – so to take advantage of each approach: benefiting from the strengths of each
method, and smoothing the corresponding limits.</p>
      <p>
        Along this line, in this paper we sketch a vision of computable law as an argumentation-based
multi-agent system. As widely recognised, agent architecture is today the reference for the
design of intelligent systems [
        <xref ref-type="bibr" rid="ref2 ref3 ref4">2, 3, 4</xref>
        ], it allows dealing with heterogeneous entities and models,
and fits perfectly with the pervasive and distributed scenarios that the computable law aims at
addressing. The vision we propose relies on the multi-agent system (MAS) abstractions and
architecture [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ] expanding them both from the point of view of norms and argumentation (as
in existing works on normative MAS, see for instance [
        <xref ref-type="bibr" rid="ref6 ref7">6, 7</xref>
        ]), and of the integration between
symbolic and sub-symbolic techniques.
      </p>
      <p>In this scenario, the very nature of the system actors – intended as agents, but also as
surrounding environments – embodies the concepts of computable law implementing and
coordinating the activities of distributed processes in order to achieve either individual or
common goals. In fact, computable law is likely to become associated with the capability of
debating about situations and about the current context, by reaching a consensus on what is
happening around and what is needed, and by triggering and directing proper decentralised
semantic conversations to decide how to collectively act in order to reach the future desirable
state of the afairs. Within this novel setting, interpretability and explainability become a
remarkable feature of the multi-agent system.</p>
      <sec id="sec-1-1">
        <title>1.1. Contributions of the Paper</title>
        <p>Based on the envisioned scenario, the contributions of this paper are the following:
• We detail the concept of computable law as argumentation-based MAS (i.e., conversational
agents), also with the help of a running example, and show how they afect the engineering
of intelligent systems, challenging traditional approaches to distributed computing and
calling for novel argumentation approaches.
• We investigate a number of approaches and technologies that should be involved in
the engineering of systems and services in that original scenario, and should become
core expertise for distributed systems engineering. Among the others, those include
knowledge representation and ontologies, machine learning, argumentation models and
technologies, human-computer interfaces.</p>
      </sec>
    </sec>
    <sec id="sec-2">
      <title>2. Motivating Scenario: self-driving cars</title>
      <p>To ground the discussion, we first examine a case study in the area of trafic management,
considering the near future of self-driving cars. In that scenario cars are capable of
communicating with each other and with the road infrastructure while cities and roads are suitably
enriched with sensors and virtual trafic signs able to dynamically interact with cars to provide
information and supervision.</p>
      <p>The choice of the case study is driven by the fact that both self-driving cars and the whole
intelligent transportation domain – including trafic management – are a natural fit for
computable law since they need real-time control and feedback – with respect to the norms and the
current state of afairs – and should both adapt to legislation as well as to possible contingencies
(also involving ethical choices and legal reasoning).</p>
      <p>Accordingly, self-driving cars need to (i) exhibit some degree of intelligence for taking
autonomous decisions, (ii) converse with the context that surrounds them, (iii) have humans in
the loop, and (iv) be deeply intertwined with the law setting characterising the environment
and the society. Generally speaking, the main success factor to address the goal of the car (i.e.,
reach a destination) is the capability to converse and dialectically interact with the surrounding
environment (other vehicles, infrastructure, humans, etc.), in order to make the best choice and
therefore actuate the best action.</p>
      <p>
        Part of this scenario is already a reality in many (smart) cities around the world. There, IT
technologies are exploited to improve both the organisation of the transportation infrastructure
and the performance of some specific parts. Adaptive trafic lights, dynamic speed and flow
metering, urban monitoring stations, and similar tools and algorithms are routinely employed by
district administrations as a means to observe and revise trafic situations in (almost) real-time
[
        <xref ref-type="bibr" rid="ref8 ref9">8, 9</xref>
        ], as well as to assist in urban planning.
      </p>
      <p>In the following we analyse the scenario from a computational perspective – i.e., discussing
in details actors and actions that come up as well as system requirements – in order to provide
the bases for its reification on the most appropriate engineering approaches and corresponding
architecture.</p>
      <sec id="sec-2-1">
        <title>2.1. Analysis &amp; requirements</title>
        <p>In the envisioned desiderata scenario, passengers simply express their desire (e.g., “bring me
to the hospital") and the car starts acting in autonomy, travelling towards the destination and
without passengers to worry about the specific actions and decisions to be undertaken. There,
autonomous goal-oriented smart agents (either software, objects, humans, etc.) are pervasive
since multiple actors come into play, such as other self-driving cars or, for instance, autonomous
intersection managers that regulate the flow of cars based on specific goals imposed by the
municipality (e.g., reduce circulation in a specific area). Moreover, the environment is one of
the main actors of such a system–providing context information, such as rules valid in that
specific area or situation, i.e., encapsulating context knowledge and intelligence.</p>
        <p>Looking in-depth at the expect behaviour of self-driving cars in an urban environment, it turns
out that autonomous cars need to undertake a complex decision-making process, in compliance
with norms and social rules, in almost the whole journey, which seamlessly integrates actions
(and perceptions) at diferent levels.</p>
        <p>The first action is the trip planning which must be done by considering the global knowledge—
for instance, local laws, regulations, policies, and average trafic conditions of the intersections
and roads along the path toward the destination, influenced by factors such as the day of the
week, the hour of the day, the weather, and the like.</p>
        <p>Planning is then modulated by considering all contingencies arising during the journey –
such as, for instance, a car crash forcing a change path, a protest causing delays in our preferred
path, etc. As a consequence, the original plan has to be adapted to the local knowledge (and
perceptions) cars gather while enacting it; once again, the decision is closely related to legal
issues: for instance, what is the best action to perform in order to avoid fines? Or, again, if an
accident cannot be avoided, how to choose the ethically-preferable option?</p>
        <p>Moreover, there are strict rules on safety we would like self-driving cars to automatically
enforce, such as slowing down when passing nearby a school, breaking and setting aside the
vehicle as soon as the horn of an ambulance is heard, etc. In other words, regardless of what
the global and local knowledge may suggest, we abide by a set of general commonsense rules
orthogonally considered valid.</p>
        <p>Generally speaking, compliance with the legal rules by the autonomous cars – including also
global, national, state, and local laws, regulations, and policies – must be guaranteed, unless
contingent priority situations occur (such as when a norm has to be violated in order to save
the life of a passenger/pedestrian).</p>
        <p>The analysis carried out so far highlights some key requirements that engineering approaches
and techniques have to fulfil. First of all, the envisioned scenario seamlessly integrates
perceptions (and actions) at two diferent scales—namely, the macro and the micro.</p>
        <p>The macro level of the system includes global knowledge and generally-valid rules, like
universal norms and legal conventions, possibly modulated by commonsense reasoning. Moreover,
with respect to the surrounding environment, the macro level deals with a mid / long term
horizon and focus on the issue of trafic flow management–including, for instance, trafic flow
forecasting and urban planning possibly learned from historical data analysis. On the other
hand, the micro level, deals with the short term horizon, and mostly focuses on intersection
management, including a few highly intertwined sub-problems—e.g. collision avoidance,
minimisation of average delay, and congestion resolution. The macro-level and the micro-level act
synergistically, exploiting some sort of integration in order to achieve individual and social
goals.</p>
        <p>As the last step, a suitable infrastructure for V2I (vehicle-to-infrastructure) and V2V
(vehicle-tovehicle) communication should be considered, for instance through the deployment of Road-side
Units (RSU). This infrastructure should make it possible to convey information from the vehicle
to the infrastructure and vice-versa. For instance, it should provide information about the road
on which the vehicle is travelling and its specific rules, environmental conditions around the
vehicle, trafic in the vicinity of the vehicle, and construction in the vicinity of the vehicle.</p>
        <p>All the abovementioned ingredients should be mixed consistently in order to achieve the goal
of the cars and therefore of the user, respecting current rules and convention. Therefore, the
system knowledge base needs to be built by taking into account both macro and micro scales,
and agents have to be able to reason and argue over it in order to achieve their goals.</p>
        <p>In the following, we show that computable law naturally is an essential ingredient in a
distributed multi-party conversation, or dialogue, based on distributed intelligence. It cannot
be easily tackled with traditional approaches to distributed computing: instead, diferent
approaches and techniques needs to be put in place and to be fruitfully integrated in order to meet
the aforementioned requirements.</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>3. Computable law as conversation and distributed (micro-)intelligence in MAS</title>
      <p>
        The wide variety of actors and requirements in the above-described scenario recall the MAS
model and architecture as intrinsically suitable for addressing issues and challenges lay ahead.
Along this line, the vision of computable law described in this paper is based on the fundamental
roles of MAS – individuals (agents), society, and environment [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ]– according to the Agents
and Artefacts (A&amp;A) meta-model [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ].
      </p>
      <p>Then, our vision enhances these fundamentals roles taking into account two more key
concepts and related abstraction: norms and e-institutions – i.e., proper roles and abstractions
for the normative environment – and micro-intelligence—i.e., rational reasoning capabilities to
enhance both environment and individuals.</p>
      <p>In a nutshell, the system is composed of several agents, each with his own personal goal to
achieve. Agents’ interaction and dialogue, on normative aspects and surrounding situations,
allow them to complete their goals. For that reason, rules and conventions must have an
active role in the system (e-institution) in order to be questioned, examined and respected. The
system’s actors – individuals, environment, e-institution – are intelligent (micro-intelligence)
in that they can reason on knowledge and context, argue and explain the rationale behind
their decision. The overall behaviour of the system transposes the social behaviour, the debate
with the society can act as a social judgment on individuals and therefore can calibrate their
individual actions.</p>
      <p>Diferent roles of e-institution and micro-intelligence and their interaction are discussed
below.</p>
      <sec id="sec-3-1">
        <title>3.1. e-Institution</title>
        <p>
          In the vision sketched above, particular attention has been given to normative concepts since
norms and laws are the basic bricks upon which to build the concept of computable law, since
individual and collective behaviours are both afected by norms. In particular, we leverage
on the well-known concepts of electronic-institution (e-institution) – as in normative MAS
[
          <xref ref-type="bibr" rid="ref6">6</xref>
          ] – and deliberative agents [
          <xref ref-type="bibr" rid="ref12">12</xref>
          ], setting up e-institutions via suitable coordination artefacts
exploited as normative abstractions.
        </p>
        <p>
          Loosely speaking, e-institutions are computational realisations of traditional institutions:
i.e., coordination artefacts providing an environment where agents can interact according to
stated laws – norms or conventions – in such a way that interactions within the e-institution
reproduce norm-based interactions in the actual world. With the term deliberative agents, we
emphasise the agents’ autonomy stressed both by e-institutions and normative systems. Indeed,
in such systems individuals possess the property of normative autonomy—i.e., can decide to
violate a norm to achieve their goals, or to change their goals so that they match the existing
norms. E-instutions can provide for real-time detection of violations, and norm-enforcement
can be envisaged via diferent enforcement technique. For instance, a simple local blocking rule
[
          <xref ref-type="bibr" rid="ref13">13</xref>
          ] can be applied (specific actions may be disallowed), or the application of the laws can be
prioritised according to the contingent situation (some actions may be discouraged), or norms
can be modelled in a game theory framework, where sanctions and rewards are provided for
agents [
          <xref ref-type="bibr" rid="ref14">14</xref>
          ], or where agents are expected to cooperate according to norms that maximise the
society’s utility.
        </p>
        <p>From an agent’s point of view, two major benefits stem from modeling environments as
e-institutions. On the one hand, e-institutions establish conventions – on behaviour, language,
and protocols – that inform agents about the law and possibly induce them to comply. In a
sense, the environment is given structure, so that the agents have an easy comprehension of its
working laws. On the other hand, norm enforcer agents endowed with capabilities for acquiring
norms dynamically and enforcing them in uncertain environments can be envisioned and spread
all over the system. Recalling our motivating scenario: drivers or trafic wardens know when
they can talk, what consequences their acts will have, and what actions are possible at each
moment in time. These restrictions contains the set of actions that agents have to consider at
each moment in time by limiting the set of options that agents have to think about.</p>
        <p>From the systems properties’ point of view, e-institution promote a clear embodiment of the
laws that govern the system, thus making it observable and more explainable in its autonomous
actions—i.e., the explicit role of institution allow to enforce a set of norms whose violation can
be perfectly observed. With respect to the case study, e-institution incarnates global system
knowledge as global, national, state, and local laws, regulations, and policies.</p>
      </sec>
      <sec id="sec-3-2">
        <title>3.2. Micro-intelligence</title>
        <p>
          In order to ensure diferent levels of knowledge – and intelligence – as highlighted in Section 2,
the MAS model is extended with the concept of micro-intelligence. Micro-intelligence is exploited
to manage precisely the micro-level of the system intelligence – i.e., local intelligence, as the
intelligence of objects, things, but also of the environment – to be integrated with the
macrolevel intelligence. We recall the micro-intelligence definition from [
          <xref ref-type="bibr" rid="ref15 ref16">15, 16</xref>
          ] as the externalised
rationality of cognitive agents, complementing their own in the sense of Clark and Chambers’
active externalism [
          <xref ref-type="bibr" rid="ref17">17</xref>
          ], and under the perspective of Hutchins’ distributed cognition [
          <xref ref-type="bibr" rid="ref18">18</xref>
          ], as
depicted in Figure 1. In short, micro-intelligence is external because it does not strictly belong
to agents, as it is a process independently executed by another entity – namely, an artefact – to
whom the agent is (possibly, temporarily) coupled—in the sense of the distributed cognition’s
“extended mind”. Moreover, it is rational because it is supposed to convey a sound inference
and argumentation process in order to provide an explanation on the decision process.
Microintelligence complements agents’ own cognitive processes because it augments the cognitive
capabilities of agents, by embodying situated knowledge about the local environment along
with the relative inference and argumentation processes. Agents may even be unaware of the
knowledge embodied in the environmental artefact delivering micro-intelligence. Under this
perspective, artefacts act as delegates for intelligent (possibly, epistemic, but surely rational)
behaviour, since they undertake inference processes on behalf of the interacting agents. Along
this lines, our vision stems from two basic premises underpinning the above definition: (i)
knowledge is locally scattered in a distributed environment, hence its situated nature; (ii)
inference and argumentation capabilities are admissible and available over this knowledge,
with the goal of extending the local knowledge through argumentation, induction, deduction,
abduction, and the like.
        </p>
        <p>Operationally, micro-intelligence is about scattering small chunks of machine intelligence
all over a distributed and situated system, capable to enable the individual intelligence and
the argumentation capability of any sort of devices. Micro-intelligence can be encapsulated in
devices of any sort, making them smart, and capable to work together in groups, aggregates,
societies. Thus, the micro-intelligence vision promotes ubiquitous distribution of intelligence
in large pervasive systems such as those belonging to the pervasive and IoT landscape, in
particular as a complement to agent-based technologies and methods, at both the individual
and the collective level.</p>
        <p>Note the micro-intelligence model – and related technologies – becomes fundamental also
for the e-institution since it provides externalised rationality for reasoning and speaking with
other system actors as well as the capability to reach properties such as interpretability and
explainability, by leveraging on symbolic approach.</p>
        <p>Moreover, micro-intelligence, as depicted in this summary, becomes the fundamental engine
enabling conversation and argumentation between the system’s actors and facilitating the
control of contingent situations establishing the possibility to reach a consensus among speaking
agents.</p>
      </sec>
      <sec id="sec-3-3">
        <title>3.3. Overall vision</title>
        <p>Mixing up all the above-mentioned ingredients, agents become capable of dialogue, exploiting
a type of rational and symbolic intelligence that allows arguing and reasoning on the acquired
knowledge. They become aware of the context – normative and social – in which they are
immersed and through the argumentative process and the dialogue they can decide the next
actions be implemented. The normative context is properly represented by the e-institution
abstraction. Distributed (micro-)intelligence embodies the enabler of conversation between
entities in the MAS infrastructure, making feasible the computation of the law and its incarnation
in a computational system.</p>
        <p>
          Dialogue is not expected only between agents but also with the environment and in particular
with the e-institution. The bi-directional dialogue and interaction process, as well as enabling
negotiation between system entities, also allow a prompt examination of each agent and of
their status in order to detect automatically and real-time a possible deviation from the rules
or policies envisaged. The issue of reaching a consensus in an ensemble of cooperating and
interacting autonomous components by distributed negotiations has been already investigated
in the field of multi-agent systems [
          <xref ref-type="bibr" rid="ref19">19</xref>
          ]. In particular, the argumentation-based negotiation
area [
          <xref ref-type="bibr" rid="ref20 ref21">20, 21</xref>
          ] shows how argumentation can help in reaching global and individual goals and
solutions, by letting agents converse and motivate their choices.
        </p>
        <p>Figure 2 (left) summarises our vision by highlighting the main roles involved in the system
as well as the two main activity flows—from data to users (left) and the opposite, from user
goals and desires to activity planning, and lower-level commands (right). The grey boxes, that
we describe more in detail in the following section, represent the technologies involved in the
vision, while arrows represent the expected provided functionalities.</p>
        <p>On one side, knowledge is collected from various sources – e.g., domain-specific
knowledge, ontologies, sensors raw data – and is then exploited by agents that live in a normative
environment (e-institutions). Agents’ activity is then regulated by norms but also addicted
to situated knowledge over which agents can reason and discuss in order to achieve goals.
The multi-agent system, also thanks to its rational reasoning and argumentation capabilities,
can provide outcomes to the users as well as explanations for their behaviours. On the other
side, humans can insert input into the system – like desires, preferences, or goal to achieve –
and these are transposed into agents’ goal, corresponding activity planning, and lower-level
commands for actuators.</p>
        <p>The law, in this vision, become an internal component of the computational processes: legal
norms, values, and principles are mapped and translated into, a computable representation of
legal information – i.e., into the system knowledge – that is directly processed by computational
entities. Agents become artificial legal agents able to comply with legal requirements and to
reason over them.</p>
        <p>Computable law assumes a vision of compliance/law by design in this architecture combining
top-down compliance with predefined rules and bottom-up learning from cases and experience
with capability to address regulatory conflicts according to legal values and principles.</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Enabling technologies</title>
      <p>The above considerations translate into diferent engineering approaches and therefore diferent
nature of algorithms and techniques that can and must be considered when building the system.
In the following, we encapsulate the main technologies involved as well as research challenges
and opportunities.</p>
      <sec id="sec-4-1">
        <title>4.1. Knowledge representation</title>
        <p>Knowledge representation and related techniques are the cornerstone of the envisioned
distributed system, to be able to argue and reason over it.</p>
        <p>Of course, system knowledge has to take into account domain-specific knowledge and
largescale ontologies as repositories to interpret the knowledge bases available to the agents and to
reason and argument over it. Knowledge could be continuously modified, adapted, and refined
by the agents, according to their experience and perception of the environment or to learning
from experience.</p>
        <p>Accordingly, the knowledge base is plausible that is assembled by two main sources: on the
one hand, ontologies and hand-crafted rules, on the other hand, rules learned from big data.
Advances in machine learning will allow extracting knowledge from this data and to merge it
with the former. Hybrid approach dealing with the integration of symbolic and sub-symbolic
approaches becomes of paramount importance.</p>
        <p>In this context, there are several issues and challenges to be tackled, to cite few, automatic
extraction of knowledge from ML models, extraction of commonsense knowledge from the
context, integration of the diverse knowledge in an appropriate logical language that allows
argumentation and inference process to be performed. Several research fields are already facing
these issues, but the general problem is far from being solved. For sure, we believe that a
suitable integration of symbolic and sub-symbolic approaches can help in the achievement of
the construction of proper system knowledge.</p>
      </sec>
      <sec id="sec-4-2">
        <title>4.2. Machine learning</title>
        <p>In our vision, a fundamental role is played by machine learning involved in diferent phases—
namely, data processing &amp; rule learning, and planning.</p>
        <p>
          Data processing &amp; rule learning. At the most straightforward level, machine learning
techniques are clearly involved in raw input data elaboration, coming from sensors and/or documents,
into more complex, high-level, structured information. Moreover, agents should be able to learn
policies from past experience, by adapting both to the changing environment, and to the
continuous progress of the society. Data aggregation, feature extraction, clustering, classification,
data and pattern mining techniques are typically employed today to reach these objectives.
We believe that hybrid approach could provide promising solutions to these tasks, by merging
logic with probabilistic models and statistical learning, so to eficiently handle advantages of
both symbolic and sub-symbolic approaches and moving towards explainable systems [
          <xref ref-type="bibr" rid="ref22">22</xref>
          ]. As
highlighted above, the ML knowledge should somehow be translated into logical knowledge
and properly merged with logical knowledge coming from ontologies or domain-expert norm
translation or similar.
        </p>
        <p>
          Planning. Distributed problem solving, planning, reinforcement learning, and cooperation
[
          <xref ref-type="bibr" rid="ref23">23</xref>
          ] are some of the well-known ML techniques exploited in MAS. Our framework adds the
challenge of integrating these techniques in the argumentation setting, so that the planning and
cooperation derive from a continuous, natural interaction between agents with the environment.
Once the user has specified his desires, the agent must be able to achieve them, interacting and
coordinating with other individuals and with the e-institution to define the actions to perform
and consequently defining appropriate plans to reify the decisions.
        </p>
      </sec>
      <sec id="sec-4-3">
        <title>4.3. MAS &amp; Normative MAS - middleware infrastructure</title>
        <p>From a more implementation-oriented perspective, given that conversations are a new means
of orchestrating the activities of distributed agents, an open research question – and a key one,
too – is to understand which services should a middleware provide in order to support such
distributed conversations.</p>
        <p>
          The multi-agent infrastructure need not only to allow coordination among system actors
but also include the possibility of customisable and reactive artefacts, capable of incorporating
regulation and norms and micro-intelligence (possibly in the form of service). Moreover, the
middleware should provide support for discussions via an open and shared discussion space,
enabling dialogue among components that do not necessarily know each other in advance, and
also providing services and or techniques for sharing knowledge, e.g., a tuple space [
          <xref ref-type="bibr" rid="ref24">24</xref>
          ].
However, unlike traditional tuple space models, the evolution of the conversation, the argumentation
process and the reached consensus should be taken into account, also to be exploited in similar
situations and/or to provide explanations. The best way to build such shared dialogue space,
also taking into account diferent source of knowledge (e.g., commonsense kb, . . . ) and diferent
artefact acting both as law enforcer and intelligence promoter is a fertile ground for research.
        </p>
      </sec>
      <sec id="sec-4-4">
        <title>4.4. Argumentation &amp; logical reasoning</title>
        <p>
          Argumentation is a necessary feature for agents to talk and discuss to reach an agreement.
Several existing works establish the maturity of argumentation models as a key enabler of our
vision [
          <xref ref-type="bibr" rid="ref25 ref26">25, 26</xref>
          ].
        </p>
        <p>
          Despite the long history of research in argumentation and the many fundamental results
achieved, much efort is still needed to efectively exploit argumentation in our envisioned
framework. First, research on argumentation has mostly been theoretical, practical applications
to real-world scenarios have only recently gained attention and are not yet reified in a
readyto-use technology [
          <xref ref-type="bibr" rid="ref27">27</xref>
          ]. Second, many open issues of existing argumentation frameworks
regard their integration with contingency situation and situated reasoning to achieve a blended
integration of the two concepts. Finally, the fundamental assumption every argumentation
framework makes – that is, there must exist either an agreement among agents about the
knowledge, or an external judge enacting some form of control over the argumentation process
– is quite challenging preserving in our envisioned highly distributed, open, and dynamic
scenario. The assumption is somehow related to the requirement of the formal model to
have a coherent and logical conclusion, but neither of the assumptions is easy to have in a
typical pervasive situation: reaching an agreement among many heterogeneous agents and
devices is already a complex task, not obviously easily scalable, an external authority may be
an unacceptable centralisation point. The argumentation architecture should be designed in
order to be highly scalable, hybrid approaches should be investigated such as:
• provide many external authorities sharing the load of negotiating argumentations among
a limited number of participants to enforce shared normative rules, possibly exploiting
some notion of proximity;
• base the agreement on temporary agreement valid only for the duration of a conversation,
defining somehow the concept of temporal locality;
• dually to the previous one, spatial locality may be the criterion to enforce partial
consistency of normative and behavioural rules—according to the concept of “argumentation
neighbourhood” where diferent distributed mediators act to enforce an agreement
respecting the law.
        </p>
        <p>In any case, we think that the concept of locality is crucial and should be considered along with
distributed argumentation—coherently with the notion of micro-intelligence. In addition, it could
be interesting to envision a framework where the specific argumentation and inference process
can be swapped at runtime, making consideration on the specific situation, and introducing,
for instance, abductive reasoning, or probabilistic argumentation or the algorithm deemed best
and necessary for such a situation. The dialogue, therefore, becomes possible by following
diferent rationales that can guide the decision-making process, possibly comparing diferent
perspectives and visions.</p>
      </sec>
      <sec id="sec-4-5">
        <title>4.5. Human-Computer interaction</title>
        <p>Finally, techniques coming from natural language processing, computer vision speech
recognition – already a reality in most everyday applications (e.g., Amazon Alexa, Google Home,. . . )
– become essential components of our vision for humans interaction. The challenge in the
envisioned scenario is always related to the distributed issues, i.e., making commands possibly
understandable to a multitude of agents and vice-versa. Existing algorithms should, therefore,
be adapted for dealing with distributed and pervasive environments.</p>
      </sec>
      <sec id="sec-4-6">
        <title>4.6. Simulation</title>
        <p>
          Validation through simulation is of paramount importance when dealing with MAS, and even
more when dealing on the integration of many diferent technologies—as in our vision.
Simulation is concerned with the truthfulness of a model with respect to its problem domain and the
correctness of its construction. In other words, simulation calls for verification, i.e., “building
the system right" and validation, i.e., “building the right system" [
          <xref ref-type="bibr" rid="ref28">28</xref>
          ]. So, our vision leverages
on consolidated simulation techniques that must become a core technology to validate and
verify the final model and its properties.
        </p>
        <p>
          In the trafic management scenario, for instance, by simulating car-following in continuous
trafic flow and comparing simulation data with the data collected from the actual road, the
reliability of the model and the architecture. Many simulation models have been proposed on
the topic [
          <xref ref-type="bibr" rid="ref29 ref30 ref31">29, 30, 31</xref>
          ].
        </p>
        <p>Also in this field, the simulation models must be able to mix ingredients from the techniques
above discussed (symbolic and sub-symbolic) and should be adapted to meet the software
engineering requirements these techniques refer to.</p>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>5. Conclusions</title>
      <p>The heterogeneous nature of intelligence required by pervasive AI systems along with fears
related to sub-symbolic techniques more and more exploited in such systems require for models
and technologies guaranteeing explainability as one of the main requirement.</p>
      <p>Within this context, we believe that argumentation can play a major role, as it allows debates
to be studied and analysed, reasoning and persuasion to be exploited in dialogues, with a
well-grounded theoretical framework.</p>
      <p>In this paper, we show how diferent dialogues can occur in such pervasive contexts,
highlighting the advantages of employing argumentation. Interpretability of decision making, tolerance
to uncertainty, adaptiveness, robustness of the system, and improved trust by end-users and
amongst interacting components, are the most notable benefits of the proposed approach.</p>
      <p>The main limit of the argumentation approach is the typical assumption to have an external
judge or authority that has to control the whole argumentation process, but this is very unlikely
in a dynamic, distributed scenario like the one we propose. This aspect will certainly be the
subject of future work.</p>
      <p>However, model and techniques that should play a key role in the engineering of intelligent
systems – even if with enhancements and extensions – have been discussed and constitute a
starting point for further researches.</p>
    </sec>
    <sec id="sec-6">
      <title>Acknowledgments</title>
      <p>Roberta Calegari and Giovanni Sartor have been supported by the H2020 ERC Project
“CompuLaw” (G.A. 833647). Andrea Omicini has been supported by the H2020 Project “AI4EU” (G.A.
825619).</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>G.</given-names>
            <surname>Sartor</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Casanovas</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Casellas</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Rubino</surname>
          </string-name>
          ,
          <article-title>Computable models of the law and ICT: State of the art and trends in european research</article-title>
          ,
          <source>in: Computable Models of the Law</source>
          , Springer,
          <year>2008</year>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>20</lpage>
          . doi:
          <volume>10</volume>
          .1007/978-3-
          <fpage>540</fpage>
          -85569-
          <issue>9</issue>
          _
          <fpage>1</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>M. J.</given-names>
            <surname>Wooldridge</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N. R.</given-names>
            <surname>Jennings</surname>
          </string-name>
          ,
          <source>Intelligent agents: Theory and practice</source>
          ,
          <source>The Knowledge Engineering Review</source>
          <volume>10</volume>
          (
          <year>1995</year>
          )
          <fpage>115</fpage>
          -
          <lpage>152</lpage>
          . doi:
          <volume>10</volume>
          .1017/S0269888900008122.
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>R.</given-names>
            <surname>Calegari</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G.</given-names>
            <surname>Ciatto</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            <surname>Denti</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Omicini</surname>
          </string-name>
          ,
          <article-title>Logic-based technologies for intelligent systems: State of the art and perspectives</article-title>
          ,
          <source>Information</source>
          <volume>11</volume>
          (
          <year>2020</year>
          )
          <fpage>1</fpage>
          -
          <lpage>29</lpage>
          . doi:
          <volume>10</volume>
          .3390/ info11030167, Special Issue “10th Anniversary of Information-Emerging Research Challenges”.
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>F.</given-names>
            <surname>Xhafa</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Patnaik</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Tavana</surname>
          </string-name>
          ,
          <source>Advances in Intelligent Systems and Interactive Applications</source>
          , volume
          <volume>1084</volume>
          , Springer International Publishing,
          <year>2020</year>
          . doi:
          <volume>10</volume>
          .1007/ 978-3-
          <fpage>030</fpage>
          -34387-3.
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>C.</given-names>
            <surname>Savaglio</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Ganzha</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Paprzycki</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Bădică</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Ivanović</surname>
          </string-name>
          , G. Fortino,
          <article-title>Agent-based internet of things: State-of-the-art and research challenges</article-title>
          ,
          <source>Future Generation Computer Systems</source>
          <volume>102</volume>
          (
          <year>2020</year>
          )
          <fpage>1038</fpage>
          -
          <lpage>1053</lpage>
          . doi:
          <volume>10</volume>
          .1016/j.future.
          <year>2019</year>
          .
          <volume>09</volume>
          .016.
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>G.</given-names>
            <surname>Andrighetto</surname>
          </string-name>
          , G. Governatori,
          <string-name>
            <given-names>P.</given-names>
            <surname>Noriega</surname>
          </string-name>
          ,
          <string-name>
            <surname>L. W. van der Torre</surname>
          </string-name>
          ,
          <article-title>Normative multi-agent systems</article-title>
          , volume
          <volume>4</volume>
          of
          <string-name>
            <given-names>Dagstuhl</given-names>
            <surname>Follow-Ups</surname>
          </string-name>
          ,
          <source>Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik</source>
          ,
          <year>2013</year>
          . URL: https://drops.dagstuhl.de/opus/volltexte/dfu-complete/dfu-vol4-complete.pdf.
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>W. W.</given-names>
            <surname>Vasconcelos</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Sabater</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Sierra</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Querol</surname>
          </string-name>
          ,
          <article-title>Skeleton-based agent development for electronic institutions</article-title>
          ,
          <source>in: 1st International Joint Conference on Autonomous Agents and Multiagent Systems: Part 2 (AAMAS '02)</source>
          , ACM, New York, NY, USA,
          <year>2002</year>
          , pp.
          <fpage>696</fpage>
          -
          <lpage>703</lpage>
          . doi:
          <volume>10</volume>
          .1145/544862.544911.
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>J.</given-names>
            <surname>Rios-Torres</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. A.</given-names>
            <surname>Malikopoulos</surname>
          </string-name>
          ,
          <article-title>A survey on the coordination of connected and automated vehicles at intersections and merging at highway on-ramps</article-title>
          ,
          <source>IEEE Transactions on Intelligent Transportation Systems</source>
          <volume>18</volume>
          (
          <year>2017</year>
          )
          <fpage>1066</fpage>
          -
          <lpage>1077</lpage>
          . doi:
          <volume>10</volume>
          .1109/TITS.
          <year>2016</year>
          .
          <volume>2600504</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>V.</given-names>
            <surname>Gradinescu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Gorgorin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Diaconescu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Cristea</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Iftode</surname>
          </string-name>
          ,
          <article-title>Adaptive trafic lights using car-to-car communication</article-title>
          ,
          <source>in: IEEE 65th Vehicular Technology Conference (VTC2007)</source>
          ,
          <year>2007</year>
          , pp.
          <fpage>21</fpage>
          -
          <lpage>25</lpage>
          . doi:
          <volume>10</volume>
          .1109/VETECS.
          <year>2007</year>
          .
          <volume>17</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>A.</given-names>
            <surname>Omicini</surname>
          </string-name>
          ,
          <article-title>SODA: Societies and infrastructures in the analysis and design of agent-based systems</article-title>
          , in: P. Ciancarini,
          <string-name>
            <given-names>M. J.</given-names>
            <surname>Wooldridge</surname>
          </string-name>
          (Eds.),
          <source>Agent-Oriented Software Engineering</source>
          , volume
          <volume>1957</volume>
          <source>of Lecture Notes in Computer Science</source>
          , Springer-Verlag,
          <year>2001</year>
          , pp.
          <fpage>185</fpage>
          -
          <lpage>193</lpage>
          . doi:
          <volume>10</volume>
          .1007/3-540-44564-1_
          <fpage>12</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>A.</given-names>
            <surname>Omicini</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Ricci</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Viroli</surname>
          </string-name>
          ,
          <article-title>Artifacts in the A&amp;A meta-model for multi-agent systems</article-title>
          ,
          <source>Autonomous Agents and Multi-Agent Systems</source>
          <volume>17</volume>
          (
          <year>2008</year>
          )
          <fpage>432</fpage>
          -
          <lpage>456</lpage>
          . doi:
          <volume>10</volume>
          .1007/ s10458-008-9053-x, Special Issue on Foundations,
          <article-title>Advanced Topics and Industrial Perspectives of Multi-Agent Systems</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>R.</given-names>
            <surname>Rubino</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Omicini</surname>
          </string-name>
          , E. Denti,
          <article-title>Computational institutions for modelling norm-regulated MAS: An approach based on coordination artifacts</article-title>
          , in: O.
          <string-name>
            <surname>Boissier</surname>
            ,
            <given-names>J. A.</given-names>
          </string-name>
          <string-name>
            <surname>Padget</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          <string-name>
            <surname>Dignum</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          <string-name>
            <surname>Lindemann</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          <string-name>
            <surname>Matson</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Ossowski</surname>
            ,
            <given-names>J. S.</given-names>
          </string-name>
          <string-name>
            <surname>Sichman</surname>
          </string-name>
          , J. Vázquez-Salceda (Eds.), Coordination, Organizations, Institutions, and
          <article-title>Norms in Multi-Agent Systems</article-title>
          , volume
          <volume>3913</volume>
          of Lecture Notes in Computer Science, Springer Berlin Heidelberg,
          <year>2006</year>
          , pp.
          <fpage>127</fpage>
          -
          <lpage>141</lpage>
          . doi:
          <volume>10</volume>
          .1007/11775331_9.
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <surname>A. Perreau de Pinninck</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          <string-name>
            <surname>Sierra</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          <string-name>
            <surname>Schorlemmer</surname>
          </string-name>
          ,
          <article-title>Distributed norm enforcement: Ostracism in open multi-agent systems</article-title>
          , in: P. Casanovas, G. Sartor,
          <string-name>
            <given-names>N.</given-names>
            <surname>Casellas</surname>
          </string-name>
          , R. Rubino (Eds.),
          <source>Computable Models of the Law</source>
          , Springer,
          <year>2008</year>
          , pp.
          <fpage>275</fpage>
          -
          <lpage>290</lpage>
          . doi:
          <volume>10</volume>
          .1007/ 978-3-
          <fpage>540</fpage>
          -85569-9_
          <fpage>18</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>L.</given-names>
            <surname>Denant-Boemont</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Masclet</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C. N.</given-names>
            <surname>Noussair</surname>
          </string-name>
          ,
          <article-title>Punishment, counterpunishment and sanction enforcement in a social dilemma experiment</article-title>
          ,
          <source>Economic Theory</source>
          <volume>33</volume>
          (
          <year>2007</year>
          )
          <fpage>145</fpage>
          -
          <lpage>167</lpage>
          . doi:
          <volume>10</volume>
          .1007/s00199-007-0212-0.
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>A.</given-names>
            <surname>Omicini</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Calegari</surname>
          </string-name>
          ,
          <article-title>Injecting (micro)intelligence in the IoT: Logic-based approaches for (M)MAS</article-title>
          , in: D.
          <string-name>
            <surname>Lin</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          <string-name>
            <surname>Ishida</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          <string-name>
            <surname>Zambonelli</surname>
          </string-name>
          , I. Noda (Eds.), Massively
          <string-name>
            <surname>Multi-Agent Systems</surname>
            <given-names>II</given-names>
          </string-name>
          , volume
          <volume>11422</volume>
          of Lecture Notes in Computer Science, Springer,
          <year>2019</year>
          , pp.
          <fpage>21</fpage>
          -
          <lpage>35</lpage>
          . doi:
          <volume>10</volume>
          .1007/978-3-
          <fpage>030</fpage>
          -20937-
          <issue>7</issue>
          _
          <fpage>2</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>R.</given-names>
            <surname>Calegari</surname>
          </string-name>
          ,
          <article-title>Micro-Intelligence for the IoT: Logic-based Models and Technologies</article-title>
          ,
          <source>Ph.D. thesis, Alma Mater Studiorum-Università di Bologna</source>
          , Bologna, Italy,
          <year>2018</year>
          . doi:
          <volume>10</volume>
          . 6092/unibo/amsdottorato/8521.
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <given-names>A.</given-names>
            <surname>Clark</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D. J.</given-names>
            <surname>Chalmers</surname>
          </string-name>
          ,
          <article-title>The extended mind</article-title>
          ,
          <source>Analysis</source>
          <volume>58</volume>
          (
          <year>1998</year>
          )
          <fpage>7</fpage>
          -
          <lpage>19</lpage>
          . URL: https: //www.jstor.org/stable/3328150.
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>J.</given-names>
            <surname>Hollan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            <surname>Hutchins</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Kirsh</surname>
          </string-name>
          ,
          <article-title>Distributed cognition: Toward a new foundation for humancomputer interaction research, ACM Transactions on Computer-Human Interaction (TOCHI) 7 (</article-title>
          <year>2000</year>
          )
          <fpage>174</fpage>
          -
          <lpage>196</lpage>
          . doi:
          <volume>10</volume>
          .1145/353485.353487.
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <given-names>S.</given-names>
            <surname>Kraus</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R. C.</given-names>
            <surname>Arkin</surname>
          </string-name>
          , Strategic Negotiation in Multiagent Environments, Intelligent Robotics and Autonomous Agents, MIT Press,
          <year>2001</year>
          . URL: https://mitpress.mit.edu/books/ strategic-negotiation
          <article-title>-multiagent-environments.</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>L.</given-names>
            <surname>Amgoud</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Dimopoulos</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Moraitis</surname>
          </string-name>
          ,
          <article-title>A unified and general framework for argumentation-based negotiation</article-title>
          ,
          <source>in: 6th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS</source>
          <year>2007</year>
          ), ACM,
          <year>2007</year>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>8</lpage>
          . doi:
          <volume>10</volume>
          .1145/ 1329125.1329317.
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <given-names>I.</given-names>
            <surname>Rahwan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S. D.</given-names>
            <surname>Ramchurn</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N. R.</given-names>
            <surname>Jennings</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Mcburney</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Parsons</surname>
          </string-name>
          , L. Sonenberg,
          <article-title>Argumentation-based negotiation</article-title>
          ,
          <source>The Knowledge Engineering Review</source>
          <volume>18</volume>
          (
          <year>2003</year>
          )
          <fpage>343</fpage>
          -
          <lpage>375</lpage>
          . doi:
          <volume>10</volume>
          .1017/S0269888904000098.
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <string-name>
            <given-names>R.</given-names>
            <surname>Calegari</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G.</given-names>
            <surname>Ciatto</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Omicini</surname>
          </string-name>
          ,
          <article-title>On the integration of symbolic and sub-symbolic techniques for XAI: A survey</article-title>
          ,
          <source>Intelligenza Artificiale</source>
          <volume>14</volume>
          (
          <year>2020</year>
          )
          <fpage>7</fpage>
          -
          <lpage>32</lpage>
          . doi:
          <volume>10</volume>
          .3233/IA-190036.
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23]
          <string-name>
            <given-names>P.</given-names>
            <surname>Stone</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Veloso</surname>
          </string-name>
          ,
          <article-title>Multiagent systems: A survey from a machine learning perspective</article-title>
          ,
          <source>Autonomous Robots</source>
          <volume>8</volume>
          (
          <year>2000</year>
          )
          <fpage>345</fpage>
          -
          <lpage>383</lpage>
          . doi:
          <volume>10</volume>
          .1023/A:
          <fpage>1008942012299</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [24]
          <string-name>
            <given-names>S.</given-names>
            <surname>Mariani</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Omicini</surname>
          </string-name>
          ,
          <article-title>Coordination in situated systems: Engineering MAS environment in TuCSoN</article-title>
          , in: G. Fortino,
          <string-name>
            <given-names>G. Di</given-names>
            <surname>Fatta</surname>
          </string-name>
          ,
          <string-name>
            <given-names>W.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Ochoa</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Cuzzocrea</surname>
          </string-name>
          , M. Pathan (Eds.),
          <source>Internet and Distributed Computing Systems</source>
          , volume
          <volume>8729</volume>
          of Lecture Notes in Computer Science, Springer International Publishing,
          <year>2014</year>
          , pp.
          <fpage>99</fpage>
          -
          <lpage>110</lpage>
          . doi:
          <volume>10</volume>
          .1007/978-3-
          <fpage>319</fpage>
          -11692-
          <issue>1</issue>
          _
          <fpage>9</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          [25]
          <string-name>
            <given-names>J.</given-names>
            <surname>Hulstijn</surname>
          </string-name>
          ,
          <string-name>
            <surname>L. W. van der Torre</surname>
          </string-name>
          ,
          <article-title>Combining goal generation and planning in an argumentation framework</article-title>
          , in: International Workshop on Non-monotonic
          <source>Reasoning (NMR'04)</source>
          ,
          <year>2004</year>
          , pp.
          <fpage>212</fpage>
          -
          <lpage>218</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          [26]
          <string-name>
            <given-names>S.</given-names>
            <surname>Modgil</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Caminada</surname>
          </string-name>
          ,
          <article-title>Proof theories and algorithms for abstract argumentation frameworks</article-title>
          ,
          <source>in: Argumentation in Artificial Intelligence</source>
          , Springer,
          <year>2009</year>
          , pp.
          <fpage>105</fpage>
          -
          <lpage>129</lpage>
          . doi:
          <volume>10</volume>
          .1007/978-0-
          <fpage>387</fpage>
          -98197-
          <issue>0</issue>
          _
          <fpage>6</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          [27]
          <string-name>
            <given-names>R.</given-names>
            <surname>Calegari</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G.</given-names>
            <surname>Contissa</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Lagioia</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Omicini</surname>
          </string-name>
          , G. Sartor,
          <article-title>Defeasible systems in legal reasoning: A comparative assessment</article-title>
          , in: M.
          <string-name>
            <surname>Araszkiewicz</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          <string-name>
            <surname>Rodríguez-Doncel</surname>
          </string-name>
          (Eds.),
          <source>Legal Knowledge and Information Systems. JURIX</source>
          <year>2019</year>
          :
          <article-title>The Thirty-second Annual Conference</article-title>
          , volume
          <volume>322</volume>
          <source>of Frontiers in Artificial Intelligence and Applications</source>
          , IOS Press,
          <year>2019</year>
          , pp.
          <fpage>169</fpage>
          -
          <lpage>174</lpage>
          . doi:
          <volume>10</volume>
          .3233/FAIA190320.
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          [28]
          <string-name>
            <given-names>D.</given-names>
            <surname>Calvaresi</surname>
          </string-name>
          , G. Albanese,
          <string-name>
            <given-names>J.-P.</given-names>
            <surname>Calbimonte</surname>
          </string-name>
          ,
          <string-name>
            <surname>M.</surname>
          </string-name>
          <article-title>Schumacher, SEAMLESS: Simulation and analysis for multi-agent system in time-constrained environments, in: Advances in Practical Applications of Agents, Multi-Agent Systems, and Trustworthiness</article-title>
          .
          <source>The PAAMS Collection</source>
          , Springer International Publishing, Cham,
          <year>2020</year>
          , pp.
          <fpage>392</fpage>
          -
          <lpage>397</lpage>
          . doi:
          <volume>10</volume>
          .1007/ 978-3-
          <fpage>030</fpage>
          -49778-1_
          <fpage>30</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          [29]
          <string-name>
            <given-names>M.</given-names>
            <surname>Mashayekhi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Du</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G. F.</given-names>
            <surname>List</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. P.</given-names>
            <surname>Singh</surname>
          </string-name>
          ,
          <string-name>
            <surname>Silk:</surname>
          </string-name>
          <article-title>A simulation study of regulating open normative multiagent systems</article-title>
          ., in: 25th
          <source>International Joint Conference on Artificial Intelligence (IJCAI-16)</source>
          ,
          <year>2016</year>
          , pp.
          <fpage>373</fpage>
          -
          <lpage>379</lpage>
          . URL: https://www.ijcai.org/Proceedings/16/ Papers/060.pdf.
        </mixed-citation>
      </ref>
      <ref id="ref30">
        <mixed-citation>
          [30]
          <string-name>
            <given-names>J.</given-names>
            <surname>Dai</surname>
          </string-name>
          ,
          <string-name>
            <surname>X.</surname>
          </string-name>
          <article-title>Li, Multi-agent systems for simulating trafic behaviors</article-title>
          ,
          <source>Chinese Science Bulletin</source>
          <volume>55</volume>
          (
          <year>2010</year>
          )
          <fpage>293</fpage>
          -
          <lpage>300</lpage>
          . doi:
          <volume>10</volume>
          .1007/s11434-009-0230-3.
        </mixed-citation>
      </ref>
      <ref id="ref31">
        <mixed-citation>
          [31]
          <string-name>
            <given-names>F.</given-names>
            <surname>Zhang</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Li</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Q.</given-names>
            <surname>Zhao</surname>
          </string-name>
          ,
          <article-title>Single-lane trafic simulation with multi-agent system</article-title>
          ,
          <source>in: 2005 IEEE Intelligent Transportation Systems</source>
          , IEEE,
          <year>2005</year>
          , pp.
          <fpage>56</fpage>
          -
          <lpage>60</lpage>
          . doi:
          <volume>10</volume>
          .1109/ITSC.
          <year>2005</year>
          .
          <volume>1520219</volume>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>