<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>MAS-DRiVe: a Practical Approach to Decentralized Runtime Verification of Agent Interaction Protocols</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Davide Ancona</string-name>
          <email>davide.ancona@unige.it</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Daniela Briola</string-name>
          <email>daniela.briola@unimib.it</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Angelo Ferrando</string-name>
          <email>angelo.ferrando@dibris.unige.it</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Viviana Mascardi</string-name>
          <email>viviana.mascardi@unige.it</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>D. Ancona, A. Ferrando, and V. Mascardi are with DIBRIS, University of Genova</institution>
          ,
          <country country="IT">Italy</country>
        </aff>
      </contrib-group>
      <fpage>35</fpage>
      <lpage>43</lpage>
      <abstract>
        <p>-We address the problem of decentralized runtime verification of interaction protocols in multiagent systems by means of MAS-DRiVe, an algorithm for partitioning a multiagent system (MAS) into sub-MASs which can be monitored independently. Given a global interaction protocol named AIP (for A“gent Interaction Protocol”) describing all the interactions which can take place in the MAS, the MAS-DRiVe algorithm extracts the interaction graph from AIP , identifies the clusters of agents which cannot be split during the decentralized monitoring as the interactions they are involved in are not independent, collapses each of those clusters into a single node in the interaction graph, and finally partitions the collapsed graph obtained so far. Although the “unsplittable agents identification” stage is still in its early design and prototyping phases and requires a better formalization and a deeper analysis, the MAS-DRiVe algorithm pipeline has been fully implemented and demonstrated on two simple MASs. Once the independently monitorable sub-MASs have been identified by MAS-DRiVe, the global interaction protocol AIP can be projected onto the subsets of agents belonging to each graph partition, thus obtaining local versions of AIP which can be monitored in a decentralized way.</p>
      </abstract>
      <kwd-group>
        <kwd>Agent Interaction Protocol</kwd>
        <kwd>Decentralized Runtime Verification</kwd>
        <kwd>Decentralized Monitoring</kwd>
        <kwd>Graph Partitioning</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>I. INTRODUCTION AND MOTIVATION</title>
      <p>Runtime verification (RV) is a software verification
technique that complements formal static verification and testing.</p>
      <p>In RV, dynamic checking of the correct behavior of a system
is performed by a monitor which is generated from a formal
specification of the properties to be verified.</p>
      <p>Distributed runtime verification (DRV), as described by S.</p>
      <p>Rajsbaum in his keynote speech at the SSS 2015 Symposium
[25], tackles the problem of building a decentralized runtime
monitor for a distributed system and involves designing a
distributed algorithm that coordinates the monitors in order
for them to correctly verify the dynamic behavior of the whole
system.</p>
      <p>With respect to RV, DRV techniques are more complex and
less developed, since they involve designing a distributed
algorithm that monitors another distributed algorithm.
Nevertheless, they are gaining more and more attention, as shown by the
recent Distributed Runtime Verification Workshop organized
in Bertinoro in May 20161.</p>
      <p>Once the formal specification of the global pattern of events
is given, however, distributing the monitoring activity can be
resorted to decomposing the global specification into
“subspecifications”, involving less events than the global one,
which can be monitored in an independent way from each
other and such that the union of their independent monitoring
gives the same guarantees as the monitoring of the whole
system w.r.t. the original specification.</p>
      <p>Given a global specification of the expected system behavior
S, the challenge to face is then to compute a set of n
subspecifications S1, . . . , Sn derived from S in such a way that
the following properties hold:
• decentralized monitoring against the sub-specifications</p>
      <p>S1, . . . , Sn is equivalent to centralized monitoring against</p>
      <p>S;
• the decomposition of S into S1, . . . , Sn is balanced
according to some weighting criterion, to get, ideally, the
best monitoring performances.</p>
      <p>These sub-specifications do not necessarily induce a partition
of the monitored events: indeed, it might be necessary that
some monitored events belong to more than one specification,
and hence must be checked by more monitors. Let us suppose
for example that the formal specification of the event pattern
states that when the alarm rings, Mary stops it and either she
wakes up, washes, has breakfast and goes to work (then the
monitoring is allowed to stop), or she continues to sleep.</p>
      <p>The events “wake up”, “wash”, “have breakfast”, “go to
work” are independent from the event “sleep”, so a good way
to decentralize the monitoring activity between two monitors
m1 and m2 could be to have m1 in charge of the “active”
behavior (m1 would check that “when the alarm rings, Mary
stops it and then either she wakes up, washes, has breakfast,
goes to work – then the monitoring can stop –, or the
monitoring can stop immediately”) and m2 in charge of the lazy
one (m2 would check that “when the alarm rings, Mary stops
it and then either she continues to sleep, or the monitoring can
stop immediately”).</p>
      <p>It would be convenient that the events “the alarm rings” and
“Mary stops it” were monitored by both m1 and m2, in order
to avoid a further monitor ensuring the correct sequence of
initial events, whatever Mary’s attitude towards the new day.</p>
      <p>Due to events which may be monitored by more than
one monitor, the total amount of monitored events in the
decentralized setting may be larger than in the centralized
case. However, decentralized monitors can be run in parallel
on different machines, so the workload is more balanced
than in the centralized setting. Also, sometimes a centralized
approach is unfeasible, as in the motivating scenario discussed
in our previous work presented at EMAS 2014 [2]. There, we of agents in a MAS which provide formal guarantees that
supposed that the safety of a humanitarian convoy was guar- verification through projected types and decentralized monitors
anteed by unmanned aerial vehicles whose behavior is critical is equivalent to verification performed by a single centralized
for the success of the mission and hence subject to runtime monitor with a single global type.
verification, but which might not be monitorable by a single This paper addresses that problem by presenting
MASmonitor for network connectivity reasons and for failures of DRiVe, an algorithm for partitioning a MAS in such a way
some computational entities. In cases like that, decentralizing that the identified subsystems can be safely monitored in
the runtime verification activity is no longer an option, but a decentralized way. The paper is structured as follows:
the only viable solution. The possibility to exchange protocol Section II discusses the related work, Section III introduces
specifications, dealt with as first class entities that can be sent background knowledge, Section IV presents the design of
and received [3], allows us to apply our approach to oppor- MAS-DRiVe, Section V discusses its implementation and
tunistic networks (Wireless Sensor Networks, drone ferries, experiments, Section VI concludes.
and the like) where the actual communication protocols are
unknown until deployment, and mobility dynamically changes II. RELATED WORK
them.</p>
      <p>Finding a set of sub-specifications S1, . . . , Sn of S DRV lies at the crossroad where decentralized algorithms
meeting the requirements for decentralizing the monitoring and formal methods meet [25]. It must cope both with the
activity heavily depends on the way S has been specified. problems intrinsic to distributed systems, and with those
RV specifications are typically expressed using trace predicate related to formalization of properties to be verified at runtime.
formalisms. Many of them, such as finite state machines, reg- Furthermore, the distribution of the monitoring activity itself
ular expressions, context free grammars, linear-time temporal poses new problems and raises new challenges.
(LTL) logic, have been originally introduced for other aims One of the most severe and most studied problems which
and then exploited for RV; others, such as trace expressions characterize decentralized systems is the lack of a global clock
[6], have been expressly devised for RV. Different formalisms and the need to tag events with a timestamp which, although
may have different expressive power, which impacts on the local to the node where the event took place or was observed,
types of patterns that can be described and on the meaning of can be compared with the timestamps of the other events to
“independent” sub-specifications. check their relative order. In this paper we assume that each</p>
      <p>Although the MAS-DRiVe algorithm proposed in this paper monitor observes the events it is in charge of, in the same
for decentralizing the monitoring activity is general enough to order as they actually took place in the environment. This
be adopted whatever the formalism and the kind of monitored condition is enough for our approach to work. As mentioned
events are, it has only been experimented for the specific in the introduction, some events may be observed by more
formalism of trace expressions, for specific decentralized than one monitor so the condition we pose implicitly means
systems, that is, multiagent systems (MASs), and for specific that event ordering is always preserved in all monitors. A large
events, namely interactions between pairs of agents. Also, as body of literature addresses this problem; the two most widely
we will see later on, the “unsplittable agents identification” is used approaches are to connect the different physical clocks
almost naif and requires a refinement and a better theoretical of the decentralized nodes to one common logical clock, by
support. exploiting causality in communication between nodes
(Lam</p>
      <p>Trace expressions are an evolution of global types [5], which ports “happened before” relation [22]), and to synchronize the
have been initially proposed for RV of interactions in MASs. local clocks to provide a full ordering of events as proposed
Trace expressions are an expressive formalism based on a set for example by M. Maro´ti et al. [23].
of operators (including prefixing, concatenation, shuffle, union, Formalizing the system properties in order to dynamically
and intersection) to denote finite and infinite traces of events. verify them requires that a suitable formal language is
availTheir semantics is based on a labeled transition system defined able. Among the first efforts in this direction we may cite one
by a simple set of rewriting rules which directly drive the paper by A. Bauer, M. Leucker, and C. Schallhart [7] where
behavior of monitors generated from them. They have been the use of LTL in the context of RV was explored. When used
used to model and monitor (in a simulated setting) both fail- for RV, the expressive power of LTL is reduced, because at
uncontrolled and ambient intelligence systems [4], as well as runtime only finite traces can be checked. To provide a formal
medical guidelines for remote patient monitoring [16]. account for this limitation, a three-valued semantics for LTL,</p>
      <p>In the already cited EMAS 2014 paper, we tackled the called LTL3, has been proposed later on [8]. A third truth
problem of projecting a global specification of an interaction value “ ?” is introduced to specify that after a finite trace of
protocol expressed in the global types formalism onto subsets events has been occurred, the outcome of a monitor can be
of agents in the MAS. The projection function is a key element inconclusive.
for moving from a global specification of the protocol to a The generation of efficient monitors for properties specified
localized one, which involves less agents and hence hopefully in a variety of formal languages was firstly addressed by
less interaction events. However, in our previous work we M. Kim et al. [21, 18] and many proposals exist for RV
faced only one of the issues involved in distributing the of object-oriented systems (jassda [11], PQL [24], JavaMOP
RV activity, that of projecting the interaction protocol, and [13], LARVA [14], SAGA [15], just to cite a few) and MASs
did not consider the problem of finding suitable partitions [1, 17]. A large community of researchers works on RV
and in 2001 the Runtime Verification Workshop/Conference
series was initiated (http://www.runtime-verification.org), and
workshops/conferences have occurred each year since then.</p>
      <p>When moving from RV to DRV, however, the situation
dramatically changes: no introductory papers and surveys on
the topic exist, which demonstrates the youth of this research
field. W.r.t. DRV of MASs, we were not able to find any related
work except for one paper of ours [10] where the projection
mechanism introduced in our previous EMAS 2014 work had
been integrated with JADE [9].</p>
    </sec>
    <sec id="sec-2">
      <title>III. BACKGROUND In this section we introduce the three pillars of our work: trace expressions, projection, and graph partitioning.</title>
      <sec id="sec-2-1">
        <title>A. Trace Expressions</title>
        <p>Trace expressions are a specification formalism expressly
designed for RV. They are based on the notions of event and
its abstraction, event type.</p>
        <p>Events. In the following we denote by E a fixed universe of
events. An event trace over E is a possibly infinite sequence
of events in E. As an example, we might have E = {
msg(bob, alice, tell, m1), msg(alice, bob, tell, a1), msg(bob,
carol, tell, m2), msg(carol, bob, tell, a2), msg(bob, dave, tell,
m3), msg(dave, bob, tell, a3) } where the interaction event
msg(S, R, P, C) corresponds to the interaction between agent
S and agent R, with S sending a message with performative
P and content C to R. In this example, contents m1, m2 and
m3 correspond to actual messages, and contents a1, a2 and
a3 correspond to acknowledges of reception.
• τ1·τ2 (concatenation), denoting the set of all traces
obtained by concatenating the traces of τ1 with those of
τ2.
• τ1∧τ2 (intersection), denoting the intersection of the
traces of τ1 and τ2.
• τ1∨τ2 (union), denoting the union of the traces of τ1 and
τ2.
• τ1|τ2 (shuffle ), denoting the set obtained by shuffling the
traces in τ1 with the traces in τ2.</p>
        <p>To support recursion without introducing an explicit construct,
trace expressions are regular (a.k.a. rational or cyclic) terms.
A regular term can be represented by a finite set of syntactic
equations, as happens, for instance, in most modern Prolog
implementations where unification supports cyclic terms. A
trace expression τ is contractive if all its infinite paths from
the root contain the prefix operator.</p>
        <p>From the operators above we can derive the filter
operator, useful for making trace expressions more compact and
readable. The expression ϑ τ denotes the set of all traces
contained in τ , when deprived of all events that do not match
ϑ. Assuming that event types are closed by complementation,
the expression above is a convenient syntactic shortcut for T |τ ,
where T = ∨ϑ:T , and ϑ is the complement event type of ϑ,
that is, JϑK = E \ JϑK.</p>
        <p>In the context of RV, where the three-valued semantics
is considered for LTL, trace expressions are strictly more
expressive than LTL [6]: every LTL formula can be encoded
into a trace expression with an equivalent three-valued
semantics, whereas the opposite property does not hold, since trace
expressions are also able to specify all context-free languages,
and also some non context-free ones.</p>
        <p>Although trace expressions can define protocols involving
events of any kind, in the sequel we will limit our investigation
to interaction events like msg(S, R, P, C) and interaction types
that we will identify with α.</p>
      </sec>
      <sec id="sec-2-2">
        <title>B. Projection</title>
        <p>Event types. To be more general, trace expressions are
built on top of event types (chosen from a set ET), rather
than of single events; an event type denotes a subset of E.</p>
        <p>For example, if we were interested only in the type of the
message content (actual message or acknowledge), we might
define the two event types Msg= { msg(bob, alice, tell,
m1), msg(bob, carol, tell, m2), msg(bob, dave, tell, m3) }
and Ack= { msg(alice, bob, tell, a1), msg(carol, bob, tell,
a2), msg(dave, bob, tell, a3) }, and use them for describing
interaction patterns.</p>
        <p>The projection function was first introduced in [2]. Given
the finite set AGS of all the agents that could play a role in
the MAS and an interaction type α, senders(α) is the set of
all the agents in AGS that could play the role of sender in
actual interactions having type α, and receivers(α) is the set
of all the agents in AGS that could play the role of receiver</p>
        <p>Trace expressions. A trace expression τ represents a set of in interactions of type α. The involves predicate holds on one
possibly infinite event traces, and is defined on top of the interaction type α and one set of agents Ags, involves(α, Ags),
following operators (binary operators associate from left, and iff (senders(α) ∩ Ags 6= ∅) ∨ (receivers(α) ∩ Ags 6= ∅).
are listed in decreasing order of precedence, that is, the first Projection can be described as a function Π : T ×
operator has the highest precedence): P(AGS) → T where T is the set of trace expressions. Π is
• (empty trace), denoting the singleton set { } containing driven by the syntax of the trace expression to project; since
the empty event trace . Π is defined on cyclic terms, the simplest way to define it
• ϑ:τ (prefix ), denoting the set of all traces whose first event would be by coinduction2 as follows:
e matches the event type ϑ (e ∈ ϑ), and the remaining (i) Π( , Ags) =
part is a trace of τ . For example, if our communication
protocol just required that an interaction from the set 2Coinduction is a technique for defining and proving properties of systems
Msg must take place, no matter which one, and then an of concurrent interacting objects. It is the mathematical dual to structural
interaction from Ack must occur, no matter which one, isnudcuhctaiosn.stCreoaimndsu.cStieveelyhtdtpe:fi//ncseedwteybp.uescsadr.eedtuy/pgircoaullpys/tiantfia mniit/ehadnadtdaesmtrousc/tduorce/s,
we could express it as Msg:Ack: . coind.htm for a more technical formalization.
(ii) Π(α : τ, Ags) = α : Π(τ, Ags) if involves(α, Ags)
(iii) Π(α : τ, Ags) = Π(τ, Ags) if ¬involves(α, Ags)
(iv) Π(τ 0 op τ 00, Ags) = Π(τ 0, Ags) op Π(τ 00, Ags), where
op ∈ {·, ∧, ∨, |}.</p>
        <p>This definition works properly only on all non cyclic terms
and on some, but not all, cyclic terms. To guarantee that
the projection function always returns a contractive type and
that the correct coinductive definition is implemented, it is
necessary to keep track of all types visited by Π along a path;
each type is associated with its depth in the path, and with
a fresh variable which will be unified with the corresponding
computed projection. During the visit, the depth DeepestSeq
of the deepest visited sequence operator is kept. If a type τ has
been already visited, then a cycle is detected: if its depth is less
than DeepestSeq then the cycle contains an occurrence of the
sequence constructor, therefore the projected type associated
with τ is contractive and, hence, is returned; otherwise, the
projection would not be contractive, therefore is returned.</p>
      </sec>
      <sec id="sec-2-3">
        <title>C. Graph Partitioning</title>
        <p>The graph partitioning problem (GPP) is defined in the
following way [12]: given a number k ∈ N&gt;1 and an
undirected graph G = (V, E) with non-negative node and edge
weights, GPP asks for a partition of V , that is, with blocks of
nodes (V1, ..., Vk) such that
all edges in the CI G have weight 1, but the approach
works also in case edge weights are different: a higher
edge weight would model a preferential communication
channel;
4) partition the new graph obtained so far using some
suitable graph partition method;
5) project the interaction protocol onto these agents’
partitions.</p>
        <p>The algorithm is explained by means of two running
examples introduced in [2].</p>
      </sec>
      <sec id="sec-2-4">
        <title>A. Examples</title>
        <p>1) V1 ∪ V2 ∪ ... ∪ Vk = V
2) Vi ∩ Vj = ∅ ∀i 6= j</p>
        <p>A balance constraint may be required, demanding that all
blocks have about equal weights. To be more precise, a good Fig. 1. The “Socks and Shoes” MAS
partition is the one where the sum of the node weights
in each Vi is “about the same” and the sum of all edge a) Socks and Shoes (SaS): Let us consider the simple
weights of edges connecting all different pairs Vi and Vj is scenario illustrated in Figure 1 where two robots (right
minimized. Applications of graph partitioning include parallel and left), two monitors (right and left) associated with
processing, road networks, image processing, VLSI design, each robot, and a plan monitor which supervises them,
social networks, and bioinformatics. Among the many existing must reach the goal to put the right and left socks and
tools for graph partitioning we opted for using METIS [20], shoes in the correct way (socks first!). As robots are
a set of serial programs for partitioning graphs, partitioning autonomous, they could perform the two actions in the wrong
finite element meshes, and producing fill reducing orderings order: monitors are there to ensure that wrong actions are
for sparse matrices (http://glaros.dtc.umn.edu/gkhome/views/ immediately rolled back. Robots communicate their actions
metis, accessed on June 2016). The choice of METIS was due to their corresponding monitors, which, in turn, notify the
to the efficiency of the multi-level Kernighan/Lin approach it plan monitor when the robots accomplish their goal. The
builds upon [19] and to the ease of its installation and usage. interaction types involving the right robot and the right node
monitor are defined as msg(right robot, right node monitor,
IV. MAS-DRIVE: DESIGN tell, put sock) ∈ put right sock; msg(right robot,
right node monitor, tell, put shoe) ∈ put right shoe;
msg(right robot, right node monitor, tell, removed shoe) ∈
removed right shoe; msg(right node monitor, right robot,
tell, oblige remove shoe) ∈ oblige remove right shoe;
msg(right node monitor, plan monitor, tell, ok) ∈ ok right.</p>
        <p>Those for the left robot and left node monitor are symmetrical.</p>
        <p>Given an interaction protocol expressed in some suitable
formalism, the MAS-DRiVe algorithm for partitioning a MAS
in such a way that the identified subsystems can be safely
monitored in a decentralized way is the following:
1) extract the interaction graph I G from the interaction</p>
        <p>protocol;
2) identify the sets of agents which cannot be split during
the decentralized monitoring (unsplittable agents), since
the interactions they are involved in are not independent;
3) compute the collapsed interaction graph CI G by
collapsing each of those sets into a single node with weight
equal to the cardinality of the collapsed set, and by
computing the collapsed edges consistently; for simplicity,</p>
        <p>The right robot (RIGHT branch in the protocol description
below) can start by putting the sock, which is the correct action
to do, or (∨ operator) by putting the shoe, which requires
a recovery by the right robot monitor and looping back to
the RIGHT branch. The left robot has the same behavior
(LEFT branch in the protocol description). The SaS protocol
is described by the shuffle of the actions of the right and left
verify MA2, and if an event has type msg3 or ack3 , then it
must verify MA3.</p>
      </sec>
      <sec id="sec-2-5">
        <title>B. Algorithm Steps</title>
        <p>
          As already introduced in the beginning of this section, the
MAS-DRiVe algorithm consists of five steps.
robots and monitors (RIGHT | LEFT):
(put right sock:put right shoe:ok right: )∨
(put right shoe:oblige remove right shoe:
removed right shoe:RIGHT)
(put left sock:put left shoe:ok left: )∨
(put left shoe:oblige remove left shoe:
removed left shoe:LEFT)
RIGHT|LEFT.
(msg MM )∧(msg ack (
          <xref ref-type="bibr" rid="ref1">1</xref>
          ) MA1)∧
(msg ack (
          <xref ref-type="bibr" rid="ref2">2</xref>
          ) MA2)∧(msg ack (
          <xref ref-type="bibr" rid="ref3">3</xref>
          )
msg1 :msg2 :msg3 :MM
msg1 :ack1 :MA1
msg2 :ack2 :MA2
msg3 :ack3 :MA2
where msg ack (i), i ∈ {1, 2, 3}, and msg denote the event
types s.t. Jmsg ack (i)K = Jmsg iK ∪ Jack iK, i ∈ {1, 2, 3}, and
Jmsg K = Jmsg1 K ∪ Jmsg2 K ∪ Jmsg3 K.
        </p>
        <p>The trace expression defined by MM corresponds to the
first constraint informally stated above, while MA1, MA2,
and MA3 formalize the second constraint. The main trace
expression ABP3 can be easily read as follows: if an event
has type msg1 or msg2 or msg3 , then it must verify MM ,
and if an event has type msg1 or ack1 , then it must verify
MA1, and if an event has type msg2 or ack2 , then it must
extract(ID, AlreadyMetTrExp, Str, Level, τ ) is defined by cases:
1) if TrExpr belongs to AlreadyMetTrExp, then return;
2) if TrExpr is , then return;
3) if τ is α uop τ1, where uop is an operator in {:, },
then
a) NewStr = Str • (uop, Level+1)
b) extract all the possible pairs of senders and
receivers (S, R) such that S and R are involved in
α
c) for each of such pairs, generate and assert edge(ID,</p>
      </sec>
      <sec id="sec-2-6">
        <title>S, R, NewStr)</title>
        <p>d) NewAlreadyMetTrExp = {τ } S AlreadyMetTrExp</p>
      </sec>
      <sec id="sec-2-7">
        <title>e) call extract(ID, NewAlreadyMetTrExp, NewStr,</title>
        <p>Level+1, τ1)
4) if τ is τ1∧τ2, then
a) NewStr = Str • (∧, Level+1)
b) NewAlreadyMetTrExp = {τ } S AlreadyMetTrExp</p>
      </sec>
      <sec id="sec-2-8">
        <title>c) call extract(ID, NewAlreadyMetTrExp, NewStr,</title>
        <p>Level+1, τ1)</p>
      </sec>
      <sec id="sec-2-9">
        <title>d) call extract(ID, NewAlreadyMetTrExp, NewStr,</title>
        <p>Level+1, τ2)
5) if τ is τ1 bop τ2, where bop is an operator in {·, ∨, |},
then
a) NewStr1 = Str • (bop, Level+1)
b) NewStr2 = Str • (bop, 2*Level+1)
c) NewAlreadyMetTrExp = {τ } S AlreadyMetTrExp</p>
      </sec>
      <sec id="sec-2-10">
        <title>d) call extract(ID, NewAlreadyMetTrExp, NewStr1,</title>
        <p>Level+1, τ1)</p>
      </sec>
      <sec id="sec-2-11">
        <title>e) call extract(ID, NewAlreadyMetTrExp, NewStr2,</title>
        <p>2*Level+1, τ2)</p>
        <p>Given a protocol identified by ID and represented by the
trace expression τ , extract(ID, {}, Λ, 1, τ ) generates and
asserts all the edges in IGext. The generation of IGext nodes
is straightforward. The idea behind this function is that edges
whose labels share the same prefix have a strong dependency
as they belong to the same intersection. In fact, the only case
where labels of edges share the same prefix is that of ∧ (case
4), where the same string and level are passed to the recursive
calls. If the operator is not an intersection (case 5), the strings
and levels passed to the recursive calls are different, meaning
that the two operands are independent from one another and
so will be the edges generated during their exploration.</p>
        <p>
          For example, the edges in the auxiliary IGext graph
extracted from the ABP3 protocol are:
(bob, alice, ((∧, 2) (
          <xref ref-type="bibr" rid="ref3">, 3</xref>
          ) (
          <xref ref-type="bibr" rid="ref4">, 4</xref>
          ) (
          <xref ref-type="bibr" rid="ref5">, 5</xref>
          )) )
        </p>
        <p>
          ∧ ∧
(bob, carol, ((∧, 2) (
          <xref ref-type="bibr" rid="ref3">, 3</xref>
          ) (
          <xref ref-type="bibr" rid="ref4">, 4</xref>
          ) (
          <xref ref-type="bibr" rid="ref5">, 5</xref>
          )) )
        </p>
        <p>
          ∧ ∧
(bob, dave, ((∧, 2) (
          <xref ref-type="bibr" rid="ref3">, 3</xref>
          ) (
          <xref ref-type="bibr" rid="ref4">, 4</xref>
          ) (
          <xref ref-type="bibr" rid="ref5">, 5</xref>
          )) )
        </p>
        <p>
          ∧ ∧
(bob, alice, ((∧, 2) (
          <xref ref-type="bibr" rid="ref3">, 3</xref>
          ) (
          <xref ref-type="bibr" rid="ref4">, 4</xref>
          ) (
          <xref ref-type="bibr" rid="ref5">, 5</xref>
          ) (:, 6)) )
        </p>
        <p>
          ∧ ∧
(bob, carol, ((∧, 2) (∧, 3) (∧, 4) (
          <xref ref-type="bibr" rid="ref5">, 5</xref>
          ) (:, 6) (:, 7)) )
(bob, dave, ((∧, 2) (∧, 3) (∧, 4) (
          <xref ref-type="bibr" rid="ref5">, 5</xref>
          ) (:, 6) (:, 7) (:, 8)) )
(alice, bob, ((∧, 2) (
          <xref ref-type="bibr" rid="ref3">, 3</xref>
          ) (∧, 4) (
          <xref ref-type="bibr" rid="ref5">, 5</xref>
          ) (:, 6) (:, 7)) )
∧
(bob, carol, ((∧, 2) (∧, 3) (
          <xref ref-type="bibr" rid="ref4">, 4</xref>
          )) )
(carol, bob, ((∧, 2) (
          <xref ref-type="bibr" rid="ref3">, 3</xref>
          ) (
          <xref ref-type="bibr" rid="ref4">, 4</xref>
          )) )
        </p>
        <p>
          ∧
(bob, carol, ((∧, 2) (
          <xref ref-type="bibr" rid="ref3">, 3</xref>
          ) (
          <xref ref-type="bibr" rid="ref4">, 4</xref>
          ) (:, 5)) )
        </p>
        <p>
          ∧
(carol, bob, ((∧, 2) (
          <xref ref-type="bibr" rid="ref3">, 3</xref>
          ) (
          <xref ref-type="bibr" rid="ref4">, 4</xref>
          ) (:, 5) (:, 6)) )
        </p>
        <p>
          ∧
(bob, dave, ((∧, 2) (
          <xref ref-type="bibr" rid="ref3">, 3</xref>
          )) )
(dave, bob, ((∧, 2) (
          <xref ref-type="bibr" rid="ref3">, 3</xref>
          )) )
(bob, dave, ((∧, 2) (
          <xref ref-type="bibr" rid="ref3">, 3</xref>
          ) (:, 4)) )
(dave, bob, ((∧, 2) (
          <xref ref-type="bibr" rid="ref3">, 3</xref>
          ) (:, 4) (:, 5)) )
The IG nodes are alice, bob, dave, carol and the edges are
(alice, bob), (dave, bob), (carol, bob), which are extracted in
a trivial way from IGext.
        </p>
      </sec>
      <sec id="sec-2-12">
        <title>2) Identification of unsplittable agents. Not all the agents</title>
        <p>can be monitored independently from one another, as some
interaction patterns that must be respected by a set of agents
seen as a whole, could be lost when looking at subsets of the
agents.</p>
        <p>As an example, alice, bob, carol and dave in the ABP3
represent an unsplittable set of agents. In fact, if we monitored
interactions involving alice and bob only, we could find that
they respect the second constraint of the protocol, but we could
never ensure that the first constraint – which also depends on
the interactions of bob with carol and dave – is respected as
well. The notions of dependence and independence are related
to the protocol but also to the formalism used for modeling
it, so no general rule for this step can be devised. When the
interaction formalism F is that of trace expressions, however,
dependencies among interactions are naturally modeled using
the intersection operator. Thus, although it may be an
overcautious approach, when using trace expressions we define
“unsplittable” the agents involved in interactions connected
by an intersection operator.</p>
        <p>
          The algorithm for performing the identification of
unsplittable agents looks at the labels in IGext and creates
sets of agents involved in interactions whose label has the
same prefix before the first∧ operator in the label, if any.
For example, in SaS no label contains ∧ and hence the set
of unsplittable agents is empty, whereas in ABP3 all the
edge labels share the (
          <xref ref-type="bibr" rid="ref2">, 2</xref>
          ) prefix, meaning that all the
∧
interactions in the protocol belong to branches connected
by the same intersection operator. As a consequence, all the
agents involved in the ABP3 protocol interactions belong to
the same set of unsplittable agents, which coincides with all
the agents in the MAS.
3) Graph collapse. In this stage, we transform the unweighted
undirected interaction graph IG into a weighted undirected
graph CIG, obtained by collapsing each node in IG
corresponding to an agent belonging to unsplittable set U , into a
single node with label U and weight equal to U cardinality.
Nodes in CIG are labeled with sets of agents, which can be
the singleton set for agents belonging to no unsplittable set.
Edges exiting from (resp. entering into) a node are all those
which, in IG, exited from (resp. entered into) one of the nodes
in the label.
        </p>
        <p>Considering the running examples above, the
weighted nodes in the SaS CIG are ({right robot},1)
({right node monitor},1) ({plan monitor},1) ({left robot},1)
({left node monitor},1) namely, no collapse took place,
while the only weighted node in the ABP3 CIG is
({alice,bob,carol,dave},4) labeled with the set of unsplittable
agents.</p>
        <p>The edges in the SaS CIG are the same as those in the
SaS IG, while there are no edges in the ABP3 CIG due to
the presence of only one node.
4) Graph partitioning. This stage of the algorithm consists in
partitioning the CIG graph obtained by the collapse stage.
The number of expected partitions is given as an input by the
user and must be grater than one. Any suitable partitioning
algorithm could be used during this stage. We will discuss
our implementative choice in Section V.</p>
        <p>The partition of the original, not collapsed graph IG, is
trivially derived by that of CIG by making the union of
the agents in the labels of the nodes in each CIG partition.
the node weights in each partition is about the same and
the number of edges connecting nodes in two different
partitions is minimized (by associating the unit weight with each
edge), we can obtain the following partition for the agents
{plan monitor, right node monitor, right robot} } .
involved in the SaS protocol: {{left node monitor, left robot},</p>
        <p>The agents involved in the ABP3 protocol cannot be
partitioned because the collapsed graph consists of only one
node.
5) Projection. Once the agents belonging to the same partition
have been devised, the projection algorithm presented in
Section III-B can be used to project the global interaction
protocol onto each of them.</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>V. IMPLEMENTATION AND EXPERIMENTS</title>
      <p>The MAS-DRiVe algorithm is fully implemented in
SWIProlog (http://www.swi-prolog.org/, accessed on June 2016),
apart from the graph partitioning stage which exploits the
METIS tool described in Section III. The extraction of the
IG from the protocol, the identification of the unsplittable
agents sets, and the generation of the collapsed graph CIG
are performed by Prolog predicates. After them, a predicate for
encoding CIG into a format suitable for METIS and saving
it into a file is called. The command line for running METIS
on that file is launched from inside Prolog, and the result is
read and decoded.</p>
      <p>In order to run the algorithm, the user must call the
partition/3 predicate with three arguments: the identifier
of the agent interaction protocol to distribute, the number
of partitions to be obtained, and the name of the file where
Prolog will write the result of the partition. For example,
given that abp3 identifies the ABP3 protocol, and that the
trace expression associated with this identifier has been
read into the Prolog knowledge base, the user should call
partition(abp3, 2, ’./abp3Out.txt’) obtaining
the following content for the abp3Out.txt file:</p>
      <sec id="sec-3-1">
        <title>Unsplittable list: [((/\),2)],[alice,bob,carol,dave]</title>
      </sec>
      <sec id="sec-3-2">
        <title>Partition failed</title>
      </sec>
      <sec id="sec-3-3">
        <title>Nodes</title>
        <p>node(abp3,bob)
node(abp3,alice)
node(abp3,carol)
node(abp3,dave)</p>
      </sec>
      <sec id="sec-3-4">
        <title>Extended edges</title>
        <p>edge(abp3,bob,alice,[ ((/\),2), ((/\),3),
((/\),4), ((&gt;&gt;),5)])
edge(abp3,bob,carol,[ ((/\),2), ((/\),3),
((/\),4), ((&gt;&gt;),5)])
....</p>
      </sec>
      <sec id="sec-3-5">
        <title>Collapsed nodes cnode(abp3,[alice,bob,carol,dave],4)</title>
      </sec>
      <sec id="sec-3-6">
        <title>The trace expressions</title>
        <p>@((S_1|S_2),
[S_1=
(put_right_sock:put_right_shoe:ok_right:lambda)\/
(put_right_shoe:oblige_remove_right_shoe:
removed_right_shoe:S_1),
S_2=
(put_left_sock:put_left_shoe:ok_left:lambda)\/
(put_left_shoe:oblige_remove_left_shoe:</p>
        <p>removed_left_shoe:S_2)])
can be partitioned into
{[left_node_monitor],[left_robot]}
{[plan_monitor],[right_node_monitor],[right_robot]}</p>
      </sec>
      <sec id="sec-3-7">
        <title>Nodes</title>
        <p>node(socks,right_robot)
node(socks,right_node_monitor)
node(socks,plan_monitor)
node(socks,left_robot)
node(socks,left_node_monitor)</p>
      </sec>
      <sec id="sec-3-8">
        <title>Extended edges</title>
        <p>edge(socks,right_robot,right_node_monitor,
[((|),2), ((\/),3), ((:),4)])
edge(socks,right_robot,right_node_monitor,
[((|),2), ((\/),3), ((:),4), ((:),5)])
........</p>
      </sec>
      <sec id="sec-3-9">
        <title>Collapsed nodes</title>
        <p>cnode(socks,[right_robot],1)
cnode(socks,[right_node_monitor],1)
cnode(socks,[plan_monitor],1)
cnode(socks,[left_robot],1)
cnode(socks,[left_node_monitor],1)</p>
      </sec>
      <sec id="sec-3-10">
        <title>Collapsed edges</title>
        <p>cedge(socks,[right_robot],[right_node_monitor])
cedge(socks,[right_node_monitor],[plan_monitor])
cedge(socks,[left_robot],[left_node_monitor])
cedge(socks,[left_node_monitor],[plan_monitor])</p>
        <p>We run experiments with protocols that involve distinct
groups of agents each following the ABP3 protocol. For
example, in the DoubleABP3, alice, bob, carol and dave follow
the ABP3 protocol, alice2, bob2, carol2 and dave2 follow the
ABP3 protocol as well, and both bob and bob2 interact with
boss. Since alice, bob, carol and dave have no interactions
with alice2, bob2, carol2 and dave2, the two groups are
independent from each other and can be safely partitioned
for the purpose of decentralized runtime verification. In such
a situation, the partition computed by MAS-DRiVe is the
following:</p>
      </sec>
      <sec id="sec-3-11">
        <title>Unsplittable list:</title>
        <p>[ ((*),3), ((\/),7), ((:),8), ((:),9), ((/\),10)],
[alice2,bob2,carol2,dave2]
[ ((*),3), ((\/),4), ((:),5), ((:),6), ((/\),7)],
[alice,bob,carol,dave]</p>
      </sec>
      <sec id="sec-3-12">
        <title>The trace expressions</title>
        <p>
          @((bobasks:lambda|bob2asks:lambda)*
((okbob:nobob2:msg3&gt;&gt;S_1/\msg_ack(
          <xref ref-type="bibr" rid="ref1">1</xref>
          )&gt;&gt;S_2/\
msg_ack(
          <xref ref-type="bibr" rid="ref2">2</xref>
          )&gt;&gt;S_3/\msg_ack(
          <xref ref-type="bibr" rid="ref3">3</xref>
          )&gt;&gt;S_4)\/
(okbob2:nobob:mmsg3&gt;&gt;S_5/\mmsg_ack(
          <xref ref-type="bibr" rid="ref1">1</xref>
          )&gt;&gt;S_6/\
By using a graph partitioning algorithm where the sum of Unsplittable list: []
The content of the socksOut.txt file that we obtain by mmsg_ack(
          <xref ref-type="bibr" rid="ref2">2</xref>
          )&gt;&gt;S_7/\mmsg_ack(
          <xref ref-type="bibr" rid="ref3">3</xref>
          )&gt;&gt;S_8)),
[S_1=m1:m2:m3:S_1,S_2=m1:a1:S_2,
calling partition(socks, 2, ’./socksOut.txt’) S_3=m2:a2:S_3,S_4=m3:a3:S_4,S_5=mm1:mm2:mm3:S_5,
is: S_6=mm1:aa1:S_6,S_7=mm2:aa2:S_7,S_8=mm3:aa3:S_8])
events. For example, an interaction between the plan monitor
can be partitioned into and the left node monitor, corresponding to a (plan monitor,
{[boss],[alice2,bob2,carol2,dave2]} left node monitor) edge in the graph, will be monitored by
{[alice,bob,carol,dave]} both the monitor in charge of {left node monitor, left robot}
tfadoholfaservToaeahgpwleeaminrciattuteihlstsg2ito,ofthobrnbieletlohoaTbmmwrr2eioi,pcnnflocgoieatruAortrnehroBecdelPdt2.lA3yWaaBlrpnleePrdcot3oogt.doggoacentvotihetzlh,e2eers.w,tNeiahtxnhoapdtetoahtcthlrthieeaceedter,tdhccbioeoosrtnbisrsn,eatcccmrttaaeirgrnoerhtlossouaulonldptndsss trraabpheinneogdeodshuestnthnimbtuedhrlpmraoealntbobabcoyeryyietren}ads,coeacfotlanhofesenacdiroatmgiiguttneetpgbosiremnooaavvfacaopetrvlim{aovcorpseaietsdlslinalteyitpnodaan,ogdrmifetebninottugoeitntoscuianttrclosiagtn.rwhno,eGrobibrrtrooiekhagtp:mphhkthitsemttoprhpnfaataaolttredattnhgseimtueiiesimommnsmni.obimacnneloTrgiiltuzhoaeoliirdssssf,
        </p>
        <p>The code of the MAS-DRiVe algorithm can be downloaded monitors to deploy, or at least guidelines may be provided in
from http://www.disi.unige.it/person/MascardiV/Software/ the future.
masdrive.html. It requires SWI Prolog and the METIS Finally, as part of our future work, we plan to make an
exsoftware installed and accessible via the command line from haustive experimentation of MAS-DRiVe on existing complex
everywhere. We tested it on Mageia Linux. protocols. METIS can easily partition graphs with hundreds
of nodes so we will model protocols involving hundreds of</p>
        <p>VI. CONCLUSIONS AND FUTURE WORK agents and test the scalability of the approach. Studying its
This paper addresses the problem of how to partition a computational properties from a theoretical viewpoint is on
MAS into agents’ subsets that can be verified at runtime, our agenda as well.
independently from one another, in such a way that the results
obtained by the decentralized runtime verification are the ACKNOWLEDGMENTS
same as those obtained by monitoring the whole MAS in a We are grateful to the anonymous reviewers for their useful
centralized way. remarks and constructive comments.</p>
        <p>We designed and implemented the MAS-DRiVe algorithm,
whose results are consistent with those discussed in [2]. In that REFERENCES
paper, the feasibility of distributing the runtime monitoring
over user-defined sets of agents was empirically validated by
checking the traces of events compliant with the decentralized
subprotocols up of a given length, and verifying whether they
were compliant with the original global protocol as well. That
approach gave a semi-decidable information: if all traces up
to a given length l were found to be compliant with the
protocol, this did not ensure that problems could not arise
with traces of length l + 1. With MAS-DRiVe we propose a
constructive way to partition agents in the MAS, rather than
letting the user decide how to partition them, and we overcome
the problems due to the empirical validation approach by
proposing partitions that are “safe” w.r.t. the trace expression
operators. In particular, we assume that agents involved in
an intersection can never be partitioned. This is an
overcautious approach, but since intersection is the construct used
to state constraints across different independent branches of the
protocol, we assume that those branches cannot be monitored
independently.</p>
        <p>In its current version, then, the MAS can not get split if
no semi-independent parts of the system can get identified.</p>
        <p>Although this is a limitation of our approach, this could have
an interesting side effect that we aim to explore in the close
future, namely the possibility to come up with suggestions
for MAS designers in terms of good practices for better
subsystems encapsulation based on interaction protocol analysis.</p>
        <p>Such suggestions could be applied to distributed systems in
general, and not only to MASs, and could prove useful in
those approaches where there is an embedded notion of some
sort of encapsulation.</p>
        <p>As far as graph partitioning is concerned, we point out
that the obtained partitions do not correspond to partitions of</p>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>M.</given-names>
            <surname>Alberti</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Gavanelli</surname>
          </string-name>
          , E. Lamma,
          <string-name>
            <given-names>P.</given-names>
            <surname>Mello</surname>
          </string-name>
          , and
          <string-name>
            <given-names>P.</given-names>
            <surname>Torroni</surname>
          </string-name>
          .
          <article-title>The SCIFF abductive proof-procedure</article-title>
          .
          <source>In AI*IA</source>
          , pages
          <fpage>135</fpage>
          -
          <lpage>147</lpage>
          ,
          <year>2005</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>D.</given-names>
            <surname>Ancona</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Briola</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>El Fallah Seghrouchni</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Mascardi</surname>
          </string-name>
          , and
          <string-name>
            <given-names>P.</given-names>
            <surname>Taillibert</surname>
          </string-name>
          .
          <article-title>Efficient verification of MASs with projections</article-title>
          .
          <source>In EMAS 2014. Revised Selected Papers</source>
          , volume
          <volume>8758</volume>
          <source>of LNCS</source>
          , pages
          <fpage>246</fpage>
          -
          <lpage>270</lpage>
          . Springer,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>D.</given-names>
            <surname>Ancona</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Briola</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Ferrando</surname>
          </string-name>
          , and
          <string-name>
            <given-names>V.</given-names>
            <surname>Mascardi</surname>
          </string-name>
          .
          <article-title>Global protocols as first class entities for self-adaptive agents</article-title>
          . In G. Weiss, P. Yolum,
          <string-name>
            <given-names>R. H.</given-names>
            <surname>Bordini</surname>
          </string-name>
          , and E. Elkind, editors,
          <source>Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2015</source>
          , pages
          <fpage>1019</fpage>
          -
          <lpage>1029</lpage>
          . ACM,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>D.</given-names>
            <surname>Ancona</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Briola</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Ferrando</surname>
          </string-name>
          , and
          <string-name>
            <given-names>V.</given-names>
            <surname>Mascardi</surname>
          </string-name>
          .
          <article-title>Runtime verification of fail-uncontrolled and ambient intelligence systems: A uniform approach</article-title>
          .
          <source>Intelligenza Artificiale</source>
          ,
          <volume>9</volume>
          (
          <issue>2</issue>
          ):
          <fpage>131</fpage>
          -
          <lpage>148</lpage>
          ,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>D.</given-names>
            <surname>Ancona</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Drossopoulou</surname>
          </string-name>
          , and
          <string-name>
            <given-names>V.</given-names>
            <surname>Mascardi</surname>
          </string-name>
          .
          <article-title>Automatic Generation of Self-Monitoring MASs from Multiparty Global Session Types in Jason</article-title>
          . In M. Baldoni,
          <string-name>
            <given-names>L. A.</given-names>
            <surname>Dennis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Mascardi</surname>
          </string-name>
          , and W. Vasconcelos, editors,
          <source>DALT 2012. Revised Selected Papers</source>
          , volume
          <volume>7784</volume>
          <source>of LNCS</source>
          . Springer,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>D.</given-names>
            <surname>Ancona</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Ferrando</surname>
          </string-name>
          , and
          <string-name>
            <given-names>V.</given-names>
            <surname>Mascardi</surname>
          </string-name>
          .
          <article-title>Comparing trace expressions and linear temporal logic for runtime verification</article-title>
          . In E. A´braha´m,
          <string-name>
            <surname>M. M. Bonsangue</surname>
          </string-name>
          , and E. B. Johnsen, editors,
          <source>Theory and Practice of Formal Methods</source>
          , volume
          <volume>9660</volume>
          <source>of LNCS</source>
          , pages
          <fpage>47</fpage>
          -
          <lpage>64</lpage>
          . Springer,
          <year>2016</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>A.</given-names>
            <surname>Bauer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Leucker</surname>
          </string-name>
          , and
          <string-name>
            <given-names>C.</given-names>
            <surname>Schallhart</surname>
          </string-name>
          .
          <article-title>Comparing LTL semantics for runtime verification</article-title>
          .
          <source>J. Log. and Comput.</source>
          ,
          <volume>20</volume>
          (
          <issue>3</issue>
          ):
          <fpage>651</fpage>
          -
          <lpage>674</lpage>
          ,
          <year>June 2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>A.</given-names>
            <surname>Bauer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Leucker</surname>
          </string-name>
          , and
          <string-name>
            <given-names>C.</given-names>
            <surname>Schallhart</surname>
          </string-name>
          .
          <article-title>Runtime verification for LTL and TLTL</article-title>
          .
          <source>ACM Trans. Softw</source>
          . Eng. Methodol.,
          <volume>20</volume>
          (
          <issue>4</issue>
          ):
          <volume>14</volume>
          :
          <fpage>1</fpage>
          -
          <lpage>14</lpage>
          :
          <fpage>64</fpage>
          ,
          <string-name>
            <surname>Sept</surname>
          </string-name>
          .
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>F. L.</given-names>
            <surname>Bellifemine</surname>
          </string-name>
          , G. Caire, and
          <string-name>
            <given-names>D.</given-names>
            <surname>Greenwood</surname>
          </string-name>
          .
          <article-title>Developing Multi-Agent Systems with JADE</article-title>
          . Wiley,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>D.</given-names>
            <surname>Briola</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Mascardi</surname>
          </string-name>
          , and
          <string-name>
            <given-names>D.</given-names>
            <surname>Ancona</surname>
          </string-name>
          .
          <article-title>Distributed runtime verification of JADE multiagent systems</article-title>
          . In D. Camacho,
          <string-name>
            <given-names>L.</given-names>
            <surname>Braubach</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Venticinque</surname>
          </string-name>
          , and C. Badica, editors,
          <source>IDC</source>
          <year>2014</year>
          , volume
          <volume>570</volume>
          <source>of Studies in Computational Intelligence</source>
          , pages
          <fpage>81</fpage>
          -
          <lpage>91</lpage>
          . Springer,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <surname>M.</surname>
          </string-name>
          <article-title>Br o¨rkens and</article-title>
          <string-name>
            <surname>M. M o</surname>
          </string-name>
          <article-title>¨ller. Dynamic event generation for runtime checking using the JDI</article-title>
          .
          <source>Electr. Notes Theor. Comput. Sci.</source>
          ,
          <volume>70</volume>
          (
          <issue>4</issue>
          ):
          <fpage>21</fpage>
          -
          <lpage>35</lpage>
          ,
          <year>2002</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>A.</given-names>
            <surname>Buluc</surname>
          </string-name>
          ¸,
          <string-name>
            <given-names>H.</given-names>
            <surname>Meyerhenke</surname>
          </string-name>
          ,
          <string-name>
            <surname>I. Safro</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Sanders</surname>
          </string-name>
          , and
          <string-name>
            <given-names>C.</given-names>
            <surname>Schulz</surname>
          </string-name>
          .
          <article-title>Recent advances in graph partitioning</article-title>
          .
          <source>In Algorithm Engineering: Selected Results and Surveys</source>
          , volume
          <volume>9220</volume>
          <source>of LNCS</source>
          . Springer,
          <year>2015</year>
          (in press).
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>F.</given-names>
            <surname>Chen</surname>
          </string-name>
          and
          <string-name>
            <given-names>G.</given-names>
            <surname>Rosu.</surname>
          </string-name>
          <article-title>Mop: an efficient and generic runtime verification framework</article-title>
          .
          <source>In OOPSLA 2007</source>
          , pages
          <fpage>569</fpage>
          -
          <lpage>588</lpage>
          ,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>C.</given-names>
            <surname>Colombo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G. J.</given-names>
            <surname>Pace</surname>
          </string-name>
          , and
          <string-name>
            <given-names>G.</given-names>
            <surname>Schneider. LARVA -</surname>
          </string-name>
          <article-title>Safer Monitoring of Real-Time Java Programs (Tool Paper)</article-title>
          .
          <source>In SEFM 2009</source>
          , pages
          <fpage>33</fpage>
          -
          <lpage>37</lpage>
          ,
          <year>2009</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <surname>F. S. de Boer</surname>
          </string-name>
          and C. P. T. de Gouw.
          <article-title>Combining Monitoring With Run-Time Assertion Checking</article-title>
          .
          <source>In SFM 14</source>
          , pages
          <fpage>217</fpage>
          -
          <lpage>262</lpage>
          . Springer,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>A.</given-names>
            <surname>Ferrando</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Mascardi</surname>
          </string-name>
          , and
          <string-name>
            <given-names>D.</given-names>
            <surname>Ancona</surname>
          </string-name>
          .
          <article-title>Monitoring patients with hypoglycemia using self-adaptive protocol-driven agents: a case study</article-title>
          .
          <source>In Workshop Proceedings of EMAS</source>
          <year>2016</year>
          ,
          <year>2016</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <given-names>L.</given-names>
            <surname>Giordano</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Martelli</surname>
          </string-name>
          , and
          <string-name>
            <given-names>C.</given-names>
            <surname>Schwind</surname>
          </string-name>
          .
          <article-title>Specifying and verifying interaction protocols in a temporal action logic</article-title>
          .
          <source>Journal of Applied Logic</source>
          ,
          <volume>5</volume>
          (
          <issue>2</issue>
          ):
          <fpage>214</fpage>
          -
          <lpage>234</lpage>
          ,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>K.</given-names>
            <surname>Havelund</surname>
          </string-name>
          and
          <string-name>
            <given-names>G.</given-names>
            <surname>Rosu</surname>
          </string-name>
          .
          <article-title>Synthesizing monitors for safety properties</article-title>
          . In J. Katoen and P. Stevens, editors,
          <source>TACAS</source>
          <year>2002</year>
          , volume
          <volume>2280</volume>
          <source>of LNCS</source>
          , pages
          <fpage>342</fpage>
          -
          <lpage>356</lpage>
          . Springer,
          <year>2002</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <given-names>G.</given-names>
            <surname>Karypis</surname>
          </string-name>
          and
          <string-name>
            <given-names>V.</given-names>
            <surname>Kumar</surname>
          </string-name>
          .
          <article-title>Analysis of multilevel graph partitioning</article-title>
          .
          <source>In Proceedings of the 1995 ACM/IEEE Conference on Supercomputing, Supercomputing '95</source>
          , New York, NY, USA,
          <year>1995</year>
          . ACM.
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <given-names>G.</given-names>
            <surname>Karypis</surname>
          </string-name>
          and
          <string-name>
            <given-names>V.</given-names>
            <surname>Kumar</surname>
          </string-name>
          .
          <article-title>A fast and high quality multilevel scheme for partitioning irregular graphs</article-title>
          .
          <source>SIAM J. Scientific Computing</source>
          ,
          <volume>20</volume>
          (
          <issue>1</issue>
          ):
          <fpage>359</fpage>
          -
          <lpage>392</lpage>
          ,
          <year>1998</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          [21]
          <string-name>
            <given-names>M.</given-names>
            <surname>Kim</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Kannan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>I.</given-names>
            <surname>Lee</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            <surname>Sokolsky</surname>
          </string-name>
          , and
          <string-name>
            <given-names>M.</given-names>
            <surname>Viswanathan</surname>
          </string-name>
          .
          <article-title>Java-mac: a run-time assurance tool for java programs</article-title>
          .
          <source>Electr. Notes Theor. Comput. Sci.</source>
          ,
          <volume>55</volume>
          (
          <issue>2</issue>
          ):
          <fpage>218</fpage>
          -
          <lpage>235</lpage>
          ,
          <year>2001</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          [22]
          <string-name>
            <given-names>L.</given-names>
            <surname>Lamport</surname>
          </string-name>
          . Time, clocks, and
          <article-title>the ordering of events in a distributed system</article-title>
          .
          <source>Commun. ACM</source>
          ,
          <volume>21</volume>
          (
          <issue>7</issue>
          ):
          <fpage>558</fpage>
          -
          <lpage>565</lpage>
          ,
          <year>July 1978</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          [23]
          <string-name>
            <given-names>M.</given-names>
            <surname>Maro</surname>
          </string-name>
          ´ti,
          <string-name>
            <given-names>B.</given-names>
            <surname>Kusy</surname>
          </string-name>
          ,
          <string-name>
            <surname>G.</surname>
          </string-name>
          <article-title>Simon, and</article-title>
          <string-name>
            <given-names>A.</given-names>
            <surname>Le</surname>
          </string-name>
          <article-title>´deczi. The flooding time synchronization protocol</article-title>
          .
          <source>In SenSys '04</source>
          , pages
          <fpage>39</fpage>
          -
          <lpage>49</lpage>
          . ACM,
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          [24]
          <string-name>
            <surname>M. C. Martin</surname>
            ,
            <given-names>V. B.</given-names>
          </string-name>
          <string-name>
            <surname>Livshits</surname>
            , and
            <given-names>M. S.</given-names>
          </string-name>
          <string-name>
            <surname>Lam</surname>
          </string-name>
          .
          <article-title>Finding application errors and security flaws using PQL: a program query language</article-title>
          .
          <source>In OOPSLA 2005</source>
          , pages
          <fpage>365</fpage>
          -
          <lpage>383</lpage>
          ,
          <year>2005</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          [25]
          <string-name>
            <given-names>S.</given-names>
            <surname>Rajsbaum</surname>
          </string-name>
          .
          <article-title>Distributed runtime verification - where combinatorics, fault-tolerance and formal methods meet</article-title>
          .
          <source>Keynote Talk at the SSS</source>
          <year>2015</year>
          ,
          <year>August 2015</year>
          ,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>