Engineering Multiagent Organizations by Accountability and Responsibility

Abstract

Basing upon multi-agent systems, we show that an explicit representation of accountabilities and responsibility assumptions gives the right abstractions for properly specifying who should provide feedback to whom, about the execution of its duties, both at the level of design and at the level of programming. For the latter, we explain a programming pattern for developing accountable agents, and illustrate the approach in JaCaMo. The paper takes into account business processes as a motivating scenario.

Keywords

• Accountability Responsibility BPMN JaCaMo

Introduction

Multiagent Systems (MAS), and in particular MAS organizations (MAO), are an approach to the design and development of complex, distributed software.

In this respect, they are promising candidates to supply the right abstractions for developing business processes. In fact, a business process (BP) is \a set of activities that are performed in coordination in an organizational and technical environment. These activities jointly realize a business goal." [ 32 ]. In general, a business goal is achieved by breaking it up into sub-goals, which are distributed to a number of actors. Each actor carries out part of the process and depends on the collaboration of others to perform its task.

In order to provide the right support to BPs, however, MAOs are still lacking a systematic way to properly handle feedback of the execution, provided in terms of good or bad functioning. Feedback will generally be of interest to (and should be handled by) an agent which is not the one that produces it. Therefore, an appropriate \infrastructure" needs to be devised. In [ 2 ] a proposal was made to introduce accountabilities and responsibility relationships inside a MAO. Building upon that work, here we explain how such concepts can be used as tools to systematize and guide the design and development of the agents. We also present a programming pattern, providing an exempli cation in JaCaMo MAOs [ 8 ], as a Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). tr o p p u lS Issue e v e L t s 1 tr o p pTicket Received u S l e v e L d n 2 r e p o l e v e D e ftraw FroRmeqSuuepsptort o S

DescArsipktion

Handle 1st Level Issue Handle 2nd Level Issue Answer Received Explain Solution

Provide Feedback for

Account Manager

Yes Can Handle? No Result?

VIP Customer

AskSu1psptoLretvel

Solved Ask 2nd Level Support 2nd Level Issue Answer Received Unsure? Yes No

Ask Developer

Result? Provide

Solved Feedback for

1st Level

Support Next Release

Insert Into Product

Backlog

Answer Received Examine Problem

Provide Feedback for 2nd Level

Support practical way to implement the feedback loop. This paper discusses some results that have been published in [ 3 ] at AAMAS 2019, as extended abstract3.

The paper is organized as follows. Section 2 broadly introduces the concepts of accountability and responsibility as tools for engineering agents in MAOs to e ectively realize BPs. To illustrate, we rely on the OMG Incident Management process, as a practical use case. Section 3 presents the actual programming pat3 The same results have been presented in more detail, as a full paper, at the 7th International Workshop on Engineering Multi-Agent Systems (EMAS 2019), co-located with AAMAS 2019. The paper also won the Best Paper Award. tern to develop accountable agents. In Section 4 we show how the pattern can be applied, in the scope of the JaCaMo framework, to the Incident Management scenario. Conclusions end the paper. 2

Responsibility and Accountability as Engineering

Concepts

Figure 1 shows OMG's Incident Management BP [ 25 ] that we use as running example to exemplify interaction issues rising when feedback produced by an actor should be reported to the most appropriate one. The process models the interaction between a customer and a company for the management of a problem reported by the customer to a Key Account Manager, who, on the basis of her experience, can either solve the problem directly, or ask for the intervention of rst-level support. The problem is, then, recursively treated at di erent support levels until, in the worst case, it is reported to the software developer. Thus, the business aim of the process (to solve the reported problem) will generally be decomposed and may be distributed over up to ve BPMN processes, whose execution requires interaction and coordination. Noticeably, as usual with BPs, the way in which goals are achieved matters: agents are expected not only to ful ll their assigned goals, but also to respect the BP { the \goal" is that the process takes place [ 1 ].

One common limitation of the kind of modularity implemented both by BPs and by MAOs is that the overall process structure of the goal is intended mainly as a way for constraining the agents' autonomy, and not as information provided to support the agents in their work: agents are expected merely to focus on the achievement of the assigned sub-goals. In this way, however, the agents loose sight of the overall process and ignore the place their achievement has within the organization. The relationship between each level of support and the following one, in the example, is emblematic: when a request of support is made, an answer containing some kind of feedback on the realization of this support is expected in order to proceed. However, since processes are independent, one cannot give for granted that another will answer. It follows that when a process does not answer, the waiting one may get stuck inde nitely. Similarly, MAOs (e.g., [ 11, 14 ]) allow a designer to structure the functional decomposition of complex, organizational goals, and the assignment of subgoals to agents. The coordinated execution of subgoals is often supported by a normative speci cation, by which the organization issues obligations towards the agents, e.g., [ 13, 17, 12, 8 ]. However, agents may operate in ways that do not t into the process speci cation and, importantly, when agents fail, the organization has no a systematic mechanism for sorting out the abnormal conditions, i.e., for a redress.

This is where accountability [ 18, 19, 10, 4, 5, 7, 6 ] and responsibility [ 29, 31, 33, 6 ] come in handy. Accountability \emerges as a primary characteristic of governance where there is a sense of agreement and certainty about the legitimacy of expectations between the community members." [ 15 ]. In particular, accountability implies that some actors have the right to hold other actors to a set of standards, to judge whether they have ful lled their responsibilities in light of these standards, and to impose sanctions if they determine that these responsibilities have not been met [ 19 ]. It presupposes a relationship between power-wielders and those holding them accountable, where there is a general recognition of the legitimacy of (1) the operative standards for accountability and (2) the authority of the parties to the relationship (one to exercise particular powers and the other to hold them to account).

Concerning responsibility, [ 31 ] proposes an ontology relating six di erent responsibility concepts (capacity, causal, role, outcome, virtue, and liability), that capture: doing the right thing, having duties, an outcome being ascribable to someone, a condition that produced something, the capacity to understand and decide what to do, something being legally attributable. In the context of Information Systems, the meta-model ReMMO [ 16 ] represents responsibility as a unique charge assigned to an agent. This perspective emerges also in the literature cited above, where most of the authors acknowledge that responsibility aims at conferring one or more obligation(s) to an actor (the responsibility owner). In this light, accountability is grounded on perceived/assumed responsibility, deriving from recognition of the claim-right to hold the responsible for a task to account for that very same task.

BPs represent an agreed behavior, introduce expectations on the behavior of the interacting parties, and require some kind of governance in order for the process to be enacted, but the accountability results hidden into some kind of collective responsibility (\many hands problem" [ 23 ]). As a consequence, the system is fragile to unexpected situations. When, instead, accountability relationships are expressed explicitly, they enable the reporting of unwanted conditions (i.e., account, feedback) to the most apt agent to handle them. 3

Engineering MAOs with Accountability/Responsibility

We refer to a setting where the organizational goal is functionally decomposed and the resulting sub-goals are assigned to agents. Since MAOs often rely on norms in order to orchestrate the executions of their agents, we imagine obligations, that are issued by the organization, to be the means through which the coordinated execution of the agents is regulated. Agents, of course, are autonomous and their ful llment of the obligations cannot be given for granted.

Following [ 2 ], we denote by R(x; q) and A(x; y; r; u) responsibility and accountability relationships. R(x; q) expresses an expectation on any agent playing role x on pursuing condition q (x is entitled and should have the capabilities of bringing about q); A(x; y; r; u) expresses that x, the account-giver (a-giver), is accountable towards y, the account-taker (a-taker), for the condition u when the condition r (context ) holds4. We see u in the context of r as the agreed standard which brings about expectations inside the organization. The proposal in [ 2 ] 4 We rely upon precedence logic [ 28 ], an event-based linear temporal logic, which deals with occurrences of events along runs. Event occurrences are assumed to be nonrepeating and persistent. The logic has three primary operators: `_' (choice), `^' suggests to complement the functional decomposition of the organizational goal with a set of accountability and responsibility speci cations. A set of accountability relationships is denoted as A, and is called an accountability speci cation. The organization designer will generally specify a set of alternative legitimate accountability speci cations which is denoted by A. Given a set of accountability speci cations A, and a set of responsibility assumptions R (responsibility distribution), the organization is properly speci ed when the accountability tting \R ts A" (denoted by R A) holds [ 2 ].

We now discuss one possible use of these concepts at design time for realizing accountable agents, that is, agents that provide an account of their conduct both when they succeed in achieving their goals, and when, for some reason, they fail in the attempt. Since accountabilities and responsibilities imply some obligations [ 19 ], we can realize them in JaCaMo by relying on the deontic primitives that such framework provides: the tting relationship represented by each pair hR(x; q); A(x; y; r; u)i in Rx Ax ( tting projection), can be mapped into a number of Jason plans of the agent playing role x by way of the following pattern, expressed in AgentSpeak(ER) [ 27 ]: +!be accountable(x, y, q) <: drop tting(x, y, q) f // Well-Doing e-plan +obligation(x, q) : r ^ c <- bodyq. // Wrong-Doing e-plan +oblUnful lled(x, q) : r ^ c0 <- bodyf .

g It is requested that the following conditions hold: 1) bodyq satis es the ttingadherence condition (below); 2) bodyf includes sending an explanation for the failure from x to y. The two e-plans encode the proactive behavior of an agent assuming a responsibility. Until the responsibility is not dropped, the agent starts reacting to obligations in accordance to the accountability speci ed in the tting. The agent will perceive, through the identity that is provided by the organizational role it plays, certain events as events it should tackle through some behavior of its own, but it will also be aware of its social position both (1) by knowing some other agent will have the right, under certain conditions, to ask for an account and (2) by including speci c behavior for building such an account. Well-doing e-plan { It is triggered when the speci ed obligation is issued by the MAO. The context r ^ c is satis ed when condition r (activating the agent accountability) holds together with some (possibly empty) local condition c that allows choosing among alternative plans, i.e., multiple ways to achieve a same result in di erent (local) circumstances. bodyq must be such to satisfy a ttingadherence condition, that is, given the responsibility assumption represented by the pair hR(x; q); A(x; y; r; u)i, there must exist an execution of bodyq that, restricted to the events that are relevant for the progression of u, is an actualization (concurrence), and ` ' (before). The before operator allows constraining the order in which two events must occur. of the responsibility q (see [ 2 ]). Intuitively, q is actually used for ful lling the obligation. In this case, in certain applications the obligation to give an account for the satisfaction of the obligation may be implicitly resolved by satisfying the very same obligation. It is interesting to note that the accountability tting is not only a functional speci cation of the organization, but it also speci es the \good" behavior of the agents. It is in fact this characteristic that justi es our programming pattern, that enriches the standard JaCaMo.

Wrong-doing e-plan { It allows the agent to provide an account when it did not complete its task. The triggering event, oblUnful lled, is generated by the MAO when an obligation is left unsatis ed. The context has the same structure as above; bodyf produces an account of the failure. This will be an explanation that the agent produces and that some other agent will use to manage the exception and to resume the execution. The correct use of the pattern guarantees, by design, that exceptional events, when occurring, are reported to the agents who can handle them properly. Accountability ful lls this purpose because, by nature, it brings about an obligation on the a-giver to give an account of what it does. 4

JaCaMo Accountable Agents

The engineering of accountable JaCaMo MAOs involves several steps: (1) Each process is mapped to an organizational role; (2) A scheme representing the overall process goal is de ned { its successful execution amounts to the achievement of such a goal; (3) For each activity to be performed in sequence, a subgoal is added to the scheme by means of the corresponding operator; (4) For each structured block including a concurrent execution, the corresponding goals, grouped by the parallel operator, are added to the scheme; (5) For each choice all the schemes representing the possible courses of action should be de ned. They will be instantiated dynamically by the agents, depending on their internal choices.

By applying the steps to Incident Management we obtain the ve roles: customer (c), key account manager (am), rst level support ( s), second level support (sls), and developer (dev). For each incident to manage, we assume there will be exactly one agent playing each role. The organizational goal is distributed into a set of schemes. The top-level scheme involves c and am, and is instantiated by the customer c when some need arises. In other terms, the scheme instantiation corresponds to the occurrence of a report-problemc event. When a problem is reported, the account manager is expected to perform ask-descriptionam (ask for a description of the problem), and c is expected to send what requested (send-descriptionc). Then, am should provide a solution: to this aim, it can take two alternatives courses of action, both leading to the same join point of the BPMN diagram. The rst alternative amounts to the case in which am can handle the problem directly, and does not require the execution of any action before the join point. The second alternative, instead, amounts to the case in which am cannot handle the problem directly and brings to the instantiantion of a new scheme. The agent will make a request (ask-support- s am) that s will manage a1 : A(am; c; report-problemc; report-problemc ask-descriptionam) a2 : A(am; c; report-problemc ask-descriptionam send-descriptionc;

report-problemc ask-descriptionam send-descriptionc explain-solutionam) . . . r1 : R(am; ask-descriptionam) r2 : R(am; explain-solutionam) . . . either directly or by involving the next level of support { by instantiating a further scheme. ability a1 states that am is accountable towards c for asking for a description of the incident, after a problem is reported. This requirement means that c can legitimately expect that, by reporting a problem to am, it will be asked for a description of the problem. The second accountability has a similar structure, and states that once the description of the problem has been provided, am should, in the end, explain the solution, no matter what happens in the meanwhile.

With the accountability speci cation Aincident as a basis, the designer can identify a suitable responsibility distribution which ts it. An excerpt of an acceptable one, concerning am, is also reported in Figure 2.

We, now, brie y explain the realization of the key account manager am agent obtained by exploiting the pattern presented in the previous section. For each pair in Ram

Aam, a g-plan must be de ned, containing the proper well-doing and wrong-doing e-plans. Let us consider, in particular, the tting involving r2

a2. It is implemented by the following plans: 1 +! be_accountable (Ag , ATaker , What ) : . my_name ( Ag ) & play ( ATaker , customer , incident_group ) & ( satisfied (... , explain_solution ) = What | done (... , explain_solution , Ag ) = What ) <: drop_fitting (Ag , ATaker , What ) { 2 3 4 5 6 7 + obligation (Ag ,_ , What ,_) : . my_name ( Ag ) & can_handle ( What ) & goalState ( sch1 , ask_description ,_ ,_ , satisfied ) & goalState ( sch1 , send_description ,_ ,_ , satisfied ) & <- goalAchieved ( explain_solution ). 28 + oblUnfulfilled (O) 29 : . my_name ( Ag ) & obligation (Ag ,_ , What ,_) = O & not can_handle ( What ) & 30 goalState ( sch1 , ask_description ,_ ,_ , satisfied ) & 31 goalState ( sch1 , send_description ,_ ,_ , satisfied ) & 32 not goalState ( sch2 , provide_feedback_am ,_ ,_ , satisfied ) & 33 <- . send ( ATaker , tell , please_call_again ). 34 35 + cancel -fls - request 36 : oblUnfulfilled (O) & obligation (_ ,_ , What ,_) = O 37 <- . send ( ATaker , tell , please_call_Again ); 38 . drop_all_intentions . 39 40 ... 41 42 }

The outer g-plan is triggered when the agent proactively decides to adhere to the tting r2 a2. Once triggered, the g-plan will remain active until the agent does not drop the tting (see Line 5). The plans in braces encode the reactive behavior corresponding to the well-doing and wrong-doing e-plans. The rst two plans, in particular, realize the well-doing part of the pattern. The plans are triggered as soon as an obligation to explain the solution to the customer's problem is issued (Line 7). The obligation's object (What ) is the satisfaction of the organizational goal explain solution. Indeed, in JaCaMo, the achievement of an organizational goal ful lls the corresponding obligation. Then, as requested by the pattern, the contexts of both plans must include the conditions speci ed in a2. In JaCaMo we represent these conditions in terms of schemes that were instantiated and in terms of organizational goals that were achieved. To respect the tting-adherence condition, both plans for well-doing, thus, need to include some actions that amount to explain-solutionam, corresponding to the achievement of the given organizational goal. This is trivially true in the example (see Lines 11 and 19). The agent modi es the organizational environment thereby constructing a sequence of facts that forms the account for the speci c goal.

The presence of two plans triggered by the same obligation re ects the internal choice inside the business process, driven by some local condition. The rst plan is executed when am can handle the problem directly. The second plan, instead, is executed when the agent decides to ask for support. In this case, before providing a feedback to the customer the agent will create an instance of the second social scheme (Line 17). The successful scheme completion will provide it with a feedback from s: am can legitimately expect such a feedback by virtue of another accountability (not reported here), in which it is a-taker. The feedback, in turn, will enable the agent to execute explain-solutionam. The third and fourth plans, instead, deal with the wrong-doing part of the pattern. Should, for any reason, the obligation be unful lled, the agent, by virtue of its accountabilities, must provide a motivation about the violation of the obligation that binds it to the account-taker. The plan at Line 21, in particular, is triggered when the obligation is unful lled because of a reason that is internal to the am agent itself. In the fourth plan the obligation becomes unful lled because am is still waiting for a feedback from sls. In both cases, a proper message encoding the explanation for the failure is sent to the a-taker. The last plan, in turn exempli es how am behaves as an a-taker when receives the account of a failure from another (a-giver) agent. Speci cally, the plan handles a possible failure coming from s raised when it has not satis ed its obligation to provide a feedback. Event cancel-fls-request corresponds to the message s sends as an account of such a failure, and am handles such a failure by asking the customer to call another time and dropping its current intention(s).

Notably, considering the accountability speci cation as a requirement, the actual implementation of the system results more robust. to be accountable, am must be capable, on the one side, of capturing exceptions from other agents (speci cally, s), and on the other side, of providing an account to its a-taker (i.e., the customer). 5

Conclusions

The systematic application of the proposed pattern makes agents aware of the process as characterization of the goal. Hence, accountabilities provide the programmer with a behavioral speci cation the agent has to satisfy. Our approach is specular to [ 33 ], whose objective is to determine whether a group of agents can be attributed the responsibility for a goal. Once the responsibility can be attributed to the agents, their accountability is implicitly modeled in the plan that has been inferred. Here, instead, we aim at developing agents that, by construction, satisfy the organization speci cation.

An interesting evolution of the present work goes in the direction of an agentoriented type checking. Having an explicit organization model, in terms of accountabilities and responsibilities, it would be possible to devise a type checking system that veri es whether, at role enacting time, an agent possesses all the necessary plans for role playing. Moreover, since agent-based approaches are typically declarative, future work will include the exploration of other alternatives to BPMN process diagrams to depict the collaborative work ows. Declarative approaches are also used in the information systems area, in particular for what concerns business process representation. Indeed, OMG has recently released the issue 1.1 of the document for the speci cation of Case Management Model and Notation (CMMN) [ 24 ], which is an extension and re nement of the GSM declarative model [ 21 ]. Further investigations of these approaches will be the objective of future research. In particular, CMMN exploits an event-condition-action language that bears similarities to the way in which agents are programmed when using Jason. Still another orthogonal approach is the RALph graphical notation [ 9 ], which focuses more on the assignment of human resources to BP activities rather than on the strict de nition of a work ow.

As an impact, the proposal moves MAOs closer to other paradigms where exceptions are handled, like the actor model [ 20 ], for instance, where actors that cannot handle an anomalous situation can report exceptions to their parent actors for management, and so forth further up. In an agent-based system such a scheme is not directly applicable because agents are independent entities, and show no parent-child relationship. Approaches for modeling exceptions in a MAS setting have been proposed (see, e.g., [ 22, 30, 26 ]). However, no consensus has been reached on the use of such a concept in agent systems. The main problems rise when trying to accomodate the usual exception handling semantics with the properties of MAS; namely autonomy, openness, heterogeneity, and encapsulation. Accountabilities provide adequate support to ll this gap.

References

1. Adamo , G. , Borgo , S. , Di Francescomarino , C. , Ghidini , C. , Guarino , N.: On the notion of goal in business process models . In: Ghidini, C. , Magnini , B. , Passerini , A. , Traverso , P. (eds.) AI*IA 2018 - Advances in Arti cial Intelligence - XVIIth International Conference of the Italian Association for Arti cial Intelligence , Trento, Italy, November 20-23 , 2018 , Proceedings. Lecture Notes in Computer Science , vol. 11298 , pp. 139 { 151 . Springer ( 2018 ). https://doi.org/10.1007/978-3- 030 -03840-3 - 11, https://doi.org/10.1007/978-3- 030 -03840-3 11

2. Baldoni , M. , Baroglio , C. , Boissier , O. , May , K.M. , Micalizio , R. , Tedeschi , S. : Accountability and Responsibility in Agents Organizations . In: Miller, T. , Oren , N. , Sakurai , Y. , Noda , I. , Savarimuthu , T. , Son , T.C. (eds.) PRIMA 2018: Principles and Practice of Multi-Agent Systems , 21st International Conference. pp. 403 { 419 . No. 11224 in Lecture Notes in Computer Science , Springer, Tokyo, Japan (October 31st{November 2nd 2018 ), http://dx.doi. org/10.1007/978-3-030-03098-8 16

3. Baldoni , M. , Baroglio , C. , Boissier , O. , Micalizio , R. , Tedeschi , S. : Engineering business processes through accountability and agents . In: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems , AAMAS ' 19 , Montreal, QC, Canada, May 13 -17, 2019 . pp. 1796 { 1798 . International Foundation for Autonomous Agents and Multiagent Systems ( 2019 )

4. Baldoni , M. , Baroglio , C. , May , K.M. , Micalizio , R. , Tedeschi , S. : Computational Accountability . In: Chesani, F. , Mello , P. , Milano , M. (eds.) Deep Understanding and Reasoning: A challenge for Next-generation Intelligent Agents , URANIA 2016 . vol. 1802 , pp. 56 { 62 . CEUR, Workshop Proceedings , Genoa, Italy (December 2016 ), http://ceur-ws. org/ Vol-1802/paper8.pdf

5. Baldoni , M. , Baroglio , C. , May , K.M. , Micalizio , R. , Tedeschi , S.: An Information Model for Computing Accountabilities . In: Ghidini, C. , Magnini , B. , Passerini , A. , Traverso , P. (eds.) AI*IA 2018: Advances in Arti cial Intelligence , XVII International Conference of the Italian Association for Arti cial Intelligence. Lecture Notes in Computer Science , vol. 11298 , pp. 30 { 44 . Springer, Trento, Italy (November 20th{23th 2018 ), https://doi.org/10.1007/978-3- 030 -03840-3 3/

6. Baldoni , M. , Baroglio , C. , May , K.M. , Micalizio , R. , Tedeschi , S. : Computational Accountability in MAS Organizations with ADOPT . Applied Sciences 8 ( 4 ) ( 2018 )

7. Baldoni , M. , Baroglio , C. , Micalizio , R. : Goal Distribution in Business Process Models . In: Ghidini, C. , Magnini , B. , Passerini , A. , Traverso , P. (eds.) AI*IA 2018: Advances in Arti cial Intelligence , XVII International Conference of the Italian Association for Arti cial Intelligence. Lecture Notes in Computer Science , vol. 11298 , pp. 252 { 265 . Springer, Trento, Italy (November 20th{23th 2018 ), https://doi.org/10.1007/978-3- 030 -03840-3 19

8. Boissier , O. , Bordini , R.H. , Hubner, J.F. , Ricci , A. , Santi , A. : Multi-agent oriented programming with JaCaMo . Science of Computer Programming 78 ( 6 ), 747 { 761 ( 2013 ). https://doi.org/http://dx.doi.org/10.1016/j.scico. 2011 . 10 .004, http://www.sciencedirect.com/science/article/pii/S016764231100181X

9. Cabanillas , C. , Knuplesch , D. , Resinas , M. , Reichert , M. , Mendling , J. , RuizCortes , A.: RALph: A Graphical Notation for Resource Assignments in Business Processes . In: Advanced Information Systems Engineering . pp. 53 { 68 . Springer International Publishing ( 2015 )

10. Chopra , A.K. , Singh , M.P. : From social machines to social protocols: Software engineering foundations for sociotechnical systems . In: Proc. of the 25th Int. Conf. on WWW ( 2016 )

11. Corkill , D.D. , Lesser , V.R.: The Use of Meta-Level Control for Coordination in Distributed Problem Solving Network . In: Bundy, A . (ed.) Proceedings of the 8th International Joint Conference on Ar ti cial Intelligence (IJCAI'83) . pp. 748 { 756 . William Kaufmann, Los Altos, CA ( 1983 )

12. Dastani , M. , Tinnemeier , N.A. , Meyer, J.J.C. : A programming language for normative multi-agent systems . In: Handbook of Research on Multi-Agent Systems: semantics and dynamics of organizational models , pp. 397 { 417 . IGI Global ( 2009 )

13. Dignum , V.: A model for organizational interaction: based on agents, founded in logic . Ph.D. thesis , Utrecht University ( 2004 ), published by SIKS

14. Dignum , V.: Handbook of research on multi-agent systems: Semantics and dynamics of organizational models ( 2009 )

15. Dubnick , M.J. , Justice , J.B. : Accounting for accountability ( September 2004 ), https://pdfs.semanticscholar.org/b204/36ed2c186568612f99cb8383711c554e7c70.pdf, annual Meeting of the American Political Science Association

16. Feltus , C. : Aligning Access Rights to Governance Needs with the Responsability MetaModel (ReMMo) in the Frame of Enterprise Architecture . Ph.D. thesis , University of Namur, Belgium ( 2014 )

17. Fornara , N. , Vigano , F. , Verdicchio , M. , Colombetti , M. : Arti cial institutions: a model of institutional reality for open multiagent systems . Arti cial Intelligence and Law 16 ( 1 ), 89 { 105 ( 2008 ). https://doi.org/10.1007/s10506-007-9055-z

18. Gar nkel, H.: Studies in ethnomethodology . Prentice-Hall Inc., Englewood Cli s, New Jersey ( 1967 )

19. Grant , R.W. , Keohane , R.O. : Accountability and Abuses of Power in World Politics. The American Political Science Review 99 ( 1 ) ( 2005 )

20. Haller , P. , Sommers , F. : Actors in Scala - concurrent programming for the multicore era . Artima ( 2011 )

21. Hull , R. , Damaggio , E. , De Masellis , R. , Fournier , F. , Gupta , M. , Fenno F. Terry Heath III , Hobson , S. , Linehan , M.H. , Maradugu , S. , Nigam , A. , Sukaviriya , P.N. , Vacul n , R.: Business artifacts with guard-stage-milestone lifecycles: managing artifact interactions with conditions and events . In: Proc. of the Fifth DEBS . pp. 51 { 62 . ACM ( 2011 ). https://doi.org/10.1145/2002259.2002270, http://doi.acm. org/10 .1145/2002259.2002270

22. Mallya , A.U. , Singh , M.P. : Modeling exceptions via commitment protocols . In: Proceedings of the Fourth International Joint Conference on Autonomous Agents and Multiagent Systems . pp. 122 { 129 . AAMAS '05, ACM ( 2005 )

23. Nissenbaum , H.: Accountability in a computerized society . Science and Engineering Ethics 2 ( 1 ), 25 { 42 ( 1996 )

24. Object Management Group (OMG): Case Management Model and Notation (CMMN), Version 1.1 . OMG Document Number formal/2016-12-01 (http://www.omg.org/spec/CMMN/1.1/PDF) ( 2006 )

25. Object Management Group (OMG): BPMN Speci cation - Business Process Model and Notation ( 2018 ), http://www.bpmn.org/, online, accessed 08 /11/2018

26. Platon , E. , Sabouret , N. , Honiden , S.: An architecture for exception management in multiagent systems . Int. J. Agent-Oriented Softw. Eng . 2 ( 3 ), 267 { 289 ( 2008 )

27. Ricci , A. , Bordini , R.H. , Hubner, J.F. , Collier , R.: AgentSpeak(ER): An Extension of AgentSpeak(L) improving Encapsulation and Reasoning about Goals . In: AAMAS . pp. 2054 { 2056 . International Foundation for Autonomous Agents and Multiagent Systems Richland , SC, USA / ACM ( 2018 )

28. Singh , M.P. : Distributed Enactment of Multiagent Work ows: Temporal Logic for Web Service Composition . In: The Second International Joint Conference on Autonomous Agents & Multiagent Systems, AAMAS 2003, July 14-18 , 2003 , Melbourne, Victoria, Australia, Proceedings. pp. 907 { 914 . ACM ( 2003 )

29. Sommerville , I. , Lock , R. , Storer , T. , Dobson , J.: Deriving information requirements from responsibility models . In: Advanced Information Systems Engineering , 21st International Conference, CAiSE 2009 , Amsterdam, The Netherlands, June 8-12, 2009 . Proceedings. pp. 515 { 529 ( 2009 )

30. Souchon , F. , Dony , C. , Urtado , C. , Vauttier , S. : Improving exception handling in multi-agent systems . In: Software Engineering for Multi-Agent Systems II . pp. 167 { 188 . Springer Berlin Heidelberg ( 2004 )

31. Vincent , N.A. : Moral Responsibility, Library of Ethics and Applied Philosophy , vol. 27 , chap . A Structured Taxonomy of Responsibility Concepts . Springer ( 2011 )

32. Weske , M. : Business Process Management: Concepts , Languages, Architectures. Springer ( 2007 )

33. Yazdanpanah , V. , Dastani , M. : Distant group responsibility in multi-agent systems . In: PRIMA 2016 : Princiles and Practice of Multi-Agent Systems - 19th International Conference, Phuket, Thailand, August 22-26 , 2016 , Proceedings. pp. 261 { 278 ( 2016 ). https://doi.org/10.1007/978-3- 319 -44832-9 16