1 Towards an architecture for self-regulating agents: a case study in international trade Brigitte Burgemeestre, Joris Hulstijn, and Yao-Hua Tan - and open agent systems. An enforcement mechanism that Abstract—Norm-enforcement models applied in human elaborates on an agent’s internal architecture to achieve societies may serve as an inspiration for the design of multi-agent compliant behavior, and does not require additional systems. Models for norm-enforcement in multi-agent systems ‘observers’ is self-regulation. Self-regulation is a control often focus either on the intra- or inter-agent level. We propose a combined approach to identify objectives for an architecture for approach in which rule making and/ or enforcement are carried self-regulating agents. In this paper we assess how changes on the out by the agent itself, instead of a regulator agent or inter-agent level affect the intra-agent level and how a generic institution. It can be an alternative or extension to direct BDI architecture IRMA can be adapted for self-regulation. The control, when external supervision and norm enforcement are approach is validated with a case study of AEO certification, a not possible at all, are ineffective or when there is a lack of European wide customs initiative to secure the supply chain while controlling resources. For example, in e-institutions it might be facilitating international trade. impossible to check all agent actions for compliance in real Index Terms—self-regulation, agent architectures, norm time. A solution then might be to do a code review up compliance forehand and determine if an agent is compliant by design. In human societies programs of self–regulation have been found I. INTRODUCTION to contribute to expanded control coverage and greater inspectorial depth [2]. Self regulation can be implemented in T o motivate autonomous agents to comply with norms various enforcement mechanisms have been proposed. Norms here define standards of behavior that are acceptable in various ways: from voluntary self regulation, where a group of agents voluntary chooses to regulate themselves, to mandated a society, indicating desirable behaviors that should be carried or enforced self-regulation, where a government agency out, as well as undesirable behaviors that should be avoided delegates some of its regulative and enforcing tasks to the [8]. Enforcement mechanisms often require the introduction of agents subjected to the norm, but retains the supervision, to a special “observers” or “regulator agents” that actively monitor combination of mandated self regulation and direct control by the behavior of the other agents [1]. Such agents are assigned regulator agents [10]. Each model of self-regulation causes to monitor the behavior of other agents and sanction them in different agent dependencies and information needs, which case of norm violations. When developing norm enforcement imposes different requirements on the IT architecture. mechanisms for multi-agent systems, the modeling is often A special case of self regulation for international trade is the focused on the inter-agent level (between agents). Such models Authorized Economic Operator (AEO) program [7]. The AEO aim to analyze agent interactions and dependencies to program is a European wide customs initiative that aims to construct norm enforcement mechanisms. The intra-level secure the supply chain while at the same time reducing the (inside the agent) is mainly treated as a black box. We argue administrative burden for companies through the use of self- that the intra- and inter-agent aspects cannot be viewed control. Companies that are reliable in the context of customs separately from each other, especially in norm enforcement related operations and have a good internal control system may where perceptions of external stimuli should motivate an agent apply for the AEO certificate and receive operational benefits to adapt its behavior and thereby its internal mechanisms. from simplified customs procedures, preferential treatment, Norm-enforcement models applied in human societies may and less physical inspections. Companies that do not have an serve as an inspiration for the design of electronic institutions AEO certificate remain subject to the current level of customs controls. Participation in the AEO program is voluntary, but effective self-control is an obligatory requirement. Manuscript received June 26, 2009. This research was partially funded by Implementing self-regulation as a control mechanism thus PGS IT Audit of the VU Amsterdam and the integrated project ITAIDE of the 6th Framework of the IST Programme of the European Commission results in a redistribution or delegation of control tasks among All authors are with the faculty of Economics and Business Administration the actors. Agents have to adapt their internal mechanisms to of the VU University, Amsterdam, The Netherlands. The third author is also cope with these tasks. We see that changes at the inter-agent with the Dept. of Technology, Policy and Management of the Technical level affect the intra-level. We therefore propose a combined University Delft. Brigitte Burgemeestre (e-mail: cburgemeestre@feweb.vu.nl). approach to develop an architecture to embed self-regulation Joris Hulstijn (e-mail: jhulstijn@feweb.vu.nl). as a control mechanism for multi-agent systems. Yao-Hua Tan (e-mail: ytan@feweb.vu.nl). 2 In this paper we present our first steps towards an norms for A. R uses information about A and A’s actions to architecture for self-regulating agents. The research questions select the appropriate norms from the norm framework that we like to answer in this paper are: 1. What objectives need to apply to A’s specific situation. R2 determines ‘control be met by an architecture on self-regulating agents? 2. How do indicators’ of A. A ‘control indicator’ is the kind of evidence we need to adapt existing Beliefs Desires Intentions (BDI) [9] required to demonstrate compliance of a norm, as well as architectures? As a starting point we propose a combination of infrastructural requirements to collect that evidence. For frameworks to cover the inter- as well as the intra-agent example: when a company sends an invoice, they always make analysis. For the inter-agent analysis the Intelligent Resource- a copy of the invoice and store the copy to be able to check if Bounded Machine Architecture (IRMA) [3] is a good starting the invoice payments are correct and complete. R3 is the point because it is a general BDI architecture that is well monitoring performed by R on A’s actions, based on accepted and has formed the basis for more recent agent information provided by A about the control indicators. R4 architectures. Software engineering methodology TROPOS describes the plan of R to sanction A in case of a norm [6] provides suitable concepts to analyze and model agents’ violation. Agent A’s model is quite simple, as A is a ‘blind’ dependencies. We analyze direct regulation and self-regulation agent that has no knowledge about the norms or control using TROPOS (Section II). Using this analysis we generalize indicators and only acts. Therefore it is possible that A the objectives for the internal architecture of a self-regulating unknowingly engages in an activity that violates a norm that is agent. We try to embed the normative objectives in IRMA imposed upon A by R. However, we do assume that A (Section III). Using the extended architecture and TROPOS remembers action-sanction relations and that it can decide to model, we analyze a case study of AEO (Section IV). We cancel an action that will lead to a sanction. Figure 1 shows the examine if our adapted version of the architecture covers the dependency analysis for direct control. findings of the case study. We identify its suitability and the shortcomings. II. INTER-AGENT ANALYSIS We first analyze the agents and the dependencies among agents. To do this we use concepts from the early requirements phase of the TROPOS methodology [4], which is derived from the i*conceptual framework[11]. The key concepts we use are: actor, goal, plan, resource and dependency. An actor can be an autonomous agent that has a goal or strategic interest. A goal can be satisfied through the execution of a plan, which is an abstract representation of a way of doing something. A resource can be a physical or informational entity. Actors can depend on each other to reach a certain goal, to execute a plan or to obtain resources. The agent that depends on another agent is called the depender, the agent he depends on is called the dependee. The object which is the subject of the dependency relation is called the dependum. We first model the direct control approach where the actions of autonomous agents are regulated by special regulator agents. After that we analyze self-regulation and assess what Figure 1: TROPOS model of direct control. The actions of changes when an autonomous agent internalizes control tasks an actor (A) are regulated by a regulator (R). Note that of the regulator agent. arrows depict dependency, not information flow. So to A. Agents’ dependencies in direct control regulate A’s actions, R depends on A for info about actor and actions. In direct control we have two types of agents: an Actor agent (A) that is carrying out an activity and a Regulator agent B. Agents’ dependencies in self-regulation (R) that is responsible for regulating A’s actions such that For self-regulation we start again with two types of agents: agent A complies with the norms that are applicable to A. An the actor agent (A) and the regulator agent (R). In self- agent can violate the norms through pursuing an illegal goal or regulation control tasks are delegated from R to A. Since A is by performing an illegitimate action. We assume that R has a autonomous, R can never be absolutely certain that A norm framework from which it derives the set of norms complies. R thus has to implement a mechanism to motivate A tailored to an agent’s specific situation. To regulate A, agent R to regulate itself appropriately. Furthermore to maintain the has to have the following plans: R1: Specify norms for actor, power of the regulator to handle non-compliant agents, the R2 ‘Determine control indicators of actor’, R3 ‘Monitor sanctioning task (R4) remains the regulators responsibility. actor’s actions’ and R4 ‘Sanction actor’. R1 generates a set of We first consider the consequences of the internalization of 3 control tasks by A. Plans R1, R2 and R3 may be internalized instead of its activities. In auditing R5 refers to a system-based by agent A as plans: A1 ‘Specify norms’, A2 ‘Determine audit, were the focus is on the control system itself instead of control indicators’ and A3 ‘Monitor actions’. A1 specifies the business transactions. Before an agent thus can enter in a norms based on a norm framework which originates from R. self-regulative relation it has to provide for its authenticated This entails a new dependency between A and R: A now control architecture or control script to the regulator. Figure 2 depends on R for communicating the norm framework. When shows the dependencies between agents A and R when they the norm specification is done by A, A is also supposed to be engage in self-regulation. When we compare direct control able to differentiate between norm violations and norm with self-regulation we see that A internalizes some of R’s compliance. A therefore no longer depends for information control activities on A. New information resources have to be about violations and permissions on R, but has to do it himself. gathered to be used within the control activities. Also new A2 defines control indicators about A’s actions, based on the goals evolve and consequently the adoption of new plans. In norms defined in A1. A3 describes the monitoring actions of A correspondence new dependencies between R and A develop which it performs in the context of the control indicators from for the acquisition of other information resources plan A2. The plans A1, A2, and A3 together, should support A Summarizing, a self-regulating agent has to have the to act compliantly with the norms. The acts of A in return capabilities to: (1) Detect, internalize and store applicable affect the nature of the control actions. If A starts doing norms in the environment, (2) Translate norms into measurable different activities the control indicators may become less control indicators, and (3) ‘Monitor, detect and mitigate effective and A therefore has to determine new control possible norm violations’. In the next section we zoom into the indicators that cover the norms. For example, if A replaces the internal architecture of the actor agent in self-regulation process of sending paper invoices to its customers by sending them electronic invoices, new control indicators are required; III. INTER-AGENT ANALYSIS e.g. log files instead of paper copies of the invoice. We now analyze how the new tasks and dependencies revealed by the TROPOS models affect an agent’s internal architecture. We acknowledge that these tasks are complex normative tasks As a basis for our model we use the Intelligent Resource-Bounded Machine Architecture (IRMA) [3]. The architecture is a BDI architecture where the intentions are structured into plans. A plan can be the plan that an agent has actually adopted, or a plan-as-recipe that is stored into the plan library. Plan options are proposed as a result of means-end reasoning or by the opportunity analyzer. The opportunity analyzer detects changes in the environment and determines new opportunities, based on the agent’s desires. The options are filtered through a compatibility filter, that checks the options to determine compatibility with the agent’s existing plans, and a filter override mechanism, in which the conditions are defined under which (portions) of plans need to be suspended and replaced by another option. The deliberation process determines the best option on the basis of current beliefs and desires. Consider an autonomous agent that likes to achieve a certain goal. The agent has already several plans of action available (in its plan-library) to reach this goal. Before deliberating on a plan, the agent engages in a filtering process. This process Figure 2: TROPOS model of self-regulation, control tasks constrains the agent’s possible plans, to plans that can be of the regulator are internalized by the actor agent. completed given its available (sub) plans in the plan library, its Now we describe the consequences of A’s internalization of beliefs and desires. The agent chooses from this selection the the control tasks of R’s goals and plans. Since A now has to best plan, given its beliefs and desires, and executes the plan. control its own actions, the goal of R to regulate A’s actions is Figure 3 shows our extension of the IRMA architecture, supposed to be met by the control activities of A. To determine adapted for self-regulation. Norm related adaptations are if this delegation of control is effective, R’s has adopted a new shown in grey and dotted lines. The ovals in the figure are goal which is to regulate the control activities of A. To reach information stores (repositories) and the rectangles are process this goal, R also has defined a new plan (R5). R5 describes the modules. activities of R to monitor and evaluate A’s control actions. R Within IRMA we like to implement the processes and now depends on A for information about its control activities information stores that are needed for self-regulation. A self- 4 regulating agent needs to internalize certain control activities implementations make it possible for an agent to decide not to to control its actions. The activities are: specify norms (A1), consider a plan option that aims at buying a snake skin determine control indicators (A2), and monitor actions (A3). handbag. The opportunity analyzer may use the norms and These control activities require input from the agent’s actions, beliefs to search for an alternative, such as a fake snake skin and the actions in turn are influenced by the norms. We first handbag. analyze what modules IRMA are possibly affected by We find that norms can impact all components of the normative reasoning architecture. To assure consistent norm application we propose a central information-storage for norms similar to what the plan library is for plans. Activity A1 updates the norm library according to the beliefs of the agent. Only norms that are considered to be applicable to the agent’s specific situation are included. To make an agent aware of a norm (violation) we connect the norm library with the reasoner module that is attached to the beliefs. If an agent then reasons about its beliefs, it takes the norms into account. Beliefs about a norm (violation) can be used as input for the means-end reasoner, opportunity analyzer and the deliberation process. Besides that, the agent may use its knowledge about norms to determine the control indicators of A2. We consider the filtering process the best location to implement the control indicators. Beliefs about norms are already included in the other reasoning processes. The filtering process and reasoning thus together consider (non-) compliant behavior. We think that the majority of the control indicators should be embedded in the compatibility filter and only severe violations should be handled by the filter override mechanism. Otherwise it could happen that the filtering is too strict. The monitoring in A3 is handled through a comparison of the beliefs about the data on the indicators with the norms. Based on results from this Figure 3: A reasoning component for self-regulating agents adapted from [3] analysis controls in the filtering process may be adapted. Norms can impact the information stores and or processes Figure 3 shows an adapted version of the rational agent of the architecture. A norm can be implemented in plans and reasoning architecture for self-regulation. function as a threshold to restrict the outcome. For example, a Our approach of embedding norms into the filtering process thermostat function that tries to keep the room heated at a is compatible with the framework that is proposed by [8]. certain temperature. Norms can also restrict the possible set of Norms can also be implemented into the goal generation plans. Plans that violate the norm are not stored in the plan mechanism as was done in the BOID architecture [5]. In library. Or in means-end reasoning: there are illegal plans BOID one can distinguish two kinds of goals: internal available in the plan library but we do not consider them as motivations (desires), representing individual wants or needs, appropriate options to reach a goal. Norms can also prevent and external motivations (obligations) to model social the actual execution of a plan. For example, a person can plan commitments and norms[5]. All these potential goals may to rob a bank, but decide not to do so. conflict with each other. To resolve conflicts among the sets of Besides that, norms affect the beliefs, and beliefs affect the beliefs, obligations, intentions and desires, a priority order is norms. An agent may realize, based on its beliefs, that it is needed. In the BOID, such a (partial) ordering is provided by acting non-compliant with the norms. Or, an agent realizes that the agent type. due to a change in activities certain norms are no longer applicable and new norms must be incorporated. When an IV. CASE STUDY AEO CERTIFICATION agent adopts a new norm, this must be known (believed). We use our models to analyze a specific case of self- Norms are also related to the desires of an agent. An agent’s regulation: AEO certification. The case study results are based desires may violate the norms. For example, an agent may on document analysis and a series of semi-structured desire a handbag that is made of the skin of a protected snake. interviews with experts from Dutch Tax and Customs A norm is that killing a protected animal is illegal. If norms are Administration, held in the period of May till November 2009. included in the compatibility filter, an agent can check if an Meeting notes were made by the authors and verified by option is compatible with its norms. If norms are part of the interview partners. Intermediate results of the case study were filter override mechanism, non-compliance can be a condition validated in a one-day workshop. under which an agent always has to reconsider its plans. Both An Authorized Economic Operator (AEO) can be defined as 5 a company that is in-control of its own business processes, and if the likelihood of a threat is limited and the risk is partially hence is reliable throughout the EU in the context of its covered, or if the costs for complete coverage are very high. customs related operations [7]. Typically, modern enterprise The company has to motivate its choices in its system of information systems (e.g ERP, CRM etc.) play an essential control measures to customs. It has to show how its risk role for companies to be in-control. AEO’s will receive several management approach contributes to being a self-controlling benefits in customs handling, such as a “Green Lane” and reliable party. The company therefore evaluates the treatment with a reduced number of inspections. These effective implementation of the proposed measures, using the benefits can lead to considerable cost-reductions for COSO internal control scoring definitions. COSO is a businesses. For non-certified enterprises customs will continue framework for risk management and internal control [12]. The to carry out the traditional supervision. Customs can thus scores range from 0 “no control measures in place”, 1 “internal direct their efforts towards non-certified companies to increase control is ad hoc and unorganized”, 2 “internal control has a the security of international supply chains, while at the same structured approach”, 3 “internal control is documented and time reducing the administrative burden for AEOs. known”, 4 “internal control is subject to internal audits and To qualify as AEO, a company must meet a number of evaluation” until 5 “internal control measures are integrated criteria, which are described in the community customs code into the business processes and continuously evaluated”. This and the AEO guidelines [7], which are developed by the scoring provides the customs with an indication of the maturity European Commission. Part of the application procedure is a level of the company’s self-controlling abilities. self-assessment on the quality of the company’s internal B. Case analysis control system for aspects that are relevant to the type of AEO certificate (‘Customs simplifications’, ‘Security and safety’ or In the AEO case study we see the implementations of tasks ‘Combined’ [7]). The company’s approach and the results of A1, A2, and A3 at the company’s side. A company has to the self-assessment are inspected by customs. The customs define a control system appropriate to handle its specific risks. determine whether the self-assessment is performed well and The company therefore translates the general AEO guidelines whether the results indicate that a company is able to control into norms that are applicable in its own practice and its business processes such that they contribute to a secure circumstances (compatible with A1). Thereby a company supply chain. If this is the case and the other requirements are determines parameters to control its business processes (A2). met an AEO certificate is issued by the customs office. Next A company with a control system of a high maturity level we focus on the self-assessment task. monitors its actions (A3) through internal audits and controls that are integrated in the processes. The customs replaces its A. The self-assessment task traditional controls of the company’s processes (R1, R2, R3) The company’s first task is to collect information related to by an assessment of the company’s self-regulating capabilities the specific nature of the company to focus the self- and monitors the control actions of the company (R5). We also assessment. This step is called ‘Understanding the business’. observe dependencies on information needs. The company The next step is to identify (potential) risks to which the depends on abstract norms (e.g. the AEO Guidelines) provided business is exposed using the AEO guidelines, which provide by the customs, which they try to apply to themselves as an overview of general risk and attention points. The company customs would do. The customs on the other hand depends on determines which sections are important according to the the company for information about their control system. nature of the business activities. A company then has to The AEO case provides us a new approach of control that identify, what risks affect the supply chain’s safety, and are could be applied to a multi agent system. It shows that norm therefore of interest of the customs authorities. The company enforcement can be a task that can be distributed between thus replaces the customs’ task of risk identification. For various types of agents. Furthermore we learned that self- example, computer components are valuable goods, which are regulation only works under certain conditions and that subject to theft. Trading valuable goods requires more security delegating control tasks is not simple. In general companies measures, than, say, trading in a mass product like fertilizer. find it difficult to do a self-assessment as they do not know However, some ingredients of fertilizer may be used to what customs expects from them. Especially the specification assemble explosives, leading to a different set of risks of abstract norms of the AEO guidelines into company specific A company then assess if appropriate internal control concrete norms proved to be hard. For companies it is thus measures are taken to mitigate these risks. The vulnerability of unclear when they have taken sufficient measures to secure a company to threats depends on its current control measures. their part of the supply chain. Companies expect from the Control measures either reduce the likelihood, by dealing with customs to indicate on a more detailed level what is sufficient: vulnerabilities (preventative controls), or reduce the impact “A fence for a chemical company should be X meters high”. (detective and corrective controls). A robust system of controls Even for customs such knowledge is often only implicitly is thus able to prevent, detect and correct threats. A robust available as “expert knowledge” that is difficult to externalize system of controls should also monitor its own functioning. and make accessible for companies. For risks that are not controlled, additional measures may be When we look at the company’s internal control system we implemented or the risk is “accepted”. Risks can be accepted, see that norms have to be internalized based on perceptions of 6 the environment. Only applicable norms are implemented. The identify objectives for an architecture for self-regulation. We norms have to be implemented in a systematic and structured identified key processes and their influence on the way such that they detect norm violations and prevent them dependencies between agents and the internal agent from occurring. In the architecture we see norms implemented architecture. The models provide insight in differences in as a filtering mechanism. In the AEO certification we see norm requirements for direct controlled agents and self-regulating control as a structured process. In addition, mature self- agents. The analysis also points out the limitations of some controlling companies may have controls integrated in the well-known existing approaches. IRMA lacked in reflective processes or audits to check the functioning of the controls. capabilities and is therefore not sufficient to model a truly self- The total control system of a company could be seen as their regulating agent: an agent that is able to learn from its implementation of the internal control architecture. Therefore experiences with norms and use these experiences as these new monitoring activities of customs in the AEO case constraints for future normative reasoning. Also unaddressed could be seen as quality assessment of such a control were aspects of norm communication. For two agents to architecture rather than the traditional role of Customs to engage in a self-regulation relation, they must able to control the specific business operations of the company. This communicate the norms effectively. Since the agents are fundamental change in the controlling role of the government autonomous we cannot simply assume that both agents use is often referred to as the transformation from operational similar vocabularies or protocols [6]. A solution for norm control to meta-control, where operational control is delegated communication should take the agent’s autonomy into account. by the Customs to the companies themselves. Future research will zoom in on the role of reflection on normative behavior and the communication of norms. Besides V. DISCUSSION that we are also interested in the evolution process of an agent The combination of TROPOS and IRMA for self-regulating from direct control to self-regulation. agents also has its limitations. However, we do not claim that these are the best approaches currently available. Instead we Acknowledgments We would like to thank the Dutch Tax and used the approaches as a means to identify requirements for Customs administration for their discussions. self-regulating agent at the intra- and inter-agent level. Below we describe the two most important limitations. REFERENCES First, the most important limitation of the architecture is that [1] G. Boella, L. van der Torre, and H. Verhagen: Introduction to normative multiagent systems. Computational and Mathematical Organization it is not reflective. By this we mean that agents cannot learn Theory, 12:71–79, 2006P from their mistakes. When the agent determines that a plan [2] J. Braithwaite: Enforced self-regulation: a new strategy for corporate contains or leads to a norm violation it is only able to cancel crime control. Michigan law review vol. 80, pp 1466-1506, 1982 [3] M. E. Bratman, D. Israel, and M. Pollack. Plans and resource-bounded this plan as a current possible option. It lacks mechanisms to practical reasoning. In R. Cummins and J. L. Pollock, editors, delete or change such plans in a plan library. Desires that Philosophy and AI: Essays at the Interface, pages 1--22. The MIT Press, violate norms can also not be changed. The agent therefore Cambridge, Massachusetts, 1991. [4] P. Bresciani, A. Perini, P. Giorgini, F. Giunchiglia, and J. Mylopoulos. keeps proposing violating plans and desires. Since norms are Tropos: An agent-oriented software development methodology. Journal context dependent it is quite complex to differentiate violating of Autonomous Agents and Multi-Agent Systems, 8, pp. 203–236, plans from non-violating plans. Plans that are allowed in one 2004. situation may be a violation under different circumstances. An [5] J. Broersen, M. Dastani, J. Hulstijn, Z. Huang and L. van der Torre. The BOID architecture - Conflicts between beliefs, Obligations, Intentions adaption of the plan mechanism is needed. and Desires. In Proceedings of the Fifth International Conference on Secondly, there seems to be fundamental problem in Autonomous Agents, pp. 9-16, Montreal Canada, May 2001 delegation of control; namely that often it is not clear how to [6] C.B. Burgemeestre, J. Liu, J. Hulstijn and Y. Tan: Early Requirements Engineering for e-Customs Decision Support: Assessing Overlap in communicate the delegated norms from the regulator agent to Mental Models. In the Forum Proceedings of the 21st CAiSE the regulated agent. For companies it is difficult to interpret conference, pp 31-36, Amsterdam , The Netherlands 2009 and implement the customs’ norms for their business activities. [7] European Commission: AEO Guidelines, TAXUD/2006/1450, 2007. http://ec.europa.eu/taxation_customs/customs/policy_issues/customs_se Should customs and companies implement protocols, a curity/aeo/ vocabulary or procedures such that they effectively can [8] F. Meneguzzi and M. Luck: Norm-based behaviour modification in BDI communicate information? How should a company make its agents. Proceedings of the 8th International Conference on Autonomous internal control system available to customs, such that they can Agents and Multiagent Systems (AAMAS), Budapest, Hungary, 2009. [9] A.S. Rao and M.P. Georgeff: Modelling rational agents within a BDI- determine the quality of a control system in a specific context architecture, in proceedings of Knowledge representation and reasoning with limited expert knowledge? These and related questions (KRR-91) Conference, San Mateo CA, 1991. have to be answered through a study of norm communication [10] J. Rees, Self Regulation: An Effective Alternative to Direct Regulation by OSHA? , Policy Studies Journal, 16:3 pp 602-614, 1988 between agents. [11] E.K.S. Yu, Towards Modelling and Reasoning Support for Early-Phase Requirements Engineering, in: Proceedings of the Third IEEE VI. CONCLUSION AND FURTHER RESEARCH International Symposium on Requirements engineering, pp. 226-235, 1997 A combined approach, that analyses the inter- (between [12] COSO enterprise risk management framework. Available: agents) and intra-agent level (inside agents), was suitable to http://www.coso.org