1 Call Graph Profiling for Multi Agent Systems Dinh Doan Van Bien, David Lillis and Rem W. Collier School of Computer Science and Informatics University College Dublin dinh@doanvanbien.com, {david.lillis, rem.collier}@ucd.ie Abstract—The design, implementation and testing of Multi introduce the concept of a call graph, and analyse how the Agent Systems is typically a very complex task. While a number traditional concept of a call graph can be applied to a MAS. of specialist agent programming languages and toolkits have been Section V presents the concrete implementation of an agent created to aid in the development of such systems, the provision of associated development tools still lags behind those available for call graph within AgentSpotter profiling tool, followed by a other programming paradigms. This includes tools such as debug- discussion of the proposed approach in Section VII. Finally, gers and profilers to help analyse system behaviour, performance we conclude and outline ideas for future work in Section VIII. and efficiency. AgentSpotter is a profiling tool designed specifically to operate on the concepts of agent-oriented programming. This paper extends previous work on AgentSpotter by discussing its II. R ELATED W ORK Call Graph View, which presents system performance information, with reference to the communication between the agents in the The work presented in this paper draws from two principal system. This is aimed at aiding developers in examining the effect research areas. Firstly, in order to provide a profiling tool for that agent communication has on the processing requirements of MASs, it is necessary to examine the concepts and features the system. of existing profiling tools for other programming paradigms, such as object-oriented programming. It is also necessary to explore the available programming tools aimed at aiding the I. I NTRODUCTION debugging and profiling of MASs. By its nature, a Multi Agent System (MAS) is a complex Initially proposed by Knuth, the key motivating factor behind system consisting of loosely-coupled autonomous software profiling tools is his observation that “less than 4% of a program entities that are required to communicate with one another in accounts for more than half of its running time” [3]. By order to achieve individual or system objectives. To facilitate identifying and improving code that represents a performance the development of such systems, a number of agent-oriented bottleneck, software developers can greatly improve the overall programming languages and MAS toolkits have been developed performance of their programs. An important motivator for by a variety of researchers [1]. However, the availability of the use of specialist profilers to identify these bottlenecks ancillary tools to aid with debugging and profiling is limited, is the frequent tendency of developers’ mental map of their particularly when compared with the available tools for other programming not matching the reality of how the program programming paradigms and languages. behaves. Thus, areas of concern that programmers may not Previous work introduced AgentSpotter, a profiling tool have considered will be identified by the profiler. designed specifically for MASs [2]. Profiling is a performance In the context of more traditional, non-MAS, programming, analysis technique that is based on the notion that in a program, developers generally have access to long-established and only a few places, called bottlenecks or hot spots, can account widely-accepted profiling tools such as gprof [4] or performance for the majority of the execution time of a program. Hence, analysis APIs such as the Java Virtual Machine Tool Interface by fixing only these sections of the code, the performance (JVMTI) [5] or ATOM [6]. However, those developing MASs of a program can be substantially improved. Profiling was do not tend to have access to such well-established tools. introduced almost 40 years ago by Donald E. Knuth in his One MAS framework that does provide the ability to glean empirical study of FORTRAN programs [3], and has since data about system performance is Cougaar [7]. This provides been successfully adapted to a variety of different languages, access to data on historical performance data, event detection, platforms and software architectures, including large distributed monitoring of ACL messages and a number of other services. systems. The LS/TS agent platform provides an administrator tool that The aim of this paper is to continue the mapping of traditional records some high-level system monitoring information [8]. The concepts to those of agent oriented software engineering so as main limitation of these systems is the lack of post-processing to facilitate the compilation of useful profiling data, presented in of the raw performance data in order to produce meaningful an intuitive, visual fashion in order to aid multi agent developers synthetic indicators like a profiler would do. in improving the performance of their systems. Besides performance analysis, most agent frameworks pro- Section II provides a brief discussion of some related tools vide a debugging tool similar to the Agent Factory Debug- that have been developed for debugging and profiling MASs. ger [9], which provides information about the mental state In Section III, we give a brief overview of the AgentSpotter and communication from the viewpoint of individual agents. agent profiling application. Following this, in Section IV we A different type of debugging tool is the Agent Viewer that 2 is provided in the Brahms toolkit [10], which displays agent the development and deployment of agent-oriented applications. timelines so as to understand when agents’ actions are taken. A more detailed description of this implementation and the As the work in this paper also requires the monitoring of data gathered by AgentSpotter can be found in [2]. inter-agent communication (see Section IV), it is also important Previously, it was shown how AgentSpotter was used to to acknowledge the availability of existing communication- map traditional profiling concepts onto agent-oriented concepts. analyis tools for MAS platforms. A number of such tools have This focused on two types of visualisation: been developed for a variety of agent frameworks and toolkits • Flat Profile: provides data on such things as agent activity, to aid developers in understanding the interaction between messages and reasoning/action duration in a tabular form. agents in their systems. An early example of such a toolkit is • Space-Time Diagram: provides a navigable visualisation Zeus [11], which contains a “society tool” that visualises the representing the data from the flat profile in a more interaction between agents, so as to help in understanding the intuitive manner. topology of the social contacts within the MAS. This type of The focus of this paper is on an agent-oriented call graph. tool also aids in debugging MASs, since developers can ensure Whereas a space-time diagram can aid in identifying the timing that the expected communication and collaboration between and extent of actions executed by agents, a call-graph is agents is indeed taking place. traditionally intended to also indicate the reasons why particular In the JADE agent development framework, a Sniffer Agent actions were undertaken at particular times. is a FIPA-compliant agent that monitors messages created with an Agent Communication Language (ACL) passed between IV. C ALL G RAPH C ONCEPT agents and presents these in a simple graphical interface [12]. A. Traditional Call Graph A more sophisticated tool, called ACLAnalyser, provides more detailed information on agent communication [13]. Again, the The concept of a call graph was introduced in 1982 in the principal aim of this is to aid in debugging errors in MASs “gprof” profiling tool [4]. This is an improvement on the popular that relate to coordination or cooperation. “prof” UNIX profiling tool. In additional to summarising the time spent in different functions, it also recursively presents III. AGENT S POTTER all the call stacks annotated with the time spent in the various functions that are called. Another name for the call graph AgentSpotter is a profiling tool designed specifically for is “hierarchical profile”, which conveys the idea that gprof gathering and displaying profiling information on MASs [2]. provides information to aid in understanding the impact of one Figure 1 illustrates the abstract architecture of the system, function in relation to all the functions that have called it. designed to be compatible with any type of agent platform. The Although the textual output of gprof is very dense and AgentSpotter Service runs within the Run-Time Environment of requires some practice to understand, modern profiler user an Agent Platform, gathering data about the agents themselves interfaces have made call graphs more tractable by presenting (actions performed, messages exchanged), along with system them as tree view controls that can be interactively explored. data such as CPU and memory usage. This is the only portion of the system that must be ported in order to be run on different agent platforms. The data gathered is logged into a Snapshot File, which allows it to be accessed and analysed offline, once the system has finished running. AgentSpotter Station is a visual application that provides a number of visualisations on various aspects of system perfor- mance, in order to help programmers to identify performance bottlenecks in their code. AGENT PLATFORM Profiled Application messages Run-Time Environment AgentSpotter Service activity events Profiler events System Monitors data AGENTSPOTTER STATION (visualisation application) QUERY ENGINE Snapshot File (.aspot) Session Summary Tables queries Session summary query Agent activity query Space-Time Diagram data Events Messages sent/received query Sessions Call graph query Call Graph Tree View Messages System activity query Agents Fig. 2. Call Graph Tree View of a fictional C program that removes duplicate lines from a file Fig. 1. AgentSpotter Architecture Figure 2 shows an example of a typical call graph. This relates to a fictional C language program that is designed to The outputs utilised in this paper are gleaned from running a remove duplicate lines from a text file. In this tree view, the concrete implementation of the AgentSpotter service within the root node is the main function, which represents 100% of the Agent Factory framework [14]. Agent Factory is a modular and total execution time of the program (including the execution extensible framework that provides comprehensive support for time of its child nodes). Each node represents a function within 3 Agent A the program, with the child nodes representing functions that are called within the parent function. The percentages represent the cumulative proportion of the program’s execution time that message Mα at time α impact of MΩ is attributable to a node and its children. In the example, the { Agent B α β Ω main function calls process file, which is then represented { as a sub-tree with leaves representing its own calls to the impact of Mα message MΩ at time Ω bubble sort, load file and remove duplicates functions. The key benefit of the call graph tree view is the extended Agent X context it gives to performance information. For instance, this simple example reveals that the program spends 90% of its time processing a file. The tree shows that one of the top- Legend: level function process file’s callees, the bubble sort operation, Message event Agent activity accounts for 80% of its caller time. A flat profile would have Agent life line shown the time for these functions separately without explicitly time showing the hierarchical link between them. Fig. 3. Agent Message Impact Concept Diagram B. Agent-Oriented Call Graph Model When constructing a flat profile for a MAS, it was necessary of Mα on agent B. Note that the outgoing message at time to map a number of concepts relating to traditional program- stamp β does not break the computation sequence. ming to equivalent concepts in the domain of agent-oriented It is now easy to determine the total impact Tx,y of all programming [2]. A similar mapping must be performed in the messages sent by a given agent x to another agent y. Let order to allow for the development of an agent-oriented call M be the total number of messages sent, 1 ≤ m ≤ M a graph. single message impact identifier, αm the reception time stamp The central measure used in the traditional call graph is of message m from x to y, and Ωm , where αm ≤ Ωm , the the function execution time. Each node represents a function, next reception time stamp message coming right after m from which can take the action of calling other functions as part any other source. The total impact Tx,y is then given by the of its execution. The consequence of this action is that some equation: amount of time is spent executing the child function. Thus M X Ωm we can say that the impact of calling a function is that this X Tx,y = bt (2) additional processing time has been incurred. m=1 t=αm In many MASs, agents tend to perform actions as a reaction to the receipt of ACL messages from other agents in the system. By applying the equations recursively, we can compute the Thus in the same way the impact of a functional call in a total impact Tx of an agent x on N other agents numbered traditional system is the execution time of that function, within 1 ≤ a ≤ N as follows: a MAS, the impact of a message can be related to the additional N X Ma X Ωm processing that must be undertaken in order to react to the Tx = X bt (3) information contained therein, formulate a response or perform a=1 m=1 t=αm a requested task. Because of this mapping, we introduce, as a first simplified approach, the agent message impact measure to Finally, the total impact TS of all the K agents numbered be used as an equivalent to the function processing time used 1 ≤ k ≤ K of a session S is given by the equation: in traditional profiling. K X Nk X Ma X Ωm The quantification of such a measure is a difficult task, X TS = bt (4) given the data typically available from MASs. One potential k=1 a=1 m=1 t=αm measurement for TMα ,B , the impact of a message Mα sent It must be noted that the total activity time AS of the session from an agent A to an agent B and received at time stamp S is given by the equation: α is to use the total amount of computation time used by the agent B until agent B receives a message MΩ from another XK αX k0 −1 agent X at time stamp Ω ≥ α. Let b be the duration of an AS = TS + bt (5) activity by agent B at time stamp t where α ≤ t ≤ Ω. The k=1 t=αS impact of message Mα on agent B, TMα ,B , is then given by where αS is the first recorded time stamp in session S and the recurrent equation: αk0 the time stamp of the very first message received by agent XΩ k. To put it differently, the total impact for each agent can be TMα ,B = bt (1) computed only after it has received its first message. t=α This proposed method of calculating agent message impact In Figure 3 we have tried to summarise this concept in is imperfect, and superior metrics are likely to be developed a graphical form. The diagram clearly shows that the three in the future. However, it does provide useful information for activity stars that lie between α and Ω make up the total impact the debugging and development of MASs. Both the drawbacks 4 and benefits of this approach are outlined in more detail in navigate easily through the tree, AgentSpotter Station offers Section VII. an advanced tree navigation user interface that expands only that part of the tree which is currently explored so as to reduce V. C ALL G RAPH V ISUALISATION S PECIFICATION the visual clutter. The currently explored part of the tree is The conceptual model we have presented deals with the highlighted in a different colour to give the developer some session level, the emitter agent level, the receiver agent visual feedback. level and the message level. The graphical translation of Moreover, to speed up the retrieval of information on the the model, outlined In Figure 4, should be a tree view system, a search feature allows developers to enter a keyword representing the levels we have previously enumerated plus an (e.g. an agent name or a performative). Doing so has the additional level for the FIPA ACL message content. A message effect of highlighting in a special colour all the visible nodes content is defined as a performative plus an expression e.g. that contain the specified keyword, significantly improving the “request:doSomeThing(123)”. This additional level should give visual retrieval speed of a node. developers necessary contextual information for the messages. Finally, developers can zoom and pan around the tree view It is important to note that this fixed-depth call graph tree to locate items even more quickly. represents a divergence from traditional call graphs, whose depth is dictated by the depth of the deepest function call stack. B. Implementation A sample screen shot of the visualisation of the call graph can be seen in Figure 5. In this figure, the element numbered 1 on the screen shot is the tree root, i.e. the session level which represents 100% of the cumulative recorded activity time. The tree root is highlighted in blue because it is the current tree selection in this specific example. As such, it determines the branch that is expanded, Fig. 4. Call Graph Tree View levels as stated in Section V-A, so as to reduce the visual clutter. In order to provide a sufficient level of detail, all the children The session at the root of the tree should add up to 100% and grandchildren of a selected node are visible. Consequently, of all emitter agents’ impact as defined by Equation 4. Then at when the tree root is selected, only the first two subsequent each level, each node should recursively total the impact of its levels are expanded, that is the emitter agent level and the child nodes down to the message leaf nodes. These leaf nodes receiver agent level. Hence, selecting an emitter agent node simply report their impact as defined by Equation 1. More should make the FIPA ACL message level visible, and so on. precisely, at each level, for each node, the following values As an illustration, the call graph numbered 3 shown in Figure 5 should be displayed: screen shot, has an agent receiver node selected; as a result, • Label: informative text associated with the node. The this branch is fully expanded down to the message impact structure of the label depends on the level as follows: level. – session: “capture date, time - duration”; The element numbered 2 is a text area used to enter a – emitter agent: “from: agent id”; search keyword. The number of nodes matching the keyword – sender agent: “to: agent id”; is displayed and all the matching nodes that are visible are – FIPA ACL: “performative: contents”; highlighted in pink. For instance, the element numbered 3 is – message: “sent: time stamp rec: time stamp”. one of the nine nodes containing the “explorer” keyword and so • Total impact time: sum of impact times of all the current is highlighted in the screenshot. In a large expanded tree, this node’s children. highlighting greatly adds to the visual effect and consequently • % parent time: percentage of the current node total to the navigability of the tree. The bottom-most highlighted impact time divided by the node’s parent total impact node in the tree represents a message sent from the “botagent3” time. agent to the agent named “explorer”. Clicking on this node • % session time: percentage of the current node total would cause the subtree rooted at that node to be expanded so impact time divided by the session total impact time. as to examine the content and timing of that message. The visualisation is completely interactive and can be con- Ideally, developers should be able to order the intermediary trolled using the mouse or the keyboard. Possible interactions tree levels differently so as to produce different call graph include panning, scrolling, expand tree branches, zooming in interpretations. For example, moving the FIPA level right above and out. the emitter agent level would list for each FIPA ACL entry One other important feature is the ability to alter the their total impact for all the emitter/receiver pairs. hierarchy of the nodes. Whereas the recursive nature of function calls means that these are inherently inflexible in the tree A. User Interface hierarchy they create, the nature of message-passing is a Despite having a fixed depth, a call graph tree view could different situation. The hierarchy above places the sender of potentially be very wide at the leaf level for sessions that each message in a higher position in the hierarchy than the produce thousands of messages. Therefore, to help developers recipient. This means that the cumulative performance data 5 Fig. 5. Call Graph Tree View Screen Shot for the higher-level nodes represents the overall impact of all will delegate the assignment of tasks to a worker agent, in messages sent by a particular agent to other agents. However, which case the worker agent becomes an overseer agent for this may not encapsulate the information that a developer a brief period. A flat profile and space-time diagram for this requires at a particular point in time. Changing the hierarchy benchmark system is contained in [2]. Figure 6 shows a portion to place the recipient agent above the sender changes the of the call graph tree view for a run of this application. Here, focus of the cumulative performance data. In this case, the the names of overseer agents begin with “master”, whereas the figures represent the contribution to overall running time of a names of worker agents begin with “agent”. particular agent, based on the message that it receives from any The benchmark application profile (displayed in Figure 6 and all sources. This may potentially identify entire individual reveals that overseer agents master1 and master2 do not have agents as bottlenecks. This may be because the system’s load the same impact on performance. Intuitively, one would expect is imbalanced, meaning that one agent may bear an inequitable each overseer agent to have an equal impact. However, in share of the processing burden. Alternatively, in a distributed reality, we can see that the impact of messages sent by the MAS, an agent may simply reside on a machine with inferior “master2” agent accounts for only 20.4% of the overall session hardware resources. By exploiting the flexible nature of this running time. Studying the call graph in more details helps hierarchy, users of the call graph tree view can alter the data in explaining this imbalance, by studying the effects of the being presented to better fit their needs. messages with the content “pleaseDoThing(20)” that were sent by both master1 and master2 to agent001. These are emphasised VI. E VALUATION in Figure 6 by means of the red rectangles. In each case, the parameter passed as part of the a “pleaseDoThing” request is To demonstrate the effectiveness of the call graph as a related to the amount of work that the agent is being requested profiling tool, a simple benchmark application was developed. to perform. This consists of two types of agents. Overseer agents request worker agents to perform small, medium or large tasks. If The call graph shows that some requests from master2 have a worker agent has recently been overloaded, it may refuse a 0.0 impact which in practice means they were ignored (no to execute the required task. Occasionally, overseer agents actions took place as a result of receiving those messages). In 6 Fig. 6. Benchmark Call Graph Tree View for master1 and master2 other words, when master1 sends a request to an agent, and It may be possible to make such a deduction from viewing immediately afterwards that master2 sends the same request to the underlying agent code itself, however the use of the call the agent, the overloaded agent simply refuses to execute the graph makes this far more easily apparent without the need for request. These “pleaseDoThing(20)” messages sent by master1 detailed examination of the code. This also means that testers are reasonably consistent in terms of their impact, are never that are not necessarily familiar with the code (or even perhaps refused and account for a total of 8.8% of the total session testers who do not understand the programming language running time. In contrast, only a single such request sent by used) can identify bottlenecks and behavioural anomalies for master2 was honoured by agent001. This action accounted for developers to address. a mere 0.5% of the session running time. It is important to note that the greater impact of master1’s VII. D ISCUSSION messages does not necessarily constitute a bottleneck, merely an imbalance in the system. This type of analysis would motivate The proposed metric for measuring the agent message impact the use of the space-time diagram to examine the timing of the outlined in Section IV-B has a number of drawbacks. It operates messages in question, so as to further find why messages from on the naive assumption that the actions of an agent are directly master2 are more likely to be ignored by the worker agent. related to the messages received by it. The impact of a message A bottleneck would be identified by comparing the impact on an agent is thus taken as the sum of the execution times of of different messages being sent (rather than the same message all actions undertaken by the agent between the receipt of that being sent by different agents). For instance, it is notable message and the receipt of the next message. that the session impact percentages for “pleaseDoThing(1)” The principal drawback with such an approach is that there messages sent by master1 to agent001 are far lower than for is no provable causal link between the receipt of messages “pleaseDoThing(20)”. In this simple benchmark application, and the execution of actions. Agents may decide to act for this is an unsurprising result, as the increased workload is reasons other than the receipt of ACL messages. For instance, explained by the messages themselves, with the latter message a perceptor may have detected changes in the environment that requesting more processing to be undertaken by the former. may require some reaction. Also, when actions are executed as However, figures such as these would indicate a bottleneck if a direct consequence of the receipt of an ACL communication, the results are unexpected (i.e. where high-impact messages there is no guarantee that all of the relevant actions have been are not intended to trigger high-cost actions on the part of the performed prior to the receipt of the next message. Thus the message recipients) and so would motivate a closer examination agent message impact arising from the receipt of a single of the longer-running actions to increase efficiency. message may not be particularly informative. 7 Ideally, the best method of measuring the impact of the For further development of the call graph view of the receipt of an ACL message would be to track the internal AgentSpotter application, there are two principal areas for reasoning process of the agent, so as to identify those actions improvement. Firstly, as we have acknowledged in Section VII, that are performed as a direct result of the receipt of a message the current measure for gauging the processing impact of a and take only these into account when calculating the message message being passed between agents is not an ideal one. We impact. This is, however, a particularly difficult task, as the intend to investigate other possible measures that will include reasoning used by agents is extremely platform-dependent and a stronger causal link between the receipt of a message and the would require a substantial amount of work to be performed resulting processing activity. In doing so, the other AgentSpotter in order to port AgentSpotter to other agent platforms and views (flat profile and space-time diagram) will be utilised to frameworks. This contravenes one of the fundamental aims of ensure that any proposed measures reflect the reality of the AgentSpotter, which is intended to be as platform-agnostic as system’s execution as closely as possible. is practicable. The second significant area of future work is in the area Even if we are to settle for a framework-specific profiling of agent conversation protocols. The work presented in this system, the task of identifying direct causal links between paper considers each ACL message to be entirely independent events is also non-trivial. Whereas some agents may contain of all other messages. The reality of agent communication straightforward agent code that reacts to the receipt of a is somewhat different. In the agent architecture presented message by always invoking a particular action, this is unlikely as the benchmark application in Section VI, some overseer to always be the case. Receipt of a message may alternatively agents request that worker agents perform certain tasks. In our lead to a refinement of an agent’s goals, or even more subtly, simple application, this is done by means of a single message an alteration of its current belief state, which in turn may result containing the work request being sent to the worker. In reality, in goal refinements. Goals may be adopted based on the entire a more complex conversation would be used. The initial request belief set, making it difficult to ascertain for certain whether for a task to be performed may be answered with an acceptance the belief triggered by the message was a cause of the change or rejection of the task being assigned, followed perhaps by in the agent’s goals or a merely coincidental occurrence. Even the communication of the result of the task. Clearly, an agent when goals have been adopted, a plan selection algorithm is accepting and performing a task will consume more processing typically used to decide upon the best path to take towards resources than when it is rejected. However, in the existing satisfying those goals. Again, this is a potentially difficult model, both scenarios will be grouped together, under the process to trace reliably. initial message requesting action. Such behaviour may mask Although the proposed measure does have drawbacks and inefficiencies in the processing code by including the low-cost is somewhat simplistic, it is also important to highlight the rejection actions in its session percentages. By introducing an benefits of such a measure. Whereas a high impact measurement additional conversation level into the tree, these situations can for a single message may not be indicative of a major be separated, meaning that actions will be grouped according system bottleneck (and may indeed be merely coincidental), to entire agent transactions rather than single messages. consistently high impact measures for similar types of messages are far more likely to be a result of a causal link between the R EFERENCES receipt of the message and the processing that follows. It is this type of analysis that makes the call graph a useful tool in [1] R. Bordini, L. Braubach, M. Dastani, A. Seghrouchni, J. Gomez-Sanz, J. Leite, G. O’Hare, A. Pokahr, and A. Ricci, “A survey of programming identifying situations that result in a high processing load and languages and platforms for multi-agent systems,” Informatica, vol. 30, thus aid in helping developers concentrate on the appropriate no. 1, pp. 33–44, 2006. portions of the code base to improve system efficiency. [2] D. Doan Van Bien, D. Lillis, and R. Collier, “Space-time diagram gener- ation for profiling multi agent systems,” in Proceedings of PROMAS’09, Budapest, Hungary, May 11-12th 2009. VIII. C ONCLUSIONS AND F UTURE W ORK [3] D. E. Knuth, “An empirical study of FORTRAN programs,” j-SPE, vol. 1, We have proposed a new visualisation, the call graph no. 2, pp. 105–133, Apr./Jun. 1971. [4] S. L. Graham, P. B. Kessler, and M. K. Mckusick, “Gprof: A call graph tree view, in order to provide detailed information about the execution profiler,” SIGPLAN Not., vol. 17, no. 6, pp. 120–126, 1982. performance impact of agents interactions. After discussing the [5] Sun Microsystems, Inc., “JVM Tool Interface (JVMTI), Version concept of call graph in a traditional programming context, we 1.0,” Web pages at http://java.sun.com/j2se/1.5.0/docs/guide/jvmti/ have then mapped it into an agent-oriented concept based on (accessed August 4th, 2008), 2004. [Online]. Available: http: //java.sun.com/j2se/1.5.0/docs/guide/jvmti/ the idea that when an agent sends an ACL message to another, [6] A. Srivastava and A. Eustace, “Atom: a system for building customized its impact on the amount of processing the recipient perform program analysis tools,” in PLDI ’94: Proceedings of the ACM SIGPLAN can be measured and used to identify system bottlenecks, 1994 conference on Programming language design and implementation. New York, NY, USA: ACM, 1994, pp. 196–205. load imbalances and efficiency issues. Although the proposed [7] A. Helsinger, M. Thome, T. Wright, B. Technol, and M. Cambridge, measure is not optimal, it does provide users with data that is “Cougaar: a scalable, distributed multi-agent architecture,” in Systems, appropriate and useful in the context of a profiler application. Man and Cybernetics, 2004 IEEE International Conference on, vol. 2, 2004. We have then extended this notion to a tree model with multiple [8] G. Rimassa, M. Calisti, and M. E. Kernland, Software Agent-Based levels: session, message emitter, message receiver, message. Applications, Platforms and Development Kits, ser. Whitestein Series in Finally, we have described the advanced user interface that Software Agent Technologies and Autonomic Computing. Birkhäuser Basel, 2005, ch. Living Systems R Technology Suite, pp. 73–93. allows developers and testers to interact with this model in the [9] R. Collier, “Debugging Agents in Agent Factory,” Lecture Notes in form of a zoomable and searchable tree view. Computer Science, vol. 4411, p. 229, 2007. 8 [10] C. Seah, M. Sierhuis, W. Clancey, and M. Cognition, “Multi-agent modeling and simulation approach for design and analysis of MER mission operations,” in Proceedings of 2005 International conference on human-computer interface advances for modeling and simulation (SIMCHI’05), 2005, pp. 73–78. [11] H. Nwana, D. Ndumu, and L. Lee, “ZEUS: An advanced tool-kit for engineering distributed multi-agent systems,” Applied AI, vol. 13, no. 1, p. 2, 1998. [12] F. Bellifemine, G. Caire, D. Greenwood, and E. Corporation, Developing multi-agent systems with JADE. Springer, 2007. [13] J. Botıa, J. Hernansaez, and F. Skarmeta, “Towards an Approach for Debugging MAS Through the Analysis of ACL Messages ,” Computer Systems Science and Engineering, vol. 20, 2005. [14] R. Collier, G. O’Hare, T. Lowen, and C. Rooney, “Beyond Prototyping in the Factory of Agents,” Multi-Agent Systems and Application III: 3rd International Central and Eastern European Conference on Multi- Agent Systems, Ceemas 2003, Prague, Czech Republic, June 16-18, 2003: Proceedings, 2003.