1 Online Search in Behavioral Programming Models Orel Moshe Weinstock Department of Computer Science, Ben-Gurion University of the Negev Abstract—We present a model based approach to Search Based 3) If unsatisfied with the execution’s choices, extend the Software Engineering (SBSE). The approach is based on the Be- model by formalizing more refined requirements. havioral Programming (BP) paradigm where independent aspects 4) Repeat steps 2 and 3 until the behavior is satisfactory. of behavior are woven at run time using a simple interaction protocol. We propose to extend the behavioral programming The behavioral programming (BP) paradigm that we focus execution mechanism with on-line heuristic search in program on in this paper is described in detail in Section II. BP state space that allows programmers to develop non-deterministic extends and generalizes scenario-based programming which programs while relying on a “smart” event selection mechanism was introduced with the language of live sequence charts to resolve non-determinism in a way that maximizes a specified heuristic function. The paper presents a new library that we have (LSC) [7], [8]. In addition to the refinement idioms that already developed in Java and in JavaScript, using Rhino, to facilitate exist in BP, which allow programmers to incrementally shape the proposed modeling approach and programming style. We give their software by adding modules that can both widen and examples, in the context of a StarCraft game bot built with the narrow the set of possible behaviors of the system [9], we library, that demonstrate how the proposed programming idioms propose in this paper to allow BP based models to also contain can simplify the code and help build robust reactive systems. specification of fitness criteria for the heuristic search function that can also be refined along the above development process. I. M OTIVATION AND BACKGROUND The idea of “smart” execution of scenario based specifi- Search Based Software Engineering (SBSE) is an emerging cations started in [10] and in [11] with proposals to apply, field of research which aims to cope with the increased de- respectively, model-checking and planning algorithms for run- mand for functionality, scalability, and robustness of computer ning a single super-step (the part of the run that spans between programs (and of reactive robotic systems in particular) using two consecutive external events) in LSC. We apply a similar heuristic search mechanisms [1]. SBSE consists of automatic mechanism in the context of a behavioral programming library resolution (using search algorithms) of complex decisions that embedded in an imperative programming language. Beyond programmers model as optimization problems. There are many running in a different setting, the main addition of our library, published papers in this area that describe various approaches when compared to these earlier contributions, is that it runs within the software engineering research community [2], [3]. the “smart” event selection mechanism at run-time, on real There are also reports that describe how SBSE has been program code rather than on a model or specification, and successfully applied to solve problems in nearly all software that it can consider a horizon beyond a single super-step. development life cycle phases [4], [5]. The main challenge in SBSE is, of course, finding a good modeling technique that II. B EHAVIORAL P ROGRAMMING P RINCIPLES facilitates the search [6]. As presented in [12], a behavioral model consists of a set Despite the research activity in the area, search methods are of independent behavior threads (b-threads for short). Each practically used only in specific domains. Harman [2] reports, b-thread is a specification of a reactive machine that can be for example, that 54% of SBSE tools are used for testing modeled, e.g., as a procedure in an imperative programming purposes, an additional 11% for maintenance, and another language. Together, the b-threads control the behavior of flow 10% for project management. It seems that the main barrier of the application via a synchronization protocol, as follows. that delays further adaptation of the technique is shortage in When a b-thread reaches a point that requires synchronization, models for online search [6]. it waits until all other b-threads reach such synchronization The goal of this this paper is to explore how SBSE can points in their own flow. At synchronization points, each b- be made accessible to modelers and programmers of reactive thread specifies three sets of events: (1) requested events - the systems, such as robotic applications and interactive game thread proposes that these events be considered for triggering, bots, as idioms that integrate with standard constructs in com- and asks to be notified when any of them occurs; (2) waited- mon modeling and programming languages. This allows for for events - the thread does not request these events, but only natural, powerful derivation from modeling languages (such asks to be notified when any of them is triggered. The platform as LSC [7], [8]) to code. Specifically, we aim at tools that will not consider these events for triggering, unless given as facilitate the following software development methodology: external input to the system; and (3) blocked events - the thread 1) Code and/or derive from models the high-level specifi- currently forbids triggering of these events. cation of the system’s behavior, using non-determinism As shown in Figure 1, when all b-threads are at a synchro- to specify free choices in execution. nization point, a legal event (an event that is requested by 2) Run the system using an engine that resolves non deter- at least one b-thread and is not blocked by any b-thread) is minism heuristically or synthesize deterministic code. chosen. This chosen event is then triggered by resuming all the 2 b-threads that either requested or waited for it. Each of these •Look-ahead subject to desired properties of the resulting resumed b-threads then proceeds with its execution, all the event sequence, as in smart play-out [10]. way to its next synchronization point, where it again presents • Planning algorithms [11], called planned play-out in LSC. sets of requested, waited-for and blocked events. The other b- • Reinforcement learning where events that have shown to threads remain at their last synchronization points, oblivious produce better expected value are selected [14]. to the triggered event, until an event is selected that they have • Allow concurrent events or split execution into parallel requested or are waiting for. When all b-threads are again at concurrent executions, as in [15]. a synchronization point, the event selection process repeats. • Synthesizing specifications into a deterministic automa- More formally, recall that a deterministic labeled transition ton (e.g. [16]). system is a quadruple hS, E, →, initi, where S is a set of In this research we adopt and extend the mechanism pro- states, E is a set of events, → is a (possibly partial) function posed in [11] as elaborated in Section IV below. from S × E to S, and init ∈ S is the initial state. The runs of such a transition system are sequences of the form Requested Events e1 e2 ei s0 −→ s1 −→ · · · −→ si · · · , where s0 = init, and for all i = 1, 2, · · · , si ∈ S, ei ∈ E, and the function → maps b-thread ei the pair hsi−1 , ei i to si , written as si−1 −→ si . We say that hS, E, →, initi is total if the transition function → is a total b-thread Blocking function. We will now give the semantics of a set of b-threads in terms of runs of a transition system. For this, we model each b-thread behavior thread as a transition system with an association of requested and blocked events to each state: b-thread Definition 1. behavior thread [12]: A behavior thread (abbr. Selected Event b-thread) is a tuple hS, E, →, init, R, Bi, where hS, E, → , initi forms a deterministic total labeled transition system, Figure 1. Collective execution of behavior threads using an enhanced publish/- R : S → 2E is a function that associates each state with the subscribe protocol: (a) all b-threads place their “bids”, specifying requested events and blocked events; (b) a synchronization mechanism chooses an event set of events requested by the b-thread when in that state, and that is requested but is not blocked; (c) b-threads waiting for the event are B : S → 2E is a function that associates each state with the notified; (d) the notified b-threads progress to their next states, where they set of events blocked by the b-thread when in that state. can place new bids. We define a composition operator on the set of b-threads and the resulting set of runs of the composite transition system as III. B EHAVIORAL P ROGRAMMING IN JAVASCRIPT follows: The intent is for programmers to use the principles intro- duced in the previous section in an imperative programming Definition 2. Runs of a set of b-threads [12]: We define the language. To illustrate this coding technique, consider a b- runs of a set of b-threads {hSi , Ei , →i , initi , Ri , Bi i}ni=1 as thread that increases water flow in a hot water tap by request- Sn system hS, E, →, initi, where the runs of the labeled transition ing five times the event AddHot which stands for turning S = S1 × · · · × Sn , E = i=1 Ei , init = hinit1 , . . . , initn i, e the tap anticlockwise for some small fixed amount. Another → hs01 , . . . , s0n i if and → includes a transition hs1 , . . . , sn i − b-thread performs a similar action, with the event AddCold, and only if on the cold water tap. To increase the water flow in both taps n n [ ^ [ in parallel, as may be desired for keeping the temperature e∈ Ri (si ) e∈ / Bi (si ) . (1) stable, one may activate the above b-threads alongside a i=1 i=1 third one, which forces the interleaving of events in the two | {z } | {z } e is requested e is not blocked scenarios. The third b-thread, for example, can be coded and as “repeatedly: block AddCold until AddHot; n block AddHot until AddCold”. This programming e style was proposed in [12] in Java and is extended to Javascript ^ →i s0i ) ∧ (e ∈ / Ei =⇒ si = s0i )  (e ∈ Ei =⇒ si − i=1 | {z } | {z } in our work. A Javascript program for the water tap application affected b-threads move unaffected b-threads do not move (2) is shown in Figure 2. As seen in Figure 2, defining b-threads is easy and concise When multiple events are requested and not blocked, the - you simply register a function with the application object, semantics of event selection may vary. The process of choosing which is embedded in the Javascript interpreter from the un- the next event, the focus of this research, is called Arbitration. derlying Java. Synchronization is induced by calling a method Various arbiters have been suggested in other works: called bsync, passing to it three sets of events: requested, • A naı̈ve arbiter would select a legal event at random, as waited-for, and blocked events (in this order). The return value in the LSC Play Engine [8]. of bsync is the triggered event that resumed the b-thread. All • Choosing a minimal event according to b-thread, event, multi-threading and concurrency issues are handled by the BP- and request order [12], [13]. Javascript engine. 3 bpjs.registerBThread("HotBt", function () { 1 function findAndHarvestMinerals(move) { while(true){ 2 while(true){ bsync(addHot, none, none); 3 var minerals = bsync(new FindMineral(this), none, none } ); }); 4 var othersHarvesting = new OthersHarvesting(this, bpjs.registerBThread("ColdBt", function () { 5 minerals.getLocation()); while(true){ 6 while(notFullyLoaded){ bsync(addCold, none, none); 7 var harvest = } 8 bsync(new Harvest(minerals.getLocation()), }); 9 none, othersHarvesting); bpjs.registerBThread("AlternatorBt", function () { 10 updateLoaded(harvest.getAmount()); while(true){ 11 } bsync(none, addHot, addCold); 12 bsync(new ReturnToBase(this), none, none); bsync(none, addCold, addHot); 13 }} } }); bpjs.registerBThread("Display", function () { Figure 3. A b-thread for a worker assignment to mineral fields. while(true){ FindMineral is an event list of all orders to harvest a visible mineral field, bsync(none, all, none); one for each such field. Harvest is a predefined event class for harvesting bplog("Event: " + lastEvent()); a given mineral field. OthersHarvesting is an event set which includes } all Harvest event by other workers. }); bpjs.start(); var totalMinerals = 0; function h(bpstate) { Figure 2. An example of using BP-Javascript. var le = bpstate.getLastEvent(); if(isMineralsCollectedEvent(le)) totalMinerals = totalMinerals + le.getValue(); The application object is not the only object available to the return totalMinerals; Javascript programmer. Other objects from the Java layer, like } the predefined event set objects none and all, and addCold and addHot can also be embedded in the interpreter and, thus, Figure 4. The heuristic function summing total collected minerals. can be accessed by the Javascript code. The ability to embed Java operations and objects within the interpreter allows us Figure 3 shows an example of using BP-Javascript with to create DSLs for sharing common functionality between b- search for optimal worker assignment to mineral fields in threads and a much easier way to extend the application or the game of StarCraft. The example is accompanied by the even embed it in another BP-Javascript application. screenshot shown in Figure 5. The function described is The b-threads can combine the full power of the Javascript registered as a b-thread for each worker. To see how the language with synchronized behavioral execution and can search mechanism works in this example, let us examine the also dynamically integrate with applications containing non- arbitration process for a behavioral program with these b- behavioral components. BP-Javascript also enjoys the flexibil- threads bsync by bsync, assuming the heuristic function’s ity and conciseness of Javascript without sacrificing any of the value is the total amount of minerals harvested as in Figure 4. semantics of the Java BP back-end. Each time a bsync is executed (lines 3,8,12), BP will apply the heuristic function to the game state. A comparison to [12], which has no backtracking, and Line 3: The worker asks for orders to mine any visible [17], which uses external tooling for backtracking, is in mineral field - it requested such an order for each visible field. order. Our implementation adapts [17] for online search by The worker will then start harvesting the location it was sent using Javascript backtracking over Rhino with a modified BPJ to at line 8. At this point, the BP system has to choose which library without exteral tooling, see V. mineral field harvest order to trigger. The system will search IV. BP-JAVASCRIPT W ITH S EARCH While BP, as presented in the preceding sections, is use- ful in allowing for relatively independent code components, the model is, by definition, a non-deterministic system that leaves a choice when there is more than one event that is requested and not blocked. To get a standard, deterministic, implementation, programmers can add b-threads that refine the specification or, as we propose in this paper, use a “smart” execution mechanism. Specifically, we now demonstrate how an extension of BP-Javascript with an application agnostic search-based event selection mechanism allows for a cleaner and more robust program. The examples from now on are based on a library we have developed to support this ap- proach. We demonstrate how it can be used in the context Figure 5. Zerg hatchery (red square) with Drones (worker units, green of a StartCraft [18] bot. The bot itself is only in an early squares) harvesting minerals (blue crystals). Drones collect and bring minerals development phase, the code here is only an outline, not part to the hatchery, adding to the player’s credit. Optimal resource collection of a full, working implementation. allows the player to pay for combat units, more workers and new buildings. 4 through the different branches of the program space, each program as a whole. Therefore, we can ignore these internal starting with a different harvest order. A sub-optimal choice workings and focus on bsyncs, as only the states at these will get a lower heuristic value than the optimal choice. Line synchronization points define the integrated system behavior. 8: The worker harvests the field he received in the preceding The only programmatic construct that captures program exe- bsync. While this worker is harvesting the mineral field cution in an immutable, re-entrant object, as required by search (receiving Harvest events), because of the blocked event algorithms, is a continuation [22], [23]. Continuations are set given to bsync, no other b-thread (worker) can receive a representations of the program at a given point in execution, Harvest event on the same field (i.e. can harvest it). While which are available to the programmer, rather than hidden by searching through program space at this point, any branch in the runtime environment. They are used here to traverse the which a worker will wait on an already taken mineral field state space by ordinary program execution, as they facilitate will get a lower heuristic value than one in which there are backtracking and resuming of execution from desired points no workers who are waiting for others to finish with the where they were captured. We can now formally define the field. Therefore, the heuristic drives the search away from state space for the search: unnecessary waits by workers. Line 12: The worker asks for Definition 3. A BT-state represents the state of a specific b- orders to head back to base with the collected minerals. thread in a bsync call during a run of the program. It is This example shows how the non-determinism created by composed of the b-thread and its captured continuation. the existence of multiple requested yet unblocked events is resolved by the search engine, using an appropriate heuristic public BEvent bsync(RequestInterface requested, EventSet waited, EventSet blocked) { function. This can be utilized to write concise programs that _request = requested; would have been longer on a regular imperative platform, _wait = waited; _block = blocked; even when employing search techniques [19], [20]. As a ... rough quantitative comparison, assume only 10% of the code Context cx = ContextFactory.getGlobal().enterContext(); _cont = cx.captureContinuation(); managing the build order (resource collection, unit training ... and building construction sequence) in the bot coded in [20] } deals with harvesting minerals. That is approximately 40KB Figure 6. Code snapshot of the implementation of the bsync function in of source code, significantly longer than the code in Figure 3. the underlying Java b-thread object. As shown in Figure 6, a BT-state (_cont) is captured V. U NDER THE H OOD at every call to bsync by the underlying Java b-thread object. This ensures that once all b-threads have reached a Defining the program-state correctly has great influence on synchronization point, their continuation object representing search results. At every synchronization point where the arbiter that state is updated, so that a complete state of the BP system has to choose the next event, we run the program with our can be captured: choices while controlling its inputs and outputs to determine the heuristic value of states reached in the run and choose Definition 4. A BP-state represents the state of the whole the event leading to the highest scored state. The model in program. It is composed of the BT-states of all b-threads in this case is the actual program state - all its variables, stack the program, captured at a bsync. frames, memory space etc. When the BP infrastructure is required to make a choice The technique of running the program in a controlled between multiple events to trigger, it creates a BP-state as environment is called sandboxing, and is often used for a root for the search. Expanding search nodes is done by testing [21]. Although using this method requires writing an triggering legal events and capturing the new BP-states created environment simulator (to generate inputs), in addition to the by the triggering. This is done by executing the code as in [17], desired application itself, this is not a significant penalty to and not offline or by running on an abstract model of the code development as a simulator is a required in order to test the as in [24]. This ensures that there are no discrepancies between application. While running the program in a sandbox with the code itself and the search results. The BT-state and BP- a simulator in its entirety may be difficult for big general state are the abstractions used by the search algorithm directly purpose applications (due to the large codebase), it is viable such that all BP specific code is encapsulated within those for control/reactive systems - the focus of this research. This objects and is completely transparent to the search algorithm. type of systems contains higher level code that concentrates With the state space defined, we can delve into the search on logical flow, which runs faster than general code. Note mechanism itself - how we run behavioral programs in a sand- that the simulation of the environment does not have to be box. The sandbox is composed of the environment simulator complete, a good search mechanism can make much use even (input generator), a search algorithm and a heuristic function. of an abstract description of the environment. An implementation of a look-ahead mechanism, beyond one We will now discuss the implementation of the program- super-step, requires that the system be able to predict the state object. Programs in BP are comprised of b-threads, so actions of the environment to some precision. For this, we their states is an aggregate of the independent state of each b- need to ask programmers to provide the search mechanism thread. Anything happening inside a b-thread between bsync with an abstract model of the environment (which can be calls - that is, between synchronization points - is by definition probabilistic), a simulator, to provide inputs to the behavioral internal to the b-thread, and so can be considered atomic to the program while in the sandbox. 5 Environment real input input output Behavioral Model simulated input Environment Model Figure 7. The architecture of the BP search sandboxing mechanism. We developed a switched system that operates in two modes. (1) In normal mode the application b-threads receive input from the environment. (2) In search mode, the application b-threads get input from a simulator and output is fed only to the simulation b-threads, so that the environment is not actuated. The Figure 8. Hydralisks (green) and Marines (yellow, red) accompanied by a figure shows the state of the switches in search mode. When in normal mode, Medic (light blue) in combat. both switches are in their other mode, i.e., the right switch is closed and the left switch is connecting ‘real input’ to ‘input’. 1 function killClosest(move) { 2 var closestEnemy = getClosestEnemy(); As shown in Figure 7, we have implemented an environment 3 bsync(new AttackCommand(closestEnemy),none,none); 4 while(closestEnemy.isAlive()){ simulator as b-threads in the program itself. In normal oper- 5 bsync(new Shoot(this,Enemy),none,none); ation, a simulator b-thread’s bsync is modified such that its 6 }} requested event sets are added to the waited-for events set, and its requested event set is empty. This ensures the simulation Figure 9. The Javascript function registered as the Marine’s b-thread. The bsync in line 3 will request and trigger the AttackCommand event, which is b-thread is made aware of all events relevant to it, so that it an output event. The bsync in line 5 will request the Shoot event, which is maintains a correct state for the next use in simulation mode. an input event, in order to keep track of the enemy’s health. When the enemy When the BP infrastructure needs to search for an event is killed, we can move to a new target. to trigger, the simulation b-threads are switched to simulation Selecting the right search algorithm for a behavioral pro- mode, in which they request events normally triggered by the gram can have great impact on the results. The algorithms we environment as modeled in their code. No manipulation of have used are depth-limited A* and minimax [26] textbook their event lists is done in simulation mode. This “Eating our implementations [27]. The architecture of our solution is such own dog food” approach greatly contributes to the robustness that it is easy to introduce other search algorithms, as there is of the simulator and program, while also providing a clear and no coupling between the algorithm itself and the BP-Javascript unified interface for programming behavioral programs. engine. The specific choice of best search algorithms for Let us now consider another example, illustrated in Figure 8. specific application domains is beyond the scope of this work. In this example, we demonstrate usage of the environment Writing a heuristic function is straightforward: the BP- simulator to encode our knowledge that the environment- state object passed to the function grants access to the entire driven Marines attack the closest enemy. We therefore code program without compromising speed or space. This includes the Marine simulator function as in Figure 9 and register it the b-thread’s bsync event sets and public access methods. as a simulation b-thread. When in normal operation, BP will The programmer, then, is given full power in the evaluation of receive the Marine’s actions from the environment. When in program state, independent of the search algorithm used. The simulation mode, the yellow Marine will request to attack the programmer can write different heuristic functions that reward top Hydralisk, and the red Marine will request to attack the desired events and b-thread properties. bottom Hydralisk. If we learn more about Marine behavior, we can change the function in Figure 9, or register more b-threads specifying their behavior as simulation b-threads. VI. C ONCLUSION An interface for sending inputs to the behavioral program In this paper we have shown how a BP engine with and for examining its outputs is then required. A convenient an embedded program-state space search mechanism can solution for this is defining the interface to and from the facilitate development in a new, more natural way using a behavioral program to be an event queue (an input queue standard, modern programming language. The BP architecture has been introduced in [25]). This way the environment and was enhanced to be more flexible and real-world ready. The the sandbox both enqueue events for triggering within the game bots we programmed in this manner accomplished their behavioral program in the input queue and read the behav- initial goals by fulfilling their function in the game while ioral program’s output from its output queue. The behavioral being relatively short and easy to code. This will allow us program does not directly perform actions on the environment to further explore and advance the use of BP, in general and - events from the output queue are fed as player actions back in AI domains particularly, using run-time search to provide into the environment by an adapter, thus enabling running in the programmer with more freedom. sandbox without extra code analysis. 6 R EFERENCES [16] D. Harel and I. Segall, “Synthesis from live sequence [1] M. Harman and B. F. Jones, “Search-based software chart specifications”, Journal of Computer System Sci- engineering”, Information and Software Technology, ences, 2011. vol. 43, no. 14, pp. 833–839, 2001. [17] D. Harel, R. Lampert, A. Marron, and G. Weiss, [2] M. Harman, S. A. Mansouri, and Y. Zhang, “Search- “Model-checking behavioral programs”, in Proceedings based software engineering: trends, techniques and ap- of the ninth ACM international conference on Embed- plications”, ACM Computing Surveys (CSUR), vol. 45, ded software, ACM, 2011, pp. 279–288. no. 1, p. 11, 2012. [18] (1998). Starcraft: brood war — wikipedia, the free en- [3] H. Jiang, Z. Ren, X. Li, and X. Lai, “Transformed cyclopedia, [Online]. Available: http://en.wikipedia.org/ search based software engineering: a new paradigm of wiki/StarCraft: Brood War (visited on 07/11/2015). sbse”, in SSBSE, 2015. [19] D. Churchill and M. Buro, “Build order optimization in [4] J. Manuel, C. Trilla, S. Poulding, and C. Runciman, starcraft.”, in AIIDE, 2011. “Weaving parallel threads: searching for useful paral- [20] ——, (2015). UAlbertaBot, Starcraft bot code, [Online]. lelism in functional programs”, in SSBSE, 2015. Available: http://github.com/davechurchill/ualbertabot [5] S. Yoo, “Amortised optimisation of non-functional (visited on 07/11/2015). property in production environment”, in SSBSE, 2015. [21] A. Fox, E. Brewer, et al., “Harvest, yield, and scalable [6] J. Ahluwalia, I. H. Krüger, W. Phillips, and M. tolerant systems”, in Hot Topics in Operating Systems, Meisinger, “Model-based run-time monitoring of end- 1999. Proceedings of the Seventh Workshop on, IEEE, to-end deadlines”, in Proceedings of the 5th ACM 1999, pp. 174–178. international conference on Embedded software, ACM, [22] G. D. Plotkin, “Call-by-name, call-by-value and the λ- 2005, pp. 100–109. calculus”, Theoretical computer science, vol. 1, no. 2, [7] W. Damm and D. Harel, “Lscs: breathing life into pp. 125–159, 1975. message sequence charts”, Formal methods in system [23] J. C. Reynolds, “The discoveries of continuations”, Lisp design, vol. 19, no. 1, pp. 45–80, 2001. and symbolic computation, vol. 6, no. 3-4, pp. 233–247, [8] D. Harel and R. Marelly, Come, let’s play: scenario- 1993. based programming using LSCs and the play-engine. [24] G. Katz, “On module-based abstraction and repair Springer Science & Business Media, 2003, vol. 1. of behavioral programs”, in Logic for Programming, [9] D. Harel, A. Marron, and G. Weiss, “Behavioral pro- Artificial Intelligence, and Reasoning, Springer, 2013, gramming”, Communications of the ACM, vol. 55, no. pp. 518–535. 7, pp. 90–100, 2012. [25] D. Harel, A. Kantor, G. Katz, A. Marron, G. Weiss, [10] D. Harel, H. Kugler, R. Marelly, and A. Pnueli, and G. Wiener, “Towards behavioral programming in “Smart play-out of behavioral requirements”, in FM- distributed architectures”, Science of Computer Pro- CAD, Springer, vol. 2, 2002, pp. 378–398. gramming, vol. 98, pp. 233–267, 2015. [11] D. Harel and I. Segall, “Planned and traversable play- [26] S. Russell and P. Norvig, “AI a Modern Approach”, out: a flexible method for executing scenario-based pro- Learning, vol. 2, no. 3, p. 4, 2005. grams”, in Tools and Algorithms for the Construction [27] (2015). Java implementation of algorithms from Norvig and Analysis of Systems, Springer, 2007, pp. 485–499. and Russell’s ”Artificial Intelligence - A Modern Ap- [12] D. Harel, A. Marron, and G. Weiss, “Programming proach”, [Online]. Available: http://github.com/aima- coordinated behavior in java”, in ECOOP 2010–Object- java/aima-java (visited on 07/11/2015). Oriented Programming, Springer, 2010, pp. 250–274. [13] G. Wiener, G. Weiss, and A. Marron, “Coordinating ACKNOWLEDGEMENTS and visualizing independent behaviors in erlang”, in This work was partially supported by the Lynne and William Proceedings of the 9th ACM SIGPLAN workshop on Frankel Center for Computer Science. Erlang, ACM, 2010, pp. 13–22. [14] N. Eitan and D. Harel, “Adaptive behavioral program- ming”, in Tools with Artificial Intelligence (ICTAI), 2011 23rd IEEE International Conference on, IEEE, 2011, pp. 685–692. [15] H. Kugler, C. Plock, and A. Roberts, “Synthesizing bio- logical theories”, in Computer Aided Verification - 23rd International Conference, CAV 2011, Snowbird, UT, USA, July 14-20, 2011. Proceedings, G. Gopalakrishnan and S. Qadeer, Eds., ser. Lecture Notes in Computer Science, vol. 6806, Springer, 2011, pp. 579–584, ISBN: 978-3-642-22109-5. DOI: 10.1007/978- 3- 642- 22110- 1 46. [Online]. Available: http://dx.doi.org/10.1007/ 978-3-642-22110-1 46.