=Paper=
{{Paper
|id=Vol-3065/paper1_171
|storemode=property
|title=Plan Simulation with PDSim
|pdfUrl=https://ceur-ws.org/Vol-3065/paper1_171.pdf
|volume=Vol-3065
|authors=Emanuele De Pellegrin,Ronald P. A. Petrick
|dblpUrl=https://dblp.org/rec/conf/aiia/PellegrinP21
}}
==Plan Simulation with PDSim==
https://ceur-ws.org/Vol-3065/paper1_171.pdf
Plan Simulation with PDSim
Emanuele De Pellegrin1 , Ronald P. A. Petrick1
1
Edinburgh Centre for Robotics, Heriot-Watt University, Edinburgh, Scotland, United Kingdom
Abstract
This paper reports on the Planning Domain Simulation (PDSim) project, an asset for the Unity game
engine to simulate plans in a 2D or 3D environment with custom animations and graphics effects. The
project aims to fill a gap in the area of planning simulation and validation by tackling the problem of
the scarcity of systems and tools to help the user quickly evaluate the validity of the planning model.
Simulating a planning problem using 3D graphics and animation techniques can help the user to quickly
evaluate the quality of a plan and improve the design of the planning domain and problem. This paper
presents an overview of PDSim, including its aims as a system for automated planning, the current state
of development, and future plans for the project.
Keywords
Automated Planning, Unity Game Engine, Simulation
1. Introduction
Verifying plan solutions and debugging planning domain models can be quite challenging,
especially for real-world planning problems that involve a large number of actions and objects.
While languages like PDDL [1] provide a standard way of representing planning models sup-
ported by a range of planners, catching modelling errors (i.e., incorrect logic in action effects and
preconditions, missed predicates in an init block, etc.) can still be difficult due to the complexity
of the knowledge that needs to be specified and the level of abstraction that is often required
for ensuring the generation of tractable solutions. Although several tools do exist to aid in
the validation of planning domain models (e.g., VAL [2]), and formal plan verification methods
are a growing area of research [3, 4, 5], approaches based on visualisation methods and visual
feedback can also play an important role in addressing the problem of correctly modelling
planning domains. Visual tools can also serve as an environment for displaying, inspecting, and
simulating the planning process, which can aid in plan explainability for human users [6].
PDSim (Planning Domain Simulation) [7] introduced a system to visualise and simulate plans
for classical planning problems defined in PDDL. While visualisation of planning domains and
plan solutions is not a new idea [8, 9, 10, 11, 12], PDSim approaches the problem by building a
graphical environment for plan visualisation and simulation within the Unity game engine [13].
PDDL is used to define the structure of the domain knowledge and the problem formulation
(e.g., planner requirements, language models used in the domain such as types and objects, plus
IPS 2021: 9TH ITALIAN WORKSHOP ON PLANNING AND SCHEDULING, November 29–30, 2021
" ed50@hw.ac.uk (E. D. Pellegrin); R.Petrick@hw.ac.uk (R. P. A. Petrick)
~ https://cryoscopic-e.github.io/ (E. D. Pellegrin); http://petrick.uk/ (R. P. A. Petrick)
0000-0002-5979-6547 (E. D. Pellegrin); 0000-0002-3386-9568 (R. P. A. Petrick)
© 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
standard definitions of the domain and problem) in a 3D visual setting. These components are
used by a planner to check that a solution exists and to generate a list of actions (a plan) satisfying
the goal. Using the plan, PDSim interprets the action effects as 3D animations and graphics
effects that can be used to assess the validity of the plan and to deliver a visual explanation of the
world and its actions during plan execution. The main “actors” involved in PDSim simulations
are therefore the properties responsible for defining the initial state and the action effects.
In this paper, we describe the aims of PDSim, the structure and core components that are
responsible for providing virtual simulations, and illustrate how PDSim can be used to simulate
classical planning problems. PDSim is built by extending the Unity game engine editor [13] and
uses the components offered by the engine such as a path planner, lighting system, and scene
management, among others. The system uses a back-end server that is responsible for parsing
PDDL files and managing plan generation. The parser supports a wide range of PDDL language
features. Thanks to the modular nature of the server side, the system is also extensible: one
planned extension is support for the Robot Operating Sytem (ROS) [14], enabling the use of
PDSim as a visualisation tool for robotics and related applications.
The rest of the paper is organised as follows. First, we review work related to planning
problem visualisation. We then describe the main components of PDSim and provide examples
of their use in practice. Finally, we conclude with future work and planned additions to PDSim.
2. Related Work
PDSim [7] is part of the small ecosystem of simulators for automated planning which use
visual cues and animations to translate the output of a plan into a 3D environment. The closest
approach to ours is Planimation [10] which uses the Unity game engine as the front end to
display objects and animate their position while following a given plan. Planimation defines
animations using an ad-hoc language (namely an animation profile) similar to PDDL. This differ
from PDSim, where animations are defined using a custom visual scripting system.
The Logic Planning Simulator (LPS) [11] also provides a planning simulation system that
represents PDDL objects with 3D models in a user-customizable environment. The approach
is integrated with a SAT-based planner and a user interface that enables plan execution to be
simulated while visualising updates to the world state and individual PDDL properties in the
3D environment. LPS is not based on Unity but provides the user with a simple interface for
plan visualisation. Several user-specified files are also required to define 3D object meshes, the
relationship between PDDL elements and 3D objects, and the specific animation effects.
Several systems also exist to help users formalize planning domains and problems through
user-friendly interfaces. For instance, systems like GIPO [15], ItSimple [9] and VIZ [16] use
graphical illustrations of the domain and problem elements, removing the requirement of PDDL
language knowledge, to help new users approach planning domain modelling for the first
time. Other software such as Web Planner [17] and Planning.Domains [18] use Gantt charts or
tree-like visualisation methods to illustrate the generated plan and the state space searched by a
particular planning algorithm. PlanCurves [12] uses a novel interface based on time curves [19]
to display timeline-based multiagent temporal plans distorted to illustrate similarity between
states. All of these tools attempt to assist users in understanding how a plan is generated and to
Figure 1: PDSim system design
help detect potential errors in the modelling process.
Simulators are also prevalent in robotics applications, and multiple systems exist that make
use of game engines to provide a virtual environment, such as MORSE [20] or Drone Sim
Lab [21]. Game engines also offer several benefits such as multiple rendering cameras, physics
engines, realistic post-processing effects, and audio engines, with no need to implement these
features from scratch [21], making them desirable tools for simulation. For instance, Unity
is being used as a tool for data visualization, architectural prototypes, robotics simulation1 ,
computer vision2 and machine learning applications [22, 23]. Interesting use cases of Unity
related to AI and planning include the Unity AI Planner3 , an integrated planner being created
by Unity as a component for developing AI solutions for videogames, and Unity’s machine
learning agents4 , a solution for training and displaying agents whose behaviours are driven by
an external python machine learning component.
3. System Implementation
We begin by presenting an overview of PDSim, its structure, and how it is integrated with Unity.
Figure 1 presents the overall system design of PDSim. In the following sections we consider
each component in more detail and how it is implemented in the system. In general, PDSim can
be imported into the Unity engine as a common asset. The simulation is initialized and handled
by a python back-end server which is responsible for parsing and building a representation of
the planning model, making it easier for Unity to understand.
Table 1 (left) shows the PDDL components used to simulate a planning problem and (right)
1
Unity Robotics: https://unity.com/solutions/automotive-transportation-manufacturing/robotics
2
Unity for CV: https://unity.com/products/computer-vision
3
Unity AI Planner: https://docs.Unity3d.com/Packages/com.Unity.ai.planner@0.0/manual/index.html
4
Machine Learning Agents: https://github.com/Unity-Technologies/ml-agents
Table 1
PDDL to Unity conversion table
PDDL Element Unity Representation
Animation graphs define object behaviour in Unity
predicate
(translation, path planning, audio emission, particle effects, etc.)
action Action effects are the animated components, using the object attributes
objects General 2D/3D model representation with custom sprites/meshes
init Pre-simulation using the predicate’s defined animations
{'objects':
[{'name': 'apn1', 'type': 'airplane'},
{'name': 'apt1', 'type': 'airport'},
...],
'predicates':
[{'name': 'in-city', 'attributes': ['place', 'city']},
{'name': 'at', 'attributes': ['physobj', 'place']},
{'name': 'in', 'attributes': ['package', 'vehicle']}],
'init':
[{'predicate': 'at', 'attributes': ['obj13', 'pos1']},
{'predicate': 'at', 'attributes': ['obj23', 'pos2']},
..]'
..
}
Figure 2: Parsed PDDL to JSON example
how Unity uses those components to represent PDDL in a graphical environment. PDDL files
are translated into a JSON map of the components needed for simulation. For example, Figure 2
shows the JSON code for the logistics domain. In Unity, the user can set the 2D or 3D models
for the constants defined in the planning problem, create animations for specific predicates and
use Unity’s internal components such as the physics engine, the planning system, etc.
3.1. Back-end
The back-end of PDSim is a python server that communicates with the Unity editor using the
ZeroMQ netwoking library 5 , in particular the python implementation package pyzmq 6 on
the server side and the C# implementation netMQ 7 on the Unity side. The Figure 3 shows the
workflow executed by the system when the user wants to create a new simulation. The user
interact with PDSim that use the Unity editor to create a form where the domain and problem
5
https://zeromq.org/
6
https://pypi.org/project/pyzmq/
7
https://github.com/zeromq/netmq/
Figure 3: Creating new simulation diagram.
Figure 4: Requesting a new plan diagram.
files are expected. Unity tries to connect, using a separate thread, with the back end server by
submitting a request using the PDDL domain and problem files. The request is sent with the
"init" header to tell the server that the request is to parse the PDDL and to create a representation
on the server of the planning problem to be used for later requests. The server handle the PDDL
parsing using the well know Tarski parser 8 . The PDDL elements are converted to a JSON
representation and send back to Unity that will create the objects and animations customizable
by the user. As shown in Figure 4, the python server can accept the "plan" request used to
generate a plan with the current domain and problem previously initialised. The PlanRequester
interface can use a local planner such as FastDownward [24] if present, or the planner web
service offered by Planning.Domains [18]. The response sent to Unity use the JSON format as
before, representing the actions to animate in PDSim.
8
https://github.com/aig-upf/tarski
Figure 5: Simulation object example with a truck type from the logistic domain.
3.2. Unity
Unity [13] is an well know state of the art game engine. Unity is used as front end of PDSim, and
it’s responsible to handle all the 2D/3D graphics and animations related to the simulation. One
of the fundamental OOP design used by Unity is the composition, which means that an object
can be composed of different types of objects. Thanks to the components system, every object in
an Unity scene can have assigned custom scripts or module, such as a rigid body for the physics
simulation, a collision volume, an audio source, etc. Every object in Unity can be scripted using
the C# language, meaning that an object can have an user-defined behaviour in the scene. For
example, an object can respond to user inputs such as mouse and keyboards, translate, rotate
and scale, change color, based on conditional events. Being able to script every object in Unity
is of an utmost importance for the modularity aspect of a simulator in particular for the custom
representation of the PDDL elements. Scripting can also be applied to the editor window itself.
The editor window is where the user interacts with the engine, where it’s possible setting game
objects properties in the scene by using the Unity’s UI.
3.2.1. Simulation Objects
A PDDL type in PDSim is represented by a SimulationObject, a prototype that share the same
structure for all the objects defined in the problem. A simulation object is defined by 2 main
components: Models and Control points. Models are used to visually represent in the virtual
word the object type (i.e. block, airport, player, robot, etc). These can be 3D meshes or 2D
textured sprites that can be imported in the Unity editor. It’s possible to add as many models the
user wants, a collision box that wraps all the models is automatically calculated to be used later
in the simulation to detect the interaction with the user inputs and the collisions calculated by
the physic engine. Control points are 3D vectors that represent particular points of interest in
the object type representation (i.e. the cardinal points of an object, a point that represent the
position of the arm of an agent, etc.). Figure 5 shows an example of how a simulation object
can be composed. The Models is composed only by one mesh representing a delivery truck, and
Figure 6: Animation definition example.
Table 2
PDSim animation nodes
Animation Behaviour
MoveObject Animation for moving a particular Object in the scene to a specific Point in the world.
Animation for moving an object using Unity’s path planning system
ObjectPathTo
(Requires setup).
Spawn Instantiate an Object in the scene.
PlayAudio Play an audio clip.
Utility Node Behaviour
The output is a vector sum of the 2 input Vectors
AddPositions If connected to a node, the inputs can be current objects positions
(Used to express custom paths)
Node mainly used to get the components of a simulation object previously defined.
GetSimObject
Used during run-time to get the current object components involved in an action.
the control point is the 3D vector position of the cargo represented by the blue dot in the scene.
The collision box calculated is represented by the green line that contains all the models.
3.2.2. Animations
One of the most important aspects of PDSim is the visual scripting animation system. As shown
in Figure 6, the user can create his own particular behaviour in the virtual scene for every
predicate he wants to animate. The example shows a stacking of 2 translation animations where
the same object is translated along the x axis first and the y axis subsequently. Every animation
has a duration, where the value can range from 0 to n seconds and can reference the previous
and the next animation. Table 2 shows the animation nodes currently implemented in PDSim.
In Unity these animations use Coroutines that allow the user to write functions that can run
concurrently in the main Unity thread and be suspended or resumed either by user choice or if
a condition is met.
Figure 7: Simulation Manager diagram.
{'status' : 'OK',
'plan':[
{
'action': 'pick-up',
'attributes' : ['b']
},
{
'action': 'stack',
'attributes' : ['b', 'a']
},
{
'action': 'pick-up',
'attributes' : ['c']
},
...]}
Figure 8: Parsed plan in JSON format.
3.2.3. Simulation Manager
Figure 7 show the overall diagram of the simulation manager, a component that handles the
simulations on the front-end side. Its main responsibilities includes:
• Holding references to the several instances to simulate
• Holding reference of all simulation objects in the scene
• Holding reference of all animation graph defined by the user
• Keep track of the existing types defined in the domain file
• Send request to the back-end server to generate a new plan if the instance changes and
no plan exist.
The simulation manager build simulation objects blueprints for all the leaf type of the type tree
that is build when the domain is parsed for the first time. These types prefabs are replicated for
(pick-up b)
(stack b a)
(pick-up c)
(stack c b)
(pick-up d)
(stack d c)
Figure 9: Problem representation for Blocks World
each object in the problem files that match the particular type, using the user configuration
explained in Section 3.2.1. The simulation manager communicates with the back-end server to
request a plan using the current instance being simulated and the domain corpus active on the
server side. The request is a simple message containing the problem file with a plan header. The
server will respond with a JSON representation of the plan as shows in Figure 8, using the plan
for the blocks word domain problem. Every action will have an associated list of animation
graphs representing the effect of a PDDL action. The simulation manager will stack and execute
those animations using the attributes in the plan representing the simulation objects involved
in the simulation of that action.
4. Tested Domains
The main functionality of the system has been tested using particular PDDL benchmark domains,
released for the IPC and available in a public repository9 . The domains selected are Blocks
World, Logistics and Sokoban. These particular domains were selected for testing the system by
increasing the complexity of the domain, starting with blocks world as the simplest to Sokoban
the hardest. To test Unity’s internal components and how they integrate into PDSim, a custom
domain called Robots and Boxes have been used also to demonstrate how the simulator can be
used with user-defined domains.
4.1. Blocks World
Blocks World (IPC 2000) is one of the most famous domains. In this domain, blocks can be
stacked one on top of the other. The agent hand can only pick, move and drop one block at a
time. The goal is achieved when the specified stack sequence is reproduced. Figure 9 shows an
example plan for blocks world and the respective simulation in PDSim.
(load-truck o23 t2 p2)
(load-truck o1 t2 p2)
(drive t2 p2 ap2 c2)
...
(fly ar1 ap2 ap1)
...
Figure 10: Problem representation for Logistics
(move p p5-5 p5-4 up)
(move p p5-4 p5-3 up)
...
(push-nongoal p [..])
...
(push-goal p [..])
Figure 11: Problem representation for Sokoban
4.2. Logistic
The Logistics (IPC 2000) domain describes a problem involving packages that need to be
transported between cities using a plane and within cities using trucks. Constraints of this
domain are that each city has one truck and one airport. This domain steps up the complexity of
the simulation environment while keeping simple definitions of predicates and actions. InCity,
At, In are predicates used to respectively describe if a location is inside a particular city, if an
object is in a particular location, and if a package is in a vehicle. Figure 10 shows the plan
simulation of the logistic domain, highlighting the translation of boxes between cities using the
vehicles.
4.3. Sokoban
The Sokoban (IPC 2008) domain describes the Sokoban game problem10 . The player needs
to perform optimal moves to move an object over a prefixed goal on a grid map. Figure 11
illustrates a typical problem level for the Sokoban game where P is the player, S the stone that
needs to be moved, G the goal. This domain adds up the complexity from the previous and it
was selected to assess the functionality and ubiquity of this simulation program as an additional
instrument for the Unity game engine as a tool to rapidly have an AI agent in-game.
(move_robot r2 a f)
(pick_up_box r2 box1 f)
(move_robot r2 f a)
(drop_box r2 box1 a)
(move_robot r1 d c)
...
Figure 12: Problem representation for Robots and Boxes
Figure 13: Problem representation for Robots and Boxes
4.4. Robots and Boxes
Finally, the Robots and Boxes domain describes a simple domain where multiple robots can be
used to move boxes around a map with rooms and connections between them. The robots can
move from a room to another (if those are connected) pick-up or drop boxes. This domain was
selected to show that PDSim can be used with custom domain definitions and to demonstrate
that can work with Unity’s internal components, in particular, the path planning system that
it’s used to find a smooth path for the robot to follow when moving between rooms. Figure 12
show the simulation for this domain from the point of view of the moving robots.
4.5. Analysis
The planning problems used as a testing ground for PDSim produced a good quality of the
simulation in terms of visual cues to evaluate the correctness of a plan. In particular, using the
blocks world domain as an example, if we introduce a logic error in the stack action effects plan
is generated but it’s not valid. The error introduced wad about missing to mark as not clear
the block below the held block. Figure 13 shows the invalid plan where a block (D) is stuck in
mid-air which might indicate an error in the domain modelling.
5. Conclusion
This paper presented the structure and operation of PDSim, a simulation system for PDDL
that can be used to animate classical plans. This project supports classical automated planning,
however, current work is extending PDSim to support temporal planning through an intuitive
visualization system for timed actions and deadlines. Future work on the project will consider
support for partial plans defined by the user and other simulation features such as following
simulation actions, replaying previous actions, modifying the simulation speed, and displaying
partial animations to show the outcome of animation while defining the structure. An important
direction for PDSim will also be to include extensions for visualising the current state of an
agent’s knowledge and beliefs to support epistemic planning.
References
[1] D. McDermott, M. Ghallab, A. Howe, C. Knoblock, A. Ram, M. Veloso, D. Weld, D. Wilkins,
PDDL—the planning domain definition language, Technical Report CVC TR-98-003/DCS
TR-1165, Yale Center for Computational Vision and Control, 1998.
[2] R. Howey, D. Long, VAL’s progress: The automatic validation tool for PDDL2.1 used in
the international planning competition, in: Proceedings of the ICAPS Workshop on The
Competition: Impact, Organization, Evaluation, Benchmarks, 2003.
[3] S. Bensalem, K. Havelund, A. Orlandini, Verification and validation meet planning and
scheduling, International Journal on Software Tools for Technology Transfer 16 (2014)
1–12.
[4] A. Cimatti, A. Micheli, M. Roveri, Validating domains and plans for temporal planning
via encoding into infinite-state linear temporal logic, in: Proceedings of AAAI, 2017, pp.
3547–3554.
[5] A. Hill, E. Komendantskaya, R. P. A. Petrick, Proof-carrying plans: A resource logic
for ai planning, in: International Symposium on Principles and Practice of Declarative
Programming (PPDP), 2020, pp. 1–13.
[6] M. Fox, D. Long, D. Magazzeni, Explainable planning, in: Proceedings of the IJCAI
Workshop on Explainable AI, 2017.
9
https://github.com/potassco/pddl-instances
10
Sokoban Game: https://en.wikipedia.org/wiki/Sokoban
[7] E. De Pellegrin, Pdsim: Planning domain simulation with the unity game engine, in: Pro-
ceedings of the ICAPS Workshop on Knowledge Engineering for Planning and Scheduling
(KEPS), 2020.
[8] D. Vrakas, I. Vlahavas, A visualization environment for planning, International Journal of
Artificial Intelligence Tools 14 (2005) 975–998.
[9] T. S. Vaquero, V. Romero, F. Tonidandel, J. R. Silva, itsimple2.0: An integrated tool for
designing planning domains, in: Proceedings of ICAPS, 2007, pp. 336–343.
[10] G. Chen, Y. Ding, H. Edwards, C. H. Chau, S. Hou, G. Johnson, M. Sharukh Syed, H. Tang,
Y. Wu, Y. Yan, T. Gil, L. Nir, Planimation, 2020. URL: https://doi.org/10.5281/zenodo.3773027.
doi:10.5281/zenodo.3773027.
[11] C. Tapia, P. San Segundo, J. Artieda, A PDDL-based simulation system, in: Proceedings of
the IADIS International Conference Intelligent Systems and Agents, 2015.
[12] P. Le Bras, Y. Carreno, A. Lindsay, R. P. A. Petrick, M. J. Chantler, Plancurves: An interface
for end-users to visualise multi-agent temporal plans, in: Proceedings of the ICAPS
Workshop on Knowledge Engineering for Planning and Scheduling (KEPS), 2020.
[13] Unity Technologies, Unity, 2021. URL: https://unity.com.
[14] M. Quigley, K. Conley, B. P. Gerkey, J. Faust, ROS: an open-source robot operating system,
in: Proceedings of the ICRA Workshop on Open Source Software, 2009.
[15] R. M. Simpson, D. E. Kitchin, T. L. McCluskey, Planning domain definition using GIPO,
The Knowledge Engineering Review 22 (2007) 117–134.
[16] J. Vodrázka, L. Chrpa, Visual design of planning domains, in: Proceedings of ICAPS
Workshop on Knowledge Engineering for Planning and Scheduling (KEPS), 2010, pp.
68–69.
[17] M. C. Magnaguagno, R. Fraga Pereira, M. D. Móre, F. R. Meneguzzi, Web planner: A tool to
develop classical planning domains and visualize heuristic state-space search, in: ICAPS
Workshop on User Interfaces and Scheduling and Planning (UISP), 2017.
[18] C. Muise, Planning.domains, ICAPS System Demonstration, 2016.
[19] B. Bach, C. Shi, N. Heulot, T. Madhyastha, T. Grabowski, P. Dragicevic, Time curves:
Folding time to visualize patterns of temporal evolution in data, IEEE transactions on
visualization and computer graphics 22 (2015) 559–568.
[20] G. Echeverria, N. Lassabe, A. Degroote, S. Lemaignan, Modular open robots simulation
engine: Morse, in: 2011 IEEE International Conference on Robotics and Automation, IEEE,
2011, pp. 46–51.
[21] O. Ganoni, R. Mukundan, A framework for visually realistic multi-robot simulation in
natural environment, arXiv preprint arXiv:1708.01938 (2017).
[22] J. K. Haas, A history of the Unity game engine, 2014.
[23] J. Craighead, J. Burke, R. Murphy, Using the unity game engine to develop sarge: a case
study, in: Proceedings of the 2008 Simulation Workshop at the International Conference
on Intelligent Robots and Systems (IROS 2008), 2008.
[24] M. Helmert, The fast downward planning system, Journal of Artificial Intelligence Research
26 (2006) 191–246.