This paper presents an integrated cognitive robotics system for industrial kitting operations in a modern factory setting. The robot system combines low-level robot control and execution monitoring with automated mission and task planning, and a logistics planner which communicates with the factory's manufacturing execution system. The system has been implemented and tested on a series of automotive kitting problems, where collections of parts are picked from a warehouse and delivered to the production line. The system has been empirically evaluated and the complete framework shown to be successful at assembling kits in a small factory environment.
The kitting paradigm, where collections of parts are gathered and delivered to a production line, is becoming increasingly common in modern factories as it enables customised products to be built one after the other. Currently, kitting is a task that is usually reserved for human workers, leading to longterm injuries from repetitive stress due to the sheer number and weight of parts picked. However, automating this process means addressing a number of challenging problems including autonomous navigation in an open warehouse environment, execution of bin picking and complex part picking, planning of robot actions to complete kitting orders in the required time, and logistics planning to interface with factory systems. This paper presents an integrated cognitive robotics system that has been developed to solve the kitting problem and deal with the aforementioned problems.
The system is split into three main parts: a high-level
logistics planner, a low-level robot control architecture, and
the automated planning subsystems. The logistics planner
provides an overview of the state of the entire system and
includes a comprehensive world model. It also provides
access for human workers to oversee operations and
interact with planned missions and the world model. The
lowlevel architecture, called SkiROS
The robot used for this work is shown in Figure 1. The robot consists of a robotic arm mounted on an automated guided vehicle (AGV) platform. The platform has enough space for two kitting boxes to be filled concurrently. SkiROS, the task planner, and the robot-level control modules are contained on the robot itself. The logistics planner and mission planner are hosted outside of the robot and communicate via Robot Operating System (ROS) services1 using wifi.
The rest of this paper is organised as follows. First, the related work is discussed, followed by a brief overview of the system architecture. The next three sections then introduce the main components of the integrated system: the logistics planner, the SkiROS system, and the two planning systems. Finally, we describe the results of the in-factory testing of the complete system, and conclude by discussing future work.
Integrated robotics systems are becoming more and more
complex, with many new architectures being proposed. At
the robot-level, hybrid systems with a deliberative high level,
a reactive low level, and a synchronisation mechanism to
mediate between the other two levels
Other approaches that seek to bridge the gap between
highlevel and low-level robot actions include ROSco
Knowledge representation also plays a key role in cognitive
robotic systems
Automated planning has been used for robot control since
Shakey
We begin by describing the system architecture used in this work (Figure 2). The logistics planner interfaces directly with the factory’s manufacturing execution systems (MES) in order to retrieve information about the current state of the factory environment and the kitting orders to be filled. This includes details of the parts to be collected for the next set of products, and also the expected locations of the parts in the warehouse. The logistics planner also interfaces with the mission planner: when kitting orders are received, they are sent to the mission planner which returns an assignment of the kitting orders (goals) to individual robots in a robot fleet.
The logistics planner also communicates with the SkiROS system on each robot. Communication is implemented via generic ROS services, except for communication between the logistics planner and MES, which must conform to the factory’s system interfaces. Once a goal is sent to a specific robot, SkiROS invokes the task planner to find a sequence of skills whose execution should lead to the successful completion of the goal. During skill execution, the internal world model of the robot is constantly updated and replanning is invoked in the case of execution failure.
Both the logistics planner and SkiROS have GUIs designed for easy access to the current system state, and for factory workers to program and update the system. The logistics planner’s user interface can be used to configure the warehouse environment, send and receive missions and tasks, and to monitor the system. The SkiROS user interface can be used to add custom skill sequences, monitor and update the robot’s world model, and initiate the execution of skills.
The system components are described in detail below.
The logistics planner forms the bridge between the robots, the mission planner, and the MES elements. It stores data about the working environment, including identifiers for all physical objects and their three dimensional models. Information is initially compiled from the MES, manual input by a kitting technician, and the states and capabilities of the robots as defined in SkiROS (see below).
The information communicated by the logistics planner mainly concerns the physical objects that are located in the environment. In our application, this equates to shelves, small and large boxes, conveyors, kits, parts, and packaging elements. For each object, its characteristics (e.g., internal organisation, geometry, and spatial location) are modelled in the logistics planner so as to provide this information to the planning and execution components.
The logistics planner’s world model specifies the position, orientation, and geometry (in the form of a volume) of all object instances. This is represented through an inverted hierarchical graph, where a node represents a physical object and the link between two nodes the containment relationship between the two corresponding physical objects. The node on the top of the hierarchy is the root node which models the entire space. This region is further decomposed into subregions for different categories of objects (e.g., shelves, large boxes, and conveyors) or to achieve specific purposes (e.g., robot navigation). Below these specific regions are a set of nodes used to model kits (boxes with compartments), large boxes (containers on the ground organised into layers with compartments in each layer), shelves (also structured into layers that contain small boxes), and conveyors. Parts are not normally represented as nodes in order to avoid excessive information in the graph (e.g., a large box containing four layers may have 80 parts) and also because there is no need to identify each part instance. Common characteristics of parts (e.g., the part’s 3D model) are represented in the world model to support robot operations. The world model is initially constructed in two phases. First, the representation of these object categories is gathered from an external information system. Second, object instances are created and positioned in the 3D space occupied by the logistics supermarket. This is achieved by a technician using the GUI.
The logistics planner supports three methods for inputting and storing information about the environment (Figure 3). First, a 2D image is created by a CAD application to visually specify the location and orientation of each object in the logistics supermarket (Figure 3a). The red rectangle represents the object being placed whilst the remaining rectangles represent objects already in place. Second, a 2D image created by the robot through its Simultaneous Localization and Mapping (SLAM) functionality can be used to visually specify an object’s location and orientation (Figure 3b). Finally, a 3D image of the supermarket is also created to visually specify object locations and orientations (Figure 3c).
A SkiROS component is located on each robot and controls robot-level execution and ontology management. It also communicates status updates and receives mission assignments from the logistics planner. SkiROS is organised into four layers, each of which is represented by a manager. The lowest layer is the device manager, which loads proxies (drivers which conform to a standard interface) and presents standard interfaces for similar devices (e.g., gripper, arm, camera, etc.). The second layer contains the primitive manager which contains motion primitives, software blocks that realise movement controlled with multi-sensor feedback, and services, software blocks that perform a generic computation. The modules are parameterised and contain a parameters specification and execution part.
The third layer, the skill manager, loads skills (see below) and provides interfaces to the layer above. It also registers the robot subsystem with the world model, specifying the hardware, available modules, and available skills. A skill’s execution is usually implemented as a finite state machine which coordinates the execution of several parameterised primitives. Finally, the fourth layer of the architecture is the task manager which monitors the presence of subsystems via the world model and acts as a general coordinator. The task manager is the interface for external systems, designed to be connected to a GUI or the MES of a factory.
A central concept in SkiROS is the idea of robot skills, which can be thought of as general and robust software constructs that model self-contained, re-occurring operations that a robot might perform. Skills are intended to be designed such that they easily map to simple intuitive tasks. For example, a system might include calibration skills, manipulation skills for operations like picking and placing, as well as driving skills for mobile robots. Skills are implemented Robot skill
Execution
Continuous evaluation by experts to contain the necessary sensing and action operations for self-contained execution on the robot platform. One benefit of a skills-based system is that non-experts can typically programme a robot task in a straightforward manner by selecting appropriate skill sequences that result in the desired state changes to the robot’s environment. Skill sequences can also be constructed in a completely automated way using planning techniques, as discussed below.
The conceptual model of a robot skill is shown in Figure 5. A skill takes as input a set of parameters and a representation of the world state; it outputs a set of state changes. A skill contains both precondition and postcondition checks which monitor the environment, either through sensing or based on the world model. These checks allow the task layer to infer the likely causes of execution failures. For example, a precondition check for a pick skill might be that the item to be picked must be visible to a camera, and a postcondition check might be that the picked item must be in the gripper.
In addition to skills, a key part of SkiROS is the world model, which acts as a knowledge integration framework. The world model is a vertical cross-layer component which links all layers together by gathering information from every subsystem at run time, allowing the modules to maintain a shared working memory, and storing the environment and skills information that are used to create the planning domain. In terms of the architecture, the world model can be read and modified by almost every part of the system.
The world state is partially defined by a human operator in the ontology, partially abstracted by the robot using perception, and completed with the procedural knowledge embedded in the skills and primitives. It is originally populated with the robot skills knowledge, robot-specific knowledge such as grasp poses or parameter settings, and with the world model provided by the logistics planner. Each skill manager in the system is responsible for keeping the world model updated with its subsystem information (e.g., hardware, available primitives, skill state, etc.). Similarly, each primitive and skill can extend the scene information with the results of robot operation or sensing. In special cases, the ontology can be extended automatically by the robot to learn new concepts (e.g., a new grasping pose).
Figure 6 shows an example of a world model. The tree constructed by the spatial relations forms the scene graph, a data structure commonly used by modern computer games to arrange the logical and spatial representation of a graphical scene. In this structure, an object’s pose is always defined with respect to the parent frame. The skills are connected to robot elements by the non-spatial relation hasSkill.
Two different planning components are used in the system: a task planner and a mission planner. The task planner forms part of SkiROS and finds skill sequences for robots to complete their missions. The mission planner distributes missions to individual robots in the fleet based on input from the MES and the world model, mediated by the logistics planner.
The task planner automatically generates its domain from
the world model and skill definitions in SkiROS. The task
planner has three main functions: it creates a PDDL
Figure 8 shows the parameters, preconditions, and postconditions for two skills (Drive and Pick), as defined in SkiROS, based on world model constraints and updates. Relations or properties for which postcondition checks are
add: RobotAt(Container, MobileBase) Pick(Gripper, Object, Container) : pre: empty(Gripper) pre: robotAt(Container, Robot) pre: objectAt(Container, Object) del: empty(Gripper) add: contains(Gripper, Manipulatable) expected to be true (similarly, false) become add (similarly, delete) effects. Action parameters are created from the inputs defined for the skill’s execution block.
An overview of the task planning process is shown in Figure 7. The central part shows the task planner’s planning library, which contains all the structures needed to create a PDDL planning problem from the world state, skills, and goals. Specifically, it includes structures for types, predicates (both ground and unground), actions, and (typed) objects.
The process of creating the initial planning representation is given in Algorithm 1, which represents the left hand side of Figure 7 and involves three main steps. The first step is to parse the skills that exist in the world model. This involves adding all types and predicates that appear in the skills definitions to the planning library and also creating an action for each skill, which has a direct copy of the preconditions and effects. All relations, properties, and types that do not appear in a skill are therefore not included in the planning library. The process of iterating through the skills and querying the world model to find true predicates may result in a planning library with less elements and types than in the world model. The omitted data can be safely ignored (and an error given for an incorrect goal) due to the following: Lemma 1: Any object with a type that does not appear in a skills definition can never appear in a solution plan. Proof Sketch: It is impossible to change the truth value of a predicate that does not appear in an action’s effects, and the truth value of a predicate that does not appear in any action’s
Input : Initial Planning Representation Output Final Planning Representation : // Add Capabilities 1 foreach Action a : actions do 2 predicates.add(can_a ?robot) 3 a.pre.add(can_a ?robot) 4 foreach Robot r : hasSkill(a, r) do 5 initState.add(can_a r)
// Spatial Relation Constraints 6 foreach Action a do 7 foreach Spatial Relation S(o; s) 2 a:add do 8 if 6 9S 2 a:pre AND 6 9S 2 a:del then 9 s:params:add(x; s:type) 10 s:pre:add(S(o; x)) 11 s:del:add(S(o; x)) 12 else if 6 9S 2 a:pre then 13 s:pre:add(S(o; s)) 14 else if 6 9S 2 a:del then 15 s:del:add(S(o; s)) (:action drive :param (?R - Agent ?T - Location
* ?preT - Location) :pre (and * (can_drive ?R) * (RobotAtLocation ?R ?preT)) :eff (and * (not (RobotAtLocation ?R ?preT))
(RobotAtLocation ?R ?T))) preconditions can never be required for a change in state.
The right hand side of Figure 7 encodes implicit properties of the world model and system. The first part makes sure that skills are only usable by the correct elements, by querying the hasSkill relation from the world model. For each action, a new predicate (can_a ?robot) is added to the planning description. This predicate is added as true to the initial state for each robot that can perform a particular skill, and is invariant. An extra precondition (can_a ?robot) is also added to each action so that it can only be instantiated by the correct robots. If the Robot parameter is missing from the skill definition then it is added to the action parameters.
The second part adds any preconditions and delete effects that are necessary to maintain the tree structure of the spatial relations in the world model. SkiROS contains methods for internally updating its world model so that it remains consistent, and these methods need to be modelled explicitly in the planning domain. For instance, with respect to the skills in Figure 8, the Drive skill only contains a single predicate which specifies the new location of the robot. This is because the input for the execution block of the Drive skill is only the goal location to which the robot has to move. The drive action must then be modified so that robotAt is true of only one grounding for the robot performing the Drive skill, so the
old instantiation must be found (it becomes a precondition) and added as a delete effect of the action.
Figure 9 shows the skills from Figure 8 after translation to PDDL. Note that in terms of implementation, the parameter added to the Drive skill is removed when returning the parameterised skill to the task manager. The translation adds three new preconditions and two new delete effects over the two actions. The following lemma shows that these additions ensure the world model’s tree structure is maintained: Lemma 2: Performing an action created by the task planner on a problem whose spatial relations form a tree will result in a state in which the spatial relations still form a tree. Proof Sketch: All that needs to be shown is that any deletion of a spatial relation property inserts it elsewhere with the same object (and therefore moves the whole subtree), and that every addition has a corresponding deletion. The former is a constraint on the skill definition. For the latter, every time a new spatial relation appears in the add effects then, by construction of the algorithm, a spatial relation with the same subject must appear in the delete effects. This spatial relation must match the only occurrence of that object in a spatial relation in the current state otherwise the action could not be performed as this must exist (again by construction) as a precondition to the action.
Once the translation is complete, the planning problem is written to domain and problem files in PDDL for use with an external planner. The planner’s output (a sequence of instantiated actions) is parsed and converted back to parameterised skills to be sent to the task manager for robot execution.
The mission planner takes goal and world state
information from the logistics planner and returns goal assignments
(missions) for each robot in the fleet. The mission planner
therefore needs to be able to perform planning for
multirobot domains with large numbers of agents. These types of
domains are generally hard to solve for traditional classical
planning approaches so a multiagent planner
As the mission planner must find plans for the entire robot fleet, creating the planning domain manually is necessary in order to remain scalable as the system increases the number of robots. Some of the more fine-grained information available in the world model is abstracted away (e.g., information used by the on-board robot task planners to find the exact skill sequences), since the mission planner only needs to return a viable assignment of goals. In particular, the mission planner focuses on making sure that plans are achievable and assigned to the most appropriate robots, not with skill orderings. For specific implementations, depending on the domain sizes, complexity, and (crucially) the number of robots, it might be possible to use the task planner’s automatically generated domain, or necessary to abstract even more. In either case, the interfaces to the logistics planner remain the same.
The complete system was deployed in a real-world environment (Figure 1) at a PSA Peugeot Citroën factory in Rennes, France. The results of the experimental testing are presented in Table 1. The tests were carried out using simulated kitting orders as input. The system success rate shows the success of the system, ignoring any errors that came from robot execution modules. In all cases, the mission planner successfully found mission assignments, and the task planner found correct sequences of skills to complete the missions.
The failure points in the system came from robot execution code for individual skills, which (for this implementation) are still undergoing design change. In some failure cases, replanning was able to recover from a failure in execution, though these cases are still marked as failures in the results. Overall, the results show that the system architecture is sound, but the execution code for the skills needs to be improved. Due to the modular nature of the system design, the skill execution code can be updated without affecting the logistics planner, automated planning, or high-level SkiROS components.
This paper described a fully implemented and integrated system for autonomous robots in an industrial factory setting. The system uses planning and a skills model called SkiROS to bridge the gap between low-level robot control and high-level planning. A logistics planner mediates between the factory MES and the mission planning component. Skills are explicitly defined, while a task planner automatically generates the corresponding PDDL planning domain. The resulting system has been shown to operate successfully in a factory environment. From an end-user perspective, the robot is programmed to perform new tasks by specifying goal conditions; new skills are added by specifying constraints on the world model with no explicit knowledge of planning required.
Work is progressing to test more skill implementations and further explore the relationship between skills and planning. Failure handling will be improved to allow for replanning in the case of unsatisfied pre/postconditions and execution failures. To optimise cycle time, the assumption of sequential skill execution will also be relaxed, allowing parallel skill execution from temporal plans.
The research leading to these results has received funding from the European Union’s Seventh Framework Programme under grant no. 610917 (STAMINA, stamina-robot.eu).