A cognitive robot may face failures during the execution of its actions. These failures are mostly due to the gap between the physical world and the constructed symbolic plans, some internal problems that may occur in its embodiment or unexpected external events. In this paper, we propose a visual scene interpretation system for extracting spatial relations among objects in a scene and using these relations to detect failures during the plan execution. Our system uses LINE-MOD and HS histograms in order to recognize textureless objects with different shapes and colors. Then, it analyzes the scene to specify the world state after each action execution. Our focus in this research is on particularly the following spatial relations: on, on table, clear and unstable. In the experiments, we test the performance of our system on recognizing objects, determining pairwise spatial relations among them, and detecting failures using these relations. Our preliminary results reveal that our system can be successfully used to extract spatial relations in a scene, and to determine failures during plan execution by using this information.
A cognitive robot possesses abilities to construct symbolic plans to solve given
problems and to execute these plans in the real world. Automated planners
are commonly used for determining a valid sequence of actions for a robot to
achieve its goals. These planners use high-level description of the problem and
the domain (i.e. initial/goal states and operators corresponding to real-world
actions) to construct a plan. After obtaining a valid plan, the robot needs to
execute the corresponding real-world actions in order to attain the desired goal.
However, it may face several types of failures during the execution of its actions
in the real world [
Ensuring robostness is crucial for a cognitive robot in order to accomplish the given goals in the real world. In this work, we investigate how spatial relations among objects are determined using visual data from an RGB-D camera and how this information is used to detect action execution failures in the real world. As a motivating example to illustrate the stated problem, consider the object manipulation task in the blocks world domain. An example plan constructed for a 3-block problem is given in Figure 1. In this toy problem, the aim is stacking three blocks on top of each other where all blocks are initially on the table and without any other objects on top of them (i.e. satisfying clear predicate). During the execution of the generated plan, the robot may fail in executing action stack. Possible reasons for this failure might be the weight of the object, improper grasp position or a vision problem. To ensure robustness in such cases, the robot needs to continuously monitor the state space for anomalies during action execution.
s0 (handempty) (clear a) (clear b) (clear c) (on_table a) (on_table b) (on_table c)
s1 (holding b) (clear a)
X (clear c) (on_taXble a) (on_table c) o0 o1
o2 pickup b stack(b,c) pickup a
stack(a,b) s2 (handempty) (clear a) (clear b)
X (on_taXble a) (on_table c) (on(b,c))
s3 (holding a)
X (clear b)
X X
X (on_table c) (on(b,c))
o3 b c b c
s4 (handempty) (clear a)
X X X
X (on_table c) (on(b,c)) (on(a,b)) a b c a b c a c
a
Throughout the paper, we first give some background information on the areas of automated planning, object recognition and scene interpretation. Then, we describe the details of our system for determining spatial relations among the objects in order to detect failures. We then give empirical results of our approach followed by the conclusions. 2
In this section, we formulate the planning problem and give a brief review of the approaches used for recognizing objects and determining spatial relations in the scene. Then, in the following section, we present our solution to the stated problem. 2.1
Cognitive robots may use automated onboard action planning for online
generation of action sequences to accomplish given tasks against exogenous events.
A planning task Π can be described on a state space S containing a finite
and discrete set of states including an initial state s0 and a goal state sG,
and a state transition function st+1 = f (ot, st) where ot ∈ O(st) is an
operator applied in state st. State transition function is realized through
planning operators o ∈ O that are defined as symbolically abstracted
representations of real world actions a ∈ A. A planning operator can be formalized as
a tuple o = {pre(o), add(o), del(o), cost(o)} where pre(o) defines the
preconditions, add(o) and del(o) define the effects of the operator and cost(o) represents
the cost of the corresponding action. ot ∈ O(st) is defined as the set of
applicable operators in a state st ∈ S determined by checking preconditions of
the operators to satisfy pre(ot) ⊆ st. By applying ot at state st, a new state
st+1 = add(ot) ∪ (st \ del(ot)) is observed. Planning task is achieved by a planner
to reach sG from s0 by selecting a sequence of operators from O(st) at
successive states st and executing the corresponding actions at ∈ A in the given
order. After searching the whole space of operators, the planner constructs a
valid plan P = o0:G by considering an optimization criteria (e.g., makespan)
and the duration/cost of each operator. Having generated a valid plan P , the
robot can execute each corresponding action ot → at ∈ A in sequence in the
physical world. If all goes well with execution, the robot successfully attains sG.
However, due to non-deterministic actions and different sources of uncertainty
in physical environments, several failures may be encountered [
There are various approaches for recognition of objects in a scene using
different types of visual clues. These approaches can be categorized as 2D object
recognition approaches based on local invariant feature descriptors and 3D
object recognition approaches based on surface normals computed from the depth
map. In the case of 2D color data, local feature descriptors are used to determine
patterns in the image which differ from the other pixels in their neighborhood.
These distinguishing parts of the image (i.e. keypoints) are generally chosen by
considering sharp changes in color intensity and texture. To store the keypoints,
descriptors are computed around them which are suitable for measuring their
similarity. The idea of using local invariant descriptors became popular when
Scale-Invariant Feature Transform (SIFT) [
Detecting and representing structures with spatial relations in a scene is known
as the scene interpretation problem. While this is a trivial task for humans,
interpreting spatial relations by processing visual information from artificial vision
systems is not a totally solved problem for autonomous agents [
Our proposed work differs from the previous studies in two ways. First, determining spatial relations is done for a higher level task of detecting failures after action executions. Second, the object recognition system used in this work is more generic as it can deal with textureless objects that do not have any distinguishing texture information. 3
Scene Interpretation for Monitoring Action Executions
We propose a failure detection system based on visual information. The system
involves three main procedures, namely, object recognition, scene interpretation
and failure detection. In the system, first, each object of interest is modelled
by creating multi-modal LINE-MOD [
After the objects are recognized and located in the scene, qualitative spatial relations are determined for failure detection. In the blocks world domain, these relations are on, on table, clear and unstable. Initially all the recognized objects are assumed to be on table and clear. Then the on relation is determined between each pair of objects as follows,
∀obji, objj , (EC(obji, objj ) ∨ P O(obji, objj )) ∧ N (obji, objj ) ⇒ on(obji, objj )
where EC(externally connected) and PO(partially overlapping) are topological
predicates of RCC8 [
To eliminate false positives in the extraction of on table relations due to recognition failures, the area under the object is checked in order to see if it is planar or not. If the area is not a horizontal plane, than it is assumed that the corresponding object is not on the table. Finally, unstable relation is determined by checking the horizontal projections of the aligned object templates. If the horizontal projection of the upper object in the on relation has an unsupported part (i.e., out of the area covered by the object below) of more than 1/4 ratio to its length, this on relation is assumed to be unstable.
Action execution failures are detected by checking the state of the world with
respect to spatial relations after each action execution. We consider three states
after executing an action [
Definition 1 (success state) If all the desired effects of the action occurs in the environment, the situation is specified as success.
Definition 2 (fail-safe state) If the state of an execution is not success but the state does not change, the situation is specified as fail-safe. For example, the robot fails in picking up an object but the state of the object is not changed. Definition 3 (fail-unsafe state) If the execution of an action fails and there is any damage and/or dangerous situation (e.g., an undesirable state is observed) or the robot cannot judge whether there is any harmful situation, the situation is specified as fail-unsafe. For example, the robot fails in picking up an object, and the object is broken into pieces or fallen down the ground out of reach of the robot. 4
In the experiments, the proposed system is evaluated in real time for different possible situations in the scene using the real-world data captured by an RGB-D sensor. The objects used in these experiments are three paper blocks, two plastic toy grapes and a single toy box (Figure 2). These objects are selected as they have different shape and/or color features.
red block green block blue block purple grapes green grapes
box
First, the overall recognition performance has been evaluated by comparing the results of LINE-MOD and our approach combining LINE-MOD with HS histograms. The results are illustrated in Table 1 as a confusion matrix for 120 different scenes (20 scenes for each object). As expected, both LINE-MOD and our approach give good recognition rate for different shaped objects. However, LINE-MOD cannot always distinguish similar shaped objects with different colors. Our approach based on checking HS histogram correlations on the results obtained using LINE-MOD leads to much better results in these situations. False negatives in recognition are slightly greater in our approach since some correct results are eliminated by checking color correlation.
Second, the performance of our system for extracting spatial relations: on, on table, clear and unstable has been tested in an experiment involving 100 scenes (50 scenes for the blocks, 50 scenes for the grapes and the box). The results are shown in Figure 3. As given in these results, our system can be used to successfully detect relations for all the objects used in our tabletop scenarios. The highest error is in determining on relation by 20% and this is caused by the objects that cannot be recognized. When the object that is located below another object cannot be recognized, this also affects the success of determining on table relation. Similarly, there are some errors in determining clear relation for an object when another object that is located on top of it cannot be recognized. Errors in unstable relation are due to the failures in recognition or bad alignment of the recognized templates.
on s n itoon_table a l e R ilta clear a p S unstable
Wrong 0% 20% 40% 60% 80% 100%
In the last set of experiments, the performance of the failure detection process has been tested on 20 different scenes that are set for each execution state: success, fail-safe and fail-unsafe. Similar to the previous set of experiments, it has been observed that, spatial relations are determined correctly in 85% of the scenes where the action is executed successfully. Moreover, the system has been observed to label 90% of the scenes with a fail-safe state correctly where the state of the world does not change after executing an action. In these experiments, when the object to be stacked is fallen down the table and this object cannot be recognized in the scene, the state is assumed to be a fail-unsafe state. With this assumption, the system has identified all fail-unsafe examples in the 3-blocks problem correctly. 5
We have presented an approach for detecting failures to ensure robust task execution in cognitive robotic applications. Our approach is based on using visual information extracted from the scene in order to determine spatial relations among the objects that are involved in manipulation scenarios. First, we have shown how our system can be used to recognize objects with different geometric shapes and colors. Then, we have given the details of the visual scene interpreter for specifying spatial relations among the objects of interest and evaluating these relations for detecting failures during action executions. The preliminary results of the conducted experiments on our system indicate that the system can be used to successfully detect states with failures in an object manipulation scenario. In our future studies, we plan to conduct experiments on larger sets of scenes involving various objects to justify our research. Our ongoing work includes the integration of temporal reasoning into spatial reasoning in order to detect the possible causes of failures from previous states (e.g., an unstable stack of blocks causing a failure when stacking another block on top of them). 6
This research is funded by a grant from the Scientific and Technological Research Council of Turkey (TUBITAK), Grant No. 111E-286. We thank Prof. Muhittin Gokmen for his recommendations on vision algorithms.