In tabletop construction scenarios, robots work with vertically or horizontally stacked object structures. In order to form such structures, they need to recognize and correctly model closely placed objects in such structures. Depending on the robot's point of view and the objects' positions, it is likely that objects closely located or in contact partially occlude each other, and as a result it is not always possible to model object stacks by relying only on object recognition. However, if the objects are added to the construction consecutively, it becomes possible to sequentially build the model of object stacks. In this work, we propose a scene interpretation system to build and maintain a consistent world model for tabletop construction scenarios. To overcome the challenge of modeling object stacks, we extend our previous scene interpretation system with a semi-closed world assumption and by preserving the models of objects in the formed structures even when they are out of sight. Our extension includes the use of spatial object relations, as well as depth-based segmentation results to model not only single objects, but object combinations. In our system, the LINE-MOD algorithm and an enhanced version with HS histograms are used for recognizing objects along with depth-based segmentation for detecting novel objects. We run numerous construction scenarios using building blocks and show that our system can be successfully used for modeling constructed objects.
In order to achieve given goals, robots often need to interact with various objects. For
successful interaction, before anything else, they need to collect correct information
about the objects in their environment. The required data includes the accurate
properties of objects, such as their size, shape and color, their locations in the world and
necessary inter-object relations. For this purpose, robots use their sensors to gather
observations from the world, which sometimes do not overlap, are not complete, and sometimes
even contradict with each other. Our previous work presents a scene interpretation
system to cope with these challenges for a ground robot [
World modeling is especially challenging when objects are in direct contact with
each other either horizontally or vertically (i.e., when they are on top of each other).
In these scenarios, it is likely that they partially occlude one another from the robot’s
point of view or the vision algorithms may fail in recognizing all objects. For example,
consider the scenario where the robot is tasked to build a rectangular prism structure
from a set of cubical blocks in different colors and sizes as in Figure 1. In this figure,
the LINE-MOD algorithm [
Given the requirements in modeling objects in construction scenarios, we propose new advancements over our previous scene interpretation system. The contributions of this work are three fold. First, the scene interpretation system is made capable of using observations taken during execution and building the model of a structure incrementally using both temporal and spatial relations extracted during runtime and prior semantic rules for handling occlusions. Second, a truth maintenance mechanism is applied to store the models of occluded objects even if they can not be recognized but to remove their models when they are believed to disappear from the scene. Third, for the identification of objects a semi closed-world assumption is applied for symbol grounding.
The rest of the paper is organized as follows. First, we mention studies related to world modeling. Then, we describe our scene interpretation system for consistent world modeling in tabletop block construction scenarios. We then give empirical results of our system followed by the conclusions.
Several recent studies address the issue of maintaining a world model from the robot’s
visual observations. Nyga, Balint-Benczedi and Beetz (2014) proposed an ensembles of
experts approach based on Markov Logic Networks [
The first step in object manipulation by autonomous robots is maintaining a consistent
and up-to-date world model about their environment. For this task, the robot has to
collect visual recognition and detection data to filter out and to reach conclusions. Our
perception system uses LINE-MOD [
The perception sources are implemented as separate processes, where their
recognition/detection results are asynchronous. The Scene Interpreter system combines these
results to create an accurate representation of the world [
Object recognition is not reliable alone for robotic manipulation tasks, since failures in
recognition or detection occur due to noisy sensor measurements, illumination changes,
dynamic environments or other agents and sensors. In order to automatically build
a consistent and up-to-date model about the environment, visual recognition and
detection outcomes should be filtered out and logical conclusions should be reached in
the face of contradictory outputs. Previous work by the same research team includes a
Scene Interpretation system for ground robots working with objects clearly separated in
the horizontal plane [
The confidence is represented with a value varying between 0 and 100, proportional to the degree of belief on the corresponding object’s existence. Confidence values are updated with every new perception outcome. An object’s confidence value increases as more consistent recognition or detection results arrive regarding the object.
The observed facts are kept in the Knowledge base (KB) of the robot, which can be defined as a collection of reached conclusions about objects, their properties, and interobject relations. The robot’s KB is initialized as empty. During run time, recognized objects are inserted into the KB and their corresponding confidence values, as well as properties, are updated with each newly received recognition message. If an object in the KB does not receive any corresponding recognition message for a period of time, even though this object is in the robot’s field of view and should be recognized, the confidence value regarding the object is gradually decreased. If this value reaches zero, it is believed that the object is no longer in the robot’s environment, and thus it is removed from the KB.
Most humanoid robots have the capability of moving their heads around, making it possible for them to observe more about their environment. As a result, their visual field of view (FOV) is bounded by the limitations of their cameras. A robot can receive reliable information about objects only within its FOV and the field of view constraints should be taken into account when updating object properties. Extending the 2D definition in the base system, 3D boundaries are empirically determined for an RGB-D camera where the objects within are expected to be recognized reliably. An example scenario regarding FOV calculations can be seen in Figure 2.
Objects in the environment are considered depending on whether they are inside the camera’s FOV or not. Objects outside the FOV are not expected to be recognized, and any data regarding them in the KB are kept static until they re-enter the FOV of the robot, and new perception data are available.
(a)
(b)
After recognition and localization of the objects in the scene, their spatial relations are
determined as in [
The first part of the study focuses on symbol grounding problem for objects ids. Ambiguities in determination of ids can arise in dynamic scenes. Objects may be displaced, removed from and put back into the scene, or the robot could be mobile and have localization problems. As a result, an object might be registered with different ids over time, which prevents creating and executing plans including object manipulation successfully.
After successfully detecting objects, the world is assumed to be closed (closed world
assumption) for identity resolution tasks. Closed world assumption [
Data: Detected object attributes Result: Object Id foreach object in KB do if attributes match and object is not in the scene then
return object:id; else end end newId generate new id ; return newId
Algorithm 1: Algorithm for Semi-Closed World Assumption
The second focus of the study is to correctly model closely located objects. Object recognition in cluttered scenes is still a challenging problem. Object detection success is low in such scenes due to their placements. An example scenario with six cubical blocks is given in Figure 4. The system can not distinguish between objects and only some of the objects can be added to KB. This is the natural result of assumptions of vision algorithms. LINE-MOD extracts surface normals on visible surfaces and color gradients around borders. Furthermore, the 3D segmentation algorithm assumes objects are clearly separable on a supporting plane.
The first solution attempt to this problem was decreasing the similarity threshold of the LINE-MOD algorithm between the object templates and real time detections (See Figure 5). The threshold is set to 95% by default, and it is decreased gradually. As a result, the system was able to detect the objects and register them into the KB. The main drawback of the approach is, as the threshold is lowered, the number of false positives, i.e. the number of misdetections increase. As the threshold reaches 80% and below it becomes harder to maintain the number of objects in the KB.
For a similar scenario where the threshold is set to 95%, even though objects are placed into the scene one by one but very closely, some of the previously recognized objects are removed from the KB due to lack of recognition messages after some point. The second proposed approach utilizes the 3D segmentation algorithm. We can rely on detections of the 3D segmentation algorithm in terms of the existence of an object in the scene. If the objects are placed into the scene one by one and clear enough to be recognized, after successfully being added to the KB, the objects can be marked as detected, if their centroid lies in one of the last segmented point clouds produced by the 3D segmentation algorithm. Thus, the objects are exempted from being removed from the KB. The proposed algorithm is given in Algorithm 2.
The segmentation algorithm is used to maintain object stacks of the same level. In order to increase the level -the height- of the structure, spatial relations among objects, namely on relations, are employed. When objects are stacked on top of each other, corresponding on relations are detected between object pairs. In a pair, the bottommost object is partially occluded, so it is not expected to be detected, which avoids the update operation on the object and thus the deletion from the KB. In addition, if the topmost object of a pair is removed from the scene, the corresponding object model and the on relation are also removed from the KB, and the bottommost object is expected to be detected again.
KB initialize empty knowledge base; upon receive Objects:; /* Objects : Recognized objects via LINEMOD and LINEMOD&HS */ foreach object in Objects do if object in KB then
update object; else end
KB
add object ; end upon receive Segments:; /* Segments : Segmented point cloud clusters foreach object in Objects do if objectcentroid in Segments then object mark object as detected */ end
end
Algorithm 2: Maintaining Closely Located Objects This section describes the experimental setup and presents the obtained results. First, we present object recognition and registration to KB during run time. Then, id tracking capabilities of our system under semi-closed world assumption is demonstrated.
For the first part, block construction scenario is considered. Red, green and blue colored blocks are placed into the scene sequentially to form horizontal, vertical and diagonal structures on the same plane. For comparison purposes, experiments are repeated with and without employing the proposed segmentation based approach. Each time, after a block is placed, the number of objects registered to the KB is recorded. Each case is repeated 10 times, and the mean is calculated. Figure 6 shows the comparison for horizontal, vertical and diagonal structures. Note that, since the results are recorded in a sequential manner, errors in the previous steps accumulated to oncoming steps.
The following conclusions can be drawn from the analysis given in Figure 6. Employing segmentation based approach fairly increases the number of objects registered to the KB. The best performance is obtained from vertical placement scenario due to the fact that the last placed object can be correctly isolated from its surroundings and thus, it is easier for the vision algorithm to recognize. However, in the diagonal placement scenario using segmentation does not provide much improvement since objects are in less contact with each other. Horizontal scenario is the most complicated one in terms of distinguishing between objects, since objects have more contact with each other. Improvements become clear when the number of objects in the structure is increased.
In another experiment, blocks are stacked on top of each other one by one to measure on relation detection success when new layers are introduced. This time, after each (b)Vertical Registered Objects in the KB (c) Diagonal
Registered Objects in the KB 6 B5 K e h ijttscenbO43 # e2 g a r1 e vA 0 2 Without Segmentation
With Segmentation 3#Objects4in the Sce5ne 6 (e)Vertical
Without Segmentation
With Segmentation 3#Objects4in the Sce5ne 6 (f) Diagonal (a) Horizontal Registered Objects in the KB
Without Segmentation
With Segmentation 3#Objects4in the Sce5ne 6 (d) Horizontal 6 B5 K e h ittscnbO4 je3 # e2 g a r1 e vA 0 2 stacking operation, the number of on relations is recorded. It is expected to detect one on relation with a stack of two objects, two on relations with a stack of 3 objects and so on. Success rates of detecting on relations are 100%, 100% and 93.3% for the number of layers 2,3 and 4 respectively. Due to object recognition failures, success rate is decreased in the 4th layer and above.
The goal of this experiment is to illustrate id tracking capabilities of the system. The object set contains red, green and blue colored, medium sized cylinders, and small and large sized blocks. In the model; type, size and color attributes are taken into account. An example scenario is visualized in Figure 7.
The KB is initialized by putting all target objects into the scene, and each object is assigned a unique id. Then, the objects are removed from the scene. Each object is put back and id assignments are observed. A confusion matrix based on id assignments is given in Figure 8. Whenever an object could not be matched with the previously encountered objects registered to KB, a new id is generated for the object. The reason of mismatches are originated from errors in recognizing objects due to illumination conditions. It is observed that if an object is failed to match with an object in the initial object set, and thus attached a new id, the consecutive recognitions are also matched to this id. Whereas, some objects could not be recognized at all, which are denoted as not detected in Figure 8.
We have presented enhancements for our scene interpretation system in order for it to be used in tabletop manipulation and construction scenarios for cognitive robots. First, (a) (b) (c) (d) the 2D model of a ground robot’s field of view was extended to 3D for a humanoid robot with a moveable head. Then, we introduced a hybrid model of open and closed world assumptions for keeping track of object ids in case of dislocations and disappearances & reappearances. This hybrid model is able to keep track of lost objects, while still allowing new objects to enter the scene. Deductions about object ids are made based on physical attributes of the objects and without using any kind of prior knowledge. Finally, for block construction scenarios, we proposed utilizing 3D segmentation on top of object recognition to maintain objects in the KB when they are in direct contact with each other and cannot be recognized. Furthermore, we employed spatial relations to maintain objects that have other objects on top of them and thus to model higher level structures. Future work includes improving the system to keep track of more complicated scenarios that include unknown objects, and modifying the system to operate on a probabilistic framework.
This research is partly funded by a grant from the Scientific and Technological Research Council of Turkey (TUBITAK), Grant No. 111E-286.