Recognising objects in everyday human environments is a challenging task for autonomous mobile robots. However, actively planning the views from which an object might be perceived can significantly improve the overall task performance. In this paper we have designed, developed, and evaluated an approach for next best view planning. Our view planning approach is based on online aspect graphs and selects the next best view after having identified an initial object candidate. The approach has two steps. First, we analyse the visibility of the object candidate from a set of candidate views that are reachable by a robot. Secondly, we analyse the visibility of object features by projecting the model of the most likely object into the scene. Experimental results on a mobile robot platform show that our approach is (I) effective at finding a next view that leads to recognition of an object in 82.5% of cases, (II) able to account for visual occlusions in 85% of the trials, and (III) able to disambiguate between objects that share a similar set of features. Hence, overall, we believe that the proposed approach can provide a general methodology that is applicable to a range of tasks beyond object recognition such as inspection, reconstruction, and task outcome classification.
Autonomous mobile robots that operate in real-world environments are often required to find and retrieve task-related objects to accomplish their tasks. However, perceiving and recognising objects in such environments poses several challenges. Firstly, object locations are continuously changing. That is, a mobile robot cannot simply rely on a fixed set of views from which it can observe an object. The robot needs to plan in a continuous space where to stand and where to look. That is, the robot has to select a view from the uncountably infinite set of possible views which allows it to observe the sought object. Matters are further complicated by dynamic obstacles which might hinder the robot to take a particular view. Or, when taking a view, relevant features of an object might be occluded by other objects and/or by the object itself (self-occlusion). Finally, other conditions such as lighting and/or sensor noise can influence the performance of object recognition tasks.
In previous work, we have enabled robots to find objects
in particular rooms
Our local view planning approach addresses all of these above mentioned challenges. It is based on using a realistic sensor model of an RGB-D camera to generate online aspect graphs (a set of poses around an object which describe object visibility at that point) and takes the kinematic constraints of a mobile robot platform into account. Hence, by changing the sensor and/or the kinematic model our approach can easily be transferred to robot platforms of different types. We further consider two environmental constraints when generating robot and camera poses: (1) dynamic obstacles which might hinder a robot to take particular views, and (2) occlusions of object features which might be hidden by other objects (or by the object itself). Finally, we consider learned object models in the planning process to predict the location and visibility of features of object candidates. An overview of the next best view planning approach is given in figure 1.
Experimental results show that our next best view planning approach, which takes multiple views of an object, allows us to improve the performance of recognition tasks when compared to single-view object recognition. Further, it enables robots to differentiate between objects that share a similar set of features.
To this end, this work contributes the following: 1. a method for analysing potential views using online aspect graphs by taking both into account: (1) occlusions, and (2) the visibility of features based on learned object models. 2. a next best view planner that selects a view based on the method above and an executive that accounts for dynamic obstacles during execution 3. a set of experiments which demonstrates (a) how robots can disambiguate objects which share similar feature sets,
and (b) how the performance of object recognition can be improved by taking multiple views.
The remainder of the paper is structured as follows. We first discuss related work in Section 2, followed by a conceptual overview of our approach in Section 3 and a detailed description of the implementation in Section 4. In Section 5, we present and analyse experimental results and discussion, before we conclude with a summary and conclusion in Section 7.
2
Hutchinson et al.
The approach used by Stampfer et al.
Early work into next view planning was based on
detecting which parts of an object were causing self occlusion
Okamoto
Callari
Wixson
Vasquez-Gomez and Stampfer
3
This section provides a conceptual overview of the proposed view planning approach. Implementation details can be found in Section 4.
Figure 2 shows the next best view planner’s place in the perception-action cycle of a robot. A hypothesis about an object’s identity and its estimated pose are the inputs into the planner. As output, the planner provides the next best view from which it determines the robot has the best chance of identifying the object.
The following briefly describes the view planning process after a candidate identity is received: (I) potential viewing locations are checked for dynamic obstacles on the local cost map. (II) Views that are reachable by the robot are subjected to an environmental analysis to determine if any visual occlusions block the view of the candidate object. (III) Views which survive environmental analysis undergo model analysis to determine the amount of visible surface area visible from each view point.
Before describing the individual components of the view planning approach in detail, we motivate when next best view planning is initiated and why.
In the context of object search tasks, a robot might seek
objects in certain rooms
When any of these conditions are met, next view planning is the initiated. The components of the approach are described in the following sections.
Streams of images received from the robot’s camera are processed to detect candidate objects. Bottom-up perception algorithms used - making estimates of object identities based on information available in the scene and no contextual information. Segmented sections of the scene are compared with a model database; any matches between the two are returned with a confidence measure as estimated object identities.
An object cannot be seen in its entirety from one
viewpoint and different viewpoints prnt different features to a
sensor. Aspect graphs are a method of simulating which
features may be visible at different viewpoints around an
object. A typical aspect graph for next view planning
consists of a sphere of poses around a model and
geometric analysis determines which features are visible from
each point. In past work, aspect graphs have been
computed offline
This project will instead build aspect graphs online which will be built in two stages. By moving online, we can account for real-time information about environmental obstructions and their effect on potential new poses. Aspect graphs are generated for both Environmental Analysis and Model Analysis, which will be discussed next. The shape of aspect graphs in this project are governed by the robot’s degrees of freedom. The robot used in this paper had 3-DOF (base movement and pan/tilt unit) and so graph nodes were arranged in a disc surrounding the candidate object. Environmental Analysis After receiving a candidate identity we need to decide which available pose offers the next best view. The first step is to determine if any part of the environment lies between the sensor and the candidate object, thus creating a visual occlusion. To achieve this, a snapshot of the current environment is taken and converted into a volumetric representation. From various points around the candidate object the sensor (oriented towards the object) is modelled. Within this model, any part of the sensor’s field of view which does not reach the estimated position of the object is discarded. This leaves a circle of poses around the object, each containing its respective remaining field of view which allows unobstructed line-of-sight to the object. These remaining parts of the field of view are then carried forward to model analysis.
Model Analysis Surviving sections of the modelled camera at each pose do not necessarily represent the visibility of the object at that location. In order to establish which view provides the most information about the object we use the model of the object from the object database along with surviving sections of the modelled cameras at each pose from the previous step.
The object database contains a manipulable model of the target object. This model is rotated to match the estimated pose of the candidate object. From here we again simulate each camera pose from the previous step, using only the surviving fields of view. Each modelled camera at each pose will be oriented towards the object model; the proportion of the field of view of each camera which is filled by the object is then saved and represents the visibility of the object from each pose around it.
After environmental and model analysis, each pose is matched with a score which represents its visual coverage of the candidate object. The pose with the highest score is determined to be the next best view; this is sent to the robot’s navigation component. Once at the new location the robot will either accept or reject the initial identity estimate or begin to determine another next best view if the first did not lead to recognition.
In this section we describe how the view planning approach is integrated with the data structures and algorithms of the robot’s perception and control components. Figure 3 provides an overview of the different components and explains the view planning process step-by-step.
In this work, we build on a state-of-the-art object modelling
and object recognition framework
In Figure 3, the input to the object recognition service, the service itself, and a visualisation of the output is depicted in Step A, B, and C respectively. In this case, the output is a hypothesis for a candidate object identity (here: a book). The object recognition service is used after moving to the next best view (see Step H, I, and J) this time, the outcome is a verified identification of the object.
After a candidate identity is received, a series of poses are generated around it. These poses are generated in two uniformly distributed rings of robot poses around the estimated location of the object, each oriented directly towards the location of the object.
The accessibility of each of these poses is assessed by collision checking the views using the local cost map. Any views in which the robot would collide with environmental obstructions are discarded. This is seen in Figure 3 (Step D), where views that make contact with the supporting surface of the object are eliminated. The remaining views are carried forward to be assessed for visual occlusions.
After a set of accessible views has been generated and
tested, environmental analysis is performed on the initial
point cloud to assess whether any regions of the environment
might occlude the view of the object. In order to do this, the
current point cloud is converted into an octree representation
(Figure 3 Step E) (using Octomap
Figure 3 (Step F), shows the model analysis step. By completing the previous two steps, the amount of possible view locations and camera fields of view have been reduced. Each 1http://www.ros.org/ surviving part of the modelled camera is simulated and directed towards a model of the candidate object. The remaining sections of each camera are simulated using ray-casting; this computes a line from the origin of the camera out in the direction of the field of view. If the line makes contact with the model it is considered that the part of the object with which it made contact would be visible to the camera from that viewpoint. After the camera is modelled at each view, we are left with a measure of the amount of object visible at each location. The view which enables the highest visibility is then considered the next best view.
Multi-object Disambiguation Note, if the recognition service returns more than one candidate hypothesis, aspect graph building is performed for every object, each with its own pose transformation. After this, each is analysed to find the best compromise view—one which gives the best chance for recognition of each model. A solution to this is, after calculating the sum of visible surface area for each view, the difference between them is then subtracted. This ensures no high surface area visibility from one pose dominates over a low score for the same pose on another object.
When the next best view has been found, the pose associated with it is sent to the robot’s navigational packages, which manoeuvres the robot to that view (Figure 3, Step G). In this case, the pose consists of movement by the base of the robot and angular movement by pan/tilt unit to centre the camera on the object’s location. Once the goal is reached, input resumes to the recognition service (Figure 3, Step I); The response of the recognition service will denote different things: 1. Verification: If a high confidence estimate of the object’s identity is returned, the next view is considered successful and the object considered identified. 2. No candidate: If the recognition service returns no high confidence identity after the next view it should either be considered successful in dispelling a false hypothesis from the first view (true negative) or unsuccessful, as it has lowered the amount of information available to the recognition service to deny further identification.
Depending on the result of the movement: the object search will end, another view is taken or the candidate will be discarded and the search continued. This gives a detailed overview of the process this next view selection undergoes in order to produce a next best view pose. The components of this system will now be assessed and results examined. 5
Experiments were conducted to test the capabilities of this next best view planner. All experiments were carried out on a Scitos G5 robot equipped with pan/tilt unit. In each experiment the robot was located in an open area with few obstacles. The centre of this area contained a tall, thin plinth, the plateau of which was just below the robot’s camera height. Test objects and obstacles were placed on this supporting table and the robot was able to move around to reach a new view. Each experiment used its own specific of target objects. Details of the set-up of individual experiments are given below.
Set-up The primary function of this planner is to take in hypotheses about potential objects and select the best pose to enable their accurate confirmation or rejection. This function was tested during this experiment. No obstacles or occlusions were used apart from the object’s supporting plane. 40 trials were carried out in total, 20 each on two different objects: a large book and a standard mug. In all cases the robot was initially positioned with a view of a the object which gave a low chance of recognition. Success in these trials were defined by the ability of the recognition service to make a high confidence identification of the target object after the next best view planner selected a new pose and the robot had moved. In each trial up to two next best view locations were permitted.
Results Table 1 shows the results of this experiment. The table shows a successful verification rate of 82.5%, meaning that in most cases the planner was able to take an uncertain hypothesis and verify it through moving to a new location. In 10% of trials, two next view poses were taken, but no recognition was achieved.
Discussion Failure to provide a verified hypothesis in 10% of cases can be attributed to the pose estimation provided by the recognition service. If the view of the candidate in the initial view is very low quality, pose estimation can be inaccurate; this leads to poorly aligned candidate poses and thus inaccurate movement. In some cases this was be compensated for: of the 33 successful trials, 7 required two views before high confidence recognition was achieved. The initial inaccurate pose estimation led to poor next view selection; however, the new view presented better quality information about the object to a point where the pose estimation improved and the subsequent view resulted in recognition. The nature of taking pose estimates from low confidence identity hypotheses does inherently hold the risk of inaccurate pose estimation, so this is to be expected.
Circled poses in figure 4 show the final positions of 10% of the trials in which no recognition could be made after two views, which can be attributed to a succession of inaccurate pose estimations. The red pose arrows in figure 4 tend to cluster in areas with a view of large surface area of the object. When using the book this was a view from which both the spine and cover were visible and where the handle and body were visible on the mug. This demonstrates that given a reasonably accurate initial pose estimation of the object, the next best view planner is able to locate the view which presents one of the largest faces of the object to the camera and that doing this leads to reliable recognition. This shows that aspect graphs can be computed online to lead to high recognition accuracy and do not require excessive analysis.
Set-up After making a high confidence identification of an object, it may be useful to take another from a different position, as two high confidence identifications provide more certainty than one. This experiment followed the same procedure as experiment 1, except for the starting position of the robot which was positioned to allow a high confidence identification. Success in this experiment was measured firstly on whether the next view led to another high confidence recognition and secondly on how much identification confidence increased from the first pose to the next. Results Table 2 shows the descriptive statistics for the second experiment. On average, moving from one verified viewpoint to another resulted in an increase in confidence in 60% of cases, with an average increase in confidence of 3.65%. The size of increase/decrease fluctuations was quite large, but the average change in confidence shows an upward trend over these 20 trials.
Discussion Increases in confidence were largest when the start position did not align with the largest face of the object; next view selection was then able to find the largest face in the subsequent view. On the contrary, in situations where confidence decreased, the initial view coincided with the largest face of the object so the subsequent view moved away from this largest face. However, instances of reduced confidence should are not necessarily failures, as this still provided two high confidence identifications of the object. This block of experiments shows the planner is able to select a view to verify an object hypotheses with a good level of reliability. Set-up Environmental obstacles potentially act as visual occlusions when selecting a view. Next view planning must be able to recognise these in one view and assess their impact on the next. Without this ability the next view planner can select a view that would theoretically lead to a recognition confidence of 100%, but but this view may be blocked by another object, thus the actual recognition confidence is closer to 0%. To test this functionality the robot is presented with a scene which contained a target object and potential occlusion to block all or large parts of the object. Over 20 trials the robot was provided with a initial view of the object which allowed low confidence recognition. Each trial would be deemed a success if the object was recognised with a high level of confidence after the first movement.
Results Results showed that in 17 of the 20 cases (85%), the planner was able to select a pose which both avoided the occlusion and lead to recognition. Of the remaining trials, the selected pose allowed a view of the target object but the view was incomplete - being partially occluded by the obstacle.
Discussion In most cases the planner was able to account for an environmental occlusion and choose a best view pose that avoided it. In the remainder of cases the next view pose lead to a partially obscured view of the object. This is due to occlusion modelling being based on the information gained from a single frame during the initial view. If the potentially occluding object is occluded by the target object then the environmental analysis is detrimentally affected by partial observability of the obstacle.
Set-up A new view of an object can be taken to differentiate between objects that share similar features, which was the basis of this experiment. When presented with a view of an object that could belong to a target object or unrelated object, it would be best to disambiguate this view to decide if the target object has been found. To test this, two custom objects were used. Figure 5 shows the two modelled objects share a face from which the are almost indistinguishable, but are structurally different from other angles. In this experiment the robot will be placed with a view of the common face of these two objects and is expected to decide on a pose which increases the difference between the number of clusters recognised from each object. In all experiments the cuboid object(figure 5) was the target object. The robot was required to select a new pose to increase the strength of one correct hypothesis and decrease that of the incorrect one in 20 trials. The number of visible features that the recognition service matched to each candidate identity is a measure of the strength of that hypothesis.
Results Figure 6 shows that when presented with a scene in which one of two objects is present, the next best view planner can strengthen the hypothesis of the correct object and weaken that of the incorrect identity. Of the clusters recognised in the initial view, an average of 367.4 belonged to the correct object and 210.65 to the incorrect object. After one movement based on the selection of the next best view selection, the average available clusters for the correct object rose to 456.4 and 152.8 for the incorrect object.
Discussion This shows that selecting one new view can
increase the differentiation between two ambiguous objects
and lead to a reliable identification. This suggests much
simpler and less computationally expensive methods of
hypothesis differentiation that in Okamoto
The time for making one movement: from receiving a candidate identity and pose estimation to arriving at the next location is 1 minute 44 seconds. For two cycles the completion time jumps to 4 minutes 58. This is due to the large amount of data needed to compute each camera model at each pose. This is clearly a number that needs to be reduced and can be a subject for future work.
Experimental results show this is a strong next view algorithm for object recognition that can work reliably in cluttered, unpredictable environments.
In order to improve this solution further some areas can be enhanced to make it more robust and generalisable. Potential next view locations are currently set at a fixed distance from the candidate object; this can be a hindrance in certain topological layouts. Developing adaptable next view locations which, rather than test which of a fixed set of locations are in free space and therefore available, the potential locations should be instead generated in exclusively free space and then environmental and model analysis take place from there.
In adopting a greedy approach, this work selected only poses with the highest visible portion of the object; future work should focus on including a cost function to form a utility between movement distance and amount of the model which is visible.
7
A summary of the contributions of this project are shown below. The results shown in the previous section will be presented in support of these.
In summary, the aims of this study were to show that: 1. a method for analysing potential views using online aspect graphs by taking both into account: (1) occlusions, and (2) the visibility of features based on learned object models. 2. a next best view planner that selects a view based on the method above and an executive that accounts for dynamic obstacles during execution 3. a set of experiments which demonstrates (a) how robots can disambiguate objects which share a similar set of features, and (b) how the performance of object recognition can be improved by taking multiple views.
Results of experiment 1 show that the online aspect graphs analysis is able to verify candidates put forward by the recognition service with an accuracy of 82.5%; however this also showed that the next best view planner is also highly dependent on the accuracy of the pose estimation provided to it. Experimentation also showed that dynamic collision detection was able to eliminate unavailable poses; removing around 17 of 38 poses on average during every trial. We can also show that in 85% of cases, the planner was able to avoid visual occlusions in the environment, but this was heavily dependent on the visibility of the obstruction during the initial view. This was dually confirmed when two identical starting poses in experiment 1 & 3 both arrived at different final poses, as the best view when no occlusions are present was unavailable when clutter was introduced. Finally, we showed that the planner was able to decrease ambiguity in objects that have identical faces.
To achieve these aims we used online aspect graph
building and octree based visual occlusion detection. These
were new ways of approaching next best view planning and
showed that online aspect graph analysis for view planning
was possible and unlike offline examples
The work presented in this paper was successful in its aims. From online aspect graph building and collision detection to camera modelling and near real time occlusion analysis, the way this planner was designed allows it to be plugged into any robot using any model based recognition system, meaning this planner is available for a variety of robots that conduct object search in cluttered environments.
The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement No 600623, STRANDS.