- . Paraphrasing Olivier Faugeras in the foreword of [1], making a robot see is still an unsolved and challenging task after several decades of research. The traditional research has been based on the geometric models of multiple views of a scene, estimating a sparse 3D map of the scene and the camera pose. Recent advances have led to fully dense and real-time 3D reconstructions. Also, there are relevant recent works on the semantic annotation of the 3D maps. This extended abstract summarizes the work of [2], [3], [4], [5], [6] in this direction; in particular using mid and high-level features to improve the accuracy of dense maps.
The inverse depth r for each pixel u in a reference image is estimated by minimizing the following energy E(r)
Z 3 E(r) = l0C(u; r) + R(u; r) + å P(u; r; rp )¶ u ; (1) p=1
C(u; r) is the photometric difference of each pixel u backprojected at an inverse depth r and projected into several overlapping images. R(u; r) is a regularization term –usually the TV-norm. Finally, the three terms in the sum 3 åp=1 P(u; r; rp ) correspond to the three mid and highlevel scene cues. For more details on each term and the optimization of the function the reader is referred to [5]. a) SUPERPIXELS (3DS): Superpixels are clusters of pixels that have been segmented based on their color and 2D distance. We will assume that such regions of homogeneous color will be planar. Specifically, we use the superpixel segmentation of [13].
We extract the planes P = (p1; : : : ; pk; : : : ; pq) that fit the superpixels by minimizing a function F of the geometric error ek of the reprojected contour of the superpixel k in the rth overlapping frame m q Pˆ = arg min å å F(ek) :
P r=1 k=1 (2)
The inverse depth r1 in equation 1 is the intersection of
the planes P with the backprojected ray from the pixel u.
For details, see [
b) DATA-DRIVEN PRIMITIVES (DDP): A data-driven 3D primitive [14] is a RGB-D pattern learnt from data. The visual part of the primitive should be discriminative enough to be detected on another images, and the depth pattern geometrically consistent.
The depth pattern is modelled by its normals and the RGB pattern by a HOG descriptor and a SVM-based classifier. At detection time, the inverse depth r2 for each pixel is extracted from the primitive normal and the depth from a multiview reconstruction. See [5] for more details.
c) LAYOUT (Lay.): The so-called layout [15] consists on the estimation of the rough geometry of a room and the classification of each pixel u into the classes wall, ceiling, floor and clutter.
We assume that the room is cuboid, so its model is composed of six planes. We estimate their normals using multiview vanishing points and their distances from a geometric reconstruction. From such layout, the inverse depth r3 is computed as the intersection of each pixel with the room boundaries if is is classified as that. If the pixel u is classified as clutter we consider that the depth is not predictable.
Figure 1 shows an illustrative view of our results in some
selected sequences from the NYU dataset [
Tables I and II show the median depth error of DTAM
[11], the sparse feature-based multiview stereo PMVS [
DPP
Layout sonable results even in the single-view case.
IV. CONCLUSIONS
In this abstract –and the associated papers [
Fig. 6 superpixels, 3rd column the data-driven primitives and 4th column the estimated layout. The 5th column is the ground truth d5e0pth from a RGB-D camera and the186th one our result. Notice the simi2la5rity between the latest two.
Median
Mean 16 40 25%−75% 14 20 showing th)eir complementary natu9r%e−.9F1%or mor)1e2 details on these and o(cmt3h0er experiments see [5]. (c10
m R R
O O8 SequeERR2n0ce DTAM [11] MeaPnMEVrSro[r1[7c]m(]%) ERR6Ours nicely, so a fusion o)f all of them improves the accuracy in a wide array of cas e(cm1s5.
R
O ACRK10NOWLEDGMENTS
R
E This research has been partially funded by projecta
5 DPI2012-32168 and DGA T04-FSE.
0
Bedroo1m01 (3DS) Bedroom1 (DDS) Bedroom1 (Lay.)
Bedroom2 (3DS) Bedroom2 (DDP) Bedroom2 (Lay.)(a) Bed7r.1oom1 15.8 7.0 (18%) 415.0 2 4.2
7.9 0 D5.T9AM PMVS LAY. 3DS DPP ALL 6.7 7.6 7.7 6.8
[1] R. I. Hartley and A. Zisserman, Multiple View Geometry in Computer 5.7 (22%) (b) Bedroom2Vision. Cambridge University(cP)reKss,itIcShBeNn: 0521540518, 2004.
Bedroom2 (All) [
KFitcihge.n 7(DDBP)ox and Whiskers plots showing t7h.7e depth error [d3]istAr.ibCuotnicohna, fWor. Hthuessainind,oLo.rMhoingtahn-op, aarnadllJa.xCisveeqrau, e“nMcaensh.attan and Kitchen (Lay.) 7.2 5.5 (20%) 5.7 ar#e4 t(DhDeP)same t4h2.3an the b28a8s.4e(l9i%n)e DT3A9.M1 and we
#4 (All) 20.9 only present results for DDP and Layout. As TABLE pIIr:evMioeuansldyespathide,rtrhorisfiosr aDcTlAeaMr,liPmMiVtaStiaonnd oofu3rsDiSn low-parallax sequences. (%) is the percentage of pixels –and in general of multiview geometry– and an
reconstructed by PMVS.
advantage of DDP and Layout, that give rea