In this work a scheme for scene-adaptive depth sensor network optimization is presented. We propose to fuse the knowledge inferred by the sensor network into a common world model while at the same time exploiting this knowledge to improve the perception and post processing algorithms themselves. Moreover, we show how our optimization scheme can be applied to improve the use cases of disparity estimation as well as people detection with multiple depth sensors.
Low cost commodity depth sensors are an emerging technology and are applied to a broad field of applications such as people detection and tracking, 3D reconstruction or emergency detection in an ambient assisted living context. However, depth sensor networks as well as modern vision algorithms have many parameters and require fine-tuned, scene-specific configurations to achieve optimal performance. Due to strongly varying scenes and changing conditions at run time it is very challenging to fine-tune those parameters manually in real world applications. To overcome the problem of scene-specific manual (re)configuration of depth sensor networks, we propose a scene-adaptive scheme which exploits the scene knowledge to improve perception and post processing vision algorithms. Our objective is not only to tune the given parameters but also to improve the vision algorithms, such as stereo block matching, detection or tracking by explicit exploitation of the scene knowledge, e.g. by building scene-specific object models. Therefore, we fuse the knowledge inferred from the sensor network into a common world model, representing our current context knowledge. This knowledge is then fed back to optimize sensor parameters and algorithms to improve the performance of a sensor network at run time. 2 2
The configuration of video sensor networks in the context of video surveillance
has been widely studied in the literature. In [
In this section we present a scheme for scene-adaptive sensor network optimization. The general information flow in a depth sensor network is depicted in Fig. 1 and separated into five different abstraction layers. The sensing layer sensing sensing sensing
sensor data postprocessing
sensor data postprocessing local 2D and 3D analysis local 2D and 3D analysis sensor data postprocessing local 2D and 3D analysis … … … data and knowledge fusion global data analysis world model scene and situation analysis contains low-level methods related to the raw sensor measurement such as synchronization, calibration and image acquisition. In the sensor data post processing layer depth estimation algorithms (e.g. stereo block matching), filtering and low-level feature extractors are included. Local data analysis covers high level vision algorithms which take the RGB-D data as input such as segmentation, recognition, object detection, local 3D object and scene reconstruction or
Scene-Adaptive Optimization Scheme for Depth Sensor Networks 3 tracking of objects. Based on the results of the data and knowledge fusion, the global data analysis layer includes methods which make use of the fused information of multiple sensors of the network. Examples are 3D scene reconstruction, 3D object localization and global tracking. The sensor network infers information about the scene across abstraction levels. Over time, information is fused into a common world model which represents the current scene knowledge. While a world model can be used to do e.g. scene and situation analysis, we use it to optimize the parameters of each individual sensor online and support the data analysis methods e.g. by building scene-specific object models gradually. 3.1
The employed knowledge representation within the world model has to be
expressive to solve the high-level task of the sensor network and the optimization
of the sensor network itself. The fusion layer might provide sensor data as well as
locally derived high-level knowledge and the world model therefore might need
to cover low-level data up to high level information. Taking these aspects into
account, several existing approaches for knowledge representations are qualified
to serve as world model. For most tasks and networks, a world model consisting
of geometric and semantic scene descriptions will be suitable. Geometric scene
knowledge thereby encompasses information about the objects contained in the
scene and their properties. This includes the object class (e.g. humans, furnature,
floor plan), the object location and orientation in a global world coordinate
system, dynamic properties e.g. a motion model, shape, material. Examples for such
a world model are object oriented world models [
Depth sensor networks involve multiple algorithms which leads to a large amount
of parameters. In this section we give an overview of parameters and methods
which are suitable for automatic scene-adaptive sensor optimization. We assume
that a suitable knowledge base (see section 3.1) exists and focus on algorithm
and parameter optimization. Following our layered scheme, we categorize the
optimization targets into three major categories, see Fig. 2. Sensing
parameters have a direct impact on the measurement quality. Parts of this category
have already been addressed. Auto exposure is state-of-the-art for decades in
consumer cameras, but sophisticated scene models [
In this section we show the applicability of our scheme on two exemplary use cases. 4.1
Our knowledge representation contains sensor knowledge in the form of a camera model and existing camera calibration parameters π, scene geometry using a ground plane assumption P (h) ⊂ R3 and a 3D morphable human surface model parameterized by β. Scene semantics are represented as segmentations of a single human sh and the ground plane sg in the image. Let Dπ(u) be a depth image computed using the estimated disparity values u from the image pair (I1, I2). Classical stereo algorithms estimate the disparity values u minimizing a cost function
E(u) = Ephotometric(u; I1, I2) + Ereg(u) , (1)
Scene-Adaptive Optimization Scheme for Depth Sensor Networks 5 where Ephotometric is the photometric error penalizing intensity deviation in the local neighborhood given u and Ereg regularizes the problem penalizing unlikely disparity values based on simple scene assumptions. We propose to employ a scene-adaptive optimization scheme reformulating (1) with
Eadaptive(u) = Ephotometric(u; I1, I2) + Emodel(u[sh], u[sg]; β, h) , (2) where Emodel uses our provided scene representation to measure the deviation from the estimated depth at the segmented pixel locations u[sh] and u[sg] to the explicit geometric scene representation consisting of the ground plane at height h and the human shape model parameterized by β. Scene-adaptive disparity estimation is then performed by estimating uˆ = arg minu Eadaptive(u). Eq.(2) can be extended in various ways, which proves the generality of the proposed approach by e.g. introducing a human motion model to enforce temporal consistency constraints. 4.2
The sensors have a top view on the scene and a significant overlap to each other. Additionally, we assume that the sensors are intrinsically and extrinsically calibrated in advance and that the common ground plane is known. We model the presence of a person on the ground floor as a discrete grid of Bernoulli random variables X = (x1, .., xn), xi ∈ {0, 1} where each xi maps to one specific ground plane grid location gi ∈ R2. Our goal is to infer the likelihood of a scene configuration X given current depth observations O = (O1, . . . , OC ) from C depth sensors. Applying Bayes’ theorem and assuming that the prior factorizes n as p(X) = Qi=1 p(xi) we get the posterior distribution n p(X|O) ∝ p(O|X) Y p(xi).
i=1
For this application we assume that the likelihood p(O|X) is given (see [
In the present work we have proposed a scheme for scene-adaptive optimization of depth sensor networks. We have given an analysis of relevant knowledge representations and categorized identified optimization targets. Moreover, we have (3)
J.Wetzel et al. exemplarily applied our scheme on the use cases of disparity estimation as well as people detection with multiple depth sensors. Future work will include the investigation of more use cases as well as proof of concept implementations.