Robots that detect changes in the environment can attain better context awareness and increased autonomy. In this work, a spatial change detection approach is presented which uses a single fixed depth camera to identify environment changes caused by human activities. The proposed method combines hand tracking and the difference between organized point clouds. Bimanual movements are recorded in real-time and encoded in Gaussian Mixture Models (GMMs). We show that GMMs enable change detection in presence of occlusions. We also show that the GMM analysis narrows down potential salient regions of space where manipulation actions are carried out. Experiments have been performed in an indoor environment for object placement, object removal and object repositioning tasks.
In this work, a method for spatial change detection is presented that identifies
environment changes due to human activities. The experimental setup includes
a single fixed depth camera (Kinect V2). Detection of the salient regions of the
environment, where human actions have been carried out, is achieved by
computing the difference between two organized point clouds, that are acquired at
the beginning and at the end of each experimental session. Moreover, bimanual
movements are tracked in real-time and encoded using GMMs. GMM analysis
enables change detection in presence of occlusions and it reduces the number
of false positives. Spatial change detection has mainly been investigated with
object-based approaches by exploiting cameras or depth sensors [
GMM-based spatial change detection
The proposed approach operates on two depth frames acquired automatically from a fixed Kinect V2, when no human motion is detected by the skeletal tracker: one (frame P ) when the user has not yet entered the area, and the other (frame N ) after the user left the scene. A difference depth image Duv is computed (Eq. 1) given the one-to-one pixel correspondence in the two frames at coordinates (u, v).
NaN if |Nuv − Puv| ≤ Th Duv = Nuv if Puv − Nuv > Th Puv if Nuv − Puv > Th (1)
Iuv = (NaN if |Nuv − Puv| > Th
Nuv otherwise (2) Th is a threshold set from to the noise model of the sensor. According to Eq. 1, in case of a significant change between the depth values the nearest point is selected. Indeed, if the user performs an object placement task in direct sight of the camera depth image N contains relevant information about the newly placed object, while depth image P contains information about the background. The opposite occurs if an object is removed in direct sight of the camera. An invariant image is also computed as in Eq. 2, containing the pixels that do not change significantly. Both Duv and Iuv are converted into point clouds called difference point cloud and invariant point cloud respectively. 2.2
Motion tracking of both hands is performed in real-time during the execution
of the experiment using the Kinect V2 skeletal tracker. The trajectory of each
hand is represented as a set of 3D points with time stamp, i.e. {xk, yk, zk, tk}.
The proposed approach first generates two separate GMMs, ML and MR, one for
each 4D hand trajectory using Expectation Maximization (EM). The number of
Gaussians of ML and MR was chosen to minimize the BIC index [
Information extracted from depth image processing and GMMs is exploited to compute regions of interests (ROIs) corresponding to human activities as illustrated in Fig. 1. First, a set of salient spheres of fixed radius r is generated, each centered at the mean position of a salient Gaussian component in M . Then, a salient point cloud is computed as the part of the difference point cloud inside any of the salient spheres. Clusters of connected components in the salient point cloud (defined as salient clusters) are extracted. Salient clusters are likely to represent ROIs for manipulated objects in direct sight of the camera, i.e. objects that have been placed in the environment, moved or removed, as both human trajectory analysis and the difference point cloud agree. Each salient cluster generates a spherical ROI, centered at the cluster centroid with radius equal to the distance between the center and the farthest point of the cluster.
Salient spheres that contain at least TI points of the invariant point cloud are also added to the list of the regions of interest. In fact, such spheres are likely to represent regions of space where user activity was detected from motion trajectory analysis although the Kinect depth frame could not locate any changes due to occlusions. Points of the invariant point cloud within a distance of r′ from any salient cluster centroid (rejection region) are not counted to prevent duplicate ROI detections. The proposed approach was evaluated by different users in an environment of size 5 by 4 meters. In each experiment the user entered the workspace, performed multiple manipulation actions and then left the scene. Fig. 2 shows an example experiment. Results of the spatial change detection algorithm are shown in Fig. 3. Fig. 4 illustrates, in a simpler experiment for clarity, the benefit of the GMM trajectory analysis when an action is performed in direct sight: the sensor noise at the depth image borders is filtered out. A quantitative evaluation was carried out on a dataset consisting of three experiments, with 10 trials for each experiment. Experiments of the first type consist of a sequence of three actions, two performed in direct sight of the sensor and one in an occluded region. The second type involves two relevant actions, both in occluded areas. Experiments of the third type consist again of three actions, all in direct sight of the sensor. Experiments have been performed with r = 80 cm, r′ = 40 cm, Th = 10 cm and TI = 50. Results are summarized in table 3. Precision and recall are above 87% for actions in direct sight of the sensor and above 80% for actions in occluded regions.