We present an approach for generic object detection and segmentation in monocular videos. In this task, we want to segment objects from a background with no prior knowledge about the possible classes of objects which we may encounter. This makes this task much harder than the classical object detection and segmentation, which can be posed as a supervised learning problem. Our approach uses an ensemble of 3 different models which are trained by different objectives and have different failure modes and therefore complement each other. We demonstrate the usefulness of our approach on a custom dataset containing 18 classes of organic objects. Using our method, we were able to recover the classes of objects in a fully unsupervised way.
Separating generic objects from a background in monocular videos is a challenging task. We believe that this problem is essential to Computer Vision and, as such, gained an unproportionally small amount of attention from the research community. The ability to separate objects from the background would vastly simplify other tasks as it can be viewed as a kind of dimensionality reduction on relevant features.
In image classification, object separation prevents a classifier from learning spurious correlations, which could arise when a certain class is often captured on a particular background. Object separation from a background automatically restricts a classifier to consider only features directly tied to the class label, as opposed to features only correlated with it.
As the generic object separation is a rather nonstandard task and still vaguely defined – it is not clear what should be considered an independent object – we focus on a simplified setting in which a camera captures a single salient object. Our solution is an ensemble of three different models trained for different objectives.
Furthermore, we study the impact of the background removal on the clustering properties of the resulting representations. The representations are obtained by training a neural network in an unsupervised way. We are able to recover the categories of objects in a fully unsupervised way, using our custom dataset containing videos of organic things.
Our main contributions are: • We present an ensemble model that can separate objects from the background in monocular videos containing one salient object. The ensembled models compensate for each other failure modes. • We demonstrate the benefits of object separation by comparing the classification accuracy of objects with and without background using clustering.
Section 2 contains related work. Section 3 describes our approach for detecting and segmenting generic objects within monocular videos. Section 4 describes how the detected objects enable unsupervised discovery of object classes. In section 5 we describe our experiments and the dataset we test our approach on, and finally we provide a conclusion in section 6. move independently. Nonetheless, we consider this as our working definition because it allows us to make progress in generic object detection and segmentation. 2
Generic object separation is a largely an unexplored area,
and therefore similar works are scarce. Our approach
is most related to DINO method [
Lu et al. introduced an approach called CO-attention
Siamese Network (COSNet) [
The task of class discovery is marginally related to
current self-supervised approaches using large amounts of
unlabeled data such as [
Lastly, our approach for class discovery can be seen as a
version of clustering with constraints. It has been heavily
studied in the past, for example, by [
This section describes our approach to generic object detection and segmentation. By a generic object we mean an object of an unknown class. We use this term to distinguish it from classical object detection and segmentation, which can deal only with a concrete set of specified classes. Classical object detection and segmentation is much easier because it can be approached as a supervised learning problem on a dataset with annotated bounding boxes and segmentation masks. With generic objects, it is not that straightforward because it is not known in advance what kind of objects we will encounter at the test time.
Moreover, at first it may not be obvious how to define what should be considered as a separate object. One useful definition would be that an object is anything that can move independently from the rest of the environment. In this view, we can understand generic object segmentation as a way to factorise the visual stream into independent components. We need to mention that this definition does not cover all cases in which we would like to detect something as a separate object. Examples include buildings, letters on a sheet of paper, and other “entities” which can not 3.1
Our approach to the problem of generic object detection and segmentation is based on an ensemble of 3 models which are trained by different objectives. Even though each of these models has its own failure modes, together they constitute a robust ensemble. Concretely, we use one model trained for depth map prediction, one model trained for optical flow estimation, and one model trained for tracking of objects. Using the model for depth prediction, we can separate foreground objects based on depth, using the model trained for optical flow estimation, we can separate moving objects, and finally, using the tracker, we can verify the temporal coherency of our predictions. The tracker is initialised with a bounding box obtained from the predictions of the two other models in the frames where these predictions are most consistent. The following paragraphs provide a high-level description of these three models. For a more complete description of these models, see the respective publications.
Depth Prediction For the depth prediction, we use the
model introduced by Ranftl et al. [
Optical Flow Estimation For optical estimation, we use
the model introduced by Teed et al. [
To obtain training data for hand segmentation, we
downloaded the following 4 datasets: GTEA,
HandOverFace, GTEA_GAZE_PLUS, and EgoHands456. The
architecture of the model is UNet with timm_regnetty_160
[
Using freely available datasets for hand segmentation mentioned above, the trained model was not working well on our dataset, probably because of a large distribution shift (most of the images in these public datasets contained hands in front of the face or were captured in the interior).
To mitigate this problem, we used a simple trick to enlarge the training data with images of hands which are similar to the images in our target dataset. Concretely, we captured our hands from a similar viewpoint as in our dataset, and then used the same model for depth prediction to produce depth maps for every 10th frame within the video. Finally, we thresholded the predicted death maps to obtain reliable segmentation masks of hands. In this way, we obtained hundreds of labeled images of hands with minimal effort. After adding this dataset to the other datasets, we obtained an accurate model for hand segmentation.
We use this model to remove hands from the mask predicted by our ensemble. More precisely, we remove the hands from the outputs of the model predicting the depth map and the model predicting the optical flow before we initialise the bounding box for the tracker. In this way we obtain masks only for the object, ignoring the hands. 3.3
As mentioned above, we initialize the tracker with a bounding box obtained from the predictions of depth and optical flow maps within each frame. These predictions are two rectangular matrices of the same shape. We rescale the range of values to the interval between 0 and 1 and denote the final matrices by f1(x) and f2(x), respectively.
We first choose k frames where the predictions from
these two models are most consistent. To measure this
consistency, we devise the following heuristic. We first
detect edges using a Canny edge detector [
å i2Pixels(x) ee1i+e2i ; (1) where e1 and e2 are the two blurred edge maps from the two predictions and i indexes individual pixels. Next, we obtain the aggregated predictions for each pixel i by: y(x)i = e f1(x)i+ f2(x)i + 1
å jPixels(x)j j2Pixels(x) e f1(x) j+ f2(x) j : (2) We exponentiate the sum of the predictions from the two models because we want these predictions to interact superlinearly. We also subtract the mean of this value taken over all pixels within the image to make the aggregated predictions centered at zero. Therefore, the pixels where no object was predicted will contain negative values.
Once we have the aggregated predictions for the selected frames, we initialize the bounding boxes for the tracker. For this, we again devise a score which captures how well a given bounding box (bbox) covers pixels with high values (signifying that an object is present) and at the same time excludes pixels with low values. It has the following form:
å i2Pixels(x) bboxScore(bbox; y(x)) = isInBbox(i; bbox) y(x)i; (3) where the function isInBbox returns 1 if the pixel is not contained in the bounding box and 1 otherwise.
Finally, we optimize the coordinates of the bounding
box using CMA-ES [
1 e2 1 ; (4) The subtraction of 1 ensures that we get 0 when all three models predict 0. Thresholding and dividing by e2 1 insures that we obtain a value close to 1 when at least two models predict values close to 1. Finally, we obtain a bounding box for each frame using the same method as in the bounding box initialization (optimization using CMAES), i.e., minimizing the objective in Equation 1.
The whole process can be viewed as certainty propagation. We first select a few frames where the first two models agree on their predictions and from these the tracker propagates the certainty to other frames.
Our final task is a discovery of classes of objects within monocular videos. We view the generic object detection and segmentation as an intermediate step towards this goal. Therefore, we evaluate the usefulness of our approach on this target task. That is, we test how well are we able to recover classes of objects from images where the objects were segmented out by our approach. We compare it to the setup where we use the same algorithm for class discovery but where we use the original images with a background. We also mention that we do not require pixel-perfect segmentation masks as our goal is only to focus on the relevant parts of the image, so that the measured similarity between images will mostly reflect the similarity of objects and not of backgrounds. The next section describes our pipeline for the task of class discovery 4
The algorithm for class discovery was proposed in [
The advantage of the Louvain algorithm is that it needs no apriori knowledge of the number of clusters/communities.
The accuracy of our approach is measured in two ways. First, by counting how many times a video was assigned to an incorrect cluster. Second, whether the algorithm discovered all clusters. It is clear that the final accuracy mostly reflects the measured similarity between individual videos. Each video is represented by a sequence of images, but to compute the similarity, we ignore the ordering and treat the sequence as a set. The similarity between the two videos is computed in the following four steps: 1. Train an autoencoder using images from all videos to obtain a low-dimensional representation zi of each image xi. 2. In each video, select n representative frames which are not correlated, described in 4.2. 3. For each pair of videos, compute all pairwise similarities d(zi; z j) with cosine distance. 4. Finally, select the l most similar pairs of images and average their similarities to obtain the final similarity between two videos.
The intuition behind step 4 is that videos of similar objects may contain only a few frames where these objects are captured from the same angle or in the same situation. 4.2
Step 2 of similarity computation takes n representative frames. If we would simply use all frames from each video, the distribution of the dataset may end up skewed because some parts of a video may be more static than others. These static parts would produce many correlated frames. Therefore, the correlated frames need to be filtered out from a given video. We first test whether the subjective visual similarity of images can be captured by cosine similarity between their low-dimensional representations obtained in step 1 of the similarity computation. As can be seen in Figure 4, it captures the visual similarity well enough.
To extract n uncorrelated frames from each video, we run k-means clustering, where k = n, on the lowdimensional representations and take the most similar frame to every centroid of the resulting clusters. This simple heuristic produces uncorrelated images.
To conclude this section, if the low-dimensional representation of individual images obtained by the autoencoder reflects the similarity between the captured objects and not some other irrelevant factors, we may expect to obtain meaningful clusters. Moreover, note the benefit of creating the similarity graph of videos instead of individual images. All images in one video are automatically linked together. If a few images in 2 videos are similar, this similarity is propagated to other frames within the video, To test the algorithm for object discovery, we assembled a custom dataset of organic objects. The dataset contains 18 classes of organic objects, some of which are depicted in Figure 5. We have chosen organic objects because they naturally produce large variability between instances. For every class, we capture ten different samples from different viewpoints. The final dataset can be downloaded at the following address – github.com/Jan21/ Organic-objects-dataset.
Using our ensemble described in Section 3, we segment the object in every frame of each video. Using the resulting bounding boxes and segmentation masks, we crop each image and blur the background to suppress the distinctive features present in the background.
To obtain the low-dimensional representations used to filter out correlated frames and construct the similarity graph, we resize all images to a fixed resolution of 64 64 pixels and train a convolutional autoencoder. The autoencoder has five convolutional layers (16, 32, 64, 128, 256 filters with stride 2) and one fully-connected layer7 with dimensions 1024 ! 96. 8 After running community detection on the similarity graph, we inspected how many image bundles were assigned to the wrong component. Out of 173 videos, only 5 in the training set were assigned to the wrong component. The result of community detection on the constructed similarity graph is shown in Figure 6. Numerical results and comparison with clustering of non-segmented images are presented in Table 1. From the accuracy and number of discovered classes, it is clear that background removal created a significant accuracy difference of 56:1% and a difference in the number of correctly discovered classes. 6
In this contribution, we present a method for generic object detection and segmentation, which uses an ensemble of 7Image is downscaled to 2 2 times 256 filters ! 1024 input vector to the fully-connected layer.
8We also tried to extract representations by using VGG16 which was pre-trained on ImageNet. These representations better discriminated very similar objects (e.g., two types of red flowers). three different models trained by three different objectives. To demonstrate the effectiveness of the approach, we have created a custom dataset of organic objects. The dataset was used in our pipeline to remove background, create low-dimensional representations, and perform class discovery and classification. We have shown that background removal significantly increases the accuracy of the classification and the number of correctly discovered classes.
In future work, we plan to optimize our approach for speed, generalize it to work with videos containing multiple objects, and make the class discovery an online process that can discover new classes on-the-fly.
Pre-segmenting the objects Yes No Number of discovered classes 18 15