Many real-world applications require accurate segmentation of images into semantically-meaningful regions. In many cases one needs to obtain accurate segment maps for a large dataset of images that depict objects of certain semantic categories. As current state-of-the art methods for semantic image segmentation do not yet achieve the accuracy required for their use in real-world applications, they are not applicable in this case. The standard solution would be to apply interactive segmentation methods, however their use for a large number of images would be laborious and time-consuming. In this work we present an online learning framework for interactive semantic image segmentation that simpli es processing of such image datasets. This framework learns to recognize and segment user-de ned target categories using the ground truth segmentations provided by user. While the user is working on ground truth image segmentation, our framework combines online-learned category models with the standard stroke-propagation mechanisms that are typically used in interactive segmentation methods. Our implementation of this framework in a software system has speci c interface features that minimize the required amount of user input. We evaluate the implementation on several datasets from completely di erent domains (Sowerby dataset containing 7 di erent semantic categories, sheep & cows dataset containing 3 categories, and 6 di erent ower datasets with 2 categories each). Usage of our system requires substantially less user e ort compared to the traditional interactive segmentation methods.
Many applications, such as aerial and space image processing, defect detection,
and medical imaging require accurate segmentation of large image datasets into
some semantically-meaningful zones. Despite the substantial progress made,
current state-of-the art methods for automatic semantic image segmentation [
In this work we consider the case when one needs to perform segmentation of
a large image dataset into a number of semantically-meaningful categories. We
aim at developing a general framework that would work with any user-de ned
target categories and learn these categories from the user input as semantic
segmentation methods do. On the other hand, we want to give the user as much
control on the segmentation results as interactive image segmentation methods
provide. Our main goal is to minimize user e ort while allowing her to produce
accurate image segmentations. Most existing methods for interactive image
segmentation work with a single image [2{4], which limits their power. For example,
segmentation of an image from MSRC dataset with our system takes less than
a minute compared to 15{60 minutes for manual annotation [
Related task of inducing segmentation from example was looked at [23]. In this approach a non-parametric model of the provided training pair is constructed by selecting a set of patch-based representatives inside each labeled region in the training image. These representatives are used to quantify the degree of resemblance between small regions in the input image and the labeled regions in the training set.
Adaptive learning of object detection was considered in [
The paper is organized as follows. Next section describes the general workow of our system and our semantic segmentation algorithm. In Section 3 we describe the online random forest. Section 4 describes the experiments, and the last section is left for conclusions.
Suppose one needs to process a large image dataset and obtain accurate segmentation of the objects of certain categories. In framework, a user examines images from the dataset in sequence. Each image is presented as a set of superpixels (Section 2.2). The rst image is segmented manually: a user should label each superpixel with one of the category labels. The newly-obtained labelling is used to update the appearance model. When the user opens one of the consequent images, segmentation is performed automatically using the current appearance model. Then the user may correct mistakes of the automatic method by changing superpixel labels. Each time the user approves the (possibly corrected) segmentation result, the system learns from the newly obtained examples of object categories and background. As training goes, user time spent on correction of category map reduces, thus the rate of image labelling increases.
Our implementation provides a set of tools to simplify the process of error correction for the user. The brush tool is used to modify superpixel labels. It is possible to change labels of groups of neighbouring superpixels by choosing the appropriate brush size. Each time the user applies it, the system also updates the global labelling using the newly observed labels as context. Therefore, one brush stroke usually changes labels of a large number of pixels. We use hierarchical clustering to obtain superpixels, so the user can switch between di erent scales of superpixels and choose appropriate scale for correction of errors in the segmentation. We also provide a user with a set of sliders, each one controls the trade-o between false positive and false negative rates of one object category. The sliders help to signi cantly reduce the amount of manual work in the beginning of the image set processing, when few images have been seen, and the classi ers are likely to be biased towards some categories.
The output of the most time-consuming operations (such as over-segmentation and feature extraction) can be cached, so in practice those operations are performed o ine, before the user starts working with the system. We use e cient methods for inference of the optimal segmentation and learning the appearance models of object categories (see Sections 3 and 2.2), thus a user gets immediate response from the system. 2.2
We obtain pixelwise object segmentation by assigning category labels to a set
of superpixels obtained by clustering the joint color and coordinate space with
mean-shift algorithm [
We use a simple pairwise conditional random eld (CRF) that allows e cient inference. The vector of superpixel labels c = fcig is determined as the one that minimizes the following energy function:
E(c) =
X i (cijI) i
X
(i;j)
ij (ci; cj jI) ;
Our variant of online random forest1 builds a set of Hoe ding trees [
In Breiman's Random Forest [
N k where the rst term sums the appearance potentials of individual superpixels, the second sum is over the neighbouring pairs of superpixels.
The unary potential for assigning the object category ci to the i-th superpixel in the image I is computed as i (cijI) = log p (cijSi; I) + (ci); where p (cijSi; I) is the probabilistic output of the online random forests (Section 3) for ci-th class on the superpixel Si. The second term (ci) is the slider value that can be treated as the prior that prefers some categories over the others.
Pairwise potentials consist of the two terms: ij (ci; cj jI) = (ci; cj jI) + (ci; cj ) : The rst term is the inverse of the boundary strength provided by the mean-shift segmentation. The second term corresponds to a fraction of neighbouring superpixels of the classes ci and cj among all neighbouring superpixels in the images seen so far. There are e cient algorithms for minimization of the energy (1), so inference can be performed every time when the user moves a slider to change the trade-o between false positives and false negative rates. 3
(1) (2) As N goes to in nity, the distribution of K converges to the Poisson distribution with the parameter equal to 1: K expk(! 1) . Therefore online bootstrapping can be performed as follows: for each base model, choose each example K Pois(1) times and update the base model accordingly. To diversify the trees in our variant of online random forest each tree operates with a random subset of features. 1 http://graphics.cs.msu.ru/en/science/research/machinelearning/bolt
Input: Example (x; y) For each base model hm = h1,. . . ,hM
Set k = P oisson(1) Do k times
hm = U pdate tree (x; y)
Return updated fh1, . . . ,hM g
As long as Hoe ding trees can handle multiclass classi cation, our Online Random Forest naturally performs multiclass classi cation without any change in the algorithm.
Handling imbalanced classes. It was proven [
Image datasets. In the rst experiment we used Sowerby dataset that contains 100 images of urban scenes. The goal in this experiment was to perform accurate multi-zone segmentation into 7 object categories provided in the ground truth annotation of this dataset.
In the second experiment we used a subset of the MSRC dataset2 composed of 60 images of cows and sheep. In this experiment we considered a 3-zone segmentation problem where the goal was to segment cows and sheep from background.
In the third experiment we used a subset of 17- ower dataset3. We considered 6 di erent owers (da odil, tigerlily, daisy, fritillary, pansy, sun ower), and 80 images of each ower. The goal in this experiment was to segment each ower from the background.
Measuring usability of the system. The typical sequence of user actions to segment an image in our system is the following. The user starts with tuning the sliders to adjust the false positive vs. false negative rates, and then corrects the segmentation errors using brush tool. In most cases the optimal strategy for error correction is to start with xing the errors in the coarsest scale of superpixels and then proceed to more detailed scales.
To quantify the user input we have implemented a robot-user that emulates the actions of a human user working with the system. Given the initial image 2 http://research.microsoft.com/en-us/projects/ObjectClassRecognition/ 3 http://www.robots.ox.ac.uk/~vgg/data/flowers/ segmentation, the robot rst nds the optimal values of (ci) for all object classes (i.e. optimal position of the sliders). For that we minimize the total area of misclassi ed superpixels using Nelder{Mead algorithm. Then we count the number superpixels that need to change labels in order to obtain correct segmentation result. We start by correcting the errors at the coarsest scale and proceed to more detailed scales of superpixels. The resulting metric characterizes overall amount of user input required to obtain correct result using our system, and we refer to it as usability metric.
To measure the gain provided by learning the appearance models of object categories, we compared two values of usability metric. First we computed the usability metric for the case of fully manual image segmentation using our brush tool, i.e assuming that all superpixels are initially labelled as background. Second, we computed the usability metric for our semantic segmentation framework.
The results of this experiment for Sowerby and sheep & cows datasets are shown in Figure 3 (a, b). The green lines show the results in fully manual case, and the blue lines show the results for our framework. The use of automatic segmentation helps to signi cantly reduce required amount of user input compared to performing fully manual segmentation.
To measure the gain of online learning we looked at the behaviour of the plots
of usability metric with respect to the the total number of images processed.
As the values of usability metric vary signi cantly for each particular image,
we computed the average over 9 subsequent images to estimate the long-term
trends of usability metric. The e ect of online learning is most clearly visible for
the owers image datasets (Figure 3 (c)), where the total number of superpixels
that require relabelling tends to decrease over time. For Sowerby and sheep &
cows image datasets this metric decreases also decreases, but more slowly.
Measuring the time. We compared the time required from a human user
to obtain high-quality image segmentation with our system and with GrowCut
interactive segmentation tool [
We have presented a framework for interactive semantic image segmentation that is based on online learning. The experiments show that online learning of object appearance models helps to signi cantly reduce user input required to obtain accurate image segmentation.