Instance-level image segmentation provides rich information crucial for scene understanding in a variety of real-world applications. In this paper, we evaluate multiple crowdsourced algorithms for the image segmentation problem, including novel worker-aggregation-based methods and retrieval-based methods from prior work. We characterize the different types of worker errors observed in crowdsourced segmentation, and present a clustering algorithm as a preprocessing step that is able to capture and eliminate errors arising due to workers having different semantic perspectives. We demonstrate that aggregation-based algorithms attain higher accuracies than existing retrieval-based approaches, while scaling better with increasing numbers of worker segmentations.
Precise, instance-level object segmentation is crucial for
identifying and tracking objects in a variety of real-world
emergent applications of autonomy, including robotics
To deal with these challenges, many have employed
heuristics indicative of crowdsourced segmentation quality to pick
the best worker-provided segmentation
As shown in Figure 1, quality evaluation methods for
crowdsourced segmentation can be classified into two categories:
Retrieval-based methods pick the “best” worker
segmentation based on some scoring criteria that evaluates the
quality of each segmentation, including vision information
Aggregation-based methods combine multiple worker segmentations to produce a final segmentation that is not restricted to any single worker segmentation. An aggregationbased majority vote approach was employed in Sameki et al. (2015) to create an expert-established gold standard for characterizing their dataset and algorithmic accuracies, rather than for segmentation quality evaluation as described here. 3
On collecting and analyzing a number of crowdsourced segmentations (described in Section 6), we found that common worker segmentation errors can be classified into three types: (1) Semantic Ambiguity: workers have differing opinions on whether particular regions belong to an object (Figure 2 left: annotations around ‘flower and vase’ when ‘vase’ is requested); (2) Semantic Error: workers annotate the wrong object entirely (Figure 2 right: annotations around ‘turtle’ and ‘monitor’ when ‘computer’ is requested.); and (3) Boundary Imperfection: workers make unintentional mistakes while drawing the boundaries, either due to low image resolution, small area of the object, or lack of drawing skills (Figure 3 left: imprecision around the ‘dog’ object).
Quality evaluation methods in prior work have largely focused on minimizing boundary imperfection issues. So, we first describe our novel aggregation-based algorithms designed to reduce boundary imperfections in Section 4. Next, in Section 5, we discuss a preprocessing method that eliminates semantic ambiguities and errors. We present our experimental evaluation in Section 6. At the heart of our aggregation techniques is the tile data representation. A tile is the smallest non-overlapping discrete unit created by overlaying all of the workers’ segmentations on top of each other. The tile representation allows us to aggregate segmentations from multiple workers, rather than being restricted to a single worker’s segmentation, allowing us to fix one worker’s errors with help from another. In Figure 3 (left), we display three worker segmentations for a toy example with 6 resulting tiles. Any subset of these tiles can contribute towards the final segmentation.
This simple but powerful idea of tiles also allows us to reformulate our problem from one of “generating a segmentation” to a setting that is much more familiar to crowdsourcing researchers. Since tiles are the lowest granularity units created by overlaying all workers’ segmentations on top of each other, each tile is either completely contained within or outside a given worker segmentation. Specifically, we can regard a worker segmentation as multiple boolean responses where the worker has voted ‘yes’ or ‘no’ to every tile independently. Intuitively, a worker votes ‘yes’ for every tile that is contained in their segmentation, and ‘no’ for every tile that is not. As shown in Figure 3 (right), tile t2 is voted ‘yes’ by worker 1, 2, and 3; tile t3 is voted ‘yes’ by worker 2 and 3. The goal of our aggregation algorithms is to pick an appropriate set of tiles that effectively trades off precision versus recall.
Now that we have modeled segmentation as a collection of worker votes for tiles, we can now develop familiar variants of standard quality evaluation algorithms for this setting.
This simple algorithm includes a tile in the output segmentation if and only if the tile has ‘yes’ votes from at least 50% of all workers.
Unlike MV, which assumes that all workers perform
uniformly, EM approaches infer the likelihood that a tile is part
of the ground truth segmentation, while simultaneously
estimating hidden worker qualities. In Section 6 we evaluate
an EM variant which assumes that each worker has a
(different) fixed probability for a correct vote. Details of this,
and more fine-grained variants can be found in our technical
report
The greedy algorithm picks tiles in descending order of the tiles’ ratios of (estimated) overlap area with the ground truth to (estimated) non-overlap area with ground truth, for as long as the (estimated) Jaccard similarity of the resulting segmentation continues to increase. Intuitively, tiles that have a high overlap area and low non-overlap area contribute to high recall with limited loss of precision. Since tile overlap and non-overlap areas, and Jaccard similarity of segmentations with ground truth are unknown, we use different heuristics to estimate these values. We discuss details of this algorithm and its theoretical guarantees in our technical report.
This algorithm picks the worker segmentation with the largest
number of control points around the segmentation boundary
(i.e., the most precise drawing) as the output segmentation
t4 As discussed in Section 3, disagreements often arise in segmentation due to differing worker perspectives on large tile regions. We developed a clustering-based preprocessing approach to resolve this issue. Based on the intuition that workers with similar perspectives will have segmentations that are close to each other, we compute the Jaccard similarity between each pair of segmentations and perform spectral clustering to separate the segmentations into clusters. Figure 2 (bottom) illustrates how spectral clustering divides the worker segmentations into clusters with meaningful semantic associations, reflecting the diversity of perspectives for the same task. Clustering results can be used as a preprocessing step for any quality evaluation algorithm by keeping only the segmentations that belong to the largest cluster, which is typically free of semantic errors.
In addition, clustering offers the additional benefit of preserving a worker’s semantic intentions. For example, while the green cluster in Figure 2 (bottom right) would be considered bad segmentations for the particular task (‘computer’), this cluster can provide more data for another segmentation task corresponding to ‘monitor’. A potential future work direction would be to crowdsource the semantic labels for the computed clusters to enable the reuse of segmentations across multiple objects to lower costs. Colors denote clusters with different worker perspectives.
We collected crowdsourced segmentations from Amazon
Mechanical Turk; each HIT consisted of one segmentation task
for a specific pre-labeled object in an image. Workers were
compensated $0.05 per task. There were a total of 46 objects
in 9 images from the MSCOCO dataset
Evaluation metrics used in our experiments measure how well the final segmentation (S) produced by these algorithms compare against ground truth (GT). We use the Jaccard score Jaccard (J) =
UIAA((SS)) , which accounts for the intersection area, IA = area(S \ GT ) and union area, U A = area(S [ GT ) between the worker and ground truth segmentations.
In Figure 5, we vary the number of worker segmentations along the x-axis and plot the average Jaccard score on the y-axis across different worker samples of a given size across different algorithms. Figure 5 (left) shows that the performance of aggregation-based algorithms (greedy, EM) exceeds the best achievable through existing retrieval-based methods (Retrieval). Then, in Figure 5 (right), we estimate the upper-bound performance of each algorithm by assuming that ‘full information’ based on ground truth is given to the algorithm. For greedy, the algorithm is aware of all the actual tile overlap and non-overlap areas against ground truth. For EM, the true worker quality parameter values (under our worker quality model) are known. For retrieval, the full information version directly picks the worker with the highest Jaccard similarity with respect to the ground truth. By making use of ground truth information (Figure 5 right), the best aggregation-based algorithm can achieve a close-toperfect average Jaccard score of 0.98 as an upper bound, far exceeding the results achievable by any single ‘best’ worker (J=0.91). This result demonstrates that aggregation-based methods are able to achieve better performance by performing inference at the tile granularity, which is guaranteed to be finer grained than any individual worker segmentation.
Intuitively, larger numbers of worker segmentations result in finer granularity tiles for the aggregation-based methods. The first row in Table 1 shows the average percentage change in performance between 5-workers and 30-workers samples. We observe that aggregation based methods typically improve in performance with an increase in number of workers, while this is not generally true for retrieval-based methods.
The second row in Table 1 shows the average percentage Jaccard change when clustering preprocessing is used. While clustering generally results in an accuracy increase, since the ‘full information’ variants are already free of semantic errors, we do not see further improvement for these variants.
Retrieval-based Aggregation-based Algorithm num pts worker* MV EM greedy greedy* Worker Scaling -6.30 2.58 2.12 1.78 2.07 5.38 Clustering Effect 5.92 -0.02 2.05 0.03 5.73 0.283 We identified three different types of errors for crowdsourced image segmentation, developed a clustering-based method to capture the semantic diversity caused by differing worker perspectives, and introduced novel aggregation-based methods that produce more accurate segmentations than existing retrieval-based methods.
Our preliminary studies show that our worker quality models are good indicators of the actual accuracy of worker segmentations. We also observe that the greedy algorithm is capable of achieving close-to-perfect segmentation accuracy with ground truth information. Given the success of aggregationbased methods, including the simple majority vote algorithm, we plan to use our worker quality insights to improve our EM and greedy algorithms. We are also working on using computer vision signals to further improve our algorithms.