We introduce a novel content-independent lter to protect privacy sensitive Regions Of Interest (ROI) in video surveillance sequences. An abstracted version of the original image is rendered such as the general appearance of shapes and colors is preserved, while obfuscating ne details carrying personal visual information. We use shapes and colors-aware, temporally coherent segmentation algorithm, combined with a color quantization and patch rendering step.
The increasing adoption of video surveillance systems has
led to a growing research interest in privacy protection
methods. A review of principles for privacy protection in video
surveillance can be found in [
Non-photorealistic rendering techniques described in the
literature achieve artistic e ects such as tooning, painting, or
sketching. For example, in [
We propose a new privacy protection lter inspired by results in image abstraction and non-photorealistic rendering elds. Our method is based on a boundaries and regionsaware segmentation algorithm, combined with a color quantization and patch rendering step, which transforms the original privacy sensitive ROI in a stylized and simpli ed version. While the general appearance of shapes and colors is preserved, to allow for people and actions detection tasks, identi cation details, such as faces and clothes traits, are obfuscated to render identi cation impossible.
Given a video sequence together with bounding boxes de ning the privacy sensitive ROIs, our algorithm proceeds in three steps. First, a segmentation algorithm divides the image in boundaries-aware patches. Second, the image is abstracted by replacing the pixels in each patch with a single color chosen from a palette. Finally, the abstracted image is rendered on top of the original frame to produce the nal output. If additional region annotations or background subtraction maps are available, the nal result can be further re ned by binary masking. Figure 1 shows an example of original and ltered frame.
The algorithm allows adaptation to the desired strength of privacy protection. By varying the number of patches, either globally or independently in certain regions, we can obtain di erent levels of abstraction. Our C++ implementation takes on average less than 0:5 seconds/frame for segmentation and about 0:3 seconds/frame for color quantization and rendering. 2.1
The intuition behind our privacy protection algorithm is
to render an abstracted version of the image by replacing
patches of pixels with a single color chosen from a palette.
To preserve intelligibility and visual pleasantness, we aim
for a region and boundary-aware process. Accordingly, we
adopt a segmentation procedure which divides the image in a
user-speci ed number N of arbitrarily shaped patches,
maximizing both their spatial and color consistency. A good
review of patch, or superpixel, segmentation methods, together
with the original description of the algorithm we adopted in
our work (SLIC ) can be found in [
The SLIC segmentation algorithm is based on K-means
clustering [
D = dLab + c
R
C N dxyt (1) Where N is the desired number of patches, (R; C) are the height and width of the ROI and c is a compactness parameter balancing the trade-o between spatial proximity and color similarity of the resulting clusters.
The output of the segmentation algorithm is a segmentation label map, where each patch is identi ed by a unique label. When additional annotation corresponding to speci c regions, such as a face, is available, we enforce a higher level of privacy protection by merging all the patches substantially overlapping with such region, to ensure proper obfuscation of shape and color details. 2.2
We keep a palette of a small xed number of colors (e.g.
8) progressively updated from the upcoming frames as
following: we rst compute the average color for each patch
and subsequently build the palette with a K-medoids
quantization [
To make the result visually more appealing and avoid ltering nonsensitive regions, we crop the abstract image with a foreground mask, inferred from the annotations and background subtraction maps, when available. Very sensitive regions such as face and skin are represented with an ellipse in the mask, to enforce maximum protection. The nal frame is computed as:
Iout = Ia \ [S(L; Mf )
T ] Iin \ [S(L; Mf ) < T ] (2) where Iout is the nal rendered image, Ia the abstract image, Iin the input image, L the segmentation labels map and Mf the foreground mask. S is a support operator which counts the number of foreground pixels for each given patch label. In such way, each patch is either fully rendered abstracted or fully rendered original in the nal image.
(a)
We applied the proposed method on selected sequences
from the PEViD dataset [
Score Average 0.563 0.502 0.576 0.665 0.385 0.56
Score Average 0.728 0.656 0.607 0.684 0.514 0.492 4.
In this paper we have proposed a novel privacy lter based on a region-aware segmentation algorithm combined with a color quantization and abstract rendering step. The result is a stylized image where the general intelligibility of shape and color is preserved, but the ne details of visual features are destroyed. 5.
This work has been conducted within the framework of the EC funded Network of Excellence VideoSense.