The steady growth in the adoption of video surveillance systems emphasizes the need for privacy protection techniques. In this paper, we present a method inspired from image abstraction and non-photorealistic rendering elds for creating privacy protection lters. The e ectiveness of the proposed lter has been demonstrated by assessing the intelligibility vs. privacy vs. pleasantness in a subjective evaluation framework using di erent videos.
In recent years, a steady growth in the adoption of
digital video surveillance systems for monitoring buildings and
public spaces has been observed. These systems raised many
concerns related to the privacy rights of the subjects being
monitored [
Current surveillance systems either do not implement any mechanism for privacy protection, or they apply traditional techniques such as masking, Gaussian blurring, and pixelization. The lack of appropriate methods to detect regions of the image which contain privacy sensitive information and to evaluate the amount of privacy protection required in a speci c scenario often causes failure in either minimizing the intrusion of the surveillance system or goes against the purpose of the surveillance itself.
Recently, much e ort has been devoted on this eld. One
big challenge in de ning privacy protection policies for video
surveillance applications is the identi cation of the correct
trade-o between intelligibility of the video, which should
be adequate to the monitoring tasks, and privacy protection
itself. Consequently, a number of recent studies have been
conducted to propose more adequate systems for privacy
protection. Among the proposed techniques, non-photorealistic
rendering ones achieve artistic results. For instance, in [
In this paper, we propose a privacy protection lter
inspired from [
The proposed algorithm essentially consists of applying a color-based segmentation in order to divide any region of interest (RoI) into di erent patches of di erent colors. Then, the RoI is abstracted by replacing the pixels belonging each patch by one single color. The privacy sensitive RoIs, are de ned in our case by the bounding boxes provided by the annotation step. These regions can be further re ned if additional foreground segmentation masks are available. An illustration of the proposed protection lter is shown in Figure1.
(a) Original frame (b) Filtered frame
To process videos, we apply a color-based segmentation
algorithm in the bounding boxes de ning the privacy
sensitive regions of interest. This algorithm proceeds as follows:
First, for each RoI, we de ne every pixel by its color in RGB
space (ri; gi; bi) and its spacial coordinates (xi; yi). So,
every pixel could be presented as this vector [ri; gi; bi; xi; yi],
where is a coe cient used to adjust the balance between
the spatial proximity and the color similarity of the
resulting clusters. is set according to the frame size and the
number of clusters. After that, the pixels belonging to that
RoI are divided into di erent groups using k-means
clustering algorithm [
Once the clustering process is nished, the pixels in the same patch are replaced by one color from this patch. This color is chosen to be the color of the centroid of every patch. 2.2
Since our primary goal is to keep the main information to enable detecting persons and actions while obscuring the sensitive data to protect personal privacy, our actual results obtained by applying k-means clustering in RoI have to be further processed. At this stage, we have as input the results of the clustering and the patch rendering steps applied on the whole RoI, which is in our case the bounding box of the annotation. These results are re ned using the foreground masks. The nal abstracted RoI is computed as: Iout = Iin \ [S(L; F g)
T ] I \ [S(L; F g) < T ] (1) where Iout, Iin, and I denote the nal abstracted RoI, the abstracted RoI resulting from the previous step, and the original RoI, respectively. Also, L is the segmentation label map, F g is the foreground mask, and S is a support operator that counts the number of foreground pixels of each patch label.
By this way, each patch is either fully rendered abstracted or fully rendered original. Finally, the re ned and ltered RoIs are added to the original frame by replacing the original ROIs. This step of binary masking is important to make the results visually more appealing and to avoid ltering nonsensitive regions.
We tested our proposed approach on di erent video
sequences from PEViD dataset [
In Table 1, we report our results, compared to the median score of all participants to the challenge. These results are the average of three subjective evaluations.
It is important to mention that our proposed lter achieved good results for both intelligibility and pleasantness, however the privacy protection level was low. This limitation of privacy protection can be mainly explained by the fact that our approach did not enforce maximum protection for sensitive regions such as faces. We would expect higher protection level if this point has been taken into account. 4.
In this paper, we present our approach for privacy protection lters based on color segmentation and binary masking. The resulting abstracted ( ltered) image has the advantage of resembling to the original image in the general shape and color appearance, while destroying ne details. 5. 6.
This work has been conducted within the framework of the EC funded Network of Excellence VideoSense.