MediaEval 2015 Drone Protect Task: Privacy Protection in Surveillance Systems Using False Coloring ∗ Serdar Çiftçi1 , Pavel Korshunov2 , Ahmet Oğuz Akyüz1 , Touradj Ebrahimi2 1. Department of Computer Engineering, Middle East Technical University, Ankara, Turkey {sciftci, akyuz@ceng.metu.edu.tr} 2. Multimedia Signal Processing Group, Ecole Polytechnique Fedéralé de Lausanne, Switzerland {pavel.korshunov, touradj.ebrahimi@epfl.ch} ABSTRACT RBS DEF This paper describes privacy protection method based on a LOCS false coloring approach for Drone Protect Task of MediaEval 2015. The aim is to obscure regions of a video that are pri- Figure 1: Color scales used in this study. vacy sensitive without sacrificing intelligibility and pleasant- ness. False coloring transforms the original colors of pixels using a color palette into a different set of colors in which implement and has little computational overhead, thus, is private information is harder to recognize. The method can applicable for real-time system [4]. False coloring preserves be applied globally to an entire frame of the video or to privacy without compromising pleasantness and intelligibil- a specific region of interest (ROI). The privacy protected ity. Furthermore, its output can be reversed to obtain a output is expected to remain pleasant, and when needed, a close approximation of the unprotected information. close approximation of the original input can be recovered. The proposed method was applied to mini-drone video Benchmarking evaluations on the mini-drone dataset show dataset [2] provided by the organizers of MediaEval 2015 promising results, especially, for intelligibility and pleasant- Drone Protect Task [1]. The dataset contains short clips ness criteria. captured by a surveillance mini-drone. Each clip is anno- tated by human observers to mark the sensitive ROIs and the privacy level for each ROI. 1. INTRODUCTION Video surveillance systems are being widely used to pro- tect the safety of public and private perimeters. An ideal 2. FALSE COLOR BASED PRIVACY PRO- surveillance system should balance well between two objec- TECTION tives: efficiently execute a security task (intelligibility) and The main idea in false color based privacy protection is in carefully preserve subjects’ privacy (privacy). The most transforming colors of pixels in a frame such that the pri- commonly used methods to protect privacy such as blur- vate information becomes unrecognizable while the impact ring, masking, and pixelization do not achieve a good bal- on intelligibility is kept as small as possible. Previous work ance. For this reason, second generation solutions such as on false coloring has demonstrated the applicability of such scrambling [5], warping [6], and in-painting [3] are proposed. an approach for privacy protection against both human ob- However, these solutions have their own weaknesses such as servers and automatic face recognition algorithms [4]. dependency on compression and format, visually disturbing This algorithm first converts a color frame into grayscale. results, negative impact on intelligibility, and irreversibility. The pixel intensities of the grayscaled frame are then used as Furthermore, most methods strongly rely on efficient com- keys to a look-up a table that represents a color palette. Op- puter vision algorithms for instance when regions that re- tionally, the grayscale frame can be compressed or quantized quire privacy protection must be automatically detected (e.g., to further distort the visual information prior to table look- faces, license plates, etc.). However, computer vision algo- up. The pixel values of the original frame are then replaced rithms are known to fail at times. If a sensitive region is by the values from the table. This algorithm can be applied missed, even in a single frame, it will severely compromise on an entire frame or on one or more ROIs. The strength of privacy. Therefore, there is a need to develop robust and ef- the protection is controlled by the color distribution of the fective algorithms for privacy protection that can efficiently selected color palette (Figure 1). cope with situations when computer vision algorithms fail. The protected frames can be reversed to obtain a close ap- We propose to protect privacy via false coloring, which proximation of the originals by performing an inverse table does not rely on computer vision and can be applied either look-up. However, due to the initial grayscale conversion, on an entire frame or a region of interest. It is simple to the recovered frames will be in grayscale. Also, if the look- ∗Currently with Idiap research institute (Switzerland) up table contains duplicate values, full recovery may not be possible due to the initial many-to-one mapping. Finally, the reversion is only possible if one knows the properties of Copyright is held by the author/owner(s). the color map used during protection. Thus, security can be MediaEval 2015 Workshop, Sept. 14-15, 2015, Wurzen, Germany enhanced by utilizing a custom color palette. (a) (c) (e) (b) (d) (f) Figure 2: False color results, a) original frame, c) protected frame, e) recovered frame. 3. EVALUATION RESULTS 4. CONCLUSION We applied false coloring to the annotated ROIs of the In this paper, we described a simple and effective method provided mini-drone dataset [2] using the following color for protecting privacy using false coloring. Benchmarking maps: Radiance default (DEF) for high, rainbow color scale evaluations indicated high preference for the pleasantness (RBS) for medium, and linearized optimal color scale (LOCS) and intelligibility of this method, whereas it was found to for low privacy regions. This selection was motivated by the be less effective for preserving privacy. Future work will effectiveness of each color map for privacy protection as de- investigate designing custom color scales to improve privacy termined by earlier work [4]. protection and the quality of reversibility while enhancing A sample result is shown in Figure 2, where (a) represents security. an original video frame, (c) its privacy protected version, and (e) its recovered version. The close-up views can be observed 5. ACKNOWLEDGEMENTS in the bottom row. It can be noted in (c) and (d) that the The work was conducted in the framework of FP7 NoE face of the individual is represented in the DEF color scale, VideoSense and TUBITAK project number 114E445. whereas his body is represented in the RBS color scale. The vehicle, on the other hand, is represented in the LOCS color scale based on expert annotations. 6. REFERENCES The recovered ROIs shown in (e) and (f) do not con- [1] A. Badii, P. Korshunov, H. Oudi, T. Ebrahimi, tain color information and have some artifacts near the ROI T. Piatrik, V. Eiselein, N. Ruchaud, C. Fedorczak, J.-L. boundaries. This is due to the compression of the protected Dugelay, and D. F. Vazquez. Overview of the frames. As the compression works at block rather than pixel MediaEval 2015 drone protect task. In MediaEval 2015 level, the false colored pixels affect the colors of the neighbor- Workshop, Wurzen, Germany, Sept. 2015. ing pixels and are not corrected during the inverse look-up. [2] M. Bonetto, P. Korshunov, G. Ramponi, and The MediaEval benchmarking results reported in Table 1 T. Ebrahimi. Privacy in Mini-drone Based Video show that our intelligibility and pleasantness scores are above Surveillance. In Workshop on De-identification for the average of all submissions whereas the privacy level score privacy protection in multimedia, 2015. is below the average. This can be explained by the fact that [3] S.-C. Cheung, M. Venkatesh, J. Paruchuri, J. Zhao, and false coloring is a point operation and, unlike most other T. Nguyen. Protecting and managing privacy methods, it does not introduce structural distortions. Nev- information in video surveillance systems. In Protecting ertheless, privacy level could be improved by using custom Privacy in Video Surveillance, pages 11–33. 2009. color palettes that are better tailored to privacy protection. [4] S. Çiftçi, P. Korshunov, A. O. Akyüz, and T. Ebrahimi. Using false colors to protect visual privacy of sensitive Table 1: Evaluation results where FC and AVG respectively content. In SPIE Human Vision and Electronic stand for false color results and the average of all submis- Imaging XX, pages 93941L–93941L–13, 2015. sions. Expert and Naı̈ve’s are participant groups which rep- [5] F. Dufaux and T. Ebrahimi. Scrambling for Privacy resent the people conducting research on visual privacy pro- Protection in Video Surveillance Systems. IEEE Trans. tection and naı̈ve observers. on Circuits and Systems for Video Technology, vol. 18(no. 8):1168–1174, 2008. Privacy Intelligibility Pleasantness [6] P. Korshunov and T. Ebrahimi. Using Warping for FC AVG FC AVG FC AVG Expert 0.39 0.49 0.76 0.59 0.73 0.60 Privacy Protection in Video Surveillance. In 18th Naı̈ve 0.34 0.48 0.75 0.58 0.75 0.61 International Conference on Digital Signal Processing Average 0.365 0.49 0.755 0.59 0.74 0.60 (DSP), 2013.