=Paper=
{{Paper
|id=Vol-1436/Paper7
|storemode=property
|title=Overview of the MediaEval 2015 Drone Protect Task
|pdfUrl=https://ceur-ws.org/Vol-1436/Paper7.pdf
|volume=Vol-1436
|dblpUrl=https://dblp.org/rec/conf/mediaeval/BadiiKOEPERFDV15
}}
==Overview of the MediaEval 2015 Drone Protect Task==
https://ceur-ws.org/Vol-1436/Paper7.pdf
Overview of the MediaEval 2015 Drone Protect Task
1 2 3 4 5 6
Atta Badii , Pavel Koshunov , Hamid Oudi , Touradj Ebrahimi , Tomas Piatrik , Volker Eiselein ,
7 8 9 10
Natacha Ruchaud , Christian Fedorczak , Jean-Luc Dugelay , Diego Fernandez Vazquez
1,3. {atta.badii, h.oudi}@reading.ac.uk, 2 , 4. {pavel.korshunov, touradj.ebrahimi}@epfl.ch,.5. t.piatrik@qmul.ac.uk,
6. eiselein@nue.tu-berlin.de, 7.Natacha.Ruchaud@eurecom.fr , 8. christian.fedorczak@thalesgroup.com,
9. jean-luc.dugelay@eurecom.fr , 10. dfvazquez@isdefe.es
ABSTRACT car park. The contents of the videos were grouped into three
This paper presents an overview of the Drone Protect Task (DPT) categories: Normal, Suspicious and Illicit behaviour. The videos
of MediaEval 2015, its objectives, related dataset, and evaluation in the Normal category involved subjects performing common
approach. Participants in this task were required to implement a social behaviours in the car park such as entering or leaving a car.
privacy filter or a combination of filters to protect various The Suspicious category included loitering, taking a picture of
personal information regions in the video sequences provided. parked cars and other questionable behaviours. On the other
The challenge was to achieve an adequate balance between the hand, Illicit behaviour included Actors stealing a car, leaving a car
degree of privacy protection, intelligibility (how much useful unattended, parking the car in forbidden areas or fighting.
information is retained after privacy filtering), and pleasantness The actors in the videos, carry specific items and so could
(how minimal were the adverse effects of filtering on the potentially reveal their identity and may therefore need to be
appearance of the video frames). The evaluation methods for this privacy-filtered appropriately. For example, the actors are
task include subjective evaluation by those working in the video featured carrying backpacks, umbrellas, wearing scarves, and
surveillance sector and also by naïve viewers. performing various actions, such as fighting, stealing, loitering, or
simply walking. Actors may be at a distance from the camera or
1. INTRODUCTION near the camera, making their faces appear with varying size and
The number of drones deployed for civil applications and quality. Despite the use of advanced stabilisation techniques for
other non-military uses such as journalism, recreation, public the Camera on board the drone, the drone maneuvering and the
safety, and precision agriculture is increasing. In particular, the variable conditions outdoors still led to some jitter effects in some
deployment of the highly mobile and versatile drones for aerial video segments. The ground truth data set has been created
surveillance in urban policing and crowd management gives rise manually by the task organisers and consists of annotations of the
to new challenges for civil liberties, privacy and safety. The bounding boxes containing the regions of High ﴾H﴿, Medium ﴾M﴿,
ubiquity and enhanced capability of such surveillance can pose or, Low ﴾L﴿ Personally Identifiable Information (PII) including
significant threats to citizens’ privacy and therefore new vehicles, persons’ faces and accessories, and, unusual events such
mitigation technologies are needed to ensure an appropriate level as fighting, stealing and bag dropping.
of privacy protection. The Drone Protect Task (VPT) of The data included such annotations that distinguished the
MediaEval 2015 has thus provided an opportunity for relative privacy sensitivity of PIIs; namely for License Plates(H),
experimentation to explore how video-analytic techniques may Skin (M), Face (H), Hair (L), Accessories (M), and for Person’s
arrive at enhanced solutions to some visual privacy problems. body (L). The dataset has been provided in accordance with the
This task focuses on privacy protection techniques that are European Data Protection and ethical compliance guidelines
responsive to the context-specific needs of persons for privacy. including informed consent and access control as required. Figure
The DroneProtect performance evaluation involves three distinct 1, shows an example of a video illustrating Illicit behaviour.
user studies aimed at developing a deeper understanding of users’
perceptions of the effects and side-effects of privacy filtering for a Figure 1 Sample of the video in the dataset [3].
user-centered evaluation of the privacy solutions offered.
2. DPT 2015 DATASET
The DPT dataset has provided 38 video clips of about 20
seconds each, in full HD resolution with sufficient number of
examples of video images depicting different typical scenarios in
a car park [3]. The bounding boxes for persons and cars are
annotated. However, the detection of the face-head area as a
region of interest and detecting a “person-entering-a-car” event
are regarded as task to be solved as would be the cased in a real-
life Car Park Security use-case scenario; this will provide an
appropriate level of challenge in the present task, especially as the
region-specific privacy filtering element has been previously
benchmarked within the MediaEval 2014 Visual Privacy Task [1].
The video data included various scenarios featuring one or
3. AIM AND OBJECTIVES
The objective of the DroneProtect: Mini-drone Video Privacy
several human subjects walking and interacting with vehicles in a
Task is to benchmark privacy filtering solutions for drone videos
Copyright is held by the author/owner(s).
MediaEval 2015 Workshop, Sept. 14-15, 2015, Wurzen, Germany.
related to public safety. The performance of solutions is judged privacy filtering; a separate evaluation process has been planned
by their ability to retain sufficient (frame-level) semantic for the results of this additional element of the task.
information about activities and situations, while at the same time
providing the required level of privacy for people appearing in the 4. SUBMISSIONS EVALUATIONS
videos. Task participants are to implement a combination of The ground truth consisted of video frames with annotations
privacy filters to protect various personal information regions in a of the bounding boxes containing description of entities in the
set of drone videos as had been provided. Privacy filtering is to video images of persons and cars plus examples of alternative
be optimised for the fulfillment of both the following criteria: filtering approaches and questionnaires used by the human
Obscure personally identifying information effectively viewers who had evaluated them, and, the final rankings achieved.
Preserving the information needed by a human viewer in Privacy Solutions Evaluation: Participants have submitted
order to interpret the video at the level required to maintain privacy protected video clips using the testing subset. The
evaluation of the submitted privacy solutions is based on the
security in the area monitored by the drone.
human-perceived level of privacy filtering i.e., the level by which
Solutions attempted to preserve the overall visual the High/Low regions of personally identifiable information, as
acceptability-attractiveness of the resulting privacy filtered video- previously annotated in the dataset, have been responsively
frames, since these factors had potential impact on interpretability obscured by appropriate filtering techniques. Thus the evaluation
and on the quality of the work experience for humans interpreting is essentially based on the overall human perception and
the videos. As a secondary goal, the task aimed to investigate interpretation of the resulting privacy filtered image in terms of
mixtures of reversible and irreversible privacy filters. the level of retained information i.e., intelligibility, and,
For this task, the use-case scenario was Car Park Security appropriateness (acceptability-attractiveness) of the privacy
and so the typical objectives of such a scenario would determine filtered image (also defined in the MediaEval 2012, MediaEval
how much of which type of information must be retained in the 2013, and, MediaEval 2014 Privacy Task descriptions [1,2]).
video to support the goal of maintaining security. The video input Participants will each receive the results of the evaluations of
for the privacy filtering process consisted of drone video clips their submission as well as the overall results and rankings for all
showing examples of: Persons walking, running, or fighting in the the submitted entries. The rankings will be based on the
car park area, Persons attacking a driver, loitering, entering or application of different weightings to the results for each of the
leaving a particular car in the car park, wrongly parked cars, and above three criteria (privacy protection level, intelligibility,
collision with cyclists. appropriateness) as calculated from the evaluation results arising
The output of the privacy filtering process was to preserve from evaluations by the surveillance security practitioners and
sufficient semantics for recognition of specific security-relevant naïve evaluators.
events unfolding in the car park scenes whilst reversibly masking The weightings will be agreed by the participants so as to
the following aspects: reflect the relative importance of each of the above three
Person’s face and silhouette evaluation criteria as perceived by each of the human evaluator
Person’s gender and race (note this does not entail groups. 6 participants from the security practitioner’s category
gender/race recognition but rendering un-classifiable) and 11 from the naïve category will be asked to complete a survey
Personal accessories with 13 questions after viewing each of 3 randomly selected and
Vehicle make and model distinct videos of results of privacy filtering as submitted by each
Vehicle license plate (if zoomed-in on) team. The 13 question will evaluate all three criteria. The score
given to each team will consist of the average score for each of
The face and the car body have high personal identification the criteria mentioned above for each evaluation category.
potential, whereas the human body outline, particularly one that
has been rendered gender-unclassifiable, has a low personal 5. ACKNOWLEDGEMENTS
identification potential. Note that gait analysis has been excluded The Drone Protect Task at MediaEval 2015 was supported by
in the formulation of the task. Accordingly all image regions as the European Commission under contract FP7-261743
listed above needed to be masked respectively with corresponding VideoSense.
filter strength, High (H), Low (L), Medium (M) so as to maintain
the appropriate privacy protection, intelligibility and 6. REFERENCES
attractiveness-acceptability of the resulting privacy filtered video [1] Badii, A., Al-Obaidi, A., and Einig, M., MediaEval 2013
frame. Thus this privacy filtering task required the detection of the Visual Privacy Task: Holistic Evaluation Framework for Privacy
human face-and-head zone within each bounding box that has by Co-Design Impact Assessment. MediaEval 2013 Workshop.
already delineated a person. CEUR-WS.org, 1043, Barcelona, Spain, October 2013.
As a secondary goal the task invited solutions that deployed
an appropriately managed mix of reversible and irreversible [2] A. Badii, T. Ebrahimi, C. Fedorczak, P. Korshunov, T. Piatrik,
privacy filters. Such filters are typically optimised responsively to V. Eiselein, and A. Al-Obaidi. Overview of the MediaEval 2014
the context of the events and persons’ behaviours occurring in the visual privacy task, In MediaEval 2014 Workshop, Barcelona,
video. Such filtering must also allow the car park staff to reverse Spain, October 2014.
the filtering to investigate any activities as deemed possibly [3] Bonetto, M., Korshunov, P., Ramponi, G., and Ebrahimi, T.,
relevant to the investigation of any security incidents within a Privacy in Mini-drone Based Video Surveillance, Workshop on
specific time frame as set by the regulations; e.g., within 7-30 De-identification for privacy protection in multimedia, May 2015.
days of any video-recording after which all videos are usually [4] Badii, A., Einig, M., Tiemann, M., Thiemert, D. and Lallah,
deleted. C., Visual context identification for privacy-respecting video
As an additional challenge a set of 5 un-annotated videos analytics, in IEEE 14th International Workshop on Multimedia
were provided for the participants, optionally to attempt blind Signal Processing (MMSP 2012), pp. 366-371, Banff, Canada,
September 2012.