These past years the world had to deal with a whole new situation brought by Covid-19. Everyone's routine changed and we started passing way more time than before on virtual meeting, virtual chats and similar. With this, many privacy problems arised from all the video data generated by a single user. Google and Zoom introduced the possibility to blur out the background while using a front face camera, but this did not solve many privacy concerns ranging from showing people in videos without their permission, to the leaking of sensible data and information from videos uploaded online. We propose a solution build over the use of computer vision techniques like image segmentation and classification for context recognition for a privacy enforcement solution capable of fitting the user's personal need, blurring out selectively specific objects from a video based on the user's preferences for each room in which they are.
to show, based on the recognition of the environment
framed, in order to blur out objects relatively to both the
In the past years there has been a solid shift for the en- user needs and the context in which they are.
tire world population towards a more active presence
online. Covid-19 has further pushed many activities to
be faced digitally. Virtual meeting application like Zoom,
had 10 million daily meeting participants in December 2. Related works
2019, but by April 2020, that number increased to reach
up to 300 million [
Sensible users’ data is crucial to be kept private. The proposed software tracks such information by means of various modules which pipeline is shown in fig.1. Memory bufers are used in between modules to guarantee lfexibility towards input videos of any , and , as well as to stabilize the output overcoming the common flickering experienced in these kind of applications. Thanks to such bufers it is possible to store past frames and reuse those to statistically smooth of the ifnal output; past frames are reused according to level of confidence the class recognition module predicted with.
Input videos can be directly uploaded to the system or
streamed from cameras(i.e. webcams). 3.1. Dataset
The proposed solution separates the overarching learning Distinct datasets have been used for the two diferent
problem in two sub problems, namely context recogni- learning tasks respectively, image segmentation and
contion and image segmentation; this approach guarantees text recognition. The choice of using the Detectron2
robustness through modularity and simplifies the overall network for the the image segmentation tasks leaves
litfunctioning of the software. tle to no choice but using the 2017 version for the COCO
Recognizing the users, their emotional state [
The performance of the network are reported in the table 1. As we can see we are capable of obtaining high F1score values for each class and an overall accuracy above 80%, making the results satisfying for our standards. We can consider the macro F1 score as a general metric of evaluation, defined as:
provides state-of-the-art detection and segmentation algorithms. It is the successor of Detectron [31] which is in turn based on the Maskrcnn-benchmark model [32]. It supports a great number of computer vision research projects thanks to its flexibility, output capabilities and available documentation.
Among the available Detectron2’ architectures maskrcnn-fpn has been chosen. Such architecture is mainly built from three modules: a Backbone Network, a Region Proposal Network and a Box Head.
The , whose role is to extract multiscale feature maps with diferent receptive fields starting from the input image, is based on the Feature Pyramid Network [33] technique. In this way areas of interest from diferent points of view are identified and passed to both the two next modules. The detects object regions (which are the so called “proposal boxes” ) based on multi-scale features, which together with the feature maps serve as input for (Region of interest) . This last module warps feature maps using proposal boxes into multiple fixed-size features, and retrieves the fine-tuned box locations and classification results via fully-connected layers.
The context recognition task belongs to a classification problem, where each frame of the video is treated as an − 1 =
1 ∑︁ 1 = 0.81 =1 Which is simply an overall F1 score among all classes. In our case, we can see a macro F1 score of 0.81.
Videos in daily life scenarios, likely happen to contain temporary blank frames, as well as artifacts, due to user or scene related conditions. A context recognition network will therefore generate a classification label that will be trivially assigned, leading to an instability problem. We dealt with this problem through the introduction of a "memory bufer", that is capable of stabilizing statistically the result.
This is achieved by endowing the system with two bufers, one for the context recognition network that stores the predicted context classes, and one used to track the instance segmentation network output and to store the classes predicted with enough accuracy. Storage of past data allows to create a time relation between successive frames, thus enforcing the output of each network and stabilizing the final one. This method allows to correlate information inside the video with the least expenditure of resources. There are two bufers. to change the output and in this short period of time is wrongly classified with the previous stable label. This is strongly compensated by the stability provided and the delay is short enough to be dificult to notice. 3.4.2. Instance segmentation memory bufer The other output stabilization technique is the instance segmentation memory bufer dedicated to the Detectron output. The rationale here is regarding the threshold used by the model to decide if an element belongs or not to a certain class. Being a privacy concern to conceal as much as possible the sensible information inside the frames, false positives in exchange of a higher number of true positives are preferable. Therefore, two kinds of thresholds are considered: the basic one and the opFigure 3: The confusion matrix generated with the use of the timal one. The first one is lower than the second one scikit library [35] and is the minimal value considered acceptable to take into account the output of the network. If the output confidence regarding a specific instance inside the frame falls below this value is considered too unclear and it is 3.4.1. Context memory bufer not counted. The second threshold instead represents The first bufer visible in the top of fig.1 is the one ded- the optimal value of confidence used by the system in icated to the output of Alexnet. The assumption taken order to properly recognize an element with enough acby this approach is that the context changes don’t take curacy. This information is used in order to track the place suddenly but instead are related to a smooth trend. last elements appeared to the network, appending them For instance, if the frame recognizes a specific context, inside the bufer. there is a high probability that also the frame + 1 will If the networks find an instance of a class inside the bufer carry out similar information and represents the same even with a lower confidence value in respect to the opcontext. Thus, doing an average among the past frames timal threshold it is still considered acceptable, therefore increases the overall accuracy by smoothing the output processed and eventually concealed. In this way it gets trend. easier for the network to work with moving objects beThe length of the bufer is set dynamically in relation to cause this method allows it to trace them even in the case the value retrieved from the video and the informa- of uncertainty due to movement. tion obtained by the frames from the last half of second The bufer length is dynamically related to the value gets stored within. and stores information regarding the set of frames conThe trade-of of this method is a small delay from the cerning the last three seconds. If an element is recogcontext recognition module because the output context nized by the network after this time interval in order to label has to stabilize for at least half of the bufer ℎ be evaluated it needs to overcome the optimal confidence threshold again. The trade-of of this system, as mentioned above, is a higher frequency of false positives, which can be misleading for the final result and inversely proportional in number to the two threshold values. Overall, the accuracy following this approach improved by a discrete percentage, mainly in the more dynamic scenarios.
The results of the system are evaluated accordingly on how many times the full procedure works consistently to specific information given as input related to a specific test video. Knowing in advance those information which are the settings inside a set of test videos as well as the list of elements inside of them, we can measure the overall accuracy of the system.
For instance, if a video displays a specific context with a certain number of known elements inside of it we can control how many times those elements are found by the two networks and by the two memory bufers in the output trend.
In order to achieve this result an evaluation procedure was implemented that given an input video follows similar steps that the system does but keeps into account the number of times the output given starting from each input frame is correct in respect to the total number of them. The test video used are three, which are providing the following scenarios: • ℎ (fig. 4.a), where we want to blur out a bowl from a table given that the system recognize the context. The instance segmentation network can identify various objects as a table, a oven and bottles. Due to the user preferences, here the stationary objects we want to blur out from all the video frames is simply one, a black bowl. • ℎ (fig. 4.b), where the user want to blur out the WC. In such scenario, the instance segmentation network recognize as a WC also the bidet given the similarity in their structure. • (fig. 4.c), In this scenario there is a bowl placed in a flat surface behind the bed, and the user wants to blur it out. This scenario can be potentially challenging since the portion of the room we are framing is very restricted, and the only object that can be considered a strong feature is the bed.
The results are shown in the following table. Here we have 5 diferent values of evaluation for each test: • accuracy of context recognition network respect the total number of frames(C.R.), indicating the percentage of success for the context recognition network applied to the frames of the video. This value does not show the improvements brought by the memory bufer. • accuracy of context recognition network + memory bufer respect the total number of frames (C.R. ∖ B.), indicating the percentage of success for the context recognition network combined with the memory bufer for the context recognition task. • Accuracy of the instance segmentation network in ifnding the objects of interest in a frame respect the total number of frame(I.S.). This indicates the percentage of success for the instance segmentation task respect the objects of interest for the user. This value does not show the improvements brought by the memory bufer. • Accuracy of the instance segmentation network + memory bufer in finding the objects of interest in a frame respect the total number of frame(I.S. ∖ B.). this indicates the percentage of success for the instance segmentation task respect the objects of interest for the user. This value does not show the improvements brought by the memory bufer. • overall accuracy of the whole system. This indicates, as the name states, the overall accuracy of the whole pipeline. This accuracy is given as a combined accuracy from the accuracy of the two tasks, obtained as the product between the accuracy of the instance segmentation considering also the bufer and the context recognition considering also the bufer.
ory bufers contribute to increase the accuracy of both tasks, translating in overall better system’s accuracy. It must be noted that accuracy can be further improved by fine tuning the thresholds required by the instance segmentation task. A general thumb rule is that, if the accuracy is similar between the system using the bufers and the system not using it, it is possible to improve the performance through such fine tuning.
Input Kitchen Bathroom
Bedroom
In this paper we presented a machine learning-powered solution for privacy enforcement in video data, a datadriven implementation to safeguard the privacy of any user that may be forced to spend plenty of hours in videos and/or in video meetings. Such solution will solve an untouched problem that wasn’t formally faced by the field in the past years, where our time started to be of-centre towards the time spent online.
Our implementation presents good performance even in presence of noisy or foggy videos, while perform almost perfectly in the most common scenarios of videos with common perspectives extracted from general recordings using mobile devices. The adaptability of the system to change with the needs of diferent users both for the objects of interest and the context of interest makes the solution we propose a solid step forward in the field of privacy enforcement for video data.