In the past years there has been a solid shift for the entire world population towards a more active presence online. Covid-19 has further pushed many activities to be faced digitally. Virtual meeting application like Zoom, had 10 million daily meeting participants in December 2019, but by April 2020, that number increased to reach up to 300 million [1]. It is estimated that in 2024 only 25% of the business meetings will take place in person [2]. Studies started during 2020 have demonstrated that nowadays people spend on average way more time in virtual meetings than before [3], leading to many concerns for the single person. Users have started experiencing stress related to not being competent in the use of the technology, but most importantly to "Zoom fatigue" due to it being always "on" [4]. Many privacy related issues have been crippling the user experience ever since, such as exposing private and personal spaces on camera, unintentionally framing a person who did not give consent to be on video or sharing sensible information leaked from careless online posting. Many solutions have been promptly developed to prevent such things from happening, providing virtual meeting room services with safeguard-privacy functionalities like blurring the background and virtual backgrounds [5,6,7,8]. We present in this paper a novel computer vision based approach for privacy enforcement in video data, capable of filter out from a video a list of objects that a user does not want
With the advancement of technology, people have been sharing a continuously growing amount of personal data online. Additionally to life-logging devices [9], social medias have recently stepped in, ending up quickly dominating the landscape of mass produced data with "visual data"(i.e. images and video). For instance, in 2020, the first year of pandemic, users have generated and shared via Facebook a total of 10.5million videos [10]. This impactful amount of data brought to the attention of experts and users to many privacy related issues; studies started identifying and observing how easily privacy could be violated just from unintentionally sharing personal data contained inside images and videos, and subsequently started proposing privacy models to formally approach and tackle said scenarios [11]. The scientific world went quickly from defining sub-fields like Privacy-Preserving Machine Learning(PPML) [12], to adopting deep learning models for image disguising [13], Context Recognition [14] as well as Image-Based Localization [15] and again computer vision based framework [16] as novel solutions for privacy preserving of first person vision image sequences, placing computer vision, Artificial Intelligence and data driven approaches as state of the art techniques for preserving privacy on line. Among the many available solutions for privacy preservation and safeguarding, it is currently missing one which allows single users to selectively censor objects from visual data depending on their personal needs and preferences.
The propose work aims therefore to provide experiencelacking users with an intuitive, easy to use tool for privacy enforcement in video data based on computer vision techniques.
Sensible users' data is crucial to be kept private. The proposed software tracks such information by means of various modules which pipeline is shown in fig. 1. Memory buffers are used in between modules to guarantee flexibility towards input videos of any ππ ππππ‘, πππ‘ππ and π ππ , as well as to stabilize the output overcoming the common flickering experienced in these kind of applications. Thanks to such buffers it is possible to store past frames and reuse those to statistically smooth of the final output; past frames are reused according to level of confidence the class recognition module predicted with. Input videos can be directly uploaded to the system or streamed from cameras(i.e. webcams). The proposed solution separates the overarching learning problem in two sub problems, namely context recognition and image segmentation; this approach guarantees robustness through modularity and simplifies the overall functioning of the software.
Recognizing the users, their emotional state [17,18,19], his attentive state [20,21,22], and the context surroundings [23,24,25] allows to selectively obscure specific elements based on preferences the same user expressed at the registration time; such data is stored into a database for later inference. Context recognition has been tackled with a neural network inspired by Alexnet [26], a famous Deep Convolutional Neural Network, designed for image classification.
Together with an RFID application [14] Detectron 2, a very powerful instance segmentation network published by Facebook in 2019 [27], is used to identify user's context specific, privacy related data, within video frames, with no ambiguity; combining the output masks produced by Detectron 2 with all the information retrieved before, a particular region of the frame is identified and filtered with a Gaussian transform. Similar or identical contexts disambiguation is possible to be tackled and solved with the support of RFID technology: with the introduction of a beacon that send a constant signal, it is possible to recognize and distinguish two apparentlysame looking environment. Such discriminatory action is essential, yet simple to be applied since it is integrable in any environment with low effort or invasiveness. A similar RFID-based solution for context recognition was already presented by another research [14] conducted some years ago. Finally the desired effect is obtained by processing and collecting all the frames of the video and setting the right frame velocity.
Distinct datasets have been used for the two different learning tasks respectively, image segmentation and context recognition. The choice of using the Detectron2 network for the the image segmentation tasks leaves little to no choice but using the 2017 version for the COCO dataset [28] which has been demonstrated to be performative with such dataset. COCO is a dataset composed of two groups of elements: images and annotations. Images contain a vast variety of objects, for a total of 80 different category of elements.
The network was capable to recognize them all, and even apparently odd objects were left untouched and not removed. Together with the set of images, COCO is composed of a set of so-called "annotations" that contain information related to the position of the object masks,
Detectron2 is Facebook AI Research's library [27] that provides state-of-the-art detection and segmentation algorithms. It is the successor of Detectron [31] which is in turn based on the Maskrcnn-benchmark model [32]. It supports a great number of computer vision research projects thanks to its flexibility, output capabilities and available documentation. Among the available Detectron2' architectures maskrcnn-fpn has been chosen. Such architecture is mainly built from three modules: a Backbone Network, a Region Proposal Network and a Box Head.
The π΅πππππππ π ππ‘π€πππ, whose role is to extract multiscale feature maps with different receptive fields starting from the input image, is based on the Feature Pyramid Network [33] technique. In this way areas of interest from different points of view are identified and passed to both the two next modules. The π πππππ π πππππ ππ π ππ‘π€πππ detects object regions (which are the so called "proposal boxes" ) based on multi-scale features, which together with the feature maps serve as input for π ππ (Region of interest) π»πππ. This last module warps feature maps using proposal boxes into multiple fixed-size features, and retrieves the fine-tuned box locations and classification results via fully-connected layers.
The context recognition task belongs to a classification problem, where each frame of the video is treated as an image to classify. For this task Alexnet has been fine tuned on our reduced dataset by means of transfer learning.
The structure of the network is shown in figure 2.
Here we report the performance with which we evaluated our model. We will give particular importance to both the accuracy and F1-score of each class. The performance of the network are reported in the table 1. As we can see we are capable of obtaining high F1score values for each class and an overall accuracy above 80%, making the results satisfying for our standards. We can consider the macro F1 score as a general metric of evaluation, defined as:
Which is simply an overall F1 score among all classes. In our case, we can see a macro F1 score of 0.81.
It is also possible to take vision of the confusion matrix in figure 3, showing how the different samples from the test set were classified during the test phase.
Videos in daily life scenarios, likely happen to contain temporary blank frames, as well as artifacts, due to user or scene related conditions. A context recognition network will therefore generate a classification label that will be trivially assigned, leading to an instability problem. We dealt with this problem through the introduction of a "memory buffer", that is capable of stabilizing statistically the result. This is achieved by endowing the system with two buffers, one for the context recognition network that stores the predicted context classes, and one used to track the instance segmentation network output and to store the classes predicted with enough accuracy. Storage of past data allows to create a time relation between successive frames, thus enforcing the output of each network and stabilizing the final one. This method allows to correlate information inside the video with the least expenditure of resources. There are two buffers.
The first buffer visible in the top of fig. 1 is the one dedicated to the output of Alexnet. The assumption taken by this approach is that the context changes don't take place suddenly but instead are related to a smooth trend. For instance, if the frame π recognizes a specific context, there is a high probability that also the frame π + 1 will carry out similar information and represents the same context. Thus, doing an average among the past frames increases the overall accuracy by smoothing the output trend. The length of the buffer is set dynamically in relation to the π ππ value retrieved from the video and the information obtained by the frames from the last half of second gets stored within. The trade-off of this method is a small delay from the context recognition module because the output context label has to stabilize for at least half of the buffer πππππ‘β to change the output and in this short period of time is wrongly classified with the previous stable label. This is strongly compensated by the stability provided and the delay is short enough to be difficult to notice.
The other output stabilization technique is the instance segmentation memory buffer dedicated to the Detectron output. The rationale here is regarding the threshold used by the model to decide if an element belongs or not to a certain class. Being a privacy concern to conceal as much as possible the sensible information inside the frames, false positives in exchange of a higher number of true positives are preferable. Therefore, two kinds of thresholds are considered: the basic one and the optimal one. The first one is lower than the second one and is the minimal value considered acceptable to take into account the output of the network. If the output confidence regarding a specific instance inside the frame falls below this value is considered too unclear and it is not counted. The second threshold instead represents the optimal value of confidence used by the system in order to properly recognize an element with enough accuracy. This information is used in order to track the last elements appeared to the network, appending them inside the buffer. If the networks find an instance of a class inside the buffer even with a lower confidence value in respect to the optimal threshold it is still considered acceptable, therefore processed and eventually concealed. In this way it gets easier for the network to work with moving objects because this method allows it to trace them even in the case of uncertainty due to movement. The buffer length is dynamically related to the π ππ value and stores information regarding the set of frames concerning the last three seconds. If an element is recognized by the network after this time interval in order to be evaluated it needs to overcome the optimal confidence threshold again. The trade-off of this system, as mentioned above, is a higher frequency of false positives, which can be misleading for the final result and inversely proportional in number to the two threshold values. Overall, the accuracy following this approach improved by a discrete percentage, mainly in the more dynamic scenarios.
The results of the system are evaluated accordingly on how many times the full procedure works consistently to specific information given as input related to a specific test video. Knowing in advance those information which are the settings inside a set of test videos as well as the list of elements inside of them, we can measure the overall accuracy of the system. For instance, if a video displays a specific context with a certain number of known elements inside of it we can control how many times those elements are found by the two networks and by the two memory buffers in the output trend. In order to achieve this result an evaluation procedure was implemented that given an input video follows similar steps that the system does but keeps into account the number of times the output given starting from each input frame is correct in respect to the total number of them. The test video used are three, which are providing the following scenarios:
β’ πππ‘πβππ (fig. 4.a), where we want to blur out a bowl from a table given that the system recognize the context. The instance segmentation network can identify various objects as a table, a oven and bottles. Due to the user preferences, here the stationary objects we want to blur out from all the video frames is simply one, a black bowl.
β’ πππ‘βππππ (fig. 4.b), where the user want to blur out the WC. In such scenario, the instance seg-mentation network recognize as a WC also the bidet given the similarity in their structure.
β’ π΅ππππππ (fig. 4.c), In this scenario there is a bowl placed in a flat surface behind the bed, and the user wants to blur it out. This scenario can be potentially challenging since the portion of the room we are framing is very restricted, and the only object that can be considered a strong feature is the bed.
The results are shown in the following table . Here we have 5 different values of evaluation for each test:
β’ accuracy of context recognition network respect the total number of frames(C.R.), indicating the percentage of success for the context recognition network applied to the frames of the video. This value does not show the improvements brought by the memory buffer.
β’ accuracy of context recognition network + memory buffer respect the total number of frames (C.R. βπ€ B.), indicating the percentage of success for the context recognition network combined with the memory buffer for the context recognition task.
β’ Accuracy of the instance segmentation network in finding the objects of interest in a frame respect the total number of frame(I.S.). This indicates the percentage of success for the instance segmentation task respect the objects of interest for the user. This value does not show the improvements brought by the memory buffer.
β’ Accuracy of the instance segmentation network + memory buffer in finding the objects of interest in a frame respect the total number of frame(I.S. βπ€ B.). this indicates the percentage of success for the instance segmentation task respect the objects of interest for the user. This value does not show the improvements brought by the memory buffer.
β’ overall accuracy of the whole system. This indicates, as the name states, the overall accuracy of the whole pipeline. This accuracy is given as a combined accuracy from the accuracy of the two tasks, obtained as the product between the accuracy of the instance segmentation considering also the buffer and the context recognition considering also the buffer.
In table 2 tests' results are reported, confirming that memory buffers contribute to increase the accuracy of both tasks, translating in overall better system's accuracy.
It must be noted that accuracy can be further improved by fine tuning the thresholds required by the instance segmentation task. A general thumb rule is that, if the accuracy is similar between the system using the buffers and the system not using it, it is possible to improve the performance through such fine tuning.
In this paper we presented a machine learning-powered solution for privacy enforcement in video data, a datadriven implementation to safeguard the privacy of any user that may be forced to spend plenty of hours in videos and/or in video meetings. Such solution will solve an untouched problem that wasn't formally faced by the field in the past years, where our time started to be off-centre towards the time spent online. Our implementation presents good performance even in presence of noisy or foggy videos, while perform almost perfectly in the most common scenarios of videos with common perspectives extracted from general recordings using mobile devices. The adaptability of the system to change with the needs of different users both for the objects of interest and the context of interest makes the solution we propose a solid step forward in the field of privacy enforcement for video data.