Video surveillance systems have become indispensable tools for the security and organization of public and private areas. In this work, we propose a novel distributed protocol for an edge-based face recognition system that takes advantage of the computational capabilities of the surveillance devices (i.e., cameras) to perform person recognition. The cameras fall back to a centralized server if their hardware capabilities are not enough to perform the recognition. We evaluate the proposed algorithm via extensive experiments on a freely available dataset. As a prototype of surveillance embedded devices, we have considered a Raspberry PI with the camera module. Using simulations, we show that our algorithm can reduce up to 50% of the load of the server with no negative impact on the quality of the surveillance service.
Video surveillance is of paramount importance in areas like law enforcement,
military and even for a commercial environment. One of the straightforward
approaches for video surveillance is to use the client-server model of
communication: the surveillance devices stream the video directly to a main powerful
server, where the data can be displayed to the human operators, who are
responsible to analyze the video [
The idea of exploiting the computation capabilities and the topology of a
distributed network of smart cameras to reduce the amount of messaging has
been proposed in [
This work summarizes the contribution presented in [
The rest of the paper is structured as follows. Section 2 provides the de nition of the system model and problem statement, whereas Section 3 presents the algorithm for the adaptive camera-assisted person recognition. Section 4 evaluates the distributed surveillance solution through simulations. Finally, Section 5 concludes the paper. 2
In our work, we model the geographical area, where the surveillance system is deployed, as divided into sectors. A set of Smart Sensing Units (SSUs) are distributed among the sectors to execute the video surveillance task (Figure 1). Such units have various sensing, computing, storage, and communication capabilities. An SSU is a logical unit, it could be a smart camera having onboard sensing, computing, storage resources, or it could be composed of a camera connected to a computer. Moreover, we assume also the existence of a Main Server (MS) unit in the network. A main server is a stand-alone SSU that is characterized by (theoretically) in nite computing, storage, and communication capabilities. It also plays the role of the central server where the devices send their requests, in case of client-server modality.
In our work, we distinguish two possible scenarios: basic and enhanced. In the basic scenario, all SSUs are connected to the central MS and their functionality is limited to video observation and data streaming to the MS. In the enhanced scenario, on the other hand, all SSUs are connected in a structured peer-topeer topology. SSUs can cooperate to execute complex tasks and build together a rich representation of the context where the infrastructure is deployed. We consider the peer-to-peer overlay connecting the SSUs to be xed. Each SSU can contact directly the MS and its geographical neighbor SSUs. We assume the communication between SSUs is reliable, i.e., no message is ever lost or corrupted. In the enhanced scenario, each SSU has computational resources and local storage memory, where it keeps a recognition library, which is a collection of recognizers, each specialized to recognize a certain person. The MS's recognition library contains the recognizers for all the recognizable persons in the surveillance area, while the recognition library of an SSU is, basically, a local cache that keeps only a subset of all recognizable persons.
As an illustrative example, let us consider the following scenario, taking place in a geographical area (the surveilled area) of appropriate size, where a number of SSUs have been deployed for video surveillance purposes. Here we consider a person that is moving in the area. The system should visually recognize the person in order to evaluate the presence of this person in the area. One of the straightforward solutions for area video surveillance is to stream all the data from the SSUs to the M S for the following processing and storage. This approach can cause some additional system issues. At rst, the interested organization needs to have a powerful server system devoted speci cally to the surveillance needs.
Algorithm 1: Active thread algorithm executed by SSU repeat recognizers.increaseAge(); events getVisibleFaces(); selection selectRecognizers(recognizers); foreach face in events do if alarm getStatus(face; selection) then send(F ACEID; face; alarm; view); startAlarmActivity(); return; if recognized getStatus(face; selection) then send(F ACEID; face; recognized; view); return; if null getStatus(face; selection) then t setRecognitionTimeout(); reclBuffer.add(face; t ); send(ALARMREQUEST; face; null; view); return; wait T ; until; This server system needs to be powerful enough to continuously process the video streaming data that is coming from the SSUs. The second problem is connected with the delays for the person recognition, caused by possible server overloading and network bandwidth overhead in case of a high rate of requests for recognition.
In order to tackle these issues, our solution exploits a distributed architecture for face recognition, where the detection and recognition tasks are moved to SSUs when it is possible. We performed some simulations in order to move the most possible computation on the camera themselves and we used optimized strategies of distributed computation to solve the most challenging resource-demanding tasks. 3
In this section, we describe our enhanced algorithm for distributing the recognition processes on the SSUs. Each camera in the system keeps the list of its neighbors SSUs, which we name view. In order to improve the e ciency of the recognition process each camera locally keeps a recognition library that contains a subset of all recognizers for faces. Using these local recognizers, each camera in the network tries to recognize the face of the detected person locally and with the help of neighbor SSUs, without involving the MS, thus reducing its computational and bandwidth consumption. The size of the library is limited to recognizersLimit according to the available storage space of an SSU.
Three threads are executed by each of the SSUs: (1) active thread (Algorithm 1) is responsible for the active area monitoring, (2) time-out thread corresponds to the timeout event in the face recognition by neighbors and (3) passive thread (Algorithm 2) processes the incoming messages received by an entity. Algorithm 2: Passive thread algorithm executed by SSU on event msg (type; face; status; sender)receive do if type == alarmrequest then result getStatus(face; selection); send(alarmreply; face; result; sender); if type == alarmreply then if sender == MS then if status == alarm then
startAlarmActivity(); integrate(recognizers; face); send(faceid; face; status; view); else recBuffer.getFace(face).counter + +; if recBuffer.contains(face) and status == null then if recBuffer.getFace(face).counter view:size then send(alarmrequest; face; null; MS); recBuffer.remove(face); else if status == alarm then
startAlarmActivity(); integrate(face; recognizers); recBuffer.remove(face); send(faceid; face; status; view); if type == faceid then
integrate(face; recognizers);
Every T each SSU executes the active thread Algorithm 1. At rst, it increases the "age" of all known recognizers in the library recognizers. Then, it processes the current surveillance area image and extracts the facial features for the detected persons. Finally, it selects the subset selection of recognizers with the lowest age parameter from the local collection.
For each f ace in events an SSU executes the recognition algorithm. In case f ace corresponds to the entry in recognizers with alarm status, the SSU (1) sends a noti cation message type faceid with a f ace features and the alarm tag to the neighbor SSUs; then (2) starts prede ned alarm activity, for example, video recording and live streaming to the operator displays.
In case a f ace is recognized, a noti cation faceid is distributed between the neighbor SSUs. If there is no correspondence in selection, the camera establishes a recognition waiting time and adds the face features into the reclassi cation bu er reclBuf f er. It also sends the recognition request alarmrequest to the neighbors' SSUs. When the recognition waiting time has passed, if the object is still in the reclBuf f er, then the camera sends the classi cation request alarmrequest to the M S and removes the features from thereclBuf f er.
When an SSU receives a message, it is processed according to the Algorithm 2. We consider three types of messages: alarmrequest, alarmreply and faceid. The alarmrequest message contains the request for a f ace classi cation. The recipient checks the status of the received f ace in the local subset of a recognizers list and sends an alarmreply containing the results of the recognition result to the sender. The faceid message is used for the face features broadcasting to the neighbor SSUs. Hence, whenever a camera receives this message it integrates the received face features to the local recognizers library.
The alarmreply message corresponds to the reply to a recognition. If it comes from M S entity, the recipient integrates the recognized face and its status to the local recognizers and sends the face noti cation message faceid to its neighbors view. Moreover, in case the received status is alarm the camera initiates some prede ned alarm activity. Each SSU keeps track of the replies of all the neighbor SSUs for each f ace in the reclBuf f er. If no neighbor SSU is able to recognize the current f ace, then the camera sends the recognition request alarmrequest to the M S and removes the f ace from the reclassi cation bu er reclBuf f er. If, on the other hand, the received status is not null it means the f ace has been recognized by the sender and the camera integrates it to the local recognizers, removes the face from the reclassi cation bu er and sends the face noti cation message faceid to its neighbors SSUs. In case the received status is alarm, the camera also initiates some prede ned alarm activity. 4
In this section, we present a simulation experiment in order to evaluate the load imposed on the M S and the performance of the distributed surveillance system.
In our simulation, we assume that the SSUs are uniformly distributed in the
monitored area. Persons in the \surveillance area" move according to the
mobility model described in [
We executed simulation of 6 hours using the peersim simulator [
In the baseline model, all SSUs stream the video directly to the M S, where the M S extracts the face features and recognizes the persons. The baseline curve in Figure 2 shows the number of recognition the M S is required to process in a 60sec time interval. Other curves on the gure show the evaluation of the recognition requests to M S when the enhanced algorithm is applied and under di erent system parameters, such as Tmax and ct, where Tmax is the maximum allowed face recognition delay and ct is the time needed to compare the extracted features with one class in the recognition library. The results show that the trssepe6h 778805050000000000000000 ct=13443655500mmmmssss u 650000 req 600000 iiiftsao 455505000000000000 n c s lcaS 345000000000 M 300000 4 6 8 10 12 14 16
Maximum al owed delay (seconds)
(b) Tmax = 10s; workload A (6ch) eaxgpgerreigmaetendt; rweoqrukelsotasdtoB SC; enhanced algorithm can reduce up to 50% of the recognition activity of the M S unit in peak hours.
Each class of the cache is a simulated sample of the face features that characterize a person. Therefore, the time needed by an SSUs to perform a single face recognition task is given by the size of the cache selection times ct. A higher number of the face features in the class increases the accuracy of the recognition, but at the same time increases also the ct.
As we can see in Figure 2a, in case of a lower Tmax the system does not have enough time for the whole algorithm recognition cycle and it is forced to rely on the M S recognition more often. Instead, a higher Tmax allows an SSU to perform the recognition using the cache and to receive the replies from the neighbor SSUs. Nevertheless, even in case of low Tmax, the enhanced algorithm signi cantly reduces the recognition load of M S compared to the baseline solution.
Figure 2b shows the in uence of ct time interval on the M S recognition load. Even in case of high accuracy of local recognition and relatively low Tmax, the load of M S by recognition requests is signi cantly lower than the baseline solution. Moreover, lower ct values further decreases the load imposed on the M S by the recognition. Figure 2c shows the impact of the system parameters on the algorithm e ectiveness. Lower values of ct and larger of Tmax signi cantly reduce the M S recognition load. One of the straightforward ways to minimize ct time is to reduce the accuracy of the local recognition algorithm by decreasing the number of features in the sample. In case fast person recognition is not a requirement, the increment of Tmax can also reduce the load of the M S. 5
In this paper, we propose a distributed edge-based protocol for area video surveillance based on image classi cation. In order to minimize the classi cation load on the main server, recognition tasks take place on the SSUs when possible. To perform person recognition, an SSU uses the local resources together with the resources of the neighbor SSUs. The surveillance devices fall back to the main server when the classi cation cannot be done in the desired time interval. In this extended abstract, we gave an overview of how our system works, we described the proposed algorithms and we presented simulations that show that by partially placing the recognition tasks on the local resources of surveillance devices can reduce the load on the main server up to 50%.