In this paper we describe the design and development of a real-world software system that integrates machine learning augmenting a pre-existing remote surveillance framework. Machine learning was embedded as a service in the system, plugged-in between back-end data flux handlers; the system has been redesigned following a microservices architecture to make it scalable and to allow a progressive adoption of the machine learning-powered assistance in the event management process. A case study of the application in an actual security company is analysed and discussed, where we show how this innovation helped human operators to better shield themselves from the "information overloading".
must try to understand the reason of the notified alarm. If it is recognized as a false alarm, the case is simply closed.
In this paper we describe the design and development On the other hand, if the alarm is improper, i.e., it is due a of a real-world software system that integrates big data system anomaly, the operator starts an anomaly handling analytics and machine learning into a pre-existing remote process. surveillance framework operated by security company The software adopted by the company to support such that monitors a number of sites through closed circuit and a process was a monolithic application that ofered only IP cameras, anti-theft sensors (e.g., volume and pressure basic functionalities such as collecting signals and data sensors, door opening sensors, etc.) and also physical streams, presenting the events in a managing console and sensors (e.g., humidity and temperature). saving them in a persistent database. Therefore, most
Figure 1 shows a fragment of the process commonly of the operations described by the event management
followed to handle events and alarms coming from a process above required a substantial amount of manual
surveillance network. When an alarm is received, first the work by the control center operators.
operators check the surveillance videos. If such videos While the human intervention cannot be avoided in
are not available or they do not clearly show the event, such a context, as in any security-related context, machine
the operator requests an on-site check to the security learning can be exploited to assist the operators in several
staf. Such action and its outcome, as well as the outcome steps of the process, leaving the humans with only the
to all the actions taken during the process, is stored in most critical steps to accomplish (see, e.g., [1, 2, 3, 4] for
the system database. Then, if the event is in progress, examples belonging to diferent surveillance contexts).
the operator starts the true alarm handling process. Oth- However, embedding machine learning in the
comerwise, if the notified event is not actually in progress, pany pre-existing software presented several challenges.
the operator must check for other alarms on the same First, we are modifying a production, real-time critical
site and, if any, restart the handling process for such new system, so we need to gradually add such a support, in
events. If no other site alarms are active, the operator order to let the operators adapt to the new
functionalities while verifying their reliability without int
In particular, while having a completely redesigned core, the system has been designed to ofer the basic functionalities of the previous one (to maintain the above service continuity) and extend them in order to classification, the data storage and the log certification; • ofer decision support tools to assist operators in
the real-time management of events.
All the three extensions above could be based on simple • acquire data and information on objects and (sometimes pre-existing) algorithms, but also take advanevents from multiple sources such as IoT devices, tage from machine learning. Thus, ML can be seen as a open systems and mobile services; (plug-in) service in the overall system architecture. This • combine and correlate the real-time big data consideration, together with a number of technical asstreams in order to make easier and faster the con- pects concerning the modular structure of the applicatrol center operators job, in particular the event tion and its deployment environment, made us opt for a service oriented architecture [5] as its development to the events of the last three days gathered from all basis. More in detail, to be as lightweight as possible, the sources and suitably aggregated (requirement 2). we adopted a microservices architecture [6] to efectively Prometheus is an open-source systems monitoring and split the complexity of the overall system into small spe- alerting software, which provides a multi-dimensional cialized units, each with its REST API and containerized data model with time series, and allows a variety of inusing Docker [7]. This allowed us to develop a scalable, teractions with other third-party software components. versatile, and easily maintainable system, as opposed to Next, there is the the Redis [12] (gray) service that the previous one. In particular, microservices allowed us wraps the well-known fast in-memory data store (supto fragment the overall system functionality in a set of porting real-time data streams), which acts as a cache meaningful basic units that, apart from the known ad- memory to provide the consumers a fast access to the vantages in term of scalability and maintanability, make event information streams. It essentially takes the role easier the progressive addition of ML support as a ser- of "live working memory" for the events and the correvice (or set of services) interacting with the core system sponding management procedures, which are stored in services to enhance their functionalities. the database until they reach the closing state. It is worth noting that the output of sensors like temperature and humidity, whose handling does not require the human 2. System Architecture for a intervention (e.g., temperature sensor alarms are based Reliable Event Handling on simple logic rules), are not stored in Redis.
Finally, the SQLite [13] service (gray) provides persisAt a macro level, the base requirements of the new system tent storage for all the data flowing through Prometheus are the following: and Redis. This is mainly needed to later extract the ev1. Acquire data, events and states from an heteroge- idences needed by the authorities (requirements 5 and neous set of sources spread in a wide geographical 6).
area and connected through a digital network; 2. aggregate such data streams in a configurable and scalable way, since the number and type of sources may vary; 3. correlate events through time and space logical
rules; 4. visualize all the collected and elaborated informa
tion on an interactive web dashboard; 5. store all the data, as well as all the actions taken by the control center operators, on a persistent memory; 6. provide a search engine with configurable queries to access such historical data.
To support requirement 4, the blue front-end services present the gathered and processed data in diferent formats, tailored for the specific needs of the diferent system users. In particular, three "IG Server" services handle the GUI for the operators, their supervisors and the system administrators. Such user interfaces present to the users the data streams coming from the sensors, read from the Redis service, as well as the data coming from Prometheus. A special "configurator " GUI is used to conifgure the system. The last two front-end services directly interact with Prometheus: the first is based on the Grafana software package [14], which provides advanced visualization and dashboards for the processed and aggregated data, whereas the alertmanager service pushes
Starting from such requirements, collected through alerts automatically generated from Prometheus through interviews with the company staf and control center PromQL queries directly into email and Telegram mesoperators, we designed a microservice-based system ar- sages. chitecture that aims to be easily testable, maintainable, Finally, the red services are at the core of the architecand extensible. Figure 2 gives an overview of the devel- ture. In general, they read from both Prometheus and oped system. Redis and apply actions, possibly modifying the Redis
The data is pushed in the system by a number of spe- data streams accordingly. In particular, gest_notifier is cialized source services (drawn in green in the figure): a critical module that notifies the supervisors (via text in the currently deployed platform, we provide services messages) about event escalation, i.e., alarms that are trigwhich support reading data from sensors using MQTT [8] gered by an event not being correctly and timely handled and SNMP [9] as well as from specific proprietary sensors by the control center operators. On the other hand, the such as Papago [10]. These services satisfy requirement gest_control service manages the QoS by monitoring that 1. the operator reaction times follow the company SLA, also
The overall system leverages on three services that pro- generating alarms in case of ineficiencies (requirement vide diferent "analysis horizons" on the event streams: 3). The gcounter service generates aggregate statistics the Prometheus [11] service (yellow), which wraps the from the Redis data and posts them back to Prometheus corresponding software package, acts as the main data to support longer-term alarms (requirement 2). Finally, collector of the overall architecture, providing access the gest_source service is the action actuator, i.e., it is called by the operator interfaces to actually apply the to the operators. actions ("take charge", "start/pause/close workflow", etc.), Clearly the human intervention cannot be avoided executes them, and generates the action events that are when trying to solve potentially dangerous events. Howstored back in Redis to log the handling process. ever, ML can help the operators in many ways, as com
The data_analysis service, also drawn in red, is de- mented in the introduction. Thus, we initially focused on scribed in the next section. It provides ML support to the an aspect that is well known to benefit from automatic overall process, and in particular assists the operators by reasoning: mitigating the efects of an event flooding on interacting with the graphical interfaces. the operators by pre-selecting or pre-classifying events.
All the microservices above were developed in Python Indeed, an operator that manages events is the classical and containerized, to be easily deployable on the com- FIFO order may spend too much time on less-significant pany’s infrastructure through an overall container de- events and delay the solution of the really critical ones. ployment script that, in particular, takes into account the To avoid this, the Data Analysis service provides an adapservice inter-dependencies. tive event classification routine that prioritizes the events so that the ones that are considered more important, i.e., that may lead to real alarms, are presented first to the 3. Machine Learning Services operators. Such a service stands between the gest_source Integration and lgo_server services, learning from the operator actions and suitably modifying his data views in order to The event handling process, in the critical context where suggest the event classification. our system works, must be timely and efective. The Operators close each event handling process by laarchitecture described in the previous section (Figure 2) belling the events as true alarms, false alarms, inapproprihas been designed to be reliable and fast, but the process ate alarms (i.e., due to a system failure) or "other alarms" (Figure 1) still includes a number of checks, calls, and (typically due to test or maintenance). We focus on the lengthy actions that require a substantial amount of work true and false ones, since the other two types of events
are a minority class that would be dificult and useless to consider in our context.
Thus, initially, we extracted a number of significant Thanks to its design and to the use of ML, the developed features from the events, relating them with the operator- system meets highest quality standards, in particular: assigned label. Such features include information such as • it allows to to acquire, aggregate and process data the unique alarm ID, the alarm central where the event and information from a variety of IoT devices, was generated, the related customer, the activated sensor which means ofering a better service in terms of name, the timestamp of the alarm, etc. quality and flexibility;
Then, we cleaned and refined these features to further focus on the information that seems to be more relevant • it guarantees high scalability and easy configurafor our classification task. As an example, we substituted bility; the alarm timestamp, which conveys too much informa- • it is fully compliant with data privacy, integrity tion, with the alarm weekday and the corresponding part and security regulations. of the day (morning, afternoon, evening, night). More- The project requirements foresaw to process about over, we performed a K-means clustering [15] on the sen- 300,000 events a year with the current number of operasor names (suitably transformed in a numerical vector tors, and manage an alarm within at most 30 seconds, as through a word embedding process), in order to extract set by the standard regulations. Currently the system has an artificial "sensor type" feature. The significance of the been deployed and is being tested by the company in a selected features was validated by calculating the mutual control room operating 24/7 on three turns of eight hours information of each feature w.r.t. the classification label each, with two or three operators per turn. The staf is (see, e.g., [16] for an overview of mutual information ap- managing about 32,500 anti-theft and intrusion detection plied to feature selection) on a set of 169,347 past events sensors and over 500 environmental IoT devices mainly provided by the company. targeted to precision agriculture. The network connects
Once the final 128 features were devised, we extracted about 30 clients and its nodes are deployed on more than an initial training dataset from the set of past events 600 diferent sources. above. Unfortunately, the dataset was heavily imbal- Our initial statistics show that the staf is now able to anced, since the false alarms were much more than the manage an average of 1,000 events per day, thus yielding true ones [17]. Thus, we tried both a random undersam- 365,000 events managed on yearly basis as a forecast, pling of the majority class and the well-known Synthetic which doubles the performances achieved using the preMinority Oversampling Technique (SMOTE, [18]) to re- vious support software. The average time from the event balance it. arrival to its classification and taking charge is now 10
Finally, we built a deep neural network [19] with 128 seconds, slightly better compared to the performance of input neurons (one for each feature), two hidden layers of the previous system but in a far more complex scenario. 64 neurons each with RELU [20] activation function, and Finally, the average event management time, includa single output neuron with sigmoid activation function. ing classification, site operations, police calls, alarm closWe trained the network on our dataset in order to obtain ing and archiving, has been dramatically improved from the correct classification given the event features. The 1,800 to 900 seconds (Figure 3 shows the current statistics classifier validation showed that the dataset re-balanced generated by the Grafana service in the application), and through random undersampling achieves a better overall the error ratio has been tackled almost to zero thanks to performance in this context, reaching an accuracy of 0.91, the ML priority classification system. a recall of 0.93, and a precision of 0.95, with a F1-Score of 0.92.
The trained network was then embedded in the Data Acknowledgments Analysis service, where each new event is classified before being presented to the operators. However, since a We would like to thank Mr. Andrea Perna who implewrong "false alarm" classification may always happen, mented the first prototype of the machine learning model we do not simply drop the events considered not harmful as part of his Master’s Thesis in Computer Science at our by the neural network from the stream, but rather we ex- Department. tract the classification probability that can be read from This work was funded by SPEE S.r.l. Company 1 in its output neuron and use it to alter the priority value that L’Aquila, ITALY. is used to sort the events on the operator dashboards. In this way, a possibly false alarm will be handled later, but never dropped. After the event is handled, the correct, ifnal classification given by the operator is sent back to the neural network to fix its previsions, if needed. 1https://www.spee.it