An integrated intelligent surveillance system for Industrial areas Francesco Camastra, Angelo Ciaramella, Angelo Casolaro, Pasquale De Trino, Alessio Ferone, Giovanni Hauber, Gennaro Iannuzzo, Vincenzo Mariano Scarrica, Antonio Junior Spoleto, Antonino Staiano∗ and Maria Concetta Vitale Department of Science and Technology, Parthenope University of Naples, Centro Direzionale Isola C4, Naples, 80143, Italy Abstract This paper presents the design and implementation phases of a software prototype developed by the University of Parthenope for the SE4I project (Smart Energy Efficiency & Environment for Industry), funded by the ”Progetti di ricerca industriale e lo Sviluppo sperimentale” (PNR 2015-2020). The prototype leverages advanced computer vision techniques based on deep learning architectures to address industrial security and monitoring needs. Specifically, the prototype tackles three key functionalities, (1) personnel and vehicle identification: The system recognizes authorized personnel and vehicle license plates within video streams captured in restricted industrial areas; (2) anomaly detection: The software can detect various anomalies in video feeds, including falls of personnel in monitored zones and unattended objects left in unauthorized areas; (3) smart parking management: The prototype identifies vacant parking spaces within camera-monitored zones, enabling efficient parking management. These functionalities are integrated into the software prototype, and its performance has been thoroughly evaluated. Keywords Plate Detection, Face Detection, Fall Detection, Parking Detection 1. Introduction with barriers. Upon arrival, an employee’s car triggers the system. The camera mounted on the smart pole cap- The SE4I project aims to improve safety within a desig- tures the RGB video stream of the scene, using AI to nated industrial area by implementing a real-time video identify the license plate and the driver’s face. Access monitoring system. This system uses strategically placed is then granted only after successful recognition. The smart poles equipped with RGB cameras to capture video combined recognition of license plate and driver verifies streams. The project focuses on three key functionalities that the vehicle and driver are authorized. If recognition (see Fig. 1): (a) authorized access control: The system is successful, access is allowed; if not, access is denied; will recognize individuals and vehicle license plates. This (b) anomaly detection: This use case focuses on detect- ensures that only authorized personnel and vehicles can ing abnormal behavior or events in the video streams enter the area, likely through controlled access points captured by the pole-mounted cameras. These anoma- lies can range from environmental violations, such as Ital-IA 2024: 4th National Conference on Artificial Intelligence, orga- illegal dumping of waste to personnel safety issues, e.g., nized by CINI, May 29-30, 2024, Naples, Italy ∗ Corresponding author. persons’ falls, and unattended objects left in restricted Envelope-Open francesco.camastra@uniparthenope.it (F. Camastra); areas. An RGB video stream from a smart pole camera angelo.ciaramella@uniparthenope.it (A. Ciaramella); continuously feeds the scene, which may include both angelo.casolaro001@studenti.uniparthenope.it (A. Casolaro); facility personnel and outsiders, to a hardware compo- pasquale.detrino001@studenti.uniparthenope.it (P. D. Trino); nent equipped with AI modules. The intelligent module alessio.ferone@uniparthenope.it (A. Ferone); giovanni.hauber@studenti.uniparthenope.it (G. Hauber); analyzes the video to identify unusual elements. Upon gennaro.iannuzzo001@studenti.uniparthenope.it (G. Iannuzzo); detection, an alert is sent to a central control station that vincenzomariano.scarrica001@studenti.uniparthenope.it specifies the type and location of the event. This allows (V. M. Scarrica); immediate assistance to personnel experiencing medical antoniojunior.spoleto001@studenti.uniparthenope.it (A. J. Spoleto); emergencies or accidents and swift intervention in case antonino.staiano@uniparthenope.it (A. Staiano); mariaconcetta.vitale001@studenti.uniparthenope.it (M. C. Vitale) of suspicious activity; (c) smart parking management: Orcid 0000-0003-4439-7583 (F. Camastra); 0000-0001-5592-7995 This use case addresses the detection and management of (A. Ciaramella); 0000-0002-7577-6765 (A. Casolaro); parking availability within the vast industrial area. Due 0009-0003-0680-4501 (P. D. Trino); 0000-0002-4883-0164 (A. Ferone); to its size, automated and intelligent parking lot mon- 0009-0007-0137-3182 (G. Hauber); 0009-0003-5962-8302 itoring is crucial. This system will inform users about (G. Iannuzzo); 0009-0008-4640-2693 (V. M. Scarrica); 0009-0007-4037-7821 (A. J. Spoleto); 0000-0002-4708-5860 free parking spaces as they approach designated park- (A. Staiano); 0000-0002-5538-9952 (M. C. Vitale) ing spaces. In the context of performing learning tasks © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). from video streams in surveillance and security appli- CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings region, the Warped Planar Object Detection Network (WPOD-NET) [3] is used to search for license plates. The WPOD-NET performs affine transformations to rectify the LP area to resemble a frontal view. These detections are then passed to an OCR network for accurate charac- ter recognition and extraction. To balance computation time and performance, we chose YoloV4 [4]. For the clas- sification problem, we treated the network as a closed system, consolidating the outputs specifically related to vehicles such as cars, buses, and motorcycles, while ig- Figure 1: Project tasks: (a) people/vehicles recognition; (b) noring outputs related to other classes. The design of fall detection; (c) anomaly detection; (d) parking lot handling. cations, the state of the art is represented by computer vision techniques based on the use of deep learning tech- niques [1, 2].In the subsequent sections, we will discuss the proposed solutions for each of the aforementioned tasks. Figure 2: The proposed pipeline at work. 2. Plate Detection The objective is to recognize vehicles and their license WPOD-NET, which is responsible for warping the license plates from a surveillance video feed and retrieve the as- plate into a rectangular shape, was influenced by insights sociated alphanumeric sequence. This sequence is subse- from YOLO, SSD [5], and Spatial Transformer Networks quently utilized for additional vehicle recognition within (STN) [6]. Finally, in our OCR module, we used Tesser- the system. act [7], a fine-tuned optical character recognition engine trained on our license plate character dataset. Tesser- act’s advantage over a simple CNN lies in its recurrent 2.1. Challenges neural network (RNN) architecture [8], which takes into Automatic license plate recognition faces several hurdles, account the sequential nature of the characters on a li- specifically, (a) Variable Lighting: Extreme brightness, cense plate. This allows for accurate recognition as the low light, and shadows can significantly reduce plate RNN captures the contextual dependencies between char- visibility. The systems address this with techniques like acters. Tesseract’s extensive training on diverse datasets adaptive thresholding and contrast enhancement; (b) Car makes it robust, handling different font styles, sizes, and Position: Vehicles approach cameras at various angles noise levels commonly found in license plate images. and distances. Sophisticated algorithms are required for accurate plate localization and perspective correction to 2.3. Execution account for these variations; (c) Occlusions: Objects like bumpers, dirt, or even other vehicles can partially or fully The modules are designed for real-time execution in an obscure the plate. Robust object detection and diverse embedded system environment, given the strict time con- training data are crucial to overcome these occlusions; straints imposed by vehicle identification. Fig. 3 illus- (d) Font Diversity: License plate formats and fonts differ trates the time required for the execution steps. significantly across countries and even regions. Train- ing models on a wide variety of datasets is essential for generalization across different plate styles. 2.2. Methods The proposed approach consists of three key steps: ve- Figure 3: Execution time of the pipeline. hicle detection, license plate (LP) detection, and optical character recognition (OCR), as shown in Figure 2. In the first step, the system detects vehicles in the scene using a dedicated module. Within each detected vehicle 3. Face Recognition Our research team has developed a framework for secure access control in the industrial area as a smart city envi- ronment. The framework leverages surveillance cameras positioned at entry points to industrial areas. It aims to match detected faces with the license plates of associ- ated vehicles. This ensures that the driver corresponds to the registered vehicle owner, improving access control security. 3.1. Challenges Figure 4: Visual results on a customized test. The framework faces different issues, particularly with on-board processing. In addition, reflective surfaces and occlusions caused by sunlight on vehicle windshields can 4.1. Challenges hinder facial recognition. Anomaly recognition in video streams presents a signifi- cant challenge: identifying rare and short-lived events 3.2. Methods that deviate from the norm. These anomalies often occur The face recognition process is divided into two main for just a few seconds, making them difficult for humans steps, namely, face localization and face classification. to detect and nearly impossible to capture in a single, In the localization step, faces are accurately localized universal model. The vast number of possible anomaly within an image by a Multi-task Cascaded Convolutional types, locations, and contexts makes defining a compre- Networks (MTCNN) algorithm [9]. Its unique cascade hensive model impractical. It would require an enormous structure consists of three stages: Proposal Network (P- amount of data and manual effort. A more effective ap- Net), Refinement Network (R-Net), and Output Network proach is to train models that can differentiate between (O-Net). By simultaneously performing multiple tasks normal and abnormal activity, regardless of the specific such as face detection, bounding box regression, and face anomaly type. This approach leverages the fact that nor- landmark localization, MTCNN ensures a thorough and mal behavior typically occurs far more frequently than accurate face identification. In particular, it excels at anomalies. detecting faces at different scales and orientations while maintaining impressive computational efficiency, making 4.2. Methods it ideal for real-time applications. For face classification, an extremely effective approach We have used a reconstruction-based method, where combines two different algorithms. The first is a face a model is trained to learn the normal patterns of the alignment with an ensemble of regression trees [10]. Us- data, to be able to reconstruct them when new frames ing an ensemble of regression trees, the algorithm pre- are presented. The model is designed to extract the spa- dicts the positions of facial landmarks directly from image tiotemporal structure of the video stream to accurately data, bypassing traditional optimization methods. The learn the pattern of normality for a scene without anoma- second uses FaceNet[11], which efficiently maps a face lies [12](without the abnormal situation highlighted in into a continuous embedding space, i.e., converting it red in Figure 5). During testing, the model provides in- into a 128-feature embedding vector. This vector is then formation about the most anomalous areas of the video matched to a face in the database using a one-shot ap- by comparing input frames with reconstructed frames. proach (see Fig. 4 as an example of qualitative result on A relatively low score indicates a normal scene, while a test image.) a high score indicates the presence of anomalies. The goal is to create an end-to-end model capable of learn- ing the spatio-temporal patterns of the analyzed data 4. Anomaly Detection and predicting when an event is anomalous compared to the learned normality. The model used for this un- Anomaly detection involves the identification of unusual supervised analysis is called CLSTM-AE, which stands items, events, or observations that are significantly differ- for Long-Short Term Memory Convolutional-Transpose ent from the norm or expected behavior, or that indicate Convolutional Autoencoder. The architecture is a spe- unusual conditions. The activities conducted in the SE4I cial type of autoencoder [13]. This approach enables project focused on identifying waste dumping where ac- real-time anomaly detection by identifying events that cess is prohibited, and fall detection. this approach is labor-intensive and requires continuous monitoring. 4.3.2. Methods Our method addresses the previous issues by enabling fall detection with a simpler setup, a standard RGB cam- era, eliminating the need for specialized equipment, and AI-powered detection that uses a single AI module run- ning on a GPU to analyze the video stream for instances of falls. This approach eliminates the need for wearable devices and reduces the reliance on human intervention. Detection is to be performed in pedestrian areas and parks, and so a dataset was created to fit this particular environment and to train the model on data representing the final context. The training dataset is illustrative of Figure 5: Anomaly detection in a parking lot. all the normal poses that people take while walking in places like pedestrian areas and parks. The scenes were therefore captured with a fixed camera about 3 meters deviate significantly from the learned representation of above the ground, pointing across an open space and normal video data. The trained model compares the re- covering walking, standing poses, and running events constructed video frames with the original input. For nor- in all directions and with or without obstacles/occlu- mal events, the reconstructed frames closely resemble the sions. The idea of training a model with only ”normal originals, with minimal differences in pixel values. How- events” is important because, in nature, abnormal events ever, when anomalies occur, the network’s reconstruc- (falls) are very rare and therefore expensive to acquire. tion becomes less accurate. This is reflected in blurry or The data preprocessing pipeline involves using the open- distorted frames compared to the originals. By analyzing pose framework [14] to extract skeleton key points from this reconstruction error (the difference between original each frame. From each skeleton, irrelevant keypoints and reconstructed frames), one can identify anomalies in are removed as they are considered noisy and skeletons real-time. with significant missing keypoints are filtered out. Key- point coordinates are then normalized using min/max 4.3. Fall Detection normalization and discretized into coarser bins to pro- vide numerical stability for the training phase. Finally, Falls, especially outdoors where help might be delayed, the data are shaped into time windows using sequences can lead to serious injuries. Traditional fall detection of 75 skeletal frames. Such windows form the basic unit systems often rely on wearable sensors, which can be on which the AI model operates. Since the video stream inconvenient or impractical. The SE4I project addresses is supposed to be captured at 25 FPS, working with 75 this challenge with a camera-based fall detection sys- frame windows means analyzing human behavior over tem using an LSTM Autoencoder. This system leverages 3-second actions. An overlap of 25 frames between con- anomaly detection techniques within a computer vision secutive windows is also included to maintain continuity framework. It essentially learns what ”normal” move- between the windows themselves. The model is based on ment looks like and identifies deviations from this norm a LSTM autoencoder [15], [16]. The execution time when as potential falls. This approach offers several advan- running on a consumer GPU allows for real-time perfor- tages, specifically, no need for wearable Sensors, only mance (see Fig. 6). Once the model has learned normal camera-based detection working with existing surveil- lance infrastructure, and the use of real-time alerts, en- abling faster response times. 4.3.1. Challenges Traditional fall detection systems typically rely on wear- Figure 6: Pipeline components execution times. able sensors or specialized depth cameras. These meth- ods can be intrusive to users and costly to deploy on a large scale. On the other hand, relying solely on human human behavior patterns, it can be used to reconstruct observation through video footage is an option. However, time windows. Reconstruction and input data are then compared; if the reconstruction error exceeds a certain threshold and deviates significantly from normal data, Parking lot detection and car detection are performed the input is intuitively flagged as a fall event. Overall, simultaneously to classify occupied or free parking lots the results highlight the effectiveness of using learned on the basis of the IoU between parking lot and car masks temporal skeletal patterns for robust anomaly detection detected by the Yolact++ module, respectively. For IoU in the context of outdoor fall detection. values greater than an IoU threshold, the system classi- fies parking lots as busy lots, as free lots, otherwise. The Yolact++ architecture is based on the RetinaNet ar- 5. Parking Detection chitecture [20], using pre-trained ResNet-101 stages. In addition, Yolact++ introduces three improvements over Parking detection for SE4I requires the development of the base model: Fast Mask Re-Scoring Network Stage, De- an automatic system that searches for free parking space formable Convolutions with Intervals, and a Optimized in one of the parking areas within the industrial area and Prediction Head. The selection of the Yolact++ architec- provides information to drivers who have requested a ture for the parking lot detection problem was motivated parking space. The Parking Guide and Information (PGI) by the runtime requirements and the accuracy achieved system [17] has been adopted as a solution for the parking by this instance segmentation model. detection task using a monitoring system. The proposed A client-server system called PGI has been developed. PGI system consists of two main parts. The former is The clients include the drivers, administrators, and ma- based on deep learning instance segmentation model to chine learning systems. Drivers can search for parking detect all available free spaces in a parking lot. The latter lots, while administrators can add, remove, and moni- is a client-server architecture that automatically guides tor parking lots.System operations are performed on the drivers to the closest parking lot with the highest number server side, which is built using PHP and MySQL for of available spaces. database storage. Clients connect to the server through a server interface using a Java Android app. The app 5.1. Challenges provides various functionalities, such as guiding drivers Parking lot detection systems using video surveillance to the nearest parking lot with available spaces using face several difficulties: (a) the impact of weather, e.g., GPS, and monitoring areas using the Google StreetView low visibility caused by fog, rain, and snow can signifi- API. The system presents favorable results with low loss cantly decrease the accuracy of these systems, or harsh values and acceptable mAP for both the box and the mask, weather conditions can obscure parking lot boundaries in determined using a 0.5 IoU threshold (see Table 1 and Fig. the video feed; (b) diverse parking lot data, i.e., training 7). robust parking detection models requires a large dataset Metrics Average Values with a wide variety of scenarios, including variations in Box Localization Loss 2.027 parking space layouts, weather conditions, camera an- Class Confidence Loss 1.604 gles, obstructions, parking lot types (e.g., open-air, multi- Mask Loss 3.185 story), and lighting conditions (day/night); (c) real-time Semantic Segmentation Loss 0.125 processing, that is, for practical applications, the system I Loss 0.116 needs to operate in real-time, this necessitates developing Total Loss 7.058 a light parking detection model that can run efficiently mAP@0.50 Box 80.5 on available hardware. mAP@0.50 Mask 76.62 Table 1 5.2. Methods Results on Yolact++ after fine-tuning and testing on the cus- tom dataset. This work conceived a model for parking lot detection using an instance segmentation approach. Yolact++ [18], which is an extension of Yolact [19], was trained with suc- cessful results on a novel dataset appropriately designed 6. Integration and Infrastructure for this task. The dataset consists of 1395 images and 23600 manually annotated parking lots, and it was built The five intelligent modules of the SE4I project, plate by using a web-scarping approach. The images, taken detection, face detection, anomaly detection, fall detection, from public access cameras, were selected to represent a and parking detection, are part of a larger system powered variety of conditions, i.e. weather and lighting conditions, by a peer-to-peer network of NVIDIA Jetson Xavier features, i.e, different camera angles, occlusions, shadows, devices mounted on multifunctional light poles. This presence of people or animals, camera heights, satellite im- setup ensures efficient and real-time processing of the agery in 2D and 3D, different types of lines and colors, data collected by the surveillance cameras, as the com- and different backgrounds. putation is performed in the field and each device shares [4] A. Bochkovskiy, C. Wang, H. M. Liao, Yolov4: Opti- mal speed and accuracy of object detection, CoRR abs/2004.10934 (2020). arXiv:2004.10934 . [5] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, A. C. Berg, Ssd: Single shot multibox detector, in: ECCV 2016, Springer Intl. Pub., 2016, pp. 21–37. [6] M. Jaderberg, K. Simonyan, A. Zisserman, K. Kavukcuoglu, Spatial transformer networks, CoRR abs/1506.02025 (2015). [7] R. Smith, An overview of the tesseract ocr engine, in: ICDAR 2007, volume 2, 2007, pp. 629–633. [8] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural computation 9 (1997) 1735–80. [9] K. Zhang, Z. Zhang, Z. Li, Y. Qiao, Joint face detec- Figure 7: Visual results on test images. Masks are applied to tion and alignment using multitask cascaded convo- the lots, each with a different color to better distinguish each lutional networks, IEEE Signal Processing Letters instance. The associated probability score is printed on each 23 (2016) 1499–1503. mask. [10] V. Kazemi, J. Sullivan, One millisecond face align- ment with an ensemble of regression trees, in: 2014 IEEE Conference on Computer Vision and Pattern data and JSON output with devices on other poles using Recognition, 2014, pp. 1867–1874. a ZMQ publisher/subscriber pattern. Therefore, a dedi- [11] F. Schroff, D. Kalenichenko, J. Philbin, Facenet: A cated module manages the surveillance camera stream unified embedding for face recognition and cluster- and associated metadata, such as brightness, frame rate, ing, in: 2015 IEEE Conference on Computer Vision contrast, etc. All modules are containerized as Docker and Pattern Recognition (CVPR), 2015, pp. 815–823. solutions, allowing for flexible portability, easy installa- [12] M. Hasan, J. Choi, J. Neumann, A. K. Roy- tion, resilience, and scalable performance. The whole Chowdhury, L. S. Davis, Learning temporal regular- system is based on Python and C++ programming lan- ity in video sequences, 2016. arXiv:1604.04574 . guages and PyTorch, OpenCV, OpenPose, Onvif libraries [13] Y. S. Chong, Y. H. Tay, Abnormal event detection are used. This infrastructure guarantees the real-time in videos using spatiotemporal autoencoder, 2017. requirements and the privacy of the video-monitored arXiv:1701.01546 . areas. [14] Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, Y. Sheikh, Openpose: Realtime multi-person 2d pose estima- tion using part affinity fields, 2019. doi:10.1109/ Acknowledgments TPAMI.2019.2929257 . arXiv:1812.08008 . [15] S. Hochreiter, J. Schmidhuber, Long short-term This research was conducted as part of the Smart En- memory, Neural Comput. 9 (1997). ergy Efficiency & Environment for Industry (SE4I) project, [16] M. A. Kramer, Nonlinear principal component anal- CUP 𝐼 66𝐺18000230005, funded by ”Progetti di ricerca ysis using autoassociative neural networks, Aiche industriale e lo Sviluppo sperimentale nelle 12 aree di Journal 37 (1991) 233–243. specializzazione individuate nel PNR 2015-2020, di cui al [17] D. Acharya, W. Yan, K. Khoshelham, Real-time D.D. del 13 luglio 2017 n. 1735”. image-based parking occupancy detection using deep learning., Research@ Locate 4 (2018) 33–40. [18] C. Zhou, Yolact++ Better Real-Time Instance Seg- References mentation, University of California, Davis, 2020. [1] A. Ferone, A. Maratea, Adaptive quick reduct for [19] D. Bolya, C. Zhou, F. Xiao, Y. J. Lee, Yolact: Real- feature drift detection, Algorithms 14 (2021). time instance segmentation, in: Proceedings of the [2] A. Maratea, A. Ferone, Deep neural networks and IEEE/CVF international conference on computer explainable machine learning, in: WILF 2018, vol- vision, 2019, pp. 9157–9166. ume 11291 LNAI, 2019, p. 253 – 256. [20] T. Lin, P. Goyal, R. B. Girshick, K. He, P. Dol- [3] S. Montazzolli, C. Jung, License plate detection and lár, Focal loss for dense object detection, CoRR recognition in unconstrained scenarios, in: ECCV abs/1708.02002 (2017). URL: http://arxiv.org/abs/ 2018, Springer Intl. Pub., 2018, pp. 593–609. 1708.02002. arXiv:1708.02002 .