Deep learning is rapidly growing, obtaining groundbreaking results in speech recognition, image processing, pattern recognition, and many other application domains. Following the success of deep learning, many automatic data analysis techniques are becoming common also in law enforcement agencies. To this end, we present a survey about the potential impact of deep learning on three application domains, peculiar to law enforcement agencies. Specifically, we analyze the findings about deep learning for Face Recognition, Fingerprint Recognition, and Violence Detection. In fact, combining 1) data from the routine procedure of collecting a subject frontal and profile pictures and her/his fingerprints, 2) the pervasiveness of surveillance cameras, and 3) the capability of learning from a huge amount of data, might support the next steps in crime prevention.
ranging from personal health systems [2, 3]
to police investigations [4], to the modeling of
From its dawn as a discipline, Artificial In- automata [5] and autonomous agents [6, 7, 8],
telligence (AI) aims to understand if we are to smart home reasoning systems [9, 10, 11]
able to implement machines with the abil- and many more. On the other side, machine
ity to think. During this unceasing explo- learning tries to give to machines the
capabilration, symbolic AI, also known as Good Old- ity of autonomously learning from examples.
Fashioned AI [
The rest of the paper is divided into sec- not take into account the intra-class variance tions dedicated to each application domain, between the face images of a single subject. i.e. Face Recognition (Section 2), Fingerprint Instead, the Fisherfaces method [17] adds to Recognition (Section 3), and Violence Detec- the PCA the Linear Discriminant Analysis tion (Section 4). Finally, Section 5 draws the (LDA), in order to minimize intra-class variconclusions of this survey, highlighting some ance. Diferently from Eigengaces and Fisheraspects which we consider worth of further faces, Ahonen et al. [18] proposed to compute research. Local Binary Patterns Histograms (LBPH) on face images, dividing it into region to com2. Deep Learning and Face pute Local Binary Patterns (LBP). Similarly to Eigenfaces and Fisherfaces, a distance funcRecognition tion based on LBPHs can be used to perform the face identification.
National police forces routinely collect two While these techniques (and those derived) pictures (commonly known as mugshots), fin- obtained a good accuracy on datasets where gerprints, and personal information of a sub- some parameters such as pose, lighting, and ject, for various purposes, ranging from re- expression are fixed, they are insuficient to leasing documents to registering criminals. extract stable identity feature invariant to realHence, there is a clear connection between world changes [19], such as in images got from videos and surveillance cameras. Therefore, they are not suitable in law enforcement, 3. Deep Learning and when comparing the two mugshots (a frontal Fingerprint Recognition and a profile pictures) collected by police agencies in ideal conditions, with images got in the The patterns created by the epidermal ridges wild. On the contrary, deep learning-based and furrows on our fingers, i.e. fingerprints, techniques demonstrated capable of extract- have been used for identification for more ing features that are invariant to changing than 2000 years [28]. As fingerprints are a conditions about facial expression, lighting, so discriminative biometric characteristic, the and pose. While there are some early methods implementation of Automated Fingerprint Idenwhich combined multiple Neural Networks tification Systems (AFIS) has been a promiand Belief Revision [20, 21] before the deep nent topic in Computer Vision in the last four learning popularity, Convolutional Neural Net- decades. Specifically, fingerprint matching to works (CNNs) significantly improved the ac- identify or verify a person’s identity is based curacy in Face Recognition under unconstrained on the presence of singularities of epidermal conditions. To this end, Taigam et al. [22] ridges called minutiae [29]. In this regard, presented DeepFace, a 8-layer CNN to pro- algorithms to extract features and perform cess 3-channels 152x152 face images, capable matching on fingerprint images focused on of getting a 97.35% accuracy on the Labeled two basic types of minutiae: bifurcations and Faces in the Wild (LFW) dataset [23]. Simi- terminations, i.e. the points where a ridge larly, Schrof et al. [ 24] proposed Facenet, a splits itself into two ridges and where a ridge 22-layer CNN trained in several experiments ends [30, 31, 32]. In addition to issues such with a varying number of face images, be- as image noise, distortions, rotations, and distween 100 and 200 million, belonging to 8 placement, large variability in diferent immillion of diferent subjects. They got 99.63% pressions of the same finger and similarity accuracy on LFW, using 220 x 220 input im- between two images from diferent fingers ages. Cao et al. [25] showed the efectiveness make fingerprint matching a very challengof the ResNet-50 [26], a 50-layer CNN based ing problem [33]. on residual learning able to get a top-1 identi- Traditional Computer Vision-based algoifcation error of 3.9% on the VGGFace2 datset rithms demonstrated their efectiveness on (composed by over 3 million of images of more fingerprint matching, and specifically, on minuthan 9 thousands subjects). tiae matching, evolving over the years. For
The listed CNN-based techniques for Face example, in 1997, Maio and Maltoni [30] proRecognition are just few examples among the posed to perform ridge line following on gray many which demonstrated they robustness to scale fingerprint images to identify terminachanging conditions and unconstrained face tions and bifurcations. Farina et al. [31] proimages (see Guo and Zhang [27] for a de- posed to identify minutiae from skeletonized tailed list of deep learning-based Face Recog- binary images. Fronthaler et al. [32] exploited nition techniques). However, to the best of symmetry features (linear and parabolic) to reour knowledge, there is a lack of research duce noise and extract minutiae on grayscale in understanding to which extent such tech- images. Cappelli et al. [34] proposed a new niques are efective in identifying a known representation for minutiae, treating the minusubject when only the two standard images tiae extraction and the fingerprint recognition of police databases are available as training as a 3D pattern matching problem instead of a samples. 2D one, obtaining top-level accuracy results.
Of course, these are just few examples of the many algorithms and techniques available in fingerprint matching. In fact, as highlighted in the survey of Peralta et al. [33], The increasing availability of technologies even if the best performing algorithms are for video-surveillance, combined to the need diferent, they are based on common features of unburdening authorities from the task of such as minutiae coordinates, angle, and type. checking hours of video recordings, boosted Which is, then, the role of deep learning in the attention of the research community toifngerprint recognition, given the maturity of wards the automatic detection of violence in the field and the good performance of tradi- videos. The violence and fight detection is tional Computer Vision-based algorithms? In considered a task of human action recognirecent years, deep learning-based techniques tion: specifically, it is a binary problem which have been proven useful to overcome some consists of recognizing the presence or the of the limitations of traditional techniques. absence of violence [44].
While traditional algorithms, such as those As violence detection is rooted in action presented, perform well on rolled and plan recognition, the early works are based on ifngerprints collected with dedicated sensors, Computer Vision techniques originally imthey failed on latent fingerprints, i.e. partial plemented for action recognition and can be ifngerprints unintentionally impressed on sur- categorized into two classes [45], using handfaces [35, 36, 37, 38]. To this end, Tang et crafted features to represent actions: al. [36], proposed to convert the traditional operations for minutiae extraction into a CNN • in local features-based techniques, the that can be trained end-to-end. Similarly, Cao representation of an action is computed et al. [38] presented a latent fingerprint recog- by using Points of Interest (POIs) across nition system based on CNNs. Li et al. [37] the frames of a video; also proposed a CNN-based architecture, but with a diferent objective: enhance latent fin- • in global features-based techniques, the gerprint images to be used for the fingerprint representation of an action is computed matching (performed with other applications). by evaluating characteristics from mul
Latent and partial fingerprint recognition tiple frames as a whole. is not the only open challenge addressed with Among the techniques which are based on deep learning in the field. In the use of finger- local features, Chen and Hauptmann [46] proprints for authentication, Lin and Kumar [39] posed MoSIFT, a technique that combines the presented a model based on CNN to learn dis- Scale-Invariant Feature Transform (SIFT) [47] criminative 3D representations of fingerprints with optical flow to represent the movement in contactless fingerprint recognition applica- of POIs. Xu et al. [45] evolved the use of tions. With the availability of high resolution MoSIFT by combining it with a non-parametric scanners, CNN-based architectures have been Kernel Density Estimation (KDE) to remove developed to recognize sweat pores in high redundant and irrelevant features. They achieved resolution fingerprints [ 40, 41]. Finally, deep good results on detecting person-to-person learning techniques are being investigated to ifghts on videos, using sparse coding to repdetect malicious attempt to authenticate via resent the extracted features. Instead, Deniz artificial fingerprints, for the development of et al. [48] proposed to compute acceleration anti-spoofing methods [42, 43]. from the power spectrum of adiacent frames to detect a large variation of speed, obtain- mance in both the Hockey Fight (96% accuing results comparable to MoSIFT, but with a racy) and Crowd Violence (98%) datasets. In faster algorithm. addition to 3D CNNs, also the ConvLSTM ar
Concerning the techniques based on global chitecture [56] has been proven efective in features, Hassner et al. [49] proposed the com- violence detection. To this end, Sudhakaran putation of the Violence Flows (VIF) descrip- and Lanz [57] proposed to aggregate the spators, an evolution of optical flow which com- tial information extracted from the frames putes the changes in the magnitude of flow by 2D CNNs with a ConvLSTM, achieving a vectors, obtaining promising results on the de- 97.1% accuracy on the Hockey Fight dataset, tection of violence in crowds. Gao et al. [50] and 94.5% on the Crowd Violence dataset. added to the VIF the orientation of the flow Therefore, deep learning-based techniques vector, proposing OVIF, improving the perfor- demonstrated their accuracy on datasets which mance on the detection of person-to-person are traditional in literature such as the Hockey ifghts, but with a lower accuracy on crowd Fight and Crowd Violence. However, there is violence. still ongoing research to validate their robust
Deep learning contributed to advance the ness against false positives [58], and with real violence detection field by overcoming some surveillance camera footages [59]. of the limitations of the optical flow, such as discontinuities and camera motion, and by getting very good performance in person-to- 5. Conclusions person fights and crowd violence with the same model. Specifically, 3D CNN have been We presented a short survey about deep learnproven capable in learning spatio-temporal ing applications for three application domains information, i.e. features which represent the connected to law enforcement: Face Recognimotion information in a video, in addition to tion, Fingerprint Recognition, and Violence the spatial information in a single frame. For Detection. These three domains have some example, Ding et al. [51] presented a 9-layer common characteristics. In fact, early meth3D CNN for violence detection, obtaining a ods to the computerization of related tasks are 91% accuracy on the Hockey Fight dataset [52]. all rooted in Computer Visions, using techSimilarly, Li et al. [53] with a 10-layer 3D CNN niques such as Principal Component Analyalternating dense and transitional layers after sis, Image Binarization and Thinning, Optical a convolutional layer, achieved 98.3% accu- Flow, etc. However, the use of deep learnracy on the Hockey Fight dataset, and 97.2% ing techniques, such as Convolutional Neural on the Crowd Violence dataset [49]. Trans- Networks (2D and 3D) and ConvLSTMs, sigfer learning approaches based on 3D CNN nificantly improved the accuracy of automatic also demonstrated good performances. For applications dealing with Face Recognition, example, in our previous work [44], we used Fingerprint Recognition, and Violence DetecC3D [54], a 3D CNN pre-trained to classify tion. sport categories, as a feature extractor, and a While some of these deep learning techSupport Vector Machine (SVM) classifier, with niques are being integrated in production sysa 98.5% and a 99.2% accuracy on the Hockey tems, at least for Face and Fingerprint RecogFight and the Crowd Violence respectively. nition1, there is still the need to investigate Similary, Ullah et al. [55] used C3D as a fea- 1See, for example, the Italian system SARI, an extenture extractor, but followed by fully connected sion of an Automated Fingerprint Identification Systems layers for classification, with a good perfor- (AFIS) which supports Face Recognition [60]. their impact in real world applications. For example, concerning Face Recognition, there is a lack of research in understanding the efectiveness of face identification when only the two mugshots per subject commonly stored in law enforcement databases are available for training. Concerning Fingerprint Recognition, research is ongoing to get an efective extraction of minutiae from latent fingerprint images, which are available in crime scenes.
Concerning Violence Detection, the accuracy of deep learning techniques with real surveillance cameras and their robustness to false positives are among the objectives of current research.
Moreover, to be efective in real applications, deep learning based techniques, as Artificial Intelligence in general, need to take into account concrete real time performances.
In fact, as pointed out in [61], an intelligent answer preserves its importance only if given in time. Finally, as the evidence collected using AI should be explainable to a judge in a court [13], also Explainable AI (XAI) methods, capable to provide human understandable explanations of their results [62], should be investigated in the presented application domains, to avoid the use of deep learning techniques as mere “black boxes”.
The presented research has been part of the Memorandum of Understanding between the Università Politecnica delle Marche, Centro “CARMELO” and the Ministero dell’Interno, Dipartimento di Pubblica Sicurezza, Direzione Centrale Anticrimine della Polizia di Stato.
[2] N. Falcionelli, P. Sernani, A. Brugués,
D. N. Mekuria, D. Calvaresi, M. Schumacher, A. F. Dragoni, S. Bromuri, Indexing the event calculus: Towards practical human-readable personal health systems, Artificial Intelligence in Medicine 96 (2019) 154–166. doi:10.
1016/j.artmed.2018.10.003. [3] N. Falcionelli, P. Sernani, A. Brugués,
D. N. Mekuria, D. Calvaresi, M. Schumacher, A. F. Dragoni, S. Bromuri, Event calculus agent minds applied to diabetes monitoring, in: Autonomous Agents and Multiagent Systems, Springer International Publishing, 2017, pp. 258–274. doi:10.1007/9783-319-70887-4_3. [4] A. F. Dragoni, S. Animali, Maximal consistency, theory of evidence, and bayesian conditioning in the investigative domain, Cybernetics and Systems 34 (2003) 419–465. doi:10.1080/ 01969720302863. [5] N. Falcionelli, P. Sernani, D. Mekuria,
A. F. Dragoni, An event calculus formalization of timed automata, in: Proceedings of the 1st International Workshop on Real-Time compliant MultiAgent Systems co-located with the Federated Artificial Intelligence Meeting, volume 2156 of CEUR Workshop Proceedings, 2018, pp. 60–76. URL: http://ceurws.org/Vol-2156/paper5.pdf . [6] A. F. Dragoni, P. Giorgini, L. Serafini,
Mental states recognition from communication, Journal of Logic and Computation 12 (2002) 119–136. doi:10.1093/ logcom/12.1.119. [7] P. Sernani, A. Claudi, A. F. Dragoni,
Combining artificial intelligence and netmedicine for ambient assisted living: A distributed bdi-based expert system, International Journal of E-Health and Medical Communications 6 (2015) 62–76. doi:10.4018/IJEHMC.2015100105. D. Koracin, R. McMahan, J. Jerald, tional long short-term memory, in: H. Zhang, S. M. Drucker, C. Kamb- 2017 14th IEEE International Conferhamettu, M. El Choubassi, Z. Deng, ence on Advanced Video and Signal M. Carlson (Eds.), Advances in Vi- Based Surveillance (AVSS), 2017, pp. 1–6. sual Computing, Springer International doi:10.1109/AVSS.2017.8078468. Publishing, 2014, pp. 551–558. doi:10. [58] M. Bianculli, N. Falcionelli, P. Sernani, 1007/978-3-319-14364-4_53. S. Tomassini, P. Contardo, M. Lombardi, [52] E. Bermejo Nievas, O. Deniz Suarez, A. F. Dragoni, A dataset for automatic G. Bueno García, R. Sukthankar, Vio- violence detection in videos, Data in lence detection in video using computer Brief 33 (2020) 106587. doi:10.1016/j. vision techniques, in: P. Real, D. Diaz- dib.2020.106587.
Pernil, H. Molina-Abril, A. Berciano, [59] M. Cheng, K. Cai, M. Li, RWF-2000: an W. Kropatsch (Eds.), Computer Anal- open large scale video database for viysis of Images and Patterns, Springer olence detection, CoRR abs/1911.05913 Berlin Heidelberg, Berlin, Heidelberg, (2019). URL: http://arxiv.org/abs/1911. 2011, pp. 332–339. doi:10.1007/978- 05913.
3-642-23678-5_39. [60] E. Sacchetto, Face to face: il com[53] J. Li, X. Jiang, T. Sun, K. Xu, Eficient vi- plesso rapporto tra automated olence detection using 3d convolutional facial recognition technology e neural networks, in: 2019 16th IEEE processo penale, La legislazione International Conference on Advanced penale (2020) 1–14. URL: https: Video and Signal Based Surveillance //iris.unito.it/retrieve/handle/2318/ (AVSS), 2019, pp. 1–8. doi:10.1109/ 1758754/668686/Sacchetto-finale.pdf.
AVSS.2019.8909883. [61] A. F. Dragoni, P. Sernani, D. Calvaresi, [54] D. Tran, L. Bourdev, R. Fergus, L. Torre- When rationality entered time and besani, M. Paluri, Learning spatiotemporal came real agent in a cyber-society, in: features with 3d convolutional networks, Proceedings of the 3rd International in: 2015 IEEE International Conference Conference on Recent Trends and Apon Computer Vision (ICCV), 2015, pp. plications in Computer Science and In4489–4497. doi:10.1109/ICCV.2015. formation Technology, volume 2280 of 510. CEUR Workshop Proceedings, 2018, pp. [55] F. U. M. Ullah, A. Ullah, K. Muham- 167–171. URL: http://ceur-ws.org/Volmad, I. U. Haq, S. W. Baik, Violence 2280/paper-24.pdf. detection using spatiotemporal features [62] D. Doran, S. Schulz, T. R. Besold, What with 3D convolutional neural network, does explainable AI really mean? a Sensors 19 (2019) 2472. doi:10.3390/ new conceptualization of perspecs19112472. tives, in: Proceedings of the First [56] X. Shi, Z. Chen, H. Wang, D. Yeung, International Workshop on ComprehenW. Wong, W. Woo, Convolutional LSTM sibility and Explanation in AI and ML network: A machine learning approach 2017, volume 2071 of CEUR Workshop for precipitation nowcasting, CoRR Proceedings, 2017, pp. 15–22. URL: abs/1506.04214 (2015). URL: http://arxiv. http://ceur-ws.org/Vol-2071/CExAIIA_ org/abs/1506.04214. 2017_paper_2.pdf. [57] S. Sudhakaran, O. Lanz, Learning to detect violent videos using convolu