A technique for detecting concealed objects in terahertz images based on information measure D M Murashov1, A A Morozov2 and F D Murashov3 1 Federal Research Center “Computer Science and Control” of RAS, Vavilov str., 44-2, Moscow, Russia, 119333 2 Kotel'nikov Institute of Radio Engineering and Electronics of RAS, Mokhovaya, 11-7, Moscow, Russia, 125009 3 Moscow Aviation Institute (National Research University), Volokolamskoe Shosse, 4, Moscow, Russia, 125993 e-mail: d_murashov@mail.ru, morozov@cplire.ru Abstract. In this paper, a new technique for detecting concealed objects in the images acquired by a passive THz imaging system is proposed. The technique is based on a method for mutual information maximization successfully used for image matching. For reducing computational expenses, we propose to analyze the mutual information at local maxima of the cross- correlation function computed in the Fourier domain. The proposed technique does not require parameter tuning. A computing experiment approved the efficiency of the proposed technique and the possibility of its implementation in security systems. 1. Introduction One of the areas of public safety protection is the development of concealed object detection systems in images obtained in the terahertz band (300 GHz - 3 THz). As a rule, these images are characterized by a low signal-to-noise ratio, low resolution, low contrast, and fuzziness of objects (see Figure 1 (a)). One of the approaches to solving the problem for detecting concealed objects in the terahertz images consists of image segmentation and recognition of selected objects. Many of the proposed segmentation methods are based on the assumption that there are three areas in the terahertz image (background, a human body, and hidden objects) characterized by their radiometric temperature range corresponding to the ranges of gray tone levels. Gaussian mixture models are usually used to represent the images. The authors of works [1-3] used these models as the basis for developing multi-level segmentation algorithms. Noise suppression algorithms are applied at the preprocessing stage to improve the segmentation quality. For example, the anisotropic diffusion algorithm and the nonlocal means (NL-means) algorithm are used in [1]. In paper [3], the authors developed a multi-level EM algorithm to localize objects. At the first level, the algorithm segments two objects: the background and the human body. At the second level, the EM algorithm highlights the hidden object over the body. In [4], the author proposed an algorithm based on the maximum likelihood method for recognizing hidden objects. In order to reduce the effects of noise and low contrast, it may be useful to combine images taken in different spectral ranges. To increase the performance of the concealed V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) Image Processing and Earth Remote Sensing D M Murashov, A A Morozov and F D Murashov objects detection, the authors of work [5] combined images obtained in the terahertz and visible spectral bands. In [6], the objects hidden under clothes are localized using the images obtained in four channels of a system operating in 3- and 8-millimeter wavelengths with vertical and horizontal polarization. A multilevel segmentation algorithm combines the k-means algorithm, the EM algorithm for estimating the parameters of a Gaussian mixture, and the Bayesian classifier. An alternative approach to object detecting in terahertz images is to use matching the template image of an object and the input image. In [7], a method based on the localization of the maximum of the correlation coefficient was applied to detect objects. In this paper, for detecting objects in terahertz images, we propose to apply an image matching method based on the mutual information maximization [8]. 2. Problem statement Suppose there is a set of images of objects that should be detected under the clothes of people, using a passive terahertz camera. Objects that are visible in the image should be compared with the templates, and it is necessary to make a decision about their identity or difference. Suppose there are N template images of objects U n , n  1,2,..., N . Let the image V be the input image. It is necessary to identify the object U n located in the image V . To solve such a problem, the segmentation of the input image is usually preliminarily performed, feature descriptions of the images of objects U n are formed, and then in the feature space, the separating surfaces between the classes of the objects under consideration should be found. The properties of terahertz images listed above complicate the segmentation of objects. Therefore, it is preferable to use methods that do not require segmentation. In this paper, we propose to solve the object detection problem as an image matching problem [9]. Let the similarity measure of the template and the objects in the image V be the criterion J (U n ,V ) . Then the problem can be formulated as follows. It is necessary to find a template image V which provides the maximum value of the similarity measure J (U n ,V ) : U m  arg max{ J (U n ,V )} , n  1,2,..., N . (1) Un Several methods that do not require segmentation of images are known, in particular, methods based on the maximization of the cross-correlation function values, maximization of the correlation coefficient [7,9], and maximization of the mutual information [8,10]. These methods do not require a priori knowledge about the relationship between the input image and the template as well as information on the statistical characteristics of the images. However, the correlation matching of terahertz images fails in some cases (see Figure 1). The mutual information is calculated directly from the joint distribution of the gray levels of these images and involves the data contained in the compared images more completely than the correlation parameters. The mutual information is a quantitative characteristic of the statistical dependence of images. The greater similarity of the template and the input image produces a greater value of the mutual information. In this work, the mutual information is used as a similarity measure to solve the problem (1). To apply the information-theoretic approach, a stochastic model of the dependency between the template image and the input image is necessary. Let the grayscale values in the compared images at the point x with coordinates ( x, y) be described as discrete random variables U (x) and V (x) with values u and v quantized into a finite number of levels K and L respectively. If images U (x) and V (x) represent the same scene, there is a relationship between the variables U (x) and V (x) . We will use a model similar to that proposed in [8,11]: V (Tr (x))  F (U (x))   (x), (2) where Tr is a coordinate transformation (for registered images V (Tr (x))  U (x) ); F is a grayscale value conversion function that simulates the relationship between two images of an object;  (x) is a random variable that simulates noise. Model (2) can be considered as a model of a discrete stochastic information system with the input U and the output V . V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 270 Image Processing and Earth Remote Sensing D M Murashov, A A Morozov and F D Murashov a) b) c) Figure 1. The result of the correlation image matching: (a) an image of a human with a hidden object obtained from the THERZ-7A complex; (b) the image of the template object; (c) the result of an erroneous correlation matching. The contour of the template is shown in red. The mutual information between the system input and output is defined by the expression: K L  p(uk , vl )  I (U ;V )   p(uk , vl )log  , (3) k 1 l 1  p(uk ) p(vl )  where the functions p(u ) , p(v) , and p(u, v) define one-dimensional and two-dimensional discrete probability distributions of the gray tone levels of the images U and V . If mutual information (3) is used as a similarity measure, the problem (1) can be formulated as follows. It is necessary to find an image of the template U n maximizing the criterion (3): U m  arg max{ I (U n ;V )} . (4) Un In the next section, an algorithm for solving the problem is proposed. 3. Comparison of the template images with the images obtained in the terahertz spectral band The direct computing of the mutual information values on the set of pixels corresponding to the intersection of the images U n and V at different positions of U n relative to V requires large computational expenses. Therefore, in practice, methods for accelerated computing of the mutual information magnitude are necessary. For example, in [8], a differential equation was obtained for searching for the extremum of function (3) in the image matching problem. In this paper, we propose to compute the mutual information values I (U n ;V ) at the points of the input image V corresponding to the local maxima of the cross-correlation function [9], where the content of the input image V matches the template image U n . The number of the local maxima is significantly less than the number of pixels in the input image V and the cross-correlation function of the images can be rapidly computed in the frequency domain. These circumstances make it possible to significantly speed up the computing of mutual information and, accordingly, solving the problem (4). Since the cross-correlation function is sensitive to changes in the amplitudes of the grayscale levels in the compared images, we will use contour preparations of the U n and V images for its computing because the boundaries of the objects have a fairly stable high grayscale value in the contour preparations. The cross-correlation function is determined as follows: cUn,V ( x, y)   CUn (s, t )CV ( x  s, y  t ) , s t (5) where CUn and CV are the contour preparations of the images U n and V , x and y are the spatial coordinates of the points of the image CV , s and t are the coordinates of the points in the intersection area of the images CUn and CV . Given the high noisiness of terahertz images, masks of V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 271 Image Processing and Earth Remote Sensing D M Murashov, A A Morozov and F D Murashov stochastic gradients [12] for signal-to-noise ratio SNR = 1 will be used to obtain contour preparations. The computing of the function (5) will be carried out in the frequency domain. Thus, the search algorithm on the terahertz images of an object corresponding to condition (4) includes the following operations: (a) smoothing the image V and obtaining contour preparations CU 1 and CV of the images U n and V respectively; (b) calculating the cross-correlation function of the contour preparations CU 1 and CV by the formula (7); (c) finding the points of local maxima of the cU 1,V ( x, y) function; (d) calculating the mutual information I (U1;V ) at the cU 1,V ( x, y) local maxima points; (e) finding the maximum of mutual information I max (U1;V ) ; (f) repeating steps (a) – (e) for the images U1 , ..., U N ; finding the maximum of the mutual information I max (U m ;V ) among the values of I max (U1;V ) , I max (U 2 ;V ) , ..., I max (U N ;V ) and the corresponding template image U m which will be the solution of the problem (4). It should be noted that an operation of impulse noise filtering [13] can be applied at step (a) for input image enhancement. The proposed algorithm has no parameters and does not require tuning. Figure 2 illustrates the main stages of localizing the objects. The input image captured by the THERZ-7A complex is shown in Figure 2 (a), and the image of the template object is given in Figure 2 (b). Contour preparations of these images are presented in Figures 2 (c) and 2 (d). Figure 2 (e) visualizes the cross-correlation function of the contour preparations. The figure shows the local maxima of the surface formed by the function cUn,V ( x, y) . The result of localizing the object in the input image is shown in Figure 2 (f). a) b) c) d) e) f) Figure 2. The main stages of localizing the objects: (a) image captured by the THERZ-7A complex; (b) the image of the template object; (c) the contour preparation of the input image; (d) the contour preparation of the template image; (e) visualized cross-correlation function of contour preparations cUn,V ( x, y) ; (f) the result of localizing the object in the input image, corresponding to the maximum of the mutual information I (U n ;V ) . 4. Experiment To estimate the applicability of the above approach to solve the problem of localizing and identifying concealed objects, a computational experiment was carried out. The experiment was organized as follows. We obtained template images of objects, which should be detected in images captured at the output of the terahertz imaging system. Template images are shown in Figure 3. a) b) c) Figure 3. Images of template objects: (a) Kalashnikov submachine gun (AK); (b) TT handgun; (c) Walther handgun. V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 272 Image Processing and Earth Remote Sensing D M Murashov, A A Morozov and F D Murashov We consistently placed objects under the clothes of a person and recorded image sequences using the THERZ-7A system. Using the proposed method, we compared the template images shown in Figure 3 (a) with frames of the obtained video sequences and fixed the values of the similarity measure (3). The shooting range was about 1.5 meters and varied slightly, so the template images were used with the constant value of the scale factor. The cross-correlation function and the mutual information are not invariant with respect to rotation. Therefore, it is also necessary to analyze the rotated and flipped variants of the template images. The results of the object localizing are shown in Figures 4 (a-e), and the computed values of mutual information are presented in Table 1. It follows from Figure 4 and Table 1 that concealed objects are successfully localized by the proposed method based on a combination of correlation matching and the mutual information maximum matching. From the data in Table 1, we can draw the following conclusion. When comparing images with concealed objects obtained at the output of the terahertz imaging system and the images of template objects, the maximum of the mutual information is achieved if the template corresponds to the concealed object. Consider an example. Suppose a terahertz image is obtained in which a hidden object is visible (for example, a Walther handgun). This image is matched with the AK submachine gun, TT handgun, and Walther handgun templates. The computed values of the mutual information between the matched images are given in the third row of Table 1. The maximum value of the mutual information is obtained when comparing the input image and the Walther template. The system then makes a decision that the concealed object is a Walther handgun. From this, it follows that the proposed method can be used to detect and recognize prohibited concealed objects in security systems. To eliminate false alarms, the calculated maximum value of the mutual information is compared with a predetermined threshold value. If the found maximum of the criterion (3) is less than the threshold, then the decision is made that the prohibited object is not found. The threshold value is determined empirically. a) b) c) d) e) Figure 4. The result of the matching images of template objects and the images of hidden objects: (a) TT handgun hidden under the clothes and its template; (b) hidden Walther handgun and its template; (c) hidden AK submachine gun and TT template; (d) hidden Walther handgun and TT template; (e) hidden AK submachine gun and its template. Table 1. The values of the mutual information computed when matching the template images and the images of concealed objects. Template objects Concealed object AK TT Walther AK 0.307398 0.295407 0.276332 TT 0.191471 0.404531 0.345492 Walther 0.162791 0.267373 0.342771 5. Conclusions The paper proposes a new algorithm for localization and identification of objects in images taken using the passive terahertz imaging system. The proposed algorithm combines matching methods based on maximizing the value of the cross-correlation function and the mutual information magnitude. To reduce computational expenses, the criterion of the mutual information maximum is V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 273 Image Processing and Earth Remote Sensing D M Murashov, A A Morozov and F D Murashov analyzed at points of local maxima of the cross-correlation function computed in the frequency domain. The proposed algorithm does not require parameter tuning. Future research will be aimed at improving the quality of detection and recognition of concealed objects using preprocessing of terahertz images and applying detection methods that are invariant to geometric transformations. 6. References [1] Shen X, Dietlein C R, Grossman E, Popovic Z and Meyer F G 2008 IEEE transactions on Image Processing 17 2465-2475 [2] Lee D S, Yeom S, Son J Y and Kim S H 2010 Optics Express 18 10659-10667 [3] Martinez O, Ferraz L, Binefa X, Gomez I and Dorronsoro C 2010 Concealed object detection and segmentation over millimetric waves images IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops 31-37 [4] Antsiperov V E 2016 Automatic target recognition algorithm for low-count terahertz images Computer Optics 40(5) 746-751 DOI: 10.18287/2412-6179-2016-40-5-746-751 [5] Kowalski M, Palka N, Piszczek M and Szustakowski M 2013 Acta Physica Polonica A 124 [6] Yeom S, Lee D, Lee H, Son J and Gushin V P 2013 Progress in Electromagnetics Research 39 169-180 [7] Trofimov V A, Trofimov V V, Shestakov I L and Blednov R G 2016 Concealed object detection using the passive thz image without its viewing Passive and Active Millimeter-Wave Imaging XIX (International Society for Optics and Photonics) 9830 98300E [8] Viola P and Wells III W M 1997 International journal of computer vision 24 137-154 [9] Gonzalez R C and Woods R E 2008 Digital Image Processing (New Jersey: Pearson Education, Inc) [10] Maes F, Collignon A, Vandermeulen D, Marchal G and Suetens P 1997 IEEE transactions on Medical Imaging 16 187-198 [11] Escolano F, Suau P and Bonev B 2009 Information theory in computer vision and pattern recognition (Springer Science & Business Media) [12] Jain A K 1989 Fundamentals of digital image processing (Englewood Cliffs, NJ: Prentice Hall) [13] Chervyakov N I, Lyakhov P A and Orazaev A R 2018 Two methods of adaptive median filtering of impulse noise in images Computer Optics 42(4) 667-678 DOI: 10.18287/2412-6179- 2018-42-4-667-678 Acknowledgments Authors are grateful to the Astrohn Technology Ltd and OOO ASoft who provided us with the THERZ-7A terahertz scanning device. This research was supported in part by the Russian Foundation for Basic Research (grants No 18-07-01295 and No 16-29-09626). V International Conference on "Information Technology and Nanotechnology" (ITNT-2019) 274