Processing of data streams in the detection of retroreflective objects Sergey M. Borzov1 , Oleg I. Potaturkin1 1 Institute of Automation and Electrometry SB RAS, Novosibirsk, Russia Abstract This work is devoted to the study of methods for detecting retroreflective objects (RRO), including optical and optoelectronic observation devices, based on the search for spatial anomalies in images formed in pulse laser location systems. Algorithms and software for detecting RRO have been developed. At the same time, special attention is paid to various methods of forming difference frames with periodic illumination, including preliminary replacing each pixel of the background images with the maximum value for the corresponding neighborhood. The effectiveness of the proposed methods for detecting retroreflective objects in conditions of intense sunlight, despite the presence of specular and diffuse reflecting surfaces in the field of view, is demonstrated. Keywords Remote detection of retroreflective objects, formation of difference images, search for spatial anomalies, pulse laser location. 1. Introduction Recently, much attention has been paid to a special class of optoelectronic laser location systems designed to detect retroreflective objects (RRO), such as photo and video equipment, optical and optoelectronic surveillance systems, various types of reflectors, etc. [1]. The operation of such systems is based on the use of the effect of retroreflection, which occurs when highlighting objects of these types, as a result of which in the field of view in the coordinates corresponding to the RRO, luminous points are formed against the background of the underlying surface, observed only from a position close to the position of the emitter. To increase the efficiency of the detection of RRO at long distances, the organization of synchronous operation of the laser emitter used to illuminate the scene and the photodetector with a high- speed shutter is carried out. Moreover, the illumination is carried out in short pulses, and the photodetector shutter opens for a time close to their duration with a delay (relative to the operation of the emitter) equal to the time of light propagation to the observed objects and back. Due to this, the photodetector perceives the radiation reflected from the objects of interest, and cuts off the light reflected from objects that are closer and further than the specified distance. As a shutter in devices operating on the basis of the described method, as a rule, a high-speed image intensifier is used. A detailed overview of such devices is provided in [2]. However, the use of image intensifier leads to a complication of their design and a decrease in the SDM-2021: All-Russian conference, August 24–27, 2021, Novosibirsk, Russia " potaturkin@iae.nsk.su (O. I. Potaturkin) Β© 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) 24 Sergey M. Borzov et al. CEUR Workshop Proceedings 24–31 spatial resolution of the generated images of the controlled scene. An alternative solution is to implement the shutter function using a special algorithm for controlling a two-section photodetector receiver with a lowercase transfer [3]. This approach is implemented in the hardware and software complex [4], the purpose of which is to check the performance of CCD arrays as part of active-pulse devices for detecting RRO without using an image intensifier tube or other external high-speed shutter in its design. To effectively solve the problem of detecting and recognizing RRO in real conditions, it is necessary to equip laser pulse location systems with built-in functions for processing recorded images, aimed at identifying fragments that potentially contain objects of interest. The pro- cessing performed in automatic mode is based on the analysis of the spatial distribution of the image brightness by a sliding window and the assessment of the presence of detectable objects at various points of the observed scene. At the same time, this assessment is traditionally performed by determining the correspondence between the analyzed fragments and the a priori description of the detected objects [5]. However, as practice has shown, this approach is not effective enough due to possible changes in illumination, low resolution of RRO images at a considerable distance and unknown angles of the observed objects, the presence of significant atmospheric distortions and a diverse dynamic background. Therefore, in this paper, to detect RRO in the field of observation, it is proposed to use a method based on the formation of a description of the background for each scene, followed by the search for fragments whose parameters do not correspond to the received description. This approach in the literature is called the search for anomalies [6], does not require the implementation of preliminary training procedures and allows you to detect in the field of view of the observation system atypical fragments in certain parameters on a non-uniform background. The aim of this work is to develop and experimentally study the effectiveness of digital data processing methods for pulse laser location systems designed for automatic detection of RRO by accumulating signals with a pulse illumination frequency and intra- and inter-frame processing of recorded images. 2. Software and hardware An optoelectronic stand consisting of sensing channels and synchronous image recording was used for experimental investigation of the RRO detection method [7]. The first of them provides illumination of the observed scene by narrow-band pulsed laser radiation, and the second-the formation of the resulting images. The control of the stand, as well as the visualization and processing of the received data, is carried out on a personal computer using an application specially developed for this purpose. When the scene is illuminated, part of the radiation reflected from the RRO returns to the light source, creating a light response (glare) of significantly greater intensity than in diffuse reflection from other objects. The location can be made both by single-frequency pulses, and their series with the accumulation of signals by the CCD receiver of the photodetector. The stand allows you to detect with high confidence RRO with a small light aperture at a significant range. As an example, Figure 1 shows an image of a scene containing a camera (the lens focus is 50 mm, the aperture value is f/16) on a natural background (soil cover, tree-shrub 25 Sergey M. Borzov et al. CEUR Workshop Proceedings 24–31 Figure 1: Images of the RRO on a natural background, registered by the stand. and herbaceous vegetation), recorded using an optoelectronic stand with the accumulation of a signal from 100 illumination pulses. It can be seen that the detection of RRO with such a small retroreflective index in this image can be performed accurately due to a significant increase in the signal-to-background ratio, even without the use of additional image processing. However, as practice has shown, with intense sunlight and the presence of objects of artificial origin in the field of view, the task is significantly more complicated. To increase the efficiency of the detection of RRO under these conditions, it is proposed to use periodic illumination of the scene, and at the stage of data analysis to perform inter-frame processing [8] of the recorded images and search for spatial anomalies [9] by a series of 2𝑛 frames of the sequence, where 𝑛 is the number of frames with and without illumination. Cross-frame processing is designed to form difference frames and is performed in two ways. The first one is implemented by a simple pixel-by-pixel subtraction of the 𝑖 frame with illumination and the 𝑖 frame without illumination [7]. 𝐼(π‘₯, 𝑦; 𝑖) = 𝐼1 (π‘₯, 𝑦; 𝑖) βˆ’ 𝐼0 (π‘₯, 𝑦; 𝑖). (1) The second one is proposed to be carried out by pre-replacing each pixel of the image obtained in the absence of laser illumination with the maximum value for its given neighborhood, i.e., by local processing. 𝐼(π‘₯, 𝑦; 𝑖) = 𝐼1 (π‘₯, 𝑦; 𝑖) βˆ’ max [𝐼0 (π‘₯, 𝑦; 𝑖)] . (2) πœ” In this case, the size of the neighborhood πœ” must correspond to the estimated size of the images of the detected objects. At the final stage of inter-frame processing, it is advisable to perform averaging of a series of difference images obtained in one way or another 1 βˆ‘οΈ 𝐼(π‘₯, 𝑦; 𝑖) = 𝑛𝐼(π‘₯, 𝑦; 𝑖). (3) 𝑛1 𝑖=1 26 Sergey M. Borzov et al. CEUR Workshop Proceedings 24–31 The procedure for searching for spatial anomalies from the generated difference images 𝐼(π‘₯, 𝑦) includes: β€” formation of the spatial distribution of the informative signal (οΈƒ )οΈƒ 1 βˆ‘οΈ 1 βˆ‘οΈ 𝐹 (π‘₯, 𝑦) = 𝐼(π‘₯ + π‘₯1 , 𝑦 + 𝑦1 ) βˆ’ 𝐼(π‘₯ + π‘₯1 , 𝑦 + 𝑦1 ) Β· π‘πœ” (4) π‘πœ” πœ” 𝑁Ω Ξ© where πœ” and π‘‚π‘šπ‘’π‘”π‘Ž are the central and peripheral zones of the fragment, π‘πœ” and 𝑁Ω are the number of pixels in them, respectively, π‘₯1 , 𝑦1 ∈ πœ”, Ω; β€” determination of local maxima in the resulting distribution 𝐹 (π‘₯, 𝑦) by processing with a sliding window πœ” 𝐹 (π‘₯, 𝑦), if 𝐹 (π‘₯, 𝑦) > maxπœ” πœ” (𝐹 (π‘₯ + π‘₯1 , 𝑦 + 𝑦1 )) {οΈ‚ 𝑀 (π‘₯, 𝑦) = (5) 0, else where π‘₯1 , 𝑦1 ∈ πœ”. Next, in the array 𝑀 (π‘₯, 𝑦), the mean πœ‡ and the standard deviation 𝜎 of non-zero values are calculated, i.e., the parameters of local maxima typical for this scene are determined (a description of the β€œbackground” class is formed), and the anomalous elements of the array 𝑀 (π‘₯, 𝑦) are selected using threshold processing 1, if 𝑀 (π‘₯, 𝑦) > πœ‡ + π‘˜πœŽ {οΈ‚ 𝑅(π‘₯, 𝑦) = (6) 0, else where π‘˜ is the threshold coefficient, which is usually 3 (it can be selected by the operator depending on the observation conditions). The position of non-zero elements in the binary array 𝑅(π‘₯, 𝑦) corresponds to the coordinates of the detected objects in the original image (π‘₯𝑖 , 𝑦𝑖 ). 3. Detection of RRO in difficult conditions To confirm the prospects of the proposed method of data processing, experimental studies were carried out on the detection of RRO in conditions of intense background radiation with the use of periodic illumination by a series of pulses and the formation of difference images. The measurements were carried out in the daytime under bright solar radiation. A number of retroreflective and mirror-reflecting objects were located in the field of view at different ranges. Among them are two RROs that are the target of detection: a television camera and a road sign with a retroreflective coating, located on the roadside at a distance of 1800 m. At the same distance, there were two objects that mirror the sunlight in the direction of the recording system (car elements). In addition, there were several other bright objects in the foreground and background of the sctne, including those with a reflective coating. Figure 2 shows images 𝐼1 (π‘₯, 𝑦) and 𝐼0 (π‘₯, 𝑦) containing the specified objects registered in the presence and absence of laser illumination. 27 Sergey M. Borzov et al. CEUR Workshop Proceedings 24–31 a b Figure 2: Images of the road situation registered in the presence (a) and absence (b) of laser illumination. It can be seen that in these images, the detected objects are difficult to distinguish without performing special processing. In the difference image (Figure 3, a), formed without accumulation according to expres- sions (1), (3) by subtracting successive images recorded in the presence and absence of laser illumination (𝑛 = 1), a large number of responses due to different objects are also observed. The brightest of them are numbered from 1 to 6. Here, 1, 2 β€” are caused by retro-reflection from the detected RROs, 3–6 β€” by mirror reflection of sunlight (3 β€” from the bumper of a car moving in the opposite direction on the roadway, 4 β€” from the body of a car at short range, 5, 6 β€” from the catafoats of a car in the foreground. It should be noted that performing inter-frame processing of the generated images even when using two frames leads to the suppression of signals caused by the reflection of solar radiation from stationary objects. The contrast of the detected RROs, as well as dynamic objects in the reflected solar radiation, increases significantly. For clarity, Figure 3, b shows the maximum column values in the resulting difference image. a b Figure 3: The difference image (a) formed from two frames registered in the presence and absence of laser illumination, and the maxima of signals in this image by columns (b). 28 Sergey M. Borzov et al. CEUR Workshop Proceedings 24–31 a b Figure 4: A difference image accumulated over 20 frames, with the result of detection for pixel-by-pixel processing (a) and the maxima of signals across columns (b). Performing procedures (4)–(6) under these conditions leads to the selection of a large number of false objects, in addition to the detected ones. The difference image 𝐼(π‘₯, 𝑦), formed according to (1), (3) from twenty frames (𝑛 = 10), the maximum column signals in the resulting difference image and the result of subsequent object detection are shown in Figure 4. Its analysis shows that the accumulation of a larger number of frames leads to a further increase in the brightness of the signals corresponding to the detected RROs (compared to the signals of static and dynamic objects in the reflected solar radiation). However, individual relatively slow dynamic objects with high brightness, which shift by no more than the size of the object during the recording of a series of images, still form quite intense responses (for example, object 4). To suppress them, it is proposed to use the formation of difference images with a preliminary substitution of each pixel of the image obtained in the absence of laser illumination for the maximum value in its given neighborhood πœ” in accordance with (2), (3). Such a difference image a b Figure 5: A difference image accumulated over 20 frames, with the result of detection for local process- ing (a) and maximums of the signal in columns (b). 29 Sergey M. Borzov et al. CEUR Workshop Proceedings 24–31 Table 1 Signals of objects with different methods of forming a difference image Method of forming a difference image 1 2 3 4 5 6 I 0.11 0.23 0.63 1.00 0.23 0.43 II 0.26 0.55 0.09 1.00 0.13 0.30 III 0.60 1.00 0.07 0.12 0.08 0.13 formed from twenty frames, and the result of subsequent detection according to (4)–(6), are shown in Figure 5, a. Figure 5, b, by analogy with the previous figures, shows the maximum column signals in the resulting difference image. In Table 1 given the relative signals of the objects in the resulting difference images for three ways of their formation: I β€” in accordance with the expressions (1) and (3) when 𝑛 = 1; II β€” in accordance with the expressions (1) and (3) when 𝑛 = 10; III β€” in accordance with the expressions (2), (3) when 𝑛 = 10. The obtained data confirm that the accumulation of frames with the preliminary use of the procedure for determining the maximum value for the neighborhood in the background frame allows you to additionally (several times) increase the relative signal level of detected objects (1), (2) and eliminate errors associated with the selection of bright objects (4) that have a slight shift from frame to frame. 4. Conclusion Algorithms are proposed and hardware and software tools are developed for complex intra- and inter-frame processing of image sequences for the detection of retroreflective objects based on the search for spatiotemporal anomalies using active pulse location by optoelectronic means. Their efficiency is demonstrated for the selection of RRO in conditions of intense sunlight in the presence of mirror and diffusely reflecting surfaces in the field of view, the signals of which are transmitted through the use of the proposed methods of digital data processing. Thus, the signals of static objects are suppressed by pixel-by-pixel inter-frame subtraction, and dynamic objects are suppressed by the accumulation of difference images. Of particular note is the method of forming difference frames with the preliminary replace- ment of each pixel of the background images by the maximum value in its neighborhood. Its use makes it possible to suppress the signals of relatively slow bright dynamic objects that shift by no more than the size of the object during the recording of a series of images. Acknowledgments The work was carried out with the support of the Ministry of Science and Higher Education as part of the implementation of the State Task No. 121022000116-0 in the Institute of Automation and Electrometry of the Siberian Branch of the Russian Academy of Sciences. 30 Sergey M. Borzov et al. CEUR Workshop Proceedings 24–31 References [1] Karasik V.E., Orlov V.M. Location laser vision systems. Moscow: Publishing House of the Bauman Moscow State Technical University, 2013. 478 p. (In Russ.) [2] Volkov V.G. Night vision devices for detecting glare elements // Specialnaya Tehnika. 2004. No. 2. P. 2–9. (In Russ.) [3] Golitsyn A.A, Seifi N.A. Active-pulse method of observation using a CCD photodetector with a lowercase transfer // Izvestiya Visshih Uchebnih Zavedenii. Priborostroenie. 2017. Vol. 60(11). P. 1040–1047. (In Russ.) [4] Alantyev D.V., Golitsyn A.A., Golitsyn A.V., Seifi N.A. Stand for the study of the possibility of using matrix photodetectors of the visible range as part of active-pulse observation devices // Opticheskii Jurnal. 2018. Vol. 85(6). P. 53-57. (In Russ.) [5] Roth P.M., Winter M. Survey of appearance-based methods for object recognition // Tech- nical Report ICG-TR-01/08. Institute for Computer Graphics and Vision, Graz University of Technology, Austria. January, 2008. 68 p. [6] Chandola V., Banerjee A., Kumar V., Anomaly detection: A survey // ACM Computing Surveys. 2009. Vol. 41(3). A. 15. 58 p. [7] Alantiev D.V., Borzov S.M., Kozik V.I., Potaturkin O.I., Uzilov S.B., Yaminov K.R. Exper- imental study of method of laser pulsed location for retroreflective objects detecting // Optoelectronics, Instrumentation and Data Processing. 2021. Vol. 56(1). P. 103–111. [8] Kirichuk V.S., Kosykh V.P., Popov S.A., Sinel’shchikov V.V. Suppression of a quasi-stationary background in a sequence of images by means of interframe processing // Optoelectronics, Instrumentation and Data Processing. 2014. Vol. 50(2). P. 109–117. [9] Borzov S.M. Detection of dynamic objects on the basis of space-time anomalies in video se- quences // Optoelectronics, Instrumentation and Data Processing. 2013. Vol. 49(1). P. 9–13. 31