Algorithm for search and recognition of fire by means of unmanned aerial vehicles Alexander Mezhenin1[0000−0002−7150−9811] , Vera 2[0000−0002−8707−9510] Izvozchikova , Vladimir Shardakov2[0000−0001−6151−6236] , 2[0000−0002−2193−9389] Veronika Zaporozhko , and Vladimir Borisov3[0000−0002−6312−4620] 1 ITMO University, Kronverksky Ave. 49, St. Petersburg, Russia mejenin@mail.ru 2 Orenburg State University, Ave. Pobedy 13, Orenburg, Russia werovulv@inbox.ru 3 Orenburg branch of the Russian State University of Oil and Gas named after I. M. Gubkin, st, of Young Leninists 20, 460047, Orenburg, Russia borisov-vps@mail.ru Abstract. The paper considers the possibility of providing early detec- tion of fires in oil and gas wells through the use of the RGB palette, as well as intelligent decision-making to eliminate the consequences of fire using the possibility of remote dynamic monitoring based on robotic platforms. The result was a simulated prototype of a hardware-software complex (APC), which allows monitoring of the territory based on the distributed processing of large stream of graphical information in real time (including the exact coordinates of the fire, its size, direction of fire, proximity to settlements). For the most effective detection and pre- vention of growth of natural and man-made fires. The proposed agro- industrial complex will be implemented on the basis of modern technolo- gies: robotics, GPS-technologies, GIS-technologies, client-server Internet technologies, video surveillance, intelligent information technologies. Ac- curate automated determination of coordinates with the help of GPS-fire signal makes it possible to timely proceed to the localization and elimi- nation of the fire, thereby preventing and reducing the negative impact on people, nature, wildlife, reducing the damage caused by the fire. The created software and hardware complex will allow to quickly develop and make the most optimal decisions on the direction of fire brigades and fire equipment to the places of fire, especially in particularly remote areas. Keywords: monitoring of territories · robotic platforms · detection of natural and man-made fires · unmanned aerial vehicles · fire coordinates. Copyright c 2019 for this paper by its authors. Use permitted under Creative Com- mons License Attribution 4.0 International (CC BY 4.0). 2 A. Mezhenin et al. 1 Introduction Fires at gas and oil facilities are a disaster that causes incalculable material, moral and environmental losses. Analysis of fire statistics shows that every fourth fire is accompanied by the explosion and the subsequent development of the combustion in the area of 5000 m2 . If a fire occurs without an explosion, the fire area in most cases is 500 m2 , and the maximum area reaches 3000 m2 , so its temporary detection of natural and man-made fires is an important factor in ensuring their rapid extinguishing. Currently, this function is performed by drones and unmanned aerial vehicles. In recent years they have become increasingly popular in the detection of fires in forests and fields. They are especially necessary in hard-to-reach places where the definition of fire without the use of aviation is impossible. This task will effectively manage the unmanned aerial vehicle of the type ”copter”. The main tasks of using quadrocopters in emergency situations are to determine the lo- calization of areas affected by natural disasters, conduct cartographic surveys, Search for missing people. Centralized control of the unmanned aerial vehicle is carried out by a human operator (one to one). The structure of the control system: - human operator; - remote terminal; - aircraft with on-Board intelligent system of information processing. The entire process of controlling the aircraft is carried out by the operator through the terminal, the on-Board information processing system displays the image on the operator’s screen, and he, based on his experience, makes the appropriate decision. The paper proposes to develop a prototype of an automated hardware and software complex, which in addition to performing the function of dynamic mon- itoring of the territory in real time provides a continuous analysis of the data coming from the sensors, and gives the operator an operational emergency alert when the parameters deviate from the specified norm, predicts possible emer- gencies, is able to calculate and form the optimal route to the source of the fire, i.e. is an expert intelligent system. The advantages of using a quadcopter for the monitoring of gas and oil wells: - ability to continuously monitor even the most remote and wild areas, in- cluding in any windy weather; - objectivity of the obtained data (absence of human factor-fatigue, falling asleep during visual observation of fires from specialized towers); - reducing the cost of one hour flight; - speed of data collection; - accuracy of data collection; - optimization of frames. Development of software and hardware complex 3 2 Related Work The analysis of modern applications and publications in the field of fire moni- toring in Russia and abroad has shown that the use of unmanned aerial vehicles (UAVs) for these purposes is in demand. However, this trend has not yet become widespread due to imperfect implementation approaches and the complexity of processing large amounts of data, as well as the following disadvantages: - not all systems have flexible integration to adapt the system to different subject areas; - high technical requirements for equipment; - have a closed program code. The relevance of this scientific problem is confirmed by theoretical and prac- tical works of foreign and Russian researchers. The General direction studying the problems of data mining is defined as ”Monitoring Wildfire UAVs”. The authors, led by Sharon Rabinovich [1], proposed a coordination strategy with a new methodology for estimating the periphery of a spreading fire using limited observations. The developed information-logical model of the information system displays the data of the subject area in the form of a set of information objects and links between them. A group of researchers from the Ivanovo fire and rescue Academy of the Ministry of emergency situations of Russia under the leadership of V. A. Smirnov considered a set of positive effects from the introduction of unmanned aerial vehicles into the monitoring system of forest fire situation, and also showed all the advantages that are possible when using unmanned aerial vehicles [2]. Scientists from America [3] have developed methods of planning the trajec- tory of unmanned air transport to monitor the boundaries of fires. The disad- vantage of the work is that the algorithm was not tested by the authors, but only an imitation of fire was proposed. The researchers of the University of Toronto [4] proposed a method for monitoring the occurrence based on the Kalman filter with the use of unmanned aerial vehicles. The authors conducted a fire simulation on the basis of which a method for assessing the behavior of forest fire propagation and the contour of the fire front was developed. Scientists Zhongjie Lin and Hugh H. T. Liu proposed a model for distributed topology-based optimization for joint monitoring of fires through the use of mul- tiple UAVs. The problem of interaction of several UAVs in a distributed network is formulated. A method is proposed that allows cooperation between UAVs at the group level [5]. The actual research is presented by the authors under the leadership of A. S. Vasilyev [6]. In their article, the scientists considered the key issues arising in the development of software and hardware complex of detection and monitoring of fires on the basis of an unmanned aerial vehicle by combining images, and also presented the architecture of the proposed software. In work [7] the model of the simulator on the basis of CAVE-technology is presented. The disadvantage of this model is the fact that the movement of the unmanned aerial vehicle must be monitored by the operator and place 4 A. Mezhenin et al. marks in the system where the fire is detected and describe the procedure for its suppression. Researchers Connie Phan and Hugh H. T. Liu describe a technique for con- trolling UAVs through mobile devices. The authors propose a methodology for the interaction of several aircraft through the use of a hierarchical platform [8]. Scientists from China presented the work on fire identification based on deep learning. The authors in their study took into account unmanned aerial vehicles equipped with global positioning systems (GPS), through which it is possible to provide direct georeferencing images, mapping the area with high resolution. The paper presents a 15-level self-learning architecture DCNN called ”F ireN et”. The disadvantage of such a system is the speed of processing a large amount of data [9]. In his scientific work V. Vipin proposed a method of classification of fire pixels using rule-based models in the RGB and YCbCr color space [10]. Researchers K. Angayarkkani and N. Radhakrishnan developed fuzzy rules for the use Of YCbCr color space for segmentation and fire image detection [11]. Scientist C. Yuan developed a set of algorithms for tracking fires, including median filtering, color space transformation [12]. These works are based on methods of image processing using elements of manual work, the results of which are highly dependent on the accuracy of the manually selected parameters. Statistical and machine learning methods are not actively used in this di- rection. The Gaussian mixture model (GMM) is used for flame detection [13]. However, using an empirical value for the number of mixtures may not lead to better results. The SVM classifier is used in [14]. It should be noted that when widely used feature descriptors such as the oriented gradient histogram (HOG) with space invariant feature transformation (SIFT) are used with these classifiers, the false alarm rate is not low enough. Satellite imagery is a common method of fire detection [15], but the long scanning period and low flexibility make fire detection difficult. Infrared thermo- graphic cameras are used to obtain thermal images of the terrain [16]. Through them it is possible to obtain reliable data on the distribution of heat for fire de- tection. However, most infrared thermographic imaging systems operate in the wavelength range of 0,75 to 100 microns. They find much less information about the environment on this strip. This information can also be very important, es- pecially when flammable and combustible materials are presented. In addition, according to Nyquist’s theorem, the recorded thermal image has a lower spa- tial resolution than the visible spectrum cameras. In addition, thermographic systems are quite expensive with high maintenance costs. 3 Experimental Part To get the coordinates of the quadcopter you need to have the coordinates of the UAV, the initial values of which are obtained by GPS. Next, we have a pixel- by-pixel shift of the x and y values. the shift Equation is shown in formulas (1) Development of software and hardware complex 5 and (2): x − Camera X= , (1) 2 −(y − Camera) Y = , (2) 2 where x and y are the coordinates of the transmitted images, the X and Y coordinates of the received image, Camera image produced on the camera quadracopter. To detect the source of fire, we use color analysis based on the RGB model. This model describes each color with a set of the following colors: red, green and blue. We describe the possibility of flame pixel detection for RGB color model by the following system:  R(x, y) > Rred  1 PK Rred = K · i=1 R(x, y)) (3) R(x, y) > G(x, y) > B(x, y)  where values Rx,y , Bx,y , Gx,y – value of red, blue and green colors in pixel on coordinates x, y, K is the total number of pixels, Rred is the average intensity of the red color. r, g and b are normalized components of RGB space defined by the following formulas: r = R/(R + G + B), (4) g = G/(R + G + B), (5) b = B/(R + G + B), (6) The algorithm for determining the ignition source is shown in fig. 1. According to the presented algorithm, the quadcopter transmits the image to a personal computer by scanning the marked GPS point. By using the RGB palette, the color that is closest to the ignition source (red) is determined. If the system detects a fire, it automatically notifies the operator, who decides to take the necessary measures. The results of the fire detection simulation are shown in fig. 2. Thus, the process of modeling the ignition of oil wells is carried out and the algorithm for early fire detection is described. The developed methods and ap- proaches for analyzing oil well ignition are planned to be used in forecasting and intellectual verification of upcoming changes in the heterogeneous infrastructure of the complex model, based on spatial and temporal distribution and proactive forecasting of the use of computing resources. 6 A. Mezhenin et al. Fig. 1. Algorithm for determining the source of fire Fig. 2. Simulation results of the detection of a fire Development of software and hardware complex 7 4 Conclusions The paper proposes an algorithm for color processing of RGB palette, which allows to control the territory on the basis of distributed processing of a large flow of graphic images in automatic real-time mode (including accurate determination of the coordinates of the fire, its area, the direction of the flame, the proximity of settlements, etc.). The socio-economic effect of the implementation of the proposed intelligent automated software and hardware complex is as follows: - remote access via the Internet to an interactive map of operational moni- toring, which allows you to obtain information about the state of the controlled territory; - the use of agriculture will significantly reduce the cost of special services to monitor fires, through the use of drones; - collection and analysis of operational data on the state of the fire situation. References 1. Rabinovich, S., Renwick, E., Gabriel H.: Toward Dynamic Monitoring and Suppress- ing Uncertainty in Wildfire by Multiple Unmanned Air Vehicle System. Journal of Robotics, 2018, p. 12 (2018). 2. Smirnov, V., Surovegin, A.: Application of unmanned aerial vehicles for monitoring forest fires. In: X International Scientific-Practical conference “EurasiaScience”, pp. 59–60, Moscow (2017) 3. Skeele, R., Hollinger, G.: Aerial Vehicle Path Planning for Monitoring Wildfire Fron- tiers. Field and Service Robotics, pp. 455–467 (2016) 4. Lin, Zhongjie, Liu, Hugh, Wotton, Mike: Kalman Filter-Based Large-Scale Wildfire Monitoring With a System of UAVs. IEEE Transactions on Industrial Electronics, 66(1), pp. 606–615 (2019) 5. Lin, Zhongjie, Liu, Hugh: Topology based distributed optimization for multi-UAV cooperative wildfire monitoring. Optim Control Appl Meth, 39, pp. 1530–1548 (2018) 6. Vasiliev, A., Krasnyashchikh, A., Korotaev, V., Lashmanov, O., Lysenko, D., Ne- narokomov, O., Shirokov, A., Yaryshev, S.: Development of software and hardware complex for forest fire detection by combining images. Proceedings of higher educa- tional institutions. Instrumentation, 55(12), pp. 50–55 (2012) 7. Kuznetsov, A., Musalimov, V., Saenko, A., Trambitsky, K.: Application of image analysis algorithms for detection of fires. Proceedings of higher educational institu- tions. Instrumentation, 55(6), pp. 51–56 (2012) 8. Connie, Phan, Liu, Hugh: A cooperative UAV/UGV platform for wildfire detection and fighting. In: 7th International Conference on System Simulation and Scientific Computing, p.21, Beijing, China (2008) 9. Zhao, Yi, Ma, Jiale, Li, Xiaohui, Zhang, Jie: Saliency Detection and Deep Learning- Based Wildfire Identification in UAV Imagery. Sensors, 18(3), p. 19, 2018. 10. Vipin, V.: Image processing based forest fire detection. Int. J. Emerg. Technol. Adv. Eng., 2, pp. 87-–95 (2012) 11. Angayarkkani, K., Radhakrishnan, N.: An Intelligent System for Effective Forest Fire Detection Using Spatial Data. Int. J. Comput. Sci. Inf. Secur., 7, pp. 202-–208 (2010) 8 A. Mezhenin et al. 12. Yuan, C., Liu, Z., Zhang, Y.: UAV-based forest fire detection and tracking using image processing techniques. In: Proceedings of the International Conference on Unmanned Aircraft Systems IEEE, pp. 639-–643, Denver, CO, USA (2015) 13. Töreyin, B., Dedeoglu, Y., Güdükbay, U., Cetin, A.E.: Computer vision based method for real-time fire and flame detection. Pattern Recognit. Lett, 27, pp. 49- –58 (2006) 14. Byoung, C., Kwang-Ho, C., Jae-Yeal, N.: Fire detection based on vision sensor and support vector machines. Fire Saf. J., 44, pp. 322-–329 (2009) 15. Li, Z., Nadon, S., Cihlar, J.: Satellite detection of Canadian boreal forest fires: Development and application of the algorithm. Int. J. Remote Sens, 21, pp. 3057— 3069 (2000) 16. Katayama, H., Naitoh, M., Suganuma, M., Harada, M., Okamura, Y., Tange, Y.: Development of the Compact InfraRed Camera (CIRC) for wildfire detection. SPIE Opt. Eng., 8, pp.745806-–745814 (2009)