The most important tasks of large (commercial) fish schools search technology using remote sensing (RS) data automated processing and analysis for solving the task of monitoring the oceans and seas to identify commercial fish accumulations that need to be solved have been formulated in [1]. For further research and development it is necessary to create a complete learning/test dataset of fish school images. Due to insufficient quantity of real RS images containing the objects of interest (OI) -fish schools, it is necessary to generate enough amount of artificially synthesized images, there OI would be present in various forms. Another important task is to identify areas of OI most likely location. This task can be solved in two ways: first is searching sea/ocean areas with favorable oceanographic and meteorological conditions using low-resolution RS images for further high-resolution space imagery. Another way is to analyze the movements of fishing vessels. During fishing, various types of fishing vessels perform specific maneuvers, this activity can be detected by analyzing the AIS data and than used to further analyze of these areas high resolution RS images in order OI detection. In this way it would be interesting trying to apply some approaches from adjacent areas for data analysis, for example, approaches of dynamic scaling [2] or queuing theory methods [3]. The solution of all these problems requires processing of large RS datasets, which requires significant computing resources. RS data allows its parallel processing [4,5]. Therefore, this task requires developing of special software and hardware complex that allow massively parallel data processing. To solve this problem, the experimental sample of the RS data processing complex [6], has been developed.
The space vehicles (satellites) that have appeared in the last 10 years with highresolution equipment provide high-quality RS images. The spatial image resolution above 1-2 m per pixel provides the tasks of the so-called object search and identification for relatively small size (meters, tens of meters) objects. Given that typical commercial fish schools near the surface of the ocean or the sea (the so-called pelagic fish schools) are from 5-10 meters to 150-200 meters in size, they will be seen on high-resolution RS images as detectable and identifiable objects [7].
In the process of fishing areas monitoring, both the collected data and the results of processing are geocoded (geographically), and, accordingly, can be aggregated within a single geospatial database. It is characteristic that technologies of processing and analysis of geospatial data developing in recent years are based on a qualitative transition from a set of arrays of numerical characteristics to geospatial objects that have both geographic and temporal dynamics. A convenient user tool for accessing and managing this geospatial dataset is a specialized GIS-system [8], providing opportunities of data sampling request, analysis, editing, visualization, modeling, etc. The main important element of monitoring implementing is OI classifier, which allows to identify commercial fish schools effectively, by RS data analysing.
A set of methods and algorithms for OI searching, detecting, classifying and identifying is a key component in the process of processing historical and operational RS data. The result of the research, using the sequential application of these methods and algorithms is information about the presence of OI in a pre-designated search area, its geographical coordinates and characteristics.
The initial data for the entire processing are: -Data on the search area (coordinates of the vertex points of the polygon that limits the part of the fishing area, necessary to analysis); -Time range (start and end date and time, indicating the time interval); -Historical oceanographic and meteorological data, estimated to calculate a commercial fish school of a designated type finding probability; -Operational data of the fishing vessels movements in the given sea area.
Data processing is performed sequentially, since at each stage the results of the methods and algorithms of the previous stage are used.
The purpose of OI searching methods and algorithms is to obtain data on the most probable locations of OI in the area of analysis. As a result of the operation of the methods in and search algorithms of the OI, the coordinates of fragments of marine areas should be obtained to request the high-resolution RS data.
When searching for OI, two main methods and related algorithms are used:
-Method of OI searching, based on oceanographic meteorological characteristics; -Method of OI searching, based on fishing vessels activities.
It is assumed that OI search on oceanographic meteorological characteristics (preliminary search for areas with high probability of containing OI) is carried out as follows.
There is a certain number of zones (in our case -the squares of the explored areas), each of which is assigned a certain vector-tag (a set of numerical characteristics containing oceanographic and meteorological parameters). Each of the squares belongs to the class "0" (it can not serve as a place for the appearance of a fish school) or "1" (it can serve as a place for the appearance of a fish school).
This problem is a typical classification problem. XGBoost classification algorithm is used from the family of algorithms "boosted trees" ("forced trees"). This algorithm over the past 1-2 years has been widely used due to its high efficiency. According to a lot of researchs, when performing tests on a wide variety of data, the task of classifying data (distribution of 2 or more classes) using this algorithm most often shows the highest quality scores, in particular, the smallest classification error in the AUC-ROC estimate (area under the error curve).
The XGBoost algorithm based on the procedure of sequentially constructing a composition of algorithms for classifying trees. The questions of the program implementation of this algorithm have been studied in sufficient detail and are not considered here. A detailed description of the algorithm and its implementation can be found in [9].
Algorithm learning (that is, the calculation of the value of the main parameters of the classification algorithm XGBoost) is based on pre-collected historical data for this search area. During the analysis of the current situation, the algorithm obtains current (up-to-date) data on the search area and classifies the squares of the explored sea areas, assigning them the values 0/1.
Such an algorithm allows you to select areas that are promising for more detailed analysis, in particular using sophisticated segmentation, classification and detection techniques.
In the field of developing methods and algorithms for image processing, the task of objects detecting is one of the most urgent in view of the wide possibilities of applied applications. That is why the history of the development of algorithms and methods for detecting objects has a long period of several decades. The classic delivery of the detection task is the processing of some visual scene, fixed in the form of a digital image (data array), where there is some background, on top of which one or many objects are represented; objects may also be absent.
In the vast majority of cases, objects of several different types can appear on the image. In this case, object detection can be performed simultaneously with their classification. When implementing a simple detection, all types of objects that must be recognized can be combined into one class.
In this form, the detection task is to recognize the presence on the image of an object of a given type with a certain probability and to predict its position on the picture in the form of a corresponding bounding box. In this case, the object can lie anywhere in the image and can have any size (scale). In some cases (as in the problem solved in the study), additional processing of images may be required for the purpose of segmentation and detection of the boundaries of objects.
The task of image segmentation, generally speaking, is more complex than the task of detecting (detecting) objects. Segmentation is understood as the division of an image into areas corresponding to different objects. It is required that the constructed areas as accurately as possible cover the objects of the image, taking into account their complex shape and the inevitable overlays.
Images obtained by remote sensing of the oceans and seas are characterized by the presence of images of objects of various nature on them. Such objects can be:
atmospheric fronts, clouds; condensation traces of aircraft in the atmosphere; zones of water surface disturbance; zone of ice accumulations in arctic latitudes; elements of the seabed relief; drifting phytoplankton and zooplankton; commercial fish schools; -Oil stains; zones of fishing vessels activity, etc.
The images of such objects are areas characterized by certain textural features and having fuzzy blurred boundaries. In addition, segmentation is complicated by the fact that the image is actually multi-layer, that is, objects of interest overlap. For example, the image of a fish cluster in a photograph can be partly covered by a shadow from the clouds and, at the same time, it is superimposed on visible algae and underwater relief elements from the air (see Figure 1). When large OIs are considered, one more technical circumstance appears that makes it difficult to directly apply known methods of detection, segmentation, and classification. When using high-resolution images, one OI can be displayed in several frames, some of which may be unavailable for some reason (for example, the boundary of the shooting area or the image is reached is damaged). In such cases it is useful to try to restore the shape of the boundary of the OI on the basis of available information.
The method of classification of squares of water areas, described in the OI Search, gives a preliminary prediction of the presence of fish accumulations, but for a more accurate analysis, deep image processing is necessary. In addition, several different types of objects can be present on the image, and for successful detection, you must first divide them among themselves, accurately defining the boundaries of objects.
These circumstances become a significant limitation for the application of the following well-known methods of image segmentation.
1. Methods based on the clustering of image points; methods based on color and brightness histograms and the choice of threshold values; the "watershed" method.
To the problem under consideration, these methods are poorly applicable due to overlapping of images of objects of interest. 2. Methods based on graph models: conditional random fields; Markov random fields.
Such methods are able to model overlapping objects of interest, but require a large marked training sample containing objects of interest of various types. Therefore, the image segmentation method based on the construction of the boundaries of the sought-for areas of interest objects was chosen.
In addition, it is possible to more accurately construct the boundary of the object of interest, making it possible to use the form of this boundary as one of the features for classifying objects of interest.
The restored boundary of the OI area also makes it possible to estimate the size of this area, and thus estimate the amount of the resource reserve.
OI classification is understood as the assignment of image fragments obtained through detection and segmentation procedures to one of the predefined types.
As shown in Table 1, in the subject domain, more than 10 varieties of OI can be identified, each with its own identifying features.
Using of detection methods by machine learning, for example, based on convolutional neural networks or conditional random fields, makes it possible to describe the desired regions in the form of sets of rectangles that limit the images of the objects sought. However, for a full solution of the problem posed in the study, it is important not only to identify the rectangular area in which the OIs are located, but also to accurately determine the boundaries of the alleged objects, since the shape of the object boundary in some cases is an important identifying feature used to classify the object of interest for a particular type. Among the objects under consideration, not only the fish accumulations that are of primary interest are represented on the photographs, but also other objects, and in many cases it is possible to distinguish the classes of objects among themselves precisely in the form of boundaries.
For example, in Figure 2 shows the accumulation of commercial fish and the zone of phytoplankton development. It can be seen that the boundaries of fish accumulations are sufficiently smooth and can have only a few special angular points. At the same time, the boundary of a typical zone of phytoplankton development is much more complex, it has more singular points, and the average degree of curvature is higher.
In addition, the border of this object of interest is an identification feature that serves to identify its unique characteristics (area, estimated volume of reserves, etc.).
We have shown above the role of constructing the boundaries of objects of interest in the procedures of image segmentation and extraction of features for subsequent classification.
For the initial delineation of boundaries, the well-known operators of Gabor, Canny, and Sobel are applied to the image. After applying this procedure, a system of lines appearing on the boundaries of various objects of interest appears on the image. As a rule, these lines intersect and interrupt. In Figure 3 shows a snapshot of an oil spill that was "cut" by a passing vessel and the boundaries identified after some filtration.
In Figure 4 shows a snapshot of the phytoplankton development area in the Barents Sea section with cloud cover. For successful processing of such situations, it is necessary to solve the task of tracing the boundary of the object of interest in conditions of overlaying the image of another object and its boundaries. This problem reduces to the task of reconstructing the interrupted curves in the image.
To solve this problem, an approach to solving the problem of recovering damaged images was used [10], which, in addition to the obtained sections of smoothed curves, takes into account also the original image itself, which improves the quality of the solution of the problem. The proposed method is universal, it can work both with a flat image and with a spherical image, i.e. defined in a region on a sphere of sufficiently large radius.
The apparatus of geometric control theory and sub-Riemannian geometry is used [11]. A corresponding mathematical model is constructed and a neurophysiological motivation for using just such a model is given.
In the problem of recognition of objects for today as the classification characteristics of the object, it is necessary to select the statistical characteristics formed at the output of some convolutional neural network (SNC) processing the image. Let's consider the method of solving problems of detecting and identifying OI on images using SNS.
Let the SNS handle some image and select a set of statistical characteristics (feature cards). The set of obtained maps is compared with the available set of reference character maps for all types of OI. The comparison is performed using a classification algorithm, usually also on the basis of a neural network. The result is a set of probabilities of belonging to the processed image to one of the types of OI; accordingly, the object class is determined by the greatest of probabilities.
If it is known that there is (with a high probability) an OI in the image being processed, then when solving the detection task, it is required to obtain the coordinates of the location of the image of the OI in the image as a fragment of the image completely containing the object; the boundaries of the fragment form a so-called bounding box or an object cover mask.
It should be noted that both detection and classification of an object in an image can rely on the same object identification features when analyzing an image, but in the case of detection, it is also required to determine the localization of identification features in the image coordinate system.
In order to avoid repeated recurrence of the operation of highlighting "feature cards" when processing images of the SNA, research and development of recent years are aimed at creating algorithms that realize detection and classification of objects simultaneously.
To define such a problem, in particular, the term "semantic segmentation" is used; In this task, when processing an image, its pixels are assigned to one of the interest classes (or to the background); a group of pixels of one class forms a mask that identifies the object (s) of the class.
A number of algorithms, based on this principle, have been analyzed recently. Conditionally they can be divided into two main groups: a) Algorithms based on the formation of "proposed regions" (proposed regions), such as: Regions With CNNs (R-CNN) [12]; Fast RCNN [13]; Faster RCNN [14]; YOLO [15]; SSD Single Shot Detector [16]. b) Algorithms based on the encoder-decoder architecture, such as: DenseNet [17];
SegNet [18]; U-net [19]. When analyzing the data of the algorithm for the purpose of their possible use in the development of this PNDI, the results of tests performed on the same type of computing equipment on a single test set of PASCAL VOC images were used. The comparative criteria included the following indicators [20]:
-Network training time -Time for searching and detecting objects on the test dataset -Object mask prediction accuracy -Accuracy of object class definition -CPU load -Graphic accelerator load -Memory usage.
Also, examples of application of these algorithms to image processing problems solutions of an applied character or with close data characteristics are considered in this paper.
As a result of the analysis, the following conclusions were drawn:
-On the standard data sets of the PASCAL VOC type (according to the testing data given in the literature or presented by the authors of the algorithms), the considered algorithms show close accuracy indicators (92-97% in the object classification, 85-95% in the object mask masking accuracy). -The type of network with the Unet architecture, which is often used in solving segmentation problems of an object with a fuzzy outline on an uneven background, for example, such as the scanning of human organs (c), can be considered as the closest application; processing of RS data [21], etc. -The algorithm, based on the architecture of the network type U-net, is able to use a smaller, in comparison with others, dataset for network "learning", which is critical for the task considered in this paper. OI detection and classification algorithm is presented in the following description: 1: Form π Γ π model images 2: Generate markup of object positions in images 3: Set the required Pixel Classification Accuracy 4: Cycle for π Γ π model images 5:
Get π masks of classification of image pixels (probabilities of π-class in range from 0 to 1) 6:
Get π binary masks of pixels belonging to the class by the Pixel Classification Accuracy criterion; The pixel belonging to the class takes the value 1 7:
Mask all adjacent pixels belonging to the same class in clusters 8:
Each cluster of pixels belonging to the same class is selected in a separate mask of the selected object of the class 9:
Calculate the accuracy of the mask of the object with the specified 10:
Calculate the accuracy of classification of objects selected by the mask 11: End of cycle 12: Remember the CNN (convolutional neural network) status parameters as a set of values π _ππ 13: Return π _ππ To carry out preliminary testing of this algorithm to solve the problem of the possibility of its application within the framework of the problem under consideration, its software implementation was implemented, which includes the following features:
As a coding part of the U-net network, it is suggested to use the implementation of the network with the VGG-19 architecture, discussed earlier in this chapter, previously trained on a large sample of objects from the standard ImageNet data set. The presence of a "pre-trained model" allows you to significantly reduce the process of configuring the network for the task at hand.
During the setup, i.e. additional training of the network, its parameters are tuned to typical objects of their selection for a given application task (in our case -objects of interest on the sea surface). This procedure is called "distillation" (transfer of knowledge) [22].
As the data sets for training, satellite images of objects of 4 classes on the sea surface, available in a small amount at the moment, were used: fish school; algae / plankton; pollution; empty sea surface without objects; as well as generated synthetic images of similar objects. To enlarge the sample of images, a so-called "augmentation" procedure was applied to each of them -image modification, resizing and rotation to a random angle. Thus, the number of images for each class was 100.
In the process of algorithm "learning", 90% of images (90 for each class) and 10 images were used for testing. The following metrics were used to assess the quality of the algorithm:
To assess the accuracy of the classification of the object in the image -the F-measure (F1 score), defined as: πΉ 1 = 2 * ππππππ πππ β’ ππππππ ππππππ πππ + ππππππ An estimation of the segmentation accuracy of an object (mask overlay) uses the intersection metric of IoU sets (or the Jacquard index), at the detection threshold of 0.5: πΌππ = π‘ππ’π_πππ ππ‘ππ£π π‘ππ’π_πππ ππ‘ππ£π + π πππ π_πππππ‘ππ£π + π πππ π_πππ ππ‘ππ£π It should be noted that at this stage of the assessment, given the available sample size, there is no sense in justifying and refining the parameters of these metrics, and they are used in the most general form.
The results of the assessment test are shown in Table 2: These estimates, as is understandable, are not satisfactory for using the obtained implementation at once to solve practical problems, but already at the stage of "aftertraining", an increase in the quality of the algorithm was obtained on a very small set of data.
In Fig. 5 presents the results of constructing object masks obtained as a result of additional network training using the available data set with subsequent testing on some test cases.
The mask of the object is constructed according to the following standard principle: for the final data array containing the probability of belonging to a certain pixel, those values that exceed a given threshold (in current estimation experiments it is 0.5) are selected for this class. Pixels assigned to an image in a given class are highlighted in color.
In the future it is expected to achieve the required quality indicators of the algorithm by improving its software implementation and, significantly, increasing the amount of data for learning the algorithm.
Within the scope of the task of OI searching and detecting, using oceans and seas RS data processing, a subtask was devised for constructing the OI areas boundaries. The following results were obtained:
The analysis of OI areas on oceans and seas RS data images has been carried out. The main features characterizing the OI are revealed, and the values of these characteristics for different types of OI are determined.
For cases of intersection of different OI in one image, a method for determining the OI boundaries is considered, based on the method of reconstructing curves on a spherical image.
The publication has been prepared with the support of the "RUDN University Program 5-100". The work is partially supported by state program 0077-2016-0002 Β«Research and development of machine learning methods for the anomalies detectionΒ».