In this paper the problem of large (commercial) fish schools detection, using remote sensing (RS) images of sea an ocean surface analysis is considered. Considered objects of interest (OI) detection and identification methods and algorithms using high-resolution space imagery. Images obtained by RS of the seas and oceans are characterized by the presence of images of objects of various types and classes. A classifier for diferent types of OI is considered. Also considered the OI searching methods and algorithms whose goal is to obtain data on the most probable locations of OI in the area of analysis. Restore the OI boundaries section describes the problem of image segmentation - splitting the image into areas corresponding to diferent objects in such a way that the constructed regions cover the objects of the image as accurately as possible, taking into account their complex shape and inevitable overlays. The OI detection and classification algorithm is presented, based on the the U-net type network architecture, which is able to use a smaller (in comparison with others) dataset for network “learning”, which is critical for the task considered in this paper.
Copyright © 2018 for the individual papers by the papers’ authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. In: K. E. Samouylov, L. A. Sevastianov, D. S. Kulyabov (eds.): Selected Papers of the 1st Workshop (Summer Session) in the framework of the Conference “Information and Telecommunication Technologies and Mathematical Modeling of High-Tech Systems”, Tampere, Finland, 20–23 August, 2018, published at http://ceur-ws.org
The most important tasks of large (commercial) fish schools search technology using
remote sensing (RS) data automated processing and analysis for solving the task of
monitoring the oceans and seas to identify commercial fish accumulations that need to be
solved have been formulated in [
The space vehicles (satellites) that have appeared in the last 10 years with
highresolution equipment provide high-quality RS images. The spatial image resolution
above 1-2 m per pixel provides the tasks of the so-called object search and identification
for relatively small size (meters, tens of meters) objects. Given that typical commercial
ifsh schools near the surface of the ocean or the sea (the so-called pelagic fish schools)
are from 5-10 meters to 150-200 meters in size, they will be seen on high-resolution RS
images as detectable and identifiable objects [
In the process of fishing areas monitoring, both the collected data and the results of
processing are geocoded (geographically), and, accordingly, can be aggregated within
a single geospatial database. It is characteristic that technologies of processing and
analysis of geospatial data developing in recent years are based on a qualitative transition
from a set of arrays of numerical characteristics to geospatial objects that have both
geographic and temporal dynamics. A convenient user tool for accessing and managing
this geospatial dataset is a specialized GIS-system [
A set of methods and algorithms for OI searching, detecting, classifying and identifying is a key component in the process of processing historical and operational RS data. The result of the research, using the sequential application of these methods and algorithms is information about the presence of OI in a pre-designated search area, its geographical coordinates and characteristics.
2.
The initial data for the entire processing are: – Data on the search area (coordinates of the vertex points of the polygon that limits the part of the fishing area, necessary to analysis); – Time range (start and end date and time, indicating the time interval); – Historical oceanographic and meteorological data, estimated to calculate a commercial fish school of a designated type finding probability; – Operational data of the fishing vessels movements in the given sea area.
Data processing is performed sequentially, since at each stage the results of the methods and algorithms of the previous stage are used.
The purpose of OI searching methods and algorithms is to obtain data on the most probable locations of OI in the area of analysis. As a result of the operation of the methods in and search algorithms of the OI, the coordinates of fragments of marine areas should be obtained to request the high-resolution RS data.
When searching for OI, two main methods and related algorithms are used: – Method of OI searching, based on oceanographic meteorological characteristics; – Method of OI searching, based on fishing vessels activities.
It is assumed that OI search on oceanographic meteorological characteristics (preliminary search for areas with high probability of containing OI) is carried out as follows.
There is a certain number of zones (in our case – the squares of the explored areas), each of which is assigned a certain vector-tag (a set of numerical characteristics containing oceanographic and meteorological parameters). Each of the squares belongs to the class “0” (it can not serve as a place for the appearance of a fish school) or “1” (it can serve as a place for the appearance of a fish school).
This problem is a typical classification problem. XGBoost classification algorithm is used from the family of algorithms “boosted trees” (“forced trees”). This algorithm over the past 1–2 years has been widely used due to its high eficiency. According to a lot of researchs, when performing tests on a wide variety of data, the task of classifying data (distribution of 2 or more classes) using this algorithm most often shows the highest quality scores, in particular, the smallest classification error in the AUC-ROC estimate (area under the error curve).
The XGBoost algorithm based on the procedure of sequentially constructing a
composition of algorithms for classifying trees. The questions of the program implementation
of this algorithm have been studied in suficient detail and are not considered here. A
detailed description of the algorithm and its implementation can be found in [
Algorithm learning (that is, the calculation of the value of the main parameters of the classification algorithm XGBoost) is based on pre-collected historical data for this search area. During the analysis of the current situation, the algorithm obtains current (up-to-date) data on the search area and classifies the squares of the explored sea areas, assigning them the values 0/1.
Such an algorithm allows you to select areas that are promising for more detailed analysis, in particular using sophisticated segmentation, classification and detection techniques.
In the field of developing methods and algorithms for image processing, the task of objects detecting is one of the most urgent in view of the wide possibilities of applied applications. That is why the history of the development of algorithms and methods for detecting objects has a long period of several decades. The classic delivery of the detection task is the processing of some visual scene, fixed in the form of a digital image (data array), where there is some background, on top of which one or many objects are represented; objects may also be absent.
In the vast majority of cases, objects of several diferent types can appear on the image. In this case, object detection can be performed simultaneously with their classification. When implementing a simple detection, all types of objects that must be recognized can be combined into one class.
In this form, the detection task is to recognize the presence on the image of an object of a given type with a certain probability and to predict its position on the picture in the form of a corresponding bounding box. In this case, the object can lie anywhere in the image and can have any size (scale). In some cases (as in the problem solved in the study), additional processing of images may be required for the purpose of segmentation and detection of the boundaries of objects.
The task of image segmentation, generally speaking, is more complex than the task of detecting (detecting) objects. Segmentation is understood as the division of an image into areas corresponding to diferent objects. It is required that the constructed areas as accurately as possible cover the objects of the image, taking into account their complex shape and the inevitable overlays.
Images obtained by remote sensing of the oceans and seas are characterized by the presence of images of objects of various nature on them. Such objects can be: – atmospheric fronts, clouds; – condensation traces of aircraft in the atmosphere; – zones of water surface disturbance; – zone of ice accumulations in arctic latitudes; – elements of the seabed relief; – drifting phytoplankton and zooplankton; – commercial fish schools; – Oil stains; – zones of fishing vessels activity, etc.
The images of such objects are areas characterized by certain textural features and having fuzzy blurred boundaries. In addition, segmentation is complicated by the fact that the image is actually multi-layer, that is, objects of interest overlap. For example, the image of a fish cluster in a photograph can be partly covered by a shadow from the clouds and, at the same time, it is superimposed on visible algae and underwater relief elements from the air (see Figure 1).
When large OIs are considered, one more technical circumstance appears that makes it dificult to directly apply known methods of detection, segmentation, and classification. When using high-resolution images, one OI can be displayed in several frames, some of which may be unavailable for some reason (for example, the boundary of the shooting area or the image is reached is damaged). In such cases it is useful to try to restore the shape of the boundary of the OI on the basis of available information.
The method of classification of squares of water areas, described in the OI Search, gives a preliminary prediction of the presence of fish accumulations, but for a more accurate analysis, deep image processing is necessary. In addition, several diferent types of objects can be present on the image, and for successful detection, you must first divide them among themselves, accurately defining the boundaries of objects.
These circumstances become a significant limitation for the application of the following well-known methods of image segmentation.
1. Methods based on the clustering of image points; methods based on color and brightness histograms and the choice of threshold values; the “watershed” method. To the problem under consideration, these methods are poorly applicable due to overlapping of images of objects of interest. 2. Methods based on graph models: conditional random fields; Markov random fields.
Such methods are able to model overlapping objects of interest, but require a large marked training sample containing objects of interest of various types.
Therefore, the image segmentation method based on the construction of the boundaries of the sought-for areas of interest objects was chosen.
In addition, it is possible to more accurately construct the boundary of the object of interest, making it possible to use the form of this boundary as one of the features for classifying objects of interest.
The restored boundary of the OI area also makes it possible to estimate the size of this area, and thus estimate the amount of the resource reserve.
OI classification is understood as the assignment of image fragments obtained through detection and segmentation procedures to one of the predefined types.
As shown in Table 1, in the subject domain, more than 10 varieties of OI can be identified, each with its own identifying features.
Using of detection methods by machine learning, for example, based on convolutional neural networks or conditional random fields, makes it possible to describe the desired regions in the form of sets of rectangles that limit the images of the objects sought. However, for a full solution of the problem posed in the study, it is important not only to identify the rectangular area in which the OIs are located, but also to accurately determine the boundaries of the alleged objects, since the shape of the object boundary in some cases is an important identifying feature used to classify the object of interest for a particular type. Among the objects under consideration, not only the fish accumulations that are of primary interest are represented on the photographs, but also other objects, and in many cases it is possible to distinguish the classes of objects among themselves precisely in the form of boundaries.
For example, in Figure 2 shows the accumulation of commercial fish and the zone of phytoplankton development. It can be seen that the boundaries of fish accumulations are suficiently smooth and can have only a few special angular points. At the same time, the boundary of a typical zone of phytoplankton development is much more complex, it has more singular points, and the average degree of curvature is higher.
In addition, the border of this object of interest is an identification feature that serves to identify its unique characteristics (area, estimated volume of reserves, etc.).
We have shown above the role of constructing the boundaries of objects of interest in the procedures of image segmentation and extraction of features for subsequent classification.
For the initial delineation of boundaries, the well-known operators of Gabor, Canny, and Sobel are applied to the image. After applying this procedure, a system of lines appearing on the boundaries of various objects of interest appears on the image. As a OI Classificator rule, these lines intersect and interrupt. In Figure 3 shows a snapshot of an oil spill that was “cut” by a passing vessel and the boundaries identified after some filtration.
In Figure 4 shows a snapshot of the phytoplankton development area in the Barents Sea section with cloud cover. For successful processing of such situations, it is necessary to solve the task of tracing the boundary of the object of interest in conditions of overlaying the image of another object and its boundaries. This problem reduces to the task of reconstructing the interrupted curves in the image.
To solve this problem, an approach to solving the problem of recovering damaged
images was used [
The apparatus of geometric control theory and sub-Riemannian geometry is used [
In the problem of recognition of objects for today as the classification characteristics of the object, it is necessary to select the statistical characteristics formed at the output of some convolutional neural network (SNC) processing the image. Let’s consider the method of solving problems of detecting and identifying OI on images using SNS.
Let the SNS handle some image and select a set of statistical characteristics (feature cards). The set of obtained maps is compared with the available set of reference character maps for all types of OI. The comparison is performed using a classification algorithm, usually also on the basis of a neural network. The result is a set of probabilities of belonging to the processed image to one of the types of OI; accordingly, the object class is determined by the greatest of probabilities.
If it is known that there is (with a high probability) an OI in the image being processed, then when solving the detection task, it is required to obtain the coordinates of the location of the image of the OI in the image as a fragment of the image completely containing the object; the boundaries of the fragment form a so-called bounding box or an object cover mask.
It should be noted that both detection and classification of an object in an image can rely on the same object identification features when analyzing an image, but in the case of detection, it is also required to determine the localization of identification features in the image coordinate system.
In order to avoid repeated recurrence of the operation of highlighting “feature cards” when processing images of the SNA, research and development of recent years are aimed at creating algorithms that realize detection and classification of objects simultaneously.
To define such a problem, in particular, the term “semantic segmentation” is used; In this task, when processing an image, its pixels are assigned to one of the interest classes (or to the background); a group of pixels of one class forms a mask that identifies the object (s) of the class.
A number of algorithms, based on this principle, have been analyzed recently.
Conditionally they can be divided into two main groups:
a) Algorithms based on the formation of "proposed regions" (proposed regions), such
as: Regions With CNNs (R-CNN) [
SegNet [
When analyzing the data of the algorithm for the purpose of their possible use in the
development of this PNDI, the results of tests performed on the same type of computing
equipment on a single test set of PASCAL VOC images were used. The comparative
criteria included the following indicators [
Also, examples of application of these algorithms to image processing problems solutions of an applied character or with close data characteristics are considered in this paper.
As a result of the analysis, the following conclusions were drawn:
– On the standard data sets of the PASCAL VOC type (according to the testing data
given in the literature or presented by the authors of the algorithms), the considered
algorithms show close accuracy indicators (92–97% in the object classification,
85–95% in the object mask masking accuracy).
– The type of network with the Unet architecture, which is often used in solving
segmentation problems of an object with a fuzzy outline on an uneven background,
for example, such as the scanning of human organs (c), can be considered as the
closest application; processing of RS data [
OI detection and classification algorithm is presented in the following description: 1: Form × model images 2: Generate markup of object positions in images 3: Set the required Pixel Classification Accuracy 4: Cycle for × model images 5: Get masks of classification of image pixels (probabilities of -class in range from 0 to 1) 6: Get binary masks of pixels belonging to the class by the Pixel Classification
Accuracy criterion; The pixel belonging to the class takes the value 1 7: Mask all adjacent pixels belonging to the same class in clusters 8: Each cluster of pixels belonging to the same class is selected in a separate mask of the selected object of the class 9: Calculate the accuracy of the mask of the object with the specified 10: Calculate the accuracy of classification of objects selected by the mask 11: End of cycle 12: Remember the CNN (convolutional neural network) status parameters as a set of values _ 13: Return _
To carry out preliminary testing of this algorithm to solve the problem of the possibility of its application within the framework of the problem under consideration, its software implementation was implemented, which includes the following features:
As a coding part of the U-net network, it is suggested to use the implementation of the network with the VGG-19 architecture, discussed earlier in this chapter, previously trained on a large sample of objects from the standard ImageNet data set. The presence of a “pre-trained model” allows you to significantly reduce the process of configuring the network for the task at hand.
During the setup, i.e. additional training of the network, its parameters are tuned
to typical objects of their selection for a given application task (in our case — objects
of interest on the sea surface). This procedure is called “distillation” (transfer of
knowledge) [
As the data sets for training, satellite images of objects of 4 classes on the sea surface, available in a small amount at the moment, were used: fish school; algae / plankton; pollution; empty sea surface without objects; as well as generated synthetic images of similar objects. To enlarge the sample of images, a so-called “augmentation” procedure was applied to each of them — image modification, resizing and rotation to a random angle. Thus, the number of images for each class was 100.
In the process of algorithm “learning”, 90% of images (90 for each class) and 10 images were used for testing. The following metrics were used to assess the quality of the algorithm:
To assess the accuracy of the classification of the object in the image — the F-measure (F1 score), defined as:
It should be noted that at this stage of the assessment, given the available sample size, there is no sense in justifying and refining the parameters of these metrics, and they are used in the most general form.
The results of the assessment test are shown in Table 2:
Within the scope of the task of OI searching and detecting, using oceans and seas RS data processing, a subtask was devised for constructing the OI areas boundaries. The following results were obtained:
The analysis of OI areas on oceans and seas RS data images has been carried out. The main features characterizing the OI are revealed, and the values of these characteristics for diferent types of OI are determined.
For cases of intersection of diferent OI in one image, a method for determining the OI boundaries is considered, based on the method of reconstructing curves on a spherical image. (a) Basic OI class (fish school) (b) “Seaweeds” OI class (c) “Pollution” OI class
An algorithm for detecting OI on satellite images is developed. Experiments were carried out to expand the learning set of the neural network by synthesizing images of OI, using the modification of the initial set of real images. The experiments showed an improvement in the quality of OI recognition with increasing the volume of the training dataset replenished in this way.
The publication has been prepared with the support of the “RUDN University Program 5-100”. The work is partially supported by state program 0077-2016-0002 «Research and development of machine learning methods for the anomalies detection».