Automated visual assessment of the state of the earth and plants, wilting and pests of leaves, plant growth indicators, using technical vision, can be used as a basis in smart (precision) agriculture (SA). This article discusses a brief review of the literature on the use of computer (technical) vision (CV) for analyzing the condition of agricultural fields and plants growing on them. The introduction of vision systems into real agricultural production practice is associated with the development of complex mathematical approaches that must be resistant to a variety of technical and weather changes. It is necessary to overcome image changes caused by atmospheric conditions and daily and seasonal variations in sunlight. An approach is proposed, which is based on an RGB image obtained using a typical digital camera. The results are given on the use of CV systems in solving individual tasks of agricultural production.
An important task of smart agriculture is to monitor the condition of plants from the moment of planting, to ripening and harvesting [1]. This segment of research, based on CV, has still weakly penetrated SA production. Control of large-sized and spatially distributed plots of agricultural land is difficult and poorly implemented in modern farms by classical methods. The problem here is the inability to study the characteristics of soil and plants on frequent spatial and temporal grids. Information obtained by classical methods is rare in time and space and is more based on the experience of agronomic workers.
The problems of agricultural development are global. The concept of sustainable development of society includes in the list of the main issues that will need to be addressed, the following: population growth; energy sources and new fuel; food, including drinking water; depletion of resources; global climate change; the problem of pollution of air, water (oceans, seas, lakes, rivers and underground sources) and soil; the problem of limiting the production and consumption of toxic and harmful products. The solution to almost all of these issues, one way or another, is associated with the successful development of agriculture, improving the quality and quantity of products.
The solution of the above problems is possible with the help of a modern monitoring base based on the use of satellite remote sensing (SRS) data and information obtained from unmanned aerial vehicles (UAVs) [2]. The information obtained in this way is unique in that it has a high temporal and spatial resolution and is informative (the presence of multispectral information). It should be noted that the advantages of remote sensing in SX production are widely known, then information about the possibility of using UAVs is just beginning to develop. The information received from the UAV provides the ability to obtain relevant information with high periodicity (several times a day), the ability to cover large areas with high spatial resolution (up to several centimeters), to receive data in a uniform form (images in RGB or multispectral).
Since the main area of agricultural land in our country is located in areas of unsustainable and risky farming, under the conditions of observed global climate change, increasing the reliability of current information on current and expected weather conditions, assessing their impact on the state and formation of crop productivity, is of paramount importance.
Factors such as weather conditions and soil quality form the conditions for plant growth and therefore determine productivity.
Changes in these factors lead to a change in productivity, which forces us to develop methods for assessing the state of plants throughout the entire time, from planting to ripening.
The development of methods for classifying and assessing the state of plants is one of the main areas of research in the field of remote sensing of the earth using satellites, aircraft and unmanned aerial vehicles. Observations of the seasonal development of plants have been carried out in the interests of agriculture for a long time (more than 30 years). Satellite measurements of the spectral characteristics of radiation reflection in the visible and infrared regions of the spectrum and the values of vegetation indices obtained on their basis allow us to describe the seasonal dynamics of various types of plants. One of the necessary conditions for determining the phenological characteristics of plants (the time of the onset of various phenological phases, the duration of the growing season and others) is the availability of time series of data from continuous satellite observations. These measurements provide diverse and accurate information on the development of plants over time.
However, there are limitations associated with the frequency (preferably at least once a week) of obtaining the necessary data (cloud exposure). Therefore, the obtained satellite information is important, but rather it is some benchmark information for methods that allow you to regularly receive data on agricultural fields and plants located on them, namely unmanned aerial vehicles.
Phenological characteristics such as start of growing season (SOS) and end of growing season (EOS) end dates, its growing season length (GSL), and maximum of growing date season MGS), seasonal amplitude and some others, are widely used in solving problems of remote sensing of plant conditions.
Information on the start date of the growing season is characterized by the widest practical relevance in solving agricultural problems. In practice, field agricultural work, the beginning of the growing season is established by the date of planting, and the completion is associated with determining the time of ripening of the plant. The duration of the phenological cycle is determined by the type of plant. Air temperature, air and soil humidity, illumination, the presence of chemical trace elements necessary for plant growth in the soil are significant factors that determine the parameters of the phenological cycle.
Since the presence of moisture in the soil at the time of planting is the main limiting factor for plant growth, the beginning of the growing season is determined primarily by rainfall, as well as filling the fields with snow.
Modern digital cameras mounted on UAVs [3] have technical characteristics that allow solving many practical problems of agricultural production. This paper describes the software necessary to solve the problem of determining the state of crops in large areas. Obtaining this information is possible due to the ability to obtain a set of separate images of the SA territory in several spectral channels of a digital RGB camera or with additional spectral channels (near IR or IR spectral region) [3].
The presence of this information allows you to determine the Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). characteristics of plants from the calculation of various indices (vegetation, soil, etc.), as well as texture or color analysis.
Carrying out measurements at different time periods and obtaining multi-temporal data allows us to determine the dynamics of changes in the characteristics of crops, which is directly related to the performed agrotechnical work. Such studies clearly allow us to determine the area of the CX of territories where there is a deviation from average values, for example, due to the degradation of soil parameters close to the surface of the water horizon, etc. The presence of field images allows us to pose the problem of obtaining cartographic information of the state of the SA [4] of territories , taking into account the fact that UAVs can be equipped with high-precision geo-referenced devices. Such geospatial information allows us to solve the problem of combining images in space and time, as well as embed images in geographic information systems (GIS).
The presence of RGB color channels allows us to consider a digital camera as a spectral device, which makes it possible to make index calculations (Greeness) associated with the normalized difference index of vegetation NDVI (Normalized Difference Vegetation Index) [5]. The only difference is that the calculation of NDVI requires the presence of spectral information in the region of 0.7-1.5 μm, and the red channel of the digital camera is located in the region of 0.6-0.7 μm.
Nevertheless, selecting digital cameras with the necessary spectral channels, it is possible to obtain reliable information about the state of crops. Using the results of the calculation of the Greeness index in monitoring tasks of assessing the dynamics of characteristics, one can obtain spatio-temporal maps. The presence of a priori information about the characteristics of the soil and meteorological information allows us to build mathematical models of changes in the state of agricultural crops (amplitude and growth rate at different periods of vegetation).
Such information allows you to predict in advance a possible crop, type of harvest (given time and territory). Note that the frequency of the survey is an important parameter that determines the accuracy of the forecast and problem solving, control of the performed agricultural work, monitoring of the harvest, etc.
Vegetation indices make it possible to quantify the state of a plant at the time of measurement from a comparison of the values of the RGB spectral channels. It is known that in the blue-green region of the spectrum, plants have low reflectivity, which grows significantly in the red and near infrared regions of the spectrum. Accordingly, by comparing the values of the RGB channels in pixels corresponding to the plant, the state of the plant can be detected. Here are some vegetation indices that are calculated based on RGB channels: GCC - Green Chromatic Coordinate, RCC - Red Chromatic Coordinate, BCC - Blue Chromatic Coordinate, ExG - Excess Green, ExR - Excess Red and NDI Normalized Difference Index [6] .
The calculation of the GCC, BCC and RCC indices is carried
out according to the formulas:
GCC=Green/(Blue+Green+Red), (
Plants in the image were distinguished using empirically
selected thresholds for each of the indices (
In the whole variety of tasks from image processing to machine vision, there are no clear boundaries, however, processes of different levels can be distinguished here. The processes of the first level (pre-processing) include only methods and algorithms for image processing to reduce noise, increase contrast or improve sharpness, geometric transformations, etc. They are characterized by the fact that there are images at the input of the process and its output. The processes following the first are associated with more complex image conversion tasks, such as segmentation (dividing images into areas and selecting objects in them), a description of the objects and their compression to give them a convenient shape during further computer processing, as well as the classification (recognition) of selected objects. Note that these processes have images in the input, and attributes and features extracted from these images, such as borders, contours, and other distinguishing features of objects that are also images (of a different type), are output. Finally, processes of a different level are involved in understanding the set of many recognized objects, correlating them with existing templates or vice versa, forming templates. This shows that the natural transition from image processing to analysis of their content is the recognition area of individual objects in the images. Thus, what is called digital image processing is associated with processes having images at the input and output, as well as with the processes of extracting certain knowledge about objects located on the image.
The incoming images for processing during measurements from an unmanned aerial vehicle have specific features that are associated with the state difference in the level of illumination and the geometry of obtaining each image of one agricultural field. When shooting, which is usually carried out for several hours, the Sun changes its position and shadows from the relief, trees or clouds may appear, the level of illumination itself, the position of the unmanned aerial vehicle changes from the magnitude and direction of the wind. Therefore, at the preprocessing stage, a lot of work arises related to the preliminary preparation of images uniform in quality (geometry and illumination).
After the images are unified in terms of quality, it is necessary to select only plants on it, excluding from the processing areas that are not related to plants (roads, buildings, machinery, trees, shrubs, etc.). One way to solve this problem is image segmentation, i.e. the selection of areas that are odd in some ways in the image. Currently, the main classes of image segmentation include the following classes: 1. Morphological methods - are mainly used to work with binary (black and white) images. These methods allow you to extract image components, which can later be used to represent the shape of the object. 2. Threshold methods - have intuitive properties and are easy to implement. There are several main types of threshold segmentation, but only two are basic: the method with an optimal threshold and the method with an adaptive threshold. All other methods of this class are derived from the two mentioned algorithms. 3. Methods of growing areas - are algorithms that recursively perform the procedure for grouping pixels in a subregion according to predetermined criteria. One of the main methods here is the watershed method. 4. Texture methods - are based on the analysis on the diffuse (color, reflectivity) surface properties of the analyzed object. The methods presented in this category are sets of complex operators that can reduce the surface recognition process to the simple task of distinguishing brightness levels. Note that such approaches are dependent on image quality and, accordingly, in our task they have the condition of confirming the result that was obtained earlier using greeness approaches.
Remote methods for monitoring agricultural fields make it possible to quickly identify areas of fields affected by the disease, to determine the degree of plant ripening, etc. Identifying problems of plant development in the early stages of development significantly reduces the cost of labor and funds to obtain a planned result. There are two main approaches to solving the problem of identifying affected areas - spectrometric and optical. The spectrometric approach allows one to determine many problems of plant growth in the early stages of development, however, it leads to the emergence of a large amount of data that needs to be processed in a very short time, which requires the development of a computing base and data storage. Optical methods are being developed in parallel with the spectrometric approach and have the property of simplifying processing tasks, since the number of spectral channels is fixed (only three - RGB). It is clear that this has its limitations on the quality of identification of plants on the field, but it allows you to more quickly find the necessary solutions. There are various types of characteristics that can be used to identify plants: geometric, morphological and color, as well as their combinations, which can reduce the space of characters, which simplifies identification. The main task when processing images for plant identification is segmentation, i.e. selection of image objects (groups of pixels), homogeneous in their color or fractal characteristics, and assigning them to one or another predefined class.
To test the operability of the proposed algorithms, the authors conducted a model experiment related to growing plants in specially prepared room conditions. Observation of plant growth (wheat) was carried out daily at noon, for two months. During this time, the plant went through all stages of its vegetation cycle, from ripening to wilting. The obtained daily images were processed using the developed software, which was developed in the C # programming language using the SimpleCV technical vision libraries [http://simplecv.org]. The results of image processing associated with the selection of plants are shown in Fig. 1, from which it is clearly seen that the plant stands out well in the image.
The phenological development of plants is based on the hereditary rhythm and periodicity of physiological processes, called biological or phenological clocks. However, the onset of phenophases, the duration of their passage depends on a number of climatic factors, soil quality, as well as on human activity. However, despite the fundamental research of domestic and foreign scientists, phenological monitoring has not yet been introduced in industrial monitoring system at the level of the organizational structure of typical agricultural services. Until now, phenological monitoring, despite the fact that the emergence of digital forms of presenting observations of weather and seasonal phenomena, is more attached to humans. Phenological monitoring now refers to the system of organizing long-term observations and recording the dates of the onset of seasonal phenomena, the centralized collection and accumulation of information, its statistical and analytical processing of data on the timing of the onset of seasonal natural phenomena. The main tasks of phenology are the observation of various changes in the annual cycle of plant development and annual registration time of occurrence of these changes.
More specifically, phenological monitoring is associated not only with the detection of plant conditions, but also with the determination of the factors that determine this state, namely, climatic factors (temperature, temperature changes, precipitation, light exposure, cloudiness, etc.), soil factors (humidity, types and amounts of trace elements needed for plant nutrition), environmental factors (nearby industrial enterprises, city, etc.). It is necessary to build mathematical models that relate various factors to the state of vegetation. Our work is the first step on this path when determining the state of the plant by the phenological cycle, namely, growth rate, ripening time, etc.
To use the obtained results in agricultural practice, we calculated the number of pixels corresponding to the plant. The calculation results for the experiment are shown in Figs. 2 and 3. It is clearly seen that the plant at the growth stage increases the leaf area, then saturation occurs (the leaf area does not change) and then the plant withers, in which the leaf area decreases.
Fig. 1. The selection of plants in the image (on the left - the original image and on the right - the selected plant) Fig. 2. The vegetation cycle of the plant in a certain number of pixels corresponding to the plant
In parallel with the calculation of the number of pixels corresponding to the plant, we carried out the calculation of one of the vegetative indexes ExG (see Fig. 3). Note that the shape of the curve showing the number of pixels corresponding to the plant is different from the shape of the vegetation index. The figure clearly shows that the curve of the vegetation cycle has a complex structure, which is associated with meteorological conditions (open window, exposure to the Sun, etc.). This indicates the sensitivity of the indices to the effects of lighting and meteorological parameters, which can be directly used in practice.
It is possible to carry out a series of calibration measurements (image acquisition) with simultaneous fixation of various meteorological conditions. Based on the measurements obtained, it is possible to obtain the functions of changing vegetation indices depending on various conditions of plant growth. Then, if there is a weather forecast, it is possible to predict the state of the plants (taking into account the data in Fig. 2 and Fig. 3), which means that it is more accurate to make decisions, for example, about harvesting.
The article briefly describes the historical aspects of the development of precision farming and the appearance of UAVs in agricultural practice. The basic elements of technical vision necessary for the analysis of the state of SA plants are shown. It is said that in order to verify the received data from the UAV, it is necessary to take into account the meteorological conditions and changes in the illumination of sunlight. The results of processing the measurement data of test growing plants under room conditions are presented. It is shown that the proposed approach, which is based on the RGB image, allows to obtain information about the state of the plant over the entire time period of the vegetation cycle. It is proposed to offer this approach for practical use in real conditions of agricultural fields.