=Paper=
{{Paper
|id=Vol-2485/paper40
|storemode=property
|title=Determination of Plant Phenological Cycle From RGB Images
|pdfUrl=https://ceur-ws.org/Vol-2485/paper40.pdf
|volume=Vol-2485
|authors=Mikhail Kataev,Kirill Yolgin
}}
==Determination of Plant Phenological Cycle From RGB Images==
https://ceur-ws.org/Vol-2485/paper40.pdf
Determination of Plant Phenological Cycle from RGB Images
M.Yu. Kataev1, K.S. Yolgin1
kmy@asu.tusur.ru
1Tomsk University of Control Systems and Radio Electronics, Tomsk, Russia
Automated visual assessment of the state of the earth and plants, wilting and pests of leaves, plant growth indicators, using technical
vision, can be used as a basis in smart (precision) agriculture (SA). This article discusses a brief review of the literature on the use of
computer (technical) vision (CV) for analyzing the condition of agricultural fields and plants growing on them. The introduction of
vision systems into real agricultural production practice is associated with the development of complex mathematical approaches that
must be resistant to a variety of technical and weather changes. It is necessary to overcome image changes caused by atmospheric
conditions and daily and seasonal variations in sunlight. An approach is proposed, which is based on an RGB image obtained using a
typical digital camera. The results are given on the use of CV systems in solving individual tasks of agricultural production.
Keywords: technical vision, mathematical methods, images, agriculture, image classification, unmanned aerial vehicle
The development of methods for classifying and assessing
1. Introduction the state of plants is one of the main areas of research in the field
of remote sensing of the earth using satellites, aircraft and
An important task of smart agriculture is to monitor the
unmanned aerial vehicles. Observations of the seasonal
condition of plants from the moment of planting, to ripening and
development of plants have been carried out in the interests of
harvesting [1]. This segment of research, based on CV, has still
agriculture for a long time (more than 30 years). Satellite
weakly penetrated SA production. Control of large-sized and
measurements of the spectral characteristics of radiation
spatially distributed plots of agricultural land is difficult and
reflection in the visible and infrared regions of the spectrum and
poorly implemented in modern farms by classical methods. The
the values of vegetation indices obtained on their basis allow us
problem here is the inability to study the characteristics of soil
to describe the seasonal dynamics of various types of plants. One
and plants on frequent spatial and temporal grids. Information
of the necessary conditions for determining the phenological
obtained by classical methods is rare in time and space and is characteristics of plants (the time of the onset of various
more based on the experience of agronomic workers.
phenological phases, the duration of the growing season and
The problems of agricultural development are global. The
others) is the availability of time series of data from continuous
concept of sustainable development of society includes in the list
satellite observations. These measurements provide diverse and
of the main issues that will need to be addressed, the following: accurate information on the development of plants over time.
population growth; energy sources and new fuel; food, including
However, there are limitations associated with the frequency
drinking water; depletion of resources; global climate change;
(preferably at least once a week) of obtaining the necessary data
the problem of pollution of air, water (oceans, seas, lakes, rivers
(cloud exposure). Therefore, the obtained satellite information is
and underground sources) and soil; the problem of limiting the important, but rather it is some benchmark information for
production and consumption of toxic and harmful products. The
methods that allow you to regularly receive data on agricultural
solution to almost all of these issues, one way or another, is
fields and plants located on them, namely unmanned aerial
associated with the successful development of agriculture,
vehicles.
improving the quality and quantity of products.
Phenological characteristics such as start of growing season
The solution of the above problems is possible with the help
(SOS) and end of growing season (EOS) end dates, its growing
of a modern monitoring base based on the use of satellite remote
season length (GSL), and maximum of growing date season -
sensing (SRS) data and information obtained from unmanned
MGS), seasonal amplitude and some others, are widely used in
aerial vehicles (UAVs) [2]. The information obtained in this way
solving problems of remote sensing of plant conditions.
is unique in that it has a high temporal and spatial resolution and
Information on the start date of the growing season is
is informative (the presence of multispectral information). It
characterized by the widest practical relevance in solving
should be noted that the advantages of remote sensing in SX
agricultural problems. In practice, field agricultural work, the
production are widely known, then information about the
beginning of the growing season is established by the date of
possibility of using UAVs is just beginning to develop. The
planting, and the completion is associated with determining the
information received from the UAV provides the ability to obtain
time of ripening of the plant. The duration of the phenological
relevant information with high periodicity (several times a day),
cycle is determined by the type of plant. Air temperature, air and
the ability to cover large areas with high spatial resolution (up to
soil humidity, illumination, the presence of chemical trace
several centimeters), to receive data in a uniform form (images
elements necessary for plant growth in the soil are significant
in RGB or multispectral).
factors that determine the parameters of the phenological cycle.
Since the presence of moisture in the soil at the time of planting
2. Formulation of the problem is the main limiting factor for plant growth, the beginning of the
Since the main area of agricultural land in our country is growing season is determined primarily by rainfall, as well as
located in areas of unsustainable and risky farming, under the filling the fields with snow.
conditions of observed global climate change, increasing the Modern digital cameras mounted on UAVs [3] have
reliability of current information on current and expected technical characteristics that allow solving many practical
weather conditions, assessing their impact on the state and problems of agricultural production. This paper describes the
formation of crop productivity, is of paramount importance. software necessary to solve the problem of determining the state
Factors such as weather conditions and soil quality form the of crops in large areas. Obtaining this information is possible due
conditions for plant growth and therefore determine productivity. to the ability to obtain a set of separate images of the SA territory
Changes in these factors lead to a change in productivity, which in several spectral channels of a digital RGB camera or with
forces us to develop methods for assessing the state of plants additional spectral channels (near IR or IR spectral region) [3].
throughout the entire time, from planting to ripening. The presence of this information allows you to determine the
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
characteristics of plants from the calculation of various indices In the whole variety of tasks from image processing to
(vegetation, soil, etc.), as well as texture or color analysis. machine vision, there are no clear boundaries, however,
Carrying out measurements at different time periods and processes of different levels can be distinguished here. The
obtaining multi-temporal data allows us to determine the processes of the first level (pre-processing) include only methods
dynamics of changes in the characteristics of crops, which is and algorithms for image processing to reduce noise, increase
directly related to the performed agrotechnical work. Such contrast or improve sharpness, geometric transformations, etc.
studies clearly allow us to determine the area of the CX of They are characterized by the fact that there are images at the
territories where there is a deviation from average values, for input of the process and its output. The processes following the
example, due to the degradation of soil parameters close to the first are associated with more complex image conversion tasks,
surface of the water horizon, etc. The presence of field images such as segmentation (dividing images into areas and selecting
allows us to pose the problem of obtaining cartographic objects in them), a description of the objects and their
information of the state of the SA [4] of territories , taking into compression to give them a convenient shape during further
account the fact that UAVs can be equipped with high-precision computer processing, as well as the classification (recognition)
geo-referenced devices. Such geospatial information allows us to of selected objects. Note that these processes have images in the
solve the problem of combining images in space and time, as well input, and attributes and features extracted from these images,
as embed images in geographic information systems (GIS). such as borders, contours, and other distinguishing features of
The presence of RGB color channels allows us to consider a objects that are also images (of a different type), are output.
digital camera as a spectral device, which makes it possible to Finally, processes of a different level are involved in
make index calculations (Greeness) associated with the understanding the set of many recognized objects, correlating
normalized difference index of vegetation NDVI (Normalized them with existing templates or vice versa, forming templates.
Difference Vegetation Index) [5]. The only difference is that the This shows that the natural transition from image processing to
calculation of NDVI requires the presence of spectral analysis of their content is the recognition area of individual
information in the region of 0.7-1.5 μm, and the red channel of objects in the images. Thus, what is called digital image
the digital camera is located in the region of 0.6-0.7 μm. processing is associated with processes having images at the
Nevertheless, selecting digital cameras with the necessary input and output, as well as with the processes of extracting
spectral channels, it is possible to obtain reliable information certain knowledge about objects located on the image.
about the state of crops. Using the results of the calculation of the The incoming images for processing during measurements
Greeness index in monitoring tasks of assessing the dynamics of from an unmanned aerial vehicle have specific features that are
characteristics, one can obtain spatio-temporal maps. The associated with the state difference in the level of illumination
presence of a priori information about the characteristics of the and the geometry of obtaining each image of one agricultural
soil and meteorological information allows us to build field. When shooting, which is usually carried out for several
mathematical models of changes in the state of agricultural crops hours, the Sun changes its position and shadows from the relief,
(amplitude and growth rate at different periods of vegetation). trees or clouds may appear, the level of illumination itself, the
Such information allows you to predict in advance a possible position of the unmanned aerial vehicle changes from the
crop, type of harvest (given time and territory). Note that the magnitude and direction of the wind. Therefore, at the pre-
frequency of the survey is an important parameter that processing stage, a lot of work arises related to the preliminary
determines the accuracy of the forecast and problem solving, preparation of images uniform in quality (geometry and
control of the performed agricultural work, monitoring of the illumination).
harvest, etc. After the images are unified in terms of quality, it is
necessary to select only plants on it, excluding from the
3. Vegetation Indices processing areas that are not related to plants (roads, buildings,
Vegetation indices make it possible to quantify the state of a machinery, trees, shrubs, etc.). One way to solve this problem is
plant at the time of measurement from a comparison of the values image segmentation, i.e. the selection of areas that are odd in
of the RGB spectral channels. It is known that in the blue-green some ways in the image. Currently, the main classes of image
region of the spectrum, plants have low reflectivity, which grows segmentation include the following classes: 1. Morphological
significantly in the red and near infrared regions of the spectrum. methods - are mainly used to work with binary (black and white)
Accordingly, by comparing the values of the RGB channels in images. These methods allow you to extract image components,
pixels corresponding to the plant, the state of the plant can be which can later be used to represent the shape of the object. 2.
detected. Here are some vegetation indices that are calculated Threshold methods - have intuitive properties and are easy to
based on RGB channels: GCC - Green Chromatic Coordinate, implement. There are several main types of threshold
RCC - Red Chromatic Coordinate, BCC - Blue Chromatic segmentation, but only two are basic: the method with an optimal
Coordinate, ExG - Excess Green, ExR - Excess Red and NDI - threshold and the method with an adaptive threshold. All other
Normalized Difference Index [6] . methods of this class are derived from the two mentioned
The calculation of the GCC, BCC and RCC indices is carried algorithms. 3. Methods of growing areas - are algorithms that
out according to the formulas: recursively perform the procedure for grouping pixels in a
GCC=Green/(Blue+Green+Red), (1) subregion according to predetermined criteria. One of the main
BCC=Blue/(Blue+Green+Red), (2) methods here is the watershed method. 4. Texture methods - are
RCC=Red/(Blue+Green+Red), (3) based on the analysis on the diffuse (color, reflectivity) surface
ExG=2∙GCC-RCC-BCC, (4) properties of the analyzed object. The methods presented in this
ExR=1.4∙RCC-GCC, (4) category are sets of complex operators that can reduce the surface
NDI=(Red-Green)/(Red+Green), (5) recognition process to the simple task of distinguishing
wherе R=Red, B=Blue, G=Green – channel values for each brightness levels. Note that such approaches are dependent on
image pixel. image quality and, accordingly, in our task they have the
Plants in the image were distinguished using empirically condition of confirming the result that was obtained earlier using
selected thresholds for each of the indices (1-5). Further, the greeness approaches.
indices were compared and among all the results, an index with Remote methods for monitoring agricultural fields make it
average characteristics was selected. possible to quickly identify areas of fields affected by the disease,
to determine the degree of plant ripening, etc. Identifying
Results problems of plant development in the early stages of
development significantly reduces the cost of labor and funds to leaf area, then saturation occurs (the leaf area does not change)
obtain a planned result. There are two main approaches to solving and then the plant withers, in which the leaf area decreases.
the problem of identifying affected areas - spectrometric and
optical. The spectrometric approach allows one to determine
many problems of plant growth in the early stages of
development, however, it leads to the emergence of a large
amount of data that needs to be processed in a very short time,
which requires the development of a computing base and data
storage. Optical methods are being developed in parallel with the
spectrometric approach and have the property of simplifying
processing tasks, since the number of spectral channels is fixed
(only three - RGB). It is clear that this has its limitations on the
quality of identification of plants on the field, but it allows you
to more quickly find the necessary solutions. There are various Fig. 1. The selection of plants in the image (on the left - the
types of characteristics that can be used to identify plants: original image and on the right - the selected plant)
geometric, morphological and color, as well as their
combinations, which can reduce the space of characters, which
simplifies identification. The main task when processing images
for plant identification is segmentation, i.e. selection of image
objects (groups of pixels), homogeneous in their color or fractal
characteristics, and assigning them to one or another predefined
class.
To test the operability of the proposed algorithms, the authors
conducted a model experiment related to growing plants in
specially prepared room conditions. Observation of plant growth
(wheat) was carried out daily at noon, for two months. During
this time, the plant went through all stages of its vegetation cycle,
from ripening to wilting. The obtained daily images were
processed using the developed software, which was developed in
Fig. 2. The vegetation cycle of the plant in a certain number of
the C # programming language using the SimpleCV technical
pixels corresponding to the plant
vision libraries [http://simplecv.org]. The results of image
processing associated with the selection of plants are shown in
In parallel with the calculation of the number of pixels
Fig. 1, from which it is clearly seen that the plant stands out well
corresponding to the plant, we carried out the calculation of one
in the image.
of the vegetative indexes ExG (see Fig. 3). Note that the shape of
The phenological development of plants is based on the
the curve showing the number of pixels corresponding to the
hereditary rhythm and periodicity of physiological processes,
plant is different from the shape of the vegetation index. The
called biological or phenological clocks. However, the onset of
figure clearly shows that the curve of the vegetation cycle has a
phenophases, the duration of their passage depends on a number
complex structure, which is associated with meteorological
of climatic factors, soil quality, as well as on human activity.
conditions (open window, exposure to the Sun, etc.). This
However, despite the fundamental research of domestic and
indicates the sensitivity of the indices to the effects of lighting
foreign scientists, phenological monitoring has not yet been
and meteorological parameters, which can be directly used in
introduced in industrial monitoring system at the level of the
practice.
organizational structure of typical agricultural services. Until
now, phenological monitoring, despite the fact that the
emergence of digital forms of presenting observations of weather
and seasonal phenomena, is more attached to humans.
Phenological monitoring now refers to the system of organizing
long-term observations and recording the dates of the onset of
seasonal phenomena, the centralized collection and accumulation
of information, its statistical and analytical processing of data on
the timing of the onset of seasonal natural phenomena. The main
tasks of phenology are the observation of various changes in the
annual cycle of plant development and annual registration time
of occurrence of these changes.
More specifically, phenological monitoring is associated not Fig. 3. The vegetation cycle of a plant in a specific ExG index
only with the detection of plant conditions, but also with the
determination of the factors that determine this state, namely, It is possible to carry out a series of calibration measurements
climatic factors (temperature, temperature changes, (image acquisition) with simultaneous fixation of various
precipitation, light exposure, cloudiness, etc.), soil factors meteorological conditions. Based on the measurements obtained,
(humidity, types and amounts of trace elements needed for plant it is possible to obtain the functions of changing vegetation
nutrition), environmental factors (nearby industrial enterprises, indices depending on various conditions of plant growth. Then,
city, etc.). It is necessary to build mathematical models that relate if there is a weather forecast, it is possible to predict the state of
various factors to the state of vegetation. Our work is the first the plants (taking into account the data in Fig. 2 and Fig. 3),
step on this path when determining the state of the plant by the which means that it is more accurate to make decisions, for
phenological cycle, namely, growth rate, ripening time, etc. example, about harvesting.
To use the obtained results in agricultural practice, we
calculated the number of pixels corresponding to the plant. The
calculation results for the experiment are shown in Figs. 2 and 3.
It is clearly seen that the plant at the growth stage increases the
3. Conclusion
The article briefly describes the historical aspects of the
development of precision farming and the appearance of UAVs
in agricultural practice. The basic elements of technical vision
necessary for the analysis of the state of SA plants are shown. It
is said that in order to verify the received data from the UAV, it
is necessary to take into account the meteorological conditions
and changes in the illumination of sunlight. The results of
processing the measurement data of test growing plants under
room conditions are presented. It is shown that the proposed
approach, which is based on the RGB image, allows to obtain
information about the state of the plant over the entire time period
of the vegetation cycle. It is proposed to offer this approach for
practical use in real conditions of agricultural fields.
4. References
[1] Silva T.S., Costa M.P., Melack J.M., Novo E.M. Remote
sensing of aquatic vegetation: theory and applications //
Environmental Monitoring and Assessment, 2008, 140: 131-145.
[2] Pantazi X.E., Moshou D., Alexandridis T., Wheton R.L.
Wheat yield prediction using machine learning and advanced
sensing techniques // Computers and Electronics and
Agriculture, 2016, V.6, P.57–65.
[3] Xue Z., Li J., Cheng L., Du P. Spectral–spatial classification
of hyperspectral data via morphological component analysis-
based image separation // IEEE Transaction, 2015, V.53, N.1,
P.70–84.
[4] Kataev M. Yu. Opportunities for space monitoring for
agriculture of the Tomsk region / M. Yu. Kataev, A. A. Skugarev,
I. B. Sorokin // TUSUR reports. - 2017. - T. 20, No. 3. - S. 186–
190.
[5] Ide R., Oguma H. Use of digital cameras for phenological
observations / R. Ide, H. Oguma // Ecological Informatics. –
2010. –N.5. – P.339-347.
[6] Woebbecke D.M., Meyer G.E., Vonbargen K., Mortensen
D.A. Color Indexes for Weed Identifiation under Various Soil,
Residue, and Lighting Conditions / D.M. Woebbecke, G.E.
Meyer, K. Vonbargen, D.A. Mortensen // Trans. ASABE. –
1995. – V.38. – P.259–269.