Determination of Plant Phenological Cycle from RGB Images M.Yu. Kataev1, K.S. Yolgin1 kmy@asu.tusur.ru 1Tomsk University of Control Systems and Radio Electronics, Tomsk, Russia Automated visual assessment of the state of the earth and plants, wilting and pests of leaves, plant growth indicators, using technical vision, can be used as a basis in smart (precision) agriculture (SA). This article discusses a brief review of the literature on the use of computer (technical) vision (CV) for analyzing the condition of agricultural fields and plants growing on them. The introduction of vision systems into real agricultural production practice is associated with the development of complex mathematical approaches that must be resistant to a variety of technical and weather changes. It is necessary to overcome image changes caused by atmospheric conditions and daily and seasonal variations in sunlight. An approach is proposed, which is based on an RGB image obtained using a typical digital camera. The results are given on the use of CV systems in solving individual tasks of agricultural production. Keywords: technical vision, mathematical methods, images, agriculture, image classification, unmanned aerial vehicle The development of methods for classifying and assessing 1. Introduction the state of plants is one of the main areas of research in the field of remote sensing of the earth using satellites, aircraft and An important task of smart agriculture is to monitor the unmanned aerial vehicles. Observations of the seasonal condition of plants from the moment of planting, to ripening and development of plants have been carried out in the interests of harvesting [1]. This segment of research, based on CV, has still agriculture for a long time (more than 30 years). Satellite weakly penetrated SA production. Control of large-sized and measurements of the spectral characteristics of radiation spatially distributed plots of agricultural land is difficult and reflection in the visible and infrared regions of the spectrum and poorly implemented in modern farms by classical methods. The the values of vegetation indices obtained on their basis allow us problem here is the inability to study the characteristics of soil to describe the seasonal dynamics of various types of plants. One and plants on frequent spatial and temporal grids. Information of the necessary conditions for determining the phenological obtained by classical methods is rare in time and space and is characteristics of plants (the time of the onset of various more based on the experience of agronomic workers. phenological phases, the duration of the growing season and The problems of agricultural development are global. The others) is the availability of time series of data from continuous concept of sustainable development of society includes in the list satellite observations. These measurements provide diverse and of the main issues that will need to be addressed, the following: accurate information on the development of plants over time. population growth; energy sources and new fuel; food, including However, there are limitations associated with the frequency drinking water; depletion of resources; global climate change; (preferably at least once a week) of obtaining the necessary data the problem of pollution of air, water (oceans, seas, lakes, rivers (cloud exposure). Therefore, the obtained satellite information is and underground sources) and soil; the problem of limiting the important, but rather it is some benchmark information for production and consumption of toxic and harmful products. The methods that allow you to regularly receive data on agricultural solution to almost all of these issues, one way or another, is fields and plants located on them, namely unmanned aerial associated with the successful development of agriculture, vehicles. improving the quality and quantity of products. Phenological characteristics such as start of growing season The solution of the above problems is possible with the help (SOS) and end of growing season (EOS) end dates, its growing of a modern monitoring base based on the use of satellite remote season length (GSL), and maximum of growing date season - sensing (SRS) data and information obtained from unmanned MGS), seasonal amplitude and some others, are widely used in aerial vehicles (UAVs) [2]. The information obtained in this way solving problems of remote sensing of plant conditions. is unique in that it has a high temporal and spatial resolution and Information on the start date of the growing season is is informative (the presence of multispectral information). It characterized by the widest practical relevance in solving should be noted that the advantages of remote sensing in SX agricultural problems. In practice, field agricultural work, the production are widely known, then information about the beginning of the growing season is established by the date of possibility of using UAVs is just beginning to develop. The planting, and the completion is associated with determining the information received from the UAV provides the ability to obtain time of ripening of the plant. The duration of the phenological relevant information with high periodicity (several times a day), cycle is determined by the type of plant. Air temperature, air and the ability to cover large areas with high spatial resolution (up to soil humidity, illumination, the presence of chemical trace several centimeters), to receive data in a uniform form (images elements necessary for plant growth in the soil are significant in RGB or multispectral). factors that determine the parameters of the phenological cycle. Since the presence of moisture in the soil at the time of planting 2. Formulation of the problem is the main limiting factor for plant growth, the beginning of the Since the main area of agricultural land in our country is growing season is determined primarily by rainfall, as well as located in areas of unsustainable and risky farming, under the filling the fields with snow. conditions of observed global climate change, increasing the Modern digital cameras mounted on UAVs [3] have reliability of current information on current and expected technical characteristics that allow solving many practical weather conditions, assessing their impact on the state and problems of agricultural production. This paper describes the formation of crop productivity, is of paramount importance. software necessary to solve the problem of determining the state Factors such as weather conditions and soil quality form the of crops in large areas. Obtaining this information is possible due conditions for plant growth and therefore determine productivity. to the ability to obtain a set of separate images of the SA territory Changes in these factors lead to a change in productivity, which in several spectral channels of a digital RGB camera or with forces us to develop methods for assessing the state of plants additional spectral channels (near IR or IR spectral region) [3]. throughout the entire time, from planting to ripening. The presence of this information allows you to determine the Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). characteristics of plants from the calculation of various indices In the whole variety of tasks from image processing to (vegetation, soil, etc.), as well as texture or color analysis. machine vision, there are no clear boundaries, however, Carrying out measurements at different time periods and processes of different levels can be distinguished here. The obtaining multi-temporal data allows us to determine the processes of the first level (pre-processing) include only methods dynamics of changes in the characteristics of crops, which is and algorithms for image processing to reduce noise, increase directly related to the performed agrotechnical work. Such contrast or improve sharpness, geometric transformations, etc. studies clearly allow us to determine the area of the CX of They are characterized by the fact that there are images at the territories where there is a deviation from average values, for input of the process and its output. The processes following the example, due to the degradation of soil parameters close to the first are associated with more complex image conversion tasks, surface of the water horizon, etc. The presence of field images such as segmentation (dividing images into areas and selecting allows us to pose the problem of obtaining cartographic objects in them), a description of the objects and their information of the state of the SA [4] of territories , taking into compression to give them a convenient shape during further account the fact that UAVs can be equipped with high-precision computer processing, as well as the classification (recognition) geo-referenced devices. Such geospatial information allows us to of selected objects. Note that these processes have images in the solve the problem of combining images in space and time, as well input, and attributes and features extracted from these images, as embed images in geographic information systems (GIS). such as borders, contours, and other distinguishing features of The presence of RGB color channels allows us to consider a objects that are also images (of a different type), are output. digital camera as a spectral device, which makes it possible to Finally, processes of a different level are involved in make index calculations (Greeness) associated with the understanding the set of many recognized objects, correlating normalized difference index of vegetation NDVI (Normalized them with existing templates or vice versa, forming templates. Difference Vegetation Index) [5]. The only difference is that the This shows that the natural transition from image processing to calculation of NDVI requires the presence of spectral analysis of their content is the recognition area of individual information in the region of 0.7-1.5 μm, and the red channel of objects in the images. Thus, what is called digital image the digital camera is located in the region of 0.6-0.7 μm. processing is associated with processes having images at the Nevertheless, selecting digital cameras with the necessary input and output, as well as with the processes of extracting spectral channels, it is possible to obtain reliable information certain knowledge about objects located on the image. about the state of crops. Using the results of the calculation of the The incoming images for processing during measurements Greeness index in monitoring tasks of assessing the dynamics of from an unmanned aerial vehicle have specific features that are characteristics, one can obtain spatio-temporal maps. The associated with the state difference in the level of illumination presence of a priori information about the characteristics of the and the geometry of obtaining each image of one agricultural soil and meteorological information allows us to build field. When shooting, which is usually carried out for several mathematical models of changes in the state of agricultural crops hours, the Sun changes its position and shadows from the relief, (amplitude and growth rate at different periods of vegetation). trees or clouds may appear, the level of illumination itself, the Such information allows you to predict in advance a possible position of the unmanned aerial vehicle changes from the crop, type of harvest (given time and territory). Note that the magnitude and direction of the wind. Therefore, at the pre- frequency of the survey is an important parameter that processing stage, a lot of work arises related to the preliminary determines the accuracy of the forecast and problem solving, preparation of images uniform in quality (geometry and control of the performed agricultural work, monitoring of the illumination). harvest, etc. After the images are unified in terms of quality, it is necessary to select only plants on it, excluding from the 3. Vegetation Indices processing areas that are not related to plants (roads, buildings, Vegetation indices make it possible to quantify the state of a machinery, trees, shrubs, etc.). One way to solve this problem is plant at the time of measurement from a comparison of the values image segmentation, i.e. the selection of areas that are odd in of the RGB spectral channels. It is known that in the blue-green some ways in the image. Currently, the main classes of image region of the spectrum, plants have low reflectivity, which grows segmentation include the following classes: 1. Morphological significantly in the red and near infrared regions of the spectrum. methods - are mainly used to work with binary (black and white) Accordingly, by comparing the values of the RGB channels in images. These methods allow you to extract image components, pixels corresponding to the plant, the state of the plant can be which can later be used to represent the shape of the object. 2. detected. Here are some vegetation indices that are calculated Threshold methods - have intuitive properties and are easy to based on RGB channels: GCC - Green Chromatic Coordinate, implement. There are several main types of threshold RCC - Red Chromatic Coordinate, BCC - Blue Chromatic segmentation, but only two are basic: the method with an optimal Coordinate, ExG - Excess Green, ExR - Excess Red and NDI - threshold and the method with an adaptive threshold. All other Normalized Difference Index [6] . methods of this class are derived from the two mentioned The calculation of the GCC, BCC and RCC indices is carried algorithms. 3. Methods of growing areas - are algorithms that out according to the formulas: recursively perform the procedure for grouping pixels in a GCC=Green/(Blue+Green+Red), (1) subregion according to predetermined criteria. One of the main BCC=Blue/(Blue+Green+Red), (2) methods here is the watershed method. 4. Texture methods - are RCC=Red/(Blue+Green+Red), (3) based on the analysis on the diffuse (color, reflectivity) surface ExG=2∙GCC-RCC-BCC, (4) properties of the analyzed object. The methods presented in this ExR=1.4∙RCC-GCC, (4) category are sets of complex operators that can reduce the surface NDI=(Red-Green)/(Red+Green), (5) recognition process to the simple task of distinguishing wherе R=Red, B=Blue, G=Green – channel values for each brightness levels. Note that such approaches are dependent on image pixel. image quality and, accordingly, in our task they have the Plants in the image were distinguished using empirically condition of confirming the result that was obtained earlier using selected thresholds for each of the indices (1-5). Further, the greeness approaches. indices were compared and among all the results, an index with Remote methods for monitoring agricultural fields make it average characteristics was selected. possible to quickly identify areas of fields affected by the disease, to determine the degree of plant ripening, etc. Identifying Results problems of plant development in the early stages of development significantly reduces the cost of labor and funds to leaf area, then saturation occurs (the leaf area does not change) obtain a planned result. There are two main approaches to solving and then the plant withers, in which the leaf area decreases. the problem of identifying affected areas - spectrometric and optical. The spectrometric approach allows one to determine many problems of plant growth in the early stages of development, however, it leads to the emergence of a large amount of data that needs to be processed in a very short time, which requires the development of a computing base and data storage. Optical methods are being developed in parallel with the spectrometric approach and have the property of simplifying processing tasks, since the number of spectral channels is fixed (only three - RGB). It is clear that this has its limitations on the quality of identification of plants on the field, but it allows you to more quickly find the necessary solutions. There are various Fig. 1. The selection of plants in the image (on the left - the types of characteristics that can be used to identify plants: original image and on the right - the selected plant) geometric, morphological and color, as well as their combinations, which can reduce the space of characters, which simplifies identification. The main task when processing images for plant identification is segmentation, i.e. selection of image objects (groups of pixels), homogeneous in their color or fractal characteristics, and assigning them to one or another predefined class. To test the operability of the proposed algorithms, the authors conducted a model experiment related to growing plants in specially prepared room conditions. Observation of plant growth (wheat) was carried out daily at noon, for two months. During this time, the plant went through all stages of its vegetation cycle, from ripening to wilting. The obtained daily images were processed using the developed software, which was developed in Fig. 2. The vegetation cycle of the plant in a certain number of the C # programming language using the SimpleCV technical pixels corresponding to the plant vision libraries [http://simplecv.org]. The results of image processing associated with the selection of plants are shown in In parallel with the calculation of the number of pixels Fig. 1, from which it is clearly seen that the plant stands out well corresponding to the plant, we carried out the calculation of one in the image. of the vegetative indexes ExG (see Fig. 3). Note that the shape of The phenological development of plants is based on the the curve showing the number of pixels corresponding to the hereditary rhythm and periodicity of physiological processes, plant is different from the shape of the vegetation index. The called biological or phenological clocks. However, the onset of figure clearly shows that the curve of the vegetation cycle has a phenophases, the duration of their passage depends on a number complex structure, which is associated with meteorological of climatic factors, soil quality, as well as on human activity. conditions (open window, exposure to the Sun, etc.). This However, despite the fundamental research of domestic and indicates the sensitivity of the indices to the effects of lighting foreign scientists, phenological monitoring has not yet been and meteorological parameters, which can be directly used in introduced in industrial monitoring system at the level of the practice. organizational structure of typical agricultural services. Until now, phenological monitoring, despite the fact that the emergence of digital forms of presenting observations of weather and seasonal phenomena, is more attached to humans. Phenological monitoring now refers to the system of organizing long-term observations and recording the dates of the onset of seasonal phenomena, the centralized collection and accumulation of information, its statistical and analytical processing of data on the timing of the onset of seasonal natural phenomena. The main tasks of phenology are the observation of various changes in the annual cycle of plant development and annual registration time of occurrence of these changes. More specifically, phenological monitoring is associated not Fig. 3. The vegetation cycle of a plant in a specific ExG index only with the detection of plant conditions, but also with the determination of the factors that determine this state, namely, It is possible to carry out a series of calibration measurements climatic factors (temperature, temperature changes, (image acquisition) with simultaneous fixation of various precipitation, light exposure, cloudiness, etc.), soil factors meteorological conditions. Based on the measurements obtained, (humidity, types and amounts of trace elements needed for plant it is possible to obtain the functions of changing vegetation nutrition), environmental factors (nearby industrial enterprises, indices depending on various conditions of plant growth. Then, city, etc.). It is necessary to build mathematical models that relate if there is a weather forecast, it is possible to predict the state of various factors to the state of vegetation. Our work is the first the plants (taking into account the data in Fig. 2 and Fig. 3), step on this path when determining the state of the plant by the which means that it is more accurate to make decisions, for phenological cycle, namely, growth rate, ripening time, etc. example, about harvesting. To use the obtained results in agricultural practice, we calculated the number of pixels corresponding to the plant. The calculation results for the experiment are shown in Figs. 2 and 3. It is clearly seen that the plant at the growth stage increases the 3. Conclusion The article briefly describes the historical aspects of the development of precision farming and the appearance of UAVs in agricultural practice. The basic elements of technical vision necessary for the analysis of the state of SA plants are shown. It is said that in order to verify the received data from the UAV, it is necessary to take into account the meteorological conditions and changes in the illumination of sunlight. The results of processing the measurement data of test growing plants under room conditions are presented. It is shown that the proposed approach, which is based on the RGB image, allows to obtain information about the state of the plant over the entire time period of the vegetation cycle. It is proposed to offer this approach for practical use in real conditions of agricultural fields. 4. References [1] Silva T.S., Costa M.P., Melack J.M., Novo E.M. Remote sensing of aquatic vegetation: theory and applications // Environmental Monitoring and Assessment, 2008, 140: 131-145. [2] Pantazi X.E., Moshou D., Alexandridis T., Wheton R.L. Wheat yield prediction using machine learning and advanced sensing techniques // Computers and Electronics and Agriculture, 2016, V.6, P.57–65. [3] Xue Z., Li J., Cheng L., Du P. Spectral–spatial classification of hyperspectral data via morphological component analysis- based image separation // IEEE Transaction, 2015, V.53, N.1, P.70–84. [4] Kataev M. Yu. Opportunities for space monitoring for agriculture of the Tomsk region / M. Yu. Kataev, A. A. Skugarev, I. B. Sorokin // TUSUR reports. - 2017. - T. 20, No. 3. - S. 186– 190. [5] Ide R., Oguma H. Use of digital cameras for phenological observations / R. Ide, H. Oguma // Ecological Informatics. – 2010. –N.5. – P.339-347. [6] Woebbecke D.M., Meyer G.E., Vonbargen K., Mortensen D.A. Color Indexes for Weed Identifiation under Various Soil, Residue, and Lighting Conditions / D.M. Woebbecke, G.E. Meyer, K. Vonbargen, D.A. Mortensen // Trans. ASABE. – 1995. – V.38. – P.259–269.