Apple Fruits Monitoring Diseases on the Tree Crown Using a Convolutional Neural Network

Abstract

The article presents a monitoring system with a mobile application based on a neural network, which allows you to identify apple fruits on the crown of trees, count their number, determine diseases and ripening rates of apple fruits and crop volume per hectare. The monitoring system consists of a photo (image) collection unit, which includes a client software tool (mobile application, digital camera), a received image processing unit, consisting of a database, a neural network, and a received data analysis unit. A neural network based on the VGG-16 and SSD architecture has been developed to identify apple fruits on the tree crown. Healthy (red and green apple fruits) affected by diseases (scab, powdery mildew, fruit rot, mechanical damage) were selected as classes of apple fruits for training the neural network. The software runs and functions on the Ubuntu operating system, a mobile application on the Android operating system. As a result of the experiments carried out on an industrial plantation of an apple orchard, it was found that the accuracy of estimating the total number of fruits on a tree crown compared to the true value was 94.72%, the accuracy of counting the number of infected fruits was 90.44%. The average speed of pattern recognition does not exceed 0.6 seconds per image, the average speed of apple fruit segmentation from the background does not exceed 0.8 seconds per image, the average speed of analyzing one image and obtaining a recognition result does not exceed 1.5 seconds, subject to the technical requirements for the server and image requirements. digital monitoring, identification of apple fruits, mobile application, neural network, disease

1. Introduction

Currently, there are no methods for accurately assessing the magnitude of the potential yield of horticultural crops. Agronomists, when conducting ground inspections of plantings to search for foci of diseases for crop planning, are forced to rely on a limited set of data obtained by the visual method, through systematic counts carried out at stationary sites and during route surveys, or using laboratory tests [ 1, 2 ]. The most common laboratory

methods for detecting horticultural diseases are visualization methods such as fluorescence spectroscopy, visible and infrared spectroscopy, and hyperspectral imaging [ 3 ]. The main problem in visual assessment of diseases is that the assessor takes on a subjective task, prone to psychological and cognitive functions, which can lead to low recognition rates [ 4, 5 ]. The industrial horticulture industry needs to develop an automated monitoring system that will predict the yield of an orchard with an accuracy of at least 80%. There are various methods for identifying diseases of horticultural crops [ 6-8 ]. Digital cameras and decision support systems (“expert systems”) are used [ 9-13 ]. However, for an “expert system”, the accuracy of pest

2023 Copyright for this paper by its authors. and disease identification mainly depends on the experience of the expert (agronomist). Currently, these tasks in different countries of the world are helped by software tools and decision support systems, such as Precision Gardening, Digital Garden [ 14,15 ], Agro Intelligent Vim [ 16,17 ], which provide prompt processing of information flows from sensors and weather modules in real time, reflecting the characteristics of growth and condition of plants in the critical phases of their development. These software tools are based on the use of global positioning systems (Global Navigation Satellite System), geographic information systems (Geographic information system), systems for remote monitoring of the state of plantations and crops using UAVs and various IT systems [ 18-20 ]. Existing approaches have proven themselves well for the cultivation of field crops to take into account the numerical indicators of the quality and quantity of vegetation on the field plot, various vegetation indices (NDVI, RVI, IPVI, WDVI) [ 21, 22 ], but for industrial horticulture, more accurate monitoring is required using high-resolution cameras to analyze each fruit and subsequent yield forecast [ 23 ]. An actual modern tool that is used in decision-making on the management of vegetation processes is machine learning. The importance of its application is due to the complexity of the task of analyzing fields and predicting a possible harvest, since many factors influence the overall results. Expert technologies are also successfully used to analyze the information obtained by monitoring the state of the environment [ 24, 25 ].

The purpose of the research is to improve the quality of identification of apple fruits on the crown of trees by developing a monitoring system with a mobile application based on a neural network, which will allow detecting, counting, determining diseases and ripening rates of apple fruits, as well as the yield per hectare.

2. Theoretical Aspects of a Research

A multi-scale network architecture for image diagnostics has been developed to identify apple fruits on the tree crown. The developed software and hardware complex consists of a photo (image) collection unit, which includes a client software tool (mobile application, digital camera), a received image processing unit, which includes a database and a neural network, which are installed on the server, and a received data analysis unit.

The developed neural network based on the architecture of VGG-16 (Very Deep Convolutional Networks) and SSD (Single Shot MultiBox Detector) recognizes and segments the fruits of an apple tree on the crown of a tree, evaluates the object presented as a photograph, classifies it and gives the result in the form of a probability in percent that it belongs to the fruit of an apple tree. The neural network discretizes the output bounding box space into a default box set with different aspect ratios and scales for each feature map (apple fruit) location. During prediction, the network generates a presence score for each category of objects in each field by default and makes adjustments to the field to better match the shape of the object.

The neural network is used in two ways of working, through the REST WEB API service (remote procedure call is an HTTP GET or POST request, and the necessary data is passed as request parameters or in the request body) and locally on the computer by specifying files for recognition. To use the neural network locally, you need to prepare images in a local folder. Full addresses (paths) of all images in the local must be placed in the file "test.txt", in which the path to each image must be on a separate line. As a result of recognition, photographs with selected regions are displayed on the screen, where the neural network considers that there is an object belonging to one or another class from the previously trained ones. At the input of the work, the neural network receives an array of bytes of the image to be recognized. At the first stage, the image received for recognition is processed, it is converted to RGB format and resized to the size necessary for the network to work. Further, in the course of work, the array of bytes is converted into matrices of the required n-dimensionality, which, in turn, are transferred to the neural network. At the output, the neural network generates a matrix in the cells of which contains information about the recognized objects and information about their belonging to a particular class.

To train the neural network, photographs were collected using a Sony Alpha ILCE-7M4 camera, the distances for shooting 0.2 m, 0.5 m and 1.0 m were determined, from angles that overlap each other. The following varieties of columnar apple trees were used: President, Currency, Chervonets, Lukomor, Malyukha. Senator, Triumph. Healthy apple fruits were chosen as the class - red, green; apple fruits affected by diseases - scab, powdery mildew, fruit rot, mechanical damage. More than 10,000 photographs of given classes of apples were taken. To prepare a sample for training, the obtained photographs were labeled. The open source online utility VGG Image Annotator was used for markup (Fig. 1).

The use of the VGG Image Annotator program for marking data made it possible to select the necessary objects (apple fruits) in the image in the frame and assign a class to each bounding box, saving the selection results in a JSON file.

The client-side software provides for receiving requests for image evaluation from the client software (mobile application), transferring the image to the server in the deep learning neural network, receiving a response from it, and issuing a response to the client software.

The database of the monitoring system stores structured information about all field measurements made, the results of calculating the number of apple fruits on the studied rows of plantations, ensures the integrity, completeness, reuse of data, and ease of updating data. The number of tables in the database is 20, the maximum table size is 32 TB, the maximum record size is 1.6 TB, the maximum field size is 1 GB, the maximum field size in a record is 250 pieces.

The software runs and functions on the Ubuntu operating system, a mobile application on the Android operating system. Software and mobile application have the ability to work based on incoming photos (images) online, as well as using previously captured photo material. The Andr oid application works on the basis of a neural network installed on a remote server to identify apple fruits on the crown of trees and search for diseases on the fruit, saving the information received in the database. It is possible to discretely count the number of fruits on a tree crown with the ability to determine the yield of one tree, a selected row of plantations or the entire field. The information obtained as a result of fruit identification and counting is sent online via GPRS mobile communication and upon request of the API for Python, stored and processed using software on the server for further forecasting and making agro-technological decisions.

The segmentation convolutional neural network of deep learning in the case of classifying the presented image as a known biological culture allows segmenting the area of culture presence with an accuracy of at least 85%, while the image segmentation speed does not exceed 0.8 seconds per image.

In addition, the network combines predictions from multiple feature maps at different resolutions to naturally process features of different sizes. A distinctive feature of the chosen SSD architecture is the ability to recognize objects in one run using a given grid of windows (default box) on the image pyramid. Image processing speed can reach up to 59 FPS (Frames Per Second, frames per second). The following list of Python libraries was used for the operation of the neural network: tensorflowgpu, Numpy, OpenCV. Nvidia libraries: CUDA, CUDA toolkit, CuDNN. Driver Libraries: Driver Nvidia Version: 515.65.01, CUDA Version: 11.7.

As a result of the research, the stages of image processing using a neural network have been developed:

1. Low-level processing (low-level processing) - the first level of image processing, which includes receiving an image from a mobile phone camera and converting it to digital form (preprocessing), which brings images to a single format, removing noise, distortion and correcting color levels.

2. Intermediate-level processing (intermediate processing) - the level of processing at which the object is selected in the image. The accuracy of the further determination of diseases depends on the accuracy of the intermediate processing. Segmentation and selection of objects can be done in three different ways: threshold segmentation, edge-based segmentation, and region-based segmentation. The result of intermediate processing is the selection of the fetus in the image.

3. High-level processing (high-level processing) - the last level of processing, in which the recognition of infected fruits and their classification takes place. High-level processing includes the use of deep learning algorithms and statistical methods (Fig. 2). In each of the three types of processing, there is a continuous interaction with the knowledge base, which stores the necessary materials (model coefficients, examples for training and classification) to perform accurate classification. To improve the accuracy of fruit identification, the knowledge base needs to be replenished with materials and periodically recalculate the coefficients of the models.

The input data necessary to perform the specified functions of the complex and the output information obtained as a result of the implementation of the complex of its functions are determined. The input data includes an image of the research object in PNG or JPEG format, distance values to the research object (measured by a digital camera or manually entered by the operator), geographic coordinates of the place where the image was obtained (latitude, longitude, altitude), information characterizing the optical system with which the image is obtained (real and imaginary size of the camera matrix, focal length and equivalent focal length, image resolution of the object, vertical and horizontal, digital zoom), information personalizing the research object (season, field, variety, vegetation phase). The web application has a structure consisting of a database, calculation models, type and maturity determination, a web interface part, and a main part that combines the part that the user sees, interacts with him, and outputs model results. The interaction between the pages of the software module is presented in the diagram (Fig. 3). The output information includes the result of the study of the object for belonging to the fruit of the apple tree - yes / no, the rate of ripening of the fruits of the apple tree, the result of counting the number of identified healthy fruits and fruits affected by diseases (yield per hectare), the result of the study of the object (selection of the area occupied by the object in the image, mask image file in PNG format), the area of the area occupied by the object in the image, calculated from the characteristics of the optical system, the distance to the object and the mask image, a log with systematized information about the received data, personalizing the object of study with saving the image files and the mask of the object of study on the server file system in the database with the ability to view the history of the results of the study of the object in the software and sharing access to the history of the results using authentication. The average absolute percentage error in measuring the number of apples on the crown of a tree (comparison of their number with the true value measured by the visual method) is determined by formula 1: (1) (2)

= + where − the actual number of fruits measured by the visual method, pcs, – the number of fetuses identified by the monitoring system, pcs, – the total number of apples for a threefold repetition of the experiment, pcs.

The number of fetuses identified by the monitoring system is determined by formula 2: eastern side of the row, pcs.

where − the number of identified fruits on the tree using monitoring system on the western side of the row, pcs, – the number of identified fruits on the tree using monitoring system on the

3. Results and discussion

Testing the operation of the monitoring system with a mobile application based on a neural network, assessing the accuracy of counting the number of apple fruits on a tree crown was carried out on an industrial plantation of an apple orchard aged 7 years. The scheme of the experiment is shown in Figure 4.

To assess the accuracy of the developed monitoring system, 5 trees in a row (trees No. 12-16) of the orchard of the Currency variety were used. Using a mobile phone with an application installed on it, based on a neural network on a remote server, the total number of apple fruits on the trees and the number of infected fruits on the western and eastern sides of the row were determined in real time. To avoid errors in estimating the number of fruits due to their mutual overlapping with leaves and branches, the study considered fruits that were visually visible and identified at least 20% of their actual size. The mobile application marks already recognized fruits, which eliminates duplication when counting apples using a neural network (Fig. 5). Matrix barcodes (Quick Response code, optical labels) were used to store and process the obtained data, containing the necessary information about the object to which they are attached (Fig. 6). QR codes are read by digital devices and store data as a series of pixels in a square grid. After scanning the QR code with the camera, the available data is displayed on the screen of the mobile phone. The obtained data are stored in the developed database Table 1).

The service allows real-time processing of photos taken from a smartphone (Fig. 5), as well as static processing of individual photos on a PC after collecting data from any device with a digital camera (Fig. 7). The results of the statistical evaluation of the obtained research results are shown in Table 2. The experiment was carried out in triplicate, the maximum and minimum number of identified fruits on the trees was determined, as well as the percentage deviation of the number of visible fruits to the number of identified fruits using a monitoring system. Based on the results of processing the obtained experimental data, graphs were constructed for assessing the accuracy of identifying apple fruits (Fig. 8). The percentage of fruit identification is calculated by the ratio of fruits recognized by the monitoring system to the total number of fruits visible on the tree. Figure 6 - Plot of an industrial plantation of an apple orchard with installed QR codes for storing and processing data

The mean absolute percentage error in counting the total number of fetuses was 5.59%, the mean absolute percentage error in counting the number of infected fetuses was 11.5% compared to the true value (visually measured). The accuracy of assessing the total number of fruits on the tree crown compared to the true value was 94.72%, the accuracy of counting the number of infected fruits was 90.44%. Measurements of the speed of identification of apple fruits on trees were carried out (Fig. 9). It has been established that the average time of pattern recognition does not exceed 0.6 seconds per image, the average time of segmentation of an apple fruit from the background does not exceed 0.8 seconds per image, the average time of analyzing one image and obtaining a recognition result does not exceed 1.5 seconds with comply with the technical requirements for the server and the requirements for images.

Figure 7 - The butt of the robotic software service with the number of known apples, including healthy and sick ones

4. Discussion

The risk of this development can be calculated using the methodology for assessing the risks of innovative projects based on fuzzy modeling (Yu. Samokhvalov, 2020) [ 26 ]. In modern conditions of horticulture, the environmental factor is essential, the solution of the problem of long-term planning according to the Leontiev-Ford ecological and economic model, taking into account the magnitude of environmental costs, was proposed by the authors H. Hnatiienko et al. in [ 27,28 ].

5. Conclusions

The developed monitoring system based on a neural network makes it possible to carry out digital monitoring both by photographic materials and by video stream online, operates stably in the conditions of industrial garden plantations, regardless of the size and interference of foliage, determines the color of the surface of the fruit, identifies the presence of diseases and fruit defects with a probability of at least 90% as a result of incremental expansion of the dataset during the operation of the complex and the gradual evolution of the solution by training the network in the process of working on new data. 94,93 96,86 95,14

93,22 13 43,67

1 0,22 0,44 46 1 100 5,34 0

14 51,33

3 0,89 0,89 53 4 75 3,25 20

15 45,67

4 0,22 0,44 48 5 80 5,11 25 16 55 1 0 0 59 1 100 7,27 0 Average number of identified fruits for three repetitions, pcs Average number of identified infected fetuses for three repetitions, pcs Dispersion in the general population of identified fruits, pcs2 Standard deviation of identified fruits, pcs The number of apple fruits found on a tree by a visual method, pcs The number of found infected apple fruits on a tree by a visual method, pcs Percentage deviation of the number of visible fetuses to the number of fetuses identified using PAC, % Percentage deviation of the number of visible infected fetuses to the number of identified infected fetuses using PAK, % Absolute percentage error in identification of the total number of fruits, % Absolute percentage error in identifying the number of infected fetuses, %

As a result of the research, it was found that for counting the number of apple fruits in industrial garden plantations, the most suitable neural network is a recurrent deep learning network, since its use allows you to recognize the contour of fruits and foci of diseases on them with high accuracy.

The monitoring system provides the ability to process at least 200 requests simultaneously and gives a result in the form of a percentage probability that the identified object belongs to an apple fruit, allows you to identify apple fruits on the crown of trees, count their number, determine diseases and ripening rates of apple fruits and crop volume per hectare. The developed neural network will expand the functionality of the monitoring system not only for monitoring the yield of fruit crops, but also for robotic harvesting of fruits [ 16, 17, 29 ] with the determination of the coordinates of each fruit or its part and the return of the coordinates of the center of the fruit and its contour indicating the areas of defects or diseases [ 30 ] on the fruit to the controller of the manipulator device.

6. References

References

[1] Wulfsohn D. , Zamora F.A. , Téllez C.P. , Lagos I.Z. , García-Fiñana M . Multilevel systematic sampling to estimate total fruit number for yield forecasts . Precis. Agric . 2012 . 13 . 256 - 275 .

[2] Font D. , Tresanchez M. , Martınez D. , Moreno J. , Clotet E. , Palacın , J. Vineyard Yield Estimation Based on the Analysis of High Resolution Images Obtained with Artificial Illumination atNight . Sensors . 2015 . 15 ( 4 ): 8284 .

[3] Zou X. , Zhao J. , Li Y. , Mel H. In-line detection of apple defects using three color cameras system . Comput. Electron. Agric . 2010 . 70 , 129 - 134 .

[4] Lak M.B. , Minaei S. , Amiriparian J. , Beheshti B . Apple fruit recognition under natural luminance using machine vision . Advance Journal of Food Science and Technology . 2010 . 2 ( 6 ): 325 - 327 .

[5] Garcia J. , Barbedo A . A review on the main challenges in automatic plant disease identification based on visible range images . Biosystems Engineering . 2016 . 144 : 52 - 60 .

[6] Ji W. , Zhao D ., Cheng F., Xu B. , Zhang Y. , Wang J . Automatic recognition vision system guided for apple harvesting robot . Computers and Electrical Engineering . 2012 . 38 ( 5 ): 1186 - 1195 .

[7] Jimenez A. R. , Ceres R. , Pons J. L. A survey of computer vision methods for locating fruiton trees . Transactions of the ASAE-American Society of Agricultural Engineers . 2000 . 43 ( 6 ): 1911 - 1920 .

[8] Kapach K. , Barnea E. , Mairon R. , Edan Y. , Ben-Shahar O. Computer Vision for FruitHarvesting Robots - state of the Art and Challenges Ahead . International Journal of Computational Vision and Robotics . 2012 . 3 ( 1 -2): 4 - 34 .

[9] Kim D. , Choi H. , Choi J. , Yoo S . J., Han D. A Novel Red Apple Detection Algorithm Basedon AdaBoost Learning . IEIE Transactions on Smart Processing & Computing . 2015 . 4 ( 4 ): 265 - 271 .

[10] Fountas S. , Sorensen C.G. , Tsiropoulos Z. , Cavalaris C. , Liakos V. , Gemtos T. Farm machinery management information system . Computers and electronics in agriculture. 2015 . Vol. 110 . 131 - 138 .

[11] Kaloxylos A. , Groumas A. , Sarris V. , Katsikas L. , Magdalinos P. , Antoniou E. , Politopoulou Z. , Wolfert S. , Brewster C. , Eigenmann R. , Maestre Terol C. Acloud-based farm management system: architecture and implementation . Computers and electronics in agriculture. 2014 . Vol. 100 . 168 - 179 .

[12] Anderson N.T. , Walsh K.B. , Wulfsohn D . Technologies for Forecasting Tree Fruit Load and Harvest Timing-From Ground , Skyand Time . Agronomy. 2021 . 11 , 1409 .

[13] Zhao P. , Liu G. , Li M.Z. Management information system for apple diseases and insect pests based on GIS . Trans. Chin. Soc. Agric. Eng . 2006 . 22 . 150 - 154 .

[14] Qu D. , Hu Y. , Zhang Y. The Investigation of the Obstacle Avoidance for Mobile Robot Based on the Multi Sensor Information Fusion technology . International Journal of Manufacturing , 2013 , 1 . - pp. 366 - 370 .

[15] Sgorbissa A. , Zaccaria R . Planning and obstacle avoidance in mobile robotics . Robotics and Autonomous Systems , 2012 , 60 . - pp. 628 - 638 .

[16] Smirnov , I. , Kutyrev , A. , Kiktev , N. ( 2021 ). Neural network for identifying apple fruits on the crown of a tree. E3S Web of Conferences. International scientific forum on computer and energy Sciences , WFCES 2021 , 01021 .

[17] Kutyrev , A. , Kiktev , N. , Kalivoshko , O. , Rakhmedov , R. Recognition and Classification Apple Fruits Based on a Convolutional Neural Network Model. Selected Papers of the IX International Scientific Conference "Information Technology and Implementation" (IT&I- 2022 ). Conference Proceedings, Kyiv, Ukraine. CEUR Workshop Proceedings , 2022 , 3347 , pp. 90 - 101 .

[18] Kaivosoja J. , Jackenkroll M. , Linkolehto R. , Weis M. , Gerhards R . Automaticcontrol of farming operations based on spatial web services . Computers and electronics in agriculture. 2014 . Vol. 100 . 110 - 115 .

[19] Paraforos D. S. , Vassiliadis V. , Kortenbruck D. , Stamkopoulos K. , Ziogas V. , Sapounas A.A. , Griepentrog H.W. Multi-level automation of farm management information systems . Computers and Electronics in Agriculture. 2017 . Vol. 142 . 504 - 514 .

[20] Ampatzidis Y. , Tan L. , Haley R. , Whiting M.D. Cloud -based harvest management information system for hand-harvested specialty crops . Computers and electronics in agriculture. 2016 . Vol. 122 . 161 - 167 .

[21] Miranda C. , Santesteban , L.G. , Urrestarazu J. , Loidi M. , Royo J.B. Sampling stratification using aerial imagery to estimate fruit load in peach tree orchards . Agriculture . 2018 . 8 . 78.

[22] Linker R. , Cohen O. , Naor A . Determination of the numberof green apples in rgb images recorded in orchards . Computers and Electronics in Agriculture . 2012 . vol. 81 , pp. 45 - 57 .

[23] Tan W.X. , Zhao C.J. , Wu H.R. CNN intelligent early warning for apple skin lesion image acquired by infrared video sensors . High Technol. Lett . 2016 . 22 . 67 - 74 .

[24] Giang , T.T.H.; Ryoo, Y.-J. Pruning Points Detection of Sweet Pepper Plants Using 3D Point Clouds and Semantic Segmentation Neural Network . Sensors 2023 , 23 , 4040. https://doi.org/10.3390/s23084040

[25] Tang , J. ; Jiang , F. ; Long , Y. ; Fu , L. ; Sun , H. Identification of the Yield of Camellia oleifera Based on Color Space by the Optimized Mean Shift Clustering Algorithm Using Terrestrial Laser Scanning . Remote Sens . 2022 , 14 , 642. https://doi.org/10.3390/rs14030642

[26] Samokhvalov Y. ( 2021 ). Risk Assessment of Innovative Projects Based on Fuzzy Modeling . In: Babichev, S. , Lytvynenko , V. , Wójcik , W. , Vyshemyrskaya , S. (eds) Lecture Notes in Computational Intelligence and Decision Making. ISDMCI 2020. Advances in Intelligent Systems and Computing , vol 1246 . Springer, Cham. https://doi.org/10.1007/978-3- 030 -54215-3_ 17

[27] Hnatiienko , H. , Domrachev , V. , Saiko , V. Monitoring the condition of agricultural crops based on the use of clustering methods . 15th International Conference Monitoring of Geological Processes and Ecological Condition of the Environment, Monitoring 2021 , 2021 . https://doi.org/10.3997/ 2214 - 4609 .20215K2049

[28] Hnatiienko , H. , Kudin , V. , Onyshchenko , A. , Snytyuk , V. and Kruhlov , A. Greenhouse Gas Emission Determination Based on the Pseudo-Base Matrix Method for Environmental Pollution Quotas Between Countries Allocation Problem /в 2020 IEEE 2nd International Conference on System Analysis & Intelligent Computing (SAIC) , Kyiv, Ukraine, 2020 , pp. 150 - 157 , doi: 10.1109/SAIC51296. 2020 . 9239125 .

[29] Kiktev , N. , Didyk , A. , Antonevych , M. ( 2020 ). Simulation of Multi-Agent Architectures for Fruit and Berry Picking Robot in Active- HDL. 2020 IEEE International Conference on Problems of Infocommunications. Science and Technology (PIC S&T) , Kharkiv, Ukraine, 2020 , 635 - 640 , doi: 10.1109/PICST51311. 2020 . 9467936 .

[30] Mohanty S.P. , Hughes D.P. , Salathé M. Using deep learning for image-based plant disease detection . Frontiers in Plant Scienc . 2016 . 7 : 1419 .