Recognition and Classification Apple Fruits Based on a Convolutional Neural Network Model

Abstract

An intelligent system with a neural network has been developed to identify fruits on tree crowns. Based on the results of research by well-known scientists on the use of neural networks in agriculture, a recurrent neural network of deep learning was used. As a result of the field experiment, it was found that the errors of the developed software and hardware complex in estimating the size of fruits were mainly caused by inaccurate image segmentation, as well as the low resolution of the camera used. It is revealed that the convolutional recurrent deep learning network is the most suitable neural network for the tasks of apple fruit analysis, since its use allows us to recognize the contour of fruits and the foci of diseases on them with high accuracy in conditions of changing climatic parameters. The developed software and hardware complex based on the developed neural network will allow digital monitoring of both photographic materials and video streams in online mode The developed intelligent fruit monitoring system using deep learning neural networks will allow the producer of garden crops to determine the three most important parameters for him: the volume of the crop per hectare, the quality of apples - their size (translated from pixels per inch) and the degree of their maturity, the uniformity of the quality of fruits relative to a number of plantings. The use of computer vision systems with neural networks in agricultural aggregates represents a huge potential for automating accounting and decision-making control.

Keywords

• 1 Neural network
• production process
• computer vision
• fruit recognition
• digital monitoring

1. Introduction

The development of intelligent systems and technologies for agriculture requires accurate methods for estimating the potential yield of fruit crops. Now, when planning the harvest, agronomists are forced to rely only on their own experience and an extremely limited set of data on the results of visual inspection of plantings. The industry needs automated intelligent systems that will give an early forecast of the crop with an accuracy of more than 80%. The developed intelligent system of fruit monitoring with the use of deep learning neural networks will allow the producer of garden crops to determine the three most important parameters for him: the volume of the crop per hectare, the quality of apples – their size (in terms of pixels per inch) and the degree of their maturity, the uniformity of the quality of fruits relative to the number of plantings. These parameters allow you to plan the number of seasonal workers, the volume of storage in distribution and logistics centers and deliveries to retail chains, as well as reduce sorting costs.

The analysis of studies shows that the most effective way to identify biological objects in agriculture is the use of complex systems developed using neural network technologies, computer vision, and various spectrometry methods [ 1,2,3 ]. The use of artificial neural networks makes it possible to recognize defects on each product unit, assess the overall condition, and report problems found in real time [ 4,5,6 ]. The use of neural networks in agricultural machinery is a relatively new direction, which helps to solve, among other things, the main tasks of robotic harvesting of garden crops: 1) navigation, moving the robot across the field, from tree to tree; 2) detection and localization of the object by coordinates; 3) determination of the degree of ripeness of the object and damage to disease and pests [ 7,8 ]. Recurrent neural networks (CRNN) and gradient recurrent neural networks (GRNN) are used to control the movement of robotic platforms. The recognition of apple fruits using a real-time technical vision system is a relatively difficult task, since the time available for image processing using a neural network when performing technological operations and making decisions based on the processing of the obtained data is limited.

2. Literature review

An overview of the results of searching for fruits on trees is presented in the article Jimenez A. R.,

Ceres R., Pons J. L. (2000) [9]. The methods of application are used, in particular, when sorting apples,

this is described by İ. Kavdyr, D.E. Guyer. (2008) [ 10 ]. Automatic sorting of bicolor apples using multispectral machine vision is described by D. Unay, B. Gosselin, et al. (2011) [ 11 ], also M.M. Sofu,

O. Er et al. (2016) [12]. The image segmentation technique for bagged green apples is described by J.

Lv, F. Wang, L. et al. (2019) [13]. Optical non-destructive methods for small berries in a review article by S. Li, H. Luo, et al. (2019) [14]. Fruit quality assessment using a machine vision system is described in the article by J. Blasco, N. Aleixos, E. Molto. (2003) [15].

The developed system of multispectral vision, which allows signs of defects on apples, described in the work of O. Kleynen, V. Leemans and M.-F. Destain (2005) [16].

A number of works appearing in different countries describe the technical means used in fruit

recognition. Mobile terrestrial laser scanner for fruit detection in an apple orchard by J. Gene-Mola, E.

Gregorio et al. (2014) [17]. A review of sensors and systems for detecting and detecting detected fruits in the work of A. Gongal, S. Amatya et al. (2015) [18].

In works [19,20], the authors of this article single out versions of the fruit and berry detection system.

In the work of D. Khort, A. Kutyrev et al. (2020) [19], a computer vision system was developed to study

the location and ripeness of strawberries, an algorithm for automatic control of the manipulator in Python 3.7.2 was implemented, including the determination of X and Y coordinates. berries, the degree of its maturity, as well as by calculating the approximation from the manipulator to the berries. In the work of I. Smirnov, A. Kutyrev, N. Kiktev (2021) [ 20 ], promising experiments with the recognition of apples on a tree, as well as the separation of images of healthy and sick apples. The development process of the developed neural network is implemented in the Python development language, the Spyder development environment, the PyTorch framework used, the architecture of the building of highprecision neural networks - MASK-RCNN. The model is trained using the TensorFlow Object

Detection API machine learning. Of interest is also the work of American traces by P. Narayanan, A.

M. Lefcourt, U. Tasch (2007) [21], which proposes a random and inexpensive methodological

derivation of fruit for the onset, onset and stop of optical fruit sorting technologies.

Clustering methods for monitoring the state of crops based on machine control in the work of

Gnatienko G., Domrachev V., Saiko V. (2021) [22]. Mathematical hardware solutions in multi-agent robotic systems are harvesting fruits that can be based on the development of Ivokhin O.V. Oletsky, O.V. (2022) [23],

3. Theoretical Aspects of a Research

3.1. Classification of neural networks and choice for solving the problem

As a result of the analysis of the existing neural networks used in agriculture, their general classification is developed (Fig. 1). Based on the results of research by well-known scientists on the use of neural networks in agriculture [ 24,25,26 ], a recurrent neural network of deep learning was chosen for the optimal speed of recognition of apple fruits, their size parameters and obtaining the maximum accuracy of the result. By the type of training-with a teacher, by the type of setting – dynamic, by the type of input information – analog, by the type of problem to be solved-classifying. The principle of operation of such a neural network is to divide (segment) the analyzed photo into classes and select specific objects (disease, apple, branch, etc.). The selected neural network by its design is one of the best models available for solving most "perception problems" (such as image classification).

To implement the learning process of the developed neural network, the Python programming

language and the Spyder development environment were chosen, and the PyTorch framework work was used. The architecture of deep convolutional neural networks – MASK-RCNN-is chosen. To train the model, the TensorFlow Object Detection API machine learning libraries, GPU computing libraries, and libraries for working with images and graphs are used.

3.2. Metrics for assessing the performance of neural networks

Currently, various metrics are used to evaluate the performance of neural networks in identifying

objects [ 27,28 ]. To assess the quality of the developed neural network, a binary classification problem was applied, in which two classes were used - healthy apple fruits and apple fruits affected by the disease. Used such metrics as Precision, Recall, True Positive Rate (TPR), True negative rate (TNR),

False negative rate (FNR), False positive rate (FPR), Positive predictive value (PPV), Negative predictive value (NPV), Accuracy, F-measure, Specificity, Overall accuracy (OA), Matthews correlation coefficient (MCC), Balanced Accuracy (BA). 92

An error matrix (inaccuracy matrix) has been constructed, in which the “predictions” of the algorithm metrics are located horizontally, along the vertical “answers” are the true class labels.

computed class is taken as the "positive" class, and all other classes as the "negative" one. In this case, the formula for the Accuracy metric takes the following form, the formula:

Average Accuracy =

∑ =1(TP + TN + FP + FN

) TP + TN

K ; where K is the number of classes, pcs.

The Precision metric takes the following form, the formula:

The Recall metric takes the following form, the formula:

The F-score metric takes the following form, the formula:

Average Precision = Average Recall =

The AUC-ROC metric takes the following form, the formula is:

where f(c) is the relative frequency of the class.

The Specificity metric takes the following form, the formula:

Average F − score = ∑ =1(2 · Precision + Recall) Precision · Recall ; AUC − ROC = ∑ AUC − ROC(c ) · f(c )

The Overall accuracy (OA) metric takes the following form, the formula is:

The Balanced Accuracy (BA) metric takes the following form, the formula is:

Average Specificity = ∑ =1(TN + FP)

TN K Average OA =

TP + TN ∑iK=1(TP + TN + FP + FN)i

K =1

∑ =1(TP + FP)

TP ∑ =1(TP + FN

) K

TP K

K ; ; ; ; Average BA = 1 ∑iK=1( 2 · (Recall + Specificity))i K ; 3.2.

Software and hardware for fruit recognition in the garden

To collect photos during training, several Nikon D3500 AF-S 18-140 VR cameras were used, the

Nikon Nikkor AF-P DX F 18-55 mm lens was determined at shooting distances of 0.2 m, 0.5 m and 1.0

m, from angles that overlap each other. More than 25,000 photos of the specified classes of apples were taken. To train the neural network, well-known algorithms for expanding the training set with distorted image variants (augmentation) of the Python imgaug 0.3.0 library are used. (shifts, small rotations, Gaussian blur, noise). Using the data markup technique allows you to select the necessary objects in the image and assign the desired class to each bounding box. As a result of the conducted research, a computational algorithm for recognizing apple fruits was created. The first stage includes creating a recognizer program: importing the libraries necessary for operation, creating a configuration file with predefined fields, forming a file with the "weights" of the trained neural network and specifying the path to its location, creating a neural network model with previously set settings, processing the base data set in the current directory, analyzing the resulting information array, activating and calling the object selection function, adjusting the object selection function.

The second stage includes contextual object recognition functions: creating a contextual function for

recognizing the required objects, and correlating the contextual function.

The third stage includes the formation of a reference and training sample, as well as the formation of the main test data set. The reference sample is formed from images (photos) obtained under ideal laboratory conditions by shooting identifiable objects (elements of the growth cover of several varieties of garden apple trees affected by different forms of plant diseases at different stages of development) under artificial lighting on a flat, monotonous, non-glare surface (screen). When the image is captured, the screen is placed behind the subject and covers the entire background in the frame. The object is strictly in the focus of the camera. The presence of light spots from flashes and light sources, shadows, the presence of foreign objects in the frame is not allowed. The training sample is formed from images (photos) obtained in real production conditions by shooting identifiable objects (elements of vegetation cover of several varieties of garden apple trees affected by different forms of plant diseases at different stages of development) under artificial and natural lighting on a flat, monotonous, non-glare surface (screen). When the image is captured, the screen is placed behind the subject and covers the entire background in the frame. The presence of light spots from flashes and light sources, defocusing, shadows, the presence of foreign objects in the frame is not allowed. The use of different substrates (backgrounds) when creating the same selection is not allowed. The main test data set is formed from images (photos) obtained in real production conditions by shooting identifiable objects (plant cover elements of several varieties of apple trees affected by different forms of plant diseases at different stages of development) under artificial and natural light without using additional means of background correction (screens, etc.). The presence of light spots from flashes and light sources, defocusing, shadows, the presence of foreign objects in the frame is not allowed.

The fourth stage involves training the neural network and implementing the object recognition function for similarity with a given mask: implementing the function of interpreting the compared objects in the form of matrices, finding key points and descriptors using the SIFT algorithm, setting up the algorithm, and pre-testing the algorithm.

The main difference between the used detection algorithm and the existing ones is to determine the

probability of the presence of the desired object in the entire image, in each of its segments, regardless of their number. The search is performed by pre-configured descriptors throughout the image field, then, based on the concentration of probabilities, a conclusion is made about the presence of the desired object in the image.

At the first stage of the analysis, the neural network checks the image for the presence of an object and selects it in a frame. To do this, we use the efficient YOLO (You only look once) algorithm, which allows you to select objects in the image. In the second stage of the analysis, the neural network determines the exact boundaries of the object. Algorithms for step-by-step reduction of image quality are used to search for known dependencies (distinctive features or patterns of the desired object in the image). The image is convoluted step by step from layer to layer by mixing neighboring pixels, depending on the task to the size of 2x1 pixels. To search for objects and their distinctive features, a neural network is trained using a prepared data set of the desired object. To prepare the sample for training, in the first approach, it was decided to divide the apples into 2 classes: the apple and the background, and make a markup of the photos. The open source program VGG ImageAnnotator (Fig.

2) was chosen as the markup.

4. Practical Implementation

4.1. Processing the results of the experiment

A Basler ace 1920-155uc camera with a GigE interface and a Sony IMX174 CMOS sensor with a

frequency of 164 frames per second was used for field research. The camera matrix has a resolution of up to 1920 x 1200 pixels, a resolution of 2.3 mega-pixels. To measure the illumination, a Radex Lupin luxmeter (Quarta Rad, China) was used, with a relative measurement error of 10%. As a result of the research, a computer vision system with a neural network for recognizing apple fruits was developed.

To configure and verify the calculated parameters, an analysis of its operation was carried out. To avoid

errors caused by partially hidden apples, only fully visible apples (from at least one side of the canopy) were considered in the study. This is done to ensure that the same apples were used to compare the number of fruits determined by the recognition system and the number measured manually. The results of identification of apple fruits on the tree crown are shown in Figure 3, 4. The results of the experiment are presented in Table 2.

It was found that the accuracy of estimating the number of apples on the tree crown compared to the

true value measured manually was at least 88.9%. With the help of the developed intelligent system, under changing climatic conditions, an average of 45 apples per tree crown were determined, with their true value of 50 pieces. The average absolute percentage error was 11.2% with a five-fold repetition of the measurements.

The main errors in estimating the number of apples are related to the segmentation of low-resolution images. This is due to the detection of only partial areas of apples or the erroneous perception of the environment and background as areas of apples, which led to inaccuracies in the identification of apples in the images. In addition, inaccuracies in the estimation of the number of apples are due to the low resolution of the images. These errors in estimating the number of apples per tree crown can potentially be reduced by increasing the camera resolution to 3840 × 2160 pixels.

Neural network quality assessment

The calculation of metrics for assessing the quality of the developed neural network was carried out on different samples, more than 5 thousand photos of an apple orchard were used for agaliz (Fig. 5).

Metric

The value of the area under the Precision-Recall curve when using two classes was 0.908. To assess

the definition of errors, an ROC curve was constructed, on which the values of the proportion of False positive rate (FPR) are set along the abscissa axis, and the proportion of true positive responses True

Positive Rate (TPR) along the ordinate axis (Fig. 7).

The higher the graph, the greater the TPR value and the higher the quality of the resulting model.

To identify how the developed model differs from the random one, the area under the ROC curve (AUC

ROC metric) was calculated. AUC-ROC is a quality metric, defined from 0 to 1. The AUC-ROC of a

random model is 0.5. Analysis of the resulting graph showed that the value of the area under the AUC

ROC curve is 0.957.

The results of processing the obtained data, the calculated values of the multi-classification metrics are presented in Table 4. The PR curve using six classes is shown in Figure 8. The value of the area under the Precision-Recall curve using six classes is 0.812. The ROC curve using six classes is shown in Figure 9. The area under the ROC curve using six AUC classes is 0.911

5. Discussion

As a result of the field experiment, it was found that errors in estimating the number of fruits were

mainly caused by inaccurate image segmentation, as well as the low resolution of the camera used.

It is revealed that the convolutional recurrent deep learning network is the most suitable neural network for the tasks of identifying apple fruits, since its use makes it possible to recognize the contour of the fruit with high accuracy in conditions of changing climatic parameters. This is in demand when implementing digital technologies in the field. The developed software and hardware complex based on the created neural network will allow for digital monitoring both by photo materials and by video stream in online mode. Using the created neural network and class allocation algorithms, the developed intelligent system will be able to function stably in industrial plants regardless of the size and interference of the foliage, determine the color of the fruit surface and the size of the fruit, identify the presence of diseases and defects of the fruit with a probability of at least 99%. This is possible as a result of the incremental expansion of the dataset during the operation of the complex and the gradual evolution of the solution by training the network in the process of working on new data. The developed system will expand the functionality not only for monitoring the productivity of fruit crops, but also for robotic fruit harvesting.

This system is already associated with robotic platforms for identifying trees and apples during their collection [29], during spraying [30], and can also be applied to large bushes and strawberries [31]. In the future, functional systems will be expanded to determine not only its fruit quality, but also flavors.

6. Conclusions

As a result of the research, an intelligent system with a neural network was developed to estimate

the number of apples on the crowns of trees. The analysis showed that the developed neural network model has high performance and high quality ordering of class objects. The area under the ROC curve and PR curve is in the range 0.5 < AUC/PR < 1, which indicates a high probability that the classifier will be able to accurately distinguish between positive and negative class values.

The harmonic mean between Precision and recall was Average F-score = 0.946, which indicates a high degree of search for relevant results. 7. References

References

[1] A. Wu , J. Zhu , T. Ren . Detection of apple defect using laser-induced light backscattering imaging and convolutional neural network . Computers and Electrical Engineering , 2020 , 81 , 106454 .

[2] B. Zhang , W. Huang , L. Gong , J. Li , C. Zhao , C. Liu , D. Huang . Computer vision detection of defective apples using automatic lightness correction and weighted RVM classifier Journal of Food Engineering , 2015 , № 146 , pp. 143 - 151 .

[3] Y. LeCun , B. Boser , J. Denker etc . Backpropagation Applied to Handwritten Zip Code Recognition, Neural Computation , vol. 1 , pp. 541 - 551 ( 1989 )

[4] P. Baranowski , W. Mazurek , J. Pastuszka-Wozniak , Supervised classification of bruised apples with respect to the timeafter bruising on the basis of hyperspectral imaging data Postharvest Biology and Technology, № 86 , pp 249 - 258 .

[5] J. Steinbrener , K. Posch , R. Leitner . Hyperspectral fruit and vegetable classification using convolutional neural networks . Computers and Electronics in Agriculture . 2019 . N162. 364 - 372

[6] A. Krizhevsky , I. Sutskever, G. Hinton. ImageNet Classification with Deep Convolutional Neural Networks , Communications of the ACM , No. 60 ( 6 ), pp. 84 - 90 . ( 2012 )

[7] Aleotti J. , Caselli S. Interactive teaching of task-oriented robot grasps . Robotics and Autonomous Systems , No. 58 ( 5 ), pp. 539 - 550 ( 2010 )

[8] Daoud N. , Gazeau J.P. , Zeqhloul S. , Arsicault M. A real-time strategy for dexterous manipulation: Fingertips motion planning, force sensing and grasp stability , Robotics and Autonomous Systems , No. 60 . 377 - 386 ( 2012 ).

[9] Jimenez A.R. , Ceres R. , Pons J.L. A survey of computer vision methods for locating fruit on trees . Trans ASAE, No. 43 , 1911 - 1920 ( 2000 ).

[10] I ̇. Kavdır , D.E. Guyer . Evaluation of different pattern recognition techniques for apple sorting Biosystems engineering , 2008 , № 99 , pp. 211 - 219 .

[11] D. Unay , B. Gosselin , O. Kleynen , V. Leemans , M.-F. Destain , O. Debeir . Automatic grading of Bicolored apples by multispectral machine vision . Computers and Electronics in Agriculture, 2011 , № 75 , pp. 204 - 212 .

[12] M.M. Sofu , O. Er , M.C. Kayacan , B. Cetisli . Design of an automatic apple sorting system using machine vision . Computers and Electronics in Agriculture. 2016 . N127. 395 - 405 .

[13] J. Lv , F. Wang , L. Xu , Z. Ma , B. Yang . A segmentation method of bagged green apple image 2019 . Scientia Horticulturae . 2019 . N246. 411 - 417 .

[14] S. Li , H. Luo , M. Hu , M. Zhang , J. Feng, Y. Liu , Q. Dong , B. Liu . Optical non-destructive techniques for small berry fruits: A review . Artificial Intelligence in Agriculture . 2019 . N2. 85 - 98 .

[15] J. Blasco , N. Aleixos , E. Moltó. Machine vision system for automatic quality grading of fruit Biosystems Engineering , 2003 , № 85 ( 4 ), pp. 415 - 423 .

[16] O. Kleynen , V. Leemans and M.-F. Destain . Development of a multi-spectral vision system for the detection of defects on apples . Journal of Food Engineering , 2005 , № 69 , pp. 41 - 49 .

[17] J. Gene-Mola , E. Gregorio , J. Guevara , F. Auat , R. Sanz-Cortiella , A. Escola , J. Lorens , J.-R. Morros , J. Ruiz-Hidalgo , V. Vilaplana , J. R. Rosell-Polo . Fruit detection in an apple orchard using a mobile terrestrial laser scanner . Biosystems engineering . 2019 . N187. 171 - 184 .

[18] A. Gongal , S. Amatya , M. Karkee , Q. Zhang , K. Lewis . Sensors and systems for fruit detection and localization: a review . Computers and Electronics in Agriculture. 2015. N116 . 8 - 19 .

[19] Khort , D. , Kutyrev , A. , Smirnov , I. , Osypenko , V. , Kiktev , N. Computer vision system for recognizing the coordinates location and ripeness of strawberries , Communications in Computer and Information Science , 2020 , 1158 , pp. 334 - 343 . DOI: 10 .1007/978-3- 030 -61656-4_ 22 .

[20] I. Smirnov , A. Kutyrev , N. Kiktev . Neural network for identifying apple fruits on the crown of a tree. E3S Web Conf ., Vol. 270 , 2021 , International scientific forum on computer and energy Sciences (WFCES 2021 ), DOI: 10.1051/e3sconf/202127001021

[21] Priya Narayanan , Alan Marc Lefcourt, U. Tasch . Tests of the Ability to Orient Apples using their Inertial Properties ( 2007 ). ASABE Annual International Meeting, Technical Papers. 13 . DOI: 10 .13031/ 2013 .22954

[22] Hnatiienko H. ; Domrachev V.; Saiko V. Monitoring the condition of agricultural crops based on the use of clustering methods , ( 2021 ) 15th International Conference Monitoring of Geological Processes and Ecological Condition of the Environment , Monitoring 2021 , doi: 10.3997/ 2214 - 4609 .20215K2049

[23] Ivokhin , O.V. Oletsky , O.V. Restructuring of the Model “State-Probability of Choice” Based on Products of Stochastic Rectangular Matrices . Cybern Syst Anal . Vol. 58 ( 2 ), pp. 242 - 250 ( 2022 ). https://doi.org/10.1007/s10559-022-00456-z

[24] A. Wu , J. Zhu , T. Ren . Detection of apple defect using laser-induced light backscattering imaging and convolutional neural network . Computers and Electrical Engineering , 2020 , 81 , 106454 .

[25] A.K. Bhatt , D. Pant . Automatic apple grading model development based on back propagation neural network and machine vision, and its performance evaluation . AI & Soc ., № 30 ( 1 ), pp. 45 - 56 .

[26] I. Kavdir , D. E. Guyer . Comparison of artificial neural networks and statistical classifiers in apple sorting using textural features . Biosystems Engineering . 2004 . N89. 331 - 344 .

[27] Maxwell , A.E. , Warner T.A. , Guillén L . A. Accuracy Assessment in Convolutional Neural NetworkBased Deep Learning Remote Sensing Studies - Part 2: Recommendations and Best Practices . Remote Sens . 2021 . 13 . 2591 DOI: 10.3390/rs13132591

[28] Maxwell , A.E. , Warner T.A. , Guillén L . A. Accuracy Assessment in Convolutional Neural NetworkBased Deep Learning Remote Sensing Studies - Part 1: Literature Review . https://doi.org/10.3390/rs13132450

[29] Kutyrev , A. ; Kiktev , N. ; Jewiarz , M. ; Khort , D. ; Smirnov , I. ; Zubina , V. ; Hutsol , T. ; Tomasik , M. ; Biliuk , M. Robotic Platform for Horticulture: Assessment Methodology and Increasing the Level of Autonomy . Sensors 2022 , 22 , 8901. https://doi.org/10.3390/s22228901

[30] Khort , D. ; Kutyrev , A. ; Kiktev , N. ; Hutsol , T. ; Glowacki , S. ; Kuboń, M. ; Nurek , T. ; Rud , A. ; GródekSzostak , Z. Automated Mobile Hot Mist Generator : A Quest for Effectiveness in Fruit Horticulture . Sensors 2022 , 22 , 3164. https://doi.org/10.3390/s22093164

[31] Khort , D. , Kutyrev , A. , Filippov , R. , Kiktev , N. , Komarchuk , D. Robotized Platform for Picking of Strawberry Berries, Science and Technology (PIC S&T) , 2019 , pp. 869 - 872 . DOI: 10 .1109/PICST47496. 2019 . 9061448 .