The article presents a system for monitoring apple trees in orchards during the flowering period. To control the technological operation of thinning trees during the flowering period, monitoring of this operation is necessary to select the most effective method and assess the quality of its implementation. The purpose of the study is to develop a method for monitoring apple blossoms on the tree crown based on machine learning algorithms to control the quality of the technological operation of thinning apple blossoms. A deep-learning neural network has been developed to recognize apple flowers and buds on received camera frames. The analysis of the results indicated that the YOLOv8 model categorized the "rosebud" class with an mAP (mean Average Precision) metric of 0.448, and the "Flowering" class with an mAP metric of 0.691.
The process of flower bud formation involves careful planning of measures to thin out flowers and stimulate the formation of ovaries [1]. Many years of research have proven that thinning flowers is an important stage in the cultivation of intensive apple orchards, which allows you to balance the number and size of fruits, and also ensures a sufficient number of flower buds for the next year without damaging the current year's harvest. Timely flower thinning allows you to avoid excessive fruiting and subsequent depletion of the tree’s carbohydrate reserves, which negatively affects the overall health of the tree [2]. Existing methods, including manual, chemical and mechanical, show high efficiency in thinning operations. However, the main difficulty in their use remains the significant degree of unpredictability of the final result. Typically, flower thining is based on a visual assessment of flower load by expert agronomists, which may be inaccurate and error-proof. In work [3] by american researchers W. Yuan et al. an algorithm developed an algorithm for mapping the density of apple flower using point clouds reconstructed using RGB images and photogrammetry obtained from unmanned aerial vehicles (UAVs).
In intensive gardening, there are many methods for calculating the optimal number of fruits and flowers per tree. The number of fruits on the tree (in pieces) after it enters the fruiting period should be equal to the distance between the trees in the row (in centimeters), with about 30 leaves per fruit. There should be no more than 4-6 fruits per 1 centimeter of shoot circumference four weeks after flowering, while the distance between the fruits on the branch should be about 15-20 centimeters (5-6 kg of fruits per 1 cm of the cross-section of the tree trunk). The optimal norm of crop load (number of fruits/tree, pcs.), depending on the age of the plantings in intensive gardens, is in the first year - 5-6 fruits, in the second year - 20-30 fruits, in the third year - 40-50 fruits, in the fourth year – 60-80 fruits, the fifth year 80-100 or more fruits. In this case, the number of fruits ranges from 4% to 8% of the number of flowers on the tree. In this regard, it is necessary to monitor the thinning operation to select the optimal method and assess the quality of the technological operation.
The purpose of the study is to develop a method for monitoring apple blossoms on the tree crown based on machine learning algorithms to control the quality of the technological operation of thinning apple blossoms.
Most developed computer vision systems for apple flower and fruit recognition focus on cluster detection and segmentation. Before the widespread use of deep learning methods, cluster segmentation was performed by automatically extracting color and texture features from RGB images using techniques such as intensity thresholding and morphological image processing (Krikeb et al., 2017 [4], Aggelopoulou et al., 2011 [5], Hocevar et al. 2014 [6]). An analysis of the studies has shown that cluster segmentation algorithms have low performance due to variability in lighting conditions, biological variability of crowns, mutual overlap of objects (for example, branches, leaves, flowers and ovaries) and changes in the shape, size, color and other parameters of objects.
The use of machine learning methods and convolutional neural networks (CNN) to detect and segment apple flowers and fruits has significantly improved the performance of their classification and recognition. For semantic segmentation, high probability cluster regions are first extracted using various CNN based architectures such as, Inception (GoogLeNet), ResNet (Residual Networks), VGG (Visual Geometry Group), DenseNet followed by classification or region detection methods such as support vector machines (SVM), region growth refinement (RGR) and shape-constrained level generation method (Dias, P. A. et al., 2018) [7,8]. In addition to semantic segmentation, methods for cluster detection and segmentation of individual flower clusters have been studied using widely accepted approaches such as Faster R-CNN (Farjon et al., 2019) [9] and Mask RCNN (Bhattarai et al., 2020) [10]. An analysis of research by famous scientists has shown that using the YOLO (You Only Look Once) algorithm to detect objects in images and videos provides high speed and accuracy when performing the task of object recognition in real time. The YOLO algorithm splits an image into a grid of cells, using many anchors (predefined boxes) to predict the objects in each cell. This method allows you to detect multiple objects and determine the coordinates, classes, and object certainties for each detected object. This method is actively used in computer vision to solve various problems, such as object detection on roads for autonomous cars, medical image analysis, video surveillance and other applications where accurate and fast object detection is required. (Wu et al., 2020) [11] and MaskScoring R-CNN (Tian et al., 2020) [12]. The YOLO model has many different architectures and versions, each of which improves detection performance and accuracy.
To develop models for apple blossom recognition and classification, the modern YOLOv8 model was used, which uses new features to improve performance and optimize training hyperparameters. The YOLOv8 model compared to previous models (YOLOv2-YOLOv7) has higher speed and accuracy.
YOLOv8 provides support for various artificial intelligence tasks, including detection, segmentation, pose estimation, tracking, and classification. Thanks to this versatility, users can leverage the capabilities of YOLOv8 in various application domains, including science and technology [13]. The visualization of the YOLOv8 network architecture in relation to the problem of recognizing flowers and buds of an apple tree is presented in Figure 1. The YOLOv8 model can be divided into two main components: the backbone and the head. The backbone is a modified version of the CSPDarknet53 architecture, comprising 53 convolutional layers and employing partial inter-stage connections to enhance information flow between layers. The YOLOv8 head includes multiple convolutional layers followed by fully connected layers. These layers are responsible for predicting bounding boxes, objectness scores, and class probabilities for objects detected in an image.
A key feature of YOLOv8 is the incorporation of a self-control mechanism in the head network. This mechanism allows the model to focus on different areas of an image and assign importance to different features based on their relevance to the task at hand. Another important characteristic of YOLOv8 is its ability to perform multi-scale object detection. The model utilizes a feature pyramid network for detecting objects of different sizes and scales in an image, consisting of multiple layers capable of detecting objects at various scales, enabling the model to efficiently detect small objects such as flowers.
YOLOv8 is an anchor-free model, meaning it predicts the center of an object directly rather than the offset from a known anchor box. This approach reduces the number of predicted bounding boxes, thereby accelerating the Non-Maximum Suppression (NMS) process a complex post-processing step that filters candidate detections after inference. In comparison to previous versions, YOLOv8 incorporates mosaic augmentation, involving the fusion of four images. This allows the model to learn objects in new spatial contexts with partial overlap and in the presence of surrounding pixels.
The YOLOv8 model delivers state-of-the-art performance, striking a balance between accuracy and speed, making it well-suited for real-time object detection tasks.
To collect a set of data, images of flowers and apple fruits, to train the neural network model, a Sony Alpha ILCE-7M3 digital camera was used using a Sony FE 24-240mm lens. Shooting parameters included an aperture of f/7.1, a lens focal length of 24mm, and an image resolution of 4000x2672 pixels. The total number of images was 3000 pieces. These images were evenly divided into two categories, each containing 1500 images, respectively representing flowers in the pink bud stage and flowers in the active bloom stage. The data set was collected in the research and production testing department of the Federal State Budgetary Institution Federal Scientific Center for Horticulture (Moscow region, Mikhnevo settlement).
To train the convolutional neural network model, a data set (images) was prepared. The RoboFlow web service was used to perform the image tagging process. The following classes were specified for the classification and recognition of objects: the “rosebud” class (flowers in the pink bud stage) and the “flowering” class (flowers in the blooming stage) (Fig. 2).
Marking of objects is carried out using rectangles - this is the process of outlining the objects of interest to us in the image with rectangles indicating the corresponding class. A JSON markup file is used to store class information, storing attributes such as "class" (object class), "x_center" and "y_center" (center coordinates), and "width" and "height" (width and object height). To expand the data set and improve the performance of the model, the augmentation method was used. The tools used are Flip, Rotation ±15º, Hue ±25º, Noise 5%, Blur 2.5 px, Brightness ±25%. Flip and Rotation help deal with the problem of non-invariance under rotations and reflections. Changing Hue helps the model better generalize color information. Adding noise and blur helps the model become more robust to artifacts in the data (Figure 3).
As a result, the dataset was expanded to 6000 images. To objectively evaluate the performance of the model, the data set is divided into training, test and validation sets in the ratio of 70% (4200 images), 30% (1200 images) and 10% (600 images), respectively.
To train a convolutional neural network, the Transfer Learning method is used, in which a
pretrained model is used to solve a new problem. A configuration file has been created that contains
information about the paths to the training and test images, as well as the paths to the markup
files. In the model configuration file, learning parameters are defined, such as the number of
epochs (Epoch) of training, data batch size (batch size), learning rate (Learning Rate). These
parameters play a key role in shaping the model's training process. The optimal values of these
parameters are determined empirically as a result of testing and tuning through a series of
experiments. This approach involves an iterative process in which multiple training runs are
conducted with varying values for each parameter. We used pre-trained weights on the COCO
(Common Objects in Context) data set, which allows us to speed up the process of training the
neural network on its own data. The COCO dataset is a rich set of images with labels containing a
variety of objects in different scenes. The training process included minimizing the cumulative
loss function, as well as optimizing model weights using gradient descent and weighting
coefficients. To update the weighting coefficients during model training using gradient descent,
formula (
The Box Loss formula includes weights to balance the importance of different loss components in model training. After completion of training, the performance of the models was assessed using a test sample (data set not used in the training process). To quantitatively assess the performance of the developed models in recognizing and classifying objects (apple flowers and fruits), the wellknown metrics of accuracy (Precision), completeness (Recall), F1-score (F-score) and mean Average Precision (mAP) were used. found using formulas 3-6.
The learning rate (α) determines the step size in the direction of the gradient. This process is
repeated over several iterations (epochs) until convergence or a specified stopping criterion for
model training is achieved. As a result of this process, the trained convolutional neural network
model is gradually tuned to the training data, improving its ability to detect objects. To assess the
accuracy of predicting the coordinates of the bounding box (box) for objects in the image, the
YOLOv8 model training algorithm uses the Box Loss function, found using formula (
Recall=
∑ 1 С 1 С ∑
C
C
TPi i=1 TPi + FPi
TPi i=1 TPi + FNi F1 (score) =
∑ 1 С
C 2 · Precisioni · Recalli i=1 Precisioni + Recalli Averageaccuracy(mAP) =
∑ 1 С
C i=1
APi where С - total number of classes, - number of correctly classified positive examples for the class , - number of false positive examples for a class , - number of false negative examples for a class , - area under the precision-recall curve for a class .
In these formulas, TPi, FPi and FNi refer to the true positive, false positive and false negative
examples for class i, respectively. Area under the curve The precision-recall curve for class i
is calculated using the Precision-Recall curve for different Classification Thresholds. The
classification threshold determines the probability value above which an object is considered to
belong to one of the given classes (“rosebud”, “flowering”). The Confidence indicator is defined as
the maximum probability of an object belonging to one of the classes according to formula 7:
(Confidenc)e = maxiP(ci|х)
where ( |х) – the model's predicted probability that object x belongs to the class , - the
operation of selecting the maximum value among all probabilities of objects belonging to different
classes.
(
To conduct the research, we used a computer system equipped with an Intel Core i9-10900X processor with twenty virtual threads. The model was trained using GPUs, using two NVIDIA GeForce RTX 2080 Ti video cards. The motherboard used is GIGABYTE X299 UD4 Pro. A 1 TB Intel PCI-E SSD was used for data storage. The system's RAM capacity was 32 GB using Kingston DDR4 DIMMs.
The Box loss-Epoch graph, constructed during the model training process, made it possible to determine the optimal number of training epochs, which achieves the best quality in determining the coordinates of the bounding boxes of objects, which amounted to 124 (Fig. 4).
To assess changes in precision and recall indicators depending on the epoch during the model training process, Precision-Epoch and Recall-Epoch curves were constructed. To assess the change in the average accuracy of the model depending on the number of epochs during the model training process, the mAP-Epoch curve was constructed. Analysis of the Precision-Epoch, Recall-Epoch, mAP-Epoch curves made it possible to determine the number of epochs when the model achieves the best combination of accuracy and completeness and to select the best hyperparameters to achieve the best performance of the model and maximum accuracy in detecting the “rosebud” and “flowering” classes, highlighting areas on strawberry leaves with calcium deficiency, which amounted to 94. The total training time for the YOLOv8 model using the CPU was 8 hours 35 minutes 25 seconds.
Examples of recognition of the “rosebud” class of the “flowering” class in images of the test data set using a trained model with the selection of affected areas in a frame are presented in Figure 5.
To assess the obtained values of accuracy and completeness when changing the threshold for making a decision in the classification problem being solved, the F1-score – Confidence, Precision – Confidence, Precision – Recall and Recall-Confidence curves were constructed (Fig. 6).
Analysis of the Precision-Recall plot resulted in a classification threshold of 0.56, which provides the best trade-off between precision and recall. The Precision-Confidence and RecallConfidence curves reflect the dependence of the accuracy and completeness of model predictions on the level of confidence used to decide about the presence of an object in the image. Analysis of the curves made it possible to estimate the optimal confidence level for the model, which was 0.52. The indicator provides optimal accuracy and completeness of class predictions with a minimum number of false positives of the neural network model. The resulting F1-Confidence graph allowed us to evaluate how changing the model’s confidence level affects the combined metrics of accuracy and recall, the ability to correctly classify objects and choose the optimal threshold for making a classification decision, which was 0.48. The F1-Confidence graph shows how the model responds to different levels of noise or the presence of outliers in the data. To evaluate the performance of the machine learning model, Confusion Matrix was built (Fig. 7). The matrix contains the main categories that reflect how the model classified the objects. The matrix allowed us to evaluate model errors and tune model parameters to improve its performance for recognizing the classes “flowering” and “rosebud”. Table 1 presents the coefficients of the three main metrics for each individual class and for the overall data set.
Analysis of the results showed that the YOLOv8 model classified the “rosebud” class with an mAP metric of 0.448, and the “Flowering” class with 0.691. Analysis of the research results showed that timely monitoring of apple flowers on an industrial plantation, carried out using a wheeled robotic platform using the YOLOv8 convolutional neural network for processing the received data, will allow recognizing and classifying apple flowers with high accuracy of up to 92.5%.
Neural networks are a powerful modern tool used in robotic platforms for various gardening operations [14,15,16]. Research on the use of machine learning to identify apples during harvest is described in [17]. Neural networks and other artificial intelligence systems are also used for other operations in agriculture, for example, in greenhouses when managing energy flows [18], monitoring the condition of agricultural crops [19], including nitrogen in wheat crops [20].
In further research, we set the task of counting the number of flowers for each row of plantings and eliminating duplication of their counting.
For this purpose, it is planned to develop a method for tracking (tracking process) objects or several objects on a video/set of frames by assigning a unique identifier (ID) to each recognized object. Tracking will allow you to determine the movement and changes in the position of flowers in images over time. Further expansion of the dataset and training of the model on new data, including images of fruits and ovaries, will allow monitoring the full cycle of the orchard production process in order to accurately predict yields.
Analysis of the obtained graphs made it possible to establish the optimal settings for the YOLOv8 convolutional neural network and select the confidence threshold at which the model shows optimal accuracy and completeness balanced with the number of detected objects. The configuration of the machine learning algorithm of the YOLOv8 model for recognizing apple flowers in the pink bud stage and apple flowers in the flowering stage has been determined: learning rate – 0.01 LR (learning rate), number of epochs – 130, mini-batch size (batch size) – 16.
The research used the method of training a convolutional neural network under conditions of a limited training sample volume obtained in the field using an RGB camera. The result showed that artificially increasing the volume of the training sample (images of apple blossoms), using tools such as Flip, Rotation ±15º, Hue ±25º, Noise 5%, Blur 2.5 px, Brightness ±25% can significantly improve the quality of neural network training, helps to adapt the system to real conditions, increases the accuracy of class feature detection by 16% compared to the data set without increasing the sample size. The conducted research shows the prospects for using the YOLOv8 convolutional neural network as part of a decision support system for monitoring and planning the thinning of flowers and the formation of ovaries.
[10] Bhattarai, U., Bhusal, S., Majeed, Y., and Karkee, M. (2020). Automatic blossom detection in
apple trees using deep learning. IFAC-PapersOnLine, 53(