Agriculture faces major challenges in sustainably feeding a growing global population, making accurate crop yield estimation essential for informed decision-making by farmers. While traditional methods such as field surveys can be helpful, they are often limited by issues of accuracy, cost, and time efficiency.
Tomato (Solanum lycopersicum) is a crucial vegetable crop globally, boasting 183 million tonnes in 2018 [1]. Native to Central and South America, the tomato was introduced to Europe in the 16th century, quickly gaining popularity for its delicious, nutrient-rich fruits loaded with vitamins, minerals, and antioxidants [2]. Major producers include China, India, the United States, and Turkey, with significant cultivation also occurring in African nations such as Nigeria, Egypt, Morocco, and Algeria, primarily for local consumption [3]. Tomatoes are generally classified into two main varieties: determinate, which have limited growth, and indeterminate, which continue growing throughout their lifecycle.
Whether cultivated in open fields or under protective covers like greenhouses, tomato farming requires careful irrigation due to the plant's deep taproot system. Furthermore, challenges such as pest infestations-like downy mildew and Botrytis necessitate the use of appropriate cultivation practices and phytosanitary measures to ensure optimal yields.
Several approaches have been investigated in the literature to address the challenge of fruit weight estimation. For instance, Yamamoto et al. [4] developed a method to accurately count individual tomato fruits from images of plants grown in a laboratory setting. This method employed decision trees to analyze pixel color characteristics, achieving precise pixel-level segmentation. Post-processing was then applied to group pixels corresponding to fruits, en-abling the extraction and counting of fruit centroids. The study reported a detection precision of 0.88 and recall of 0.80, demonstrating the method's efficacy in controlled environments for tomato detection and counting.
In Indonesia, the increasing demand for tomatoes necessitates efficient post-harvest handling. A study by Sari et al. [5] proposed a sorting system that categorizes tomatoes based on color, size, and weight using image processing with the OpenCV [6] library. The system sorts tomatoes into red, yellow, and green categories and measures dimensions by identifying the outermost points of the detected fruits. It utilizes a weight sensor for mass measurement. The prototype, which incorporates a webcam, Arduino, and conveyor system, achieved 100% accuracy in color detection and 95% in weight measurement, although dimensional measurement accuracy was only 5%.
Van Daalen et al. [7] examined the application of augmented reality (AR) in agriculture, focusing on detecting tomato ripeness using the 3D scanning capabilities of the HoloLens [8]. Their experimental setup, which included various tomato varieties, highlighted both the opportunities and challenges of using AR for hands-free tasks like training and harvesting in greenhouse environments.
Similarly, Lee et al. [9] proposed an artificial intelligencebased system for tomato detection and mass estimation, utilizing multi-class detection and instance-wise segmentation. By analyzing a tomato image dataset with a calibrated vision system, the study demonstrated a high correlation between fruit dimensions and mass. Their method achieved a mean absolute percentage error of 7.09%, showcasing the effectiveness of computer vision and machine learning for automating tasks such as yield monitoring and fruit sizing.
In another study, Nyalala et al. [10] developed seven regression models, including Support Vector Regression (SVR) [11] and artificial neural networks (ANNs) [12] with different training algorithms. These models effectively estimated fruit weight and volume, offering significant potential for improvements in fruit sorting and grading processes.
Basak et al. [13] introduced a non-destructive method for estimating strawberry fruit weight using machine learning models. By analyzing 900 samples from three different strawberry cultivars, they used image processing to calcu-late pixel numbers. Linear regression (LR) and non-linear SVR models were applied, resulting in training and testing accuracies of 96.3% and 89.6%, respectively.
This study focuses on applying recent advancements in computer vision, particularly object detection, and machine learning algorithms to estimate tomato weight from realworld images. The subsequent sections describe the equipment used, the structure and composition of the dataset, and the methodology employed to generate accurate quantitative measures such as projected surface area and total weight for detected fruits. Our findings demonstrate the effectiveness of this approach. Additionally, we discuss the challenges faced and propose recommendations for future research.
The data used in this study consists of tomato fruit images collected both online and in the field under real-world conditions. The dataset includes a total of 180 images obtained online and 100 images taken in the field, containing a total of 1143 tomato fruit instances. Table 1 illustrates the composition of our dataset.
Images captured in the field helped to collect additional information such as actual fruit area and actual fruit weight, which enriches the dataset by providing accurate and relevant measurements for tomato fruit weight estimation. Table 2 presents additional insights concerning field-captured images. Upon analysis of the table, the average fruit weight is 35.30 g , with a standard deviation of 14.56 g . The average true area is 2673.48 mm 2 , with a standard deviation of 873.68 mm 2 . Quartile values provide insights into the distribution of the data. Thus, 25% of the fruits have a weight of less than 25.21 g, 50% have a weight of less than 37.00 g , and 75% have a weight of less than 43.49 g. For the actual surface area, 25% of fruits have an area less than 2, 024.93 mm 2 , 50% have an area less than 2, 779.53 mm 2 , and 75% have an area less than 3, 219.12 mm 2 .
To estimate tomato fruit weights, we developed a four steps approach (see figure 1)
To train our segmentation model, we prepared a dataset of tomato images, labeled in the COCO format. The dataset consisted of 180 images containing 1043 instances of tomatoes, sourced from both the internet and field photography, and annotated using the ROboflow platform. We employed the Mask R-CNN instance segmentation model through the Detectron2 framework, selecting the mask_rcnn_R_50_FPN_3x configuration developed by Facebook AI Research. This model, pre-trained on the COCO dataset, combines the Mask R-CNN architecture with a ResNet-50 backbone and Feature Pyramid Network (FPN) for high-performance, multi-scale object detection.
To evaluate the projected area of each tomato from images, a dataset was constructed, including individual images of tomatoes, their actual weight in grams, the total number of pixels in the image, the number of pixels corresponding tomato (obtained by semantic segmentation), and the total area of the image in square meters, obtained by camera calibration.
The estimation of the projected area took place in two steps: first, the segmentation mask allows us to calculate the area in pixels occupied by the tomato in the image. Then, a camera calibration converted this pixel area into an actual metric area, using a coin as a reference object. By photographing the tomatoes under the same conditions as the reference piece, the resulting conversion factor was used to convert the pixel area of each fruit into a measure of its actual projected area in metric units. This method uses a rule of three, where the actual surface area of the tomato (𝐴 𝑡𝑜𝑚𝑎𝑡𝑜 ) is estimated based on the number of pixels corresponding to the tomato in the image (𝑃 𝑡𝑜𝑚𝑎𝑡𝑜 ), using the conversion factor established during calibration:
With this method, we were able to estimate the real surface area of each tomato in physical space from segmentation in image space, thanks to precise calibration using a reference object.
To estimate the weight of the tomatoes based on their projected surface area, we tested several regression models, including Simple Linear Regression (SLR), Multiple Linear Regression (MLR), and Partial Least Squares Regression (PLSR). These models aimed to establish a mathematical relationship between the surface area (independent variable) and the weight (dependent variable) of the tomatoes. We also applied 10 -fold cross-validation to each model to reduce the likelihood of overfitting. Figure 1 depicts the summary of the methodology adopted in this study.
Figure 2 illustrates the model's accuracy, while Figure 3 depicts the evolution of the cost function
The performance of the model was evaluated on the test set consisting of 19 images containing a total of 149 tomato annotations. The Average Precision (AP) metric was used to quantify the model's ability to correctly detect and segment tomatoes under various conditions.
Table 3 presents the results obtained for the detection and semantic segmentation tasks. We observe an average AP of 55.9% for detection and 54.6% for segmentation on different IoU thresholds between 0.5 and 0.95. The model achieves better performance on large fruits (AP of 66.1% in detection) than on small tomatoes (AP of 30.3%).
These results confirm the model's effectiveness in detecting and segmenting tomatoes in real-world conditions. Further data annotation and model optimization are expected The projected surface area of each fruit was derived from the segmented mask by calculating the pixel area, then converting it to real-world units using camera calibration information as defined in Equation 1. This method achieved a precision of approximately 95.
For tomato weight estimation, a subset of the dataset containing real-world images was used, which included precise data on both the actual weight of each tomato and their projected surface area. A mathematical relationship between the weight and projected area was established through the evaluation of several regression methods. The algorithms tested included Least Squares Regression (LSR), Multiple Linear Regression (MLR), and Support Vector Machines (SVM), and their performance was compared using cross-validation and Mean Square Error (MSE) as the evaluation metric.
Table 4 highlights the performance metrics of the tested models.
Among the evaluated models, Lasso Regression achieved the best performance, with a MAE of 5,776 and an MSE of 62.99.
The corresponding model equation is:
Table 3.1 presents the prediction results on the test dataset, where our model achieved a relative error of 7.09% in estimating the total weight. When applied in an autonomous field system, this method shows great potential to enhance yield estimation efficiency, helping farmers save time and reduce labor costs.
The study employed a multi-step methodology to estimate tomato fruit weights from images. First, a Mask R-CNN model, using the mask_rcnn_R_50_FPN_3x configuration, was trained on a dataset of 180 images containing 1043 tomato instances. After detection and segmentation, the projected surface area of each tomato was estimated using a calibrated conversion from pixel area to metric units, achieving approximately 95% accuracy. For weight estimation, several regression models were evaluated on a subset of real-world images with known weights and projected areas. Among the regression models evaluated, the Lasso Regression algorithm demonstrated superior performance in estimating tomato weights. This model achieved a Mean Absolute Error (MAE) of 5.776 grams and a Mean Squared Error (MSE) of 62.99 grams2. Our model outperformed the approach described by Lee et al. [9], which reported an MAE of 7.09 grams for a similar tomato weight estimation task.
When applied to the test dataset, this model achieved a relative error of 7.09% in estimating the total weight of tomatoes. These results demonstrate the potential of this combined approach for automated tomato yield estimation, although the ideal conditions of the study (fully visible fruits) suggest that further research is needed to address real-world challenges such as occlusion.
While this study yielded promising results, it's important to acknowledge its primary limitation: the experiments were conducted under idealized conditions that do not fully represent real-world agricultural environments. All tomatoes in the study were fully visible and unobstructed, which rarely occurs in actual fields where fruits are often partially hidden by leaves, branches, or other fruits. This idealization may lead to overly optimistic performance estimates.
To bridge this gap and enhance the model's practical applicability, future research will focus on developing robust occlusion handling techniques, such as implementing advanced image processing algorithms for reconstructing partially obscured fruits or using ellipse fitting methods to estimate the full shape of partially visible tomatoes.
Additionally, creating more representative datasets that reflect the challenging conditions found in real agricultural settings, including various levels of occlusion and diverse growth stages, will be crucial. By addressing these limitations and training on more diverse and challenging datasets, future iterations of this system could significantly improve in accuracy and robustness, making it a more reliable tool for automated agricultural yield estimation in real-world scenarios.
This study successfully introduced an innovative approach for accurately assessing tomato crop yields through the use of advanced image processing, computer vision, and artificial intelligence techniques. The results align closely with the objectives of estimating both the quantity and total weight of fruits, highlighting the practical benefits of this methodology for farmers.
Looking ahead, future enhancements will focus on refining the approach by integrating multispectral imaging to improve data acquisition. Additionally, algorithmic advancements, including image generation and ellipse fitting techniques, will be employed to tackle challenges related to occlusion. These developments will enhance the model's scalability and robustness, facilitating large-scale deployment in real-world agricultural settings. The anticipated implementation of this approach in automated systems that utilize drones and ground-based robots presents exciting opportunities for digital agriculture, paving the way for precise, efficient, and automated yield estimation.