=Paper=
{{Paper
|id=Vol-3304/paper37
|storemode=property
|title=Classification of Fruit Varieties Based on Deep Learning
|pdfUrl=https://ceur-ws.org/Vol-3304/paper37.pdf
|volume=Vol-3304
|authors=Peipei Cao,Yuekun Pei,Zhi Chai,Jingyu Liu,Zuming Wang
}}
==Classification of Fruit Varieties Based on Deep Learning==
https://ceur-ws.org/Vol-3304/paper37.pdf
Classification of Fruit Varieties Based on Deep Learning
Peipei Cao, Yuekun Pei, Zhi Chai, Jingyu Liu and Zuming Wang
Dalian University, No. 10, Xuefu Street, Jinzhou District, Dalian City, Liaoning Province, Dalian, Index, China
Abstract
With the development of economy and the improvement of consumption level, consumers'
demand for high-quality fruits is gradually increasing. During the ripening period of fruit, fruit
defects determine its quality grading, and different varieties of fruit are liked by consumers
differently, and these factors determine their market sales prices. At present, the management,
quality grading and variety classification of fruit in my country mainly rely on manual labor,
which is greatly influenced by subjective factors. Therefore, a professional and effective
method is needed. Deep learning has the advantages of high recognition accuracy, short
recognition time, and strong anti-interference ability, and has received extensive attention from
scholars at home and abroad. This paper summarizes the research results published in recent
years on the application of deep learning in fruit defect detection and variety classification.
Keywords 1
Fruit grading; fruit detection; variety classification; deep learning
1. Introduction
Fruits are good for human health. Fruit is rich in vitamin A, vitamin B, vitamin C, various inorganic
salts, calcium, phosphorus, iron, iodine, etc., and contains a variety of amino acids, which are all
necessary nutrients for the human body. Fruits are indispensable in daily life. With the development of
economy and the improvement of consumption level, consumers' demand for high-quality fruits is
gradually increasing. During fruit ripening, fruit quality is mainly determined by the presence or
absence of fruit defects and size grading. At present, fruit grading and sorting mainly rely on manual
labor, which is low in efficiency, and is greatly affected by subjective factors and has large errors, which
is difficult to meet the needs of large-scale production. In addition, due to the advancement of
agricultural technology, multiple varieties of each fruit are bred with different characteristics to meet
the needs of consumers. Therefore, the accurate identification of fruit varieties and the selection of the
favorite varieties of consumers also have certain application value.
With the development of deep learning, it has been applied in commercial retail, smart agriculture
and other fields. In terms of commercial retail, more and more fresh food supermarkets have begun to
try to use smart fresh food scales to automatically identify the type of fruit and calculate the price to
realize one-stop payment. In smart agriculture, fruit can be monitored in real time and accurately, fruit
quality can be detected, fruit classification and sorting, maturity identification and defect detection, so
as to improve the yield and quality of fruit. The application of deep learning provides solutions for
expensive manual labor costs, increasing food demand, improving fruit quality, and more. This paper
aims to carry out research on the application of deep learning in fruit sales, to explore the detection of
fruit defects and classification of varieties, and to provide some references for researchers in this field.
ICBASE2022@3rd International Conference on Big Data & Artificial Intelligence & Software Engineering, October 21-
23, 2022, Guangzhou, China
EMAIL: 2099689793@qq.com(peipei Cao)
ORCID: 0000-0002-0311-0120(peipei Cao);
© 2022 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
300
2. Classification of fruit varieties based on deep learning
In the natural environment, there are many kinds of fruits, and there are more and more high-quality
varieties. Consumers have more and more choices of high-quality varieties, and different varieties also
determine their market prices. The traditional identification of fruit varieties is based on the shape, color,
surface texture, size and other characteristics of the fruit. Some fruit varieties are relatively similar in
maturity, size, color and taste, and the accuracy of identification by non-professionals is not high. The
solution is to automate the identification process to minimize human error from subjectivity. We
summarize recent papers on deep learning-based fruit variety classification. Table 1 summarizes the
deep learning models applied to fruit variety classification, including variety categories, data set
division, and accuracy.
Table 1
Classification of fruit varieties based on deep learning
fruit Variety years DL model Data set partitioning Accuracy
75%training 25%test
Peach 5 2020 3-layer CNN Take 20% of the training 94.4%
set as the validation set
tomato 7 2020 4-layer CNN 70%training 30%validation 93%
Lprtnrl 82% training 6% test
hazelnut 17 2021 98.63%
（4-layer CNN） 12% validation
date 9 2022 5-layer CNN - 94.8%
Plum 3 2020 Alexnet 80% training 20% test 91%-97%
320 training 60 test
litchi 4 2020 VGG16 98.35%
60 validation
Grape 6 2021 Modified VGG16 80% training 20% test 99%
grape
5 2018 ExtResnet 3667 training 300 test 99%
bunch
Inception- 62.5% training
olives 7 2019 95.91%
ResNetV2 37.5% validation
80% training
Kiwi 4 2021 Densenet121 97.79%
20% validation
EfficientNet-B4- 60% training 20% test
Oil Tea 4 2021 97.01%
CBAM 20% validation
mango 3 2019 Faster RCNN - 80%
banana 5 2021 RBF 420 training 50 test 98.81%
x-axis 89.1%
x-axis training 426 test55
y-axis92.7%
hazelnut 11 2021 ANN y-axis training 421 test55
z-axis86.8%
z-axis training 419 test53
In the recently published research on fruit varieties, some scholars use their own convolutional
neural networks to achieve variety classification. Among them, Dian Rong et al. [1] realized the
identification of peach varieties by constructing a one-dimensional convolutional neural network and
five kinds of peach VIS-NIR spectral databases. The accuracy is 100% in the validation dataset and
94.4% in the test dataset. This study shows that peach cultivars can be successfully distinguished using
VIS-NIR spectroscopy and deep learning. Mahmoud A. Alajrami et al. [2] studied seven tomato
classification methods. Taking the original image as input, a Convolutional Neural Network (CNN) is
used to extract features. The network consists of 4 convolutional layers with Relu activation functions,
301
and each convolutional layer is followed by a Max Pooling layer, which accepts 7 Different varieties
of tomato images are used as input. The test accuracy was 93%. [3] designed a CNN model named
Lprtnr1 to classify 17 hazelnuts. The Lprtnr1 model consists of an input layer, four convolutional layers,
a flattening layer, a fully connected layer, and an output layer. The proposed model is evaluated by
comparison with pretrained models (VGG16, VGG19, Resnet50 and InceptionV3). The proposed
model yielded 98.63% accuracy on the test set, and the classification accuracies of the VGG16, VGG19,
InceptionV3 and Resnet50 models were 73.14%, 72.14%, 61.18% and 80.00%, respectively. The
proposed model is found to perform better than the pretrained model in terms of performance evaluation
criteria. Khaled Marji Alresheedi et al. [4] classified nine jujube species based on classical machine
learning and deep learning, including Bayesian networks, support vector machines, random forests, and
multilayer perceptrons (MLPs). Convolutional Neural Networks (CNN) for deep learning. The feature
set includes color layout functions, blurred color and texture histograms, Gabor filtering, and pyramid
histograms of oriented gradients. The fusion of various features has also been extensively explored. The
fused feature set includes Color Layout+Gabor (SVM has the highest accuracy of 88%), PHOG+Gabor
(Random Forest has the highest accuracy of 89.6%), and PHOG+FCTH (MLP has the highest of 93.8%).
The performance of deep learning is also explored using innovative CNN models. The detection results
of deep learning are 1% higher than MLP (PHOG+FCTH classic feature set) and 2% higher than
random forest detection accuracy. To sum up, the general feature of these methods is that they act
directly on the original image, learn layer-by-layer features, and then use multi-layer networks to obtain
feature information [1-3]. Or combine the self-built CNN model with machine learning to find the
optimal recognition model [4].
Some scholars use the constructed convolutional neural network model to achieve variety
classification. For example, Francisco J. Rodríguez et al. [5] chose the Alexnet network model to
identify plum varieties, and the results showed that the accuracy rate was 91% to 97%. Yutaro Osako
et al. [6] implemented litchi variety classification through the VGG16 model. Got 98.33% accuracy.
Grad-CAM visualization shows that the model uses different breed-dependent regions for breed
identification. The study shows that deep learning can be used to distinguish lychee varieties from
images. Also Amin Nasiri et al. [7] adopted a modified VGG16 model with modifications in the global
average pooling layer, Dense layer, batch normalization layer, and Dropout layer, replacing the last
three Dense layers with a classifier to Modify the original VGG16. The average classification accuracy
is over 99%. This model provides a rapid, low-cost, high-throughput method for grape variety
identification. Bogdan Franczyk et al. [8] proposed a method to identify clusters of different varieties
of grapes using KSM, Resnet and ExtResnet to classify five types of grapes from a given image dataset.
KSM classification achieves 47% accuracy and Resnet classification achieves 89%. ExtResnet is a
combination of a deep learning Resnet model and a multi-layer perceptron that achieves 99% accuracy
for correctly classified grape clusters. Juan M. Ponce et al. [9] proposed a deep learning-based method
for olive variety classification. Verify the implementation of six different convolutional neural network
frameworks Alexnet, InceptionV1, InceptionV3, Resnet-50, Resnet-101, Inception-ResnetV2, and the
obtained accuracy rates are: 89.90%, 91.81%, 94.86%, 95.33%, 94.00% , 95.91%. The highest accuracy
was obtained when using Inception-ResnetV2. Inception-ResnetV2 is based on the Inception
architecture but inspired by Resnet and uses residual connections for training to speed up processing.
Qilong Wang et al. [10] studied the classification of kiwifruit varieties based on deep learning, and
studied the deployment and application of the kiwifruit classification model on the Jetson Nano artificial
intelligence development board. Use transfer learning based on three pretrained models Xception,
Resnet50, Densenet121. Comparing and analyzing the model size, training speed, convergence and
recognition accuracy, it was concluded that the transfer learning pre-training model based on
Densenet121 had the best classification effect on kiwifruit varieties, with fast convergence speed and
the smallest model, with a recognition accuracy of 97.79%. Xueyan Zhu et al. [11] classified nectarine
varieties, selected EfficientNet-B4 as the basic model for nectarine variety identification, and integrated
the convolutional block Attention module (CBAM) into EfficientNet-B4 to construct EfficientNet-B4-
CBAM, thereby Improve the focus ability and information expression ability of fruit area. Comparisons
are made with InceptionV3, VGG16, Resnet50, EfficientNet-B4 and EfficientNet-B4-SE. The
experimental results show that the accuracy of the EfficientNet-B4-CBAM model reaches 97.02%, and
the kappa coefficient reaches 0.96, which is higher than other methods in the comparative experiments.
Philippe Borianne et al. [12] used a Faster R-CNN network to identify mango varieties on a tree, and
302
the Faster R-CNN network had some limitations in detecting mango fruits and identifying the respective
varieties of mangoes simultaneously. The varietal identification rate of the detected mango fruits was
around 80%, although some errors were proven in fruit detection. Finally, we classify the methods used
by the above researchers. Some scholars use the trained models for direct training [5-6]. Others include
replacing a layer of the trained model with a classifier [7], combining the trained model with a
perceptron [8], continuing to transfer the trained model [10], and adding a convolution block [12] to
Get the best model, and find the most suitable pre-trained model after comparison.
Some scholars use other network models of deep learning to solve variety classification. Zilvanhisna
Emka Fitri et al. [13] classified banana varieties based on artificial neural network. The banana variety
classification system adopts the combination of image processing technology and artificial neural
network. The intelligent systems used are backpropagation and radial basis function neural networks.
Based on the confusion matrix calculation, the RBF accuracy is 98.81% from the test data. Omer Keles
et al. [14] classified hazelnut varieties by using artificial neural network ANN and discriminant analysis
DA. The physical, mechanical and optical properties of three axes of 11 hazelnut varieties were
determined. Artificial Neural Network (ANN) classification success rate: 89.1% on the X-axis, 92.7%
on the Y-axis, and 86.8% on the Z-axis.
3. Fruit defect detection based on deep learning
Fruit defects determine their quality grading, and also determine the market price and consumers'
desire to buy. It is very important to identify defects at the time of fruit harvest. Defective (rot, damage,
pests, etc.) fruit may spread the disease throughout the sequence during subsequent processing or
transportation . Defect severity is a key parameter affecting yield and quality. , timely control the spread
of bad fruit , which can reduce the waste caused by fruit loss. In recent years, deep learning has been
successfully applied in the fields of automatic recognition and classification of complex images . Many
scholars have hotly discussed the use of deep learning to explore the agricultural field . A particular
application of deep learning is the detection of fruit defects, and we summarize recent papers based on
deep learning for fruit defect detection. Table 2 is a summary of deep learning models applied to defect
detection, including defect types, dataset division, sensitivity, and accuracy.
Table 2
Fruit defect detection based on deep learning
Data set result
fruit Years fruit defect DL model
partitioning ACC Sensitivity
Physical,
damage, 6-layer 80% training
apple 2020 96.5% 100%
rot,scars, CNN 20%validation
pests
rot,fleshy
damage,
6-layer 80% training
apple 2021 bruises, 88% 90%
CNN 20% test
orchard
damage, tan
training：mild
737
moderate774se mild100%
Disease: mild99%
vere625 moderate
mild, moderate97%
tangerine 2022 VGG16 normal 1173 100%
moderate, severe98%
test：mild184 severe96%
severe normal96%
moderate194se normal84%
vere156
normal 293
303
training：
rot 3100
spot 3200
scar 1120 rot 99.25% rot 99%
crack 640 spots 93% scar spot 93%
rot, spots, improved
green plum 2020 ordinary 1840 84.29% scar 84%
scars, cracks VGG16
test：rot 800 crack 78.13% crack 78%
spot 800 normal95.65% normal 96%
scar 280
crack 160
normal 460
calyx end Inception- 70%training
persimmon 2020 90% -
dehiscence ResnetV2 30%test
improved 3102training
apple 2020 rot 97.54% -
UNet 933validation
60%training
blueberry 2020 bruises FCN 20%test 81.2% -
20%validation
In recent years of research on fruit defect detection based on deep learning, some scholars implement
defect detection through self-constructed convolutional neural networks. Among them, Shuxiang Fan
et al. [15] adopted a CNN-based deep learning model for detecting 4-row apples on a defective fruit
sorter. Move at 5 apples per second to get images of normal and defective apples. The CNN-based
model was trained and tested, and the accuracy, sensitivity, and specificity of the test set were 96.5%,
100.0%, and 92.9%, respectively. He Jiang et al. [16] proposed a method to detect infected apples using
deep neural networks. A convolutional neural network extracts important features of apple images and
uses these features for classification. Apple images are divided into two categories: infected and
uninfected, and a neural network with three convolutional layers and two fully connected layers is
designed. The resulting accuracy, sensitivity, and specificity were 88%, 90%, and 98%, respectively.
The above scholars use their self-constructed convolutional neural networks for defect feature
extraction and expression [15-16].
Some scholars use the constructed convolutional neural network model to detect fruit defects. For
example, Poonam Dhiman et al. [17] used the VGG16 model to diagnose four severities (high, medium,
low and healthy) of diseases present in citrus fruits. The pretrained VGG16 is updated by replacing its
bottom layer with an expanded convolutional layer consisting of dense layers with ReLu activations
and sparse categorical cross-entropy as a loss function used to determine the performance of the
classification model. Test accuracies achieved on randomly selected images of healthy, low, high, and
moderate disease levels were 96%, 99%, 98%, and 97%, respectively. The results showed that the
method was effective in detecting four severities of citrus fruit diseases. Haiyan Zhou et al. [18]
improved the VGG network using a stochastic weight averaging (SWA) optimizer and w-softmax loss
function to generate a green plum defect detection network model. The average recognition accuracy
of green plum defects was 93.8%, the test time per image was 84.69 ms, the recognition rate of rotten
defects was 99.25%, and the recognition rate of normal green plums was 95.65%. The results are further
compared with VGG network, Resnet18 network and Ome defect network. The results show that the
recognition accuracy of the green plum defect detection network is increased by 9.8% and 16.6%, and
the test speed is increased by 1.87 and 6.21 ms, respectively, which has certain advantages. Takashi
Akagi et al. [19] proposed to classify persimmon calyx end dehiscence using five convolutional neural
network models with different layer structures, and successfully performed binary classification of
different degrees of disease with an accuracy rate as high as 90%. Among the neural network models,
InceptionResnetV2 achieves the highest accuracy in the classification of calyx tip dehiscence and
control. These results not only provide new insights into diseases within fruits, but also demonstrate the
potential applicability of deep neural networks in plant biology. Kyamelia Roy et al. [20] realized the
detection of rotten apples and fresh apples based on the defects existing in the peel. A deep learning-
based semantic segmentation of rotten parts in apple RGB images is presented. Rotten apples are
identified using segmentation techniques, where the rotten parts of the apple are segmented. The
304
proposed En-Unet model produces stronger output than Unet with 97.54% training-validation accuracy
and 95.36% accuracy for Unet as an infrastructure. The best average IoU score for En-Unet at the 0.95
threshold is 0.866 compared to 0.66 for Unet. The experimental results show that the model can be well
used for real-time segmentation, detection and classification of rotten apples and fresh apples. The
above scholars obtained the most suitable defect detection model by improving the existing
convolutional neural network model.
Some scholars use other network models of deep learning to solve defect detection, Mengyun Zhang
et al. [21] used a deep learning-based fully convolutional network (FCN) method to detect internal
bruises in blueberries. Three classes (bruised tissue, unbruised tissue, and calyx of blueberry) input
HSTI were used to evaluate the FCN model using pretrained weights (transfer learning) and random
initialization. The results showed that when the deep learning method was used, the bruise and calyx
end of the blueberry could be separated from the blueberry fruit 30 minutes after the blueberry was
mechanically damaged. The new full-wavelength model with random initialization has the highest
accuracy of 81.2% to study the resistance of blueberry fruit to mechanical damage.
4. Summary and Outlook
Fruit classification and detection is an important direction in the field of intelligent agriculture and
has great practical significance. In recent years, it has also attracted the attention of the majority of
researchers. In view of the deficiencies in the existing work, this paper makes the following conclusions:
1. The classification of fruit varieties is an important direction in the field of intelligent agriculture
and has great practical significance. In recent years, it has also attracted the attention of the majority of
researchers. In view of the deficiencies in the existing work, this paper makes the following conclusions:
2.The given image is under controlled conditions. The best image quality is obtained when the sun
is full, and taking pictures on a cloudy day will increase the complexity of image preprocessing and
reduce the recognition effect. Creating images under different environmental conditions, such as night
and morning, rain and drought, allows for reliable predictions in cultivated environments. It is necessary
to combine meteorological and temperature, humidity and other data to achieve a more realistic forecast.
3. The amount of data required for deep learning training models is very large, while the data sample
size of researchers is small, and the number of varieties is far from enough.
4. Most of the researches are still in the experimental stage, and have not been industrialized and
commercialized. There are few studies on online detection, and there is still a lot of room for
improvement.
To sum up, the defect detection and fruit variety classification methods based on deep learning have
many advantages over traditional methods. First of all, the classification and detection accuracy is high,
such as the accuracy of grape [7] is as high as 99%. Secondly, deep learning has strong anti-interference
ability, such as in tomato[2], image features are efficiently extracted.Finally, the recognition time is
very short, as in the detection of apple defects on the sorter [15], the processing time per apple is less
than 72 ms.
In view of the deficiencies in the existing technology, this paper points out its future development
trend, which is summarized as follows:
1. Develop and export mobile applications to mobile phones, making it a stand-alone tool suitable
for working on smartphones. Both farmers and consumers can use it to identify and classify fruit.
2. For the situation that there are few data samples in the data set and the samples are not balanced,
a large number of samples can be generated by using the Generative adversarial network (GAN).
3. In fruit classification and detection research, convolutional neural network CNN is mostly used,
or improvements are made on the basis of classic network models. Cross-integration with other
powerful deep learning models in future research will provide countless possibilities and application
potentials for future development.
It is expected that deep learning can achieve better results in the agricultural field in the future, and
put the research results into practice for industrialization and commercial use.
305
5. References
[1] Rong D, Wang H, Ying Y, et al. Peach variety detection using VIS-NIR spectroscopy and deep
learning[J]. Computers and Electronics in Agriculture, 2020, 175: 105553.
[2] Alajrami M A, Abu-Naser S S. Type of tomato classification using deep learning[J]. International
Journal of Academic Pedagogical Research (IJAPR), 2020, 3(12).
[3] Taner A, Öztekin Y B, Duran H. Performance analysis of deep learning CNN models for variety
classification in hazelnut[J]. Sustainability, 2021, 13(12): 6527.
[4] Alresheedi K M, Aladhadh S, Khan R U, et al. Dates Fruit Recognition: From Classical Fusion to
Deep Learning[J]. COMPUTER SYSTEMS SCIENCE AND ENGINEERING, 2022, 40(1): 151-
166.
[5] Rodríguez F J, García A, Pardo P J, et al. Study and classification of plum varieties using image
analysis and deep learning techniques[J]. Progress in Artificial Intelligence, 2018, 7(2): 119-127.
[6] Osako Y, Yamane H, Lin S Y, et al. Cultivar discrimination of litchi fruit images using deep
learning[J]. Scientia Horticulturae, 2020, 269: 109360.
[7] Nasiri A, Taheri-Garavand A, Fanourakis D, et al. Automated grapevine cultivar identification via
leaf imaging and deep convolutional neural networks: a proof-of-concept study employing primary
iranian varieties[J]. Plants, 2021, 10(8): 1628.
[8] Franczyk B, Hernes M, Kozierkiewicz A, et al. Deep learning for grape variety recognition[J].
Procedia Computer Science, 2020, 176: 1211-1220.
[9] Ponce J M, Aquino A, Andújar J M. Olive-fruit variety classification by means of image processing
and convolutional neural networks[J]. IEEE Access, 2019, 7: 147629-147641.
[10] Wang Q, Zhao L, Niu Z. Portable Kiwi Variety Classification Equipment Based on Transfer
Learning[C]//Journal of Physics: Conference Series. IOP Publishing, 2021, 1865(4): 042069.
[11] Zhu X, Zhang X, Sun Z, et al. Identification of Oil Tea (Camellia oleifera C. Abel) Cultivars Using
EfficientNet-B4 CNN Model with Attention Mechanism[J]. Forests, 2021, 13(1): 1.
[12] Borianne P, Borne F, Sarron J, et al. Deep Mangoes: from fruit detection to cultivar identification
in colour images of mango trees[J]. arXiv preprint arXiv:1909.10939, 2019.
[13] Fitri Z E, Nugroho W B, Madjid A, et al. Comparison of Neural Network Methods for
Classification of Banana Varieties (Musa paradiasaca)[J]. Jurnal Rekayasa Elektrika, 2021, 17(2).
[14] Keles O, Taner A. Classification of hazelnut varieties by using artificial neural network and
discriminant analysis[J]. Spanish Journal of Agricultural Research, 2021, 19(4): e0211-e0211.
[15] Fan S , Li J , Zhang Y , et al. On line detection of defective apples using computer vision system
combined with deep learning methods[J]. Journal of Food Engineering, 2020, 286:110102.
[16] Jiang H, Li X, Safara F. IoT-based agriculture: Deep learning in detecting apple fruit diseases[J].
Microprocessors and Microsystems, 2021: 104321.
[17] Dhiman P, Kukreja V, Manoharan P, et al. A Novel Deep Learning Model for Detection of Severity
Level of the Disease in Citrus Fruits[J]. Electronics, 2022, 11(3): 495.
[18] Zhou H , Zhuang Z , Liu Y , et al. Defect Classification of Green Plums Based on Deep
Learning[J]. Sensors (Basel, Switzerland), 20(23):6993.
[19] Akagi T, Onishi M, Masuda K, et al. Explainable Deep Learning Reproduces a ‘Professional
Eye’on the Diagnosis of Internal Disorders in Persimmon Fruit[J]. Plant and Cell Physiology, 2020,
61(11): 1967-1973.
[20] Roy K , Chaudhuri S S , Pramanik S . Deep learning based real-time Industrial framework for
rotten and fresh fruit detection using semantic segmentation[J]. Microsystem Technologies,
2020:1-11.
[21] Zhang M , Jiang Y , Li C , et al. Fully convolutional networks for blueberry bruising and calyx
segmentation using hyperspectral transmittance imaging[J]. Biosystems Engineering, 2020,
192:159.
306