Classification of Fruit Varieties Based on Deep Learning Peipei Cao, Yuekun Pei, Zhi Chai, Jingyu Liu and Zuming Wang Dalian University, No. 10, Xuefu Street, Jinzhou District, Dalian City, Liaoning Province, Dalian, Index, China Abstract With the development of economy and the improvement of consumption level, consumers' demand for high-quality fruits is gradually increasing. During the ripening period of fruit, fruit defects determine its quality grading, and different varieties of fruit are liked by consumers differently, and these factors determine their market sales prices. At present, the management, quality grading and variety classification of fruit in my country mainly rely on manual labor, which is greatly influenced by subjective factors. Therefore, a professional and effective method is needed. Deep learning has the advantages of high recognition accuracy, short recognition time, and strong anti-interference ability, and has received extensive attention from scholars at home and abroad. This paper summarizes the research results published in recent years on the application of deep learning in fruit defect detection and variety classification. Keywords 1 Fruit grading; fruit detection; variety classification; deep learning 1. Introduction Fruits are good for human health. Fruit is rich in vitamin A, vitamin B, vitamin C, various inorganic salts, calcium, phosphorus, iron, iodine, etc., and contains a variety of amino acids, which are all necessary nutrients for the human body. Fruits are indispensable in daily life. With the development of economy and the improvement of consumption level, consumers' demand for high-quality fruits is gradually increasing. During fruit ripening, fruit quality is mainly determined by the presence or absence of fruit defects and size grading. At present, fruit grading and sorting mainly rely on manual labor, which is low in efficiency, and is greatly affected by subjective factors and has large errors, which is difficult to meet the needs of large-scale production. In addition, due to the advancement of agricultural technology, multiple varieties of each fruit are bred with different characteristics to meet the needs of consumers. Therefore, the accurate identification of fruit varieties and the selection of the favorite varieties of consumers also have certain application value. With the development of deep learning, it has been applied in commercial retail, smart agriculture and other fields. In terms of commercial retail, more and more fresh food supermarkets have begun to try to use smart fresh food scales to automatically identify the type of fruit and calculate the price to realize one-stop payment. In smart agriculture, fruit can be monitored in real time and accurately, fruit quality can be detected, fruit classification and sorting, maturity identification and defect detection, so as to improve the yield and quality of fruit. The application of deep learning provides solutions for expensive manual labor costs, increasing food demand, improving fruit quality, and more. This paper aims to carry out research on the application of deep learning in fruit sales, to explore the detection of fruit defects and classification of varieties, and to provide some references for researchers in this field. ICBASE2022@3rd International Conference on Big Data & Artificial Intelligence & Software Engineering, October 21- 23, 2022, Guangzhou, China EMAIL: 2099689793@qq.com(peipei Cao) ORCID: 0000-0002-0311-0120(peipei Cao); © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) 300 2. Classification of fruit varieties based on deep learning In the natural environment, there are many kinds of fruits, and there are more and more high-quality varieties. Consumers have more and more choices of high-quality varieties, and different varieties also determine their market prices. The traditional identification of fruit varieties is based on the shape, color, surface texture, size and other characteristics of the fruit. Some fruit varieties are relatively similar in maturity, size, color and taste, and the accuracy of identification by non-professionals is not high. The solution is to automate the identification process to minimize human error from subjectivity. We summarize recent papers on deep learning-based fruit variety classification. Table 1 summarizes the deep learning models applied to fruit variety classification, including variety categories, data set division, and accuracy. Table 1 Classification of fruit varieties based on deep learning fruit Variety years DL model Data set partitioning Accuracy 75%training 25%test Peach 5 2020 3-layer CNN Take 20% of the training 94.4% set as the validation set tomato 7 2020 4-layer CNN 70%training 30%validation 93% Lprtnrl 82% training 6% test hazelnut 17 2021 98.63% （4-layer CNN） 12% validation date 9 2022 5-layer CNN - 94.8% Plum 3 2020 Alexnet 80% training 20% test 91%-97% 320 training 60 test litchi 4 2020 VGG16 98.35% 60 validation Grape 6 2021 Modified VGG16 80% training 20% test 99% grape 5 2018 ExtResnet 3667 training 300 test 99% bunch Inception- 62.5% training olives 7 2019 95.91% ResNetV2 37.5% validation 80% training Kiwi 4 2021 Densenet121 97.79% 20% validation EfficientNet-B4- 60% training 20% test Oil Tea 4 2021 97.01% CBAM 20% validation mango 3 2019 Faster RCNN - 80% banana 5 2021 RBF 420 training 50 test 98.81% x-axis 89.1% x-axis training 426 test55 y-axis92.7% hazelnut 11 2021 ANN y-axis training 421 test55 z-axis86.8% z-axis training 419 test53 In the recently published research on fruit varieties, some scholars use their own convolutional neural networks to achieve variety classification. Among them, Dian Rong et al. [1] realized the identification of peach varieties by constructing a one-dimensional convolutional neural network and five kinds of peach VIS-NIR spectral databases. The accuracy is 100% in the validation dataset and 94.4% in the test dataset. This study shows that peach cultivars can be successfully distinguished using VIS-NIR spectroscopy and deep learning. Mahmoud A. Alajrami et al. [2] studied seven tomato classification methods. Taking the original image as input, a Convolutional Neural Network (CNN) is used to extract features. The network consists of 4 convolutional layers with Relu activation functions, 301 and each convolutional layer is followed by a Max Pooling layer, which accepts 7 Different varieties of tomato images are used as input. The test accuracy was 93%. [3] designed a CNN model named Lprtnr1 to classify 17 hazelnuts. The Lprtnr1 model consists of an input layer, four convolutional layers, a flattening layer, a fully connected layer, and an output layer. The proposed model is evaluated by comparison with pretrained models (VGG16, VGG19, Resnet50 and InceptionV3). The proposed model yielded 98.63% accuracy on the test set, and the classification accuracies of the VGG16, VGG19, InceptionV3 and Resnet50 models were 73.14%, 72.14%, 61.18% and 80.00%, respectively. The proposed model is found to perform better than the pretrained model in terms of performance evaluation criteria. Khaled Marji Alresheedi et al. [4] classified nine jujube species based on classical machine learning and deep learning, including Bayesian networks, support vector machines, random forests, and multilayer perceptrons (MLPs). Convolutional Neural Networks (CNN) for deep learning. The feature set includes color layout functions, blurred color and texture histograms, Gabor filtering, and pyramid histograms of oriented gradients. The fusion of various features has also been extensively explored. The fused feature set includes Color Layout+Gabor (SVM has the highest accuracy of 88%), PHOG+Gabor (Random Forest has the highest accuracy of 89.6%), and PHOG+FCTH (MLP has the highest of 93.8%). The performance of deep learning is also explored using innovative CNN models. The detection results of deep learning are 1% higher than MLP (PHOG+FCTH classic feature set) and 2% higher than random forest detection accuracy. To sum up, the general feature of these methods is that they act directly on the original image, learn layer-by-layer features, and then use multi-layer networks to obtain feature information [1-3]. Or combine the self-built CNN model with machine learning to find the optimal recognition model [4]. Some scholars use the constructed convolutional neural network model to achieve variety classification. For example, Francisco J. Rodríguez et al. [5] chose the Alexnet network model to identify plum varieties, and the results showed that the accuracy rate was 91% to 97%. Yutaro Osako et al. [6] implemented litchi variety classification through the VGG16 model. Got 98.33% accuracy. Grad-CAM visualization shows that the model uses different breed-dependent regions for breed identification. The study shows that deep learning can be used to distinguish lychee varieties from images. Also Amin Nasiri et al. [7] adopted a modified VGG16 model with modifications in the global average pooling layer, Dense layer, batch normalization layer, and Dropout layer, replacing the last three Dense layers with a classifier to Modify the original VGG16. The average classification accuracy is over 99%. This model provides a rapid, low-cost, high-throughput method for grape variety identification. Bogdan Franczyk et al. [8] proposed a method to identify clusters of different varieties of grapes using KSM, Resnet and ExtResnet to classify five types of grapes from a given image dataset. KSM classification achieves 47% accuracy and Resnet classification achieves 89%. ExtResnet is a combination of a deep learning Resnet model and a multi-layer perceptron that achieves 99% accuracy for correctly classified grape clusters. Juan M. Ponce et al. [9] proposed a deep learning-based method for olive variety classification. Verify the implementation of six different convolutional neural network frameworks Alexnet, InceptionV1, InceptionV3, Resnet-50, Resnet-101, Inception-ResnetV2, and the obtained accuracy rates are: 89.90%, 91.81%, 94.86%, 95.33%, 94.00% , 95.91%. The highest accuracy was obtained when using Inception-ResnetV2. Inception-ResnetV2 is based on the Inception architecture but inspired by Resnet and uses residual connections for training to speed up processing. Qilong Wang et al. [10] studied the classification of kiwifruit varieties based on deep learning, and studied the deployment and application of the kiwifruit classification model on the Jetson Nano artificial intelligence development board. Use transfer learning based on three pretrained models Xception, Resnet50, Densenet121. Comparing and analyzing the model size, training speed, convergence and recognition accuracy, it was concluded that the transfer learning pre-training model based on Densenet121 had the best classification effect on kiwifruit varieties, with fast convergence speed and the smallest model, with a recognition accuracy of 97.79%. Xueyan Zhu et al. [11] classified nectarine varieties, selected EfficientNet-B4 as the basic model for nectarine variety identification, and integrated the convolutional block Attention module (CBAM) into EfficientNet-B4 to construct EfficientNet-B4- CBAM, thereby Improve the focus ability and information expression ability of fruit area. Comparisons are made with InceptionV3, VGG16, Resnet50, EfficientNet-B4 and EfficientNet-B4-SE. The experimental results show that the accuracy of the EfficientNet-B4-CBAM model reaches 97.02%, and the kappa coefficient reaches 0.96, which is higher than other methods in the comparative experiments. Philippe Borianne et al. [12] used a Faster R-CNN network to identify mango varieties on a tree, and 302 the Faster R-CNN network had some limitations in detecting mango fruits and identifying the respective varieties of mangoes simultaneously. The varietal identification rate of the detected mango fruits was around 80%, although some errors were proven in fruit detection. Finally, we classify the methods used by the above researchers. Some scholars use the trained models for direct training [5-6]. Others include replacing a layer of the trained model with a classifier [7], combining the trained model with a perceptron [8], continuing to transfer the trained model [10], and adding a convolution block [12] to Get the best model, and find the most suitable pre-trained model after comparison. Some scholars use other network models of deep learning to solve variety classification. Zilvanhisna Emka Fitri et al. [13] classified banana varieties based on artificial neural network. The banana variety classification system adopts the combination of image processing technology and artificial neural network. The intelligent systems used are backpropagation and radial basis function neural networks. Based on the confusion matrix calculation, the RBF accuracy is 98.81% from the test data. Omer Keles et al. [14] classified hazelnut varieties by using artificial neural network ANN and discriminant analysis DA. The physical, mechanical and optical properties of three axes of 11 hazelnut varieties were determined. Artificial Neural Network (ANN) classification success rate: 89.1% on the X-axis, 92.7% on the Y-axis, and 86.8% on the Z-axis. 3. Fruit defect detection based on deep learning Fruit defects determine their quality grading, and also determine the market price and consumers' desire to buy. It is very important to identify defects at the time of fruit harvest. Defective (rot, damage, pests, etc.) fruit may spread the disease throughout the sequence during subsequent processing or transportation . Defect severity is a key parameter affecting yield and quality. , timely control the spread of bad fruit , which can reduce the waste caused by fruit loss. In recent years, deep learning has been successfully applied in the fields of automatic recognition and classification of complex images . Many scholars have hotly discussed the use of deep learning to explore the agricultural field . A particular application of deep learning is the detection of fruit defects, and we summarize recent papers based on deep learning for fruit defect detection. Table 2 is a summary of deep learning models applied to defect detection, including defect types, dataset division, sensitivity, and accuracy. Table 2 Fruit defect detection based on deep learning Data set result fruit Years fruit defect DL model partitioning ACC Sensitivity Physical, damage, 6-layer 80% training apple 2020 96.5% 100% rot,scars, CNN 20%validation pests rot,fleshy damage, 6-layer 80% training apple 2021 bruises, 88% 90% CNN 20% test orchard damage, tan training：mild 737 moderate774se mild100% Disease: mild99% vere625 moderate mild, moderate97% tangerine 2022 VGG16 normal 1173 100% moderate, severe98% test：mild184 severe96% severe normal96% moderate194se normal84% vere156 normal 293 303 training： rot 3100 spot 3200 scar 1120 rot 99.25% rot 99% crack 640 spots 93% scar spot 93% rot, spots, improved green plum 2020 ordinary 1840 84.29% scar 84% scars, cracks VGG16 test：rot 800 crack 78.13% crack 78% spot 800 normal95.65% normal 96% scar 280 crack 160 normal 460 calyx end Inception- 70%training persimmon 2020 90% - dehiscence ResnetV2 30%test improved 3102training apple 2020 rot 97.54% - UNet 933validation 60%training blueberry 2020 bruises FCN 20%test 81.2% - 20%validation In recent years of research on fruit defect detection based on deep learning, some scholars implement defect detection through self-constructed convolutional neural networks. Among them, Shuxiang Fan et al. [15] adopted a CNN-based deep learning model for detecting 4-row apples on a defective fruit sorter. Move at 5 apples per second to get images of normal and defective apples. The CNN-based model was trained and tested, and the accuracy, sensitivity, and specificity of the test set were 96.5%, 100.0%, and 92.9%, respectively. He Jiang et al. [16] proposed a method to detect infected apples using deep neural networks. A convolutional neural network extracts important features of apple images and uses these features for classification. Apple images are divided into two categories: infected and uninfected, and a neural network with three convolutional layers and two fully connected layers is designed. The resulting accuracy, sensitivity, and specificity were 88%, 90%, and 98%, respectively. The above scholars use their self-constructed convolutional neural networks for defect feature extraction and expression [15-16]. Some scholars use the constructed convolutional neural network model to detect fruit defects. For example, Poonam Dhiman et al. [17] used the VGG16 model to diagnose four severities (high, medium, low and healthy) of diseases present in citrus fruits. The pretrained VGG16 is updated by replacing its bottom layer with an expanded convolutional layer consisting of dense layers with ReLu activations and sparse categorical cross-entropy as a loss function used to determine the performance of the classification model. Test accuracies achieved on randomly selected images of healthy, low, high, and moderate disease levels were 96%, 99%, 98%, and 97%, respectively. The results showed that the method was effective in detecting four severities of citrus fruit diseases. Haiyan Zhou et al. [18] improved the VGG network using a stochastic weight averaging (SWA) optimizer and w-softmax loss function to generate a green plum defect detection network model. The average recognition accuracy of green plum defects was 93.8%, the test time per image was 84.69 ms, the recognition rate of rotten defects was 99.25%, and the recognition rate of normal green plums was 95.65%. The results are further compared with VGG network, Resnet18 network and Ome defect network. The results show that the recognition accuracy of the green plum defect detection network is increased by 9.8% and 16.6%, and the test speed is increased by 1.87 and 6.21 ms, respectively, which has certain advantages. Takashi Akagi et al. [19] proposed to classify persimmon calyx end dehiscence using five convolutional neural network models with different layer structures, and successfully performed binary classification of different degrees of disease with an accuracy rate as high as 90%. Among the neural network models, InceptionResnetV2 achieves the highest accuracy in the classification of calyx tip dehiscence and control. These results not only provide new insights into diseases within fruits, but also demonstrate the potential applicability of deep neural networks in plant biology. Kyamelia Roy et al. [20] realized the detection of rotten apples and fresh apples based on the defects existing in the peel. A deep learning- based semantic segmentation of rotten parts in apple RGB images is presented. Rotten apples are identified using segmentation techniques, where the rotten parts of the apple are segmented. The 304 proposed En-Unet model produces stronger output than Unet with 97.54% training-validation accuracy and 95.36% accuracy for Unet as an infrastructure. The best average IoU score for En-Unet at the 0.95 threshold is 0.866 compared to 0.66 for Unet. The experimental results show that the model can be well used for real-time segmentation, detection and classification of rotten apples and fresh apples. The above scholars obtained the most suitable defect detection model by improving the existing convolutional neural network model. Some scholars use other network models of deep learning to solve defect detection, Mengyun Zhang et al. [21] used a deep learning-based fully convolutional network (FCN) method to detect internal bruises in blueberries. Three classes (bruised tissue, unbruised tissue, and calyx of blueberry) input HSTI were used to evaluate the FCN model using pretrained weights (transfer learning) and random initialization. The results showed that when the deep learning method was used, the bruise and calyx end of the blueberry could be separated from the blueberry fruit 30 minutes after the blueberry was mechanically damaged. The new full-wavelength model with random initialization has the highest accuracy of 81.2% to study the resistance of blueberry fruit to mechanical damage. 4. Summary and Outlook Fruit classification and detection is an important direction in the field of intelligent agriculture and has great practical significance. In recent years, it has also attracted the attention of the majority of researchers. In view of the deficiencies in the existing work, this paper makes the following conclusions: 1. The classification of fruit varieties is an important direction in the field of intelligent agriculture and has great practical significance. In recent years, it has also attracted the attention of the majority of researchers. In view of the deficiencies in the existing work, this paper makes the following conclusions: 2.The given image is under controlled conditions. The best image quality is obtained when the sun is full, and taking pictures on a cloudy day will increase the complexity of image preprocessing and reduce the recognition effect. Creating images under different environmental conditions, such as night and morning, rain and drought, allows for reliable predictions in cultivated environments. It is necessary to combine meteorological and temperature, humidity and other data to achieve a more realistic forecast. 3. The amount of data required for deep learning training models is very large, while the data sample size of researchers is small, and the number of varieties is far from enough. 4. Most of the researches are still in the experimental stage, and have not been industrialized and commercialized. There are few studies on online detection, and there is still a lot of room for improvement. To sum up, the defect detection and fruit variety classification methods based on deep learning have many advantages over traditional methods. First of all, the classification and detection accuracy is high, such as the accuracy of grape [7] is as high as 99%. Secondly, deep learning has strong anti-interference ability, such as in tomato[2], image features are efficiently extracted.Finally, the recognition time is very short, as in the detection of apple defects on the sorter [15], the processing time per apple is less than 72 ms. In view of the deficiencies in the existing technology, this paper points out its future development trend, which is summarized as follows: 1. Develop and export mobile applications to mobile phones, making it a stand-alone tool suitable for working on smartphones. Both farmers and consumers can use it to identify and classify fruit. 2. For the situation that there are few data samples in the data set and the samples are not balanced, a large number of samples can be generated by using the Generative adversarial network (GAN). 3. In fruit classification and detection research, convolutional neural network CNN is mostly used, or improvements are made on the basis of classic network models. Cross-integration with other powerful deep learning models in future research will provide countless possibilities and application potentials for future development. It is expected that deep learning can achieve better results in the agricultural field in the future, and put the research results into practice for industrialization and commercial use. 305 5. References [1] Rong D, Wang H, Ying Y, et al. Peach variety detection using VIS-NIR spectroscopy and deep learning[J]. Computers and Electronics in Agriculture, 2020, 175: 105553. [2] Alajrami M A, Abu-Naser S S. Type of tomato classification using deep learning[J]. International Journal of Academic Pedagogical Research (IJAPR), 2020, 3(12). [3] Taner A, Öztekin Y B, Duran H. Performance analysis of deep learning CNN models for variety classification in hazelnut[J]. Sustainability, 2021, 13(12): 6527. [4] Alresheedi K M, Aladhadh S, Khan R U, et al. Dates Fruit Recognition: From Classical Fusion to Deep Learning[J]. COMPUTER SYSTEMS SCIENCE AND ENGINEERING, 2022, 40(1): 151- 166. [5] Rodríguez F J, García A, Pardo P J, et al. Study and classification of plum varieties using image analysis and deep learning techniques[J]. Progress in Artificial Intelligence, 2018, 7(2): 119-127. [6] Osako Y, Yamane H, Lin S Y, et al. Cultivar discrimination of litchi fruit images using deep learning[J]. Scientia Horticulturae, 2020, 269: 109360. [7] Nasiri A, Taheri-Garavand A, Fanourakis D, et al. Automated grapevine cultivar identification via leaf imaging and deep convolutional neural networks: a proof-of-concept study employing primary iranian varieties[J]. Plants, 2021, 10(8): 1628. [8] Franczyk B, Hernes M, Kozierkiewicz A, et al. Deep learning for grape variety recognition[J]. Procedia Computer Science, 2020, 176: 1211-1220. [9] Ponce J M, Aquino A, Andújar J M. Olive-fruit variety classification by means of image processing and convolutional neural networks[J]. IEEE Access, 2019, 7: 147629-147641. [10] Wang Q, Zhao L, Niu Z. Portable Kiwi Variety Classification Equipment Based on Transfer Learning[C]//Journal of Physics: Conference Series. IOP Publishing, 2021, 1865(4): 042069. [11] Zhu X, Zhang X, Sun Z, et al. Identification of Oil Tea (Camellia oleifera C. Abel) Cultivars Using EfficientNet-B4 CNN Model with Attention Mechanism[J]. Forests, 2021, 13(1): 1. [12] Borianne P, Borne F, Sarron J, et al. Deep Mangoes: from fruit detection to cultivar identification in colour images of mango trees[J]. arXiv preprint arXiv:1909.10939, 2019. [13] Fitri Z E, Nugroho W B, Madjid A, et al. Comparison of Neural Network Methods for Classification of Banana Varieties (Musa paradiasaca)[J]. Jurnal Rekayasa Elektrika, 2021, 17(2). [14] Keles O, Taner A. Classification of hazelnut varieties by using artificial neural network and discriminant analysis[J]. Spanish Journal of Agricultural Research, 2021, 19(4): e0211-e0211. [15] Fan S , Li J , Zhang Y , et al. On line detection of defective apples using computer vision system combined with deep learning methods[J]. Journal of Food Engineering, 2020, 286:110102. [16] Jiang H, Li X, Safara F. IoT-based agriculture: Deep learning in detecting apple fruit diseases[J]. Microprocessors and Microsystems, 2021: 104321. [17] Dhiman P, Kukreja V, Manoharan P, et al. A Novel Deep Learning Model for Detection of Severity Level of the Disease in Citrus Fruits[J]. Electronics, 2022, 11(3): 495. [18] Zhou H , Zhuang Z , Liu Y , et al. Defect Classification of Green Plums Based on Deep Learning[J]. Sensors (Basel, Switzerland), 20(23):6993. [19] Akagi T, Onishi M, Masuda K, et al. Explainable Deep Learning Reproduces a ‘Professional Eye’on the Diagnosis of Internal Disorders in Persimmon Fruit[J]. Plant and Cell Physiology, 2020, 61(11): 1967-1973. [20] Roy K , Chaudhuri S S , Pramanik S . Deep learning based real-time Industrial framework for rotten and fresh fruit detection using semantic segmentation[J]. Microsystem Technologies, 2020:1-11. [21] Zhang M , Jiang Y , Li C , et al. Fully convolutional networks for blueberry bruising and calyx segmentation using hyperspectral transmittance imaging[J]. Biosystems Engineering, 2020, 192:159. 306