Contactless Classification of Strawberry Using Hyperspectral Imaging Binu Melit Devassy1 and Sony George1 1 Department of Computer Science, Norwegian University of Science and Technology, Gjøvik 2802, Norway; binu.m.devassy@ntnu.no, sony.george@ntnu.no Abstract. Rapid non-contact estimation of fruit quality parameters is an essential factor for an efficient food processing pipeline. We propose a novel workflow for the contactless classification of strawberries based on their sugar content, us- ing Hyperspectral Imaging (HSI) and One-Dimensional Convolutional Neural Network (1D - CNN). Sugar content is an important quality aspect of strawber- ries, hence classification based on sugar content gives more yield to the fruit pro- ducers. We used Visible and Near Infrared (VNIR) hyperspectral camera to ac- quire HSI data of 50 ripe strawberries and applied the proposed method to clas- sify them. To verify the advantage of the proposed method, the results from 1D- CNN are compared against other standard classification methods such as Spectral Angle Mapper (SAM), and Spectral Information Divergence (SID). The results show that the 1D-CNN outperformed other methods by achieving 96.6% classi- fication accuracy. Keywords: Hyperspectral Imaging, Strawberry classification, Fruits classifica- tion using CNN. 1 Introduction Non-invasive measurement of different food attributes is of great interest. The sugar content is one of the characteristics that enhances the customer experience and influ- ence the market value of the fruit [1]. Sugar content is one of the key factors that is useful for grading the fruits [2], measurement of this attribute with the application of imaging technologies eases the sorting process and offer several advantages like con- tactless and non-destructive measurement, the possibility for automation, and high ac- curacy within a certain limit. Hyperspectral imaging (HSI) is one of the most suitable imaging modalities that is proven to be useful for this purpose [3]. There has been many extensive studies reported on the usefulness of HSI for non-invasive measurement of several types of fruits such as apples, oranges, and kiwifruits [4][5][6]. Hyperspectral imaging systems capture both spatial and spectral information simulta- neously, which enables HSI technology to make a more reliable classification compared to the traditional three channel imaging methods. In addition to this, HSI already proven Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Colour and Visual Computing Symposium 2020, Gjøvik, Norway, September 16-17, 2020. 2 B. M. Devassy and S. George to be one of the effective non-invasive methods to study material properties compared to the chemical analysis, which are generally invasive in nature. HSI finds applications in many different fields like remote sensing [7], cultural heritage [8][9], forensics [10], etc. Food science is an important field where hyperspectral imaging used extensively to monitor the quality, diseases, and stages of development in variety of food products. Strawberry is one of the favorite fruits for many people and the interest is reflected in the increased production in every year [11]. Motivated by this fact that lack of HSI and predictive sugar analysis for strawberries, we have studied and evaluated the straw- berry’s classification based on sugar content using HSI. Customer interest in strawberry is mainly driven by the taste of the fruit and most of the times the sweetness of the fruit. Convolutional Neural Networks (CNNs) has been used in many computer vision and image classification applications, CNNs were designed to learn features from the train- ing data and which can be used to classify the test samples. The most common CNNs are two-dimensional (2D-CNN), however in this work we proposed a one-dimensional CNN (1D-CNN) because the spectral signals are One-Dimensional (1D) signals with varying amplitude. 1D-CNNs can provide reliable solutions in many 1D signal-pro- cessing applications such as Electrocardiographic (ECG) signals [12], audio[13], and other 1D signals [14][15]. We can also find a few attempts to use 1D-CNN for spectral classification [16], this experiment used 1D-CNN to classify ink spectra. To verify the effectiveness of the proposed method, we used two well-known classification methods in HSI domain; they are Spectral Angle Mapper (SAM) and Spectral Information Di- vergence (SID). Later part of this paper is organized as follows. Section 2 will present the details of the fruit samples used, hyperspectral acquisition of the fruits and processing of the HSI data. Followed by Section 3, which presents and discusses the results from this study and finally the conclusions in Section 4. 2 Materials and Methods 2.1 Fruit Samples Strawberries were purchased from local markets in Norway and fifty of them were se- lected as candidates by avoiding the fruits having any defects, bruises, and infections. All the strawberries were belong to the same class and produced in the same environ- ment. The fruits were kept in ideal storage conditions, and taken to the lab environment an hour before the hyperspectral image acquisition, in a controlled room temperature. The strawberries were washed to remove any contaminations and the water drops from the fruits were wiped away using a dry clean cloth, before measurement. Contactless Classification of Strawberry Using Hyperspectral Imaging 3 2.2 Hyperspectral Acquisition Hyperspectral acquisition system used for this experiment is shown in Fig.1, which is the same setup used in this experiment [17] except for the samples used. HySpex VNIR- 1800 [18] push broom hyperspectral camera was used for hyperspectral image acquisi- tion of the fruit samples. The camera placed at right angles to a moving translator stage where the fruit samples were placed and two halogen light sources were used to illumi- nate the scene with 45°:0° geometry with respect to the camera to minimize shadowing. This camera has a spectral sampling of 3.18 nm along with its spectral range, which divides the supporting spectral range (400 nm to 1000 nm) into 186 bands. The image acquisition resulted in a hyperspectral data cube with spatial (X and Y) and spectral (Z) directions. Here, the size of X- axis was 1800 pixels, the size of the Y-axis depends on the size and number of strawberries used in a single scan, and the size Z-axis was 186. A reference target with known reflectance (Contrast Multi-Step Target [19]) values was present in the scene, which will be used to convert the radiance to the reflectance while processing the data. Fig. 1 Hyperspectral image acquisition setup 2.3 Sugar Content Measurement The sugar content of each strawberry was measured immediately after spectral meas- urement using a refractometer (PAL 1, Atago Co., Ltd., Japan). This method of sugar content estimation requires the fruits to be squeezed to get the juice, from which the refractometer analyses the degrees of Brix (o Bx). The degree Brix represents “the per- centage of water-soluble solids in fruit juice and can be affected by many factors in- cluding variety, growth region, growth year, and maturity level of the fruit” [20]. In this case, the degree of Brix represents the sugar content in the strawberry juice; one degree of brix can be defined as the one gram of sucrose in 100 grams of fruit juice. Fig.2 represents the sugar level distribution measured from the samples used. 4 B. M. Devassy and S. George Fig. 2 Histogram of sugar values measured from the strawberries used in the present study 2.4 Proposed method The figure (Fig.3) shows the proposed 1D-CNNs architecture, the input spectra will pass through a series of hidden layers, each hidden layer consists of 1D-Convolution layer with ReLU (Rectified Linear Unit) activation, the ‘n’ will be finalized after parameter tuning. A drop out layer with a rate ‘0.5’ will follow the convolution lay- ers and then by a max-pooling layer with a pool size of ‘2’. Then a flatten layer will flatten the max-pooled output, followed by two dense layers, the last dense layer use ‘softmax’ activation to generate the classification result. The network used the cate- gorical cross-entropy as loss function, which is a proven technique for learning a multi-class classification problem and used Adam [21] optimization. Contactless Classification of Strawberry Using Hyperspectral Imaging 5 Fig. 3 Proposed 1D-CNN architecture 2.5 Spectral Angle Mapper (SAM) The Spectral Angle Mapper (SAM) is one of the important spectral similarity criteria used to estimates the spectral match between the reference and target spectra by meas- uring the angular difference in radian between the reference and test spectrum [22] as in Equation 1. In this process, both spectra were considered as vectors having dimen- sionality equal to the number of bands (nb) in the spectrum. The angle alpha (α) defines the similarity between spectra, where t and r be the test and reference spectrum respec- tively. ∑𝑛𝑛𝑛𝑛 𝑖𝑖=1 𝑡𝑡𝑖𝑖 𝑟𝑟𝑖𝑖 𝛼𝛼 = cos −1 � 1 1 � (1) �∑𝑛𝑛𝑛𝑛 2 �2�∑𝑛𝑛𝑛𝑛 𝑟𝑟 2 � �2 𝑖𝑖=1 𝑡𝑡𝑖𝑖 � 𝑖𝑖=1 𝑖𝑖 2.6 Spectral Information Divergence (SID) Spectral Information Divergence (SID) estimates the similarity between two spectra using divergence measures [23]. The reference and target spectra will be normalized to a range of [0,1], by using Equation 2 where ‘a’ is the spectrum (vector) a = (𝑎𝑎𝑖𝑖 … … … . 𝑎𝑎𝐿𝐿 ) each member of ‘a’ represents a reflectance value corresponds to the wavelength λi and L denotes the total number of bands in the spectrum. 𝑎𝑎𝑗𝑗 𝑝𝑝𝑗𝑗 = ∑𝐿𝐿 (2) 𝑖𝑖=1 𝑎𝑎𝑖𝑖 Using Equation 2, we can define a normalized vector as in Equation 3 p = {𝑝𝑝}𝐿𝐿j=1 (3) Finally, SID can be defined as Equation 4, where x and y are the normalized vectors generated from reference (r) and target (t) 6 B. M. Devassy and S. George 𝑆𝑆𝑆𝑆𝑆𝑆(𝑥𝑥, 𝑦𝑦) = 𝐷𝐷(𝑥𝑥 ∥ 𝑦𝑦 ) + 𝐷𝐷(𝑦𝑦 ∥ 𝑥𝑥 ) (4) Where 𝑟𝑟 𝐷𝐷(𝑥𝑥 ∥ 𝑦𝑦 ) = ∑𝐿𝐿𝑖𝑖=1 𝑟𝑟𝑖𝑖 log � 𝑖𝑖�𝑡𝑡 � (5) 𝑖𝑖 𝐷𝐷(𝑦𝑦 ∥ 𝑥𝑥 ) = ∑𝐿𝐿𝑖𝑖=1 𝑡𝑡𝑖𝑖 𝑙𝑙𝑙𝑙𝑙𝑙 �𝑡𝑡𝑖𝑖�𝑟𝑟𝑖𝑖 � (6) 2.7 Data Processing The major steps in the data processing pipeline are preprocessing, calculating normal- ized reflectance, and sample segmentation. The camera software performs the prepro- cessing of the data, which includes dark current reduction, sensor corrections, and ra- diometric calibration. After preprocessing, the HSI data were converted to normalized reflectance by utilizing the known reflectance of the reference target present in the scene. Then manual selection of the region of interest (ROI) will be done for each strawberry for segmentation to avoid saturation areas of the data. The saturation areas are part of the fruit possess abnormal spectra due to the glossiness of the strawberry. For faster processing of data, here we decided the ROI size as 5x5 pixels. 2.8 CNN Implementation and Parameter Tuning The proposed CNN architecture was implemented in Python using Keras [24] and a Python framework known as SHERPA [25] was used for parameter tuning. The number of filters, kernel size for convolution layer, the batch size for training, learning rate, number of hidden layers, and number of epochs of the proposed CNN model were tuned using SHERPA. All those parameters were initialized with random Gaussian distribu- tions and optimized using Bayesian optimization for hyperparameters tuning [26]. 2.9 Training and Evaluation The entire strawberries were classified into two groups based on a threshold sugar value. The berries have sugar values greater than the threshold was considered as high sugar content and the berries having lower sugar values than the threshold were con- sidered as low sugar content. The sugar values of the strawberries were varies between 6oBx and 12oBx, hence we used multiple threshold values starting from 7oBx to 10oBx with 0.5oBx increment. The usage of varying threshold caused imbalanced data sets and used random oversampling to compensate for the imbalanced data. The oversampled data were divided randomly into the train and test data, with train data, contains 80% of the total data, and the remaining were used for evaluation of 1D-CNN. To evaluate SAM and SID, the reference spectra were generated from the training data set by cal- culating the mean spectra[27]. The K-Fold technique with shuffle on and split count five was used to calculate the cross-validation result. Accuracy was used as the parameter for comparing the classification capability of the proposed method against SAM and SID. Accuracy is defined as the ratio between truly predicted outcomes (true positives + true negatives) and the sum of all predictions. Contactless Classification of Strawberry Using Hyperspectral Imaging 7 Fig. 4 Reflectance spectra of 50 strawberries, each averaged across a spatial region 3 Results and Discussion HSI of 50 strawberries were acquired and processed using the setup and processing methods described in Section 2.2 and 2,7. The average spectra for all strawberries ob- tained from their ROIs are presented in Fig.4, we can observe that they appear nearly similar in visible region and differ in near infrared (NIR) region. From the average spectra, it is difficult to classify them visually; hence, we required some reliable method for achieving this. The proposed 1D-CNN is implemented and tuned for hyperparameters, the final archi- tecture with fine-tuned values are shown in Fig.5 The number of hidden layers required was determined as three, the input and output data sizes for each block based on the final parameters are updated in the diagram. The details of the parameters and final tuned values are displayed in Table 1. Fig. 5 Fine-tuned 1D-CNN architecture 8 B. M. Devassy and S. George Table 1 CNN parameter values used in this study Parameter name Optimum Range used for value tuning Number of filters 8 8 to 32 Kernel Size 3 3 to 13 Batch Size 32 32 to 128 Learning Rate 0.003 0.001 to 0.05 Hidden Layers 2 2 to 10 Epochs 10 5 to 50 The fine-tuned 1D-CNN is trained, tested, and compared the result against SID and SAM. Fig.6 shows the variation in the accuracy and loss against epochs, and it can be observed that accuracy and loss flattens in a few epochs. Table 2 provides the summary of accuracy results obtained after cross-validation for each threshold values. From these results, it is clear that the proposed method outperformed the traditional methods, the 1D-CNN possesses a high average accuracy score of 0.96 compared to 0.58 of SID and 0.6 of SAM. Also, it is not fare to compare this result against the previous studies re- lated to sugar and spectra of strawberries because they were mainly focused on predict- ing sugar values rather than classification [28][29] Fig. 6 Accuracy and loss variation against training epochs Contactless Classification of Strawberry Using Hyperspectral Imaging 9 Table 2. Accuracy obtained for 1D-CNN, SAM and SID. Threshold (oBx) SID SAM 1D-CNN 7 0.623 0.686 0.975 7.5 0.526 0.609 0.965 8 0.501 0.547 0.978 8.5 0.586 0.564 0.967 9 0.619 0.617 0.948 9.5 0.562 0.619 0.953 10 0.650 0.578 0.978 Average Accuracy 0.581 0.602 0.966 Fig. 7 Relation between accuracy and threshold sugar value To evaluate the effectiveness of the method, we varied the threshold sugar values and executed cross-validation for each threshold value. The average accuracy obtained from cross-validation is plotted in Fig.7, which showed that the threshold sugar value has a negligible effect on the accuracy of the classification in 1D-CNN method compared to SID and SAM. 10 B. M. Devassy and S. George Fig. 8 Reference spectrum used for SAM and SID classification The performance of SAM and SID was low because of the nearly identical reference spectra, they need a reference spectrum to compare against the test spectra and the ref- erence spectra were generated from the training spectra by calculating the average. The sample reference spectra were presented in Fig.8 and we can observe that the reference spectrum for low and high sugar values possess a nearly identical spectrum. Hence, these methods such as SAM and SID, which relies on geometry of the spectrum, fails to predict correctly in most cases. However, 1D-CNN that can extract features from every sample spectrum in the training set can learn effectively and make a precise pre- diction. 4 Conclusion Hyperspectral acquisition of the strawberries were performed and created an HSI data- base of 50 strawberries. The proposed 1D-CNN method was implemented and tested on the strawberry’s HSI data set. In addition, validated the classification accuracy of the proposed method against SID and SAM and the proposed method produced a higher accuracy in classifying strawberries based on sugar levels. In future research, we aim to extend these results into wide verities and larger sample count in order to make an industrial application. References 1. Tewari, J.C., Dixit, V., Cho, B.K., Malik, K.A.: Determination of origin and sugars of citrus fruits using genetic algorithm, correspondence analysis and partial least square combined with fiber optic NIR spectroscopy. Spectrochim. Acta - Part A Mol. Biomol. Spectrosc. (2008). https://doi.org/10.1016/j.saa.2008.03.005. Contactless Classification of Strawberry Using Hyperspectral Imaging 11 2. Choi, K., Lee, K., Kim, G.: Nondestructive quality evaluation technology for fruits and vegetables using near-infrared spectroscopy. Int. Semin. Enhancing Export Compet. Asian Fruits. (2006). 3. Manolakis, D., Marden, D., Shaw, G. a: Hyperspectral Image Processing for Automatic Target Detection Applications. Lincoln Lab. J. (2003). 4. Pu, Y.Y., Feng, Y.Z., Sun, D.W.: Recent progress of hyperspectral imaging on quality and safety inspection of fruits and vegetables: A review. Compr. Rev. Food Sci. Food Saf. (2015). https://doi.org/10.1111/1541-4337.12123. 5. Zhu, H., Chu, B., Fan, Y., Tao, X., Yin, W., He, Y.: Hyperspectral Imaging for Predicting the Internal Quality of Kiwifruits Based on Variable Selection Algorithms and Chemometric Models. Sci. Rep. (2017). https://doi.org/10.1038/s41598-017-08509-6. 6. Li, J., Huang, W., Tian, X., Wang, C., Fan, S., Zhao, C.: Fast detection and visualization of early decay in citrus using Vis-NIR hyperspectral imaging. Comput. Electron. Agric. (2016). https://doi.org/10.1016/j.compag.2016.07.016. 7. Adão, T., Hruška, J., Pádua, L., Bessa, J., Peres, E., Morais, R., Sousa, J.J.: Hyperspectral imaging: A review on UAV-based sensors, data processing and applications for agriculture and forestry. Remote Sens. (2017). https://doi.org/10.3390/rs9111110. 8. Deborah, H., George, S., Hardeberg, J.Y.: Spectral-divergence based pigment discrimination and mapping: A case study on The Scream (1893) by Edvard Munch. J. Am. Inst. Conserv. (2019). https://doi.org/10.1080/01971360.2018.1560756. 9. Melit Devassy, B., George, S., Hardeberg, J.Y.: Comparison of Ink Classification Capabilities of Classic Hyperspectral Similarity Features. Presented at the (2019). https://doi.org/10.1109/icdarw.2019.70137. 10. Melit Devassy, B., George, S.: Dimensionality reduction and visualisation of hyperspectral ink data using t-SNE. Forensic Sci. Int. (2020). https://doi.org/10.1016/j.forsciint.2020.110194. 11. Food and Agriculture Organization of the United Nations, http://www.fao.org/home/en/, last accessed 2020/06/10. 12. Donida Labati, R., Muñoz, E., Piuri, V., Sassi, R., Scotti, F.: Deep-ECG: Convolutional Neural Networks for ECG biometric recognition, (2018). https://doi.org/10.1016/j.patrec.2018.03.028. 13. Hershey, S., Chaudhuri, S., Ellis, D.P.W., Gemmeke, J.F., Jansen, A., Moore, R.C., Plakal, M., Platt, D., Saurous, R.A., Seybold, B., Slaney, M., Weiss, R.J., Wilson, K.: CNN architectures for large-scale audio classification. In: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings. pp. 131–135 (2017). https://doi.org/10.1109/ICASSP.2017.7952132. 14. Abdeljaber, O., Avci, O., Kiranyaz, S., Gabbouj, M., Inman, D.J.: Real-time vibration-based structural damage detection using one-dimensional convolutional neural networks. J. Sound Vib. 388, 154–170 (2017). https://doi.org/10.1016/j.jsv.2016.10.043. 12 B. M. Devassy and S. George 15. Manganaro, G., De Gyvez, J.P.: One-dimensional discrete-time CNN with multiplexed template-hardware. IEEE Trans. Circuits Syst. I Fundam. Theory Appl. 47, 764–769 (2000). https://doi.org/10.1109/81.847884. 16. Devassy, B.M., George, S.: Ink Classification Using Convolutional Neural Network. Nis. J. 12, (2019). 17. Melit Devassy, B., George, S., Nussbaum, P.: Unsupervised Clustering of Hyperspectral Paper Data Using t-SNE. J. Imaging. 6, 29 (2020). https://doi.org/10.3390/jimaging6050029. 18. HySpex VNIR 1800, https://www.hyspex.no, last accessed 2020/06/03. 19. Contrast Multi-Step Target, https://www.labspherestore.com/, last accessed 2020/06/03. 20. Türkmen, L., Ekşi, A.: Brix degree and sorbitol/xylitol level of authentic pomegranate (Punica granatum) juice. Food Chem. 127, 1404–1407 (2011). https://doi.org/10.1016/j.foodchem.2010.12.118. 21. Kingma, D.P., Ba, J.L.: Adam: A method for stochastic gradient descent. ICLR Int. Conf. Learn. Represent. (2015). 22. Yuhas, R., Goetz, A.F.H., Boardman, J.W.: Descrimination among semi-arid landscape endmembers using the Spectral Angle Mapper (SAM) algorithm. Summ. Third Annu. JPL Airborne Geosci. Work. JPL Publ. 92–14, Vol. 1. (1992). 23. Chang, C.I.: Spectral information divergence for hyperspectral image analysis. In: International Geoscience and Remote Sensing Symposium (IGARSS) (1999). https://doi.org/10.1109/igarss.1999.773549. 24. Keras, https://keras.io/, last accessed 2020/06/05. 25. SHERPA, https://parameter-sherpa.readthedocs.io/en/latest/, last accessed 2020/06/05. 26. Snoek, B.J., Larochelle, H., Adams, R.P., Snoek, J.: PRACTICAL BAYESIAN OPTIMIZATION OF MACHINE LEARNING. Arxiv. 1–12 (2001). 27. Kruse, F.A., Lefkoff, A.B., Boardman, J.W., Heidebrecht, K.B., Shapiro, A.T., Barloon, P.J., Goetz, A.F.H.: The spectral image processing system (SIPS)- interactive visualization and analysis of imaging spectrometer data. Remote Sens. Environ. (1993). https://doi.org/10.1016/0034-4257(93)90013-N. 28. Masateru Nagata, Jasper G. Tallada, Taiichi Kobayashi, Hiroshi Toyoda: NIR Hyperspectral Imaging for Measurement of Internal Quality in Strawberries. Presented at the (2013). https://doi.org/10.13031/2013.19105. 29. ElMasry, G., Wang, N., ElSayed, A., Ngadi, M.: Hyperspectral imaging for nondestructive determination of some quality attributes for strawberry. J. Food Eng. (2007). https://doi.org/10.1016/j.jfoodeng.2006.10.016.