=Paper=
{{Paper
|id=Vol-2688/paper9
|storemode=property
|title=Contactless Classification of Strawberry Using Hyperspectral Imaging
|pdfUrl=https://ceur-ws.org/Vol-2688/paper9.pdf
|volume=Vol-2688
|authors=Binu Melit Devassy,Sony George
|dblpUrl=https://dblp.org/rec/conf/cvcs/DevassyG20
}}
==Contactless Classification of Strawberry Using Hyperspectral Imaging==
https://ceur-ws.org/Vol-2688/paper9.pdf
Contactless Classification of Strawberry Using
Hyperspectral Imaging
Binu Melit Devassy1 and Sony George1
1 Department of Computer Science, Norwegian University of Science and Technology, Gjøvik
2802, Norway;
binu.m.devassy@ntnu.no, sony.george@ntnu.no
Abstract. Rapid non-contact estimation of fruit quality parameters is an essential
factor for an efficient food processing pipeline. We propose a novel workflow
for the contactless classification of strawberries based on their sugar content, us-
ing Hyperspectral Imaging (HSI) and One-Dimensional Convolutional Neural
Network (1D - CNN). Sugar content is an important quality aspect of strawber-
ries, hence classification based on sugar content gives more yield to the fruit pro-
ducers. We used Visible and Near Infrared (VNIR) hyperspectral camera to ac-
quire HSI data of 50 ripe strawberries and applied the proposed method to clas-
sify them. To verify the advantage of the proposed method, the results from 1D-
CNN are compared against other standard classification methods such as Spectral
Angle Mapper (SAM), and Spectral Information Divergence (SID). The results
show that the 1D-CNN outperformed other methods by achieving 96.6% classi-
fication accuracy.
Keywords: Hyperspectral Imaging, Strawberry classification, Fruits classifica-
tion using CNN.
1 Introduction
Non-invasive measurement of different food attributes is of great interest. The sugar
content is one of the characteristics that enhances the customer experience and influ-
ence the market value of the fruit [1]. Sugar content is one of the key factors that is
useful for grading the fruits [2], measurement of this attribute with the application of
imaging technologies eases the sorting process and offer several advantages like con-
tactless and non-destructive measurement, the possibility for automation, and high ac-
curacy within a certain limit. Hyperspectral imaging (HSI) is one of the most suitable
imaging modalities that is proven to be useful for this purpose [3]. There has been many
extensive studies reported on the usefulness of HSI for non-invasive measurement of
several types of fruits such as apples, oranges, and kiwifruits [4][5][6].
Hyperspectral imaging systems capture both spatial and spectral information simulta-
neously, which enables HSI technology to make a more reliable classification compared
to the traditional three channel imaging methods. In addition to this, HSI already proven
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0). Colour and Visual Computing
Symposium 2020, Gjøvik, Norway, September 16-17, 2020.
2 B. M. Devassy and S. George
to be one of the effective non-invasive methods to study material properties compared
to the chemical analysis, which are generally invasive in nature. HSI finds applications
in many different fields like remote sensing [7], cultural heritage [8][9], forensics [10],
etc. Food science is an important field where hyperspectral imaging used extensively
to monitor the quality, diseases, and stages of development in variety of food products.
Strawberry is one of the favorite fruits for many people and the interest is reflected in
the increased production in every year [11]. Motivated by this fact that lack of HSI and
predictive sugar analysis for strawberries, we have studied and evaluated the straw-
berry’s classification based on sugar content using HSI. Customer interest in strawberry
is mainly driven by the taste of the fruit and most of the times the sweetness of the fruit.
Convolutional Neural Networks (CNNs) has been used in many computer vision and
image classification applications, CNNs were designed to learn features from the train-
ing data and which can be used to classify the test samples. The most common CNNs
are two-dimensional (2D-CNN), however in this work we proposed a one-dimensional
CNN (1D-CNN) because the spectral signals are One-Dimensional (1D) signals with
varying amplitude. 1D-CNNs can provide reliable solutions in many 1D signal-pro-
cessing applications such as Electrocardiographic (ECG) signals [12], audio[13], and
other 1D signals [14][15]. We can also find a few attempts to use 1D-CNN for spectral
classification [16], this experiment used 1D-CNN to classify ink spectra. To verify the
effectiveness of the proposed method, we used two well-known classification methods
in HSI domain; they are Spectral Angle Mapper (SAM) and Spectral Information Di-
vergence (SID).
Later part of this paper is organized as follows. Section 2 will present the details of the
fruit samples used, hyperspectral acquisition of the fruits and processing of the HSI
data. Followed by Section 3, which presents and discusses the results from this study
and finally the conclusions in Section 4.
2 Materials and Methods
2.1 Fruit Samples
Strawberries were purchased from local markets in Norway and fifty of them were se-
lected as candidates by avoiding the fruits having any defects, bruises, and infections.
All the strawberries were belong to the same class and produced in the same environ-
ment. The fruits were kept in ideal storage conditions, and taken to the lab environment
an hour before the hyperspectral image acquisition, in a controlled room temperature.
The strawberries were washed to remove any contaminations and the water drops from
the fruits were wiped away using a dry clean cloth, before measurement.
Contactless Classification of Strawberry Using Hyperspectral Imaging 3
2.2 Hyperspectral Acquisition
Hyperspectral acquisition system used for this experiment is shown in Fig.1, which is
the same setup used in this experiment [17] except for the samples used. HySpex VNIR-
1800 [18] push broom hyperspectral camera was used for hyperspectral image acquisi-
tion of the fruit samples. The camera placed at right angles to a moving translator stage
where the fruit samples were placed and two halogen light sources were used to illumi-
nate the scene with 45°:0° geometry with respect to the camera to minimize shadowing.
This camera has a spectral sampling of 3.18 nm along with its spectral range, which
divides the supporting spectral range (400 nm to 1000 nm) into 186 bands. The image
acquisition resulted in a hyperspectral data cube with spatial (X and Y) and spectral (Z)
directions. Here, the size of X- axis was 1800 pixels, the size of the Y-axis depends on
the size and number of strawberries used in a single scan, and the size Z-axis was 186.
A reference target with known reflectance (Contrast Multi-Step Target [19]) values was
present in the scene, which will be used to convert the radiance to the reflectance while
processing the data.
Fig. 1 Hyperspectral image acquisition setup
2.3 Sugar Content Measurement
The sugar content of each strawberry was measured immediately after spectral meas-
urement using a refractometer (PAL 1, Atago Co., Ltd., Japan). This method of sugar
content estimation requires the fruits to be squeezed to get the juice, from which the
refractometer analyses the degrees of Brix (o Bx). The degree Brix represents “the per-
centage of water-soluble solids in fruit juice and can be affected by many factors in-
cluding variety, growth region, growth year, and maturity level of the fruit” [20]. In this
case, the degree of Brix represents the sugar content in the strawberry juice; one degree
of brix can be defined as the one gram of sucrose in 100 grams of fruit juice. Fig.2
represents the sugar level distribution measured from the samples used.
4 B. M. Devassy and S. George
Fig. 2 Histogram of sugar values measured from the strawberries used in the present study
2.4 Proposed method
The figure (Fig.3) shows the proposed 1D-CNNs architecture, the input spectra will
pass through a series of hidden layers, each hidden layer consists of 1D-Convolution
layer with ReLU (Rectified Linear Unit) activation, the ‘n’ will be finalized after
parameter tuning. A drop out layer with a rate ‘0.5’ will follow the convolution lay-
ers and then by a max-pooling layer with a pool size of ‘2’. Then a flatten layer will
flatten the max-pooled output, followed by two dense layers, the last dense layer use
‘softmax’ activation to generate the classification result. The network used the cate-
gorical cross-entropy as loss function, which is a proven technique for learning a
multi-class classification problem and used Adam [21] optimization.
Contactless Classification of Strawberry Using Hyperspectral Imaging 5
Fig. 3 Proposed 1D-CNN architecture
2.5 Spectral Angle Mapper (SAM)
The Spectral Angle Mapper (SAM) is one of the important spectral similarity criteria
used to estimates the spectral match between the reference and target spectra by meas-
uring the angular difference in radian between the reference and test spectrum [22] as
in Equation 1. In this process, both spectra were considered as vectors having dimen-
sionality equal to the number of bands (nb) in the spectrum. The angle alpha (α) defines
the similarity between spectra, where t and r be the test and reference spectrum respec-
tively.
∑𝑛𝑛𝑛𝑛
𝑖𝑖=1 𝑡𝑡𝑖𝑖 𝑟𝑟𝑖𝑖
𝛼𝛼 = cos −1 � 1 1 � (1)
�∑𝑛𝑛𝑛𝑛 2 �2�∑𝑛𝑛𝑛𝑛 𝑟𝑟 2 � �2
𝑖𝑖=1 𝑡𝑡𝑖𝑖 � 𝑖𝑖=1 𝑖𝑖
2.6 Spectral Information Divergence (SID)
Spectral Information Divergence (SID) estimates the similarity between two spectra
using divergence measures [23]. The reference and target spectra will be normalized to
a range of [0,1], by using Equation 2 where ‘a’ is the spectrum (vector)
a = (𝑎𝑎𝑖𝑖 … … … . 𝑎𝑎𝐿𝐿 ) each member of ‘a’ represents a reflectance value corresponds to
the wavelength λi and L denotes the total number of bands in the spectrum.
𝑎𝑎𝑗𝑗
𝑝𝑝𝑗𝑗 = ∑𝐿𝐿 (2)
𝑖𝑖=1 𝑎𝑎𝑖𝑖
Using Equation 2, we can define a normalized vector as in Equation 3
p = {𝑝𝑝}𝐿𝐿j=1 (3)
Finally, SID can be defined as Equation 4, where x and y are the normalized vectors
generated from reference (r) and target (t)
6 B. M. Devassy and S. George
𝑆𝑆𝑆𝑆𝑆𝑆(𝑥𝑥, 𝑦𝑦) = 𝐷𝐷(𝑥𝑥 ∥ 𝑦𝑦 ) + 𝐷𝐷(𝑦𝑦 ∥ 𝑥𝑥 ) (4)
Where
𝑟𝑟
𝐷𝐷(𝑥𝑥 ∥ 𝑦𝑦 ) = ∑𝐿𝐿𝑖𝑖=1 𝑟𝑟𝑖𝑖 log � 𝑖𝑖�𝑡𝑡 � (5)
𝑖𝑖
𝐷𝐷(𝑦𝑦 ∥ 𝑥𝑥 ) = ∑𝐿𝐿𝑖𝑖=1 𝑡𝑡𝑖𝑖 𝑙𝑙𝑙𝑙𝑙𝑙 �𝑡𝑡𝑖𝑖�𝑟𝑟𝑖𝑖 � (6)
2.7 Data Processing
The major steps in the data processing pipeline are preprocessing, calculating normal-
ized reflectance, and sample segmentation. The camera software performs the prepro-
cessing of the data, which includes dark current reduction, sensor corrections, and ra-
diometric calibration. After preprocessing, the HSI data were converted to normalized
reflectance by utilizing the known reflectance of the reference target present in the
scene. Then manual selection of the region of interest (ROI) will be done for each
strawberry for segmentation to avoid saturation areas of the data. The saturation areas
are part of the fruit possess abnormal spectra due to the glossiness of the strawberry.
For faster processing of data, here we decided the ROI size as 5x5 pixels.
2.8 CNN Implementation and Parameter Tuning
The proposed CNN architecture was implemented in Python using Keras [24] and a
Python framework known as SHERPA [25] was used for parameter tuning. The number
of filters, kernel size for convolution layer, the batch size for training, learning rate,
number of hidden layers, and number of epochs of the proposed CNN model were tuned
using SHERPA. All those parameters were initialized with random Gaussian distribu-
tions and optimized using Bayesian optimization for hyperparameters tuning [26].
2.9 Training and Evaluation
The entire strawberries were classified into two groups based on a threshold sugar
value. The berries have sugar values greater than the threshold was considered as high
sugar content and the berries having lower sugar values than the threshold were con-
sidered as low sugar content. The sugar values of the strawberries were varies between
6oBx and 12oBx, hence we used multiple threshold values starting from 7oBx to 10oBx
with 0.5oBx increment. The usage of varying threshold caused imbalanced data sets and
used random oversampling to compensate for the imbalanced data. The oversampled
data were divided randomly into the train and test data, with train data, contains 80%
of the total data, and the remaining were used for evaluation of 1D-CNN. To evaluate
SAM and SID, the reference spectra were generated from the training data set by cal-
culating the mean spectra[27]. The K-Fold technique with shuffle on and split count
five was used to calculate the cross-validation result.
Accuracy was used as the parameter for comparing the classification capability of the
proposed method against SAM and SID. Accuracy is defined as the ratio between truly
predicted outcomes (true positives + true negatives) and the sum of all predictions.
Contactless Classification of Strawberry Using Hyperspectral Imaging 7
Fig. 4 Reflectance spectra of 50 strawberries, each averaged across a spatial region
3 Results and Discussion
HSI of 50 strawberries were acquired and processed using the setup and processing
methods described in Section 2.2 and 2,7. The average spectra for all strawberries ob-
tained from their ROIs are presented in Fig.4, we can observe that they appear nearly
similar in visible region and differ in near infrared (NIR) region. From the average
spectra, it is difficult to classify them visually; hence, we required some reliable method
for achieving this.
The proposed 1D-CNN is implemented and tuned for hyperparameters, the final archi-
tecture with fine-tuned values are shown in Fig.5 The number of hidden layers required
was determined as three, the input and output data sizes for each block based on the
final parameters are updated in the diagram. The details of the parameters and final
tuned values are displayed in Table 1.
Fig. 5 Fine-tuned 1D-CNN architecture
8 B. M. Devassy and S. George
Table 1 CNN parameter values used in this study
Parameter name Optimum Range used for
value tuning
Number of filters 8 8 to 32
Kernel Size 3 3 to 13
Batch Size 32 32 to 128
Learning Rate 0.003 0.001 to 0.05
Hidden Layers 2 2 to 10
Epochs 10 5 to 50
The fine-tuned 1D-CNN is trained, tested, and compared the result against SID and
SAM. Fig.6 shows the variation in the accuracy and loss against epochs, and it can be
observed that accuracy and loss flattens in a few epochs. Table 2 provides the summary
of accuracy results obtained after cross-validation for each threshold values. From these
results, it is clear that the proposed method outperformed the traditional methods, the
1D-CNN possesses a high average accuracy score of 0.96 compared to 0.58 of SID and
0.6 of SAM. Also, it is not fare to compare this result against the previous studies re-
lated to sugar and spectra of strawberries because they were mainly focused on predict-
ing sugar values rather than classification [28][29]
Fig. 6 Accuracy and loss variation against training epochs
Contactless Classification of Strawberry Using Hyperspectral Imaging 9
Table 2. Accuracy obtained for 1D-CNN, SAM and SID.
Threshold (oBx) SID SAM 1D-CNN
7 0.623 0.686 0.975
7.5 0.526 0.609 0.965
8 0.501 0.547 0.978
8.5 0.586 0.564 0.967
9 0.619 0.617 0.948
9.5 0.562 0.619 0.953
10 0.650 0.578 0.978
Average Accuracy 0.581 0.602 0.966
Fig. 7 Relation between accuracy and threshold sugar value
To evaluate the effectiveness of the method, we varied the threshold sugar values and
executed cross-validation for each threshold value. The average accuracy obtained from
cross-validation is plotted in Fig.7, which showed that the threshold sugar value has a
negligible effect on the accuracy of the classification in 1D-CNN method compared to
SID and SAM.
10 B. M. Devassy and S. George
Fig. 8 Reference spectrum used for SAM and SID classification
The performance of SAM and SID was low because of the nearly identical reference
spectra, they need a reference spectrum to compare against the test spectra and the ref-
erence spectra were generated from the training spectra by calculating the average. The
sample reference spectra were presented in Fig.8 and we can observe that the reference
spectrum for low and high sugar values possess a nearly identical spectrum. Hence,
these methods such as SAM and SID, which relies on geometry of the spectrum, fails
to predict correctly in most cases. However, 1D-CNN that can extract features from
every sample spectrum in the training set can learn effectively and make a precise pre-
diction.
4 Conclusion
Hyperspectral acquisition of the strawberries were performed and created an HSI data-
base of 50 strawberries. The proposed 1D-CNN method was implemented and tested
on the strawberry’s HSI data set. In addition, validated the classification accuracy of
the proposed method against SID and SAM and the proposed method produced a higher
accuracy in classifying strawberries based on sugar levels. In future research, we aim
to extend these results into wide verities and larger sample count in order to make an
industrial application.
References
1. Tewari, J.C., Dixit, V., Cho, B.K., Malik, K.A.: Determination of origin and
sugars of citrus fruits using genetic algorithm, correspondence analysis and
partial least square combined with fiber optic NIR spectroscopy. Spectrochim.
Acta - Part A Mol. Biomol. Spectrosc. (2008).
https://doi.org/10.1016/j.saa.2008.03.005.
Contactless Classification of Strawberry Using Hyperspectral Imaging 11
2. Choi, K., Lee, K., Kim, G.: Nondestructive quality evaluation technology for
fruits and vegetables using near-infrared spectroscopy. Int. Semin. Enhancing
Export Compet. Asian Fruits. (2006).
3. Manolakis, D., Marden, D., Shaw, G. a: Hyperspectral Image Processing for
Automatic Target Detection Applications. Lincoln Lab. J. (2003).
4. Pu, Y.Y., Feng, Y.Z., Sun, D.W.: Recent progress of hyperspectral imaging on
quality and safety inspection of fruits and vegetables: A review. Compr. Rev.
Food Sci. Food Saf. (2015). https://doi.org/10.1111/1541-4337.12123.
5. Zhu, H., Chu, B., Fan, Y., Tao, X., Yin, W., He, Y.: Hyperspectral Imaging for
Predicting the Internal Quality of Kiwifruits Based on Variable Selection
Algorithms and Chemometric Models. Sci. Rep. (2017).
https://doi.org/10.1038/s41598-017-08509-6.
6. Li, J., Huang, W., Tian, X., Wang, C., Fan, S., Zhao, C.: Fast detection and
visualization of early decay in citrus using Vis-NIR hyperspectral imaging.
Comput. Electron. Agric. (2016).
https://doi.org/10.1016/j.compag.2016.07.016.
7. Adão, T., Hruška, J., Pádua, L., Bessa, J., Peres, E., Morais, R., Sousa, J.J.:
Hyperspectral imaging: A review on UAV-based sensors, data processing and
applications for agriculture and forestry. Remote Sens. (2017).
https://doi.org/10.3390/rs9111110.
8. Deborah, H., George, S., Hardeberg, J.Y.: Spectral-divergence based pigment
discrimination and mapping: A case study on The Scream (1893) by Edvard
Munch. J. Am. Inst. Conserv. (2019).
https://doi.org/10.1080/01971360.2018.1560756.
9. Melit Devassy, B., George, S., Hardeberg, J.Y.: Comparison of Ink
Classification Capabilities of Classic Hyperspectral Similarity Features.
Presented at the (2019). https://doi.org/10.1109/icdarw.2019.70137.
10. Melit Devassy, B., George, S.: Dimensionality reduction and visualisation of
hyperspectral ink data using t-SNE. Forensic Sci. Int. (2020).
https://doi.org/10.1016/j.forsciint.2020.110194.
11. Food and Agriculture Organization of the United Nations,
http://www.fao.org/home/en/, last accessed 2020/06/10.
12. Donida Labati, R., Muñoz, E., Piuri, V., Sassi, R., Scotti, F.: Deep-ECG:
Convolutional Neural Networks for ECG biometric recognition, (2018).
https://doi.org/10.1016/j.patrec.2018.03.028.
13. Hershey, S., Chaudhuri, S., Ellis, D.P.W., Gemmeke, J.F., Jansen, A., Moore,
R.C., Plakal, M., Platt, D., Saurous, R.A., Seybold, B., Slaney, M., Weiss, R.J.,
Wilson, K.: CNN architectures for large-scale audio classification. In: ICASSP,
IEEE International Conference on Acoustics, Speech and Signal Processing -
Proceedings. pp. 131–135 (2017).
https://doi.org/10.1109/ICASSP.2017.7952132.
14. Abdeljaber, O., Avci, O., Kiranyaz, S., Gabbouj, M., Inman, D.J.: Real-time
vibration-based structural damage detection using one-dimensional
convolutional neural networks. J. Sound Vib. 388, 154–170 (2017).
https://doi.org/10.1016/j.jsv.2016.10.043.
12 B. M. Devassy and S. George
15. Manganaro, G., De Gyvez, J.P.: One-dimensional discrete-time CNN with
multiplexed template-hardware. IEEE Trans. Circuits Syst. I Fundam. Theory
Appl. 47, 764–769 (2000). https://doi.org/10.1109/81.847884.
16. Devassy, B.M., George, S.: Ink Classification Using Convolutional Neural
Network. Nis. J. 12, (2019).
17. Melit Devassy, B., George, S., Nussbaum, P.: Unsupervised Clustering of
Hyperspectral Paper Data Using t-SNE. J. Imaging. 6, 29 (2020).
https://doi.org/10.3390/jimaging6050029.
18. HySpex VNIR 1800, https://www.hyspex.no, last accessed 2020/06/03.
19. Contrast Multi-Step Target, https://www.labspherestore.com/, last accessed
2020/06/03.
20. Türkmen, L., Ekşi, A.: Brix degree and sorbitol/xylitol level of authentic
pomegranate (Punica granatum) juice. Food Chem. 127, 1404–1407 (2011).
https://doi.org/10.1016/j.foodchem.2010.12.118.
21. Kingma, D.P., Ba, J.L.: Adam: A method for stochastic gradient descent. ICLR
Int. Conf. Learn. Represent. (2015).
22. Yuhas, R., Goetz, A.F.H., Boardman, J.W.: Descrimination among semi-arid
landscape endmembers using the Spectral Angle Mapper (SAM) algorithm.
Summ. Third Annu. JPL Airborne Geosci. Work. JPL Publ. 92–14, Vol. 1.
(1992).
23. Chang, C.I.: Spectral information divergence for hyperspectral image analysis.
In: International Geoscience and Remote Sensing Symposium (IGARSS)
(1999). https://doi.org/10.1109/igarss.1999.773549.
24. Keras, https://keras.io/, last accessed 2020/06/05.
25. SHERPA, https://parameter-sherpa.readthedocs.io/en/latest/, last accessed
2020/06/05.
26. Snoek, B.J., Larochelle, H., Adams, R.P., Snoek, J.: PRACTICAL BAYESIAN
OPTIMIZATION OF MACHINE LEARNING. Arxiv. 1–12 (2001).
27. Kruse, F.A., Lefkoff, A.B., Boardman, J.W., Heidebrecht, K.B., Shapiro, A.T.,
Barloon, P.J., Goetz, A.F.H.: The spectral image processing system (SIPS)-
interactive visualization and analysis of imaging spectrometer data. Remote
Sens. Environ. (1993). https://doi.org/10.1016/0034-4257(93)90013-N.
28. Masateru Nagata, Jasper G. Tallada, Taiichi Kobayashi, Hiroshi Toyoda: NIR
Hyperspectral Imaging for Measurement of Internal Quality in Strawberries.
Presented at the (2013). https://doi.org/10.13031/2013.19105.
29. ElMasry, G., Wang, N., ElSayed, A., Ngadi, M.: Hyperspectral imaging for
nondestructive determination of some quality attributes for strawberry. J. Food
Eng. (2007). https://doi.org/10.1016/j.jfoodeng.2006.10.016.