Non-invasive measurement of different food attributes is of great interest. The sugar content is one of the characteristics that enhances the customer experience and influence the market value of the fruit [1]. Sugar content is one of the key factors that is useful for grading the fruits [2], measurement of this attribute with the application of imaging technologies eases the sorting process and offer several advantages like contactless and non-destructive measurement, the possibility for automation, and high accuracy within a certain limit. Hyperspectral imaging (HSI) is one of the most suitable imaging modalities that is proven to be useful for this purpose [3]. There has been many extensive studies reported on the usefulness of HSI for non-invasive measurement of several types of fruits such as apples, oranges, and kiwifruits [4][5] [6].
Hyperspectral imaging systems capture both spatial and spectral information simultaneously, which enables HSI technology to make a more reliable classification compared to the traditional three channel imaging methods. In addition to this, HSI already proven to be one of the effective non-invasive methods to study material properties compared to the chemical analysis, which are generally invasive in nature. HSI finds applications in many different fields like remote sensing [7], cultural heritage [8][9], forensics [10], etc. Food science is an important field where hyperspectral imaging used extensively to monitor the quality, diseases, and stages of development in variety of food products.
Strawberry is one of the favorite fruits for many people and the interest is reflected in the increased production in every year [11]. Motivated by this fact that lack of HSI and predictive sugar analysis for strawberries, we have studied and evaluated the strawberry's classification based on sugar content using HSI. Customer interest in strawberry is mainly driven by the taste of the fruit and most of the times the sweetness of the fruit.
Convolutional Neural Networks (CNNs) has been used in many computer vision and image classification applications, CNNs were designed to learn features from the training data and which can be used to classify the test samples. The most common CNNs are two-dimensional (2D-CNN), however in this work we proposed a one-dimensional CNN (1D-CNN) because the spectral signals are One-Dimensional (1D) signals with varying amplitude. 1D-CNNs can provide reliable solutions in many 1D signal-processing applications such as Electrocardiographic (ECG) signals [12], audio [13], and other 1D signals [14][15]. We can also find a few attempts to use 1D-CNN for spectral classification [16], this experiment used 1D-CNN to classify ink spectra. To verify the effectiveness of the proposed method, we used two well-known classification methods in HSI domain; they are Spectral Angle Mapper (SAM) and Spectral Information Divergence (SID).
Later part of this paper is organized as follows. Section 2 will present the details of the fruit samples used, hyperspectral acquisition of the fruits and processing of the HSI data. Followed by Section 3, which presents and discusses the results from this study and finally the conclusions in Section 4.
Strawberries were purchased from local markets in Norway and fifty of them were selected as candidates by avoiding the fruits having any defects, bruises, and infections. All the strawberries were belong to the same class and produced in the same environment. The fruits were kept in ideal storage conditions, and taken to the lab environment an hour before the hyperspectral image acquisition, in a controlled room temperature. The strawberries were washed to remove any contaminations and the water drops from the fruits were wiped away using a dry clean cloth, before measurement.
Hyperspectral acquisition system used for this experiment is shown in Fig. 1, which is the same setup used in this experiment [17] except for the samples used. HySpex VNIR-1800 [18] push broom hyperspectral camera was used for hyperspectral image acquisition of the fruit samples. The camera placed at right angles to a moving translator stage where the fruit samples were placed and two halogen light sources were used to illuminate the scene with 45°:0° geometry with respect to the camera to minimize shadowing. This camera has a spectral sampling of 3.18 nm along with its spectral range, which divides the supporting spectral range (400 nm to 1000 nm) into 186 bands. The image acquisition resulted in a hyperspectral data cube with spatial (X and Y) and spectral (Z) directions. Here, the size of X-axis was 1800 pixels, the size of the Y-axis depends on the size and number of strawberries used in a single scan, and the size Z-axis was 186.
A reference target with known reflectance (Contrast Multi-Step Target [19]) values was present in the scene, which will be used to convert the radiance to the reflectance while processing the data.
Fig. 1 Hyperspectral image acquisition setup
The sugar content of each strawberry was measured immediately after spectral measurement using a refractometer (PAL 1, Atago Co., Ltd., Japan). This method of sugar content estimation requires the fruits to be squeezed to get the juice, from which the refractometer analyses the degrees of Brix ( o Bx). The degree Brix represents "the percentage of water-soluble solids in fruit juice and can be affected by many factors including variety, growth region, growth year, and maturity level of the fruit" [20]. In this case, the degree of Brix represents the sugar content in the strawberry juice; one degree of brix can be defined as the one gram of sucrose in 100 grams of fruit juice. Fig. 2 represents the sugar level distribution measured from the samples used.
Fig. 2 Histogram of sugar values measured from the strawberries used in the present study
The figure (Fig. 3) shows the proposed 1D-CNNs architecture, the input spectra will pass through a series of hidden layers, each hidden layer consists of 1D-Convolution layer with ReLU (Rectified Linear Unit) activation, the 'n' will be finalized after parameter tuning. A drop out layer with a rate '0.5' will follow the convolution layers and then by a max-pooling layer with a pool size of '2'. Then a flatten layer will flatten the max-pooled output, followed by two dense layers, the last dense layer use 'softmax' activation to generate the classification result. The network used the categorical cross-entropy as loss function, which is a proven technique for learning a multi-class classification problem and used Adam [21] optimization.
Fig. 3 Proposed 1D-CNN architecture
The Spectral Angle Mapper (SAM) is one of the important spectral similarity criteria used to estimates the spectral match between the reference and target spectra by measuring the angular difference in radian between the reference and test spectrum [22] as in Equation 1. In this process, both spectra were considered as vectors having dimensionality equal to the number of bands (nb) in the spectrum. The angle alpha (α) defines the similarity between spectra, where t and r be the test and reference spectrum respectively.
Spectral Information Divergence (SID) estimates the similarity between two spectra using divergence measures [23]. The reference and target spectra will be normalized to a range of [0,1], by using Equation 2where 'a' is the spectrum (vector) a = (𝑎𝑎 𝑖𝑖 … … … . 𝑎𝑎 𝐿𝐿 ) each member of 'a' represents a reflectance value corresponds to the wavelength λi and L denotes the total number of bands in the spectrum.
Using Equation 2, we can define a normalized vector as in Equation 3
Finally, SID can be defined as Equation 4, where x and y are the normalized vectors generated from reference (r) and target (t)
Where
(5)
The major steps in the data processing pipeline are preprocessing, calculating normalized reflectance, and sample The camera software performs the preprocessing of the data, which includes dark current reduction, sensor corrections, and radiometric calibration. After preprocessing, the HSI data were converted to normalized reflectance by utilizing the known reflectance of the reference target present in the scene. Then manual selection of the region of interest (ROI) will be done for each strawberry for segmentation to avoid saturation areas of the data. The saturation areas are part of the fruit possess abnormal spectra due to the glossiness of the strawberry.
For faster processing of data, here we decided the ROI size as 5x5 pixels.
The proposed CNN architecture was implemented in Python using Keras [24] and a Python framework known as SHERPA [25] was used for parameter tuning. The number of filters, kernel size for convolution layer, the batch size for training, learning rate, number of hidden layers, and number of epochs of the proposed CNN model were tuned using SHERPA. All those parameters were initialized with random Gaussian distributions and optimized using Bayesian optimization for hyperparameters tuning [26].
The entire strawberries were classified into two groups based on a threshold sugar value. The berries have sugar values greater than the threshold was considered as high sugar content and the berries having lower sugar values than the threshold were considered as low sugar content. The sugar values of the strawberries were varies between 6 o Bx and 12 o Bx, hence we used multiple threshold values starting from 7 o Bx to 10 o Bx with 0.5 o Bx increment. The usage of varying threshold caused imbalanced data sets and used random oversampling to compensate for the imbalanced data. The oversampled data were divided randomly into the train and test data, with train data, contains 80% of the total data, and the remaining were used for evaluation of 1D-CNN. To evaluate SAM and SID, the reference spectra were generated from the training data set by calculating the mean spectra [27]. The K-Fold technique with shuffle on and split count five was used to calculate the cross-validation result.
Accuracy was used as the parameter for comparing the classification capability of the proposed method against SAM and SID. Accuracy is defined as the ratio between truly predicted outcomes (true positives + true negatives) and the sum of all predictions.
Fig. 4 Reflectance spectra of 50 strawberries, each averaged across a spatial region 3
HSI of 50 strawberries were acquired and processed using the setup and processing methods described in Section 2.2 and 2,7. The average spectra for all strawberries obtained from their ROIs are presented in Fig. 4, we can observe that they appear nearly similar in visible region and differ in near infrared (NIR) region. From the average spectra, it is difficult to classify them visually; hence, we required some reliable method for achieving this.
The proposed 1D-CNN is implemented and tuned for hyperparameters, the final architecture with fine-tuned values are shown in Fig. 5 The number of hidden layers required was determined as three, the input and output data sizes for each block based on the final parameters are updated in the diagram. The details of the parameters and final tuned values are displayed in Table 1. The fine-tuned 1D-CNN is trained, tested, and compared the result against SID and SAM. Fig. 6 shows the variation in the accuracy and loss against epochs, and it can be observed that accuracy and loss flattens in a few epochs. Table 2 provides the summary of accuracy results obtained after cross-validation for each threshold values. From these results, it is clear that the proposed method outperformed the traditional methods, the 1D-CNN possesses a high average accuracy score of 0.96 compared to 0.58 of SID and 0.6 of SAM. Also, it is not fare to compare this result against the previous studies related to sugar and spectra of strawberries because they were mainly focused on predicting sugar values rather than classification [28][29]
To evaluate the effectiveness of the method, we varied the threshold sugar values and executed cross-validation for each threshold value. The average accuracy obtained from cross-validation is plotted in Fig. 7, which showed that the threshold sugar value has a negligible effect on the accuracy of the classification in 1D-CNN method compared to SID and SAM. The performance of SAM and SID was low because of the nearly identical reference spectra, they need a reference spectrum to compare against the test spectra and the reference spectra were generated from the training spectra by calculating the average. The sample reference spectra were presented in Fig. 8 and we can observe that the reference spectrum for low and high sugar values possess a nearly identical spectrum. Hence, these methods such as SAM and SID, which relies on geometry of the spectrum, fails to predict correctly in most cases. However, 1D-CNN that can extract features from every sample spectrum in the training set can learn effectively and make a precise prediction.
Hyperspectral acquisition of the strawberries were performed and created an HSI database of 50 strawberries. The proposed 1D-CNN method was implemented and tested on the strawberry's HSI data set. In addition, validated the classification accuracy of the proposed method against SID and SAM and the proposed method produced a higher accuracy in classifying strawberries based on sugar levels. In future research, we aim to extend these results into wide verities and larger sample count in order to make an industrial application.