The possibilities of explainable artificial intelligence (XAI) in the early diagnosis of drought in plants based on hyperspectral images (HSI) are investigated. To provide the explainability and high accuracy to the result, we used the markup of HSI by superimposed Thermal IR (TIR) images of the last day of the experiment. Traditional HSI-based NDVI (Normalized Diference Vegetation Index) images were also constructed. The markup of HSIs based on their clustering by the k-means method into 5 classes was also objectified: wet plants; plants in a state of drought; wet soil; dry soil; background. For HSI, on the day of the experiment started, the number of clusters was set to 2 less to reflect the absence of drought circumstances. For use in training and testing, all HSIs channels are marked up with the results of clustering. The HIS-TIR-combination made it possible to determine the temperature for each plant pixel in HSI, and as the result to determine the number of days without watering. A fully connected Double Layer Perceptron (DLP) neural network was used to solve classification and regression problems. The trained DLP-regressor showed the average accuracy of predicting the temperature of plants on the control days of the experiment RMSE = 0.52 degrees, providing an error in predicting the day of the beginning of the drought for near 2 days. The DLP-classifier was able to classify the drought of the plant in the early stages (the fifth day) with an accuracy of 97.3%. Software tools: pytorch, scikit-learn, pysptools.
Early diagnosis of the state of drought based on the use of artificial intelligence methods is a trend of today. Moreover, it is interesting to build models of explicable artificial intelligence (XAI), which make the diagnosis and forecast of plant drought reliable and visual.
Today, most methods of detecting stress and pathologies in plants using artificial intelligence
are based on RGB image data due to their availability. There are many examples of using
artificial intelligence methods [
However, there is a significant disadvantage in detecting plant stresses, including drought
detection, based on RGB data. They reliably detect drought even when the efects of stress are
visible to the naked eye. This approach undoubtedly gives high accuracy, but it does not give a
chance to prevent stress in a timely manner and cure the plant. Early detection of plant stress
can provide a significant reduction in crop losses in precision farming [
The presence of hyperspectral (HSI) data with hundreds of channels instead of 3 in RGB
expands the possibilities of early diagnosis due to the possibility of constructing special indexes
based on the data of HSI channels, such as NDVI (Normalized Diference Vegetation Index),
focused on monitoring the state of plants. After creating NDVI images in grayscale (or
pseudocolors), you can apply the usual processing methods to the image. Direct HSI processing
methods can be divided into pixel and sub-pixel. Sub-pixel algorithms allow us to estimate the
composition of the material within a given pixel. Their goal is to decompose the image cube into
pure spectral signatures of reference materials, and determine the proportion of each material in
each pixel. Such methods include Linear Spectral Mixing (LSU), Matched Filtering (MF), Mixture
Tuned Matched Filtering (MTMF) [
One of the obstacles to efective HSI classification is the lack of data markup or pure spectra for the objects under study. Despite the existence of a number of methods for automatic extraction of pure signatures, such as: Pixel Purity Index (PPI), Automatic Target Generation Process (ATGP), N-FINDR, Fast Iterative Pixel Purity Index (FIPPI) and others [8], there is still a need for full-scale verification of the decomposition result.
The high eficiency of classification in biology is demonstrated by deep learning, namely convolutional neural networks, including those with 3d convolutions [9, 10]. Possible options are methods based on cluster analysis, for example, a regression model without a teacher [11], discretization into ordinal classes [12], the "Unsupervised domain adaptation" model, presented in [13] or a data-driven method that combines clustering, band allocation, and a measure of spectral similarity [14]. Classical clustering by the k-means method is widely used [15].
The following sequence of classification of plant diseases based on HSI data is widely used: pretreatment (noise removal, radical reduction of the number of channels based on PCA); segmentation (separation of the background using spectral indexes or a classifier); extracting features (PCA, DWT - discrete wavelet transform; LESC - local embedding based on spatial coherence algorithm); the use of a classifier [ 9]. Image segmentation is used as a pre-processing stage and is usually performed before formal spectral analysis in order to extract target objects from the background and form an area of interest for further analysis.
Distinguishing the type of stress (disease) of a plant may require the use of textural (or spatial) features in addition to spectral ones. Thus, the selection of features can be considered as the most important step in hyperspectral classification. Its purpose is to extract and form the most relevant new feature vectors for the detection of plant diseases by combining and optimizing spectral and spatial features. Methods of image segmentation and feature extraction are usually used to improve the eficiency of data analysis, which may not always be necessary when detecting plant diseases [9].
In the work [16], the problem of early detection of drought in plants was solved using a single-layer perceptron (SLP), for which RGB and TIR images were used for training. The results of their preprocessing were fed to the SLP input instead of the images themselves: statistical parameters, quantized histogram values, as well as textural features for solving regression and classification problems. The achieved accuracy of the SLP regressor in terms of RMSE = 0.617day. The possibilities of using TIR and HSI separately for detecting plant stresses are also investigated in the review [17]. The high eficiency of TIR images in detecting plant stress is beginning to be actively used in precision agriculture, since it allows detecting drought in plants at the earliest possible time by the temperature of the plants themselves. A direct competitor of this work is [18], in which derivative spectra instead of the full spectra were used for training of drought detection model, and the achieved accuracy, and the accuracy above 97.5% was achieved only for late drought detection while for the early stress it was much lower.
Our research was carried out on the example of wheat plants (Triticum aestivum L.), a variety of Zlata. After soaking, the seeds were planted in pots filled with soil. The plants were grown up to the age of 14 days, after which they were divided into control and experimental ones. In the experimental group, watering was stopped, in the control group it was continued according to the standard schedule. Thus, the diference in both dry weight and temperature accumulated between the control and experimental groups. The experiment lasted 25 days, during which the diference in temperature and weight increased between the control and experimental samples. The photos were recorded every few days so that the diference in features was noticeable. The images were extracted using three devices: the Nikon D5100 RGB SLR camera (Nikon Corporation, Japan); thermal imager Testo 885-2 (Testo SE & Co, KGaA, Germany), with a measuring range from -30°C to 100°C; Specim IQ hyperspectral camera: range: 400-1000 nm; spectral resolution: 7 nm; channels: 204; 512 x 512 pixels. For the study, wheat plants were placed in three boxes, 30 pots in each. Thus, 45 pots were placed for the experimental group, and the same number for the control group. The pots inside the boxes were placed in such a way that the control and experimental groups of plants were located from diferent sides. For each box, 11 images of each type were obtained at an angle of 90° (top view). Such images correspond to the acquisition of images by sensors installed on the UAV.
The images were taken from a distance of 100 cm between the sensor and the object. For the RGB camera, the images were formed in the "JPG" format with a size of 5184x3456 pixels. Images in the "BMP" format of 320x240 pixels were obtained for the TIR sensor. The HSI sensor (Specim IQ camera) recorded images using the " dat " format and with a header file of the "HDR" format, having a size of 512x512 pixels and with the presence of 204 image channels in the range from 397nm to 1003nm. Figure 1 shows an example of images obtained from an RGB sensor.
On the days of image fixation, the temperature of the leaves and the diference in water content were measured for control and experimental plants. The graphs in Fig. 2 show that a noticeable change in the water content in the plant appears only on the 11th day of the drought. But according to the leaf temperature obtained using the TIR image, it is possible to notice the beginning of the stress state of the plant on the 5th day.
The values of each hyperspectral image should be reduced to the reflection values. When shooting HSI, the Specim IQ hyperspectral camera receives a three-dimensional array of hyperspectral image, as well as a white standard and a dark frame, also presented as three-dimensional arrays. This data is used to convert the original HSI data into reflection coeficient values. These values are necessary in order to compare the data measured in diferent environments. Formula 1 presents the transformation of the reflection coeficient values, where "Reflectance" is the reflection coeficient values, "data" is the raw HSI values, "Dark" is the values of the dark frame, "White" is the values of the white standard, and "1" and "2" are additional coeficients. The Specim IQ hyperspectral camera automatically calculates the values of the reflection coeficient, and saves the result. Thus, the HSI obtained with the help of this hyperspectral camera is a three-dimensional data array ready for subsequent processing.
= 1 − 1 2 ℎ2 − 2 * 1 (1)
However, the resulting HSI may contain noise and glare pixels, so the image requires additional processing. Thus, the image was processed using the Savitzky-Golay filter [ 19]. The SavitzkyGolay filter removes high-frequency noise from the data, preserving the original shape and features of the signal better than other types of filtering methods, thanks to the polynomial smoothing of the result. This filter with window width = 7 is applied to the signature of each HSI pixel using a smoothing polynomial of the 3rd degree.
HSI after noise removal is used to calculate the spectral vegetation index NDVI by the formula (2), where is the reflection value in the - nanometer HSI channel.
= 800 − 680 (2)
800 + 680
Figure 3 shows a NDVI image in pseudo-color before noise removal (using the Savitzky-Golay iflter), and after removal.
In the pre-processed HSI, the area of interest is highlighted, for which 2 background masks are built. The first mask excluded everything that is not a plant and was constructed using the NDVI index (formula 2) according to the rule: pixels with a value above the specified threshold belong to the plant, and the remaining pixels belong to the background. The second mask is based on the spectral index given by the formula (3).
= 450 − 550 (3)
450 + 550
She was supposed to add soil to the area of interest in plants. This could also be done by choosing a threshold, since the background was specially draped with a dark fabric. The separation of soil and plants was provided by a combination of masks shown in (Fig. 4).
The first task of the markup is to bind the temperature data of the TIR images to NDVI and HSI images using 1 mask. As a result of combining TIR and HSI, the temperature marking of wheat plants on RGB, HSI and NDVI images was obtained (Fig. 5), which allows to qualitatively increase the accuracy of the selection of the drying part of the plant and the prediction of the day since the beginning of the drought (Fig. 2). The most informative marking is the temperature of the HSI signatures.
Within the area of interest specified by mask 2, the k-means [ 20] method, marking of 4 classes (clusters) necessary for diagnostics was performed on all HSI channels: wet and dry soil, wet and drying wheat (Fig. 6). The scikit-learn software tool was used. This clustering problem can also be solved on NDVI images. However, using HSI signatures instead of NDVI values allows you to build the most correct class separation. To predict the day of drought, a DLP-regressor model is further constructed.
To solve the problem of predicting the day of drought, a fully connected neural network of a double layer perceptron was used — DLP regressor. The HSI pixel signatures containing 204 channels are fed to the input layer of the regressor, and the output layer contains 35 neurons corresponding to the 34th temperature values (calculated with an accuracy of 0.1 degrees) and the background (Fig. 7). An image of the 25th day of the experiment was used to train the network (Fig. 6), since drought is observed especially clearly on it due to the maximum temperature diference between control and experimental plants. 50% of randomly selected signatures of this image were used as a training sample.
The training was conducted on an Intel Core i3-8130U processor with a frequency of 2.20 GHz, 4 cores, 4 GB. The training time of the model was 5 hours and lasted 100 epochs. The accuracy of the temperature prediction was 56.1%, and the RMSE was 0.52 degrees. The result of the temperature prediction is shown in (Fig. 8), and the error matrix of this prediction is shown in (Fig. 9). Using a graph of temperature changes over the days of the experiment, it is possible to estimate the error of early prediction of a drought day by a value of about 2 days, which will not cause crop losses due to late detection of the onset of drought.
The predictive ability of the trained DLP-regressor was tested on the HSI of all days of the experiment. Figure 10 shows the temperature labels for HSI on the 1st, 6th, 8th, 12th, 19th and 25th days of the experiment predicted by the network. According to the values of the labels, starting from the 6th day, the dynamics of temperature increase in the experimental group is traced, which corresponds to the defeat of drought, which is easily reversible after watering until the 10th day inclusive.
To solve the classification problem, a model similar to the DLP regressor was used. The diference between this model is only in the output layer containing 5 neurons corresponding to 5 classes. To train the network, 2 groups of 3 marked HSI, the start day and the end day of the experiment were selected, each of which included 15 pots with control plants and 15 with experimental plants. As a training sample, 25% of the HSI signatures of the 25th day of the experiment (the 25th day in Fig. 12), marked up into 5 classes, as well as 25% of the HSI signatures of the 1st day of the experiment (the 1st day in Fig. 12), marked up into 3 classes (background, wet wheat, wet soil) - it was assumed that the plant on the 1st day without watering is not subject to drought. The training took place on the same equipment, lasted 1 hour and lasted 15 epochs. The result of training the networks were tested on 25% of the HSI signatures of the 1st day and 25% of the HSI signatures of the 25th day, which were not used for training. The accuracy of the network operation was 97.3% (Fig. 11).
After using diferent parts of the images of the 1st (containing "wet" classes) and the 25th (containing "drying up" classes) days for DLP training, the network was able to predict the necessary labels on other HSI. The results of DLP-classifier, which was trained on HSI of one box, after using on the other 2 boxes from 25th-experiment-day are shown on (Fig.12). On (Fig. 13) are shown the DLP-classifier results for the diferent experiment days from first to 25th.
The construction possibility of explainable artificial intelligence (XAI) models for the early diagnosis of plants drought based on hyperspectral images (HSI) data are investigated. The problem was solving on the basis of data from an experiment to observe the drought stress of wheat plants during for 25 days, with an interval of 2-3 days. The experiment volume included of 3 boxes with 30 pots with plants in each: 15 on the left were watered; 15 on the right were not watered. The states of plants were recorded using HSI and Thermal IR (TIR) cameras. During experiment the dependencies of mean temperature of plants (via TIR images) and water losses (%, via plants weighting) from experiment-day were constructed. By these dependencies the increase of mean plant temperature on 0.2 deg on 5th day, and water losses about 8% on 11th day were fixed, which are the earliest drought evidence by criteria of plant temperature and water losses. To give the needed explainability to the result of AI-methods applying, the HSI signatures were trained on temperature values from the Thermal IR (TIR) images. It has done on the last-experiment-day data, for which the NDVI-based masks of plants, soil, and background were automatically constructed. Such trained HSI-TIR fusion put as the basis for plant temperature prediction. The regressor and classifier in this work were constructed as double-layer perceptrons. DLP-regressor and DLP-classifier have the HSI-channels as input nodes. The output DLP-regressor layer have 35 neurons to provide the temperature resolution equals 0.1 deg. After training achieved RMSE value equals 0.52 deg which provide the dayprediction error about 2 days and prediction drought state of plant before water losses. For the classification task, in the masks limits, the decomposition of plants and soil onto two classes: wet and drying up, has executed. It was done via their clustering by the k-means method, using diferent types of images: NDVI and HSI. HSI-based clustering was evaluated as more accurate after its testing on the other experiment days. The k-means method was applied on the 1st-day-HSI for 3 clusters, including background (without drying up), and the 25th-day-HSI for 5 clusters. The DLP-classifier had 5 output and its accuracy for early drought detection was 97.3%. It is noticeably more than in [18]. The constructed on HSI input data DLP-regressor and DLP-classifier, can be considered as XAI networks due to they have the property of explainability of their results provided by temperature marking of signatures, and will be useful for precision agriculture.
The work was supported by the Ministry of Science and Higher Education of the Russian Federation, agreement No 075-15-2020-808. [8] C.-I. Chang, A. Plaza, A fast iterative algorithm for implementation of pixel purity index, IEEE Geoscience and Remote Sensing Letters 3 (2006) 63–67. doi:10.1109/LGRS.2005. 856701. [9] N. Zhang, Y. Guijun, Y. Pan, Y. Xiaodong, L. Chen, C. Zhao, A review of advanced technologies and development for hyperspectral-based plant disease detection in the past three decades, Remote Sensing 12 (2020) 3188. doi:10.3390/rs12193188. [10] K. Nagasubramanian, S. Jones, A. K. Singh, A. Singh, B. Ganapathysubramanian, S. Sarkar, Explaining hyperspectral imaging based plant disease identification: 3d cnn and saliency maps, in: 31st Conference on Neural Information Processing Systems (NIPS 2017), Curran Associates Inc., Long Beach, CA, USA, 2018. [11] C. Römer, K. Bürling, M. Hunsche, T. Rumpf, G. J. Noga, L. Plümer, Robust fitting of lfuorescence spectra for pre-symptomatic wheat leaf rust detection with support vector machines, Computers and Electronics in Agriculture 79 (2011) 180–188. doi:10.1016/j. compag.2011.09.011. [12] J. Behmann, J. Steinrücken, L. Plümer, Detection of early plant stress responses in hyperspectral images, ISPRS Journal of Photogrammetry and Remote Sensing 93 (2014) 98–111. doi:10.1016/j.isprsjprs.2014.03.016. [13] P. Schmitter, J. Steinrücken, C. Römer, A. Ballvora, J. Léon, U. Rascher, L. Plümer, Unsupervised domain adaptation for early detection of drought stress in hyperspectral images, ISPRS Journal of Photogrammetry and Remote Sensing 131 (2017) 65–76. doi:10.1016/j.isprsjprs.2017.07.003. [14] M. Shahrimie, M. Asaari, Detection of plant responses to drought using close-range hyperspectral imaging in a high-throughput phenotyping platform, in: 9th Workshop on Hyperspectral Image and Signal Processing : Evolution in Remote, IEEE, Amsterdam, The Netherlands, 2018. [15] A.-K. Mahlein, M. T. Kuska, S. Thomas, D. Bohnenkamp, E. Alisaac, J. Behmann, M. Wahabzada, K. Kersting, Plant disease detection by hyperspectral imaging: from the lab to the field, Advanced in Animal Biosciences 8 (2017) 238–243. doi: 10.1017/ S2040470017001248. [16] I. Maximova, E. Vasiliev, A. Getmanskaya, D. Kior, V. Sukhov, V. Vodeneev, V. Turlapov, Study of xai-capabilities for early diagnosis of plant drought, in: IJCNN 2021 : International Joint Conference on Neural Networks, INNS, IEEE, Shenzhen, China, 2021. [17] R. Hernández-Clemente, A. Hornero, M. Mottus, J. Penuelas, V. González-Dugo, J. C.
Jiménez, L. Suárez, Early diagnosis of vegetation health from high-resolution hyperspectral and thermal imagery: Lessons learned from empirical relationships and radiative transfer modelling, Current Forestry Reports 5 (2019) 169–183. doi:10.1007/ s40725-019-00096-1. [18] P. D. Dao, Y. He, C. Proctor, Plant drought impact detection using ultra-high spatial resolution hyperspectral images and machine learning, International Journal of Applied Earth Observation and Geoinformation 102 (2021). doi:10.1016/j.jag.2021.102364. [19] C. Rufin, R. L. King, The analysis of hyperspectral data using savitzky-golay filteringtheoretical basis. 1, in: Geoscience and Remote Sensing Symposium, 1999. IGARSS ’99 Proceedings. IEEE 1999 International, IEEE, Hamburg, Germany, 1999, pp. 756–758. doi:10.1109/IGARSS.1999.774430. [20] J. MacQueen, Some methods for classification and analysis of multivariate observations, in: In Proc. 5th Berkeley Symp. on Math. Statistics and Probability, California University Press, Berkeley, Los Angeles, 1967, pp. 281–297.