Accurate crop type identification from remote sensing data in tropical regions is a challenging task. The more favorable weather conditions permit more flexible agricultural practices, which creates complex crop dynamics. Synthetic Aperture Radar (SAR) systems provide high-resolution images independent of daylight, cloud coverage, and climate conditions, turning into a cost-efective tool for crop mapping in tropical regions. Conditional random fields (CRFs) models have been used with great success to exploit spatial and temporal contexts in crop type classification. Even if such approaches deliver high accuracy, they often rely on manually designed features requiring domain-specific knowledge. In this context, deep learning methods such as convolutional neural networks (CNNs) proved to be an appealing alternative for feature learning in the context of remote sensing image classification. This work introduces an end-to-end model that combines CNNs and CRFs for crop recognition in areas characterized by complex spatio-temporal dynamics, typical of tropical regions. The proposed framework consists of two modules: the first implements a CNN that models spatial and temporal contexts from the input data, and the second implements a CRF module that models temporal dynamics considering label dependencies. Experiments are presented for an agricultural region in Brazil using multi-temporal SAR images sequences. The experiments showed significant improvements in the F1 score against a baseline model that does not include temporal dependencies in the learning process.
discrimination.
Probabilistic Graphical Models, such as Hidden The expected increase in human population from 7.7 Markov Models (HMM) and Conditional Random Fields billion in 2019 to 8.5 billion in 2030 [1], coupled with the (CRFs), have been successfully used to exploit both predicted worldwide growth of per capita income, spatial and temporal contexts in crop type classification pressures the demand for food in the future [2, 3] and [5, 6, 7]. These approaches deliver high accuracies but reinforces the need for eficient and sustainable often rely on feature engineering and require expert agricultural policies that ensure food security. In this knowledge. context, accurate estimation of crop area extents is Alternatively, convolutional neural networks (CNNs) indispensable for farm monitoring and yield prediction. proved to be a robust option for remote sensing image The Synthetic Aperture Radar (SAR) is a crucial classification, as they can learn optimal features directly cost-efective tool that generates high-resolution from raw data. In [8], the authors used CNNs for imagery independent from daylight, cloud coverage, and multi-date crop recognition considering a SAR climate conditions. multi-temporal image sequence in a tropical region and
In temperate regions, agricultural practices are applied a post-processing algorithm based on an HMM. characterized by a single crop type per parcel during the The so-called Most-Likely Class Sequence (MLCS) productive season, simplifying crop dynamics analysis. enforces prior knowledge about crop occurrence over Conversely, in tropical regions, crop dynamics are more time in the target region. 3D CNN have been also complex due to the multiple harvests per year, crop employed for crop mapping from multi-temporal remote rotation, and other agricultural practices [4]. The sensing images considering the temporal and spatial diverse crop dynamics observed in tropical regions often context [9]. In [10], the authors proposed a bidirectional require multitemporal approaches for eficient crop convolutional recurrent neural network (ConvLSTM) that takes into account the spatio-temporal context and CDCEO 2021: 1st Workshop on Complex Data Challenges in Earth delivers a prediction for each date of interest. The Observation, November 1, 2021, Virtual Event, QLD, Australia. authors also applied the MLCS algorithm that further $ lauracue@aluno.puc-rio.br (L. E. C. L. Rosa); improved the per-date accuracy by obtaining the (dRa.rQio.bFoe@it©ob2sr0.a2i1b)Cmop.ycroigmht fo(rDth.isApa.pBer.byOitlsiavuethiorras.)U;sreapuerml@itteedluen.dperuCcr-eartiivoe.br cmoonsstt-rlaikinetlsy. Becslaidsess ascehqiueveinncgescombpaseetidtiveopnerftoermmpaonrcael, CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmmUoRns LWiceonsrekAstthribouptionP4r.0oIncteerenadtiionnagl s(CC(CBYE4U.0)R.-WS.org) the ConvLSTM classifier does not consider these 1 2
T x ∈ × × ×
CNN agdn n o i t c a if s s a l c n i n r a e l e r u t a e f (x, y1; ) (x, y2; ) (x, y ; )
Tr[1] Tr[2] Tr[ ] (y|x)
Tr[1] Tr[2] constraints during training. bands, respectively.
Recently, CNNs and RNNs have been combined with CRFs in an end-to-end framework to model the intra-class 2.1. 3D CNN module temporal progression. These models were first proposed for sequence tagging for natural language processing The sequence of images is fed to the CNN module that and then extended for video action segmentation [11, 12]. learns spatio-temporal features from the input sequence However, they are yet to be explored for crop mapping and produces a score for each label at each time step. from multi-temporal remote sensing images sequences We denote the output of the CNN at each time-step as to the best of our knowledge. This work seizes that (x, y; ), where is the network parameters. These opportunity and proposes a novel method that combines are the emission scores (also known as unaries) that deep learning and graphical models in an end-to-end serve as input to the CRF module and are related to framework from crop mapping in tropical regions from the input sequence’s posterior probability at time-step multi-temporal image sequences. . The CNN module can consist of any CNN capable of modeling spatial and temporal context, such as 3D CNN and Convolutional LSTM networks that have been used 2. METHOD with success for crop mapping [9, 10]. Our work presents a method for making dynamic decisions for multi-date crop recognition combining linear-chain CRFs and CNNs in an end-to-end formulation. The proposed framework named CNN-CRF is presented in Figure 1 and consists of three modules: a CNN module, a linear-chain CRF module, and Viterbi decoding. It takes as inputs a multi-temporal remote sensing image sequence x ∈ × × × where each image at each time-step covers the same region; , and represent the height and width, respectively; and and are the length of the sequence and the number of
Using the unaries from the CNN module, we applied a sequential CRF to decode labels for the whole multi-temporal input sequence jointly. The linear-chain CRF uses features that depend on the input, i.e. the predicted unaries, and on a transition matrix that models the temporal relation among pairs of adjacent labels. We define the transition matrix Tr with size ( − 1 × | | × | |), where each Tr is the transition score matrix considering the two adjacent epochs − 1 and . For each Tr, elements in row and column relates to the probability of the input sequence being the -th crop type at time-step , considering the -th crop type in the previous time-step − probabilistic model defines the family of conditional 1.
The CRF probability (y|x) as (y|x) = , where (x) is the partition function (a normalization constant) that ensures the normalization between 0 and 1. We proposed to train the CNN and CRFs modules end-to-end by minimizing the negative log-likelihood (NLL), which is equivalent to maximizing our data’s logsequences, the NLL loss function reads as: likelihood. Given a set of training samples pair A = { x, y}=1 : y ∈ , where is the set of all label ℒ(, Tr) = − ∑︁ log (y|x).
We explored two variants for learning the transitions scores automatically from the training data. We first tested a variant that has been widely used in natural language processing [11], that we called CNN-CRFG hereafter. CNN-CRFG assumes that the transition matrix is shared over time, i.e. a global matrix whose transitions scores are independent of the time-step. Hence, the
CNN-CRFG model considers Tr[1] = Tr[2] = ... = Tr[ ], and the transition scores are estimated directly from training data.
Despite the success of similar models for sequence tagging in natural language problems, learning a global transition
matrix can be too restrictive for crop phenology changes, principally in tropical regions where crop transitions are subject to agricultural practices that may vary according to the productive season. In this sense, we propose a second variant called CNN-CRFA, that learns a transition matrix for each pair of adjacent epochs conditioned to the observed transitions in the training set.
During inference, the objective is finding the most likely sequence given an unseen input sequence, i.e., the maximum scoring sequence according to our model.
That corresponds to solving the following equation: yˆ = require computing the probability for all possible
arg maxy∈ (y|x). The naive solution would we can solve the above equation more eficiently using the Viterbi algorithm [13]. sequences, which is computationally infeasible. Instead, and vertical flips during training. 0 Oct/15 Nov/15 Dec/15 Jan/16 Feb/16 Mar/16 May/16 Jun/16 Jul/16
Images Soybean
Maize
Cotton
Sorghum
Beans
NCC Pasture
Eucalyptus
Soil
Turf grass
Cerrado
We evaluate the proposed models in Campo Verde, an agricultural region located in Brazil. Campo Verde [4] is a ublic dataset that provides the land-use classes by month (between October 2015 and July 2016) and 14 pre-processed
SAR
Sentinel-1 with
VV and
VH polarization. The major crops found in the region are soybean, maize and cotton. Others minor crops are beans and sorghum.
The rest of the land-use classes are composed by Non-Commercial Crops (NCC), pasture, eucalyptus, turfgrass, cerrado and soil. The agricultural practices in the region consist of two seeding periods for the major crops, soybeans span from October to February, and maize and cotton from Mars to July. Figure 2 shows how the area is distributed among diferent crops along the months which allow to observed the complex crops’ dynamics characteristic of the region. The reference data consisted of 608 crop fields split into two disjoint sets, 50% of each class selected for training and the other 50% for testing, using stratified random sampling.
The CNN-CRF model takes as input an image patch and computes the class score for the central pixel of the patch. From the SAR images sequences, in each epoch, 15 ×
15 × we randomly cropped 50, 000 image patches of 14 ×
2 on the fly to train the network. To alleviate the class imbalance problem, we replicated samples from minority classes and employed rotations and horizontal
Our work primarily focused on how the inclusion of a
CRF module afects the classification performance; classification decision at each time-step is conditionally therefore, we do not evaluate various architectures for independent of its neighbors in such schema. We trained the CNN module and experiment with a simple 3D-CNN all models with Adam optimizer for 50 epochs with 64 only. As fully-connected layers often contain large image patches per batch. numbers of parameters, we only employed convolutional layers. The network consists of three 3.3. Accuracy Assessment processing blocks that comprehend two successive 3D convolutions + batch normalization + LeakyReLU The performance of the evaluated methods was operations, followed by an average pooling that reduces expressed in terms of Overall Accuracy (OA), producer’s the spatial dimension by 2. The 3D convolutions accuracy (PA), user’s accuracy (UA), and F1 score (F1). consider a spatial context of 3 × 3 and a temporal The OA represents the proportion of correctly classified context of 5 epochs. Our dataset has several images per samples; hence it is a global metric that depends on epoch/month, but we are interested in a unique larger classes. The PA (also known as recall) represents prediction for each month. Hence we implemented the the probability that a particular class on the reference is last processing block to map the temporal dimension to correctly classified. The UA (also known as precision) is the desired length. Finally, a last convolutional layer the probability that a pixel classified into a given class with linear activation with || kernels of size 1 × 1 × 1 actually represents that class on the reference. Finally, delivers the emission scores. The second term of the CRF the F1 score is the harmonic mean of UA and PA. The F1 model is implemented by defining the transition matrix score is a more suitable metric in scenarios where class of size ( − 1) × | | × | |. That matrix is estimated distribution is uneven, as in our dataset. during training from the training data. At inference time, the high dimensional remote sensing input images 3.4. Results were cropped in densely overlapping images patches and the final crop map was constructed by predicting Figure 3 summarizes the per-month results obtained for each patch associated with each image coordinate. As a CNN-CRFG and CNN-CRFA, in terms of OA, average F1 baseline, we trained a 3D-CNN that delivers per epoch score (avgF1), average producer’s accuracy (avgPA) and class probability, called CNN hereafter. The architecture average user’s accuracy (avgUA), for a sequence is the same as the CNN-CRF variants without the comprising the 14 SAR images from October 2015 to July transition matrix. We trained the model using a per-date 2016. The horizontal axis contains the month being categorical cross-entropy loss function. Note that the classified. In this figure, we report just one result per t c O v o N v o N c e D n a J b e F r a M n u J l u J
Table 2 misclassify pasture as others classes such as soybeans F1 score per class for July (best values are highlighted in bold). and soil in the sequence. These changes are inconsistent with the temporal dynamic observed in the region. In
CNN CNN-CRFG CNN-CRFA contrast, the CNN-CRF variants correctly identify the maize 46.3 50.0 48.9 class pasture for almost all pixels within this field along cotton 89.1 89.9 89.9 the whole sequence. In addition we observed that for sorghum 61.4 52.6 62.3 some fields all models erroneously classify pasture as NCC 35.0 32.2 47.9 cerrado. These errors were more frequently observed for epuacsatulyreptus 9764..93 9751..23 7983..99 the CNN-CRFG model. soil 76.4 75.4 74.4 Our results indicate that learning a transition matrix turfgrass 74.0 71.9 85.5 for each pair of adjacent months enables training a more cerrado 73.0 67.2 74.2 robust model for modeling the complex crop dynamics observed in our tested region. Our results also revealed Average 69.6 67.3 72.9 that jointly decoding a chain of labels ensures temporal consistency and improves the per-month classification.
Finally, we observed that the inclusion of the temporal context for adjacent dates brings more benefits for less abundant classes, in which any error in classification may cause a significant variation in the analyzed metrics. month, considering the nine annotated months. In addition, the figure presents the results for the baseline model where we report the performance for the CNN output (first bar in the figure).
The results revealed that CNN-CRFA consistently outperformed CNN-CRFG method in terms of OA and 4. Conclusions and future work avgUA. In contrast, CNN-CRFG outperforms CNN-CRFA in terms of avgPA. Nonetheless, looking at the F1 score, This work introduced a hybrid deep learning CNN-CRFA overcomes CNN-CRFG in 5 out of the 9 architecture for multi-temporal crop recognition, which months by a high margin, and for the other 4 months, combines the spatio-temporal context encoding of 3D we observed a drop in performance by a shallow margin. CNNs and the temporal modeling capabilities of CRFs, We also observed that CNN-CRFA model significantly in an end-to-end framework. In contrast to existing outperforms the baseline model in 8 out of the 9 months similar approaches, which learn a global transition considering the F1 score, achieving up to 4.4% of matrix that models the temporal dynamics, our method improvement. Similar behavior was observed for avgUA, proposes to learn diferent transitions matrices, one for with CNN-CRFA reporting the best results for all months. each pair of adjacent dates. We tested the models using CNN-CRFG reported higher values in terms of PA. In a publicly available multi-temporal SAR dataset from a contrast, CNN-CRFG presented worse results compared tropical region with highly complex spatio-temporal to CNN model in terms of OA for almost all months. crop dynamics.
Considering F1 score, CNN-CRFG also presented low The experiments indicated that the proposed end-toperformance in 4 months compared to CNN. end framework consistently outperformed a baseline
Table 2 reports the F1 score for all crop types for July, model that disregards temporal modeling. As further when the higher drop in performance was observed for research, we intend to test an approach that penalizes CNN-CRFG model compared with the baseline. Notice the transitions that may not occur according to the expert that CNN-CRFG reported a significant drop for sorghum knowledge in the target region. In addition, further and cerrado. Figure 2 shows that these are two minority detailed studies will be conducted to assess the influence classes with very diferent crop’s dynamics. That result of the temporal context modeled by the CNN and the CRF indicates that global transitions could potentially favor modules in the classification accuracy. In a subsequent more abundant classes. In contrast, CNN-CRFA had step, we will test other CNN architectures (e.g., Fully higher robustness, achieving better results for 7 out of Convolutional LSTM networks) for learning the emission the 9 classes when compared with the baseline method. scores.
Figure 4 presents the classification maps for each method and month in a selected area. Notice that for all Acknowledgments methods, the prediction maps were more accurate for those classes with a higher number of samples: soil for October, November, and March; soybeans from December to February; and maize from May to June.
Looking at the field located on top-right in the selected area, we observed that the CNN method frequently We gratefully acknowledge the financial support ofered by the Brazilian National Council for Scientific and Technological Development (CNPq) and the Foundation for Support of Research and Innovation, Rio de Janeiro State (FAPERJ). Journal of Photogrammetry and Remote Sensing 171 (2021) 188–201. [1] U. Nations, World population prospects 2019: [11] L. Liu, J. Shang, X. Ren, F. Xu, H. Gui, J. Peng, J. Han, Highlights., Department of Economic and Social Empower sequence labeling with task-aware neural Afairs, Population Division ST/ESA/SER.A/423 language model, in: Proceedings of the AAAI (2019). Conference on Artificial Intelligence, volume 32, [2] N. Ramankutty, Z. Mehrabi, K. Waha, L. Jarvis, 2018.
C. Kremen, M. Herrero, L. H. Rieseberg, Trends [12] J. Li, S. Todorovic, Set-constrained viterbi for setin global agricultural land use: implications for supervised action segmentation, in: Proceedings of environmental health and food security, Annual the IEEE/CVF Conference on Computer Vision and review of plant biology 69 (2018) 789–815. Pattern Recognition, 2020, pp. 10820–10829. [3] B. L. Bodirsky, J. P. Dietrich, E. Martinelli, [13] A. Viterbi, Error bounds for convolutional A. Stenstad, P. Pradhan, S. Gabrysch, A. Mishra, codes and an asymptotically optimum decoding I. Weindl, C. Le Mouël, S. Rolinski, et al., algorithm, IEEE transactions on Information The ongoing nutrition transition thwarts long- Theory 13 (1967) 260–269. term targets for food security, public health and environmental protection, Scientific reports 10 (2020) 1–14. [4] I. Del’Arco Sanches, R. Q. Feitosa, P. M.
Achanccaray Diaz, M. Dias Soares, A. J. Barreto Luiz, B. Schultz, L. E. Pinheiro Maurano, Campo verde database: Seeking to improve agricultural remote sensing of tropical areas, IEEE Geoscience and Remote Sensing Letters 15 (2018) 369–373.
doi:10.1109/LGRS.2017.2789120. [5] P. B. C. Leite, R. Q. Feitosa, A. R. Formaggio, G. A.
O. P. da Costa, K. Pakzad, I. D. Sanches, Hidden markov models for crop recognition in remote sensing image sequences, Pattern Recognition
Letters 32 (2011) 19–26. [6] S. Siachalou, G. Mallinis, M. Tsakiri-Strati, A hidden markov models approach for crop classification: Linking crop phenology to time series of multisensor remote sensing data, Remote Sensing 7 (2015) 3633–3650. [7] P. Achanccaray, R. Q. Feitosa, F. Rottensteiner,
I. Sanches, C. Heipke, Spatial-temporal conditional random field based model for crop recognition in tropical regions, in: 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), IEEE, 2017, pp. 3007–3010. [8] L. E. Cué La Rosa, R. Queiroz Feitosa, P. Nigri Happ,
I. Del’Arco Sanches, G. A. Ostwald Pedro da Costa, Combining deep learning and prior knowledge for crop mapping in tropical regions from multitemporal sar image sequences, Remote
Sensing 11 (2019) 2029. [9] S. Ji, C. Zhang, A. Xu, Y. Shi, Y. Duan, 3d convolutional neural networks for crop classification with multi-temporal remote sensing images, Remote Sensing 10 (2018) 75. [10] J. A. C. Martinez, L. E. C. La Rosa, R. Q. Feitosa,
I. D. Sanches, P. N. Happ, Fully convolutional recurrent networks for multidate crop recognition from multitemporal image sequences, ISPRS