The impressive results of deep learning in many different fields, specifically in remote sensing, together with the growing availability of open Earth Observation data creates new opportunities to address global problems. One such global problem is associated with the simplification and intensification of agricultural systems which threatens the worldwide sustainability of crop production. Despite the fact that a plethora of satellite images describe a given location on earth every year, very few deep learning-based solutions have harnessed the temporal and sequential dynamics of land use to map sustainable and unsustainable cropping practices. In this paper, we present the preliminary results of a set of experiments conducted using one-dimensional Convolutional Neural Networks (CNN) for classifying multispectral time series derived from Landsat satellites constellation. The experimental data is related to agricultural practices in Sacramento County, California, United States of America. We discuss the applicability of this approach for mapping sustainable crop rotationbased practices which have been proven to mitigate the environmental impact of agricultural land use dynamics.
Confronted with important global problems related to agriculture sustainability, food
security, climate change, and biodiversity loss, new ecological movements across the
world are promoting a set of “ecological intensification” principles, as an alternative
paradigm to mainstream agricultural practices [
Recent applications of deep learning in many fields, including remote sensing,
together with the increasing availability of free satellite images with higher spectral,
spatial and temporal resolutions, creates new opportunities to tackle global
challenges. Deep learning-based models have the ability to learn feature representations
exclusively from raw data without the need for domain-specific knowledge. This fact,
together with the advances in computational power, has encouraged the use of deep
neural networks for many tasks, including image classification, object detection,
semantic segmentation and anomaly detection [
In this work, we present a set of experiments using Convolutional Neural Networks (CNNs) with convolutions in the temporal dimension and more than 400,000 remotely sensed time series data to classify land use and agricultural practices. 2
Zhong, Liheng et al. [
The setting of the study is a surface of 4466 km2 or 1724 square miles in Sacramento
County, in the west part of the United States of America and encompasses a one-year
period ranging from January 2015 to December 2015. Fig. 1 shows the delimited
surface. The annual mean temperature in this area is 16.1 °C with a monthly daily
average temperature ranging from 8.0 °C in December and 24.2 °C in July. The wet
season period extends from October to April. The region of Sacramento has a strong
agricultural tradition and remains an important economic force not only in California
but also at a national level [
Fig. 1. Study Area, Sacramento County, United States of America. The red thick line delimits the region of interest for this study and the green polygons represent the agricultural fields our experiments and analysis will be based on. 3.2
In this study, the 2015 Sacramento County land use survey was used as a source of “ground truth” data. The Survey was developed by the State of California, Department of Water Resources (DWR). The main goal of this survey is to map agricultural fields. In this regard, a surveyor visited almost every delineated field, providing a high land use assessment accuracy.
The dataset is distributed in a Shapefile format and consists of a total of 40205 polygons, each of them containing 29 different attributes that describe the agricultural land use practices at the field level. 3.3
Different satellite constellations provide freely distributed images of the world surface
in a continues manner and at different spatial, spectral and time resolutions. For
instance, the two Sentinel-2 satellites provide 10m resolution imagery of the planet
surface every five days, whereas a combination of Landsat-7 and Landsat-8 satellites
offer an eight days revisit time with a 30m spatial resolution [
Among the available Landsat products, Landsat Level 2 is a research-quality,
application-ready science product derived from Landsat Level 1 data [
A total of 178 Surface Reflectance image products were downloaded for the region of interest delimited with red in Fig. 1 for the year 2015. From this set, 88 images correspond to Landsat-8 and a total of 90 images correspond to Landsat-7. Three bands were selected for the application of this study. The green band emphasizes peak vegetation, which is useful for assessing plant vigor. Red Band discriminates vegetation slopes while Near Infrared (NIR) emphasizes biomass 2. 1 summarizes the bands' information per satellite.
As we can see from Table 1, the spectral ranges of the different bands are slightly
different between Landsat-7 and Landsat-8. These differences have been studied in
[
In order to create pixel-level labeled time-series we followed the pipeline we proposed in a previous paper. In that contribution, we created a set of tools leveraging the open-source Orfeo ToolBox (OTB)3 tool and we presented an end to end pipeline that 1 https://earthexplorer.usgs.gov/ 2 https://www.usgs.gov/faqs/what-are-best-landsat-spectral-bands-use-my-research?qtnews_science_products=7#qt-news_science_products 3 https://www.orfeo-toolbox.org can consume a collection of satellite images and a ground in-situ shapefile dataset to create labeled, temporal-sampled and linearly interpolated time series at the pixel level. The code assets created in that study where made available for others to reuse and can be found in https://github.com/agustingp/remoteSensingTimeSeries. 4
During this study, we designed two experiments to assess the performance of CNNs for classifying land-use practices that can inform decision making to achieve a more sustainable agriculture. Below, we explain the aim of each experiment and describe the process for creating the labeled time-series dataset. 4.1
The first experiment was designed aiming at classifying two agricultural practices:
single cropping and double cropping, within the same year. Double cropping practice
is an important sustainable practice that aims at reducing the fallow periods of the
land, exploiting solar energy to enhance the quality of the soil and preventing soil
erosion [
For the experiment, we first filtered the ground truth data removing classes that were not representing agricultural fields. Then we identified the fields where double cropping practice and single cropping practice took place during the year. Table 2 presents the total amount of polygons and the total amount of pixels sampled. For each pixel, a labeled time series was created using the three spectral bands from the satellite images. This process is explained in Section 3.4. In this case, as the doublecropping class was under-represented, we selected the total amount of double cropping pixels as the limit to be sampled from the single cropping polygons. The sampling was done randomly, maximizing the diversity of single cropping polygons and not exceeding the limit of 29596 pixels. In the end, the total number of pixels sampled was 29596 for both classes.
Class name Double cropping Single cropping
Total
The second experiment consists of the classification of 20 different agricultural land
use classes. During this practical experience, we focused on the classification of
different crops that were grown in a “Single Cropping” approach. The rotation of crops
across different years often leads to better yields due to soil fertility improvements
[
For the experiment, we first filtered the ground truth data removing all the classes that did not represent an agricultural field. Following this, we removed the doublecropping practice ones, to focus on the fields where only one crop was grown during the year. From a list of 47 different crops (also including agricultural classes such as “fallow”), we selected the 20 classes that were best represented in terms of the number of pixels available. However, class imbalances are present in our dataset with the highest number of pixels for “Mixed Pasture” class, and the lowest number of pixels for “Grain Sorghum” class. Table 3 presents the total amount of polygons and the total amount of pixels sampled. Finally, we followed the same criteria for time series creation as for Experiment 1.
Class name Eucalyptus Walnuts
Pears Almonds Cherries Safflower
Corn Grain sorghum
Sudan Beans Hay Rest Alfalfa
Clover Mixed Pasture
Melons
Potatoes
Tomatoes
Flowers
In this study, we employed the CNN architecture proposed by Pelletier et al. [
221 4154 2758 360285 The models for both experiments were trained using the Azure cloud infrastructure provided by Microsoft AI for Earth grant program. The virtual machine uses an NVIDIA Tesla K80 GPU card. Each dataset created for Experiment 1 and Experiment 2, respectively, was split in two, 80% for the training set and 20% for the testing set. The training set was also split in runtime to separate some data for validation. We used a validation rate of 0.05, which means that 5% of the training set was used to validate the model performance during training. During each partitioning step, the
division of data was done at the polygon level, meaning that no pixels from the same polygon are in the training, testing or validation set at the same time.
In this section, we present the results of the data preprocessing pipeline as well as the trained models' performance. 5.1
After the preprocessing process described in Section 3.4, we obtained 419,477 pixels’ time-series. A sample of this time-series was plotted for both experiments. Fig. 3 presents the temporal and multispectral timeseries data for two different pixels, the left-hand side one corresponds to the “Single cropping” class, whereas the right-hand side corresponds to the “Double cropping” class. Fig. 4 presents this information for other two pixels classes, “Corn” and “Alfalfa”, respectively.
The overall classification accuracy for Experiment 1 was 88% and Experiment 2 was 89%. To achieve an extra performance verification of the model developed in Experiment 1, we used it to classify the entire dataset from Experiment 2. As we explained before, Experiment 2 dataset contains only pixels that belongs to single cropping practice. The results of the classification show 97% for this dataset. The highly accurate results can be explained because the model in Experiment 1 was trained with a sampling of pixels that maximized the diversity of polygons represented in the dataset. Then, most of the polygons in Experiment 2 where sampled in the dataset of Experiment 1. These results show that even the use of a small number of pixels coming from the same polygon are representative enough for the network to learn the specific time-patterns of the polygon.
In Fig. 5 and Fig. 6 we present the recall, precision and F-score for each experiment
classes.
The paucity of up-to-date ground truth data presents a problem for utilizing newer
satellite imagery with a higher spatial, spectral and temporal resolution for training
supervised AI models, as it is the European Sentinel constellation. On the other hand,
initiatives like the Harmonized Landsat and Sentinel-2 [
In this paper, we presented a novel approach to processing satellite imagery validated in a set of two distinct experiments using Convolutional Neural Networks with convolutions in the temporal dimension. Most of the recent AI models or classifiers used in operational mapping use single date spectral data for classification and do not harness the temporal resolution of remotely sensed time-series images. Therefore, we argue that our solution provides an important contribution to the domain. Our evaluation showed promising results with our models achieving 88% and 89% accuracy for experiments 1 and 2 respectively. Our future work will focus on improving these results, covering more classes and with a more general architecture.