Accurate forest monitoring data are essential for understanding and conserving forest ecosystems. However, the remoteness of forests and the scarcity of ground truth make it hard to identify data quality issues. We present two state-of-the-art forest monitoring datasets, Annual Forest Change (AFC) and GEDI, and highlight their data quality problems. We then introduce a novel method that leverages GEDI to identify data quality issues in AFC. We show that our approach can identify subsets with three times more errors than a random sample, thus prioritizing expert resources in validating AFC and allowing for more accurate deforestation detection.
Data-intensive models are only as good as their training
data. As a result, the past two decades have seen a great
deal of research and industry efort toward monitoring
and improving data quality. Solutions exist for
deduplication, missing data imputation, and identifying and
repairing incorrect data [
We present a novel approach to finding data errors
in one such application area: forest monitoring. Forests
have a significant impact on the Earth’s climate and
biodiversity [
ages [8, 9], leading to new data quality problems related to sensor limitations, cloud cover, and low resolution.
Bases (VLDBW’23) — the 12th International Workshop on Quality Canada ∗Corresponding author. †These authors contributed equally. task due to the limited availability of ground truth data, as collecting forest condition data through field visits is expensive and does not scale. As a result, there is no simple way of identifying errors in forest change maps.
we use GEDI [10], a recent spaceborne LiDAR dataset that provides information about the 3D structure of forests not available in optical images. Specifically, we identify nonforested areas, either deforested or non-forest vegetation, GEDI’s limited spatial and historical coverage, we show that GEDI’s estimates of canopy height (the height of the top of the forest) can identify parts of the forest change map that are three times more likely to contain errors than a random sample. Our approach can be used to prioritize resources for validating a forest change map and assist in more accurate detection of deforestation.
• We describe data quality issues associated with two datasets: a state-of-the-art forest change map (AFC) [8] (Section 2), and GEDI [10] (Section 3). • We propose and evaluate a method that leverages
The Annual Forest Change (AFC) dataset (Figure 1a) tracks annual changes in tropical moist forests (TMFs) from 1990 to 2021 [8]. It segments diferent land catewell as identifying changes in land cover, such as degranEvelop-O
• Missing Data: There could be gaps in satellite observations for several reasons, including cloud We introduce a method that directs experts’ attention cover and atmospheric conditions (Figure 1b), sen- toward a subset of samples that are more likely to contain sor limitations or failures, and intentional pauses errors. To achieve this goal, we use a new forest height
tus of TMFs. An AFC map is a 2D grid of pixels, each corresponding to a 30 m × 30 m (0.09 ha) area on the Earth’s surface at the equator. It classifies each pixel into one of six categories: Undisturbed TMF, Degraded TMF, Deforested land, TMF Regrowth, Water, and Other Land Covers.
Source AFC maps are derived from optical satellite imagery of the Landsat satellites [11] (Figure 1b). A Landsat satellite captures images of the Earth’s surface from 705 kilometers above, revisiting each location every 16 days. These images are taken by cameras onboard the satellites that capture diferent wavelengths of light including, visible light, infrared, and other wavelengths. Vegetation types are recognized by how they absorb or reflect light.
Methodology The AFC dataset is based on per-pixel classifications of Landsat images. Each pixel is classified using expert rules as either potential moist forest, potential non-forest, or invalid (cloud, shadows, noise). Each pixel is then assigned a final class (from the above six categories) based on the changes over time.
Accuracy AFC is reported to be 91.4% accurate but underestimates forest disturbance by 11.8% [8]. This corresponds to over 38 million hectares of land, which is a significant area [ 8].
Data Quality Challenges Forest maps, including AFC, face the following challenges.
in data collection. • Noisy Data: Satellite imagery is prone to sensor noise, miscalibration, and atmospheric noise. • Spatial and Temporal Resolution: The resolution of a forest map is determined by the resolution of the source data. For instance, AFC cannot tell the precise location of disruptions or changes smaller than 0.09 hectares. • Spectral Mixing: Satellite images often have mixed pixels containing diferent land cover types (e.g., half forest and half deforested). This issue occurs frequently in complex vegetation covers or at the boundaries between diferent land cover types. • Spectral Confusion: This occurs when diferent types of land cover have similar appearance when viewed from space. For instance, Figure 2 shows how a cocoa agroforest looks similar to a forest in optical satellite imagery [12]. • Lack of 3D Information: Optical satellite images lack 3D information such as forest height, limiting their ability to distinguish between some land cover types. For example, height information can distinguish forests from grasslands. • Limited Ground Truth: Collecting data by visiting forests ranges from expensive to impossible (for remote and inaccessible locations). As a result, experts rely on remote sensing to create a reference dataset. • Noisy Data: As a laser-based technology, GEDI is sensitive to atmospheric conditions, including dataset, described below, with its own unique data quality issues and challenges. 3. Forest Height Data cloud cover and moisture. Sensor noise and miscalibration also contribute to errors in the data. • Spatial and Temporal Resolution: GEDI footprints cover a limited portion of the Earth (around 4% in two years of operation) [10], and gridded maps have a relatively coarse resolution of 1 km. Additionally, operating from 2018 to 2023, GEDI does not ofer extensive historical information. Finally, there are no guaranteed revisits of the same location, making it dificult to monitor changes. • Terrain: Sloped or complex terrain introduces additional errors in GEDI data [16]. • Geolocation Error: Slight geolocation uncertainties ( 0 m mean, 10 m standard deviation exist in the reported coordinates. This uncertainty can significantly afect RH metrics in mixed canopies and forest edges [17].
LiDAR (Light Detection and Ranging) instrument that collects data about the Earth’s forests from space [10].
LiDAR emits laser beams and measures the time it takes for the light to return to the sensor. GEDI LiDAR is designed to penetrate the canopy, allowing scientists to study the 3D structure of forests. GEDI operated on the International Space Station (ISS) from 2019 to 2023.
Figure 3 illustrates a GEDI return waveform.
Structure A GEDI observation corresponds to a fragment of the Earth’s surface with a 25 m diameter called a footprint (Figure 3). GEDI was estimated to collect over 10 billion cloud-free observations in two years [10].
Data Products Raw GEDI waveforms are processed GEDI’s spatiotemporal limitations prevent scientists into higher-level data products that describe the 3D fea- from creating high-resolution forest change maps based tures of forests. For example, GEDI Level 2A data include solely on GEDI data. Nevertheless, GEDI can help admeasurements of ground elevation and relative height dress the lack of 3D information, spectral confusion, and (RH) [13]. RH is the height above ground at which a cer- limited ground truth problems in AFC. GEDI ofers 3D tain quantile of cumulative energy was returned (Figure information for remote unreachable forests. Therefore, 3), and the RH95 (95% quantile) has been shown to esti- GEDI data (like canopy height) can help distinguish formate canopy height (height of the top of the forest) [14]. est covers that may look the same in optical satellite These products are used to create 1 km × 1 km gridded imagery. While the spatial limitations of GEDI prevent maps. us from evaluating the entire AFC dataset, we present a
Accuracy GEDI was calibrated and validated using a novel method to identify data quality issues in AFC more ground truth dataset in which the evergreen broadleaf eficiently than random sampling while accounting for forests of South America were well represented [10]. geolocation error and noise in GEDI data. Studies show that GEDI can accurately estimate RH95 with RMSE of 2.08 m [15].
Data Quality Challenges Similar to other data col- 4. Approach lected from space, GEDI data has the following data quality issues:
are more likely to be instances of deforestation or other land covers such as grasslands.
We define areas with an undisturbed label in AFC and short canopy height in GEDI as conflicts or outliers. We suggest these conflicts can be more efective in identifying errors in the AFC dataset than randomly selected samples. Note that these conflicts represent potential errors that could arise from noise in either the GEDI or AFC data. Thus, several challenges need to be addressed: observations are more trusted than a single outlier. • Conflicts occurring close together can belong to the same area and cover type, corresponding to spatially correlated errors [19]. For instance, two consecutive GEDI shots are only 60 m apart, and both may be from a grassland misclassified as an undisturbed TMF in AFC. Using clustering, we avoid reporting these points separately. • Integrating the two datasets, the AFC map and Hierarchical clustering has two parameters: linkage and GEDI footprint data, while accounting for geolo- distance threshold. Linkage determines how the distance cation errors. between two clusters is calculated; e.g., single-linkage • Accounting for the noise in GEDI data and finding uses the minimum distance between members of the two samples that are more trusted. clusters. The distance threshold determines if clusters • Prioritizing some outliers when dealing with should be combined, merging only those closer than the thousands of conflicts, as manually examining threshold. all of them is too time-consuming. Step 4: Filtering Clusters. Clusters with few conlficts are less likely to represent areas with AFC errors than ones with many conflicts. Hence, we prioritize clusters larger than a certain threshold, . Additionally, clusters containing GEDI shots from multiple satellite orbits are more reliable and less susceptible to systematic errors. This is because consecutive shots within a single orbit could all be incorrect due to atmospheric conditions or sensor issues.
Step 1: Data Fusion. We identify the nine nearest AFC pixels to each GEDI shot. These pixels form a 3 × 3 window on the AFC map, with the center pixel containing the GEDI shot center (Figure 4). We only consider GEDI shots within homogeneous windows. This accounts for potential geolocation errors in the GEDI shot.
Step 2: Finding Outliers. We select GEDI shots with 95 < ℎ , where ℎ is a tuneable parameter. These shots are undisturbed TMFs with an abnormally short canopy. The parameter ℎ can be selected based on expert knowledge or the RH95 distribution.
Step 3: Clustering Outliers. We merge nearby outliers into clusters using hierarchical clustering for two reasons: • Reducing data noise. Several nearby conflicting
threshold (ℎ), clustering distance ( ), and minimum cluster size ( ). We discuss the efects and trade-ofs of these parameters in Section 5.
AFC map of the Brazilian Amazon region. We used RH95 from quality-filtered GEDI shots, as detailed in Burns et al. [20], collected during the second half of 2021. In this section, we describe two results. The first relied on ground truth from a validated forest cover map, while We use higher-resolution satellite images with approxthe second was from our own visual interpretation of imately 4 m per pixel resolution from the Planet NICFI high-resolution satellite images. data program [23, 24]. Specifically, we used the last cloud
Study Region We focus on the Brazilian Amazon free Planet images of 2021. Each cluster is assigned one rainforest, which plays a vital role in global climate sta- of three validation labels: Ambiguous (if no cloud-free bility, and is home to unique plant and animal species. observations are available or if it is unclear whether the The availability of numerous forest maps and freely ac- area is an AFC error), AFC Error, or False Positive (if the cessible satellite data makes the Brazilian Amazon an pixels are correctly classified in AFC). Analyzing 3 × 3 ideal region for our studies. windows of the map is similar to AFC’s validation method
Parameters Height Threshold (ℎ) is a tradeof be- [8]. tween precision and recall. A lower threshold reduces the Results We identified 23,306 conflicts (i.e., marked number of outliers, which can reduce false positives but undisturbed forest in AFC with 95 < 3.44 m) in Step 2. may afect recall. A lower Clustering Distance Threshold After filtering clusters in Step 4, 5,740 samples remain, of ( ) leads to smaller clusters that may represent correlated which 1.88% are labeled non-forest in MapBiomas. This errors. A higher distance threshold can merge unrelated gives 240 clusters, 12.08% of which have at least one clusters or create clusters of multiple noisy samples. Min- outlier with a non-forest MapBiomas label. Since manual imum Cluster Size ( ) also impacts precision and recall. evaluation is time-consuming, we evaluate 100 random Although small clusters are more likely to be false posi- clusters out of the 240 clusters using Planet images. Out tives, choosing a large afects the recall of small-scale of the 100 clusters, 33 were found to be AFC errors, 63 errors. were Ambiguous, and 4 were False Positives (see Figure
Based on empirical fine-tuning, we selected ℎ = 3.44 5 for examples). Assuming that all Ambiguous cases are meters to mark 0.3% of the GEDI shots in undisturbed False Positives, the precision of our method is at least 33%, TMFs as outliers. A lower threshold (e.g., 2 meters) elimi- which is almost three times greater than the precision of nated some evident AFC errors, whereas a higher thresh- random sampling reported by [8]. old (e.g., 4 meters) included many shots that were am- Discussion Visual interpretation revealed cases biguous as to whether they were AFC errors. We apply where both AFC and MapBiomas were inaccurate. This single-linkage hierarchical clustering with a distance of can be because of MapBiomas limitations due to the lack = 700 meters to group outliers that are from the same of 3D information in Landsat images. While MapBiomas GEDI orbit. We also filter clusters with fewer than = 8 has the advantage of evaluating every pixel in AFC, GEDI, shots. despite its limited coverage, uncovers errors that Map
MapBiomas Evaluation MapBiomas [21, 22] is an Biomas may not detect. Additionally, there are some annual dataset of Brazil’s land cover maps from 1985 vegetation types that should be classified as non-forest to 2021 at a 30-metre resolution. It uses a hierarchical covers in AFC but are considered forests in MapBiomas. classification system with four levels to categorize land For instance, MapBiomas assigns wooded savannah and covers. At Level 1, land covers are classified into six cate- tropical evergreens to the same class, while AFC refers to gories: forest, non-forest, farming, non-vegetated, water, the former as other land covers. Therefore, per-pixel evaland not observed. Level 2 expands Level 1 classes into uation cannot identify AFC errors in wooded savannahs, 22 subcategories. MapBiomas is created from Landsat but using canopy height can. images, primarily using a Random Forest classifier, and It is useful to study where AFC made errors. We found it is validated annually on over 75,000 samples. Level 1 many outlier clusters in the Brazilian Amazon’s northand 2 classification error is estimated to be 7.5% and 9.3%, west region, with a vegetation cover known as camprespectively [21]. inarana that can be dificult to distinguish in satellite
In this analysis, we use MapBiomas as a ground truth images [25]. This region is remote and challenging to dataset. Specifically, we assume that if an outlier shot is access, making it dificult to obtain field data. Various labeled as undisturbed TMF in the AFC map but classi- types of campinarana include forest, wood, shrub, and ifed as non-forest in MapBiomas, then we consider Map- grass on sandy infertile soil [26]. Forest campinaranas Biomas to be correct, meaning that the outlier is an error are up to 20 meters tall, whereas canopy height in the in the AFC map. We report two validation metrics: (1) non-forest classes does not exceed 4 meters [26]. Shrub the percentage of outliers with non-forest Mapbiomas campinarana appears green in satellite images which labels and (2) the percentage of clusters with at least one might lead to AFC errors. such outlier. Some False Positives were located near shores and wa
Visual Interpretation After finding outlier clusters, ter that could potentially be covered by mangroves (Figwe randomly select one GEDI shot per cluster. Then, we ure 5d). Mangroves have a distinct structure that difers determine if this represents an AFC error by analyzing from evergreen or semi-evergreen forests. However, all the 3 × 3 surrounding AFC pixels in a cloud-free image. three types are categorized as TMFs in AFC. Excluding such areas from the analysis could improve precision.
We also attempted to identify TMFs misclassified as deforested by filtering deforested samples with 95 > 20 metres. However, we were unsuccessful for several reasons. First, this approach does not reflect the complex nature of forests. Seeing a few square meters of trees does not indicate the presence of a forest. Second, the lack of historical height data prevents us from analyzing changes to distinguish primary forests from replacing tree plantations. Furthermore, 95 is prone to obstacle noise, such as from a flock of birds.
Holcomb et al. [27] recently discovered discrepancies between GEDI biomass (amount of organic matter in a forest) data and AFC label. They observed instances where areas with near-zero biomass were classified as regrowing forests (6+ years old), and other regions with substantial (> 200 Mg/ha) biomass, were classified as young (< 3 years old) regrowing forests after deforestation.
In this study, we used canopy height to find data quality issues in a forest change map. A recent study explored the use of canopy height to distinguish forested and unforested tropical wetlands [28]. They used a global canopy height map with a 30 metre resolution, created by combining GEDI RH metrics and Landsat images [14]. In contrast, our approach relies solely on raw GEDI height measurements.
In addition to creating a forest change map that estimates the year of forest loss, Hansen et al. [9] studied the relationship between loss year and canopy height using an older spaceborne LiDAR dataset. They observed that samples from undisturbed forests, on average, had greater canopy height than disturbed forests. This is consistent with our work.
Assessing the pixel-level agreement of two forest change maps is another way to find errors. However, this can be challenging due to variations in the map legends and diferences in resolution. Moreover, two maps based on Landsat optical images can be subject to the same data quality problems. Cohen et al. compared seven forest change maps based on Landsat at pixel level, revealing a low level of agreement [29]. On the other hand, GEDI allows us to leverage 3D information that does not exist in Landsat.
toring datasets, and their data quality issues. Although GEDI alone cannot be used to create a forest change map, it provides 3D information about forests missing from optical satellite imagery. We proposed a novel approach to find data quality issues in AFC using GEDI data, specifically, areas marked as TMF in the AFC map but with low RH95. Our findings show that this information can be used to create more accurate forest change maps.
Since no ground truth data were available, we used ancillary data in our evaluation [26, 22]. This process was limited by the interpretation of a single evaluator, and future studies could benefit from using a voting technique and involving experts. Our method could be used to identify errors in other land cover classification maps by finding inconsistencies between GEDI data and the target class. For instance, we can apply this method to ifnd errors in forest/non-forest maps. Furthermore, GEDI data products provide additional measurements, such as various RH metrics and Leaf Area Index (LAI). Exploring these metrics in future work can reveal other data quality errors in existing datasets. (a) AFC Error. The colour diference, texture diference, and geometric shape suggest that this area is not an undisturbed TMF.
(b) False Positive. This sample appears to belong to a highly dense forest cover.
(c) Ambiguous. The available context is insuficient to determine whether this sample is an error.
(d) False Positive. This area could potentially be covered by mangroves, which are classified as TMF.
up to year 2021, Publications Ofice of the European canopy height estimation and change monitoring, Union, Luxembourg, 2022. doi:10.2760/398695. Science of Remote Sensing 4 (2021) 100024. [8] C. Vancutsem, F. Achard, J.-F. Pekel, G. Vieilledent, [18] A. Pekkarinen, Global forest resources assessment S. Carboni, D. Simonetti, J. Gallego, L. E. Aragao, 2020, 2020. URL: https://www.fao.org/3/I8661EN/ R. Nasi, Long-term (1990–2019) monitoring of for- i8661en.pdf. est cover changes in the humid tropics, Science [19] P. Ploton, F. Mortier, M. Réjou-Méchain, N. Barbier, Advances 7 (2021) eabe1603. N. Picard, V. Rossi, C. Dormann, G. Cornu, G. Vi[9] M. C. Hansen, P. V. Potapov, R. Moore, M. Hancher, ennois, N. Bayol, A. Lyapustin, S. Gourlet-Fleury, S. A. Turubanova, A. Tyukavina, D. Thau, S. V. R. Pélissier, Spatial validation reveals poor predicStehman, S. J. Goetz, T. R. Loveland, et al., High- tive performance of large-scale ecological mapping resolution global maps of 21st-century forest cover models, Nature Communications 11 (2020) 4540. change, science 342 (2013) 850–853. URL: https://doi.org/10.1038/s41467-020-18321-y. [10] R. Dubayah, J. B. Blair, S. Goetz, L. Fatoyinbo, doi:10.1038/s41467- 020- 18321- y.
M. Hansen, S. Healey, M. Hofton, G. Hurtt, J. Kellner, [20] P. Burns, C. Hakkenberg, S. Goetz, Gedi l2ab + l4a S. Luthcke, et al., The global ecosystem dynamics global processing from april 2019 to december 2022, investigation: High-resolution laser ranging of the 2023. Manuscript in Preparation. earth’s forests and topography, Science of remote [21] MapBiomas, Algorithm theoretical basensing 1 (2020) 100002. sis document (atbd): Collection 7, 2022. [11] M. A. Wulder, D. P. Roy, V. C. Radelof, T. R. Love- https://mapbiomas-br-site.s3.amazonaws.com/ land, M. C. Anderson, D. M. Johnson, S. Healey, ATBD_Collection_7_v2.pdf.
Z. Zhu, T. A. Scambos, N. Pahlevan, et al., Fifty [22] M. A. Project, Collection 7.0 of annual land cover years of landsat science and impacts, Remote Sens- and land use maps, 2021. Accesed on 05.2023 via ing of Environment 280 (2022) 113195. link http://amazonia.mapbiomas.org. [12] J. E. Batista, N. M. Rodrigues, A. I. Cabral, M. J. Vas- [23] P. L. PBC, Planet application program interface: concelos, A. Venturieri, L. G. Silva, S. Silva, Optical In space for life on earth, 2018–. URL: https://api. time series for the separation of land cover types planet.com. with similar spectral signatures: cocoa agroforest [24] NICFI, Norway’s international climate and forest and forest, International Journal of Remote Sensing initiative, 2022. URL: https://www.nicfi.no/. 43 (2022) 3298–3319. [25] C. L. d. O. Cordeiro, D. d. F. Rossetti, Mapping veg[13] R. Dubayah, M. Hofton, J. Blair, J. Armston, H. Tang, etation in a late quaternary landform of the amaS. Luthcke, Gedi l2a elevation and height metrics zonian wetlands using object-based image analysis data global footprint level v002, 2021. URL: https: and decision tree classification, International Jour//lpdaac.usgs.gov/products/gedi02_av002/. doi:10. nal of Remote Sensing 36 (2015) 3397–3422. 5067/GEDI/GEDI02_A.002. [26] IBGE, Manual técnico da vegetação brasileira, 2012. [14] P. Potapov, X. Li, A. Hernandez-Serna, A. Tyukav- [27] A. Holcomb, S. Mathis, S. Keshav, D. A. Coomes, ina, M. C. Hansen, A. Kommareddy, A. Pickens, Computational tools for assessing forest recovery S. Turubanova, H. Tang, C. E. Silva, et al., Mapping with gedi and forest change maps., 2023. In Review. global forest canopy height through integration of [28] K. Verhelst, Y. Gou, M. Herold, J. Reiche, Improvgedi and landsat data, Remote Sensing of Environ- ing forest baseline maps in tropical wetlands using ment 253 (2021) 112165. gedi-based forest height information and sentinel-1, [15] C. Wang, A. J. Elmore, I. Numata, M. A. Cochrane, Forests 12 (2021) 1374.
L. Shaogang, J. Huang, Y. Zhao, Y. Li, Factors af- [29] W. B. Cohen, S. P. Healey, Z. Yang, S. V. Stehman, fecting relative height and ground elevation estima- C. K. Brewer, E. B. Brooks, N. Gorelick, C. Huang, tions of gedi among forest types across the conter- M. J. Hughes, R. E. Kennedy, et al., How similar minous usa, GIScience & Remote Sensing 59 (2022) are forest disturbance maps derived from diferent 975–999. landsat time series algorithms?, Forests 8 (2017) 98. [16] P. V. Oliveira, X. Zhang, B. Peterson, J. P. Ometto,
Using simulated gedi waveforms to evaluate the efects of beam sensitivity and terrain slope on gedi l2a relative height metrics over the brazilian amazon forest, Science of Remote Sensing 7 (2023) 100083. doi:https://doi.org/10.1016/j.
srs.2023.100083. [17] D. P. Roy, H. B. Kashongwe, J. Armston, The impact of geolocation uncertainty on gedi tropical forest