This study explores object detection in historical aerial photographs of Namibia to identify long-term environmental changes. Specifically, we aim to identify key objects - Waterholes, Omuti homesteads, and Big trees - around Oshikango in Namibia using sub-meter gray-scale aerial imagery from 1943 and 1972. In this work, we propose a workflow for analyzing historical aerial imagery using a deep semantic segmentation model on sparse handlabels. To this end, we employ a number of strategies including class-weighting, pseudo-labeling and empirical p-value-based filtering to balance skewed and sparse representations of objects in the ground truth data. Results demonstrate the benefits of these different training strategies resulting in an average 1 = 0.661 and 1 = 0.755 over the three objects of interest for the 1943 and 1972 imagery, respectively. We also identified that the average size of Waterhole and Big trees increased while the average size of Omutis decreased between 1943 and 1972 reflecting some of the local effects of the massive post-Second World War economic, agricultural, demographic, and environmental changes. This work also highlights the untapped potential of historical aerial photographs in understanding long-term environmental changes beyond Namibia (and Africa). With the lack of adequate satellite technology in the past, archival aerial photography offers a great alternative to uncover decades-long environmental changes.
Satellite imagery is a valuable source of data that can shed light on the long-term impacts of climate
change [
In this work, we aim to utilize archival aerial photos from north-central Namibia, taken in 1943 and 1972, to uncover the decades long changes on the ground predating the introduction of high-resolution satellite imagery. The 1943 aerial photos are assumed to be the first instance where aerial photography
E
Angola Namibia
Semantic Segmentation Framework
B
Train and Test region
C
Big Tree
Omuti
Waterhole
technology was systematically used in capturing the landscape of northern Namibia [
With the encouraging potential of machine learning (ML) algorithms to decipher large collection of
data and identify patterns, we employ a deep learning framework that aims to take the digitized aerial
photos as input and detected these objects of interest. Specifically, the framework contains a U-Net-based
segmentation model [
The remainder of the paper is organized as follows. Section 2 presents the methodology, with details on the main contributions. Section 3 describes the experimental setup including the specifics of the datasets used, segmentation model employed and its setting, and evaluation metrics. We present the notable results and follow up discussions in Section 4. Finally, Section 5 concludes the paper with next steps as a future work.
The overview of our approach is shown in Fig. 1. Given aerial photos from the Oshikango region in Namibia from 1943 and 1972, we aim to detect specific objects of interest: Big Trees, Omuti homesteads and Waterholes at each of the time stamps to uncover long-term environmental changes. To this end, we Annotated
Data
Train/Test Split
Training Train Set Test Set
Class Weighting Pseudo Labeling
Inference Multi-level Inference
Postprocessing
Evaluation Performance
Metrics Evaluation
Sets
employed a pre-processing step that aims to digitize and geo-reference each of the photos and merge
them into a large orthomosaic input following the steps in [
Let Dt represents an orthomosaic of multiple aerial photos taken in a year, , after each photo is digitized and geo-referenced. The problem is related to evaluating the potential of these aerial photos to quantify the long-term environment changes by detecting a set of objects of interest - : Big tree, : Omuti and : Waterhole at = 1943 and = 1972. To this end, we employ a deep learning framework to detect these objects at each Dt with a dedicated model, Θt. We assume a few examples of these objects are available as polygons or mask data, Mt, annotated by an experienced expert in the region. Each pixel, ∈ Mt, assumed to be one of the classes: = {, , , }, where represents unknown or background pixels. Due to the sparse nature of annotation performed in a smaller region, i.e., |Mt|) << |Dt|), where | · | represents dimension, a key aspect of the framework involves effective usage of Mt where the number of labeled pixels, , is quite small compared to the unknown pixels, . Furthermore, there is a large degree of imbalance in annotated pixels among classes , and . This also poses a critical question on how to utilize the larger number of pixels in Mt thereby assisting the training process and enhancing detection performance. Mt is, first, split into train ( Mtr) and test (Mte) splits with no overlapping between the two splits. The model, Θ, is trained using Mtr and evaluated for both Mtr and Mte. We further extend Mtr by incorporating new masks derived from predicted polygons from previously unannotated regions in Dt as pseudo-labels.
Class imbalance is a common challenge in geospatial machine learning as it is often resource-demanding
to do manual annotations, resulting in a sparse set of annotated regions. This is also partly due to
the different observation frequencies of objects of interest. For example, in the related aerial photo
Oshikango region in this work, we observed a higher occurrence of Big trees compared to Omuti
homesteads. In addition, the coverage area of each class may vary (see Table 1) resulting imbalanced
number of pixels across classes. Varieties of solutions have been employed to address class imbalance
challenges over the years that could be clustered under re-sampling [
In this work, we propose a simple class weighting strategy that considers both the sparsity of annotated regions) (compared to unannotated regions) and the imbalance of pixels annotated across the classes of interest. Our weighting strategy follows a re-weighting approach that aims to provide a class weight that is inversely proportional to the ratio of pixels annotated per each class (compared to the remaining classes). Let , and be the number of pixels annotated with Waterhole, Omuti and Big tree classes in the training set, Mtr, respectively. The number of unlabeled pixels is denoted by . The total number of pixels in is + , where = + + . Thus, the weight of each class is formulated as follows: = /( + ), = /, = /, and = /,
To further improve the efficiency of our training steps, we propose to incorporate the weak labels generated
from the inference of the model on the previously un-annotated pixels in training – i.e. a pseudo-labeling
based approach [
However, deep learning models are known to provide over-confident predictions even in cases where
the predicted classes are not correct [
In this work, we aim to utilize area feature and discard predicted polygons with area values that are out-of-distributions from areas of training polygons per each class. Typical threshold-based filtering could be applied directly on the histogram of the area values. However, it has been found that the distribution is skewed heavily (see Fig. 3 (a)) and hence a threshold based filtering will be very sensitive to the threshold value. On the other hand, the distributions of empirical p-values (see Fig. 3 (b)) is relatively less skewed and hence more stable for threshold-based filtering. The pseudo-code of the proposed empirical p-value based post-processing is shown in Algorithm 1. Let’s assume that we are given the set of annotated training polygons, , and predicted polygons, , for each ℎ class of interest in ^ = {, , }. Then we compute the area of each annotated polygon in , e.g., . Similarly, we compute the area of each polygon in , e.g., . The empirical p-value of each predicted polygon, , is calculated by counting the number of polygons in that are greater than or equal with . Thus, a predicted polygon with out-of-distribution area will have extreme empirical p-value, i.e., ≈ 0 and ≈ 1 for predicted polygons with very small area and very large area, compared to the training set, respectively. Finally, the predicted polygons that satisfy the empirical-value-based threshold, ℎ, are considered as pseudo labels to be used in the follow up recursive training steps. 2500 5000 Area (m2)
25000 Area (m2) BigTree 50000 00.0 00.0 y c en2000 u q e r F
0.5 Empirical p-value Omuti
In this section, we describe the data sources used in this work, along with the distribution of annotations across classes, the deep learning model architecture and hyper-parameters set for our experiments, and performance evaluation metrics.
We have used aerial photos taken in Northern Namibia in the years 1943 and 1972. See Fig. 4 for an example of these photos and the types of classes annotated in these photos. Note the annotations were done manually by a domain expert. Table 1 shows the distributions of annotations in pixels and polygons. The aggregated percentage of annotated pixels is < 1% in 1943 imagery and < 4% in 1972 imagery demonstrating the sparsity of annotated pixels (regions) compared to the unannotated regions - a typical challenge in geospatial imagery. Furthermore, Table 1 demonstrates the nature of the imbalanced number of annotated pixels or polygons across classes, e.g., ≈ 90% or more of these annotated polygons belong to Tree class whereas Waterhole constitute only < 4% of the polygons in both 1943 and 1972 images. 1 for in ^ do 2 ← CountPolygons ( );
FilterByThreshold ( , ℎ);
We have employed a U-Net-based [
Inference is performed at each pixel level in the test set, which can also be aggregated to polygon-level inference. Thus, the evaluation metrics are also computed corresponding to the level of inference. Generally, we employed Accuracy, Precision, Recall, and 1 score to evaluate how well each class’s annotated pixels (regions) were detected during inference. All the four metrics represent detection performance based on true positive (tp), true negative (tn), false positive (fp) and false negative (fn) values. For pixel-level performance metrics, true positive is when the class pixel is correctly identified; true negative is when the pixels associated with the remaining classes are correctly identified as negative; false positive is the case when pixels corresponding to the remaining classes are incorrectly detected as the class pixel, and false negative refers to the case when class pixels are incorrectly detected as the remaining class pixels. For polygon-level performance metrics, tp, tn, fp, fn are computed from a threshold-based overlapping of regions, e.g., 5%, between the predicted and ground truth polygons. Note that we have not computed the evaluation metrics for the background class as it could still be a real background class or any of the classes but left unlabeled during annotation.
Table 2 shows the results derived from different training strategies employed in our semantic segmentation framework across two imagery timestamps: 1943 and 192. Compared to 1 = 0.549 in 1943 imagery, we achieved a higher 1 = 0.706 on the 1972 imagery. This is partly due to the higher number of examples from 1972 imagery to train its model (see Table 1). Furthermore, our different training strategies, i.e., 1943 1972 class weighting, pseudo labeling and post-processed pseudo-labeling, outperformed the Baseline that does not include any of these strategies. Particularly, the class weighting strategy alone improved the Recall values from 0.261 to 0.656 in 1943 imagery and from 0.663 to 0.809 in 1972 imagery by effectively weighting the cross-entropy loss by the inverse of the observation of each class. Pseudo-labeling that aims to utilize high-confident predictions in a semi-supervised learning fashion is also shown to further improve the Precision (by reducing the false positives) and then the 1 score in both imagery sources. Additional difference between 1943 and 1972 images involve the impact of using pseudo-labels after empirical p-value based post-processing (i.e., Post-processed Pseudo Labeling). Since the ground truth data of 1943 imagery suffers from very few number of training samples per class (i.e., < 1% of the imagery is annotated), discarding the predicted polygons based on a threshold did not result in an improved performance. On the other hand, filtering the pseudo-labels before recursive training improved all the metrics in 1972 imagery, resulting in the highest 1 = 0.706.
Tables 3 demonstrates the need of lfitering predicted polygons as a part of our empirical area p-value based post-processing even for evaluating the performance metrics. The highest average 1 score across the three classes is achieved in 1943 (1 = 0.661) imagery using a p-value threshold of ℎ = 0.5 STD. Similarly, the post-processing has improved the average 1 score from 0.706 to 0.755 using an p-value threshold of ℎ = 1.0 STD, which is partly due to a larger and more balanced set of ground truth data and hence does not require a strong threshold that would discard more polygons.
Among the metrics employed to evaluate our detection performance, Recall values are found to be consistently higher than Precision values for both imagery sources (see Table 2). This is partly due to the higher observation of false positives compared to false negatives. Further analysis demonstrates that all false positives are not actually falsely identified objects of interest but previously unlabeled objects. Figure 5 shows such an instance, where two objects were shown unlabeled in the ground truth polygons in Figure 5 (a). But these two objects are detected as Waterholes during inference time, thereby suggesting the framework could also help to discover objects of interest that were not labeled during annotation though they might still be evaluated as false positives. This further motivates the need of pseudo-labeling (a) Ground truth polygons
(b) Predicted polygons in our framework that aimed to utilize such objects that were left unlabeled during annotation but later found out to be objects of interest with high confidence. Moreover, our analysis facilitates understanding of past events with less/limited data, and provides added quantitative and qualitative details on historic reports (see Fig. 6). For example, the number and location of Waterholes and homesteads confirm the sharing of Waterholes by neighbors, low-yielding reports of these water holes, and increased population density. Furthermore, class-based changes (e.g., Waterholes) in number, area, coverage, locations and/or proximity to other objects of interest reveal further insights on the changes that took place between 1943 and 1972.
Furthermore, we have utilized the trained semantic segmentation model in a larger area covering ≈ 5000 2. See Fig. 1(B) to visualize the scale of a region compared to the relatively smaller annotated region (≈ 45 2) used as a ground truth. Such large scale implementation of the framework helps to benefit domain experts by reducing the resource necessary to make exhaustive annotation, and generate large scale insights.
Understanding long-term environmental changes requires the use of old and remotely sensed images
such as satellite imagery. However, the resolutions of satellite images were not at sub-meter level a
few decades back. Old aerial photos, on the other hand, satisfy these requirements though they are
often stored unused in archives and museums across the world. In this work, we aim to demonstrate the
capabilities for understanding long-term environmental changes in Namibia using aerial photos taken
during 1943 and 1972 using deep learning to detect the following objects: Waterholes, Omuti homesteads
and Big trees. To this end, we employed a deep semantic segmentation framework that includes U-Net [
Future work aims to investigate further use cases where lower detection performance metrics, both in precision and recall, were observed. In addition, we aim to scale up the validation of the proposed approach beyond the use case in Namibia. Deploying under-utilized archival aerial photographs is, potentially, a promising alternative to validate current understanding of the past and uncover new insights, which is critical to ensure sustainability in Africa, where climate change poses a significant and disproportionate risk compared to its emission.