Classification of Dead Trees in Urban Parks Using Aerial Imagery and Convolutional Neural Networks

Abstract

In Mexico City, urban parks are facing a significant challenge due to a growing infestation of a hemi-parasitic plant known as mistletoe, which is detrimental to trees. Identifying trees killed by mistletoe is crucial for estimating infestation levels and determining the urgency of phytosanitary interventions. Traditional methods for monitoring tree health are manual, costly, time-consuming, and are realized by forestry specialists. This research introduces an approach using Convolutional Neural Networks (CNNs) to classify dead trees based on aerial imagery registered over an urban park in Mexico City. For this purpose, we collected 460 image sets and employed two CNN models, ResNet-34 and DenseNet-121. Our findings indicate that these models efectively discern unique vegetation patterns that could be associated with mistletoe infestation, and they perform with a high classification accuracy and a reduced computational cost.

Keywords

• eol>Dead trees classification
• CNNs models
• UAV imagery
• ResNet-34
• DenseNet-121
• Forest monitoring

1. Introduction

Urban parks, also known as urban green spaces, are vital to cities for ofering recreational areas, mitigating stress, reducing air pollution, and providing natural beauty along with other environmental services [ 1, 2 ]. These spaces also serve as sanctuaries for various animal and vegetation species, thereby enhancing the quality of life for city residents [ 3 ]. Over the past decade, the utilization of Unmanned Aerial Vehicles (UAVs) for imagery has gained popularity in monitoring forests and urban parks [ 4 ]. This technology has been instrumental in areas such as forest insect and pest control, including bark beetles, and disease monitoring, like pine wilt disease [ 5 ]. A distinctive advantage of UAVs over other remote sensing technologies is the high spatial resolution of the images they capture, which can be on the order of centimeters. This allows for the detailed identification of pest infestations and parasitic species, among other concerns [ 6 ]. Given the detailed nature of the data collected, the development of eficient, accurate, and flexible methodologies for analyzing these results is essential [ 7 ].

In Mexico City, urban parks are being increasingly plagued by the rapid spread of a hemi-parasitic plant known as mistletoe, posing significant challenges to tree health. Mistletoe seeds are dispersed through a combination of their inherent ballistic propulsion mechanism and transportation by birds and other fauna [ 8, 9 ]. Upon successful establishment on a host tree, mistletoe integrates into the tree’s structure, consuming vital nutrients and resources. Without timely and efective management, this parasitic relationship can lead to the host tree’s decline and eventual death. Therefore, there is an urgent need for strategic control measures to mitigate the infestation and preserve the ecological balance of urban green spaces [ 10, 11 ], including Mexico City.

Although mistletoe species exhibit distinctive characteristics with respect to the host tree, their visual identification often requires expertise, typically from a forestry specialist [ 12, 13 ]. Consequently, efectively controlling new foci of infestation remains a challenging task. Moreover, pinpointing trees deceased due to mistletoe infestation can serve as a means to gauge the extent of infestation within a designated area, where a substantial number of dead trees signal the urgency for phytosanitary intervention. Despite the existence of various manual methodologies for the surveillance of individual trees, the integration of UAV imagery alongside classification algorithms naturally emerges as a promising approach for tackling this issue [ 14, 15 ].

In the current literature oriented to trees classification based on remote sensing data, Support Vector Machines and Random Forests have emerged as the most frequently used machine learning techniques. However, a significant challenge associated with these techniques is the selection of image features that enhance classification accuracy [ 16 ]. In contrast, deep learning strategies, capable of autonomously learning from data, obviate the need for manual feature engineering [ 17 ]. Consequently, widely-used deep learning techniques, such as Convolutional Neural Networks (CNN), have become high-performance tools suitable for various remote sensing applications [ 18 ]. Despite these advances, few studies focus on automatic methodologies for dead trees inventory, crucial for efective forest management and ecological monitoring. Notably, the studies by [ 19 ] and [ 20 ] ofer innovative approaches for utilizing remote sensing data to detect and classify dead trees, emphasizing the need for expanded research in this critical area.

In this work, we propose the use of CNNs for the classification of trees dead by mistletoe, utilizing multispectral aerial imagery from an urban park in Mexico City. Specifically, we employed two CNN models, ResNet-34 and DenseNet-121, which have been previously recognized for their ability to identify unique vegetation patterns, significantly enhancing the classification accuracy of images [ 21 ]. This investigation marks an initial endeavor to assess mistletoe infestation levels in Mexico’s urban parks. The creation of automated classification tools for identifying mistletoe species and dead trees holds the potential to significantly streamline the extensive work typically conducted by forestry experts.

The structure of this document is outlined as follows. Section 2 details the specifications of the imagery utilized in compiling our dataset and provides an overview of the CNNs employed in our study. Section 3 elaborates on the methodology for training the models and the experiments conducted to evaluate their performance. Section 4 delivers the key findings, highlighting the performance metrics achieved by both models. Finally, Section 5 ofers a discussion on the implications of our research ifndings and draws conclusions.

2. Materials and Methods

2.1. Image dataset

Our study area encompasses an urban green space in Mexico City, known as Ramón López Velarde Garden, located at coordinates [19.4096∘ , -99.1563∘ ]. This park hosts a variety of tree species, e.g., Casuarina sp., Eucalyptus sp., Fraxinus sp., Cupressus sp., Ligustrum sp., and Grevillea sp. [ 22 ]. The mistletoe species Sthruthantus Interruptus is prevalent among many of these trees, significantly altering the natural landscape. Consequently, the mistletoe infestation has led to the demise of several trees, as illustrated in Figure 1-(a).

To monitor mistletoe propagation, multispectral images were collected on September 22, 2022, as described in [ 23 ]. For this purpose, an UAV equipped with a five CMOS sensors camera was employed, covering the spectral regions of Blue (450nm ± 16nm), Green (560nm ± 16nm), Red (650nm ± 16nm), Red-edge (730nm ± 16nm), and Near-infrared (840nm ± 26nm). In total, 460 images with dimensions of 1600 × 1300 pixels were registered in both orthogonal and oblique modes along the study area. Of these, only 111 images revealed the presence of dead trees.

As discussed in [ 23 ], the image sets were co-registered prior to analysis to ensure that every pixel corresponds to the same geographical coordinate across the five bands. Later, binary masks delineating the presence of three classes —Dead Trees (DT), Green Vegetation (GV), and Man-made Structures (MS)— were created based on a manual segmentation process from the RGB versions of the original images; color images were uploaded and annotated within the CVAT application (www.cvat.ai). The binary masks were then employed for the subsequent training and validation of our models. Figure 1-(b) shows a representative example of the segmented areas for the classes mentioned. (a) RGB color image. (b) Segmented image.

2.2. Convolutional neural network models

In similar studies oriented towards the classification of tree species, convolutional models such as Residual Networks (ResNet) and Densely Connected Networks (DenseNet) have shown promising results based on remote sensing data [ 17, 24, 25 ]. Given that both models are pre-trained, they are capable of reducing computational costs while extracting distinctive vegetation patterns from the collected images. These models are usually trained on 3-channel images, which could be a limitation, especially when multispectral sets are available [ 26 ].

The unique structure of ResNet’s residual blocks is particularly beneficial for object classification, enabling the eficient training of deep networks without incurring performance loss as their depth increases [ 27 ]. In addition to consecutive convolutions and activations within each block, ResNet features skip connections that utilize identity or convolutional shortcuts to mitigate the vanishing gradient problem. This design not only facilitates the construction of deeper networks, but also reduces the complexity by minimizing the number of max-pooling layers required, typically incorporating only one [ 26 ]. Figure 2 illustrates the architecture of ResNet-34, showcasing its overall structure, which includes: 32 convolutional layers with a 3 × 3 filter and ReLU activation functions, an average pooling layer, and a fully connected layer. Observe that a shortcut connection is inserted to every pair of 3 × 3 iflters.

On the other hand, unlike traditional convolutional networks, DenseNet introduces a dense connectivity structure among layers; that is, each layer is directly connected to all subsequent layers. This approach ensures eficient information flow and mitigates data loss, a common issue with residual connections. Additionally, DenseNet controls feature growth through a bottleneck strategy and employs compression to maintain an optimal number of parameters. This architecture has demonstrated strong performance across various datasets, attributed to its efective management of information throughout the network.

As illustrated in Figure 3, DenseNet features densely interconnected blocks where each layer is directly connected to every subsequent one. Within each block, the fundamental layers include convolutions, batch normalization, and ReLU activation functions. The architecture incorporates transition layers that perform dimensionality reduction, including both spatial and feature-channel compression, to eficiently manage the flow of information through the network. Additionally, DenseNet generates compact models that are parameter-eficient and straightforward to train through features reusing. By concatenating feature maps from diferent layers, the network enriches the feature set available to each layer, enhancing its representational power. This ability to leverage a rich feature set from across the network could potentially improve the classification of dead trees by providing a more diverse range of information for decision-making.

2.3. Hardware and Software

The experiments described in this manuscript were executed on a workstation with the following characteristics: Intel(R) Xeon(R) CPU e5-2620 v2 2.10 GHz, 128 GB RAM, Windows 11 (64-bits) as the operating system.

Manual segmentation and labeling of aerial images were conducted on the annotation platform known as CVAT (http://www.cvat.ai). The image preprocessing steps and classification routines were developed using Python 3.8 and implemented in the PyTorch framework, version 2.2.1.

3. Experiments

Given that only 111 images were labeled with the presence of dead trees, a data augmentation process was applied to the RGB versions of these images. This process included image transformations, such as rotations by 90∘ , 180∘ , and 270∘ , as well as horizontal and vertical reflections. As a result, a total of 666 images were generated for our experiments.

For the training and validation of our models, each image in the augmented set was divided into 16 × 16 tiles to enhance and increase the variability of the input data. We then carefully selected tiles that contained at least 90% of color and texture information of the original images. This selection process yielded 43,844 tiles for the Dead Tree class, 237,689 for Green Vegetation, and 111,110 for Man-made Structures. It is noteworthy that the quantities of these last two classes account for about 80% of the total data in our dataset.

ResNet-34 and DenseNet-121 models were adapted to suit our specific application, with modifications primarily to their output layers to classify the target classes, while leveraging the pre-trained weights of the other layers. We conducted an initial experiment using a subset of 5000 tiles per class to assess the models’ overall performance. This assessment utilized a holdout method, allocating 80% of the samples for training and 20% for validation, and a 5-fold cross-validation process. Table 1 shows the performance metrics from this preliminary training, including Precision, Accuracy, F1-score and Recall.

From the previous results, the overall accuracy with 5-fold cross-validation was slightly higher than the obtained with the holdout method. For instance, ResNet-34 achieved an accuracy of 0.95 with the former compared to approximately 0.9 with the latter validation method. However, the main drawback of 5-fold cross-validation relies on its high computational cost. Therefore, throughout this research, we opted for the holdout validation method. For both models, we used Cross Entropy Loss as our loss function, set the batch size to 64, and employed the Adam Optimizer. Table 2 details these and additional CNN training parameters.

4. Results

ResNet-34 and DenseNet-121 were trained on 105,225 tiles, constituting 80% of the entire dataset. The remaining 26,307 tiles, which make up approximately 20% of the dataset, were allocated for validation. The models were configured according to the parameters outlined in Table 2 and underwent a training regimen spanning 100 epochs. The training and validation performances, as depicted in Figure 4-(a) for ResNet-34 and Figure 4-(b) for DenseNet-121, reveal that the training loss trends closely align with those of the validation loss across epochs. Similarly, the training and validation accuracy trends show parallelism. This consistency between training and validation phases indicates an absence of overfitting and underscores the models’ capability to accurately classify the input images.

(a) Performance graph obtained from training using ResNet-34.

(b) Performance graph obtained from training using DenseNet-121.

Table 3 presents the confusion matrices for both ResNet-34 and DenseNet-121, based on a validation set consisting of 26,307 tiles. In particular, for ResNet-34 the overall accuracy reached 96.68%, with the DT class achieving an accuracy of 95.80%, primarily due to confusions with the MS class (about 3.4%). The accuracies for the GV and MS classes were 98.89% and 95.33%, respectively. Comparatively, DenseNet-121 showed an improvement in classification accuracy across all classes. Specifically, the DT class had 97.89% of its samples correctly classified, the MS class reached an accuracy of around 96.00%, and the GV class achieved 98.72% of accuracy. Therefore, the overall performance of DenseNet-121 was approximately 97.53%.

Additionally, Table 4 presents the performance metrics for ResNet-34 and DenseNet-121, obtained using a holdout validation on the entire dataset. Coincidentally, both models exhibit almost the same metric values: 0.96 for ResNet-34 and 0.97 for DenseNet-121. These results indicate that DenseNet-121

It is interesting to denote that the classification process conducted by the trained models can be visualized as heat maps, aimed at demonstrating how the models identify specific classes of interest, such as dead trees. In order to generate a heat map, every pixel of an input image is processed through the softmax function for the DT class, which then assigns a probability value between [ 0,1 ] to it. This process results in the creation of a map, whose red tones represent maximum probabilities, while purple ones correspond to low values. Examples of heat maps produced for the DT class are displayed in Figure 5 for ResNet-34 and DesNet-121, respectively.

5. Discussion and Conclusion

This study addresses the growing concern of mistletoe infestation within urban green spaces, particularly focusing on an urban park in Mexico City. The detrimental impact of mistletoe on tree health and, consequently, on the ecological balance and aesthetic value of urban parks requires for innovative monitoring and management strategies. By taking advantage of the high spatial resolution capabilities of aerial imagery, we relied on the pattern recognition potential of CNNs, specifically ResNet-34 and DenseNet-121, for the automated classification of dead trees afected by mistletoe. To compensate for the limited availability of labeled images, we realized a data augmentation process, which was followed by the segmentation of these images into tiles to enhance the variability and richness of the input data for model training. Through this approach, we aimed to develop a robust, eficient, and scalable solution for the classification task.

The challenge posed by the limited quantity of labeled images depicting dead trees required the implementation of a data augmentation strategy to enhance our dataset. Hence, by applying rotations and reflections to the original images, we expanded our dataset sixfold, ensuring a more comprehensive representation of the classes of interest. Further refinement was achieved by segmenting these augmented images into 16 × 16 tiles, a process that not only amplified the diversity of the training data, but also allowed for a more granular analysis of the imagery. The selection process of those tiles preserving at least 90% of color and texture information of the original images was instrumental in curating a dataset that closely mimics the variability encountered in natural settings and allows a more eficient training for CNN models.

In adapting the ResNet-34 and DenseNet-121 models to our specific classification task, we tailored the output layers to distinguish between Dead Trees, Green Vegetation, and Man-made Structures, (a) ResNet-34.

(b) DenseNet-121. while retaining the pre-trained weights of other layers to capitalize on the models’ inherent strengths. This strategic use of transfer learning facilitated an eficient training process, enabling the models to adeptly identify distinctive vegetation patterns. The initial experiment, which utilized a balanced subset of 5000 tiles per class, revealed the capabilities of both models. DenseNet-121, in particular, demonstrated a slight edge over ResNet-34, achieving higher performance metrics across Precision, Accuracy, Recall, and F1-score. For the task described in this work, DenseNet-121 emerged as a better option than ResNet-34 in terms of the ability to manage and leverage feature information, likely due to its densely connected architecture.

The evaluation of model performance employed two validation methods: holdout and 5-fold crossvalidation. Despite the slightly superior accuracy achieved by a 5-fold cross-validation, the significant computational resources it demanded led us to predominantly utilize the holdout method for our experiments. This pragmatic approach balanced eficiency with eficacy, ensuring robust model assessment without unduly taxing computational resources. The eficiency of the models, particularly DenseNet-121, is evident by the resulting performance metrics, but the closely aligned training and validation trends point also to the model’s robust generalization capabilities.

The utility of the trained models extends beyond numerical metrics to ofer intuitive visual insights through the generation of heat maps; these maps serve as a visual proxy, showing the models’ interpretative process by highlighting areas within the input images deemed significant for classification, a fundamental aim in explainable artificial intelligence (XAI). Particularly for the DT class, the heat maps efectively demarcate regions identified by the models as indicative of dead vegetation, with varying intensities reflecting the confidence levels of the models. By translating the models’ complex decision-making into a comprehensible format, these heat maps not only validate the accuracy of the models, but also provide a valuable tool for forestry specialists. Such visual aids facilitate a deeper understanding of mistletoe spread patterns and can significantly augment traditional surveillance methods, ofering a novel lens through which to assess and strategize phytosanitary interventions in urban green spaces.

This research demonstrates the efective application of CNNs, specifically ResNet-34 and DenseNet121, in classifying dead trees within urban parks using UAV-derived imagery. Through data augmentation and tile-based analysis, we have established a robust framework that not only enhances the dataset’s diversity, but also refines the training process of the models. The superior performance of DenseNet-121 suggests the potential of densely connected architectures in handling complex classification tasks. The complementary use of heat maps for visual interpretation further accentuates the practical utility of these models in urban forestry management. Future research could explore the integration of additional spectral bands to leverage the full potential of multispectral imagery, the inclusion of larger and more varied datasets to improve model robustness, and the application of these methodologies to other urban ecological environments.

References

[1] V. Mehta , B. Mahato , Designing urban parks for inclusion, equity, and diversity , Journal of Urbanism: International Research on Placemaking and Urban Sustainability 14 ( 2020 ) 457 - 489 . doi: 10 .1080/17549175. 2020 . 1816563 .

[2] J. Fernandez , Y. Song , M. Padua , P.-C. Liu, A framework for urban parks , Landscape Journal 41 ( 2022 ) 15 - 29 . doi: 10 .3368/lj.41.1.15.

[3] V. Kasyanov , R. Silin , Method for multi-criteria evaluation of urban parks , IOP Conference Series: Materials Science and Engineering 687 ( 2019 ). doi: 10 .1088/ 1757 -899X/687/5/055040.

[4] Z. Xu , X. Gao , Z. Wang , J. Fan , Big data-based evaluation of urban parks: A chinese case study , Sustainability ( 2019 ). doi: 10 .3390/SU11072125.

[5] K. Pristouris , H. Nakos , Y. Stavrakas , K. I. Kotsopoulos , T. Alexandridis , M. S. Barda , K. P. Ferentinos , An integrated system for urban parks touring and management , Urban Science ( 2021 ). doi: 10 . 3390/urbansci5040091.

[6] J. Zhu , H. Lu , T. Zheng , Y. Rong , C. Wang , W. Zhang , Y. Yan , L. Tang , Vitality of urban parks and its influencing factors from the perspective of recreational service supply, demand, and spatial links , International Journal of Environmental Research and Public Health 17 ( 2020 ). doi: 10 .3390/ijerph17051615.

[7] G. D. Moore , J. Hopkins , Urban parks and protected areas: on the front lines of a pandemic ( 2021 ) 73 - 84 . doi: 10 .2305/IUCN.CH. 2021 .PARKS-27-SIGM.EN.

[8] L. Skrypnik , P. Maslennikov , P. Feduraev , A. Pungin , N. Belov , Ecological and landscape factors afecting the spread of european mistletoe (viscum album l.) in urban areas (a case study of the kaliningrad city , russia), Plants 9 ( 2020 ). doi: 10 .3390/plants9030394.

[9] D. Alvarado-Rosales , L. D. L. Saavedra-Romero , Tree damage and mistletoe impact on urban green areas , Revista Árvore ( 2021 ). doi: 10 .1590/1806-908820210000030.

[10] L. Skrypnik , P. Maslennikov , P. Feduraev , A. Pungin , N. Belov , Specific features of the response of the antioxidant system of urban trees to mistletoe infection, E3S Web of Conferences ( 2021 ). doi: 10 .1051/e3sconf/202129102013.

[11] E. Sreenivasan , Occurrence of mistletoe (loranthus spp.) infestation on garden croton (codiaeum variegatum) and other host trees ( 2020 ). doi: 10 .33564/ijeast. 2020 . v04i10 . 010 .

[12] M. Miraki , H. Sohrabi , P. Fatehi , M. Kneubuehler , Detection of mistletoe infected trees using uav high spatial resolution images , Journal of Plant Diseases and Protection 128 ( 2021 ) 1679 - 1689 . doi: 10 .1007/s41348-021-00502-6.

[13] P. A. Mejia-Zuluaga , L. Dozal , J. C. Valdiviezo-N. , Genetic programming approach for the detection of mistletoe based on uav multispectral imagery in the conservation area of mexico city , Remote Sensing 14 ( 2022 ). doi: 10 .3390/rs14030801.

[14] A. Abdollahnejad , D. Panagiotidis , Tree species classification and health status assessment for a mixed broadleaf-conifer forest with uas multispectral imaging , Remote. Sens . 12 ( 2020 ) 3722 . doi: 10 .3390/rs12223722.

[15] A. Safonova , Y. Hamad , E. Dmitriev, G. Georgiev , V. Trenkin , M. Georgieva , S. Dimitrov , M. Iliev , Individual tree crown delineation for the species classification and assessment of vital status of forest stands from uav images , Drones ( 2021 ). doi: 10 .3390/drones5030077.

[16] P. A. Mejia-Zuluaga , L. F. Dozal-García , J. C. Valdiviezo-Navarro , Detection of phoradendron velutinum implementing genetic programming in multispectral aerial images in mexico city , Lecture Notes in Geoinformation and Cartography ( 2022 ) 109 - 129 . doi: 10 .1007/978-3- 030 -98096- 2 _ 9 .

[17] L. Hui , H. Baoxin , L. Qian , J. Linhai , Cnn-based individual tree species classification using high-resolution satellite imagery and airborne lidar data , Forests 12 ( 2021 ).

[18] Z. X. , T. D. , M. L. , G. Xia , Z. L. , X. F. , F. F. , Deep learning in remote sensing: a comprehensive review and list of resources , IEEE Geoscience and Remote Sensing Magazine 5 ( 2017 ) 8 - 36 .

[19] V. Mosin , R. Aguilar , A. Platonov , A. Vasiliev , A. Kedrov , A. Ivanov , Remote sensing and machine learning for tree detection and classification in forestry applications , Proc. SPIE 11155 , Image and Signal Processing for Remote Sensing XXV, 111550F ( 2019 ). doi: 10 .1117/12.2531820.

[20] M.-C. Jutras-Perreault, T. Gobakken , E. Naesset, H. Ørka , Comparison of diferent remotely sensed data sources for detection of presence of standing dead trees using a tree-based approach , Remote Sensing 15 ( 2023 ) 2223 . doi: 10 .3390/rs15092223.

[21] K. P. Ferentinos , Deep learning models for plant disease detection and diagnosis, Computers and Electronics in Agriculture 145 ( 2018 ) 311 - 318 . doi: 10 .1016/j.compag. 2018 . 01 .009.

[22] L. Muñoz-Gutiérrez , R. Pérez-Miranda , J. Reséndiz-Martínez , R. Reyes-Robles , Caracterización de árboles de riesgo en el parque nacional viveros de coyoacán, ciudad de méxico, Revista Mexicana de Ciencias Forestales 13 ( 2022 ).

[23] P. A. Mejia-Zuluaga , J. C. V. -N. B , L. Dozal, Texture descriptors and machine learning algorithms for mistletoe detection in urban forests using multispectral imagery, Remote Sensing for Agriculture, Ecosystems, and Hydrology XXV 12727 ( 2023 ). doi: 10 .1117/12.2684136.

[24] S. Natesan , C. Armenakis , U. Vepakomma, Resnet-based tree species classification using uav images, in: The International Archives of the Photogrammetry , Remote Sensing, and Spatial Information Sciences, volume XLII-2/W13 , 2019 .

[25] G. Huang , Z. Liu , G. Pleiss , L. Maaten , K. Weinberger , Convolutional netwroks with dense connectivity , IEEE Transactions on Pattern Analysis and Machine Intelligence ( 2019 ) 8704 - 8716 .

[26] T. Kattenborn , J. Leitlof , F. Schiefer , S. Hinz , Review on convolutional neural networks (cnn) in vegetation remote sensing , ISPRS Journal of Photogrammetry and Remote Sensing 173 ( 2021 ) 24 - 49 . doi: 10 .1016/j.isprsjprs. 2020 . 12 .010.

[27] A. Dimou , D. Ataloglou , K. Dimitropoulos , F. Álvarez , P. Daras , Lds-inspired residual networks , IEEE Transactions on Circuits and Systems for Video Technology 29 ( 2019 ) 2363 - 2375 . doi: 10 . 1109/TCSVT. 2018 . 2869680 .