Insects have the largest percentage of all living organisms worldwide, playing a pivotal role in maintaining essential ecosystem services such as pollination, pest control, nutrient cycling, and food provisioning. However, recent studies have reported alarming declines in insect populations globally, highlighting an urgent need for automated methods to estimate and quantify these populations, to better understand the reasons of their decline and to take proper measures. The ability to automatically estimate insect populations is crucial for shaping appropriate environmental policies. Advances in AI and computer vision techniques are revolutionizing the study of insects through non-invasive camera traps. However, the diversity of insects, close resemblances of many species, and multi-level taxa classification remain significant challenges in image-based insect monitoring. In this work, we propose TaxaNet a deep learning model for multi-level insect taxa classification, utilizing a pretrained EficientNet as a feature extractor, followed by six classification blocks. Each block predicts one of the six taxonomic levels: Kingdom, Class, Order, Family, Genus, and Species. This hierarchical design and the loss function improves lower-level taxa predictions by leveraging the higher-level features available. A class-weighted hierarchical loss function, alongside the standard class-wise loss, allows the model to understand the relationships between taxonomic levels while maintaining classification accuracy. Trained on the Diopsis insect camera trap dataset containing 31,000 training images and 7,900 test images, the model achieved an average precision of 0.85 and a recall of 0.86 across five taxonomic levels. These results demonstrate the efectiveness of our approach in harnessing multi-level insect taxonomy to achieve multi-level insect classification.
Insects, constitute the largest group of organisms on Earth and they play critical roles in sustaining
essential ecosystem services such as pollination, pest control, nutrient cycling, and food provisioning [
Traditional methods for insect population monitoring include manual counting and identification, but
they are labor-intensive, time-consuming, and prone to human error [
In this work, we propose TaxaNet, which is a computer vision model based on deep learning for
multi-level insect classification that utilizes hierarchical classification properties of insects. Our TaxaNet
model is built upon a pretrained EficientNet architecture, which serves as a feature extractor, followed
by a series of classification blocks designed to predict taxonomic levels such as Kingdom, Class, Order,
Family, Genus, and Species. Our main contribution is the introduction of a hierarchical loss function
that complements the standard class-wise loss which is mainly used in literature, enhancing a model’s
ability to capture the hierarchical relationships between levels, thereby guiding the model to make
more accurate and consistent predictions across the taxonomic hierarchy [
Deep learning (DL) has become an essential technique for classifying insects and other biological
organisms, particularly with the rise of convolutional neural networks (CNNs). Traditional methods often
employ "single-level" classification models that do not consider the inherent hierarchical relationships
between taxa, as seen in studies such as [
In the context of animal monitoring, taxonomic hierarchical classification ofers a more nuanced approach by considering the evolutionary relationships between species, which are typically structured in a tree-like hierarchy. In the context of the classification of insects and other arthropods, hierarchical classification has been explored using Deep Learning (DL) models like YOLOv3 [ 17], R-CNN [18], Faster R-CNN [19], SSD [20], and EficientDet [ 21]. The work in [22] highlights the efectiveness of using hierarchical classification in the context of automated image analysis and deep learning, particularly for challenging biological datasets that often feature class imbalance, high similarity between species, and limited training examples. The study employed a two-step process where YOLOv3 was first used to detect and classify arthropods into broad taxonomic categories, followed by a separate model for species-level classification. This approach, tested on a custom dataset with 22 classes of arthropods, demonstrated significant improvements over a global detector trained without hierarchical classification, achieving a mAP of 75% compared to 48%. Despite its success, the method leverages hierarchical information in a segmented manner, with separate models for each taxonomic rank, rather than fully integrating hierarchical relationships within a single model.
The concept of integrating hierarchical classification within a single model has gained attention due to its potential to enhance the predictability performance by harnessing multiple taxonomic levels at the same time. This is achieved by incorporating class labels at each level of the hierarchy within a unified model. A crucial component in this approach is the design of a loss function that accounts for the hierarchical dependencies, penalizing incorrect predictions more severely when they occur higher in the hierarchy. In the realm of hierarchical classification, various loss functions have been designed to leverage the structured information inherent in taxonomies. For instance, triplet loss has been explored to capture hierarchical relationships by ensuring that examples closer in the hierarchy are embedded more closely together. However, it faces challenges such as requiring careful construction of triplets and being limited in the extent of hierarchical information utilized at each iteration. To address these limitations, rank-based loss functions have been introduced, which generalize well across hierarchies with any number of levels and can handle incomplete hierarchical labels efectively [23].
For instance, [
LaGrassa et al. [25] introduced a hierarchical classification architecture using deep linear layers
with cross-entropy and center loss, optimizing local and global hierarchical relationships while
penalizing hierarchy violations. Their model was tested on two datasets: VQA-Med 2019 and the Animals
Taxonomy8 dataset, which features a three-level taxonomic structure. The results highlighted the
efectiveness of their hierarchical loss combined with cross-entropy and center loss in improving
multilevel classification tasks. In another approach, Bjerge et al. [
Our approach difers from [
Based on the literature review, our contributions can be summarized as follows:
1. We propose a pre-processing method to address datasets with high imbalance among insect
classes by employing a combination of under-sampling and over-sampling techniques. This
approach efectively balances the dataset, ensuring that it captures the diversity across all classes
while ensuring adequate representation for minority classes.
2. Inspired by the work in [
Our approach has been efective, as shown by our position as the leader in the classification phase of the ARISE Diopsis Challenge. This result shows that our methods and model can handle complex hierarchical classification tasks well.
The Insect classification data set from Arise Biodiversity - Diopsis challenge dataset [
The dataset presents several challenges, including the following: a) a significant imbalance in the number of training examples per species, b) a relatively poor image quality due to trade-ofs between camera cost and resolution. To address these challenges, we implemented a combination of data under-sampling and over-sampling techniques, explained below. 1. Undersampling: To address the issue of data imbalance, we employed a data under-sampling technique to reduce redundancy while maintaining the diversity of the dataset. The RGB images were subjected to Principal Component Analysis (PCA) to reduce dimensionality, retaining the ifrst four principal components, covering a variance of 0.8193. These four components were then used to create clusters using K-Means clustering, with the number of clusters experimentally determined to be 5. Within each cluster, the distance of samples to their respective cluster centers were calculated as in Fig:2. To maximize diversity and eliminate redundancy, samples that were close to each other within the same cluster were removed. This undersampling technique reduced the dataset size from 39,446 to 27,273, ensuring that we retained the most diverse and representative samples. 2. Oversampling: To further address class imbalance, we implemented additionally an oversampling technique based on weighted random sampling. This approach was applied at the level 4 of the taxonomic hierarchy (Order), where the dataset contains the largest number of classes. The core idea behind this method was to assign sampling probabilities inversely proportional to the class frequencies. Specifically, class weights were calculated as the inverse of the total number of items per class, ensuring that under-represented classes were given higher sampling probabilities. This technique efectively balanced the dataset by increasing the representation of minority classes without altering the original data distribution. By generating a sample set with replacement, we maintained the total number of samples while ensuring a more uniform class distribution across the dataset.
These preprocessing steps helped to address data imbalance and was crucial in enhancing the model’s ability to learn efectively from all classes, particularly those with fewer examples, thereby improving the overall classification performance. 3.2. Model We named our classification model as TaxaNet. It is based on the EficientNet-B0 architecture [ 26], a convolutional neural network (CNN) known for its eficiency in both computational resources and accuracy. EficientNet-B0 was pretrained on the ImageNet dataset, to exploit transfer learning in our specific insect classification task. TaxaNet leverages the EficientNet-B0 backbone, which outputs a 1280-dimensional feature vector after passing through the series of its convolutional layers. To adapt this backbone for our multi-level classification task, we modified the classifier head of the network. The original fully connected layer of EficientNet-B0, which outputs a 1000-dimensional vector for ImageNet classification, was replaced by a custom classifier. This custom classifier is a sequential block that first reduces the feature vector to 512 dimensions using a linear layer, followed by a ReLU activation and a dropout layer with a dropout rate of 0.5, to prevent overfitting. Finally, another linear layer maps the 512-dimensional vector to the desired number of classes.
Given the hierarchical nature of the dataset, where class labels are organized across multiple taxonomic levels, the TaxaNet model is designed to output predictions corresponding to these levels, such as species, genus, and family. The output vector is split into segments corresponding to these diferent taxonomic levels by connecting the feature vector to separate fully connected layers, each representing a diferent taxonomic level. The size of each segment, and the number of neurons in each fully connected layer, is determined by the number of classes at that particular taxonomic level, with the size directly corresponding to the one-hot encoded labels. This one-hot encoding of class labels ensures that each segment accurately represents the classes at its respective level, facilitating precise class predictions. This design not only ensures eficient computation but also efectively captures the hierarchical relationships between diferent classes, enhancing the model’s performance across various taxonomic levels. Additionally, the architecture is specifically tailored to handle a hierarchical loss function, which will be discussed in Sec: 3.3, ensuring that the training process is aligned with the hierarchical structure of the dataset.
The loss function used in TaxaNet is designed to handle the hierarchical nature of the classification
task, ensuring that the predictions are accurate at each taxonomic level and consistent across levels. We
implemented a combination of Cross Entropy Loss and a Hierarchical Dependency Loss, inspired by the
approach proposed in [
As a baseline, we initially trained our model using only Cross Entropy Loss, a standard choice for classification tasks. For each taxonomic level , the Cross Entropy Loss lloss is defined as: lloss = − ∑︁ log(ˆ ) =0 where is the true label and ˆ is the predicted probability for the th sample. The predicted probabilities ˆ are derived from a softmax function applied to the model’s output at level .
In addition to Cross Entropy Loss, we incorporated a Hierarchical Dependency Loss [
dloss = exp() − 1, = {1 if ˆ ̸→ ˆ− 1, 0 otherwise}
Here, checks if the predicted label at level is a child of the predicted label at level − 1. A violation of this relationship results in a penalty, guiding the model to respect and consider the hierarchical consistency.
The method proposed in [
5 Total Loss = ∑︁ · (Entropy Loss + Dependency Loss)
=1
Where: 1 > 2 > · · · > 5, ensuring higher levels receive more emphasis. After some trialand-error experiments, we chose = [0.25, 0.25, 0.25, 0.15, 0.1] to prioritize higher levels while still considering the lower levels. Entropy and dependency losses at each level are scaled according to these weights.
This approach allows explicit control over the importance of each taxonomic level, which is particularly advantageous given the incomplete labeling in our dataset. The flexibility of this method helps the model learn efectively from the available data, improving overall classification performance.
We trained the TaxaNet model using the Adam optimizer with a learning rate of 0.001. The training
involved three separate models, each using a diferent loss function as described in the previous section:
the standard Cross Entropy Loss (M-L1), the hierarchical loss proposed by [
The model trained with Cross Entropy Loss (M-L1) exhibits the lowest performance, in terms of
precision and recall at deeper taxonomic levels. This outcome is well expected, as Cross Entropy Loss
treats each class independently, without considering the hierarchical relationships between classes. As
a result, the M-L1 may accurately predict lower-level classes but fail to maintain consistency across
levels, leading to reduced overall performance. In contrast, the models trained with hierarchical loss
functions (M-L2 and 3) demonstrated significantly higher performance. The hierarchical loss proposed
by [
The model trained with our custom hierarchical loss function with decaying weights (TaxaNet, M-L3)
slightly outperformed the hierarchical loss from [
The results in Table 1 are based on a dataset that was split into a 90:10 ratio for training and validation, respectively. While the validation set is slightly smaller than usual, this was a deliberate choice due to the limited data available for training. The aim was to encourage the model to see as many samples as possible during training to improve its accuracy. Despite the small validation set, we performed a more robust evaluation by testing the three models on a private test dataset provided by the ARISE Diopsis Challenge portal. This final evaluation on the private dataset is important for assessing the models’ generalization in real-world challenges.
The results, summarized in Table 2, show the accuracy of our models on the ARISE private test set.
Table 2 includes two key metrics: overall accuracy and taxon distance. Accuracy refers to the percentage
of insects correctly classified, providing an overall measure of model performance, though it tends to be
influenced by the majority classes. Taxon distance, on the other hand, is a custom metric specific to
hierarchical labels. It quantifies the number of edges that need to be traversed within the taxonomic tree
to move from the true taxon to the predicted taxon. This metric accounts for the nature of hierarchical
misclassifications. Lower taxon distances indicate predictions that are closer to the correct classification,
even if not entirely accurate, thus providing a more nuanced view of model performance. As shown in
the table, TaxaNet (Model 3), which utilizes our custom hierarchical loss function with decaying weights,
achieved the highest accuracy and the lowest taxon distance. We currently lead the competition at
the ARISE Diopsis Challenge, outperforming the other participants in both accuracy and taxonomy
distance [
As future work, we intend to further experiment with the loss function by incorporating taxon distance together with the existing entropy and dependency loss, either as a diferentiable term or as a regularization term. The taxon distance, used in the Diopsis challenge, seems to be an interesting metric we intend to investigate integrating into our loss function, as it could potentially improve accuracy. By explicitly penalizing predictions that deviate significantly from the correct classification within the taxonomic hierarchy, we aim to enforce greater hierarchical consistency.
Our work has already been compared with the approach by Bjerge et al. [
We will also work on preprocessing the data to better handle the varying image resolutions in our dataset. Currently, we resize all images to a uniform size, which may result in the loss of important information. Future work will involve exploring techniques that can preserve more detail and improve model performance.
Additionally, we plan to extend our work on new datasets, which can also be incomplete, and focus on other insect species. This will allow us to assess how well TaxaNet generalizes to other problems, challenges, and datasets. A candidate such dataset is one that we are currently collecting at our field sites. This new dataset will present unique challenges and opportunities, particularly in dealing with diverse environmental conditions, insect species, and variations in image quality.
In this work, we proposed a method for hierarchical insect classification using the TaxaNet model, built on top of the EficientNet architecture. Our approach involved data preprocessing based on under- and over-sampling techniques, as well as the proposal of a novel loss function, which is based on a custom hierarchical loss with decaying weights, to improve classification accuracy and maintain consistency across taxonomic levels. The model was trained and evaluated on an incomplete dataset with many challenges, demonstrating superior performance, in comparison to existing approaches.
The experimental results showed that TaxaNet performs particularly well in scenarios where hierarchical relationships between classes are critical and at the same time the datasets have incomplete information in some taxonomical levels. The inclusion of hierarchical dependencies in the loss function proved to be highly efective, with our model achieving higher accuracy and lower taxon distance compared to models trained with standard Cross Entropy Loss. This indicates that TaxaNet not only predicts the correct classes but also respects and harnesses the hierarchical structure, resulting in more meaningful and accurate predictions. As the current leader in the open competition of the ARISE Diopsis Challenge, the proposed method demonstrates significant potential for application in large-scale insect classification and conservation eforts, providing a robust tool for biologists and ecologists to monitor species with greater accuracy and eficiency.
This research work has been partially funded from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 739578 and from the Government of the Republic of Cyprus through the Deputy Ministry of Research, Innovation and Digital Policy.
The authors would like to express their gratitude to Savvas Karatsiolis for his invaluable assistance and insights throughout the project.
The code used in this study is hosted on GitHub at https://github.com/cpadubidri/insect-classification. While the repository is currently private to protect the integrity of an ongoing challenge, it will be made publicly accessible by October 2024 to ensure reproducibility. //onlinelibrary.wiley.com/doi/abs/10.1002/ece3.5921. doi:https://doi.org/10.1002/ece3. 5921. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/ece3.5921. [15] T. Kasinathan, D. Singaraju, S. R. Uyyala, Insect classification and detection in field crops using modern machine learning techniques, Information Processing in Agriculture 8 (2021) 446–457. URL: https://www.sciencedirect.com/science/article/pii/S2214317320302067. doi:https://doi. org/10.1016/j.inpa.2020.09.006. [16] V. Kittichai, T. Pengsakul, K. Chumchuen, Y. Samung, P. Sriwichai, N. Phatthamolrat, T. Tongloy, K. Jaksukam, S. Chuwongin, S. Boonsang, Deep learning approaches for challenging species and gender identification of mosquito vectors, Scientific Reports 11 (2021) 4838. URL: https: //doi.org/10.1038/s41598-021-84219-4. doi:10.1038/s41598-021-84219-4. [17] J. Redmon, A. Farhadi, Yolov3: An incremental improvement, in: arXiv preprint arXiv:1804.02767, 2018. [18] R. Girshick, J. Donahue, T. Darrell, J. Malik, Rich feature hierarchies for accurate object detection and semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 580–587. [19] S. Ren, K. He, R. Girshick, J. Sun, Faster r-cnn: Towards real-time object detection with region proposal networks, in: Advances in Neural Information Processing Systems (NIPS), 2015, pp. 91–99. [20] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, A. C. Berg, Ssd: Single shot multibox detector, in: Proceedings of the European Conference on Computer Vision (ECCV), 2016, pp. 21–37. [21] M. Tan, R. Pang, Q. V. Le, Eficientdet: Scalable and eficient object detection, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 10781–10790. [22] P. Tresson, D. Carval, P. Tixier, W. Puech, Hierarchical Classification of Very Small Objects: Application to the Detection of Arthropod Species, IEEE Access 9 (2021) 63925–63932. URL: https://hal-lirmm.ccsd.cnrs.fr/lirmm-03477540. doi:10.1109/ACCESS.2021.3075293. [23] I. Nolasco, D. Stowell, Rank-based loss for learning hierarchical representations, in: ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2022. URL: http://dx.doi.org/10.1109/ICASSP43922.2022.9746907. doi:10.1109/icassp43922. 2022.9746907. [24] D. Gao, Deep hierarchical classification for category prediction in E-commerce system, in: S. Malmasi, S. Kallumadi, N. Uefing, O. Rokhlenko, E. Agichtein, I. Guy (Eds.), Proceedings of the 3rd Workshop on e-Commerce and NLP, Association for Computational Linguistics, Seattle, WA, USA, 2020, pp. 64–68. URL: https://aclanthology.org/2020.ecnlp-1.10. doi:10.18653/v1/2020. ecnlp-1.10. [25] R. L. Grassa, I. Gallo, N. Landro, Learn class hierarchy using convolutional neural networks, Applied Intelligence 51 (2021) 6622–6632. URL: https://doi.org/10.1007/s10489-020-02103-6. doi:10. 1007/s10489-020-02103-6. [26] M. Tan, Q. V. Le, Eficientnet: Rethinking model scaling for convolutional neural networks, CoRR abs/1905.11946 (2019). URL: http://arxiv.org/abs/1905.11946. arXiv:1905.11946.