Image classification has been one of the most popular tasks in Deep Learning, seeing an abundance of impressive implementations each year. However, there is a lot of criticism tied to promoting complex architectures that continuously push performance metrics higher and higher. Robustness tests can uncover several vulnerabilities and biases which go unnoticed during the typical model evaluation stage. So far, model robustness under distribution shifts has mainly been examined within carefully curated datasets. Nevertheless, such approaches do not test the real response of classifiers in the wild, e.g. when uncurated web-crawled image data of corresponding classes are provided. In our work, we perform ifne-grained classification on closely related categories, which are identified with the help of hierarchical knowledge. Extensive experimentation on a variety of convolutional and transformer-based architectures reveals model robustness in this novel setting. Finally, hierarchical knowledge is again employed to evaluate and explain misclassifications, providing an information-rich evaluation scheme adaptable to any classifier.
Even though so much efort is invested to perpetually improve model performance by
employing more and more refined architectures and techniques, inevitably increasing the demand for
computational resources necessary for training, there are still some open questions regarding the
ability of such models to properly handle distribution shifts. Distribution shifts refer to testing
an already trained model on a data distribution that diverges from the one the model was trained
on. The analysis of distribution shifts has gained interest in recent years [
So far, there is no approach testing image classification ’in the wild’, where uncurated images corresponding to pre-defined dataset labels are encountered. We argue that this is a real-world user-oriented scenario, where totally new images corresponding to ImageNet labels need to be appropriately classified. For example, an image of a cat found on the web may significantly difer from ImageNet cat instances, even when popular distribution shifts are taken into account. Even though a human can identify a cat present in an image with satisfactory confidence, we question whether an image classifier can do so; the unrestricted space of possible variations of uncurated images demands advanced generalization capabilities to properly understand the real discriminative characteristics of an ImageNet class without getting distracted from extraneous features.
The problem of classification ’in the wild’ becomes even more dificult when fine-grained classification needs to be performed, as distinguishing between closely related categories relies on detailed discriminative characteristics, which may be less prevalent in uncurated settings. For example, siamese and persian cat races present many visual similarities, increasing the potential risk of learning and reproducing dataset biases, especially when distribution shifts are present. We can attribute this risk to the fact that existing classifiers lack external or domain knowledge, which can help humans discriminate between closely related categories.
To sum up, in our current paper we aspire to answer the following questions:
1. How do diferent models, pre-trained on ImageNet or web images, behave on uncurated
image sets crawled from Google images (given ImageNet labels as Google queries)? We
target this question by producing a novel natural distribution shift based on uncurated
web images upon which we evaluate various image classifiers.
2. How does hierarchical knowledge help with evaluating classification results since several
ImageNet categories are hierarchically related? We attempt to verify to which extent
the assumption that the lack of external knowledge limits the generalization capabilities of
classifiers holds. Thus, we leverage WordNet [
Our code can be found at https://github.com/marialymperaiou/classification-in-the-wild.
Image classifiers With the outburst of neural architectures for classification tasks, Computer
Vision has been one of the fields most benefited from recent developments. Convolutional
classifiers (CNN) is a well-established backbone, with first successful endeavors [
The general workflow of our method (Figure 1) consists of three stages. First, the dataset should be constructed by gathering common terms (queries) and their subcategories which exist as ImageNet classes. Images corresponding to those terms are crawled from Google search. In the second stage, various pre-trained classifiers are utilized to classify crawled images. The hierarchical relationships between the given classes are reported to enrich the evaluation process. Finally, all semantic relationships between misclassified samples are gathered to extract explanations and quantify how much, falsely predicted classes, diverge from their ground truth.
Dataset creation We start by gathering user-defined common words regarding visual
concepts as queries, which will act as starting points towards extracting subcategories. The WordNet
hierarchy [
The filtered categories of among the initial query are provided as search terms to a web crawler suitable for searching Google images. We set a predefined threshold for the number of Google images returned so that we evaluate classifiers on categories containing almost equal numbers of samples. This is necessary since some popular categories may return way more Google images compared to others. We will experiment with several values of k, thus influencing the tradeof between relevance to the keyword and adequate dataset size. The retrieved images comprise a labeled dataset , with the keywords as labels.
Classification We consider a variety of image classifiers to test their ability for fine-grained
classification on uncurated web images. We commence our experimentation with
convolutionalbased models as baselines, which have generally been considered to be less robust against
distribution shifts and other perturbations [
For each model, we perform inference on the crawled images that constitute our dataset, as explained in the previous paragraph. We implement a rich evaluation scheme to capture various insights of the classification process. Accuracy is useful as a benchmark metric to compare our findings with expected classification results. WordNet similarity functions ofer valuable information about misclassifications; for example, let’s assume that the true label of a sample is ’cat’ and the classifier predicts the label ’dog’ in one case and the label ’airplane’ in another case. Intuitively, we hypothesize that a ’cat’ is more closely related to a ’dog’ than an ’airplane’ since they are both animals. This human intuition is reflected in the WordNet hierarchy, thus assigning a diferent penalty depending on the concept relevance within the hierarchy.
This concept-based evaluation can be realized using the following WordNet functions: path similarity, Leacock-Chodorow Similarity (LCS), and Wu-Palmer Similarity (WUPS). Path similarity evaluates how similar two concepts are, based on the shortest path that connects them within the WordNet hierarchy. It can provide values between 0 and 1, with 1 denoting the maximum possible similarity score. LCS also seeks for the shortest path between two concepts but additionally regards the depth of the taxonomy. Specifically, equation 1 mathematically describes LCS between two concepts 1 and 2: = − log ℎ(1, 2) 2 · (1) where ℎ(1, 2) denotes the shortest path connecting 1 and 2 and refers to the taxonomy depth. Higher LCS values indicate higher similarity between concepts. WUPS takes into account the depth that the two concepts 1 and 2 appear in WordNet taxonomy and the depth of their most specific common ancestor node, called Least Common Subsumer. Higher WUPS scores refer to more similar concepts. For each of the path similarity, LCS, and WUPS metrics we obtain an average value over the total number of samples of the constructed dataset .
Moreover, we report the percentage of sibling concepts among misclassifications. Two concepts are considered to be siblings if they share an immediate (1 hop) parent. For example, the concepts ’tabby cat’ and ’egyptian cat’ share the same parent node (’domestic cat’). It is highly likely that a classifier is more easily confused between two sibling classes, thus providing false positive (FP) predictions closely related to the ground truth (GT) label. Therefore, a lower number of siblings denotes reduced classification capacity compared to models of higher siblings percentage. Explanations are provided during the evaluation stage, aiming to answer why a pre-trained classifier cannot correctly classify uncurated images belonging to a class .
FP predictions contain valuable information regarding which classes are confused with the GT. The per-class misclassification frequency (MF) refers to the percentage of occurrences of each false positive class within the total number of false positive instances. Thus, given a dataset with classes, as the ground truth class and as one of the false positive classes, the misclassification frequency for the → misclassification is:
= = = ∑︁ =0 · 100% (2) scores can be extracted for all ̸= FP classes so that the most influential misclassifications are discovered. Higher scores denote some classifier tendency to choose the FP class over the GT one, therefore indicating either a classifier bias or an annotation error in the dataset. Specifically, a classifier bias refers to consistently classifying samples from class as samples of class , given that the annotation is the best possible. Of course, such a requirement cannot be always satisfied, especially when expert annotators are needed, as may happen in the case of fine-grained classification. On the other hand, since our explainable evaluation approach is able to capture such misclassification patterns, it is not necessary to attribute the source of misclassification beforehand. Human annotators can be employed at a later stage, identifying and verifying the source of misclassifications.
In all following experiments, we selected a threshold of =50 crawled images per class. We will present results on a random initial query as a proof-of-concept to demonstrate our findings. For this reason, we provide the query ’cat’, which returns the following WordNet hyponyms (also corresponding to ImageNet labels):
={’angora cat’, ’cougar cat’, ’egyptian cat’, ’leopard cat’, ’lynx cat’, ’persian cat’, ’siamese cat’, ’tabby cat’, ’tiger cat’}
The same experimentation can be replicated for other selected queries, as long as they can be mapped on WordNet.
We leveraged the following CNN classifiers: VGG16/19, [
Other results that can be extracted from Table 1 is that some categories can be easily classified (’siamese cat’, ’lynx cat’, ’cougar cat’, ’persian cat’, ’cat’) contrary to others (’tabby cat’, ’tiger cat’, ’egyptian cat’, ’leopard cat’, ’angora cat’). Since we have no specific knowledge of animal species, we will once again leverage WordNet to obtain explanations regarding this behavior. Sibling percentages ofer a first glance at the degree of confusion between similar classes in the fine-grained setting. For example, even though ’siamese cat’ and ’cougar cat’ classes demonstrate high accuracy scores, we observe a completely diferent behavior regarding the sibling percentages: most CNN classifiers return some sibling false positives for ’siamese cat’ ground truth label, while the opposite happens for the ’cougar cat’ ground truth label, which mostly receives zero sibling misclassifications. This behavior indicates that for ’siamese cat’ if a sample is misclassified, it is likely that it belongs to a conceptually similar class, while for ’cougar cat’ misclassifications, false positives belong to more semantically distant categories.
Regarding model capabilities, we observe that for both ’siamese’ and ’cougar cat’ classes, all ResNet50 false positives belong to non-sibling classes, contrary to EficientNet false positives, which all belong to sibling classes. By also looking to other categories, we observe that in general, EficientNet achieves a higher sibling percentage compared to ResNet50, meaning that EficientNet misclassifications are more justified compared to ResNet50 misclassifications.
The following transformer-based image classifiers were used: ViT [
As for model performance, we examine sibling percentage apart from exclusively evaluating accuracy. The behavior of transformer-based models regarding sibling misclassification is harder to be interpreted compared to CNN models, because models that return high sibling percentages for some categories may present low sibling percentages on other categories and vice versa. For example, BeiT scores low on sibling percentages for ’tabby cat’ (3.45%), ’siamese cat’ (0%) and ’persian cat’ (10%) compared to other models for the same classes; on the other hand, it returns best sibling scores for ’leopard cat’ (78.72%), ’tiger cat’ (22.45%) and ’egyptian cat’ (22.50%). More results about the explainability of results are provided in Section 4.3.
In Tables 3,4 & 5 we report the top-3 misclassifications per ground truth (GT) category and per model, as well as the misclassification frequency (MF) for each false positive (FP) label. GT column refers to cat species exclusively, even if the word ’cat’ is omitted (for example, ’tiger’ GT entry refers to ’tiger cat’). We highlight with red irrelevant FP classes, which are semantically distant compared to the GT label, while misconceptions involving sibling classes are highlighted with blue. Moreover, magenta indicates that an FP is actually an immediate (1 hop) hypernym of the GT. Due to space constraints, we present here all transformer-based models, but only a subset of the CNN models tested in total; more results can be found in the Appendix.
Interestingly, we can spot some surprising frequent misconceptions, such as confusing cat species with the ’mexican hairless’ dog breed. For CNN classifiers, we spot this peculiarity for all models under investigation: 10.53% of ResNet50 FP for ’egyptian cat’ GT label belong to the ’mexican hairless’ class; the same applies to 14.29% of ResNet101 FP, 18.18% of ResNet152 FP
GT tabby angora
lynx siamese
tiger persian cougar leopard egyptian cat
FP and 8.33% of VGG16 FP. More animals such as ’wallaby’, ’jaguar’, ’sea lion’, ’cheetah’, ’arctic fox’, ’coyote’ etc appear as frequent FPs.
For transformer models, the ’egyptian cat’ → ’mexican hairless’ abnormality is observed for all
classifiers when ’egyptian cat’ GT label is provided, resulting in the following ’mexican hairless’
FP percentages: 26.67% for CLIP, 10% for BeiT, 15.62% for DeiT, 15.38% for xRegNet, 20.83% for
Swin, and 16.33% for ViT. Obviously, regardless of whether the CNN or transformer classifier is
being used, images of ’egyptian cats’ are often erroneously perceived as ’mexican hairless dogs’.
A qualitative analysis between ’egyptian cat’ images and ’mexican hairless dog’ images indicates
that these animals are obviously distinct, even though they present similar ear shapes and rather
hairless, thin bodies. Therefore, we can assume that the transformer-based classifiers are biased
towards texture, verifying relevant observations reported for CNNs [
Even more surprising are misclassifications not including animal species. For example, CNN classifiers predict ’web site’ instead of ’tabby cat’, ’hatched’ instead of ’persian cat’, ’barbershop’ instead of ’cat’, ’menu’ instead of ’cougar’ etc. All ResNet50/101/152 and VGG16 make at least one such misclassification, something that highly questions which features of cat species contribute to such predictions.
Misclassifications involving non-animal classes using transformers (Tables 4, 5) provide the following interesting abnormalities: ’cat’ is classified as ’fur coat’ for 50% of the FP instances using DeiT. This non-negligible misclassification rate once again verifies the aforementioned texture bias. In a similar sense, xRegNet classifies ’egyptian cat’ images as ’mask’ and as ’comic book’ 7.69% of the FPs respectively. Such categories had also appeared in CNN misclassifications. We cannot provide a human-interpretable explanation about the ’mask’ misclassification, since the term ’mask’ may refer to many diferent objects. We hypothesize that ’mask’ ImageNet instances may contain carnival masks looking similar to cats, therefore the lack of context confused xRegNet. ’Comic book’ appears 9.38% of the times an ’egyptian cat’ image is misclassified by DeiT, 33.33% of the times a ’cat’ photo is misclassified by xRegNet, and 16.67% of the times an ’egyptian cat’ is misclassified by Swin. This can be attributed to the fact that crawled images may contain cartoon-like instances, which cannot be clearly regarded as cats. Other interesting misclassifications involving irrelevant categories are ’cat’ →’washer’ (25% of FPs using ViT), ’leopard cat’→’web site’ (2.27% of FPs using ViT, 15% of FPs using DeiT), ’persian cat’→’plastic bag’ (25% of FPs using ViT), ’cat’→’jersey’ (25% of FPs using Swin), ’egyptian cat’→’table lamp’ (8.33% of FPs using Swin), ’cat’→’tub (33.33% of FPs using xRegNet), and others.
An interesting observation revolves around the ’egyptian cat’ label. For CNN models, almost all top-3 FP of ’egyptian cat’ GT label correspond to irrelevant ImageNet categories. On the contrary, ’tabby cat’, ’angora cat’, and ’tiger cat’ present more sensible FPs, which usually involve sibling categories (highlighted with blue). As for transformer models, we observe that ’egyptian cat’ label is always being confused with at least one irrelevant ImageNet category, while ’angora cat’ is only confused with other cat species, and not with conceptually distant classes. Thus, ’egyptian cat’ crawled images seem to contain some misleading visual features that frequently derail the classification process. Indeed, when viewing ’egyptian cat’ crawled images, some of them are drawings or photos of cat souvenirs; however, misconceptions such as ’table lamp’ or ’armadillo’ cannot be visually explained by human inspectors, unraveling more questions on the topic. A comparison between CNN classifiers (Table 3) and transformer-based classifiers (Table 4, 5) denotes that transformers are more capable of retrieving similar categories to the GT; this becomes obvious by observing the higher number or irrelevant misclassifications highlighted with red for CNNs, compared to transformer results.
By combining Tables 3, 4 & 5 with Tables 1& 2, we obtain some very interesting findings: how are low classification metric scores connected to the relevance between misclassified categories? We start with categories presenting low accuracy scores (’tabby cat’, ’tiger cat’, ’egyptian cat’), and we compare them with categories ofering frequent extraneous misclassifications (’egyptian cat’ and ’cat’, followed by ’tabby cat’ and ’lynx’). Classifying ’egyptian cat’ images both yields low classification scores and returns irrelevant false positives. On the other hand, even though ’cat’ images present high accuracy scores, misclassifications are highly unrelated when they happen. ’Tiger cat’ scores low in accuracy, however, misclassifications are rather justified, since other cat species are returned. Surprisingly, ’tiger cat’ also scores low in siblings percentage, indicating that false positives are not immediately related to the GT ’tiger cat’ class. In this case, we assume that false positives (’egyptian cat’, ’tabby cat’, ’leopard cat’ etc) belong to more distant relatives of the ’tiger cat’ concept class, even though bearing some similar features.
Overall, throughout this analysis we prove that classification accuracy is unable to reveal the whole truth behind the way classifiers behave; to this end, knowledge sources are able to shed some light on the inner workings of this process. By analyzing a constraint family of related ImageNet labels (cat species) we already disentangled the classification accuracy from the classification relevance: false positives can be highly relevant to the ground truth (such as ’tiger cat’ misclassifications) or not (’cat’ misclassifications). We, therefore, argue that ifne-grained classification also demands ifne-grained evaluation , which can provide insightful information when driven by knowledge. The human interpretable insights of Tables 4, 5 are cat going to be quantified and verified in the next Section.
The aforementioned claim regarding the need for fine-grained evaluation is supported by demonstrating results using knowledge-driven metrics based on conceptual distance as provided by WordNet (Tables 6& 7). Since higher path similarity/LCH, WUPS scores are better, we denote with bold best (higher) scores for each category.
By comparing path similarity, LCH, and WUPS metrics across categories, we observe that categories having a large number of irrelevant FP (marked in red in Tables 4, 5), such as ’cougar cat cat’ and ’lynx cat’, followed by ’egyptian cat’ and ’cat’ also present low knowledge-driven metric scores in Tables 6, 7, as expected. Other categories such as ’angora cat’, ’leopard cat’, and ’tiger cat’ that present misclassifications of related (sibling or parent) categories also present higher knowledge-driven metric scores. Therefore, we can safely assume that employing knowledge-driven metrics for evaluating fine-grained classification results is highly correlated with human-interpretable notions of similarity and therefore trustworthy.
Model performance is rather clear when examining CNN classifiers. EficientNet achieves predicting more relevant FP images compared to other classifiers for the majority of the categories. On the other hand, it is harder to draw a similar conclusion for Transformer-based classifiers, as diferent models perform better for diferent categories; however, compared to CNN classifiers the results of knowledge-driven metrics are the same or higher for most categories. Even though this diference is not impressive, transformer-based models showcase an improved capability of predicting more relevant classes, when failing to return the GT one.
In this work, we implemented a novel distribution shift involving uncurated web images, upon which we tested convolutional and transformer-based image classifiers. Selecting closely related categories for classification is instructed by hierarchical knowledge, which is again employed to evaluate the quality of results. We prove that accuracy-related metrics can only scratch the surface of classification evaluation since they cannot capture semantic relationships between misclassified samples and ground truth labels. To this end, we propose an explainable, knowledge-driven evaluation scheme, able to quantify misclassification relevance by providing the semantic distance between false positive and real labels. The same scheme is also used to compare the classification capabilities of CNN vs transformer-based models on the implemented distribution shift. As future work, we plan to extend our analysis to more query terms in order to examine the extend of our current findings, and also combine the uncurated image classification setting with artificial corruptions to enhance our insights.
The research work was supported by the Hellenic Foundation for Research and Innovation (HFRI) under the 3rd Call for HFRI PhD Fellowships (Fellowship Number 5537).
In Table 8, we present the continuation of the results present in Table 3 for the rest of the CNN models presenting non-zero accuracy. It becomes evident that the capacity of the classifier plays an important role in identifying relevant FP: MobileNetV2, which already demonstrated low accuracy scores, also fail to retrieve semantically related FP classes. This can be easily observed from the numerous red entries corresponding to this model.
Other than that, the results agree with the observations analyzed in Table 3, where ’egyptian cat’ label demonstrated many irrelevant FP, contrary to ’tabby cat’ or ’tiger cat’ labels. tabby angora
lynx siamese
tiger persian cougar leopard egyptian
cat siamese
tiger persian cougar leopard egyptian
cat tabby angora
lynx siamese
tiger persian cougar leopard egyptian
cat egyptian cat persian cat egyptian cat whippet tabby cat lynx lynx egyptian cat
lynx fur coat comic book shower curtain west highland white terrier shower curtain
zebra spotlight comic book
knot windsor tie shower curtain
tiger cat persian cat egyptian cat egyptian cat egyptian cat tabby cat tiger cat egyptian cat comic book macaque tiger cat persian cat tabby cat egyptian cat tabby cat siamese cat
web site egyptian cat mexican hairless fur coat
MF
Top-2
FP tiger cat arctic fox coyote fur coat egyptian cat pekinese coyote lynx mask snow leopard
mask window screen tiger sock mask shower curtain mask tiger theater curtain window screen egyptian cat egyptian cat tiger cat
tabby cat lynx mexican hairless mexican hairless 20.00% 10.87% 20.00% 16.67% 16.28% 25.00% 18.18% 16.00% 5.41% 11.11%
Top-3 FP lynx egyptian cat timber wolf egyptian cat
tiger fur coat timber wolf
jaguar book jacket mousetrap
sock spotlight trafic light mask maze ant theater curtain
mask spotlight
teddy persian cat tabby cat tabby cat
tiger tiger cat lampshade indigo bunting
MF