Concept drift in machine learning refers to changes in the underlying data distribution over time, which can lead to a degradation in the performance of predictive models. Although many methods have been proposed to detect and adapt to concept drift, efective methods to explain it in a human-understandable manner remain lacking. To address this, we propose the use of neuro-symbolic rules to explain the reason for drift. We applied recent rule extraction methods to convolutional neural networks (CNNs) to shed light on the model's internal behavior and promote interpretability of the outputs, while also proposing two novel automated approaches for semantic kernel labeling. We conducted preliminary experiments to assess the applicability and efectiveness of these rules in explaining concept drift, and the eficacy of the kernel labeling strategies. Under the optimality assumption, our method was able to extract rules that can facilitate the identification of the causes of drift, through either rule inspection or antecedents activation frequencies analysis. Moreover, the proposed strategies for kernel labeling ofer a more reliable and scalable alternatives to the state-of-the-art solutions.
1. Methodology (Section 3): We propose the use of neuro-symbolic rules to explain the reasons for concept drift. By applying rule extraction methods to Convolutional Neural Networks (CNNs), we derive an interpretable set of rules that can be leveraged to formulate hypotheses regarding the causes of drift. We also introduce two scalable approaches for semantic kernel labeling which aim at mitigate the key limitations of the current systems. 2. Experiment (Section 4): We conducted a preliminary experiment using a deep learning classifier trained to predict the gender of individuals from input images. To simulate drift, we removed samples of male individuals wearing earrings from the training data and then introduced such images during inference. This drift led to a noticeable decrease in classifier performance on the drifted samples (accuracy drop = 0.09). Our method successfully extracts human-readable explanations in the form of logical rules, which explain the cause of the classifier’s drift and its performance degradation (e.g., Wearing Earrings ∧ ¬Wearing Lipstick ∧ ¬No Beard → Drift).1 We conclude this work by discussing the limitations of our current approach and experiments, as well as outlining directions for future research (Section 5).
In this section, we first provide background on concept drift and review prior work in the field, with a particular focus on drift explanation (Section 2.1). We then discuss how to extract rules from neural networks, as we propose the use of these rules to explain concept drift (Section 2.2).
Concept drift in machine learning refers to a change in the underlying data distribution over time,
which can lead to a degradation in model performance [
Concept drift localization Drift localization or segmentation techniques aim to identify the drift
data points in the data space (where)–whether a given data point is afected by drift [
In this paper, we focus on drift explanation only, assuming detection and localization have been completed. We target concept drift in CNN for image classification, aiming to provide efective, humanreadable explanations by extracting rules that explain the reasons for changes.
Deep convolutional neural networks (CNNs) have achieved remarkable performance in computer vision tasks, yet their internal decision-making remains largely opaque. To enhance transparency and accountability, research on explainable AI seeks methods that translate complex model behavior into human-understandable forms. One promising direction is rule extraction, which approximates network logic with a set of global, interpretable if–then rules. In contrast to local attribution methods that highlight individual/groups of pixels or neurons [44, 45, 46, 47], rule extraction provides a holistic description of the features and conditions driving model predictions. These rules serve as an interpretable surrogate for the original CNN, enabling practitioners to assess whether the model relies on semantically meaningful patterns or spurious correlations [48]. Early rule-extraction frameworks introduced taxonomies such as pedagogical (treating the network as a black box) and decompositional (leveraging internal structures), along with surrogate algorithms like the C4.5 decision tree [49]. More recent approaches extend these ideas to deep CNNs by training decision trees on the network’s logits or feature activations to derive high-fidelity logical rules. For example, Padalkar et al. [ 50] present NeSyFOLD, a neurosymbolic architecture that replaces the final CNN layers with a response schedule of interpretable rules. Similarly, symbolic rule extraction has been applied to Vision Transformers (ViTs), where sparse attention maps and concept-level features yield human-readable rule sets [51].
In this section, we describe the proposed concept drift explanation method, tailored to CNNs for image classification. We consider the original CNN model as-is and build upon the most recent advances in the field of neuro-symbolic rule extraction from neural networks [ 48, 50]. We assume that concept drift has already been detected and that the samples responsible for it have been identified—drift detection and localization are considered completed. Our objective is to extract neuro-symbolic rules that explain why such drifted samples difer from the normal data distribution, causing a drift.
Consider an image ∈ with ⊂ R3× × and a label ∈ {0, 1} for a binary classification problem. We assume , which represents the main concept , as a composition of multiple concepts cnodeE NCN r
Feature maps
...
Concepts
BCE Loss
: {}=1 → . For instance, in a problem of gender classification, some concepts that may imply the gender Male are the presence of beard, absence of makeup or lipstick (see Section 4). From these concepts, it is possible to derive classification rules that are easily interpretable by humans, such as: Beard ∧ ¬Makeup ∧ ¬Lipstick → Male Similarly, the approach extends to other tasks, such as scene classification, where a given class (e.g., camping) is generally correlated to specific concept presence (e.g., tent, lawn, woods, mountains, etc.).
Pre-trained models, such as CNNs, may demonstrate unpredictable behavior when exposed to out-ofdistribution inputs and data drift, leading to significant performance degradation. From this perspective, we focus on identifying the rule set { }=1 that mimics the behavior of the model, and can highlight in a more interpretable way changes in predictive patterns caused by concept or data drift.
Each rule : ⋀︀
=1 → is satisfied if all antecedents, expressed as positive or negative ¬, evaluate true. In our setting, antecedents represent the presence (or absence) of a specific concept in the image, while the consequent is the final predicted class. Therefore, given a pre-trained model , we aim to extract an interpretable rule-based model ˆ : { }=1 → ˆ ≈ . This enables the identification of the satisfied rule that guides the model’s final decision, and facilitates the detection of potentially failing antecedents–highlighting the corresponding parts of the network.
As stated above, the method considers the original CNN as-is and, starting from the feature maps from the last convolutional layer, it tries to infer the presence or absence of predefined concepts within the original image, producing a binarized concept presence table. Then, a rule extractor algorithm uses the sparse information of this table, pre-computed for the whole training dataset, to derive the rules that approximate the original model behavior. The complete architecture is shown in Figure 1a. Concept extraction and labeling The extraction of concepts is performed using the feature maps from the last convolutional layer of the original CNN. We select this layer because it yields a more semantically rich representation. Typical approaches consist in binarizing the feature maps by computing the norms for all input samples and then applying a specific threshold for each kernel, defined as the mean [ 48] or the weighted sum of the mean and standard deviation [50] of the kernel norms distribution. A label is ultimately attributed to each kernel reflecting the most prominent concept among the images it responds to, by means of visual inspection [48] or by employing a semantic segmentation model [50] to detect the most relevant concept in the image. We propose two alternative approaches for concept extraction and labeling that leverage an external source of knowledge—an Oracle—defining the presence of predefined concepts in the training images. The Oracle can be manually annotated by humans or synthetically generated through pre-trained task-specific models for object detection, semantic segmentation, Visual Question Answering (VQA), or Vision Language Models (VLMs).
The first method (Figure 1b) is inspired by existing approaches, but introduces two major variations in the choice of thresholds for binarization and in the kernel labeling procedure. Specifically, starting from the feature maps { }=1, we extract the kernel activations using the 1-norm { }=1 = {|| ||1}=1, where is the number of kernels. We then compute the point-biserial [52] correlation coeficient = (, ) between the dichotomous variable from the Oracle and the continuous variable representing the kernel norms, pre-computed for the whole training set. We repeat the process for all combinations and then select the pairs of concepts-kernels that exhibit the highest correlation: , {(, )}=1 = {arg max()}=1,=1 , where is the number of concepts in the Oracle. In this way, we find the most representative kernel for each concept while, at the same time, achieving the kernel labeling. Binarization was finally achieved using per-concept percentiles computed on the Oracle to take into account the concept imbalance in the training data, and then the thresholds { }=1 were selected to segment the bimodal norm distributions in a way that preserves the original cardinalities of each split.
The second method (Figure 1c) employs an MLP to infer the presence of concepts from feature maps, in a pure data-driven approach. Specifically, the results of the global average pooling were fed to the new concept head, which outputs binary values, one for each of the predefined concepts. The network was trained to match the Oracle using a BinaryCrossEntropy objective. Although the correlation-based extractor appears to be a viable solution, the second one is generally more accurate. Rule extraction Once the binary concept presence table is available, the set of predicted concepts that define the antecedents, can be used to derive the prediction rules. Similar to [48], we employ a tree-based extraction algorithm, near to C4.5 [49], to derive rules that outline the conditions driving the model decisions. Specifically, to obtain an approximation ˆ of the original model , the algorithm was fitted to follow its predictions ˆ = (). Although, as proposed in [48], the rule extraction could be expanded to multiple layers, we limited the analysis to the final classifier of the CNN, which only exploits the high-level features of the final convolutional layer, as in [ 50]. In addition to the original model, we explored generating drift explanations by extracting rules that explain the behavior of a drift location model, which diferentiates between drifted and non-drifted samples (see Section 4.2). Inference Inference is straightforward, as it consists of extracting the feature maps using the original CNN, identifying the concepts using the Concept extractor module, and finding the satisfied rule based on the combination of activated antecedents.
Listing 1: Rule extracted to approximate the gender classifier (optimal case). For visualization purposes, the maximum depth of the decision tree was set to 5, ensuring a fidelity of about 90%. 1 ¬Wearing_Earrings → ¬Drift (conf=1.00) 2 Wearing_Earrings & ¬Wearing_Lipstick & ¬No_Beard → Drift (conf=1.00) 3 Wearing_Earrings & ¬Wearing_Lipstick & No_Beard & ¬Brown_Hair & ¬Bangs → Drift (conf=0.97) 4 Wearing_Earrings & ¬Wearing_Lipstick & No_Beard & ¬Brown_Hair & Bangs → Drift (conf=0.81) 5 Wearing_Earrings & ¬Wearing_Lipstick & No_Beard & Brown_Hair & ¬Smiling → Drift (conf=0.91) 6 Wearing_Earrings & ¬Wearing_Lipstick & No_Beard & Brown_Hair & Smiling → Drift (conf=0.56) 7 Wearing_Earrings & Wearing_Lipstick & ¬Bushy_Eyebrows & ¬Pointy_Nose & ¬Attractive → ¬Drift (conf=0.78) 8 Wearing_Earrings & Wearing_Lipstick & ¬Bushy_Eyebrows & ¬Pointy_Nose & Attractive → ¬Drift (conf=0.96) 9 Wearing_Earrings & Wearing_Lipstick & ¬Bushy_Eyebrows & Pointy_Nose → ¬Drift (conf=1.00) 10 Wearing_Earrings & Wearing_Lipstick & Bushy_Eyebrows & ¬Pointy_Nose & ¬Wavy_Hair → Drift (conf=0.80) 11 Wearing_Earrings & Wearing_Lipstick & Bushy_Eyebrows & ¬Pointy_Nose & Wavy_Hair → ¬Drift (conf=1.00) 12 Wearing_Earrings & Wearing_Lipstick & Bushy_Eyebrows & Pointy_Nose & ¬Oval_Face → ¬Drift (conf=0.50) 13 Wearing_Earrings & Wearing_Lipstick & Bushy_Eyebrows & Pointy_Nose & Oval_Face → ¬Drift (conf=1.00)
Listing 2: Rule extracted to approximate the drift classifier (optimal case). For visualization purposes, the maximum depth of the decision tree was set to 5, ensuring a fidelity of about 97%.
In this section, we first introduce the experimental setting (Section 4.1). Then, we discuss the drift explanation results obtained by rules extraction (Section 4.2).
Dataset We conduct our experiments using the CelebA dataset [53], which consists of about 202k images and provides human-annotated labels for 40 facial attributes, that can serve as the concept Oracle in the proposed framework.
Task definition and drift simulation We define the main task as a gender classification problem by choosing the attribute Male as the target. We simulate a drift by isolating all instances from the class Male that wear earrings and then injecting them into the data stream. In particular, the dataset was divided into 4 splits preserving the original distribution: Historical train of about 156k samples (used for training the CNN), Historical test of about 19k samples (used for testing the CNN), Datastream of Attribute correlation dataset [53]. (b): Correlation patterns between top-1 binarized kernels and attributes. samples (2), and data stream with 100% of drift samples (3).
Per-concept Acc ↑ Method Corr1 Corr2 Corr3 MLP1 MLP2 MLP3 then evaluate the performance of the two proposed methods for Concept extraction and labeling to get an estimate of the gap with respect to the optimal case.
Starting from the optimal Concept presence table we extract the rules considering two diferent settings:
(i) we fit the tree-based extraction algorithm to match the original model predictions and obtain an
approximation of its behavior; (ii) we proceed with the assumption that drift has been successfully
detected and localized (e.g., through established drift detectors [
We aim to provide two complementary perspectives on model behavior: (i) Human interpretability: We extract interpretable rules that enable direct inspection to identify problematic associations that degrade model performance on concept drift and out-of-distribution samples (e.g., the model incorrectly relies on the presence of Earrings to predict that gender is not Male), and identify the specific responsible network component (kernel); and (ii) Automated diagnosis: We analyze antecedent activation frequencies to systematically identify faulty antecedents driving incorrect predictions, under the assumption that the most frequently activated antecedent is most likely responsible for the observed drift.
Listings 1 and 2 show the rules extracted in the two considered settings under the optimality assumption. Their interpretability facilitates an understanding of the antecedents primarily influencing the decision process. For instance, in Listing 1, two of the most critical rules are Rule 5 and 10, which state that the presence of earrings, regardless of the presence of beard, is suficient to predict that the gender is not Male. In Listing 2, instead, one of the most informative is Rule 1 which states that in absence of earrings no drift is detected while, in the other case, the presence of drift depends on other factors that exhibit correlation with the gender class. For instance, Rule 2 states that the presence of earrings and beard, along with the absence of lipstick, is a suficient condition to identify a drifted subset. Figure 2 illustrates how the extracted rules approximate the original models for varying tree depths in the evaluated settings under the optimal assumption. In both cases, the rule fidelity remains high, ensuring a close representation of the original model’s behavior with minimal degradation. Tables 2 and 3 present the activation frequencies of antecedents on data streams containing 0% and 100% drift samples, in both experimental settings. The results indicate that the satisfied rule leading to the final prediction consistently includes the antecedent Wearing Earring—the concept used to simulate the drift.
Finally, we evaluate the efectiveness of the Concept Extractor with respect to both concept sensitivity and concept labeling. Figure 3 shows the correlation in the CelebA dataset (a) and the correlation levels between Oracle concepts and kernels (top-1 pairs) in the correlation-based solution (b). The heatmap in Figure 3b highlights the problem of the polysemantic nature of kernel activations, which, in general, are sensitive to diferent correlated concepts. We attribute this problem to the lower performance of the correlation-based solution with respect to the MLP-based solution, which can leverage all the kernels and learn to weight them efectively to further reduce the error. Table 4, instead, shows the accuracies of the two proposed methods. Although generally both methods appear to be more reliable on larger concepts, results highlight the dificulties on very small concepts represented with few pixels—such as male earrings, which are typically much smaller than their female counterparts. Such fine details can easily be lost during convolutional operations.
Our preliminary results show the potential relationship between model interpretability and concept drift explanation in the context of trustworthy AI systems. Through neuro-symbolic rule extraction, the proposed approach clarifies the model decision pathway and helps pinpoint the origin of mispredictions, with traceability down to the responsible network component (kernel). Under the optimality assumption, the extracted rules appear highly informative, making it easier to formulate hypotheses about the possible causes of drift. However, in a real scenario, the interpretability of those rules may be afected by the concept extraction and naming procedure due to the inherent flaws still present in these systems. We proposed two efective automated approaches to address the problem of concept labeling, which is one of the most challenging and still open problems of rule extraction systems, and provide an estimation measure of their reliability. By considering the annotations (Oracle) previously extracted on the whole training set, they not only allow to avoid the search for the top-k images to which each kernel reacts most and the visual inspection phase for the labeling process, but they also enable to enlarge the set of samples considered for the kernel-concept association, resulting in a more reliable and scalable solution. Limitations and future works The eficacy of the method is inherently constrained by the a priori selection of concepts, which may obscure the true cause of the drift. Those systems can be susceptible to errors in the kernel labeling procedure, which can produce erroneously named antecedents that make the rule unreasonable, although the final classification is correct. The issue becomes more pronounced when small concepts are present in the images, as kernels in the final layers may lack suficient sensitivity to detect them. Moreover, since the encoder was originally trained for a diferent objective, individual feature maps may not necessarily be sensitive to a single concept, a challenge only partially mitigated by the proposed naming strategies. In future work, we plan to (1) enhance the robustness of the rule extraction process for drift explanation, including the exploration of automatic concept labeling methods, (2) extend the methodology to additional data modalities (e.g., text, audio), and (3) conduct a more comprehensive experimental evaluation across diverse drift scenarios, models, and data types.
During the preparation of this work, the author(s) used ChatGPT and Grammarly in order to: Grammar and spelling check. Transactions on Knowledge and Data Engineering (2015). doi:10.1109/TKDE.2014.2345382. [20] S. Rabanser, S. Günnemann, Z. C. Lipton, Failing loudly: An empirical study of methods for detecting dataset shift, in: Neural Information Processing Systems, 2018. URL: https: //api.semanticscholar.org/CorpusID:53096511. [21] S. Greco, T. Cerquitelli, Drift lens: Real-time unsupervised concept drift detection by evaluating per-label embedding distributions, in: 2021 International Conference on Data Mining Workshops (ICDMW), 2021, pp. 341–349. doi:10.1109/ICDMW53433.2021.00049. [22] L. Bu, C. Alippi, D. Zhao, A pdf-free change detection test based on density diference estimation, IEEE Transactions on Neural Networks and Learning Systems 29 (2018) 324–334. doi:10.1109/ TNNLS.2016.2619909. [23] S. Greco, B. Vacchetti, D. Apiletti, T. Cerquitelli, Driftlens: A concept drift detection tool, in: Proceedings 27th International Conference on Extending Database Technology, EDBT 2024, Paestum, Italy, March 25 - March 28, OpenProceedings.org, 2024, pp. 806–809. URL: https: //doi.org/10.48786/edbt.2024.75. doi:10.48786/edbt.2024.75. [24] E. Lughofer, E. Weigl, W. Heidl, C. Eitzinger, T. Radauer, Drift detection in data stream classification without fully labelled instances, in: 2015 IEEE International Conference on Evolving and Adaptive Intelligent Systems (EAIS), 2015, pp. 1–8. doi:10.1109/EAIS.2015.7368802. [25] A. Suprem, J. Arulraj, C. Pu, J. Ferreira, Odin: automated drift detection and recovery in video analytics, Proc. VLDB Endow. (2020). URL: https://doi.org/10.14778/3407790.3407837. doi:10. 14778/3407790.3407837. [26] M. Hushchyn, A. Ustyuzhanin, Generalization of change-point detection in time series data based on direct density ratio estimation, CoRR abs/2001.06386 (2020). URL: https://arxiv.org/abs/2001. 06386. arXiv:2001.06386. [27] K. Yamanishi, J.-i. Takeuchi, A unifying framework for detecting outliers and change points from non-stationary time series data, in: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’02, 2002. URL: https://doi.org/10. 1145/775047.775148. doi:10.1145/775047.775148. [28] O. Gözüaçık, A. Büyükçakır, H. Bonab, F. Can, Unsupervised concept drift detection with a discriminative classifier, in: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 2365–2368. URL: https://doi.org/10.1145/3357384.3358144. doi:10.1145/3357384. 3358144. [29] A. Liu, Y. Song, G. Zhang, J. Lu, Regional concept drift detection and density synchronized drift adaptation, 2017, pp. 2280–2286. doi:10.24963/ijcai.2017/317. [30] S. Hido, T. Idé, H. Kashima, H. Kubo, H. Matsuzawa, Unsupervised change analysis using supervised learning, in: Advances in Knowledge Discovery and Data Mining, Springer Berlin Heidelberg, Berlin, Heidelberg, 2008, pp. 148–159. [31] F. Hinder, V. Vaquet, J. Brinkrolf, A. Artelt, B. Hammer, Localization of concept drift: Identifying the drifting datapoints, in: 2022 International Joint Conference on Neural Networks (IJCNN), 2022, pp. 1–9. doi:10.1109/IJCNN55064.2022.9892374. [32] G. I. Webb, R. Hyde, H. Cao, H. L. Nguyen, F. Petitjean, Characterizing concept drift, Data Mining and Knowledge Discovery 30 (2016) 964–994. URL: https://doi.org/10.1007/s10618-015-0448-4. doi:10.1007/s10618-015-0448-4. [33] X. Wang, W. Chen, J. Xia, Z. Chen, D. Xu, X. Wu, M. Xu, T. Schreck, Conceptexplorer: Visual analysis of concept drifts in multi-source time-series data, in: 2020 IEEE Conference on Visual Analytics Science and Technology (VAST), 2020, pp. 1–11. doi:10.1109/VAST50239.2020.00006. [34] K. B. Pratt, G. Tschapek, Visualizing concept drift, in: Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’03, Association for Computing Machinery, New York, NY, USA, 2003, p. 735–740. URL: https://doi.org/10.1145/956750. 956849. doi:10.1145/956750.956849. [35] J. N. Adams, S. J. van Zelst, L. Quack, K. Hausmann, W. M. P. van der Aalst, T. Rose, A framework for explainable concept drift detection in process mining, in: Business Process Management, Springer International Publishing, Cham, 2021, pp. 400–416. [36] F. Hinder, V. Vaquet, B. Hammer, Feature-based analyses of concept drift, Neurocomputing 600 (2024) 127968. URL: https://www.sciencedirect.com/science/article/pii/S0925231224007392. doi:https://doi.org/10.1016/j.neucom.2024.127968. [37] C. Duckworth, F. P. Chmiel, D. K. Burns, Z. D. Zlatev, N. M. White, T. W. V. Daniels, M. Kiuber, M. J. Boniface, Using explainable machine learning to characterise data drift and detect emergent health risks for emergency department admissions during covid-19, Scientific Reports 11 (2021) 23017. URL: https://doi.org/10.1038/s41598-021-02481-y. doi:10.1038/s41598-021-02481-y. [38] Susnjak, Teo, Maddigan, Paula, Forecasting patient flows with pandemic induced concept drift using explainable machine learning, EPJ Data Sci. 12 (2023) 11. URL: https://doi.org/10.1140/epjds/ s13688-023-00387-5. doi:10.1140/epjds/s13688-023-00387-5. [39] S. M. Lundberg, S.-I. Lee, A unified approach to interpreting model predictions, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Curran Associates Inc., Red Hook, NY, USA, 2017, p. 4768–4777. [40] S. Greco, B. Vacchetti, D. Apiletti, T. Cerquitelli, Unsupervised concept drift detection from deep learning representations in real-time, IEEE Transactions on Knowledge and Data Engineering 37 (2025) 6232–6245. doi:10.1109/TKDE.2025.3593123. [41] J. a. Gama, I. Žliobaitundefined, A. Bifet, M. Pechenizkiy, A. Bouchachia, A survey on concept drift adaptation, ACM Comput. Surv. 46 (2014). doi:10.1145/2523813. [42] L. Yuan, H. Li, B. Xia, C. Gao, M. Liu, W. Yuan, X. You, Recent advances in concept drift adaptation methods for deep learning, in: L. D. Raedt (Ed.), Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, International Joint Conferences on Artificial Intelligence Organization, 2022, pp. 5654–5661. URL: https://doi.org/10.24963/ijcai.2022/788. doi:10. 24963/ijcai.2022/788, survey Track. [43] Q. Xiang, L. Zi, X. Cong, Y. Wang, Concept drift adaptation methods under the deep learning framework: A literature review, Applied Sciences 13 (2023). URL: https://www.mdpi.com/2076-3417/13/ 11/6515. doi:10.3390/app13116515. [44] M. Sundararajan, A. Taly, Q. Yan, Axiomatic attribution for deep networks, 2017. URL: https: //arxiv.org/abs/1703.01365. arXiv:1703.01365. [45] K. Simonyan, A. Vedaldi, A. Zisserman, Deep inside convolutional networks: Visualising image classification models and saliency maps, 2014. URL: https://arxiv.org/abs/1312.6034. arXiv:1312.6034. [46] F. Ventura, S. Greco, D. Apiletti, T. Cerquitelli, Explaining deep convolutional models by measuring the influence of interpretable features in image classification, Data Mining and Knowledge Discovery 38 (2024) 3169–3226. URL: https://doi.org/10.1007/s10618-023-00915-x. doi:10.1007/ s10618-023-00915-x. [47] R. R. Selvaraju, A. Das, R. Vedantam, M. Cogswell, D. Parikh, D. Batra, Grad-cam: Why did you say that?, 2017. URL: https://arxiv.org/abs/1611.07450. arXiv:1611.07450. [48] J. Townsend, T. Kasioumis, H. Inakoshi, Eric: Extracting relations inferred from convolutions, in:
Proceedings of the Asian Conference on Computer Vision, 2020. [49] J. R. Quinlan, C4. 5: programs for machine learning, Elsevier, 2014. [50] P. Padalkar, H. Wang, G. Gupta, Nesyfold: a framework for interpretable image classification, in:
Proceedings of the AAAI Conference On Artificial Intelligence, volume 38, 2024, pp. 4378–4387. [51] P. Padalkar, G. Gupta, Symbolic rule extraction from attention-guided sparse representations in vision transformers, arXiv preprint arXiv:2505.06745 (2025). [52] R. F. Tate, Correlation between a discrete and a continuous variable. point-biserial correlation,
The Annals of mathematical statistics 25 (1954) 603–607. [53] Z. Liu, P. Luo, X. Wang, X. Tang, Deep learning face attributes in the wild, in: Proceedings of International Conference on Computer Vision (ICCV), 2015.