Introduction

Deep learning has revolutionized various fields such as image classification [1], speech recognition [2], translation [3], drug design [4], medical diagnosis [5], climate sciences [6]. However, the opaque nature of deep learning systems poses challenges in applications involving automated decisions and safety-critical systems. For instance, concerns arise from incidents like Steve Wozniak's accusation of gender discrimination in Apple Card credit limits and biased image search results for "CEOs" [7]. Safety-critical areas like self-driving cars [8] and [9,10] are also vulnerable to adversarial attacks [11], including altering classification results [11] and manipulating the order of training images [12]. Some attacks are hard to detect post facto, posing significant risks [13,14].

Problem Statement: While statistical evaluations are standard for assessing deep learning performance, they fall short in providing explanations for specific system behaviors [15]. Therefore, developing robust explanation methods for deep learning systems remains crucial. Despite significant progress in this area (see Section 4), current approaches often rely on a limited set of predefined explanation categories. This reliance on human-selected categories is problematic, as it assumes they are suitable for explaining deep learning systems, which lacks evidence. Some methods leverage deep learning models, such as LLMs, to generate explanations [16], introducing another layer of opacity. Additionally, state-of-the-art explanation systems often require modified deep learning architectures, which can lead to reduced system performance compared to unmodified versions [17].

Importance: The importance of solving this challenge cannot be overstated. Transparent and interpretable AI systems are crucial for building trust, especially in domains like healthcare, finance, and autonomous vehicles. By providing explanations, we empower users, including non-experts, to understand AI decisions, fostering better acceptance and adoption. Advancing explainable AI contributes to interdisciplinary collaboration and can enhance societal benefits while mitigating ethical risks associated with AI deployment. Therefore, it is imperative to address the challenge of developing transparent and interpretable explanation methods for deep learning systems.

The subsequent section presents the research question and objectives, building on the above core principles. 2.1 describe the contributions we have made, focusing on methods we use or plan to use to support these contributions and then describing the results 3 thus far from them.

Research Question and Contributions

Research Question: How can we develop an effective approach to explainable deep learning that can be used to assign human-understandable interpretations to the activations of hidden neurons in the deep learning model?

This proposal outlines an approach to use Concept Induction, i.e., formal logical deductive reasoning [18] to automatically provide meaningful explanations for hidden neuron activation in a Convolutional Neural Network (CNN) architecture for image scene classification (on the ADE20K dataset [19]), using a class hierarchy consisting of about 2 • 10 6 classes, derived from Wikipedia, as the pool of categories [20]. Stating the hypothesis clearly that drives the work outlined in this proposal.

Hypothesis: Concept Induction analysis with large-scale background knowledge yields meaningful labels that stably explain neuron activation in the hidden layer of CNN architecture.

Contributions and Methodology

To achieve the above-stated hypothesis, the following objectives with the methodology followed or planned to follow are outlined:

Objective 1: Employing Concept Induction and a Wikipedia Knowledge Graph to Assign Meaningful Labels to Hidden Neurons' Activation.

We explored and evaluated three concrete methods (Concept Induction, CLIP-Dissect [16], GPT-4 [21]) to generate high-level concepts for explaining hidden neuron activations. Our comprehensive methodology for Objective 1 is detailed in our paper [22].

1. Prep: Scenario and CNN Training -Utilizing the annotated ADE20K dataset [19], we trained Resnet50V2 for scene classification, achieving an accuracy of (86.46%). The annotations are only used for generating label hypotheses, not for CNN training. While highest accuracy isn't critical for our investigation, it's important for models to be practically applicable. 2. Concept Induction - [18] system accepts three inputs: positive set 𝑃 and negative set 𝑁 of images from ADE20K, and a knowledge base 𝐾, all expressed as description logic theories, and all examples 𝑥 ∈ 𝑃 ∪ 𝑁 occur as individuals (constants) in 𝐾. It returns description logic class expressions 𝐸 such that 𝐾 |= 𝐸(𝑝) for all 𝑝 ∈ 𝑃 and 𝐾 ̸ |= 𝐸(𝑞) for all 𝑞 ∈ 𝑁 . For scalability, we used ECII [23] heuristic Concept Induction system with Wikipedia [20]. We included the images in the background knowledge by associating object annotations from ADE20K images with classes in the hierarchy, using the Levenshtein string similarity metric [24] with edit distance 0.

Generating Label Hypothesesa)

In Concept Induction, we used 1,370 ADE20K images with our trained ResNet50V2, extracting activations from the dense layer with 64 neurons. Positive examples (𝑃 ) are images activating the neuron with > 80% of its max activation, negative examples (𝑁 ) are those activating it with < 20% of its max or not at all. ECII generates the target label for each neuron based on these sets and background knowledge. b) CLIP-Dissect employs the top 20,000 English vocabulary words as concepts. Subsequently, activations from our trained ResNet50v2 model for ADE20K test images were collected, resulting in a matrix (Number of Images × 64). Utilizing these inputs, CLIP-Dissect assigns a label to each neuron such that the neuron is most activated when the corresponding concept is present in the image, resulting in 22 distinct concepts across 64 neurons. c) GPT-4 Leveraging GPT-4, we adopt a methodology akin to [25] for concept generation to differentiate image classes [26]. We input image annotations from positive (𝑃 ) and negative (𝑁 ) sets into GPT-4 with prompts to discern concepts unique to 𝑃 . The prompt "Generate top three classes of objects/general scenarios that better represent what images in the positive set (𝑃 ) have but the images in the negative set (𝑁 ) do not," yields three concepts per neuron, from which we select one per class for assessment.

Objective 2: Automate Concept Label Association for Input Images using Neuron Ensembles and Non-target Activation Probabilities.

1. Concept Associations and Non-Target Activations -In pursuit of Objective 1, Step 3 generates labels for neuron activation. Each neuron's label is the target concept, with all other images considered as non-target concepts. This analysis focuses on the top three ECII responses, assessing neuron activation for non-target concepts at various cut-off values relative to each neuron's maximum activation value: > 0, > 20% of max, > 40% of max, and > 60% of max. The goal is to establish strong associations between concepts and neuron activations, understanding which concepts trigger specific neurons and to what extent. 2. Neuron Ensembles for Concept Associations -Input information can be distributed across simultaneously activated neurons, necessitating the examination of neuron ensemble activations using previously established cut-off values. However, the scale challenge arises with 2 64 potential neuron ensembles for just 64 neurons. To address this, we propose combining neurons activated for semantically related labels (with top-3 responses from ECII). For instance, if "building" activates both neuron 0 and neuron 63. We assess all images activating both neurons 0 and 63 for specified cut-off values. In cases where a concept activates more than two neurons, our analysis encompasses all possible combinations of pairs, evaluating target and non-target activations. We proceed with concepts, including neuron ensembles, that exhibit target activation exceeding 80% for further analysis 3. Validating Neuron-Concept Associations -After completing Step 1 and Step 2, we obtain probabilities for non-target concepts across all concepts, including those activating single neurons as well as neuron ensembles. This allows for identifying potential concepts and assessing associated error margins. To verify or reject these concepts, we revisit the ADE20K dataset. Using a subset of 1050 randomly chosen images, we conduct a user study via Amazon Mechanical Turk (MTurk) [27] to annotate images with target concepts. We then cross-reference these designated concepts with image annotations obtained from the MTurk study. We evaluate the likelihood of neuron activations for non-target concepts. 4. Developing an Automated System -We propose developing an automated system to streamline the entire process, enabling scalability to larger datasets and exploration of a broader parameter range. The system would comprise: Concept induction: Generates class expressions/responses ranked by coverage score. Neuron activation: Calculates activation for target and non-target concepts (including neuron ensembles) at various cut-off values. Concept validation: Validates generated concepts. This automated system would analyze new images, generating a list of potential concepts with associated probabilities. Users could review the concepts and select the most relevant ones for the image. The automated approach offers several advantages, including speed, efficiency, scalability to larger datasets, and exploration of diverse parameter settings.

Evaluation and Results

Objective 1: The three approaches generate label hypotheses for all studied neurons, which we validated using new images. We search Google Images using each target label as keywords and collect 200 images per label with Imageye1 . These images are split into 80% for evaluation and 20% for statistical analysis. We then determine if the target neuron activates when the retrieval label matches the target 1(presents selective representation due to space constraints, complete version is available at [22].) show the percentage of target images that activated each neuron. A target label is confirmed if it activates for ≥ 80% of its target images, regardless of its activation for non-target images. Detailed paper can be found at [22]. Statistical Evaluation and Result:-After generating confirmed labels from all three approaches, we assess node labeling using the remaining images, treating each neuron-label pair in Table 1 as a hypothesis. Concept Induction, CLIP-Dissect, and GPT-4 produce 20, 8, and 27 hypotheses, respectively, based on confirmed labels. Using the Mann-Whitney U test, we compared activation strengths between images retrieved using the target label and those retrieved using other keywords. Table 2 shows the selective representation of results obtained through Mann-Whitney U test. Concept Induction consistently outperforms other methods, as evidenced by Mann-Whitney U results and statistical analysis. For most neurons, activation values of target images significantly exceed those of non-target images (with 𝑝 < 0.00001). Concept Induction rejects 19 out of 20 null hypotheses at 𝑝 < 0.05, CLIP-Dissect rejects all 8 null hypotheses, and GPT-4 rejects 25 out of 27 null hypotheses at 𝑝 < 0.05. More details in [22].

Objective 2: We will conduct a comprehensive statistical evaluation using the Mann-Whitney U (MWU) test for each concept across different cut-off values. This evaluation aims to compare the activation strengths of non-target concepts retrieved through Google Images(from Objective 1) with those retrieved from the ADE20K dataset. The hypothesis under consideration is that the activation strength of non-target concepts from Google Images exceeds that from the ADE20K dataset. Conversely, the null hypothesis (H0) posits that the activation strength of non-target concepts from Google Images equals that from the ADE20K dataset. For each category of cut-off values, concepts exhibiting a significant difference in activation strengths (p-value < 0.005) will undergo further validation through the Wilcoxon signed-rank test across all cut-off values as a collective unit. We refine our approach and enhance concept label associations' accuracy by identifying concepts with significantly higher activation strengths.

Related Work

With the recent advances in deep learning [28], its wide usage in nearly every field, and its opaque nature make explainable AI more important than ever, and there are multiple ongoing efforts to demystify deep learning [29,30,31]. Existing explainable methods can be categorized based on input data (feature) understanding, e.g., feature summarizing [32,33], or based on the model's internal unit representation, e.g., node summarizing [34,11]. Those methods can be further categorized as model-specific [32] or model-agnostic [33]. Another kind of approach relies on human interpretation of explanatory data returned, such as counterfactual questions [35].

We focus on the understanding of internal units of the neural network-based deep learning models. Prior work has shown that internal units may indeed represent human-understandable concepts [34,11], but these approaches often require resource-intensive methods like semantic segmentation [36] or explicit concept annotations [37]. There has been research utilizing Semantic Web data for explaining deep learning models [38,39], and Concept Induction for generating explanations [40,41]. However, they mainly focused on analyzing how inputs relate to outputs and generating explanations for the whole system, while we focused on understanding internal node activations.

CLIP-Dissect [16], similar to our work, takes a different approach. It utilizes the CLIP pre-trained model, employing zero-shot learning to associate images with labels. Another related work, Label-Free Concept Bottleneck Models [26], builds upon CLIP-Dissect, using GPT-4 [21] for concept set generation. However, CLIP-Dissect faces challenges in accurately predicting output labels based on concepts in the last hidden layer and transferring to other modalities or domain-specific applications. The Label-Free approach inherits these limitations and may compromise explainability due to its use of a concept derivation method that lacks inherent explainability.

Conclusion

Concept Induction, leveraging large-scale ontological background knowledge, provides meaningful labeling of hidden neuron activations, validated by statistical analysis. This allows us to pinpoint concepts that strongly trigger neuron responses, effectively explaining neuron activations. Our approach introduces novel possibilities for diverse label categories. Comparative analysis against CLIP-Dissect and GPT-4 showcases Concept Induction's superiority, especially in settings with labeled data. Ultimately, our work aims to thoroughly analyze hidden layers in deep learning systems, facilitating the interpretation of activations as implicit input features and explaining system input-output behavior. Moving forward, future work will focus on enhancing Concept Induction's scalability and efficiency, enabling its broader applicability across various domains.