<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Understanding CNN Hidden Neuron Activations using Concept Induction over Background Knowledge</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Abhilekha Dalal</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Kansas State University</institution>
          ,
          <addr-line>Manhattan KS</addr-line>
          ,
          <country country="US">USA</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>A major challenge in Explainable AI is interpreting hidden neuron activations accurately. These interpretations can reveal what a deep learning system perceives as relevant in the input data, thereby addressing the black-box nature of such systems. The state of the art indicates that hidden node activations can be interpretable by humans, but there's a lack of systematic automated methods to verify these interpretations, especially those that utilize substantial background knowledge and inherently explainable methods. In this proposal, we introduce a novel model-agnostic post-hoc Explainable AI method based on a Wikipedia-derived concept hierarchy with approximately 2 million classes. Our approach utilizes OWL-reasoning-based Concept Induction for explanation generation and compares with of-the-shelf pre-trained multimodal-based explainable methods. Our results demonstrate that our method automatically provides meaningful class expressions as explanations to individual neurons in the dense layer of a Convolutional Neural Network, outperforming prior work in both quantitative and qualitative aspects.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;Explainable AI</kwd>
        <kwd>Concept Induction</kwd>
        <kwd>Convolutional Neural Network</kwd>
        <kwd>Knowledge Graph</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        Deep learning has revolutionized various fields such as image classification [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ], speech recognition [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ],
translation [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ], drug design [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ], medical diagnosis [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ], climate sciences [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ]. However, the opaque
nature of deep learning systems poses challenges in applications involving automated decisions and
safety-critical systems. For instance, concerns arise from incidents like Steve Wozniak’s accusation
of gender discrimination in Apple Card credit limits and biased image search results for "CEOs" [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ].
Safety-critical areas like self-driving cars [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ] and [
        <xref ref-type="bibr" rid="ref10 ref9">9, 10</xref>
        ] are also vulnerable to adversarial attacks [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ],
including altering classification results [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ] and manipulating the order of training images [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ]. Some
attacks are hard to detect post facto, posing significant risks [
        <xref ref-type="bibr" rid="ref13 ref14">13, 14</xref>
        ].
      </p>
      <p>
        Problem Statement: While statistical evaluations are standard for assessing deep learning
performance, they fall short in providing explanations for specific system behaviors [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ]. Therefore,
developing robust explanation methods for deep learning systems remains crucial. Despite significant
progress in this area (see Section 4), current approaches often rely on a limited set of predefined
explanation categories. This reliance on human-selected categories is problematic, as it assumes they
are suitable for explaining deep learning systems, which lacks evidence. Some methods leverage deep
learning models, such as LLMs, to generate explanations [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ], introducing another layer of opacity.
Additionally, state-of-the-art explanation systems often require modified deep learning architectures,
which can lead to reduced system performance compared to unmodified versions [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ].
      </p>
      <p>Importance: The importance of solving this challenge cannot be overstated. Transparent and
interpretable AI systems are crucial for building trust, especially in domains like healthcare, finance,
and autonomous vehicles. By providing explanations, we empower users, including non-experts,
to understand AI decisions, fostering better acceptance and adoption. Advancing explainable AI
contributes to interdisciplinary collaboration and can enhance societal benefits while mitigating ethical
risks associated with AI deployment. Therefore, it is imperative to address the challenge of developing
transparent and interpretable explanation methods for deep learning systems.</p>
      <p>The subsequent section presents the research question and objectives, building on the above core
principles. 2.1 describe the contributions we have made, focusing on methods we use or plan to use to
support these contributions and then describing the results 3 thus far from them.</p>
    </sec>
    <sec id="sec-2">
      <title>2. Research Question and Contributions</title>
      <p>Research Question: How can we develop an efective approach to explainable deep learning that can
be used to assign human-understandable interpretations to the activations of hidden neurons in the
deep learning model?</p>
      <p>
        This proposal outlines an approach to use Concept Induction, i.e., formal logical deductive
reasoning [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ] to automatically provide meaningful explanations for hidden neuron activation in a
Convolutional Neural Network (CNN) architecture for image scene classification (on the ADE20K dataset [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ]),
using a class hierarchy consisting of about 2 · 106 classes, derived from Wikipedia, as the pool of
categories [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ]. Stating the hypothesis clearly that drives the work outlined in this proposal.
      </p>
      <p>Hypothesis: Concept Induction analysis with large-scale background knowledge yields meaningful
labels that stably explain neuron activation in the hidden layer of CNN architecture.
2.1. Contributions and Methodology
To achieve the above-stated hypothesis, the following objectives with the methodology followed or
planned to follow are outlined:</p>
      <p>Objective 1: Employing Concept Induction and a Wikipedia Knowledge Graph to Assign Meaningful
Labels to Hidden Neurons’ Activation.</p>
      <p>
        We explored and evaluated three concrete methods (Concept Induction, CLIP-Dissect [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ],
GPT4 [21]) to generate high-level concepts for explaining hidden neuron activations. Our comprehensive
methodology for Objective 1 is detailed in our paper [22].
      </p>
      <p>
        1. Prep: Scenario and CNN Training - Utilizing the annotated ADE20K dataset [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ], we trained
Resnet50V2 for scene classification, achieving an accuracy of ( 86.46%). The annotations are only
used for generating label hypotheses, not for CNN training. While highest accuracy isn’t critical for
our investigation, it’s important for models to be practically applicable.
2. Concept Induction - [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ] system accepts three inputs: positive set  and negative set  of
images from ADE20K, and a knowledge base , all expressed as description logic theories, and
all examples  ∈  ∪  occur as individuals (constants) in . It returns description logic class
expressions  such that  |= () for all  ∈  and  ̸|= () for all  ∈  . For scalability,
we used ECII [23] heuristic Concept Induction system with Wikipedia [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ]. We included the
images in the background knowledge by associating object annotations from ADE20K images
with classes in the hierarchy, using the Levenshtein string similarity metric [24] with edit distance
0.
3. Generating Label Hypotheses
a) In Concept Induction, we used 1,370 ADE20K images with our trained ResNet50V2,
extracting activations from the dense layer with 64 neurons. Positive examples ( ) are images
activating the neuron with &gt; 80% of its max activation, negative examples ( ) are those
activating it with &lt; 20% of its max or not at all. ECII generates the target label for each
neuron based on these sets and background knowledge.
b) CLIP-Dissect employs the top 20,000 English vocabulary words as concepts. Subsequently,
activations from our trained ResNet50v2 model for ADE20K test images were collected,
resulting in a matrix (Number of Images × 64). Utilizing these inputs, CLIP-Dissect assigns a
label to each neuron such that the neuron is most activated when the corresponding concept
is present in the image, resulting in 22 distinct concepts across 64 neurons.
c) GPT-4 Leveraging GPT-4, we adopt a methodology akin to [25] for concept generation to
diferentiate image classes [ 26]. We input image annotations from positive ( ) and negative
( ) sets into GPT-4 with prompts to discern concepts unique to  . The prompt "Generate
top three classes of objects/general scenarios that better represent what images in the
positive set ( ) have but the images in the negative set ( ) do not," yields three concepts
per neuron, from which we select one per class for assessment.
      </p>
      <p>Objective 2: Automate Concept Label Association for Input Images using Neuron Ensembles and
Non-target Activation Probabilities.</p>
      <p>1. Concept Associations and Non-Target Activations - In pursuit of Objective 1, Step 3 generates
labels for neuron activation. Each neuron’s label is the target concept, with all other images
considered as non-target concepts. This analysis focuses on the top three ECII responses, assessing
neuron activation for non-target concepts at various cut-of values relative to each neuron’s
maximum activation value: &gt; 0, &gt; 20% of max, &gt; 40% of max, and &gt; 60% of max. The goal is
to establish strong associations between concepts and neuron activations, understanding which
concepts trigger specific neurons and to what extent.
2. Neuron Ensembles for Concept Associations - Input information can be distributed across
simultaneously activated neurons, necessitating the examination of neuron ensemble activations
using previously established cut-of values. However, the scale challenge arises with 264 potential
neuron ensembles for just 64 neurons. To address this, we propose combining neurons activated
for semantically related labels (with top-3 responses from ECII). For instance, if "building"
activates both neuron 0 and neuron 63. We assess all images activating both neurons 0 and 63 for
specified cut-of values. In cases where a concept activates more than two neurons, our analysis
encompasses all possible combinations of pairs, evaluating target and non-target activations. We
proceed with concepts, including neuron ensembles, that exhibit target activation exceeding 80%
for further analysis
3. Validating Neuron-Concept Associations - After completing Step 1 and Step 2, we obtain
probabilities for non-target concepts across all concepts, including those activating single
neurons as well as neuron ensembles. This allows for identifying potential concepts and assessing
associated error margins. To verify or reject these concepts, we revisit the ADE20K dataset. Using
a subset of 1050 randomly chosen images, we conduct a user study via Amazon Mechanical Turk
(MTurk) [27] to annotate images with target concepts. We then cross-reference these designated
concepts with image annotations obtained from the MTurk study. We evaluate the likelihood of
neuron activations for non-target concepts.
4. Developing an Automated System - We propose developing an automated system to streamline
the entire process, enabling scalability to larger datasets and exploration of a broader parameter
range. The system would comprise: Concept induction: Generates class expressions/responses
ranked by coverage score. Neuron activation: Calculates activation for target and non-target
concepts (including neuron ensembles) at various cut-of values. Concept validation: Validates
generated concepts. This automated system would analyze new images, generating a list of
potential concepts with associated probabilities. Users could review the concepts and select the
most relevant ones for the image. The automated approach ofers several advantages, including
speed, eficiency, scalability to larger datasets, and exploration of diverse parameter settings.</p>
    </sec>
    <sec id="sec-3">
      <title>3. Evaluation and Results</title>
      <p>Objective 1: The three approaches generate label hypotheses for all studied neurons, which we
validated using new images. We search Google Images using each target label as keywords and collect
200 images per label with Imageye1. These images are split into 80% for evaluation and 20% for statistical
analysis. We then determine if the target neuron activates when the retrieval label matches the target
1https://chrome.google.com/webstore/detail/image-downloader-imageye/agionbommeaifngbhincahgmoflcikhm
label and if any other neurons activate. Table 1(presents selective representation due to space constraints,
complete version is available at [22].) show the percentage of target images that activated each neuron.
A target label is confirmed if it activates for ≥ 80% of its target images, regardless of its activation for
non-target images. Detailed paper can be found at [22].</p>
      <p>Statistical Evaluation and Result:- After generating confirmed labels from all three approaches,
we assess node labeling using the remaining images, treating each neuron-label pair in Table 1 as a
hypothesis. Concept Induction, CLIP-Dissect, and GPT-4 produce 20, 8, and 27 hypotheses, respectively,
based on confirmed labels. Using the Mann-Whitney U test, we compared activation strengths between
images retrieved using the target label and those retrieved using other keywords. Table 2 shows
the selective representation of results obtained through Mann-Whitney U test. Concept Induction
consistently outperforms other methods, as evidenced by Mann-Whitney U results and statistical
analysis. For most neurons, activation values of target images significantly exceed those of non-target
images (with  &lt; 0.00001). Concept Induction rejects 19 out of 20 null hypotheses at  &lt; 0.05,
CLIP-Dissect rejects all 8 null hypotheses, and GPT-4 rejects 25 out of 27 null hypotheses at  &lt; 0.05.
More details in [22].</p>
      <p>Objective 2: We will conduct a comprehensive statistical evaluation using the Mann-Whitney U
(MWU) test for each concept across diferent cut-of values. This evaluation aims to compare the
activation strengths of non-target concepts retrieved through Google Images(from Objective 1) with
those retrieved from the ADE20K dataset. The hypothesis under consideration is that the activation
strength of non-target concepts from Google Images exceeds that from the ADE20K dataset. Conversely,
the null hypothesis (H0) posits that the activation strength of non-target concepts from Google Images
equals that from the ADE20K dataset. For each category of cut-of values, concepts exhibiting a
significant diference in activation strengths (p-value &lt; 0.005) will undergo further validation through
the Wilcoxon signed-rank test across all cut-of values as a collective unit. We refine our approach
and enhance concept label associations’ accuracy by identifying concepts with significantly higher
activation strengths.</p>
    </sec>
    <sec id="sec-4">
      <title>4. Related Work</title>
      <p>
        With the recent advances in deep learning [28], its wide usage in nearly every field, and its opaque nature
make explainable AI more important than ever, and there are multiple ongoing eforts to demystify
deep learning [29, 30, 31]. Existing explainable methods can be categorized based on input data (feature)
understanding, e.g., feature summarizing [32, 33], or based on the model’s internal unit representation,
e.g., node summarizing [
        <xref ref-type="bibr" rid="ref11">34, 11</xref>
        ]. Those methods can be further categorized as model-specific [ 32] or
model-agnostic [33]. Another kind of approach relies on human interpretation of explanatory data
returned, such as counterfactual questions [35].
      </p>
      <p>
        We focus on the understanding of internal units of the neural network-based deep learning models.
Prior work has shown that internal units may indeed represent human-understandable concepts [
        <xref ref-type="bibr" rid="ref11">34, 11</xref>
        ],
but these approaches often require resource-intensive methods like semantic segmentation [36] or
explicit concept annotations [37]. There has been research utilizing Semantic Web data for explaining
deep learning models [38, 39], and Concept Induction for generating explanations [40, 41]. However,
they mainly focused on analyzing how inputs relate to outputs and generating explanations for the
whole system, while we focused on understanding internal node activations.
      </p>
      <p>
        CLIP-Dissect [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ], similar to our work, takes a diferent approach. It utilizes the CLIP pre-trained
model, employing zero-shot learning to associate images with labels. Another related work, Label-Free
Concept Bottleneck Models [26], builds upon CLIP-Dissect, using GPT-4 [21] for concept set generation.
However, CLIP-Dissect faces challenges in accurately predicting output labels based on concepts in the
last hidden layer and transferring to other modalities or domain-specific applications. The Label-Free
approach inherits these limitations and may compromise explainability due to its use of a concept
derivation method that lacks inherent explainability.
      </p>
    </sec>
    <sec id="sec-5">
      <title>5. Conclusion</title>
      <p>Concept Induction, leveraging large-scale ontological background knowledge, provides meaningful
labeling of hidden neuron activations, validated by statistical analysis. This allows us to pinpoint
concepts that strongly trigger neuron responses, efectively explaining neuron activations. Our approach
introduces novel possibilities for diverse label categories. Comparative analysis against CLIP-Dissect
and GPT-4 showcases Concept Induction’s superiority, especially in settings with labeled data.
Ultimately, our work aims to thoroughly analyze hidden layers in deep learning systems, facilitating the
interpretation of activations as implicit input features and explaining system input-output behavior.
Moving forward, future work will focus on enhancing Concept Induction’s scalability and eficiency,
enabling its broader applicability across various domains.</p>
    </sec>
    <sec id="sec-6">
      <title>Acknowledgments</title>
      <p>The author acknowledge advisor Dr. Pascal Hitzler and partial funding under National Science
Foundation grants 2119753 and 2333782.
W. R. Aue, Wikipedia knowledge graph for explainable AI, in: B. Villazón-Terrazas, F.
OrtizRodríguez, S. M. Tiwari, S. K. Shandilya (Eds.), Proceedings of the Knowledge Graphs and Semantic
Web Second Iberoamerican Conference and First Indo-American Conference (KGSWC), volume
1232 of Communications in Computer and Information Science, Springer, 2020, pp. 72–87. URL:
https://doi.org/10.1007/978-3-030-65384-2_6. doi:10.1007/978-3-030-65384-2\_6.
[21] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,</p>
      <p>S. Altman, S. Anadkat, et al., GPT-4 technical report, arXiv preprint arXiv:2303.08774 (2023).
[22] A. Dalal, R. Rayan, A. Barua, E. Y. Vasserman, M. K. Sarker, P. Hitzler, On the value of labeled data
and symbolic methods for hidden neuron activation analysis, 2024. arXiv:2404.13567.
[23] M. K. Sarker, P. Hitzler, Eficient concept induction for description logics, in: The Thirty-Third
AAAI Conference on Artificial Intelligence (AAAI) The Thirty-First Innovative Applications of
Artificial Intelligence Conference (IAAI), The Ninth AAAI Symposium on Educational Advances
in Artificial Intelligence (EAAI), AAAI Press, 2019, pp. 3036–3043. URL: https://doi.org/10.1609/
aaai.v33i01.33013036. doi:10.1609/aaai.v33i01.33013036.
[24] V. I. Levenshtein, On the minimal redundancy of binary error-correcting codes, Inf.</p>
      <p>Control. 28 (1975) 268–291. URL: https://doi.org/10.1016/S0019-9958(75)90300-9. doi:10.1016/
S0019-9958(75)90300-9.
[25] A. Barua, C. Widmer, P. Hitzler, Concept induction using LLMs: a user experiment for assessment,
2024. URL: https://arxiv.org/abs/2404.11875. arXiv:2404.11875.
[26] T. Oikarinen, S. Das, L. M. Nguyen, T.-W. Weng, Label-free concept bottleneck models, in:
The Eleventh International Conference on Learning Representations, ICLR, 2023. URL: https:
//openreview.net/forum?id=FlCg47MNvBA.
[27] K. Crowston, Amazon mechanical turk: A research tool for organizations and information systems
scholars, in: A. Bhattacherjee, B. Fitzgerald (Eds.), Shaping the Future of ICT Research. Methods
and Approaches, Springer Berlin Heidelberg, Berlin, Heidelberg, 2012, pp. 210–221.
[28] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (2015) 436–444.
[29] D. Gunning, M. Stefik, J. Choi, T. Miller, S. Stumpf, G.-Z. Yang, XAI – explainable artificial
intelligence, Science robotics 4 (2019) eaay7120.
[30] A. Adadi, M. Berrada, Peeking inside the black-box: a survey on explainable artificial intelligence
(xai), IEEE access 6 (2018) 52138–52160.
[31] D. Minh, H. X. Wang, Y. F. Li, T. N. Nguyen, Explainable artificial intelligence: a comprehensive
review, Artificial Intelligence Review (2022) 1–66.
[32] R. R. Selvaraju, A. Das, R. Vedantam, M. Cogswell, D. Parikh, D. Batra, Grad-CAM: Why did you
say that?, 2016. URL: http://arxiv.org/abs/1611.07450. arXiv:1611.07450.
[33] M. T. Ribeiro, S. Singh, C. Guestrin, "Why Should I Trust You?": Explaining the Predictions of Any
Classifier, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining, KDD ’16, Association for Computing Machinery, New York, NY,
USA, 2016, p. 1135–1144. URL: https://doi.org/10.1145/2939672.2939778. doi:10.1145/2939672.
2939778.
[34] B. Zhou, D. Bau, A. Oliva, A. Torralba, Interpreting deep visual representations via network
dissection, IEEE transactions on pattern analysis and machine intelligence 41 (2018) 2131–2145.
[35] S. Wachter, B. D. Mittelstadt, C. Russell, Counterfactual explanations without opening the black
box: Automated decisions and the GDPR, CoRR abs/1711.00399 (2017). URL: http://arxiv.org/abs/
1711.00399. arXiv:1711.00399.
[36] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, J. Sun, Unified perceptual parsing for scene understanding, in:</p>
      <p>Proceedings of the European conference on computer vision (ECCV), 2018, pp. 418–434.
[37] B. Kim, M. Wattenberg, J. Gilmer, C. J. Cai, J. Wexler, F. B. Viégas, R. Sayres, Interpretability
beyond feature attribution: Quantitative testing with concept activation vectors (TCAV), in:
J. G. Dy, A. Krause (Eds.), Proceedings of the International Conference on Machine Learning
(ICML), volume 80 of Proceedings of Machine Learning Research, PMLR, 2018, pp. 2673–2682. URL:
http://proceedings.mlr.press/v80/kim18d.html.
[38] R. Confalonieri, T. Weyde, T. R. Besold, F. M. del Prado Martín, Using ontologies to enhance human
understandability of global post-hoc explanations of black-box models, Artificial Intelligence 296
(2021) 103471.
[39] N. Díaz-Rodríguez, A. Lamas, J. Sanchez, G. Franchi, I. Donadello, S. Tabik, D. Filliat, P. Cruz,
R. Montes, F. Herrera, Explainable neural-symbolic learning (x-nesyl) methodology to fuse deep
learning representations with expert knowledge graphs: The monumai cultural heritage use case,
Information Fusion 79 (2022) 58–83.
[40] M. K. Sarker, N. Xie, D. Doran, M. L. Raymer, P. Hitzler, Explaining trained neural networks
with semantic web technologies: First steps, in: T. R. Besold, A. S. d’Avila Garcez, I. Noble
(Eds.), Proceedings of the Twelfth International Workshop on Neural-Symbolic Learning and
Reasoning (NeSy), volume 2003 of CEUR Workshop Proceedings, CEUR-WS.org, 2017. URL: https:
//ceur-ws.org/Vol-2003/NeSy17_paper4.pdf.
[41] T. Procko, T. Elvira, O. Ochoa, N. D. Rio, An exploration of explainable machine learning using
semantic web technology, in: 2022 IEEE 16th International Conference on Semantic Computing
(ICSC), IEEE Computer Society, 2022, pp. 143–146. doi:10.1109/ICSC52841.2022.00029.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>M.</given-names>
            <surname>Ramprasath</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. V.</given-names>
            <surname>Anand</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Hariharan</surname>
          </string-name>
          ,
          <article-title>Image classification using convolutional neural networks</article-title>
          ,
          <source>International Journal of Pure and Applied Mathematics</source>
          <volume>119</volume>
          (
          <year>2018</year>
          )
          <fpage>1307</fpage>
          -
          <lpage>1319</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>A.</given-names>
            <surname>Graves</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Jaitly</surname>
          </string-name>
          ,
          <article-title>Towards end-to-end speech recognition with recurrent neural networks</article-title>
          ,
          <source>in: International conference on machine learning</source>
          ,
          <source>The Proceedings of Machine Learning Research</source>
          ,
          <year>2014</year>
          , pp.
          <fpage>1764</fpage>
          -
          <lpage>1772</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>M.</given-names>
            <surname>Auli</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Galley</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Quirk</surname>
          </string-name>
          , G. Zweig,
          <article-title>Joint language and translation modeling with recurrent neural networks</article-title>
          ,
          <source>in: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing</source>
          ,
          <year>2013</year>
          , pp.
          <fpage>1044</fpage>
          -
          <lpage>1054</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>M. H.</given-names>
            <surname>Segler</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Kogej</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Tyrchan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. P.</given-names>
            <surname>Waller</surname>
          </string-name>
          ,
          <article-title>Generating focused molecule libraries for drug discovery with recurrent neural networks</article-title>
          ,
          <source>ACS central science 4</source>
          (
          <year>2018</year>
          )
          <fpage>120</fpage>
          -
          <lpage>131</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>H.-I.</given-names>
            <surname>Choi</surname>
          </string-name>
          , S.
          <article-title>-</article-title>
          <string-name>
            <surname>K. Jung</surname>
            ,
            <given-names>S.-H.</given-names>
          </string-name>
          <string-name>
            <surname>Baek</surname>
            ,
            <given-names>W. H.</given-names>
          </string-name>
          <string-name>
            <surname>Lim</surname>
            ,
            <given-names>S.-J.</given-names>
          </string-name>
          <string-name>
            <surname>Ahn</surname>
            ,
            <given-names>I.-H.</given-names>
          </string-name>
          <string-name>
            <surname>Yang</surname>
          </string-name>
          , T.-W. Kim,
          <article-title>Artificial intelligent model with neural network machine learning for the diagnosis of orthognathic surgery</article-title>
          ,
          <source>Journal of Craniofacial Surgery</source>
          <volume>30</volume>
          (
          <year>2019</year>
          )
          <fpage>1986</fpage>
          -
          <lpage>1989</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>Y.</given-names>
            <surname>Liu</surname>
          </string-name>
          , E. Racah, Prabhat,
          <string-name>
            <given-names>J.</given-names>
            <surname>Correa</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Khosrowshahi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Lavers</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Kunkel</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. F.</given-names>
            <surname>Wehner</surname>
          </string-name>
          , W. D. Collins,
          <article-title>Application of deep convolutional neural networks for detecting extreme weather in climate datasets</article-title>
          ,
          <year>2016</year>
          . URL: http://arxiv.org/abs/1605.01156. arXiv:
          <volume>1605</volume>
          .
          <fpage>01156</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>I. A.</given-names>
            <surname>Hamilton</surname>
          </string-name>
          ,
          <article-title>Apple cofounder Steve Wozniak says Apple Card ofered his wife a lower credit limit</article-title>
          , Business
          <string-name>
            <surname>Insider</surname>
          </string-name>
          (
          <year>2019</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>Z.</given-names>
            <surname>Chen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Huang</surname>
          </string-name>
          ,
          <article-title>End-to-end learning for lane keeping of self-driving cars</article-title>
          ,
          <source>in: 2017 IEEE Intelligent Vehicles Symposium (IV)</source>
          , IEEE,
          <year>2017</year>
          , pp.
          <fpage>1856</fpage>
          -
          <lpage>1860</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>A. S.</given-names>
            <surname>Rifaioglu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            <surname>Nalbat</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Atalay</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. J.</given-names>
            <surname>Martin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Cetin-Atalay</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Doğan</surname>
          </string-name>
          ,
          <article-title>Deepscreen: high performance drug-target interaction prediction with convolutional neural networks using 2-d structural compound representations</article-title>
          ,
          <source>Chemical science 11</source>
          (
          <year>2020</year>
          )
          <fpage>2531</fpage>
          -
          <lpage>2557</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>W.</given-names>
            <surname>Hariri</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Narin</surname>
          </string-name>
          ,
          <article-title>Deep neural networks for covid-19 detection and diagnosis using images and acoustic-based techniques: a recent review</article-title>
          ,
          <source>Soft computing 25</source>
          (
          <year>2021</year>
          )
          <fpage>15345</fpage>
          -
          <lpage>15362</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>D.</given-names>
            <surname>Bau</surname>
          </string-name>
          , J.-Y. Zhu,
          <string-name>
            <given-names>H.</given-names>
            <surname>Strobelt</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Lapedriza</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Zhou</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Torralba</surname>
          </string-name>
          ,
          <article-title>Understanding the role of individual units in a deep neural network</article-title>
          ,
          <source>Proceedings of the National Academy of Sciences</source>
          <volume>117</volume>
          (
          <year>2020</year>
          )
          <fpage>30071</fpage>
          -
          <lpage>30078</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>I.</given-names>
            <surname>Shumailov</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Shumaylov</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Kazhdan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Zhao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Papernot</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. A.</given-names>
            <surname>Erdogdu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R. J.</given-names>
            <surname>Anderson</surname>
          </string-name>
          ,
          <article-title>Manipulating SGD with data ordering attacks</article-title>
          ,
          <source>in: Advances in Neural Information Processing Systems (NeurIPS)</source>
          , volume
          <volume>34</volume>
          ,
          <year>2021</year>
          , pp.
          <fpage>18021</fpage>
          -
          <lpage>18032</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>S.</given-names>
            <surname>Goldwasser</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. P.</given-names>
            <surname>Kim</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Vaikuntanathan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            <surname>Zamir</surname>
          </string-name>
          ,
          <article-title>Planting undetectable backdoors in machine learning models: [extended abstract]</article-title>
          ,
          <source>in: IEEE Annual Symposium on Foundations of Computer Science (FOCS)</source>
          ,
          <year>2022</year>
          , pp.
          <fpage>931</fpage>
          -
          <lpage>942</lpage>
          . doi:
          <volume>10</volume>
          .1109/FOCS54457.
          <year>2022</year>
          .
          <volume>00092</volume>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>T.</given-names>
            <surname>Cliford</surname>
          </string-name>
          , I. Shumailov,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Zhao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Anderson</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Mullins</surname>
          </string-name>
          ,
          <article-title>ImpNet: Imperceptible and blackboxundetectable backdoors in compiled neural networks</article-title>
          ,
          <year>2022</year>
          . arXiv:
          <volume>2210</volume>
          .
          <fpage>00108</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>D.</given-names>
            <surname>Doran</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Schulz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T. R.</given-names>
            <surname>Besold</surname>
          </string-name>
          ,
          <article-title>What does explainable AI really mean? a new conceptualization of perspectives</article-title>
          ,
          <source>in: CEUR Workshop Proceedings</source>
          , volume
          <year>2071</year>
          , CEUR,
          <year>2018</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>T.</given-names>
            <surname>Oikarinen</surname>
          </string-name>
          , T.-W. Weng, CLIP-Dissect:
          <article-title>Automatic description of neuron representations in deep vision networks</article-title>
          ,
          <source>in: International Conference on Learning Representations, ICLR</source>
          ,
          <year>2023</year>
          . URL: https://openreview.net/forum?id=iPWiwWHc1V.
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <given-names>M. E.</given-names>
            <surname>Zarlenga</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Barbiero</surname>
          </string-name>
          , G. Ciravegna,
          <string-name>
            <given-names>G.</given-names>
            <surname>Marra</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Giannini</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Diligenti</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Shams</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Precioso</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Melacci</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Weller</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Lió</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Jamnik</surname>
          </string-name>
          ,
          <article-title>Concept embedding models: Beyond the accuracyexplainability trade-of</article-title>
          ,
          <source>in: Advances in Neural Information Processing Systems (NeurIPS)</source>
          ,
          <year>2022</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          [18]
          <string-name>
            <given-names>J.</given-names>
            <surname>Lehmann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Hitzler</surname>
          </string-name>
          ,
          <article-title>Concept learning in description logics using refinement operators</article-title>
          ,
          <source>Mach. Learn</source>
          .
          <volume>78</volume>
          (
          <year>2010</year>
          )
          <fpage>203</fpage>
          -
          <lpage>250</lpage>
          . URL: https://doi.org/10.1007/s10994-009-5146-2. doi:
          <volume>10</volume>
          .1007/ s10994-009-5146-2.
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          [19]
          <string-name>
            <given-names>B.</given-names>
            <surname>Zhou</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Zhao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>X.</given-names>
            <surname>Puig</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Xiao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Fidler</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Barriuso</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Torralba</surname>
          </string-name>
          ,
          <article-title>Semantic understanding of scenes through the ADE20K dataset</article-title>
          ,
          <source>International Journal of Computer Vision</source>
          <volume>127</volume>
          (
          <year>2019</year>
          )
          <fpage>302</fpage>
          -
          <lpage>321</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          [20]
          <string-name>
            <surname>M. K. Sarker</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <string-name>
            <surname>Schwartz</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          <string-name>
            <surname>Hitzler</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          <string-name>
            <surname>Zhou</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          <string-name>
            <surname>Nadella</surname>
            ,
            <given-names>B. S.</given-names>
          </string-name>
          <string-name>
            <surname>Minnery</surname>
            , I. Juvina,
            <given-names>M. L.</given-names>
          </string-name>
          <string-name>
            <surname>Raymer</surname>
          </string-name>
          ,
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>