We extract propositional conditional belief bases from multilayer perceptrons, a basic type of feedforward neural networks, and investigate the relation between these two prevalent formalisms from knowledge representation and reasoning (KRR) and machine learning (ML), respectively. The ultimate goal of our work is to imitate with the extracted belief base the main information flow in the original multilayer perceptron detached from specific input data. For this, we introduce a notion of suficient (in)activators of neurons which reflect the most relevant connections within the multilayer perceptron that lead to the (in)activation of the subsequent neurons. While focusing on the binary multi-class classification task, we show that our approach produces consistent belief bases from which principled inferences can be drawn, for instance under System Z. In particular, no inferences are invented by the System Z ranking model that are not in accordance with the initial neural network.
Neural networks [
In contrast to neural networks, knowledge-based
systems [
In this paper, we extract conditional belief bases from a specific type of neural networks called multilayer perceptrons. Multilayer perceptrons are feedforward networks in which information is always processed towards the output, hence there are no cycles in the network. In contrast to general feedforward networks, the neurons in multilayer 22nd International Workshop on Nonmonotonic Reasoning, November 2-4, 2024, Hanoi, Vietnam $ marco.wilhelm@tu-dortmund.de (M. Wilhelm); alexander.hahn@tu-dortmund.de (A. Hahn); gabriele.kern-isberner@tu-dortmund.de (G. Kern-Isberner) 0000-0003-0266-2334 (M. Wilhelm); 0009-0008-6114-2594 (A. Hahn); 0000-0001-8689-5391 (G. Kern-Isberner) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). perceptrons are arranged to at least three fully connected layers with neurons connected to the other neurons from the neighboring layers. The extracted belief base reflects the main information flow within such a multilayer perceptron.
The basic idea of our approach is to identify sets of
predecessors of a neuron the (in)activation of which is
sufifcient to (in)activate . Hereby, the (in)activation of a
neuron means that an input of the multilayer perceptron
triggers the neuron more (less) than a predefined threshold,
i.e., the output value of the neuron is larger (smaller) than
this threshold. Therewith, our approach is related to the
work in [
In more detail, the main contributions of the present paper are as follows: • We introduce a notion of suficient (in)activators of neurons (Definitions 6 and 7). • We show that suficient (in)activators are independent of the input of the multilayer perceptron (Propositions 2 and 3). • Based on the notion of suficient (in)activators, we extract belief bases from multilayer perceptrons (Definition 9). The extracted belief bases are provably consistent with respect to ranking semantics (Proposition 5). • We use the extracted belief bases and their System Z ranking models for binary classification and relate their classification behavior to the direct classification with the initial multilayer perceptrons (Proposition 6).
With our approach we abstract from specific input data
and also from overlay efects of less relevant connections
in the neural networks. The most relevant connections are
formalized in form of easy to understand conditionals. Note
that establishing such formal bridges between neural- and
logic-based models is a very old enterprise and has been
pursued in the first papers on neural networks already [
The rest of the paper is organized as follows. First we recall basics on multilayer perceptrons, in particular with respect to binary multi-class classification, and conditional 1We thank the anonymous referees for their valuable comments.
Activation function
Specification reasoning based on ranking functions (Section 2). Then, we discuss related work on extracting belief bases from multilayer perceptrons within a Description Logic context and show that a naïve translation to propositional conditional belief bases works only to a limited extent (Section 3). Eventually, we propose our novel approach on extracting belief bases based on suficient (in)activators (Section 4) and use this approach for principled binary classification (Section 5). We close the paper with a conclusion that points to future work (Section 6).
In this section, we recall preliminaries on multilayer perceptrons with an application to binary multi-class classification ifrst (Section 2.1). Then, we explain basics on reasoning with conditionals, in particular based on System Z (Section 2.2).
Multilayer perceptrons (MLPs) constitute a widely used type of neural networks which expand single perceptrons to several fully connected layers. We give a brief introduction to neural networks in general and to MLPs in particular. Afterwards, we discuss their application to binary multi-class classification.
Neural Networks Neural networks [
, · )
∑︁
∈pa
to their child nodes/neurons. Hereby, , ∈ R is the
weight of the edge from to (cf. Figure 1). Neurons
without child nodes return the output of the neural network.
Typical activation functions of neural networks are shown
in Table 1. The weights of a neural network and the biases
of the neurons are usually derived from training data, i.e.,
input data for which the expected output is known. Here, we
solely consider neural networks which are already trained.
Multilayer Perceptrons In neural networks, neurons
are usually assigned to layers with diferent functionalities.
Neurons in the first layer, the input layer, receive the input
of the network, and neurons in the last layer, the output
layer, return the output. The layers in-between are called
hidden layers. If a neural network is represented by an
acyclic directed graph, it is called a feedforward network.
In feedforward networks information is always processed
towards the output layer. Multilayer perceptrons constitute
an important subclass of feedforward networks with edges
only between adjacent layers and, taking this condition into
account, fully connected neurons. Multilayer perceptrons
have at least one hidden layer. This hidden layer (as well as
a non-linear activation function) is necessary to distinguish
data that is not linearly separable [
Definition 1 (Multilayer Perceptron). A multilayer perceptron ℳ is a special neural network which is represented by a directed graph (ℳ , ℰℳ ) consisting of a set of vertices ℳ = {, | ∈ [], ∈ []}, 2 the neurons in ℳ, and a set of edges ℰℳ = {(, , +1,)
| ∈ [ − 1], ∈ [], ∈ [+1]}, where ∈ N≥ 2, and ∈ N for ∈ []. Every edge (, , +1,) ∈ ℰℳ is assigned a real-valued weight ,, = , ,+1, , every neuron 0, , ∈ [0], in the input layer is assigned the identity function 0, : R → R with 0, () = , and every further neuron , with > 0, ∈ [], is assigned a function , : R− 1+1 → R with , (⃗) = ( , + − 1,ℎ, · − 1,ℎ (⃗)), (1) ∑︁ ℎ∈[− 1] where is the activation function of ℳ and , ∈ R is the bias of , . The input of ℳ is any vector ⃗ ∈ R0+1 whereby the -th component of ⃗ is passed to the neuron 0, , and the output of ℳ is ℳ(⃗) = (,0 (⃗), . . . , , (⃗)) ∈ R+1.
Figure 2 shows a schema of a multilayer perceptron with one hidden layer ( = 2). For a neuron ∈ ℳ, we will denote the set of its parent nodes by pa which will help us to avoid indices. 2For ∈ N, we abbreviate [] = {0, 1, . . . , }. 0,0 0, ...
...
0,0
1,0
1,
Binary Multi-Class Classification A possible
application of neural networks in general and multilayer
perceptrons in particular is binary (multi-class) classification [
In practice, one usually uses sigmoid functions like the logistic function (cf. Table 1) for classification, instead, which range over the interval (0, 1) and, thus, allow for a gradual answer behavior. Furthermore, the Heaviside function cannot be used for gradient-based training because it is not diferentiable at 0 and the derivative is 0 at all other points, while the logistics function can be diferentiated any number of times which makes it particularly suited for numerical methods. In this paper, we equip multilayer perceptrons with the logistic function as an activation function and denote this by ℳlog. Our approach works with any sigmoid function, though. We consider the following three-valued interpretation of the output of neurons in ℳlog. Definition 2 ((In)active Neurons). Let ℳlog be a multilayer perceptron, let be a neuron in ℳlog, let ⃗ be an input vector of ℳlog, and let ∈ [0, 0.5). We call a tolerance factor, and say that neuron is (cf. (1)) • activated by ⃗ wrt. , or active for short, if • inactivated by ⃗ wrt. , or inactive for short, if (⃗) ≥ 1 − ,
(⃗) ≤ , • ambiguous otherwise.
With Definition 2, we can say that an input vector ⃗ of ℳlog is classified as an instance of class , represented by the neuron in the output layer of ℳlog, if is activated by ⃗, and ⃗ is declassified as an instance of class if is inactivated by ⃗. Otherwise, the membership to is ambiguous. We give an example. 0,0 0,1 0,2
Within the field of nonmonotonic reasoning, conditionals [
guage defined over a finite signature Σ as usual.3 A
conditional (|) with , ∈ ℒ(Σ) is a formal representation
of the defeasible statement: “If holds, then usually
holds, too.” Finite sets of conditionals serve as belief bases.
The semantics of conditionals is based on possible worlds.
Here, possible worlds ∈ Ω(Σ) are the propositional
interpretations of ℒ(Σ) represented as complete conjunctions
of literals. That is, every atom from Σ occurs in a possible
world once, either positive or negated. A ranking
function : Ω(Σ) → N0 ∪ {∞} [
min ∈Ω(Σ) : |= () and accept a conditional (|) if () < (). A ranking function is a ranking model of a belief base Δ if accepts all conditionals in Δ. If Δ has a ranking model, then it is called consistent. Ranking models of Δ yield a nonmonotonic inference relation between Δ and conditionals (|) in the following sense:
Δ |∼ (|) if () < () or () = ∞.
System Z A sophisticated ranking model of consistent
belief bases is provided by System Z [
Example 2. A typical example to illustrate System Z is the Tweety example. Let Δ = { 1, 2, 3} with 1 = (|), 2 = ( |), 3 = ( |), state that penguins like Tweety are usually birds and birds usually fly, but penguins usually do not fly. The System Z tolerance partition of Δ is (Δ) = (Δ0, Δ1) with Δ0 = { 2}, Δ1 = { 1, 3}. 3In order to shorten logical expressions, we use the abbreviations for conjunctions ∧ and for negations ¬ where , ∈ ℒ(Σ). The resulting System Z ranking model is Δ () = ⎧⎪0, ∈ { , , } ⎨
1, ∈ { , } ⎪⎩2, ∈ { , , } .
System Z coincides with rational closure [
In this section, we briefly recall the extraction of beliefs from
neural networks as presented in [
In [
We formalize the extraction of propositional conditionals
in analogy to the defeasible subsumptions in [
Definition 4 (Synaptic Conditionals). Let be a neural network. Then we define for each neuron ∈ the backward synaptic conditionals as follows: Analogously, we define forward synaptic conditionals: Δ←+ ( ) = {︀ ( ′| ) | ′ ∈ pa+ }︀ , Δ−← ( ) = {︀ ( ′| ) | ′ ∈ pa− }︀ . Δ+→( ) = {︀ ( | ′) | ′ ∈ pa+ }︀ , Δ−→( ) = {︀ ( | ′) | ′ ∈ pa− }︀ .
Note that backward synaptic conditionals are abductive
in nature. The idea of backward synaptic conditionals is that
4Please see [
We can now define belief bases containing synaptic conditionals.
Definition 5 (Synaptic Belief Bases). Let be a neural network. We define the backward/forward synaptic belief bases as the union of all synaptic conditionals that share the same direction, i.e., Δ← = Δ→ = ∈ ∈ ⋃︁ (︀ Δ←+ ( ) ∪ Δ←− ( ))︀ , ⋃︁ (︀ Δ+→( ) ∪ Δ−→( ))︀ .
The synaptic belief bases capture the information that is immediately available from the structure of the neural network, namely the positive or negative influence neurons have on each other based on the trained synaptic weights. From a formal perspective, the direction of the conditionals is arbitrary. As long as the two directions are not mixed, the synaptic belief base extracted from a multilayer perceptron is consistent.
Proposition 1. For every multilayer perceptron ℳ, the synaptic belief bases Δ←ℳ and Δ→ℳ are consistent. Proof. We prove the proposition for Δ←ℳ by showing that the layers of the multilayer perceptron ℳ induce a tolerance partition of Δ←ℳ . Let ( + 1) ∈ N be the number of layers in ℳ and let be the set of neurons in the -th layer of ℳ. Then, (Δ0, . . . , Δ− 1) defined by Δ = {(˙ ′| ) ∈ Δ←ℳ | ∈ +1} partitions Δ←ℳ . Now, we show that every conditional in Δ is tolerated by ⋃︀ : ≤ < Δ. Let Δ and ∈ +1 be arbitrary but fixed. We choose a possible world with the following properties: (1) |= , (2) |= ′ if ( ′| ) ∈ Δ for every ′ ∈ , and (3) |= ′′ for every ′′ ∈ with < ≤ and ̸= ′′. It can be quickly checked that all three properties concern diferent neurons and, hence, can be satisfied by at the same time. The properties (1) and (2) together ensure that verifies all conditionals with antecedent ; property (3) ensures that is indiferent with respect to all other conditionals in all Δ with ≤ < . Since Δ and were chosen arbitrarily, this proves that every conditional in every Δ is tolerated by all Δ (with 0 ≤ ≤ < ).
The proof for Δ→ℳ is analogous; only the order of the partition needs to be reversed.
In contrast to [
Δ←ℳleoxg = {(0,0|1,0), (0,1|1,0), (0,2|1,0), and Δ→ℳleoxg = {(1,0|0,0), (1,1|0,0), (1,2|0,0), (0,0|1,1), (0,1|1,1), (0,2|1,1), (0,0|1,2), (0,1|1,2), (0,2|1,2), (1,0|2,0), (1,1|2,0), (1,2|2,0), (1,0|2,1), (1,1|2,1), (1,2|2,1), (1,0|2,2), (1,1|2,2), (1,2|2,2)}, (1,0|0,1), (1,1|0,1), (1,2|0,1), (1,0|0,2), (1,1|0,2), (1,2|0,2), (2,0|1,0), (2,1|1,0), (2,2|1,0), (2,0|1,1), (2,1|1,1), (2,2|1,1), (2,0|1,2), (2,1|1,2), (2,2|1,2)}.
In both cases (backward/forward), the Z-partition collapses: (Δ←ℳleoxg ) = (Δ←ℳleoxg ), (Δ→ℳleoxg ) = (Δ→ℳleoxg ), and we have, with (2,2) = 0,0 ∧ 0,1 ∧ 0,2, Δ |̸∼ Δ (2,2| (2,2)) regardless of whether Δ = Δ←ℳleoxg or Δ = Δ→ℳleoxg because Δ (2,2 ∧ (2,2)) = 1 ̸< 0 = Δ (2,2 ∧ (2,2)) for Δ = Δ←ℳleoxg , and
Δ (2,2 ∧ (2,2)) = 1 ̸< 1 = Δ (2,2 ∧ (2,2)) for Δ = Δ→ℳleoxg . Thus, In both cases this contradicts the fact that the input vector ⃗ = (0.9, 0.8, 0.1) triggers the neurons 0,0, 0,1, and 0,2 and is classified as an instance ex of 2,2 by ℳlog (cf. Example 1). Hence, we come to diferent ex conclusions if we either classify ⃗ = (0.9, 0.8, 0.1) by ℳlog directly or classify ⃗ based on the synaptic belief bases.
The example above shows that belief bases consisting of synaptic conditionals (only) are too basic to give any guarantees with respect to reasoning behavior when using System Z. It is to be expected that a qualitative belief base cannot provide inferences on the same level of detail like the original neural network. The example also shows that the belief base introduces new inferences which cannot be obtained from the neural network. This can be considered undesirable. Therefore, in order to make better use of the quantitative information learned by the neural network, we make the extracted conditionals more complex to capture relevant influences among the neurons better in the next section.
Now, we propose a more sophisticated approach than synaptic conditionals for extracting conditional belief bases from ( + ∑︀ = ( + ∑︀ ≥ 1 − .
+ ∑︀ + ∑︀ + ∑︀ + · + ∑︀
′, · ′ ) ′∈pa ′∈pa+ ∩+ ′, · ′ ′∈pa+ ∖+ ′, · ′ ′∈pa− ∩− ′, · ′ ′∈pa− ∖− ′, · ′ ) ∑︀
′∈pa− ∩− ′, ′∈pa− ∖− ′, ) ≥ ( +(1 − ) · ∑︀ ′∈pa+ ∩+ ′, Proof. (⇐) Assume that (2) and ′ ≥ 1 and ′ ≤ for ′ ∈ ′ hold. Then, − for ′ ∈ + With Hereby, we used ∑︀ (+, − ) is a suficient activator of . ′∈pa+ ∖+ ′, · ′ ≥ (⇒) We prove the contraposition. Assume that
( + (1 − ) · + ·
∑︁
′∈pa− ∩−
′, +
∑︁
∑︁
′∈pa+ ∩+
′∈pa− ∖−
′,
′, ) < 1
−
∑︁
holds. We have to show that there is ′ ∈ [
− . if ′ ∈ pa+ ∩ + if ′ ∈ pa+ ∖ + if ′ ∈ pa− ∩ − if ′ ∈ pa− ∖ − multilayer perceptrons. On the one hand, this means an abstraction from specific input data to generalized defeasible rules, here conditionals. On the other hand, the embedding of the essential information flow of multilayer perceptrons into a logical framework allows us to draw principled inferences of verifiable quality.
The basic idea of our method is to extract conditionals , + = ( | , +
) from a multilayer perceptron ℳlog where the consequence refers to a neuron from and the premise , + to sets of parent nodes of which are (in combination) “most relevant” for the activation of . Relevance here means that the conditional ( | , + fective, i.e., , + is true, only if it is guaranteed that the ) is efneuron is suficiently highly activated. Hence, it is reliℳlog ably justified to infer . Analogously, we extract conditionals , −
= ( | ,− ) wrt. the inactivation of . The “most based on the notion of suficient (in)activators . relevant” parents nodes of neurons in ℳlog are identified
We assume that the input of the multilayer
perceptron ℳlog is normalized to ⃗ ∈ [
Based on the concept of active and inactive neurons, we deifne (sets of) parent nodes of neurons in a multilayer perceptron ℳlog which are suficient to activate resp. deactivate the neurons, independent of the specific input vector ⃗. classification scheme. Further, let be a neuron in Definition 6 (Suficient Activator) . Let (ℳlog, ) be a from a hidden layer or the output layer. We call a tuple ℳlog (+, − ) ⊆ pa2 with + ∩ − = ∅ a suficient activator of wrt. , if the activation of the neurons in + and the inactivation of the neurons in − implies the activation − for ′ ∈ + and ′ ≤ of ; formally, if ′ ≥ 1 for ′ ∈ − implies the output of the neurons ′ ∈ pa with ′ ∈/ + irrelevant for the activation of , regardless of the concrete ∪− is input of ℳlog, as captured in the next proposition. Proposition 2. Let (ℳlog, ) be a classification scheme, and let be a neuron in output layer. Then, (+, − ) ⊆ is a suficient activator of if ℳlog from a hidden layer or the pa2 with +
∩ − = ∅ ( ) is that ( + (1 − ) · ∑︀
′∈pa+ ∩+ ′, + · + ∑︀ ∑︀
′∈pa− ∩− ′, ′∈pa− ∖− ′, ) ≥ 1 − .
(2) + ∑︀ + ∑︀ + ∑︀ + · + ∑︀
′, · ′ ) ′∈pa ′∈pa+ ∩+ ′, · ′ ′∈pa+ ∖+ ′, · ′ ′∈pa− ∩− ′, · ′ ′∈pa− ∖− ′, · ′ ) ∑︀
′∈pa− ∩− ′, ′∈pa− ∖− ′, ) = ( +(1 − ) · ∑︀
′∈pa+ ∩+ ′, < 1 − , to one can rewrite ( + ∑︀ ′, = 0 holds in this case anyway. which finishes the proof. Note that the choice of ′ = 0 in case of ′ ∈ pa ∖ (pa+ ∪ pa− ) is not mandatory because
In this proof of Proposition 2 we have exploited that the logistic function is non-negative. If one wants to apply similar techniques to arbitrary sigmoid functions which are not necessarily non-negative but bounded by (, ) ⊂ ℛ ′∈pa ′, · ′ ) beforehand (( − )( ′ +
′, · ′′ )) ∑︁ with ′ = − 1 · ( + · ∑︀ ∈pa ′, ) and ′′ = ′ − where ′′ is bounded by (0, 1) for all ∈ pa . − Note that in this case the thresholds for neurons being (in)active have to be adjusted from 1 + as well, now with ∈ [0, − 2 ). − and to − and + not.
Proposition 2 can be used to compute suficient activators. For a neuron one generates each pair (+, − ) with ∈ pa+ and −
∈ pa− and tests whether (2) holds or of 1,1 because ex Example 4. We consider the multilayer perceptron ℳlog from Example 1 (cf. Table 2) and the tolerance factor = 0.3. Then, for instance, ({0,0, 0,1}, ∅) is a suficient activator (0.7 · (0.91 + 0.81) − 0.09) ≈ 0.753 ≥ 0.7, cause where is the logistic function (cf. Table 1). Note that ({0,0}, ∅) is not a suficient activator of 1,1, instead, be(0.7 · (0.91) − 0.09) ≈ 0.633 < 0.7.
Analogously to suficient activators, we can define suficient inactivators.
Definition 7 (Suficient Inactivator) . Let (ℳlog, ) be a den layer or the output layer. We call a tuple (+, − ) classification scheme, and let be a neuron in ℳ from a hidpa2 with + ∩ −
= ∅ a suficient inactivator the activation of the neurons in + and the inactivation of the neurons in − implies the inactivation of ; formally, if ′ ≥ 1 − for ′ ∈ + and ′ ≤ for ′ ∈ − implies ⊆ of wrt. if, and let be a neuron in ℳlog from a hidden layer or the pa2 with + ∩ − = ∅ is a ( + · ∑︀
′∈pa+ ∩− ′, + ∑︀ + (1 − ) · ′∈pa+ ∖− ′, ∑︀ ′∈pa− ∩+ ′, ) ≤ .
(3) Proof. The proof is similar to the proof of Proposition 2. For the direction (⇐) note that ∑︀ ′∈pa− ∖+ ′, · ′ ≤ 0.
For the proof of the contraposition of (⇒), we select
Again, this proposition can be used to compute suficient inactivators as the next example shows. inactivator of 1,0 because tor Example 5. Again, we consider the multilayer perceptron ℳleoxg from Example 1 (cf. Table 2) and the tolerance fac= 0.3. Then, ({0,0, 0,2}, {0,1}) is a suficient cause
(0.7 · (− 1.27 − 0.91) + 0.3 · 1.23) ≈ 0.239 ≤ 0.3, where is the logistic function (cf. Table 1). Note that ({0,0, 0,2}, ∅) is not a suficient inactivator of 1,0 be(0.7 · (− 1.27 − 0.91) + 1.23) ≈ 0.427 > 0.3.
For tuples of sets (1, 2) and (1, 2) we write (1, 2)
⊑ (1, 2) if 1 ⊆ ously, if (+, − ) is a suficient activator of 1 and 2 ⊆ 2. Obvi and, for (′+, ′− ) ∈
pa( )2, (+, − ) ⊑ (′+, ′− ) holds, then (′+, ′− ) is a suficient activator of , too. A similar result holds for suficient inactivators. output layer. Then, Proposition 4. Let (ℳlog, ) be a classification scheme, and let be a neuron in ℳlog from a hidden layer or the is a suficient activator of
, too, • if (+, − ) is a suficient activator of , then (′+, ′− ) ∈ pa2 with (+, − ) ⊑ (′+, ′− ) a suficient inactivator of
, too. • if (+, − ) is a suficient inactivator of , then (′+, ′− ) ∈ pa2 with (+, − ) ⊑ (′+, ′− ) is let ′ ≥ 1 From + ⊆ Proof. Let (+, − ) be a suficient activator of , and let (′+, ′− ) ∈ pa2 with (+, − ) ⊑ (′+, ′− ). Further, − for ′ ∈ ′+ and ′ ≤ for ′ ∈ ′− . ′+ and − ⊆ ′− it follows that ′ ≥ 1− for ′ ∈ + and ′ ≤
for ′ ∈ − Then, because (+, − ) is a suficient activator, holds as well.
∑︁ (ℳlog, ) be a classification scheme, and let be a neuron in ℳlog from a hidden layer or the output layer. Then, of . and (+, − ) ̸= (′+ of , • A suficient activator
(+, − ) of is minimal if no (′+, ′− ) ∈ pa2 with (′+, ′− ) ⊑ (+, − ) and (′+, ′− ) ̸= (+, − ) is a suficient activator • A suficient inactivator
(+, − ) of is minimal if no (′+, ′− ) ∈ pa2 with (+, − ) ⊑ ′+, ′− ) , ′− ) is a suficient inactivator inactivators of wrt. with ℐmin( ).
We denote the set of the minimal suficient activators of wrt. with min( ) and the set of the minimal suficient
We consider our running example.
Example 6. The minimal suficient (in)activators of the neurons in ℳleoxg from Example 1 (cf. Table 2) with respect to the tolerance factor = 0.3 are shown in Table 3 resp. Table 4. Minimal suficient (in)activators allow for a graphical representation (cf. Figure 4). For instance, the minimal Minimal suficient inactivators of the neurons in the hidden resp. output layer of ℳleoxg from Example 1 wrt. = 0.3. suficient inactivator
({0,0, 0,2}, {0,1}) of 1,0 can be visualized as three outgoing edges from 0,0, 0,1, and 0,2, respectively, which conjointly result in 1,0. The dashed line in Figure 4 after these three edges have met indicates that ({0,0, 0,2}, {0,1}) is a suficient inactivator (and not an activator) of 1,0 and the dashed line from 0,1 indicates that 0,1 has a negative influence on the inactivation of 1,0 (because the weight 0,1,0 is positive).
Altogether, (minimal) suficient activators and inactivators make it possible to abstract from the concrete input data of a multilayer perceptron ℳlog and reveal the essential streams of information within ℳlog. This is the motivation for our following extraction of conditional belief bases from multilayer perceptrons.
on suficient (in)activators. In Now, we describe our approach on extracting a conditional belief base Δℳlog from a multilayer perceptron ℳlog based Δℳlog we formalize for
ℳleoxg from Example 1. Solid lines indicate activation and dashed ⋀︁ min( ) ̸= ∅ in case of , +
, ℐmin( ) ̸= ∅ in case of ,− .
(* ) define the extraction of the belief base Note that the conditionals depend on the tolerance factor because the sets of (minimal) suficient (in)activators depend on . However, the conditionals are not dependent on any input vector of ℳlog, since abstracts from that. Based on that, we Δℳlog from ℳlog via Δℳlog = { ,+ | ∈ +} ∪ { ,− | ∈ − },
, + is the set of neurons for which the condiwhere tional , + exists, and where
for which the conditional , −
, − is the set of neurons exists, i.e., (* ) applies.
The number of conditionals in Δℳlog is bounded by the number of neurons in
ℳlog (minus the input layer) which means a higher degree of abstraction than prevalent in synaptic belief bases (cf. Definition 5) the cardinality of which is bounded by the number of edges in ℳlog. Furthermore, the condition (* ) in Definition 9 ensures that the conditionals , + (resp. ,− ) are added to Δℳlog only if has suficient activators (inactivators). This prevents from conditionals of the form ( |⊥) and ( |⊥) in Δℳlog which would cause inconsistencies according to our acceptance definition of conditionals. If there is a neuron with improve the chance of obtaining such a conditional. , +, , −
∈/ Δℳlog , then one can increase in order to in Δ0ℳ.3log are Example 7. We consider ℳleoxg from Example 1 and the tolerance factor = 0.3. The minimal suficient (in)activators of the neurons in ℳleoxg are shown in Table 3 resp. Table 4 from which we can derive the belief base Δ0ℳ.3log . The conditionals 0.13,,0− = (1,0|0,0 ∧ 0,2 ∧ 0,1), 0.13,,1+ = (1,1|0,0 ∧ 0,1), 0.13,,2+ = (1,2|0,2), 0.23,,0+ = (2,0|1,0 ∧ 1,2 ∧ 1,1), 0.23,,1− = (2,1|1,0 ∧ 1,2 ∧ 1,1), 0.23,,2+ = (2,2|1,0 ∧ 1,2 ∨ 1,1).
In particular, note the disjunction in the premise of 0.23,,2+ because of the two (diferent) minimal suficient activators of 2,2. make use of the following lemma.
The belief base Δℳlog is consistent. To show this, we Lemma 1. Let (ℳlog, ) be a classification scheme. Then, for every neuron from a hidden layer or the output layer of ℳlog it holds that (cf. Definition 9) ,+
∧ ,− ≡ ⊥ .
Proof. Assume that , +
∧ ,− is a possible world , a suficient activator ≢ ⊥ holds. Then, there (+, − ) of wrt. , and a suficient inactivator (+, − ) of wrt. such that |=
⋀︁ ′∈+ which its consistency follows. Let + 1 be the number of layers in ℳlog and, for = 0, 1, . . . , , let be the set of neurons in the -th layer. Then, (Δ1, . . . , Δ) with has a tolerance partition from Δ = { , +
∈ Δℳlog | ∈ } ∪ { , −
∈ Δℳlog | ∈ } and − are disjoint). ∈ Ω(− 1) with |= ⋀︀ show that is tolerated by ⋃︀ for = 1, . . . , is a partition of Δℳlog (modulo empty sets). Let ∈ Δ , provided hat Δ ̸= ∅ . We have to = Δ. For this, let be of the form the form , + for some ∈ . The proof for of , − is analogous. By construction of ,+, there is (+, − ) ∈ min( ) and a (partial) possible world ′∈+ ′ ∧ ⋀︀ ′∈− ′ (+
Thanks to Lemma 1, we can extend to a (partial) possible world ′ ∈ Ω(− 1 ∪ ) such that all conditionals in Δ are either not applicable or verified by concatenating ′ to in case of |= ,+′ or concatenating ′ to in case of |= ,−′ for ′ ∈ . In particular, ′ verifies ,+.
By a repeated application of this argument, we can construct a (partial) possible world ′′ ∈ Ω(⋃︀=− 1 ) which veriifes , + and falsifies no conditional from ⋃︀ = Δ. Eventually, this (partial) possible world can be extended to a possible world in Ω(⋃︀
=0 ) by the concatenation of the remaining ground atoms, either positive or negated which can be chosen freely.
Note that the belief base Δℳlog might be empty, namely if for all neurons in ℳlog there is no suficient (in)activator. On the contrary, if a neuron can be (in)activated, then there is a suficient (in)activator of
conditional wrt. in Δℳlog most important information flow in . Thus, Δℳlog ℳlog. so that there is a reflects the
Now, we discuss how to perform binary (multi-class) classification based on the belief base extracted from a multilayer perceptron Δℳlog which we have ℳlog (cf. Definition 9). Recall that, following Definition 2, we can say that an input vector ⃗ of ℳlog is classified is a tolerance factor. We denote this with as represented by the neuron from the output layer of ℳlog if ℳlog(⃗) ≥ 1 − (resp. ℳlog(⃗) ≤ ) where (resp. declassified ) ℳlog ℳlog, ⃗ |∼ if ℳlog, ⃗ |∼ if ℳlog(⃗) ≥ 1
− ℳlog(⃗) ≤ . model Δ ℳlog
of Δℳlog .
We lift this idea of classifying ⃗ from
base Δℳlog . Thereby, we make use of the System Z ranking ℳlog to the belief Definition 10 (Z-Classification) . Let (ℳlog, ) be a classification scheme, let from ℳlog, and let the input layer of ℳlog which are activated by ⃗ wrt. , and ⃗ the set of neurons which are inactivated. Then, we say that an input vector ⃗ of ℳlog is • Z-classified as wrt. a neuron from the output layer of ℳlog, denoted by Δℳlog , ⃗ |∼ , if ℳlog, accepts ( |
⋀︁ ′), • Z-declassified as , denoted by Δℳlog , ⃗ |∼ , if ℳlog, accepts ( |
⋀︁ inferences drawn from ℳlog, can be understood, in some input layer of ℳlog which are activated resp. inactivated by the input ⃗ wrt. (cf. Definition 10). Further, let + 1 be the number of layers in ℳlog, and, for = 0, 1, . . . , , let be the set of neurons in the -th layer of ℳlog. We prove that Δℳlog , ⃗ |∼ implies ℳlog, ⃗ |∼ . The proof that Δℳlog , ⃗ |∼ implies ℳlog, ⃗ |∼ is anal
Let Δℳlog , ⃗ |∼ , i.e., by definition, ℳlog, accepts the conditional ( | ) with =
⋀︁ ′∈⃗ is falsified. Hence, ℳlog, (′) = 0. Because ℳlog, accepts the conditional ( | ), none of these extensions ′ ℳlog, ( ∧ ) = 0 would hold which contradicts the acceptance of ( | ). As a consequence, the conditional , + (cf. Definition 9) must be in Δℳlog which is the only possibility to exclude from the extensions ′ (and which is also accepted in all the extensions ′). Otherwise, there is no reason why not to have an extension ′ with ′ |= .
In more detail, either there is an extension ′ of with ′ |= and
ℳlog, (′) = 0 which contradicts the acceptance of ( | ), or ℳlog, (′) > 0 for all such exten sion ′ which requires a conditional in Δℳlog that is falsiifed in
′. The only candidate for such a conditional would be , +
. As a consequence of the acceptance of ,+, the input vector ⃗ activates at least one suficient activator of .
From this, it follows that ⃗ also activates in ℳlog.
We recall our running example to illustrate this proposition.
Example 8. We consider the same scenario as in Example 1, i.e., the multilayer perceptron ℳleoxg, the tolerance factor = 0.3, and the input vector ⃗ = (0.9, 0.8, 0.1). Then, ⃗ 0.3 = {0,0, 0,1}, ℐ⃗ Further, the Z-partition of Δ0ℳ.3log is (Δ0ℳ.3leoxg ) = (Δ0ℳ.3leoxg ), so that, for (2,2| 2,2 ) with 2,2 = 0,0 ∧ 0,1 ∧ 0,2, Δ (2,2 ∧ 2,2 ) = 0 < 1 = Δ (2,2 ∧ 2,2 ) the result from Example 1.
Thus, we classify ⃗ as an instance of 2,2 in accordance with
Our approach focuses attention on the main dependencies among the neurons in multilayer perceptrons. In contrast to the synaptic conditionals in Section 3, the influence of several parent nodes on a neuron is aggregated, with the guarantee that the aggregated parent nodes are able to (in)active . A depiction of these aggregated influences is shown in Figure 4 for our running example. Figure 4 can be understood as a visualization of the main information flow in ℳleoxg.
We proposed an approach on extracting propositional conditional belief bases from multilayer perceptrons (MLPs) for binary multi-class classification. The conditionals relate to the main information flow in the multilayer perceptron detached from specific input vectors. Therewith, our approach abstracts from both the input data as well as overlay efects in the network and rebuilds the backbone of the network within a prevalent KRR formalism. The main idea of our approach is to exploit suficient (in)activators of neurons the (in)activation of which guarantees that is (in)activated as well. The extracted conditional belief base allows for drawing inferences in a principled way, for instance, under System Z. It is guaranteed that the belief base is consistent and does not invent inferences that cannot be drawn from the multilayer perceptron.
In recent work [
Also in future work, we want to extract conditionals
from multilayer perceptrons that are based on “necessary
(in)activators” and can be used for explaining classifications
that are made by the multilayer perceptrons. Therewith,
we expect to be able to bound all possible classifications
from two directions (upper and lower bound) which, as we
hope, can help to better understand the essence of binary
multi-class classification based on multilayer perceptrons.
Further research directions could be to investigate how the
choice of the tolerance factor influences the shape of the
conditional belief base and how diferent inference
operators, e.g., based on System P [
This work was supported by DFG Grant KE 1413/14-1 of the German Research Foundation (DFG) awarded to Gabriele Kern-Isberner.