Neuro-symbolic rule learning has attracted lots of attention as it ofers better interpretability than pure neural models and scales better than symbolic rule learning. A recent approach named pix2rule proposes a neural Disjunctive Normal Form (neural DNF) module to learn symbolic rules with feed-forward layers. Although proved to be efective in synthetic binary classification, pix2rule has not been applied to more challenging tasks such as multi-label and multi-class classifications over real-world data. In this paper, we address this limitation by extending the neural DNF module to (i) support rule learning in real-world multi-class and multi-label classification tasks, (ii) enforce the symbolic property of mutual exclusivity (i.e. predicting exactly one class) in multi-class classification, and (iii) explore its scalability over large inputs and outputs. We train a vanilla neural DNF model similar to pix2rule's neural DNF module for multi-label classification, and we propose a novel extended model called neural DNF-EO (Exactly One) which enforces mutual exclusivity in multi-class classification. We evaluate the classification performance, scalability and interpretability of our neural DNF-based models, and compare them against pure neural models and a state-of-the-art symbolic rule learner named FastLAS. We demonstrate that our neural DNF-based models perform similarly to neural networks, but provide better interpretability by enabling the extraction of logical rules. Our models also scale well when the rule search space grows in size, in contrast to FastLAS, which fails to learn in multi-class classification tasks with 200 classes and in all multi-label settings.
The recent success of deep learning has started to bring transformative impacts in diferent
sectors of our society. However, they are not interpretable and cannot be used to solve
realworld problems that require safe and sound reasoning. There has been an increasing interest in
neuro-symbolic approaches that combine symbolic reasoning with neural networks [
In this paper, we build upon pix2rule’s neural DNF module to support rule learning in
realworld multi-class and multi-label classification tasks. We investigate the learning capability of
the vanilla neural DNF model similar to the neural DNF module in pix2rule on real-world
multilabel classification tasks. We also propose a novel model called neural DNF-EO (Exactly-One)
for multi-class classification to address its requirement of mutual exclusivity, i.e. having exactly
one class to be true at a time. In a symbolic learner, logical constraints enforce mutual exclusion,
but a vanilla MLP is not able to strictly enforce it with a cross-entropy loss. Our neural DNF-EO
model is trained with a cross-entropy loss but enforces mutual exclusion, realising the symbolic
constraint in a diferentiable fashion. We evaluate our models over two real-world datasets, the
multi-class dataset CUB-200-2011 [
This chapter provides a summary of the neural Disjunctive Normal Form (neural DNF) module
presented in pix2rule [
= (max || −
∑︁ ||)
(1)
(2)
where are trainable weights, is the activation function tanh, and is the layer output.
By setting to be 1 or -1, the layer can approximate conjunction or disjunction. Using this
formulation instead of a t-norm fuzzy logic with a continuous input space of [
Based on the semi-symbolic layers and the neural DNF module proposed in pix2rule [
Figure 1 consists of a semi-symbolic conjunctive layer followed by a disjunctive one, sharing the same architectural design as pix2rule’s neural DNF module. It is suitable for independent rule learning in both binary and multi-label classification tasks. In the latter case, the target (binary) predicates are independent of each other.
A multi-class classification task requires exactly one class to be true at a time. This can be written as a logical constraint expressed as follows: ← (), ( ), ̸= . It states that two diferent classes cannot be true at the same time.
This constraint is not directly satisfiable by an MLP trained with a cross-entropy loss. An MLP trained attr1 attr2 under a cross-entropy loss takes the arg max of the output when predicting a class at inference time and does not need to enforce mutual exclusion over its outputs. Hence, it is not capable of capturing the logical constraint. Similarly, training our vanilla neural DNF model with a cross-entropy loss does not enforce mutual exclusivity on the model. At inference time, a neural DNF-based model’s tanh output of a class is treated as True (⊤) if the value is greater than 0 and otherwise False (⊥). Since the softmax function in the cross-entropy loss only amplifies the diference between the output logits of each class, it does not have the concept of a value being True if it goes above a threshold, which is required for us to interpret our neural DNF-based model’s outputs. The vanilla neural DNF model’s output logits under the cross-entropy loss are only trained to be diferent from each other and not to be extreme values that fall on diferent sides of our desired threshold value. For example, if the raw output logits of our vanilla neural DNF model are ˜ = [− 2, 1.5, 3] with a target class of 2 (zero-indexed), then softmax(˜) = [0.006, 0.181, 0.813] which is very diferent between each pair, and the cross-entropy loss would be 0.207. However, the tanh output would be tanh(˜) = [− 0.964, 0.905, 0.995], which would be interpreted as [⊥, ⊤, ⊤] and violate the ‘exactly-one-class’ constraint in the multi-class setting. Our experiments of using cross-entropy loss to train vanilla neural DNF models on multi-class classification tasks (in Appendix B.3.2) show this issue of multiple classes being predicted as true, by interpreting the output to be discrete True and False value at threshold value 0. This motivates our second model, which (i) can be trained with cross-entropy loss, even though it only pushes output logits to be diferent; (ii) but still have the property of mutual exclusivity guaranteed.
The neural DNF-EO model shown (inlayFeriguCr2e in2 thhaes figaurec)ontshtartainretflecltasyer ConLjauynecrtive DisLj(uaCny1ce)trive CoCEnxosantr(cjaCutilnn2yct-)otLinvaeeyer t∧he,̸ =‘exnaocttly-one’’ cwonhsitcrhainentf‘orcesm← u- attr1 conj1 disj1 wFeiixgehdts const1 class 1 (tiunatlheexdcoluttseiodnyeolnlotwhebopxlaiinnthnee ufigruarle)D. NIfF attr2 conj2 disj2 const2 class 2 jmuonrcetitvhealnayoenreCo1u,tpnuotcilsatsrsuweoafutledr btheetrduise- ... conj3 after the constraint layer C2. This con- attr34 ... disj3 const3 class 3 straint layer C2 is initialised with all connections being -61, representing the logi- conj9 cal negation ‘not’, and its weights are not updated. During training, the constraint Plain Neural DNF layer C2’s output logits are pushed to be diferent and to match with the expected Figure 2: An example of a neural DNF-EO model. class by the cross-entropy loss, but the The plain neural DNF in the dotted yellow output of disjunctive layer C1 of the plain box is identical to a vanilla neural DNF neural DNF is pushed to have only one model. value being true and all the rest being false. At inference time, we remove the constraint layer C2 and inference with the plain neural DNF that is enforced to output exactly one class true at
1), and the output should be approximately 1 or -1. a time, thus providing mutual exclusivity.
In order to extract rules from these neural DNF-based models, we follow and expand on the post-training processes used in pix2rule. Our post-training pipeline consists of 4 stages: pruning, finetuning, thresholding and extraction. Pruning, thresholding and extraction remain mostly the same as in pix2rule except for the metrics we use, while we add finetuning as a new step to adjust for learning and extracting rules in real-world datasets.
Pruning process sets the weights that do not heavily afect the performance of the neural DNF-based model to 0, where the accepted performance drop is controlled by a hyperparameter . We prune the disjunctive layer first followed by the conjunctive layer. We also remove any disjunct that uses a conjunction with an empty body, i.e. the conjunction has weight 0 for all its conjuncts. For the neural DNF-EO model, we remove the constraint layer C2 and prune only the plain neural DNF.
Finetuning process adjusts the model’s weights by re-training the neural DNF-based model on the training set without updating the pruned weights. For the neural DNF-EO model, we add back the constraint layer after the pruned plain neural DNF during the finetuning process to maintain mutual exclusivity.
Thresholding process goes through a range of possible threshold values, such that if the magnitude of the weight is greater than that threshold, the magnitude of the weight would be set to 6. By setting the value of weights to 6 or -6, tanh output can be saturated close to 1 and -1. We choose the threshold value that gives the best performance based on the validation set. Again, the constraint layer C2 in the neural DNF-EO model is dropped and the thresholding is done on the plain neural DNF.
Symbolic rule extraction is done by taking input attributes/conjunctions connected to weights with value ± 6 as the conjunct/disjunct of their connected conjunction/disjunction. And each disjunct of the disjunction can be treated as a separate rule to classify a class or label. We express the extracted rules as an ASP program.
We experiment on two datasets: CUB-200-2011 [
The attribute annotation has two components. The ‘present’ label represents whether the
sample has that attribute, where 0 means attribute not present and 1 means attribute present.
The other one is the certainty label which indicates the degree of confidence of the ‘present’ label
Sample id: 64
Class: Laysan Albatross
iiAddtt21ributes:hhaass__bbiillll__sshhaappee::::cduargvgeedr_(up_or_down) pprreesseenntt 00 cceerrttaaiinnttyy 33
i..d.... 92 has_upper_tail_color::white present 1 certainty 3
id 312 has_wing_pattern::multi-colored present 0 certainty 4
being true. The certainty goes from 1 to 4 where 1 is not visible, 2 is guessing, 3 is probably and 4
is definitely. These attribute annotations are not consistent throughout the dataset. To de-noise
it, we follow the same pre-processing procedure used in the Concept Bottleneck Models paper
[
Table 1a provides the statistics of the CUB dataset (before and after pre-processing) and the
pre-processed subsets we experiment on. These data subsets are created for the purpose of
scalability check, and are generated by taking samples of the first classes (class 1 to ) in the
original dataset, e.g. class 1 to 3 for CUB-3 subset. We apply the data pre-processing procedure
with diferent values due to the diferent numbers of classes in the subsets. All our multi-class
classification experiments use pre-processed CUB datasets/subsets, and, for simplicity, from
now on we do not explicitly mention ‘after pre-processing’ for each CUB dataset/subset.
TMC2007-500 [
Similar to the CUB dataset, we create subsets and apply the pre-processing on each subset. Table 1b shows the statistics of the datasets/subsets after pre-processing. We apply diferent MI thresholds during pre-processing on diferent subsets, resulting in diferent numbers of attributes used in each subset.
Sample id: 18643 Present attributes: 2 (above), 92 (climb), 107 (control), ..., 480 (we), 481 (weather)
Present labels: 1, 5, 7, 12
We evaluate our neural DNF-based models in terms of classification performance, scalability,
and interpretability. The classification task can be treated as a propositional ILP task with no
background knowledge. We include a two-layer MLP as a baseline for comparison to a pure
neural method, and a state-of-the-art symbolic learner FastLAS [
Dataset
Train Acc.
Comparison to baselines For the baseline methods, we train the two-layer MLP identically as our neural DNF-EO model but without the parameter adjustment. We specify FastLAS’s scoring function to be on body length so that FastLAS learns the most compact rules.
Accuracy comparison We measure the accuracy of an MLP and a trained plain neural DNF (without the constraint layer C2) by taking the arg max. A trained decision tree gives one class as the prediction after following the decision path and reaching the leaf node. All three models perform equally perfectly across all experiments, as shown in Table 4.
Jaccard index score and scalability comparison We interpret the tanh output of the plain neural DNF to be True if the value is greater than 0, giving the possibility of multiple class being True at the same time. Both the ASP rules learned by FastLAS and the ASP rules extracted from the plain neural DNF after the post-training pipeline can give more than 1 class in an answer set. Thus we choose the Jaccard index score as a measurement of mutual exclusivity for comparing these three approaches. We also attempt to interpret the 2-layer MLP to see if there is any mutual exclusivity at all: we expect some similarity between the MLP and our neural DNF-EO model as both models have tanh as the activation function at hidden layers and their weights are initialised the same. Thus our attempt at interpreting the MLP follows the same way as we interpret our neural DNF-EO model by thresholding at 0 after tanh. However, as discussed in Chapter 3, softmax only pushes the MLP to have diferent output logits and no need to push the logits towards a threshold such that they have symbolic meaning. Thus we expect the MLP to have poor Jaccard index scores. Table 5 shows the average test Jaccard score of all three (neuro-)symbolic approaches as well as our symbolic interpretation attempt on MLP. As expected, the MLP displays little to no symbolic mutual exclusivity for tasks with 10 classes and onwards. FastLAS learns perfect mutually-exclusive rules for all subsets but fails to learn from the full dataset of CUB-200, where it runs out of memory when computing the opt-suficient search space. Our trained plain neural DNF without any post-training processing learns perfectly in all settings regardless of the number of classes. This shows that our model provides mutual exclusion enforcement just like FastLAS, but also better scalability as it does not sufer from hypothesis search space growth. However, we cannot always extract well-performed rules from it, as there is a performance gap between the rules and the plain neural DNF without constraint layer C2 from CUB-25 and onwards.
Interpretability comparison We use the average and maximum rule length as a measurement of interpretability. To measure the rule length of a decision tree, we treat each decision path to a leaf node as the body of a rule, and the number of steps for a decision path can be seen as the number of body atoms. Figure 5 shows the comparison of average and maximum rule length across FastLAS’ rules, rules extracted from neural DNF-EO model (without layer C2), decision trees without pruning, and pruned decision trees with 80% accuracy on train sets. FastLAS learns the most compact rules but has scalability issues with a large hypothesis search space, as it fails to learn with 200 classes present. While the maximum rule length for rules extracted from the plain neural DNF is the highest in most experiments, the average length increases linearly compared to decision trees’ exponential growth. Rules from our neural DNFEO are more compact than the decision tree in half of the settings, especially in experiments with large numbers of classes.
16 14 12 h tg10 n e ll e u re 8 g a r e vA 6 4 2
FastLAS Rules extracted from neural DNF-EO without C2 Decision Tree (no prune) Decision Tree (pruned)
FastLAS Rules extracted from neural DNF-EO without C2 Decision Tree (no prune)
Decision Tree (pruned) 30 25 th20 g n e llr e u x15 a M 10 5 0
CUB-3 CUB-10 CUB-15 CUB-20 CUB-25 CUB-50 CUB-100 CUB-200 CUB-3 CUB-10 CUB-15 CUB-20 CUB-25 CUB-50 CUB-100 CUB-200 Figure 5: Average rule length and maximum rule length comparison across rules learned from FastLAS, extracted rules from trained neural DNF-EO model and decision tree (both pruned and without pruning). The pruned decision trees still maintain an accuracy of 80% in train sets. FastLAS runs out of memory for CUB-200. 5.2. TMC dataset/subsets We use macro F1 as the metric for the performance of vanilla neural DNF models and the extracted rules. We train the vanilla neural DNF model with binary cross-entropy across diferent TMC subsets/dataset, and the classification performance is shown in Table 6. The results suggest that our vanilla neural DNF model struggles to learn when more labels are present in the subsets/dataset. However, we speculate that the lack of general patterns/regularities in the TMC dataset/subsets could be the reason for its poor performance. We further discuss this issue in the following section where we compare our model to other baselines. On the other hand, we observe that the rules extracted through the post-training process show very similar performance as the corresponding trained vanilla neural DNF model. Compared to CUB experiments, the thresholding process does not need to maintain mutual exclusion across diferent outputs and thus does not cause a significant performance drop when ‘discretising’ the continuous weights.
Comparison to baselines We report the test macro F1 score of the 2-layer MLP trained with binary cross-entropy loss and the decision tree (without limiting depth) in Table 7. There is an increasing performance gap between our vanilla neural DNF model and the MLP as the number of labels increases. Figure 6 shows the change in macro F1 and magnitude in the
Train macro F1 0.745 0.615 0.391 0.177 0.094 vanilla neural DNF model’s symbolic layers over 300 epochs of training on TMC-22, and we see that the model achieves a similar macro F1 (around 0.45 compared to MLP’s 0.545) when the logical biases are not heavily enforced. But as the magnitude increases, especially over 0.5, the performance of the model drops and never recovers. This suggests that the logical biases in the semy-symbolic layers make it more dificult for the vanilla neural DNF model to learn. 0 50 100 200 250
300 150 Epoch
On the other hand, the decision tree learns significantly better than both our vanilla neural DNF model and MLP in all subsets/dataset, as shown in Table 7. This suggests that there are very few general patterns to learn in this dataset and that the decision is overfitting to the data. For the neural DNF-based model, if we allow an infinite number of conjunctions (i.e. infinite width of the conjunctive layer), we can encode every possible rule body for a label to cover all cases. But if we do so, the model will overfit: it will learn individual rules that are only ‘fired’ for a small number of data points. The same applies to MLP: an MLP with a wide hidden layer approximates a more complex function that might overfit than an MLP with a small hidden layer. By limiting the number of conjunctions/the width of the hidden layer, the model learns to generalise among large quantities of samples. If the data have very few general patterns, meaning they cannot be encoded using features with low dimensionality, the model will need a wider hidden layer to increase the feature space’s dimension and the chance of overfitting increases. The decision tree splits based on input space, and if not limiting the maximum depth of the tree, it can encode every case with nested if-else clauses and overfits the data. To see the diference in model size, we compare the number of trainable parameters of the MLP/vanilla neural DNF model and the number of non-leaf nodes (split nodes) in a decision without limiting depth in Table 8. The decision tree without depth limit has more split nodes (parameters) than its corresponding MLP/vanilla neural DNF in all the experiments. And when we limit the decision tree’s max depth to 15 in TMC-22, the tree has 2810 non-leaf nodes and the test macro F1 drops to 0.565 which is close to MLP’s performance, confirming that the decision tree without a limit on depth is overfitting. This also shows that the dataset is too complex to learn without increasing the parameters.
The last baseline approach, FastLAS, fails in every TMC subsets/dataset, suefring from the same memory issue as the CUB-200 experiment. Without a mutual exclusivity constraint, the hypothesis search space is exponentially larger since every label can be fired at the same time, causing the memory issue for FastLAS.
MLP/Vanilla Neural DNF
Architecture (Input, Hidden, Output) (59, 9, 3) (60, 15, 5) (34, 30, 10) (80, 34, 15) (103, 66, 22)
Num. Parameters (Excluding bias)
Decision Tree Max. depth
No. Non-leaf Nodes
We analyse our neural DNF-based models further in the three areas of classification performance ,
interpretability, and scalability to compare against the baselines of MLPs, decision trees and
FastLAS [
Classification performance Our neural DNF-based models learns as well as their MLP counterparts when general patterns exist in the dataset2. In CUB experiments, our neural DNF-EO model always achieves test accuracy of 1 as the MLP (Table 4), but also gives test Jaccard index score of 1 and provides mutual exclusion which an MLP trained with cross-entropy cannot provide. Although the TMC experiments show a diference in performance, we speculate that the dataset lacks general patterns/regularities as discussed in Chapter 5.2, and the loss 2To further support this statement, we also construct synthetic datasets for both multi-class and multi-label classification tasks. The data points are generated based on a set of rules such that the data points follow general patterns/show regularities. We train neural DNF-based models on them, and in both scenarios, our models learn almost perfectly, and we can extract symbolic rules. See more details in Appendix B.5 plot (Figure 6) suggests that the logically constrained bias in our vanilla neural DNF model increases the dificulty of training. In regards to decision trees, our models have a diferent learning mechanism, and TMC dataset/subsets do not fully reflect the end-to-end learning capability of our models: the data instances in the dataset/subsets cannot be easily encoded in features with low dimensionality, making it hard for our models (and also MLPs) to generalise without overfitting, while decision trees without a depth limit overfit. But when data points in the dataset/subset follow some general patterns, such as the CUB dataset/subsets, our models perform equally well as decision trees (Table 4).
Interpretability The logically constrained bias in the semi-symbolic layers makes it possible to extract symbolic rules from our models through post-training processes, contrasting to the lack of interpretability in MLPs. However, as shown in the CUB experiments (Table 2), our current process does not always result in a successful rule extraction for multi-class classification. The thresholding process is the current bottleneck of the pipeline, as it cannot maintain mutual exclusivity across a large number of classes. Our rules are also not guaranteed to be in the most compact form compared to FastLAS (Figure 5). FastLAS with a scoring function on body length will guarantee the most compact hypothesis, while we do not have a strong signal for the neural DNF-based models to learn the most compact rules, which may further restrict gradient-based learning. Although we do include an L1 regulariser during training and prune the model at post-training processes, it does not necessarily reject hypotheses with longer rules. On the other hand, decision trees show less compactness than rules extracted from our neural DNF-based models, and also are prone to overfitting.
Scalability Our neural DNF-based models scale better than FastLAS. The conjunctive and disjunctive layers of the neural DNF-based model create a hypothesis space that allows rules with any combination of body atoms, and through gradient descent, the model moves towards an approximated hypothesis. FastLAS first needs to compute a hypothesis sub-space that guarantees to contain the optimal solution and may run out of memory during this computation process, which is what happens in the CUB-200 experiment and all TMC experiments. Comparing our model to MLP and decision tree, all three are trainable under the same setting regardless of the input/output size. However, training the neural DNF-based model is more dificult than an MLP (Figure 6), and the thresholding process in the post-training pipeline does not yield perfect rules from the trained model anymore after 20 classes (Table 2).
To summarise, (i) our neural DNF-based model performs as well as an MLP and a decision tree when general patterns exist in the dataset; (ii) it has better interpretability than an MLP by providing ASP rules, and the rules are in average more compact than a decision tree’s decision paths; (iii) it also scales better than FastLAS when the hypothesis search space is large.
Our work relates to the research of diferentiable Inductive Logic Programming (diferentiable ILP). Diferentiable ILP methods approximate discrete logical connectives and input with continuous ones and enable continuous logic reasoning, so that the model can be trained via gradient descent. In this category, we can further split the methods into two branches: rule-templatebased diferentiable ILP and free-form diferentiable ILP.
Rule-template-based diferentiable ILP methods rely on a set of rule templates (or meta-rules).
The rule templates specify the form of rules to learn, and they help with scaling by restricting
the hypothesis search space. They also guide the symbolic interpretation of the model. -ILP
[
Free-form diferentiable ILP methods do not restrict the form of rules to learn, avoiding the
need for human-engineered templates. Neural Logic Machine (NLM) [
Another design pattern in the field of neuro-symbolic rule learning is to use separate neural
and logical models, which generally does not allow full back-propagation. Two recent works in
this category are [17] and NSIL [
Building upon pix2rule [
In future work, we aim towards end-to-end neuro-symbolic rule learning with our neural
DNF-based models from unstructured input, such as the task of learning from images in the
CUB dataset [
URL: https://github.com/metagol/metagol. [19] Z. Yang, A. Ishay, J. Lee, Neurasp: Embracing neural networks into answer set programming, in: C. Bessiere (Ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, International Joint Conferences on Artificial Intelligence Organization, 2020, pp. 1755–1762. URL: https://doi.org/10.24963/ijcai.2020/243. doi:10.24963/ijcai.2020/243, main track. [20] E. Leurent, An environment for autonomous driving decision-making, https://github.com/ eleurent/highway-env, 2018. [21] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, 2014. URL: https://arxiv.
org/abs/1412.6980. doi:10.48550/ARXIV.1412.6980.
A.1. Procedure on CUB Dataset/subsets
Algorithm 1 shows the pseudo-code of the data pre-processing procedure used by us and also in
the Concept Bottleneck Models paper [
A.2. Procedure on TMC Dataset/subsets We calculate the mutual information (MI) of each attribute with respect to the output label combinations, and filter out attributes with MI value less than a threshold . The values we choose for TMC-3, -5, -10, -15, and -22 are 0.01, 0.02, 0.04, 0.04 and 0.05 respectively. Below we provide the mathematical formulae to calculate the mutual information.
Mutual information of two discrete random variables and can be defined as: And the conditional entropy H(| ) is calculated as: as: where H() and H( ) are entropies, and H(| ) is conditional entropy. H() is calculated I(; ) ≡
H() − H(| ) H() = − ∑︁ () log () ∈ H(| ) = − ∑︁ () ∑︁ (|) log (|) ∈
∈
For each attribute , the mutual information of it with respect to the output label combination is calculated as the following, using Equation 3, 4 and 5: ⎛ ⎞ I(; ) = − ∑︁ () log () − ⎝−
∑︁ () ∑︁ (|) log (|)⎠ ∈ ∈ ∈
Since each attribute only has two outcomes: present (1) or not present (0), Equation 6 is equivalent to: (3) (4) (5) (6) ︁) () ( = 1|)log ( = 1|) + ( = 0|)log ( = 0|) ︁) )︃ (7) /* Find where most and least frequent, set most frequent to 1 9 __ ← indices where
___ == ___; 10 ___[__] ← 1; /* Count number of times each attribute is mostly present for a class 11 __ ← sum(___, = 0); /* Select the attributes that are present most of the time on a class level, in at least classes 12 ← indices where __ ≥
/* Get the attribute median 13 __ ← ___[:, ]; */ */ */ */ */ */ */ */
B.1. Code
available at https://github.com/kittykg/neural-dnf-cub, and the code for our TMC experiments is available at https://github.com/kittykg/neural-dnf-tmc. Both repositories contain our implementation of the semi-symbolic layer and the neural DNF-based models.
B.2. Machine Specification The machine specifications for our experiments are shown below in Table 9.
Model Neural DNF-based models
MLP Decision Tree
B.3. CUB Experiments B.3.1. Hyperparameters and Model Architecture All CUB experiments are run under a random seed of 73. We use Adam [21] as the optimiser with a learning rate of 0.001 and weight decay of 0.00004. The schedule steps in every 100 epochs. The model architectures, training batch size and training epoch of our neural DNF-EO models are shown in Table 10. The untrainable constraint layer C2 is of _ × _, where relevant connections are set to -6 and rest as 0. We set the pruning threshold = 0.005 in the post-training pipeline.
Dataset CUB-3 CUB-10 CUB-15 CUB-20 CUB-25 CUB-50 CUB-100 CUB-200
In x Conj x Out 34 * 9 * 3 41 * 30 * 10 40 * 45 * 15 48 * 60 * 20 50 * 75 * 25 61 * 150 * 50 82 * 300 * 100 112 * 600 * 200
Batch size 4 16 32 32 32 32 32 32
The MLP model in each experiment has the same architectural design and training settings as the neural DNF-EO model, but each layer is a normal linear feed-forward layer instead of the semi-symbolic layer.
B.3.2. Vanilla Neural DNF Models in Multi-class Classification We have also carried out experiments with the vanilla neural DNF model on CUB-3, CUB-10 and CUB-200, and the results are shown in Table 11. With binary cross entropy as a loss function on only CUB-3, the vanilla neural DNF model is able to learn and give a set of rules which results in an average of 1 Jaccard index score on the test set. This is expected since there are only 3 classes to predict (no need for a shortcut by predicting everything as false) and each class is pushed to be binary (true or false). Yet binary cross entropy as a loss function does not work when more classes are present, since it is easier to ‘cheat’ by predicting every class to be false. Cross-entropy loss emphasises the diference between output but does not push the output value to be mutually exclusive, as discussed in Chapter 3. Without any enforcement on exactly one class being true, the vanilla neural DNF model with cross-entropy loss is able to predict the correct class under picking the max output (average of 1 accuracy on the test set), but cannot give an output where only one class is close to 1 and everything else is -1 (less than 1 average Jaccard score). These results show the importance of the constraint layer C2 in our neural DNF-EO model.
Model + Loss Fn.
B.4. TMC Experiments B.4.1. Hyperparameters and Model Architecture All TMC experiments are run under a random seed of 73. We use Adam [21] as the optimiser with a learning rate of 0.001 and weight decay of 0.00004. The schedule steps in every 100 epochs. Table 12 shows the model architectures, training batch size and training epoch of our vanilla neural DNF models. We set the pruning threshold = 0.005 in the post-training pipeline.
The MLP in each experiment has the same architectural design as the vanilla neural DNF and training settings, but each layer is a normal feed-forward linear layer instead of the semisymbolic layer.
B.5. Synthetic Datasets Experiments We generate synthetic multi-class and multi-label datasets to show that the neural DNF-based models are capable of learning rules from datasets with underlying patterns. For each synthetic dataset, we first randomly generated a set of rules that guarantee mutual exclusivity if under a multi-class setting. Then, 10,000 samples are generated based on the rule set without noise. 2,000 of the samples are randomly selected to form the test set, 1,600 for the validation set and 6,400 for the train set. Following the same training and post-training process but without the tuning stage, we evaluate a neural DNF-based model on each of the synthetic datasets. The result is shown in Table 13 below. We see that our neural DNF-based models are learning well after training in all experiments, while we still observe the problem of multi-class classification performance drop after thresholding.
These synthetic experiments can be found under the synthetic_test/notebooks/ folder in the two GitHub repositories we mentioned in Appendix B.1.