The computer-science field of Knowledge Representation and Reasoning (KRR) aims to understand, reason, and interpret knowledge as eficiently as human beings do. Because many logical formalisms and reasoning methods in the area have shown the capability of higher-order learning, such as abstract concept learning, integrating artificial neural networks (ANNs) with KRR methods for learning complex and practical tasks has received much attention. For example, Neural Tensor Networks (NTNs) are neural-network models capable of transforming symbolic representations into vector spaces where reasoning can be performed through matrix computation; when used in Logic Tensor Networks (LTNs), they are able to embed first-order logic symbols such as constants, facts, and rules into real-valued tensors. The integration of KRR and ANN suggests a potential avenue for bringing biological inspiration from neuroscience into KRR. However, higher-order learning is not exclusive to human brains. Insects, such as fruit flies and honey bees, can solve simple associative learning tasks and learn abstract concepts such as “sameness” and “diference,” which is viewed as a higher-order cognitive function and typically thought to depend on top-down neocortical processing. Empirical research with fruit flies strongly supports that a randomized representational architecture is used in olfactory processing in insect brains. Based on these results, we propose a Randomly Weighted Feature Network (RWFN) that incorporates randomly drawn, untrained weights in a encoder that uses an adapted linear model as a decoder. The randomized projections between input neurons and higher-order processing centers in the input brain is mimicked in RWFN by a single-hidden-layer neural network that specially structures latent representations in the hidden layer using random Fourier features that better represent complex relationships between inputs using kernel approximation. Because of this special representation, RWFNs can efectively learn the degree of relationship among inputs by training only a linear decoder model. We compare the performance of RWFNs to LTNs for Semantic Image Interpretation (SII) tasks that have been used as a representative example of how LTNs utilize reasoning over first-order logic to surpass the performance of solely data-driven methods. We demonstrate that compared to LTNs, RWFNs can achieve better or similar performance for both object classification and detection of the part-of relations between objects in SII tasks while using much far fewer learnable parameters (1:62 ratio) and a faster learning process (1:2 ratio of running speed). Furthermore, we show that because the randomized weights do not depend on the data, several decoders can share a single randomized encoder, giving RWFNs a unique economy of spatial scale for simultaneous classification tasks.
The human brain has an extraordinary ability to memorize and learn new things to solve a
variety of problems with dificulty ranging from trivial to complex. To understand the cognitive
architecture of the brain, research on producing a wiring diagram of the connections among
all neurons, called Connectomics [
Several recent studies across statistical relational learning (SRL), neural-symbolic computing,
knowledge completion, and approximate inference [
Although KRR aims to lift the reasoning ability of computers to that of humans, such
higherorder learning and reasoning capabilities are not unique to humans. Insect neuroscience has
shown that insects show sophisticated and complex behaviors even though they possess
miniature central nervous systems compared to the human brain [
NeSy’20/21 @ IJCLR: 15th International Workshop on Neural-Symbolic Learning and Reasoning, October 25–27, 2021 nEvelop-O LGOBE rOcid https://www.linkedin.com/in/jyhong0304/ (J. Hong); https://isearch.asu.edu/profile/1995237 (T. P. Pavlic) 0000-0003-4429-3311 (J. Hong); 0000-0002-7073-6932 (T. P. Pavlic)
© 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Workshop Proceedings htp:/ceur-ws.org CEUR Workshop Proceedings (CEUR-WS.org)
ISN1613-073
In this paper, we propose Randomly Weighted Feature Networks (RWFNs), an insect-braininspired single-hidden-layer neural network for relational embedding that incorporates randomly drawn, untrained weights in its encoder with a trained linear model as a decoder. Our approach is mainly motivated by neural circuits in the insect brain centered around the Mushroom Body (MB). The MB, analogous to the neocortex in humans, is a vital region of the insect brain supporting concept learning because it is responsible for stimulus identification, categorization, and element learning [21, 22, 23]. We can model the MB as a neural network model with three layers: Input Neurons (INs) – Kenyon Cells (KCs) – mushroom body Extrinsic Neurons (ENs). One of the remarkable properties of MB is that the connections between INs and KCs are relatively random and sparse [24]. To mimic this characteristic, we used a random weight matrix to transform the input between the input and hidden layers to generate the latent representation of the relationship between real-valued input entities. By doing so, the learning process involves only the training of the weights between the hidden and output layers, which is simple and fast. In contrast, a conventional LTN would incorporate an NTN specially trained to capture logical relationships present in data, which requires more learning parameters and a more complex learning process.
Our method is also influenced by random Fourier features [ 25], a kernel approximation method that overcomes the issues of conventional kernel machines or kernel methods [26]. Kernel methods are one of the most powerful and theoretically grounded approaches for nonlinear statistical learning problems, including classification, regression, clustering, and others [27, 28, 29]. However, the main issue of the kernel method is the lack of scalability for large datasets and a slow training process [25, 30]. Random Fourier features can address these issues by approximating the kernel function by using an explicit feature mapping that projects the input data into a randomized feature space and by applying faster linear models to learn. Interestingly, the random Fourier features model can also be viewed as a class of a single-hidden-layer neural network model with a fixed weight between the input and hidden layers. Thus, we leveraged this to substitute the tensor operations in conventional NTNs that model the linear interactions between entities and utilized the projection from input into another space as another feature representation in the hidden layer of our model to learn relationships.
Thus, our proposed model is an insect-inspired single-hidden-layer network with latent representation derived by the integration of the input transformation between INs and KCs and random Fourier features, and it only requires training of a linear decoder. By applying the model to solve the Semantic Image Interpretation (SII) tasks, we show that a trained linear decoder in RWFNs can efectively capture the likelihood of part-of relationships at a level of performance exceeding that of traditional LTNs, even with far fewer parameters and a faster learning process. To the best of our knowledge, this is the first research to integrate both insect neuroscience and neuro-symbolic approaches for reasoning under uncertainty and for learning in the presence of data and rich knowledge. Furthermore, because the encoder weights in our model do not depend upon the data, the single encoder can be shared among several decoders, each trained for a diferent classifier, giving RWFNs an economy of spatial scale in our model applications where several classifiers need to be used simultaneously.
Insect Neuroscience The MB in the insect brain receives processed olfactory, visual, and mechanosensory stimuli [31] and is viewed as the critical region responsible for multimodal associative learning [21]. In the fruit fly, thousands of Kenyon Cells (KCs) in the MB each receive a set of random ∼7 inputs from INs [24, 32], and this is similarly true for honey bees [33]. A simplified neural circuit modeling the MB is a neural network with three layers consisting of: i) INs that provide olfactory, visual, and mechanosensory inputs, ii) KCs generating the sparse-encoding of sensory stimuli, and iii) ENs for activating several diferent behavioral responses [34]. In particular, INs receive various inputs from Antennal Lobe (AL) glomeruli, Medulla, and Lobula optic neuropils [34]. For simplicity, we focus on the olfactory pathway between glomeruli in the AL and KCs in the MB [34, 33]. The insect olfactory neural circuit has a divergence–convergence structure where ∼800 AL glomeruli form a coded feature vector that expands into a spare representation across ∼170,000 KCs, and these are decoded by ∼400 ENs that actuate motor pathways based on this information-processing pipeline [33]. This general divergence–convergence structure applies equally as well to honey bees and fruit flies [ 35]; therefore, for modeling the architecture of our methods for brevity, we interchangeably leverage the neural circuits of olfactory nervous systems between these insects.
In this paper, the input transformation of odorant representation in the AL to the higherorder representation across the KCs in the MB is shown in the projection between the input and hidden layers of our model, and this representation plays the critical role in learning the relationships presented on the input.
Random Fourier Features Kernel machines, e.g., Support Vector Machines (SVMs) [36, 37], have received significant attention due to their capability for function approximation and excellent performance of detecting decision boundaries with enough training data. These methods use transformations, as with a lifting function , that help to better discriminate among diferent inputs. Given dataset vector inputs x, y ∈ ℝ , the kernel function ( x, y) = ⟨( x), ( y)⟩ represents the similarity (i.e., inner product) between x and y in the transformed space. However, because of the potential complexity of the transformation , learning the kernel function may require significant computational and storage costs.
Random Fourier features [25], instead, provide a data transformation that permits using a far less expensive approximation of the kernel function. For each vector input x ∈ ℝ , the technique applies a randomized feature function z ∶ ℝ → ℝ (generally, ≫ with sample size ≫ ) that maps x to evaluations of random Fourier basis from the Fourier transform of kernel . In this transformed space, kernel evaluations can be approximated by linear operations, as in: ( x, y) = ⟨( x), ( y)⟩ ≈z(x)⊤z(y) (1) Thus, by transforming the input with z, fast linear learning methods can be leveraged to approximate the evaluations of nonlinear kernel machines.
In this paper, we use random Fourier features as latent representations that reduce the complexity of learning relations among real-valued entities. As described in Section 1, adaptable NTNs within LTNs have been used to encode relationships among real-valued entities. We replace the adaptable NTNs with random Fourier features that have high expressiveness with low decoding overhead. Details and intuitions will be given in Section 3.
Logic Tensor Networks (LTNs) The RWFNs we propose are meant to improve upon LTNs
for statistical relational learning tasks. LTNs integrate learning based on NTNs [
Although a first-order-logic (FOL) language ℒ and its signature are defined by consisting of three disjoint sets – i) (constants), ii) ℱ (functions) and iii) (predicate) – we ignore function symbols ℱ because they are not used in SII tasks that we focus on here. For any predicate symbol , () can be described as its arity, and logical formulas in ℒ enable the description of relational knowledge. The objects being reasoned over with FOL are mapped to an interpretation domain ⊆ ℝ so that every object is associated with an -dimensional vector of real numbers. Intuitively, this -tuple indicates numerical features of an object. Thus, predicates are interpreted as fuzzy relations on real vectors. With this numerical background, we can now define the numerical grounding of FOL with the following semantics; this grounding is necessary for NTNs to reason over logical statements.
Let ∈ ℕ . An -grounding, or simply grounding, for a FOL ℒ is a function defined on the signature of ℒ satisfying the following conditions:
() ∈ ℝ for every constant symbol ∈ ( ) ∈ ℝ ⋅( )
→ [
A partial grounding ̂ can be defined on a subset of the signature of ℒ. A grounding is said to be a completion of ̂ if is a grounding for ℒ and coincides with ̂ on the symbols where ̂ is defined. Let GT be a grounded theory which is a pair ⟨ , ⟩ ̂ with a set of closed formulas and a partial grounding .̂ A grounding satisfies a GT ⟨ , ⟩ ̂ if completes ̂ and () = 1 for all ∈ . A GT ⟨ , ⟩ ̂ is satisfiable if there exists a grounding that satisfies ⟨ , ⟩ ̂ . In other words, deciding the satisfiability of ⟨ , ⟩ ̂ amounts to searching for a grounding such that all the formulas of are mapped to 1. If a GT is not satisfiable, the best possible satisfaction that we can reach with a grounding is of our interest.
Grounding ∗ captures the implicit correlation between quantitative features of objects and
their categorical/relational properties. The grounding of an -ary predicate , namely ( ) , is
defined as a generalization of the NTN [
, ∈ ℝ and ∈ ℝ . Because our RWFN model can be used to ground ( ) , we can directly compare the performance of RWFNs for the SII tasks
In this section, we introduce the details of Randomly Weighted Feature Networks (RWFNs). The underlying intuition behind the development of this model can be found in Appendix A.
Let the input vector v be [v1⊤, … , v⊤ ]⊤, the -ary vector where is arity and is the input dimension. We first define the two kinds of latent representations: i) the input transformation between AL glomeruli and KCs inspired by the insect brain, and ii) the transformed input using a randomized feature mapping z(⋅)in random Fourier features.
For the bio-inspired representation, we select ∈ [1, ) indices of the input at random without replacement for each hidden node1. In other words, the output ̄ of each hidden node ∈ {1, ..., } is a weighted combination of all inputs where only < inputs have , = 1 and all other inputs have , = 0. So the inputs are efectively gated by the weights on each hidden node and the weight matrix W ∈ ℝ× in this computation is random, binary, and sparse.
1 < to prevent hidden-unit outputs becoming trivially 0 by Eq. (3). In our setting, = 7.
Once the th hidden node has produced weighted sum ̄ , the post-processing step to produce intermediate output is performed in the hidden layer. Mimicking Eq. (6) with = 1 , the th intermediate output ̂ is: ̂ = ̄ − , =
1 =1 ∑ ̄ kinds of random Fourier features. from ℝ
to [
(0, 2 ) , which is Gaussian kernel approximation.
Consequently, the output vector h
2 can be considered as another latent representation of
relationship among input. Rahimi et al. [25], Sutherland and Schneider [
2 h be defined as ℎ(1)
vector as h
1 = [ℎ(11), … , ℎ(1)]⊤. where is the number of hidden units. Therefore, the sparse output of the -th KC node can = ( ̂ )where is the ReLU function [38] that allows the model to produce sparse hidden output, which is more biologically plausible. By doing so, we define the output
On the other hand, to generate random Fourier features, we used a randomized feature
function z(⋅)in [
In the insect brain, extrinsic neurons from the MB are processed by several small, downstream neuropils that ultimately lead to decision-making outcomes, such as muscle actuation. If we view these small neuropils as decoding the complex representations in the MB, then diferent decoders responsible for diferent decisions all use information sourced from the same randomized representations in the MB. The MB can be viewed as a generalized encoder that is not tailored for a particular task; consequently, it provides a shared resource to reduce the complexity of these downstream neuropils.
Because the weights of the randomized encoder of an RWFN are independent of the training data, they can also serve as a shared resource for multiple relatively simple (i.e., linear) downstream decoders trained for diferent classifiers. We refer to this property as weight sharing.
Fig. 1b shows a visualization of the structure of our model applied with weight sharing to the learning of diferent classifiers. The large solid box surrounds a single encoder that serves as a common feature extractor for all classifiers.
The entities in the original definition of RWFNs in Fig. 1a become the placeholder to be injected by the input for each classifier. Instead of generating the randomized encoder for each classifier, each classifier uses the same encoder, and training only requires learning the weights of that classifier’s highly simple linear decoder. This approach increases reusability and cost eficiency in a way beyond what is possible with LTNs, which must train all encoder and decoder networks separately for each classifier.
To evaluate the performance of our proposed RWFNs over LTNs, we employ both for SII tasks,
which extract structured semantic descriptions from images. Very few SRL applications have
been applied to SII tasks because of the high complexity involved with image learning. Donadello
et al. [
Here, we provide details of our experimental comparison of RWFNs and LTNs. We utilize the
formalization of SII in first-order logic from Donadello et al. [
Defining the Grounded Theories for RWFNs and LTNs A set of bounding boxes of images correctly labelled with the classes that they belong to and pairs of bounding boxes that properly labelled with the part-of relation were provided. These datasets can be considered as a training set, and a grounded theory LTN ≜ ⟨ , ̂ ⟩ can be constructed. In particular, contains: (i) the set of closed literals () and p a r t O f (, ′)for every bounding box labelled with and for every pair of bounding boxes ⟨, ′⟩ connected by the p a r t O f relation, and (ii) the set of the mereological constraints for the part-of relation, including asymmetric constraints, lists of several parts of an object, or restrictions that whole objects cannot be part of other objects and every part object cannot be divided further into parts. Furthermore, the partial grounding ̂ is defined on all bounding boxes of all the images in the training set where both ( , ) and the bounding box coordinates are computed by the Fast R-CNN object detector. ̂ is not defined for the predicate symbols in and is to be learned. (a) RWFNs achieve similar performance for object type classification compared to LTNs, achieving an Area Under the Curve (AUC) of 0.772 (compared to 0.770). (b) RWTNs outperform LTNs on the detection of part-of relations, achieving AUC of 0.647 (compared to 0.613).
A grounded theory RWTN ≜ ⟨ , ̂ ⟩ where a partial grounding ̂ can be described for predicates using Eq. (5). Thus, we can easily compare the performance between ̂ (Eq. (5)) and ̂ (Eq. (2)).
Our experiments mainly focus on the comparison of the performance between our model
and LTN, but figures also include results with Fast-RCNN [
Results for indoor objects are shown in Fig. 2 where AUC is the area under the precision–recall curve. The results show that, for the part-of relation and object types classification, RWFNs achieve better performance than LTNs. However, there is some variance in the results because of the stochastic nature of the experiments. Consequently, we carried out five such experiments for each task, for which the sample averages and 95% confidence intervals are shown in Table 1. These results confirm that our model can achieve similar performance as LTNs for object-task classification and superior performance for detection of part-of relations.
In Table 1, we only included AUC numbers for RWFNs with weight sharing (third column) for object-type classification because part-of relations only require a single classifier. The performance of RWFNs with weight sharing for the object-type classification task (which requires 11 classifiers for indoor objects, 23 for vehicles, and 26 for animals) shows only a marginal gap in performance compared to other models, which demonstrates the efectiveness and eficiency of the approach of using a single shared encoder in RWFNs with weight sharing.
As summarized in Appendix C, we also conducted ablation studies to assess the degree to which the AL–MB input transformation and the random Fourier features each contribute to the performance of the model. We have also included, in Appendix D, a detailed comparison of performance among LTNs, RWFNs, and RWFNs with weight sharing. Specifically, we compare LTNs and RWFNs in terms of numbers of learnable parameters and running times; we also compare RWFNs with and without weight sharing in terms of space complexity.
In this paper, we introduced Randomly Weighted Feature Networks, which incorporate the insect-brain-inspired neuronal feature representation and unique random features derived by random Fourier features. The RWFN encoder acts as a generalized feature extractor with greater relational expressiveness and a learning model with relatively simpler structure. We demonstrated how insights from the insect nervous system can be applied to the fields of neuralsymbolic computing and knowledge representation and reasoning for relational learning.
Our work can be advanced in several ways. For one, RWFNs can be applied to other variants
of SII problems proposed by Donadello and Serafini [
This work was supported in part by NSF SES-1735579. Sciences 109 (2012) 7481–7486. [20] A. Avarguès-Weber, M. Giurfa, Conceptual learning by miniature brains, Proceedings of the Royal Society B: Biological Sciences 280 (2013) 20131907. [21] R. Menzel, Searching for the memory trace in a mini-brain, the honeybee, Learning & memory 8 (2001) 53–62. [22] C. G. Galizia, Olfactory coding in the insect brain: data and conjectures, European Journal of Neuroscience 39 (2014) 1784–1795. [23] M. Bazhenov, R. Huerta, B. H. Smith, A computational framework for understanding decision making through integration of basic learning rules, Journal of Neuroscience 33 (2013) 5686–5697. [24] S. J. Caron, V. Ruta, L. Abbott, R. Axel, Random convergence of olfactory inputs in the drosophila mushroom body, Nature 497 (2013) 113–117. [25] A. Rahimi, B. Recht, et al., Random features for large-scale kernel machines., in: NIPS, volume 3, Citeseer, 2007, p. 5. [26] A. J. Smola, B. Schölkopf, Learning with kernels, volume 4, Citeseer, 1998. [27] J. Zhu, T. Hastie, Kernel logistic regression and the import vector machine, Journal of
Computational and Graphical Statistics 14 (2005) 185–205. [28] H. Drucker, C. J. Burges, L. Kaufman, A. Smola, V. Vapnik, et al., Support vector regression machines, Advances in neural information processing systems 9 (1997) 155–161. [29] I. S. Dhillon, Y. Guan, B. Kulis, Kernel k-means: spectral clustering and normalized cuts, in: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, 2004, pp. 551–556. [30] F. Liu, X. Huang, Y. Chen, J. A. Suykens, Random features for kernel approximation: A survey in algorithms, theory, and beyond, arXiv preprint arXiv:2004.11154 (2020). [31] P. Mobbs, The brain of the honeybee apis mellifera. i. the connections and spatial organization of the mushroom bodies, Philosophical Transactions of the Royal Society of London.
B, Biological Sciences 298 (1982) 309–354. [32] K. Inada, Y. Tsuchimoto, H. Kazama, Origins of cell-type-specific olfactory processing in the drosophila mushroom body circuit, Neuron 95 (2017) 357–367. [33] F. Peng, L. Chittka, A simple computational model of the bee mushroom body can explain seemingly complex forms of olfactory learning and memory, Current Biology 27 (2017) 224–230. [34] A. J. Cope, E. Vasilaki, D. Minors, C. Sabo, J. A. Marshall, A. B. Barron, Abstract concept learning in a simple neural network inspired by the insect brain, PLoS computational biology 14 (2018) e1006435. [35] K. Endo, Y. Tsuchimoto, H. Kazama, Synthesis of conserved odor object representations in a random, divergent-convergent network, Neuron 108 (2020) 367–381. [36] B. E. Boser, I. M. Guyon, V. N. Vapnik, A training algorithm for optimal margin classifiers, in: Proceedings of the fith annual workshop on Computational learning theory, 1992, pp. 144–152. [37] C. Cortes, V. Vapnik, Support-vector networks, Machine learning 20 (1995) 273–297. [38] X. Glorot, A. Bordes, Y. Bengio, Deep sparse rectifier neural networks, in: Proceedings of the fourteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, 2011, pp. 315–323.
For the bio-inspired representation in our model, we concentrated on implementing: (i) how to
build random connections between the AL glomeruli (input layer) and the KCs (hidden layer),
and (ii) how to guarantee hidden-layer sparsity to best diferentiate one odor stimulus from
another. We used a sparse, binary, and random matrix to define an arbitrary set of inputs
for each KC in the model with inspiration from Caron et al. [24], Peng and Chittka [33], and
Dasgupta et al. [
1
∑ ) where
indicates the weighted sum of input from 7 random indices of the input vector, is the ReLU, is the strength of global inhibition, and is the total number of KCs. The parameters = 1.0 and
= 2000 were chosen so as to match the values best calibrated to
real KC responses [
For another hidden representation using random Fourier features in our model, based on Eq. (1), we can define a decision function ( x)given a dataset including data samples x, y ∈ ℝ and a randomized feature mapping z ∶ ℝ → ℝ as follows: (6) (7) =1 =1 =1 ( x) = ∑ ( x , x) = ∑ ⟨( x ), ( x)⟩
≈ ∑ z(x )⊤z(x) = ⊤z(x) This indicates that if z(⋅)can approximate (⋅) well, we can simply map our data using z(⋅)and then use a linear model to learn because both and z(⋅)in the above equation are -vectors. Therefore, the task that we will describe in the next section is how to find a random projection function z(⋅)that can approximate the corresponding nonlinear kernel machine appropriately.
The reason why we leverage the random Fourier feature is conciseness and eficiency of computing linear interactions among input, which can be replaced with the bilinear model in Eq. (2). In Eq. (2), the bilinear tensor was used to compute the relation, which seems intuitive because each slice of the tensor serves as being responsible for one type of relation. However, this computation requires high computational cost with large number of parameters. In contrast, the random Fourier features in Eq (7) can do the similar task with a much faster learning process and fewer number of parameters.
Considering how Eq. (6) and Eq. (7) can be used in our model, the hidden representation can be expressed with the concatenation of
and z(⋅ ).
Hardware specification of the server The hardware specification of the server that we used to experiment is as follows: • CPU: Intel® CoreTM i7-6950X CPU @ 3.00GHz (up to 3.50 GHz) • RAM: 128 GB (DDR4 2400MHz) • GPU: NVIDIA GeForce Titan Xp GP102 (Pascal architecture, 3840 CUDA Cores @ 1.6
GHz, 384 bit bus width, 12 GB GDDR G5X memory) Source codes All source codes, trained models, and figures in this paper are available at https://github.com/jyhong0304/SII.
Datasets The PASCAL-Part-dataset [
Hyperparameter Setting To compare the performance between RWFNs and LTNs, we
trained two models separately. For LTN, we configure the experimental environment
following Donadello et al. [
Hyperparameter Searching for RWFNs To find out the best number of hidden nodes
, we used the Optuna framework [
Table 2 shows the results of ablation studies. In order to show how much two hidden representations – the input transformation between AL–MB and random Fourier features – contribute to the performance of our model, we built two separate RWFN models: one using the AL–MB input transformation only and another using random Fourier features only. Then, we performed ifve experiments and averaged AUCs of each model for object classification and part-of detection. The number of hyperparameter for each model was set to the same as the number of hyperparameter for the original RWFN, which is 200.
For the object-type classification and part-of detection tasks using Indoor label, the random Fourier features outperform the AL–MB input transformation. On the other hand, the AL–MB input transformation for part-of detection tasks using Vehicle and Animal labels show better performance than the random Fourier features. Therefore, these ablation studies show that the model architecture of RWFNs in Eq. (5) can fully utilize both hidden representations and contribute to their good performance shown in Table. 1 by compensating for each other.
Relative Complexity of RWFNs and LTNs To better appreciate the relative performance of RWFNs and LTNs, we can compare the number of parameters for grounding a unary predicate for each model. The dimension of the input in the dataset for both RWFNs and LTNs is = 64 . As shown in Eq. (2), the parameters to learn in LTNs are { ∈ ℝ , [1∶] ∈ ℝ×× , ∈ ℝ× , ∈ ℝ }, where = 6 following the configuration of the LTNs. Thus, the number of parameters in LTNs is ( 2 + + 2) ⋅ = (64 2 + 64 + 2) ⋅ 6 = 24972. On the other hand, in Eq. (3) and Eq. (4), the number of parameters in RWFNs are {W ∈ ℝ× , R ∈ ℝ× , b ∈ ℝ , ∈ ℝ2 }, where = 200 following the configuration of the RWFNs. Therefore, the number of parameters in RWFNs is (2 + 3) ⋅ = (2 ⋅ 64 + 3) ⋅ 200 = 26200 . Although our method requires more space complexity compared to LTNs (26200 > 24972), the parameters {W, R, b} in RWFNs are randomly drawn and fixed weights. Thus, it is also necessary to compare the number of learnable parameters across the two models.
All of the above parameters of LTNs must be adaptable, whereas the parameters to learn in RWFNs for object type classification are only ∈ ℝ2 . Thus, the number of learnable parameters is 400, which is much smaller than that of LTNs. This means that the ratio of the two numbers of parameters to learn is about 400 ∶ 24972 ≈ 1 ∶ 62. Consequently, non-adaptable parameters in RWFNs can have significant power to represent the latent relationship among objects so that the model can eficiently extract relational knowledge even though using fewer adaptable parameters. Furthermore, the number of LTN parameters heavily depends on the number of features, whereas RWFNs are independent of the number of features. In principle, this could allow the learning process in our model to be accelerated if the feature representation from the encoder model is pre-processed and stored.
Running Time Fig. 3 depicts the comparison of running time, including data configuration time and training time for LTNs and RWFNs. The running time of RWFNs is roughly as half of that of LTNs. This is because the number of learnable parameters in RWFNs is far smaller than LTNs, and RWFNs have linear models to learn, which is much simpler compared to models used in LTNs.
of RWFNs, which can greatly reduce necessary space complexity when multiple classifiers are used simultaneously. In the depicted case of learning classifiers in Fig. 1b, the space complexity for RWFNs without using weight sharing is (2 ⋅ ⋅ + 3 ⋅ ) ⋅ = (2 ⋅ + 3) ⋅ ⋅ ≈ ( ⋅ ⋅ ) . However, with weight sharing, RWFNs can achieve much better space complexity, which is 2 ⋅ ⋅ + + 2 ⋅ ⋅ ≈ ( ⋅ ) because ⋅ < ⋅ for the experiments conducted in the SII task, but it can also be ( ⋅ ) for the diferent task. This indicates that the one of the factors that highly influence to the space complexity of the original RWFNs can be negligible when using weight sharing, which makes the model cost eficient and economical.