Momentum is surging behind the consensus that neural-symbolic AI is the right road for AI to take today. We propose to travel this road using Semantic Web technologies to represent the symbolic AI tradition. Our objective is to investigate and compare the eficacy of a variety of strategies for combining the capabilities of deep neural networks for statistical learning from data with those of OWL ontologies and knowledge graphs for symbolic knowledge representation and reasoning. Our application area is visual relationship detection within images. Deep learning is data hungry and struggles to generalise to examples outside the training distribution. We seek to show that combining Semantic Web domain knowledge and reasoning with deep learning can deliver superior performance, can substitute for plentiful training data, and can deliver robust generalisation in few-shot/zero-shot learning scenarios.
At the micro-level, our problem space is the application area of visual relationship detection
within images, using a small (5,000 images) dataset prepared for this purpose in 2016: the VRD
dataset [
Deep learning is known to depend on big data for good performance, and to struggle to generalise and extrapolate to examples that lie outside the distribution of data seen during training. The small size of the VRD image dataset, and the long tail on the highly skewed frequency distribution of the VR annotations of the images, are likely to provoke these limitations of deep learning. Our research considers how to leverage the symbolic knowledge representation and reasoning capabilities of a KG hosting our OWL VRD-world ontology so as to overcome the limitations of deep learning and deliver superior performance at detecting VRs in images.
Our research has relevance and benefits for several groups: the AI community, industry, society
generally and the SW community. Prominent voices from the AI community such as those
of Marcus [
AI has transformed many aspects of everyday life, in industry and for society generally, and in ways we all now take for granted. We all have expectations of continuing positive innovations and transformative efects from AI. Hence, as neural-symbolic AI research advances AI along the road to AGI, industry and society generally will be impacted directly and benefit proportionately.
Finally, our research has the potential to show that OWL ontologies and KGs can be used
to integrate neural, statistical learning with symbolic background knowledge and reasoning
in concrete, tangible ways. In doing so, it can demonstrate that these SW technologies are
exemplars of the types of symbol manipulating tools and abstraction and reasoning modules
that Marcus [
Work relating to neural-symbolic AI in general can be reviewed in surveys such as [
Work relating to neural-symbolic AI that uses SW technologies to represent the domain
of symbolism is rapidly accumulating. Myklebust et al. [
Work relating to using neural-symbolic AI for VR detection on the VRD dataset also exists.
The original (2016) VRD paper by Lu et al., [
RQ1: How can we combine learning and reasoning to get the best of both worlds? Here, ‘learning’ refers to the the statistical ‘learning from data’ capabilities of deep NNs, ‘reasoning’ refers to the symbolic knowledge representation and reasoning capabilities of OWL ontologies and KGs, and ‘best of both worlds’ refers to improved VR predictive performance. We hypothesise that each of the several distinct NN-KG integration (NN-KG-I) strategies that we have conceived (and which we describe shortly) will deliver VR predictive performance that is superior to whatever baseline performance our deep NNs are able to deliver by themselves. We aim to experiment with each of our NN-KG-I strategies individually, rank them, and explain their relative eficacy by analysing the nature of the interactions between deep learning and symbolic knowledge representation and reasoning that they exercise.
RQ2: Some of our NN-KG-I strategies will be compatible with one another. How will they perform when used in diferent combinations? We hypothesise that we will find the contributions to improved VR predictive performance (beyond the baseline) that they make when used individually are additive when used in certain combinations, but that when used in other combinations there are interesting interaction efects between the integration strategies which either amplify or diminish their collective efect on VR predictive performance. Analysis of the results of these experiments is expected to yield further insights into the nature of the interactions between deep learning and symbolic knowledge representation and reasoning. RQ3: How best and to what extent can the symbolic knowledge representation and reasoning capabilities of OWL ontologies and KGs be leveraged by deep NNs to substitute for plentiful training data and enable robust generalisation on out-of-distribution examples? Within the VRD image annotations, many VR types have just a few training instances, and many have test instances but zero training instances. Hence, the VRD dataset lends itself to the examination of the sort of small dataset and few-shot/zero-shot learning scenarios in which deep learning alone struggles to perform well. We hypothesise that our NN-KG-I strategies will show that SW technologies such as our OWL VRD-world ontology, together with the reasoning capabilities of a KG, can be used to construct hybrid, neural-symbolic systems that out-perform deep learning alone in such settings.
The architecture of our baseline hybrid, neural-symbolic system consists of a NN pipeline and a
KG populated with our VRD-world ontology. The NN pipeline consists of an object detection
NN followed by a multi-label predicate prediction NN that takes ordered pairs of detected
objects, plus geometric features derived from their bounding boxes (an idea borrowed from
[
2
Object Detection
Neural Network
Predicate Prediction
Neural
Network < person, ride, bike >, < bike, carry, person >, < person, hold, phone >, < person, wear, jacket > process KG feedback 4 < error: KG state inconsistent >
insert candidate 3 VR prediction in KG < subjectX, rdf:type, Person > < objectY, rdf:type, Phone > < subjectX, wear, objectY >
Knowledge
Graph with VRD-world ontology S1: Strategy S1 is about leveraging a SW KG’s ability to automatically enforce the semantic rules of an OWL ontology. The only predicted VRs that stand a chance of matching with annotated (ground truth) VRs are those that are semantically valid according to our VRD-world ontology. So it makes sense to help our predicate prediction NN learn the relevant semantic rules of our VRD-world ontology. This is analogous to a checkers-playing system learning the legal checkers moves. It does not fully solve the problem of finding the best move, but by narrowing the search space to the legal moves, learning to find the best move becomes significantly easier. One way to utilise strategy S1 is to take the VR predictions emitted by the predicate prediction NN during training, insert them into a KG populated with the VRD-world ontology, and then communicate any feedback from the KG regarding invalid VRs back to the NN (e.g. by penalising its loss function). This scenario is depicted in Figure 1.
There is also potential to utilise strategy S1 during inference. Despite having trained our predicate prediction NN as best we can to only predict semantically valid VRs, it may still sometimes predict semantically invalid VRs during inference on test set images. But the KG is an active agent capable of participating in determining the final predictions of the hybrid, neural-symbolic system. Specifically, we can use it as a final filter to suppress bad VR predictions. Rather than submit the VR predictions of the predicate prediction NN directly (on behalf of the hybrid, neural-symbolic system), we may first insert them into the KG and then submit only those that the KG does not flag as being semantically invalid.
Our strategy S1 is related conceptually to the approach taken in [
S2: Strategy S2 involves using common-sense, Datalog-like rules to leverage and augment the reasoning capabilities of our KG. For example, a rule expressing the plausibility of VR pattern (X, wear, Y) can be described as follows:
wear(X,Y) :- Person(X), WearableThing(Y), ir(Y,X) ~ 1
Triples asserting the predicted classes of the detected objects and will have been inserted
into our KG before the rule is evaluated: (X, rdf:type, ), (Y, rdf:type, ).
Determining if the first goal of the body of this rule is satisfied is accomplished by a KG query
to check whether X is a member of class Person. Determining if the second goal is satisfied is
accomplished similarly but depends on the KG having deduced whether Y is a WearableThing
(which is not a VRD object class). The third goal represents a novel reuse of a bounding box
geometric feature function (per [
We will define a collection (base) of such rules using the knowledge of the VRD-world domain gained from having analysed the VR annotations so as to design the ontology. The impact of strategy S2 will likely be proportional to the comprehensiveness of this rule base, but on a diminishing scale. Analysis of the frequency distribution of the VR annotation types will help us identify those rules likely to have the greatest impact. We can minimise the number of rules required to deliver a measurable performance efect by focusing on high-impact rules.
We have identified two approaches for implementing our common-sense rules. Approach S2-A is to use Python to build custom rules and a simple (non-recursive) rule engine component for evaluating them. This rule engine component would mediate interaction between the predicate prediction NN and the KG. The description of S2 given above presumes this approach. Approach S2-B is to define proper Datalog rules and to use a KG tool whose support for Datalog includes support for basic arithmetic functions suficient for emulating bounding box geometric feature functions (such as the inclusion ratio).
A further approach to S2, S2-C, would share S2-A’s Python-based rules and rule engine while allowing us to explore an additional dimension of NN-KG integration. This approach involves developing methods for transferring and representing the structure of the class hierarchy of our VRD-world ontology within supplementary layers (and their associated weight matrices) of our object detection NN. The objective is to enable the object detection NN to not just detect objects of base classes (e.g. jacket) but to perform the generalisations needed to convey all the parent classes for each object (e.g. clothing, wearable thing, etc..) as well. This way, our Python-based rule engine need not query the KG because any class membership information required to determine rule goal satisfaction will have been supplied by the NN. S1 and S2 combined: NN-KG-I strategy S1 should be good at identifying poor VR predictions (negative cases), while S2 should be good at identifying plausible VR predictions (positive cases). So, in theory, they are complementary and may work well together, delivering a combined boost to VR predictive performance.
Some other NN-KG-I strategies: Another integration strategy involves using KG embeddings to train a NN to score the plausibility of VR predictions. This scoring NN would then be used to help train the predicate prediction NN.
A further strategy involves leveraging the training set VR annotations (KG data). Each ordered pair of object classes will have some number of annotated VR instances involving some subset of the 70 predicates. These can be transformed into discrete probability distributions over the predicates. The VR predictions generated by our predicate prediction NN during training can similarly be transformed into corresponding discrete probability distributions. The dissimilarity of corresponding pairs of predicted and annotated VR probability distributions can then be measured (using a metric identified for this purpose), and these measures can be aggregated to produce a penalty term with which to augment the loss function of the predicate prediction NN. Tactics for leveraging the ontology so as to intelligently redistribute probability in the target distributions can also be explored to better facilitate few-shot and zero-shot learning. Evaluation: The NN pipeline of our system architecture will be capable of delivering some measure of VR predictive performance on its own. This is the baseline performance measure against which all NN-KG-I strategies will be judged.
The authors of [
As per [
A key principle of our evaluation strategy is to keep the baseline architecture of our hybrid system unchanged across investigations of diferent NN-KG-I strategies. This will enable us to attribute changes in VR predictive performance to a NN-KG-I strategy alone. It will also best enable us to compare and rank our strategies in terms of VR predictive performance eficacy.
We use multiple metrics so our evaluation strategy cross-validates. If the multiple measures of performance corroborate one another, we can interpret the efect of a given NN-KG-I strategy with confidence. If not, this will signal the need for caution and investigation.
We are still assembling the infrastructure to enable experimentation, so we discuss preliminary results in the sense of things accomplished. The original, crowd-sourced VR annotations of the VRD dataset are full of inconsistencies and errors. For example, object class ‘bear’ refers to real bears and teddy bears; class ‘plate’ refers to dishware plates, license plates (on vehicles) and baseball (home) plates; too many instances of VR pattern (person, wear, Y) have Y on a diferent person. Apart from making object detection and relationship prediction noisily problematic, the semantic variability of the object classes made precise ontology design infeasible. No class hierarchy felt credible, and few opportunities existed to define useful domain/range restrictions on object properties (VRD predicates). We therefore undertook a comprehensive VR analysis and customisation exercise to strengthen the semantic consistency of the VR annotations. In time, our VRD-world ontology, our customised VR annotations, our protocol for specifying VR customisations textually, and our code for applying them in an automated, repeatable fashion, will be made publicly available as a contribution to the AI and SW communities.
Object detection training and experimentation is underway. Our predicate prediction NN has been designed. Our evaluation metrics have been implemented and proof-of-concept testing confirms they behave as expected. Proof-of-concept exercises confirming the feasibility of NN-KG-I strategies S1 and S2 (described above) have been completed successfully. A customised binary cross-entropy loss function has been conceived for training our multi-label predicate prediction NN that provides parameters for influencing the loss attracted by predicted VRs that have no matching annotated VR. Many of these predicted VRs will be entirely plausible and yet will be treated as false positives simply due to the unavoidable sparsity and arbitrariness of the annotated VRs. We aim to explore the efect of influencing the magnitude of the loss attracted by such plausible false positive cases, based on judgements derived from our KG.
Reflection: First, our research is about combining the use of KGs with deep learning in hybrid, neural-symbolic systems. The application task of VR detection within images is simply a context for exploring combination/integration strategies. We believe our NN-KG-I strategies to be generic and widely applicable. However, we plan to continue looking for other dataset/ontology pairs with which to apply our strategies so as to further demonstrate their generality.
Second, as we have described, we chose to heavily customise the original, crowd-sourced
VR annotations of the VRD images in order to enable the design of a precise ontology (and
to correct egregious errors). A consequence of this choice is that we sacrifice the ability to
directly compare the predictive performance results of our various hybrid VR detection systems
with those of the systems of previous researchers (such as [
Third, NN-KG-I strategies such as S1 and S2 that rely on real-time interaction with a KG are
likely to increase NN training times considerably, particularly if the KG is out-of-process (even
online) and accessed via a SPARQL endpoint. We do not, however, believe this consideration
to be a major concern. In our case, the small VRD dataset (4,000 training images) means no
issues should arise. More generally, we surmise that the growth rate of NN training times
relative to dataset size, , will be linear (() time) and that, on this basis, the computational
complexity implications of real-time KG access should always be manageable. Further, tactics
such as caching may well be exploitable to help keep KG access to a minimum.
Future work: Our research can readily extend in multiple directions. One direction is to
pursue the goal of contributing to the development of a theory to help formalise the foundations
of neural-symbolic AI, as advocated by van Harmelen in [
Thank you to my supervisors Dr. Ernesto Jiménez-Ruiz and Dr. Tillman Weyde for their guidance and support.