Since the 2001 envisioning of the Semantic Web (SW), the main research focus in SW reasoning has been on the soundness and completeness of reasoners. While these reasoners assume the veracity of input data, the reality is that the Web of data is inherently noisy. Although there has been recent work on noise-tolerant reasoning, it has focused on type inference rather than full RDFS reasoning. Even though RDFS closure generation can be seen as a Knowledge Graph (KG) completion problem, the problem setting is di erent| making KG embedding techniques that were designed for link prediction not suitable for RDFS reasoning. This paper documents a novel approach that extends noisetolerance in the SW to full RDFS reasoning. Our embedding technique| that is tailored for RDFS reasoning| consists of layering RDF graphs and encoding them in the form of 3D adjacency matrices where each layer layout forms a graph word. Each input graph and its entailments are then represented as sequences of graph words, and RDFS inference can be formulated as translation of these graph words sequences, achieved through neural machine translation. Our evaluation on LUBM1 synthetic dataset shows 97% validation accuracy and 87:76% on a subset of DBpedia while demonstrating a noise-tolerance unavailable with rule-based reasoners.
The Web is inherently noisy and as such its extension is noisy as well. This noise is as a result of inevitable human error when creating the content, designing the tools that facilitate the data exchange, conceptualizing the ontologies that allow machines to understand the data content, mapping concepts from di erent ontologies, etc. This paper documents a novel approach that takes previous research e orts on noise-tolerance to the next level of full RDF Schema (RDFS) reasoning. The proposed approach utilizes the recent advances in deep learningthat showed robustness to noise in other machine learning applications such as computer vision and natural language understanding- for semantic reasoning. The rst step towards bridging the Neural-Symbolic gap for RDFS reasoning is ? The code, models and datasets for this https://github.com/Bassem-Makni/NMT4RDFS paper are available at: to represent Resource Description Framework (RDF) graphs in a format that can be fed to neural networks. The most intuitive representation to use is graph representation. However, RDF graphs di er from simple graphs as de ned in the graph theory in a number of ways. The di erent graph models for RDF in the literature were neither designed for RDFS reasoning requirements nor are they suitable for neural network input. The proposed graph model for RDF consists of layering RDF graphs and encoding them in the form of 3D adjacency matrices. Each layer layout in the 3D adjacency matrices forms what we termed as a graph word. Every input graph and its corresponding inference are then represented as sequences of graph words. The RDFS inference becomes equivalent to the translation of graph words that is achieved through neural network translation. The main contributions in this paper are: { Noise Intolerance Conditions. A taxonomy for noise types in SW data according to the impact of the noise on the inference is drawn along with the necessary conditions for a noise type to be propagable (i.e a ect the inference). { Layered Graph Model for RDF. We propose a layered graph model for
RDF that is tailored for RDFS reasoning. { Graph Words. Using the layered graph model, we propose a novel way of representing RDF graphs as a sequence of graph words. { Graph-to-Graph Learning. By representing RDF graphs as a sequence of graph words, we were able to use neural network translation techniques for translation of graph words. { Full RDFS reasoning with noise tolerance. Our evaluation shows not only comparable results with rule-based reasoners on intact data but also exceptional noise-tolerance compared to them: 99% for the deep reasoner vs 0% (by design) for Jena in the U GS100 dataset.
In Section 2, we use three aspects to position our research with respect to the related work. Section 3 draws a taxonomy for noise types in SW data and illustrates the process of ground truthing and noise induction for LUBM and a subset of DBpedia. We describe the layered graph model for RDF in Section 4. Then, the overall approach including the creation of the RDF tensors and the RDF graph words is presented in Section 5. The results of the experiments are described in the Section 6. 2
In this section we use three aspects to position our research with respect to
related work:
{ Noise handling strategies: active vs adaptive. Active noise handling
consists of detecting noise and cleansing the data before performing any tasks
that might be a ected by the presence of noise, while adaptive noise handling
approaches focus rather on building techniques that are noise-tolerant. The
research described in this paper falls into the latter category.
{ Knowledge graph completion categories: schema-guided vs
datadriven RDFS closure can be seen as a Knowledge Graph Completion (KGC)
problem| multi-relational link prediction problem in particular| where
each RDFS rule generates di erent types of links. We refer to the RDFS
closure computation as schema-guided KGC because the links are generated
according to the ontology (TBox), unlike data-driven KGC where the links
are predicted based on the analysis of the existing links in the KG.
{ Graph embedding output: Node/Edge embedding vs whole-graph
embedding Graph embedding approaches can be classi ed using several
criteria. One particular criterion of interest is the \problem setting" [
The majority of graph embedding approaches yield node representation in a low dimensional space. This is why graph embedding and node embedding are often used interchangeably. However, there are other types of graph embedding outputs such as edge embedding and whole graph embedding| where the output is a vector representation of the whole graph not only node or edge vectors. The embedding vectors of similar graphs should be neighbors in the embedding space. The embedding of RDF graphs| in order to learn their inference| falls under this category. 2.1
Existing embedding techniques for KGs were not designed for RDFS reasoning and they raise two main challenges if they were to be used for this task. 1. The rst challenge is the need to check the validity of every possible triple using the scoring function in order to generate the full materialization. 2. The second challenge is the embedding of the relations that are seen only in the inference such as the super-properties. 3
For this research, the input is from one of two types of datasets: a synthetic
dataset from LUBM and a real-world dataset from DBpedia [
The literature contains a few taxonomies for the types of noise that can impact RDF graphs; however they are not drawn with respect to the impact of the noise on the inference. The taxonomy illustrated in Fig. 1 serves this purpose. TBox Noise is the type of noise that resides within the ontology, such as in the class hierarchy or domain and range properties. This type of Semantic Web noise impacts inference over the whole noise dataset. Reasoning with tolerance to TBox noise is outside the scope of this ABox noise research because using rule-based reasoners for ground truthing with noise TBox noise Propagable noise Non-propagable in the TBox biases the whole ground noise truth. Non-propagable noise consists of any corrupted triple in the input graph that does not have any impact Fig. 1: Semantic Web noise taxonomy on the inference|for example when the original triple does not generate any inference nor does the corrupted triple. Propagable noise, on the other hand, is a corrupted triple in the input graph that changes the inference. The necessary conditions for RDFS rules to generate a noisy inference from a corrupted triple are identi ed in the extended journal paper. 3.2
LUBM1 was generated according to the LUBM [
Noise Induction in LUBM1: In [
As [
From DBpedia [
Even though the RDF conceptual model is designed as a graph, it di ers from
the graph theory de nition of graphs in a number of ways. RDF graphs are
heterogeneous multigraphs. Moreover, an edge in the ABox can be a node in the
TBox (describing the properties hierarchy for example). Current research e orts
to represent the RDF model as graphs| based on a: bipartite graph model
[
The proposed layered graph model for RDF consists of an ordered sequence of directed graphs, where each directed graph represents the relations between the resources of the RDF graph according to one property (from the set of properties in the ontology plus rdf:Type). It is important to note that the transformation of an RDF graph into its layered directed graph representation is not bijective as two non-isomorphic RDF graphs can have the same layered directed graph representation. However this transformation guarantees that if two RDF graphs have the same layered directed graph representation then their RDFS inference graphs according to the ontology O are isomorphic. 5
The overview of our approach is depicted in Fig. 2. It can be summarized in the following steps: 1. Layering the RDF graphs: Using the layered graph model for RDF, we generate the layered graph version for each input and inference graph in the training set. The order of the properties corresponding to each layer can be chosen arbitrarily but it is crucial to maintain the same order across the dataset. In the simple version, each property in the ontology plus rdf:Type has its corresponding layer. While in the more e cient version, only the subset of "active" properties is considered|where an active property is a property from the ontology that is used in the dataset. This reduces the number of layers dramatically in the case of the scientists dataset where only a small subset of the DBpedia properties is used. 2. RDF tensors creation Using the layered version of the RDF graphs, an ID must be assigned to each resource in the RDF graph to allow it to be represented as a 3D adjacency matrix. In the simpli ed version, two dictionaries were created: one for the subject and object IDs| which is split into a global and a local dictionary| and one for the property IDs. The global resources dictionary contains the subject and object resources that are used throughout the G set (which are basically the RDFS classes in the ontology). While the local resources' dictionaries contain the IDs for the resources speci c to each graph. In the more advanced version|required for complex ontologies| instead of using the same local dictionary for all the layers, each group of properties share their own dictionaries of local resources. 3. Graph words generation At this stage, every RDF graph is represented as a 3D adjacency matrix of size: (active properties size, max number of resources, max number of resources). In theory the maximum number of possible layer layouts in a dataset of size dataset size is:
min(2max number of resources2 ; dataset size active properties size) However, in practice, the number of layouts is much smaller than this theoretical bound. For instance, in the LUBM1 dataset, we obtained 131 layouts versus 309; 402 possible layouts. This observation is a good indication that the encoding algorithm has achieved one of its major goals of having similar encodings for \similar" graphs. By assigning an ID for each layer's layout, the 3D adjacency matrix can be represented as a sequence of layouts' IDs as shown in Fig. 3. The layouts are termed \graph words", as the sequence (or phrase) of graph words represents a 3D adjacency matrix and thus an RDF graph. Representing an RDF graph as a sequence of graph words has two main advantages: (a) Reducing the size of the encoded dataset: only the ID of the layer's layout along with a catalog of layouts is saved. (b) Exploitation of the research results in neural machine translation.
ub:subOrganizationOf ub:softwareVersion ub:publicationAuthor ub:advisor
RDF:type
0 0 0 0 .... 0000 0000 u..0000b..:0000ta.k.0000.e.0000.s.0001C....0000o......u..........0100r.....s...0000e...0000.....0010...................0000.........1000...0000.....0100...................0010.........0010...000.....1000...................000.........0001....010........................110.. ................ .... .... .... .... ....
0 1 2 3 4 5 ...
Layers catalog
(4,5,3,1,0,0) Graph sentence of graph words The overall architecture of the model as well as its hyperparameters are detailed in the journal paper. 6
cFeisgs. o4n sthhoewLsUtBhMe1trdaaintainsget.pArof-- 100 Best accuracy: 97.67% 1 ValiTdraatiinoinngaclcoussracy ter approximately 12 minutes of 0.8 training, 98:8% training accu- 80 trhaceytrwaainseadchmieovdeedl. oWnhtehneteinsttaincgt rcccayuA 6400 00..64 lsso LUBM1 test set, an overall pergraph accuracy of 97:7% was ob- 20 0.2 tained. 00 20 40 60 80 100 120 140 160 180 2000 Epochs
The trained model was then tested on the noisy datasets cre- Fig. 4: Training results on intact LUBM1 ated as described in Section 3. data Two metrics were designed: { Macroscopic metric: Per-graph accuracy : Inferences in this metric are scored correct when dr and i are isomorphic| in other words, when the deep reasoner inference from the corrupted graph is isomorphic to the Jena inference from the intact graph. { Microscopic metric: Per-triple precision/recall. The previous metric overlooks the fact that some triples, generated by the deep reasoner and not by Jena, were in fact valid.
The macro and micro evaluation on the 5 noisy datasets (Fig. 5) shows exceptional noise-tolerance compared to rule-based reasoners: 99% for the deep reasoner vs 0% (by design) for Jena in the U GS100 dataset.
The model used for the scientists dataset is like the LUBM1 model, except for the hyper-parameters. Trainiftrdpneaeaergakrtesesantoossocnneeoate-vprgvcelraaworalcneipe1dtrha6eansitonihpgoiseosenreun9fareeo4scrcu.atcgt,tuTreoracdhafopcenwhbytsyh`asoiicwctfnihh8eiitnen73hgt:d87iset6eitehnx%spe--' 93 99 99 87tNyop92iesy91assertion76s 18 59 prNo64opies100ryty09assertions 51 64 186400000 )(rcccayuA% actly the inference from Jena minus 20 the noisy triple. For the remaining 0 0 "pfearlssoenp-polsaicteivien"fetrreinpcleess,naotfegwencoernattaeidn UGS100 RATA100 GCC100 TEPA100 WOAD100 by Jena. For example, the deep rea- Per-graph accuracy Per-triple precision Per-triple recall soner inferred that dbr:Big Ben is of type dbo:HistoricPlace even though this information is not explicitly (i.e. Fig. 5: Macro and micro evaluation on embedded in the DBpedia graph of noisy LUBM1 datasets the the resource dbr:Big Ben) nor implicitly (i.e. can be inferred). The deep reasoner inferred this information by capturing the generalization that resources with similar links to dbr:Big Ben are usually of type dbo:HistoricPlace. 7
The main contribution of this paper is the empirical evidence that deep learning (neural networks translation in particular) can in fact be used to learn semantic reasoning| RDFS rules speci cally. The goal was not to reinvent the wheel and design a Yet another Semantic Reasoner (YaSR) using a new technology; it was rather to ll a gap that existing rule-based semantic reasoners could not satisfy, which is noise-tolerance. This research can be extended in a few directions. Currently the trained model on a speci c domain with a given ontology cannot be used to generate inferences from a di erent domain. One promising research direction is to investigate transfer learning to bootstrap the adaption for new domains.