We report on new evaluations regarding the use of a Memory Networks deep learning architecture for deductive RDF reasoning. Evaluation on the SemRec CaLiGraph dataset show performance consistent with previously reported results, even without re-training on the CaLiGraph dataset. We also report on a preliminary evaluation regarding performance of the Memory Networks system with imperfect data. It shows that the system gracefully degrades.
as the training data. While this can be useful in some cases, deductive reasoning as defined is not dependant on the names of identifiers, i.e. the reasoning remains the same after a rewriting of identifier names into diferent ones. Due to this, we believe that it is only fair to say that a deep learning system has achieved deductive reasoning, if it performs well on completely new input RDF graphs including new vocabulary (i.e., IRIs). In other words, we prefer a Deep Deductive Reasoner to be able to transfer to completely new datasets.
The Memory Network system we report here is indeed capable of transfer. In fact, for part of
the evaluation, the system was not re-trained – it is the trained system reported in [
The plan of the paper is as follows. In Section 2 we will briefly recap the architecture of the Memory Network system we use in our experiments and recall previously published evaluation results. In Section 3 we present our new evaluation results. And in Section 4 we conclude.
Our system is an end-to-end Memory Network (MemN2N) [
The evaluation of classification as reported in [
In summary, we see that for the natural unmodified datasets the f-measure falls between 0.68 and 0.96. For the artificially harder datasets, it falls between 0.27 and 0.77 (with only one value lower than 0.54), and for the synthetic datasets it falls between 0.25 and 0.62.
We performed three evaluations of the Memory Network system. The first was on the SemRec
2022 CaLiGraph dataset, including training on the same. The second was using our pre-trained
system, exactly as used for the experiments reported in [
1We actually used a cloud version of the identical algorithm that was put in place for improved training times. 2http://lodlaundromat.org 3https://lod-cloud.net
Training A A (90%) A A A C C (90%) A C A C
The SemRec 2022 CaLiGraph dataset4 was provided as part of the ISWC 2022 Semantic Reasoning
Evaluation Challenge.5 CaLiGraph is a large cross-domain knowledge graph generated from
categories, list pages and other lists in Wikipedia [
The current Memory Network system accepts RDF graphs consisting of maximum 1,000 triples. In order to process the CaLiGraph datasets, they were split into disjoint subsets of maximum 1,000 triples each. Inferences were obtained for each of the resulting datasets, thus providing the required ground truth for training and validation. We note that this means that the system does not provide all inferences over the full dataset, it is limited to graphs with maximum 1,000 triples, and combining all inferences from the smaller subsets does not in general provide all infereneces over the full graph.
The evaluation results we received are shown in Table 2 (left) for the correctly inferrable triples. We note that the values are well in line with previously reported results, see Section 2.
In order to assess transfer capability of our Memory Network system, we also used the SemRec CaLiGraph validation datasets to assess how well our pre-trained system can reason over this 4http://data.dws.informatik.uni-mannheim.de/CaLiGraph/CaLiGraph-for-SemREC/SemREC-2022-Datasets/ 5https://semrec.github.io/
Dataset Clg_Full Clg_10e4 Clg_10e5 completely new dataset. The results can be found in Table 2 (right). We see that the system manages transfer well to this dataset. The values are not quite as good as for the the system trained on the CaLiGraph data, but the correctness loss is moderate.
We finally ran an evaluation on data into which we introduced random errors, in order to assess whether system performance degrades gracefully. We ran a series of test with diferent error rates; e.g., an error rate of 10% means that we modified 10% of the IRIs by replacing them with randomly chosen IRIs from the same pool. Evaluation, however was done with respect to the inferences in the unmodified data (i.e., no introduced errors). The base datasets used were those from the original evaluation reported on in Section 2.
Results of the experiments can be found in Table 3, for correctly inferrable triples. We see that the correctness values are lower than for data without introduced errors, but precision, recall and f-measure are only moderately lower. We also notice that higher error rates hardly decrease correctness, which is somewhat surprising; further evaluation and analysis will be needed to understand this behavior.
Performance of our Memory Network based deep deductive reasoner on the SemRec CaLiGraph data was consistent with our previous findings. In addition, we note that the system transferred well to the new dataset without re-training. Regarding reasoning over data with introduced errors, the system showed graceful degradation, but follow-up analysis will be needed.
A major limitation of our system is that it can only deal with input RDF graphs of at most 1,000 triples. However, training time is already rather substantial for this size, i.e. a significant increase in processable knowledge graph size does currently not appear to be reasonable with this exact approach. Another limitation of the Memory Network based system is that it is query based, i.e. the system does not quickly produce a graph with inferred triples; rather, it has to be queried about each triple whether it is inferred or not. A generative system that would be able to quickly produce a graph with inferred triples would be much preferred, however at this time we know of no published work that would describe a Deep Deductive Reasoning system that is both generative and that can transfer, even for as simple a logic as RDFS.