Combining neural networks (NN) methods with symbolic methods is an area that is exciting in terms both of basic research and practical application. We discuss here the opportunities and challenges involved in combining NN (“neural”) with logical knowledge representation and reasoning (KRR) that has high expressive power and fundamental scalability (“high-power” KRR). We describe how Rulelog KRR fits pretty well these challenges. (Note “logical” here includes probabilistic.) The domain-independent core of artificial intelligence (AI) consists of both KRR and machine learning (ML), as has been widely recognized within the AI research community since at least the 1980s. There are a number of ways in which it is useful, or even required, to combine KRR methods with ML methods. The prediction step of ML requires reasoning. The target of ML is a representation. Getting business value from ML usually requires reasoning for analysis and decisions. KRR is required to effectively combine the results of ML from multiple ML episodes, sources, or methods. KRR is required to accumulate knowledge coherently, as knowledge ongoingly originates from ML (as well as from non-ML origins). KRR is required to explain knowledge understandably. Explanations are often part of required or desired analysis functionality for their own sake; they are also needed for humans to trust an automated system, check and debug knowledge/reasoning, and to help ask drill-down follow-up questions. ML-based technology today often - arguably, usually - has weaknesses in regard to such effective combination, coherent accumulation, and understandable explaining. Reasoning is useful to supply derived facts for ML to chew on. Reasoning is useful to focus ML's tasks and conjecture schemas: e.g., to provide sets of relevant features (perhaps weightedly) and/or important questions (reasoning “(sub)goals”), so as as to drive ML. Last but not least, humans know stuff beyond what is available via ML training data, and such knowledge is often pretty complex to state. Programming is expensive, so KRR methods are often a more cost-effective approach to entry/capture of such knowledge.
Copyright © 2016 for this paper by its authors. Copying permitted for private and academic purposes.
Symbolic Side’s Representational Challenges and Requirements For the (KRR-based) symbolic side to combine most effectively with the neural side, there are a number of representational challenges and requirements. Highly flexible expressiveness is needed; ideally any kind of knowledge can be ingested by the KRR.
Thus it should be equipped feature-wise with higher-order syntax, logically quantified formulas, and strong meta (statements about statements). The KRR should represent numeric weighting and uncertainty, including (but not limited to) probabilistic and fuzzy. The KRR should treat the evolving character of knowledge and of the world; technically, it should have the defeasibility feature. The KRR should provide reasoning that is deep, including (but not limited to) multi-step in its logical chaining. The KRR should be scalable despite being expressive: not only computationally scalable to large amounts of asserted and concluded knowledge (“volume” and “velocity”) but also “socially scalable” in regard to the diverse multiplicity of ML/other info sources, algorithmic methods, and underlying data samples (“variety”). 4
Rulelog KRR Technology, its Advantages and Limitations Rulelog methods meet the above set of neural-driven representational challenges pretty well overall: not only pareto-optimally among the set of available KRR approaches, but arguably better than any other KRR approach. Rulelog methods are especially strong on the meta and higher-order features, while providing scalability. They are also well suited to orchestrating / federating multiple knowledge sources and components so as to assemble and compose multiple analysis results.
Rulelog is a leading approach to semantic rules KRR. It is expressively powerful,
computationally affordable, and has capable efficient implementations. A large subset
of Rulelog is in draft as an industry standard1 to be submitted to RuleML2 and W3C3
as a dialect of Rule Interchange Format (RIF) [
Rulelog [
the logical language); – explanations of inferences; – efficient higher-order defaults, including “argumentation theories”; – flexible probabilistic reasoning — including distribution semantics and evidential
probability; – bounded rationality, including restraint — a “control knob” to ensure that the com
putational complexity of inference is worst-case polynomial time; – “omni-directional” disjunction and existential quantifiers in the rule heads; – object-orientation and frame syntax, which subsumes RDF triples; 1 http://ruleml.org/rif/rulelog/rif/RIF-Rulelog.html 2 http://www.ruleml.org 3 http://www.w3.org
High-Power Logical Representation via Rulelog, for Neural-Symbolic – sound tight integration of first-order-logic ontologies including OWL; and several
other lesser features, including aggregation operators and integrity constraints.
Probabilistic reasoning and tight integration with (inductive) ML is a key area of recent technology progress and ongoing R&D on Rulelog.
Rulelog also combines closely with natural language processing (NLP), in the
Textual Rulelog approach [
Implementation techniques for Rulelog inferencing include transformational
compilations and extensions of tabling algorithms from logic programming. “Tabling” here
means smart caching of subgoals and conclusions together with incremental revision
of the cached conclusions when facts or rules are dynamically added or deleted [
There are both open-source and commercial tools for Rulelog that vary in their range of expressive completeness and of user convenience. They are interoperable with databases and spreadsheets, and complement inductive machine learning and natural language processing techniques. The most complete system today for Rulelog is Ergo4, a commercial platform suite from Coherent Knowledge5. Ergo Lite, a.k.a. Flora-26, is an open source system that implements a significant subset of Rulelog reasoning. Ergo’s ErgoText feature is a commercial realization of Textual Rulelog.
Rulelog has some important limitations. One is that it lacks “reasoning-by-cases”, a.k.a. it is “intuitionistic”; it only concludes a disjunction if it concludes one of the disjuncts. Another limitation is that Rulelog methods are not yet optimized for probabilistic reasoning.
As has been extensively discussed in the ML and KRR literature, classical logic (e.g., first-order logic but also higher-order logic) has reasoning-by-cases but is brittle in the face of conflicting/evolving knowledge, and lacks computational scalability.
Markov Logic Networks [
Applications to date of Rulelog technology include a wide range of tasks and domains in business, government, and science. Examples include: legal/policy compliance, particularly in financial services; personalized tutoring about science; and e-commerce marketing. Rulelog shines especially on representing, and deeply reasoning with, complex commonly-arising kinds of knowledge such as: mappings among terminologies, ontologies, and data schemas; policies, regulations, and contracts; and causal pathways (e.g., in science). 4 http://coherentknowledge.com/ergo-suite-platform-technology/ 5 http://coherentknowledge.com 6 http://http://flora.sourceforge.net
An immediate direction for future R&D is to hook up Rulelog implementations to NN systems. There are many potential applications for this combination of Rulelog KRR with NN – and/or with other ML. One realm is compliance and fraud. Another realm is NL understanding in intelligence analysis and in search.
There are a number of other interesting future research directions in terms of core technology and experiments, per the overall discussion in section 1. Next, we highlight a few of these directions. One is to feed derived data from Rulelog to NN. Another is to combine the results of neural with other ML and structured info, including humanauthored complex knowledge, e.g., that started life as English sentences. Terminology mappings and source trustworthiness are two interesting kinds of such complex knowledge represented in Rulelog. A third direction is to combine NN word-vector distributed representations with Textual Rulelog.
Related future directions for research on Rulelog KRR itself include to optimize its reasoning with probabilistic/weighted knowledge, and to extend Rulelog’s expressiveness to selective reasoning-by-cases.