This paper proposes a domain-independent method to evaluate inferences for analogical reasoning, via a prototype system. The system assigns analogical inferences confidences based on the quality of the mapping and the system's confidence in the facts used to generate the inference. An initial implementation is applied to two domains.
Any reasoning system which asserts facts through the processing and manipulation of
previously known information ought to have a measure of confidence in the accuracy
of those newly asserted facts. Even if a given reasoning technique is sound, inferred
facts are only as accurate as the assumptions upon which they are based. For example,
systems that reason via formal logic produce reliable inferences, but if the reasoning
environment is complex enough, or a particular axiom is missing, contradictions may
pass undetected. Furthermore, forward chaining systems overgenerate inferences, while
backchaining systems are directed, but require a known goal for reasoning. On the
other hand, probabilistic systems such as Bayes Nets [
This paper proposes a unified approach to evaluating and scoring analogical inferences. It integrates logical reasoning, analogical reasoning, and probabilistic reasoning to provide confidence estimates for analogical inferences. We present an initial implementation and some experimental results as a proof of concept of these ideas.
Copyright © 2017 for this paper by its authors. Copying permitted for private and academic purpose. In Proceedings of the ICCBR 2017 Workshops. Trondheim, Norway 1.1
The principles underlying our system are domain general. Our implementation uses
the Structure-mapping Engine (SME, [
SME [
When the system reasons its way to a new fact in a context, it can either be certain it is true, certain it is false, or somewhere in between. The system uses disjointness reasoning, logical contradiction and implication, and the parameters of SME mappings to determine the system’s confidence that an inference is true. All reasoning is done with respect to the context in which the inference is to be asserted. 2.1
If the system has inferred that an entity is of a certain type, and there is already a contextualized assertion that it is of another type that is by definition disjoint from the first, the system simply rejects that inference. For example, if Fluffy is a dog, it cannot assert that it is a cat unless it first retracts that it is a dog. In the Cyc knowledge base, certain collections are marked as disjoint collection types, such that if an entity is an instance of one of those types, it cannot be an instance of another. When our system detects that an inference is of the form (in-Context ?context (isa ?entity ?newType)), it gathers all the other declarations of that entity’s type in context ?context. If any of those other types are disjoint with ?newType, then the system rejects the inference. Inferences can also be rejected if they are contradicted by known implication rules. If there is a rule of the form A -> ~I, where I is the analogical inference, and A is known to be true, the inference can be rejected. Similarly, if there is a rule of the form I -> A, and A is explicitly known to be false, the inference can be rejected.
Implication is similar: If there is a rule of the form A -> I, and A is true, the inference has been confirmed. Similarly, if there is the rule ~A -> I, and A is known to be false. The confidence in the implied fact is a function of the confidence assigned to the facts used to imply it. Contrapositives of the rules for implication and contradiction are generated on-the-fly. We do not assume rules are sufficiently complete to generate all inferences generated by analogy. Even if they were, analogy would be useful for focusing logical reasoning. The system makes use of forward chaining in a targeted fashion, only for verification, which is more efficient than simply forward-chaining. 2.2
When the system derives an inference using an analogical mapping, it may be able to directly prove or disprove it. Failing that, it is desirable to have a measure of the extent to which the inference is trusted. The normalized SME match score is one such signal. Another is the degree to which the facts the inference is based on (in the base and target cases) are trusted. If the base case is a SAGE generalization, then the fact probability in the generalization tells us how likely that fact is within that generalization. For a nongeneralized case, the system does not know the extent to which the case itself is an outlier or whether any one fact in the case is core to the overall concept that the case encodes. Inferences from individual cases should be trusted less than high-probability generalization facts, since there is evidence from the generalization that the high-probability facts are more common.
Putting it all together, analogical inferences are assigned confidence scores thus: P Inference = ℎ ∗ ∗ ! "#$$% The BaseTrust is as described above: If the base case is a SAGE generalization then: BaseTrust = And otherwise it is set to the default normalizing value (currently, 0.7).
Given this formula, confidence scores are always on the interval (0, 1): normalized match scores are on that interval, and fact confidence scores are on the interval (0, 1]. Confidence cannot be zero since zero-confidence facts are simply suppressed, rather than asserted with confidence zero. Normalized match scores are a measure of the degree of overlap between cases, rather than the total amount of information being mapped from one case to another. These can be low but are never zero: for a mapping to be generated at all, there must be some degree of overlap.
We use a product, rather than, say, a sum, of these components to keep confidence on the (0,1) interval. Intuitively it makes sense that a fact inferred from many facts should be trusted less than one inferred from only a few (if we are equally uncertain of the supporting facts). The more facts used to support an inference, the greater the chance that one of them is false and that the inference is therefore invalid. If the confidence scores were allowed to be greater than one, then the confidence of inferences might become greater as we moved further out along inference chains.
The system uses a Truth Maintenance System, which has a single argument to mark a belief as in or out. This renders combining evidence from multiple arguments moot. 2.3
In the current implementation, facts that are assumed (for example, the details of the case that is to be reasoned about) have a confidence of 1. Our inference evaluator first tries to determine whether an inference is contradicted or implied; if it fails, it checks whether that inference is from analogy and scores it appropriately, and otherwise, assigns it the default normalizing score. Contradiction and implication are handled using backward chaining from axioms in the knowledge base, using resource bounds.
In our implementation, all inferences are given a confidence score and a reason for that score. The reason is the facts and axioms that were used to generate the score. For implied facts, the score is the product of the confidences of the facts that imply it (because perhaps those antecedents are not trustworthy). Contradicted facts are currently simply rejected, although in future implementations they will be scored based on the likelihood of the facts used to reject them. Confidence scoring for analogical inferences is described above. SME mapping scores can be normalized in three different ways, all of which are on in the interval [0:1]. The base normalized score is a measure of how much of the base case is mapped in the mapping, that is, how much of the base case overlaps with the target case. If the target is much larger than the base but the base is highly alignable with a sub-set of the target, the base normalized score will be quite high even if the match score is low. The target normalized score is the corresponding measure for how much of the target case is mapped, and the normalized score is the average of the base and target normalization scores. The default is the average normal score. Base normalization tends to be used in recognition tasks, where covering the entire base is the criteria, whereas target normalization tends to be used in reasoning tasks, where finding precedents that can lead to inferences within a more complex situation is important.
We tested the confidence scoring and contradiction components of this initial
implementation on two tasks: Analogical Chaining and Moral Reasoning. Analogical
Chaining is a commonsense reasoning technique that elaborates a case description by
repeated analogy to small cases called Common Sense Units (CSUs) [
We examined the performance of the inference evaluation system on 11 COPA
questions, whose internal representations were automatically extracted from the English text
of the question using EA-NLU [
One question asks: “The egg splattered. What was the cause of this?” The answers are “I dropped it” and “I boiled it.” The system first hypothesized that the egg splattering was caused by some unknown violent impact, and assigned that inference a confidence score of 0.01 (low inference scores are discussed below). It then hypothesized, as an alternative explanation for the egg splattering, that the egg hit the floor. This did not involve the abstract impact from the first inference, but was based only on the question facts. However, the mapping had a lower match score than the first, so it was given a confidence of 0.0004. The system then pursued, in separate reasoning contexts, explanations for the first two inferences. In the system’s case library was a case describing how an object was violently impacted when it was hit with a rock, so it hypothesized that perhaps the unknown impact on the egg was caused by a rock. Despite being based on a higher confidence inference (the first inference asserted, where p=0.01), this inference had a low match score and therefore resulted in a score of 2×10-6. Finally, the system used a fourth case to explain the inference that the egg hit the floor by hypothesizing that it was dropped. Despite being based on a lower-confidence fact than the inference about the rock, this inference had a higher match score and thus received a confidence of 2×10-5. While low, this score is still an order of magnitude higher than the dead-end hypothesis about the rock based on more highly-trusted initial inference.
Another question asks: “The truck crashed into the motorcycle on the bridge. What happens as a result?” The answers are “The motorcyclist died” and “The bridge collapsed”. The automatically constructed question representations involve only one statement about a motorcycle (and no motorcyclist) but several statements about the crashing event (who was involved, where it happened, etc.). The system retrieves cases based on what is present in the case, so it began by reasoning about a familiar case involving a vehicle crashing. In that case the vehicle was an airplane, so the system first posited that perhaps the crash in question involved an airplane malfunctioning (p= 8×10-5). The system then retrieved a story about a child falling out of bed and crashing onto the floor. It used this case to posit that the crash was caused when the truck fell out of bed (p = 9×10-5). Building on the airplane inference, it hypothesized that the airplane lost power (p=1×10-6), then that the motorcycle lost power, (0.012), and finally, having exhausted its knowledge of crashes, that the motorcyclist dies (p=2×10-5). In this case the correct inference has a lower confidence score than all but one of the dead-end inferences.
This example illustrates the pitfalls both of Analogical Chaining and of the inference evaluation system. After the system had posited the airplane, it was all too happy to continue reasoning about it, and the match scores were high enough along that reasoning chain (and low enough for the case that gave it the answer) that those erroneous inferences were scored much higher than the one that seems obvious to humans (humans of course have much prior knowledge the system lacks). The system can be led astray and mask the utility of useful inferences if it marks even one incorrect inference as highly probable. Furthermore, it seems wrong to give the system a hard-and-fast rule stating that airplanes cannot be involved in car crashes. Such a situation may be extremely unlikely, which could be recognized by accumulating cases in a generalization about car and motorcycle crashes, but, as Hollywood has shown us, it’s not impossible. This raises an important point about the interplay between analogical reasoning and first-principles reasoning. Analogical learning can provide explicit evidence of what can happen, because analogical generalizations provide structured, relational probabilistic representations of what has happened. But analogical learning only implicitly gathers evidence about what cannot happen. First principles reasoning is better at ruling out the kinds of things that are impossible (e.g. vehicles cannot fall out of beds because they cannot fit in them).
MoralDM is a computational moral reasoning system that makes decisions by
analogy to moral generalizations [
Most previous work on analogical inference validation has been domain specific. For
example, Ouyang and Forbus used first principles reasoning within the physics problem
solving domain to validate candidate inferences produced by SME [
Similar intermediate rankings have been used to evaluate inferences derived by
nonanalogical reasoning systems. Examples include fuzzy logic networks [
BLOG models and MLNs calculate numerical probabilities for inferences. BLOG
models do so by defining a probability distribution over a set of possible worlds
determined by prewritten axioms. A Metropolis-Hastings Markov chain Monte Carlo
approach can then be used to make inferences from the distribution [
The presented inference evaluation technique could be used in other analogical
reasoning systems that score (or could score) the quality of their matches (that is, which
have a measure similar to SME’s match score). For example, inferences in AMBR
([
In HDTP (e.g. [
Even mappings with high unnormalized match scores, indicating a high quality match, may have low normalized match scores, depending on the relative size of the cases and how much information is left out of the mapping. In the current implementation, low confidence scores assigned to analogical inferences were driven largely by low normalized mapping scores. Small cases with little structural overlap should yield low-confidence scores, since the mappings used to generate the inferences are not seen by the system as being particularly reliable, informative mappings (as indicated by the low score). However, while analogical inferences should have lower scores than logically implied inferences, they should not be vanishingly low. One possibility is to use the highest normalized score as a multiplier in calculating inference confidence scores, rather than always using the same mapping score normalizing function. Each function provides different information, but a high score in either indicates that the mapping includes a high degree of overlap from one case to another. Scoring inferences using the highest normalization score will still involve incorporating the score of the match, the score of the justifying target facts, and the probability of the generalization facts.
Given the ubiquity of certain role predicates (objectActedOn, performedBy, etc.)
analogical chaining can make some inferences that, to a human, seem quite silly.
Having the ability to rule out those silly inferences using logical forms of commonsense is
desirable but is not being done in the current implementation. The Cyc knowledge base
contains millions of axioms, but we are currently only using a small subset (the
disjointWith axioms). We plan to explore reasoning techniques that enable us to exploit
more of this knowledge, especially horn clauses and implication statements, for
constraint-checking (e.g. [
Contradictions should perhaps be asserted with a confidence proportional to the scores of the facts contradicting them, rather than suppressed entirely. If facts are seen as relatively likely, then the contradiction is also likely. If contradictions are asserted, they will must signal which facts contradict them, to keep reasoning consistent.
Many analogy inferences involve positing skolem entities. These are entities present in the base and participating in the candidate inference but which are not present in the target. For example, the event in which the egg was impacted in the above example was posited as a skolem variable. Fundamentally, however, these are open variables, and implication can help resolve them. Contradiction works in a similar way, but instead can only rule out resolutions: just because a rule says that a particular individual cannot fill a role does not mean that it says that no one can.
Finally, further testing is needed on a wider range of domains, as well as further empirical testing of the analogical inference confidence scoring. While we have verified the implication and contradiction through disjointness components of the inference evaluation system are functioning properly, these need to be empirically tested. We can thereafter examine accruing and weighting evidence for and against facts. 6
We presented an initial implementation of a system to evaluate analogical inferences, which have no guarantee of being correct. The system can identify certain inferences as being more likely than others, but further evaluation and extension of the system is needed. Nonetheless, this seems to be a promising direction for inference validation and assessment, and points towards a method for resolving skolem variables in analogical inferences. 7
This research was supported by the Socio-Cognitive Architectures for Adaptable Autonomous Systems Program of the Office of Naval Research, N00014-13-1-0470.