Ontology of Evidence Kathryn B. Laskey, David A. Schum, Paulo C. G. Costa Terry Janssen The Volgenau School of Information Technology and Engineering, Lockheed Martin, IS&GS/GSS, George Mason University, Fairfax, VA 22030 USA Herndon, VA 20171 USA [klaskey, dschum, pcosta]@ gmu.edu terry.janssen@lmco.com categories of things that can exist and the relationships they Abstract— Intelligence analysts rely on reports that are subject can bear to one another. In the field of information systems, to many varieties of uncertainty, such as noise in sensors; the term has come to mean the engineering discipline of deception or error by human sources; or cultural constructing computational representations of various domains misunderstanding. To be effective, intelligence analysts must of application. By contrast, epistemology is the study of understand the relationship between reports, the events or situations reported upon, and the hypotheses of interest to which knowledge: how agents come to know about things that exist. those events or situations are evidential. Computerized support The ontologies we construct, the argument goes, should be for intelligence analysts must provide assistance for managing about what is, not what might or might not be, or what agents evidential reasoning. For this purpose, computational can reasonably infer from available evidence. representations are needed for categories and relationships Computational support for intelligence analysts requires the related to evidential reasoning, such as hypotheses, evidence, ability to represent, store, and manipulate evidence, arguments, sources, and credibility. This paper describes some of the entities and relationships that belong in an ontology of hypotheses, and arguments relating evidence to hypotheses. evidence, and makes the case for the fundamental importance of Such representations must be stored in a computational a carefully engineered ontology of evidence to the enterprise of structure, which, for want of a better term, we might call an intelligence analysis. epistemological repository. Let us consider what such an epistemological repository might contain. It would represent Index Terms— Evidence, probabilistic ontologies, intelligence concepts such as hypothesis, evidence, source, and report. It analysis, inferential reasoning, source credibility would contain relationships such as relevance of evidence to hypothesis, or the source-of relationship connecting a source I. INTRODUCTION with a report produced by the source. It would be quite natural E vidential reasoning is fundamental to the practice of intelligence analysis. Much of an intelligence analyst’s time is spent constructing complex chains of argument from to construct the representation using the languages and tools commonly applied in the discipline of ontological engineering. In other words, this epistemological repository would look evidence to conclusion, weighing the force of each argument rather like a domain ontology, where the domain being and the credibility of its component sources, and arriving at represented is epistemology – the field devoted to how we use overall judgments that, while falling short of certainty, provide evidence obtained from the world around us to arrive at useful inputs to decision makers. Reports that give rise to knowledge about the world. The natural person to build this intelligence assessments are characterized by many varieties repository would be someone schooled in constructing such of uncertainty: noise in sensors; deception or error by human representations – that is, an ontological engineer. To call such sources; poor understanding of situation or context. To be a repository an ontology of evidence would hardly seem effective, intelligence analysts must understand the unreasonable. relationship between reports, the events or situations reported In this paper, we argue for the fundamental importance of a upon, and the hypotheses of interest to which those events or carefully engineered ontology of evidence to the enterprise of situations are evidential. intelligence analysis for the need for an ontology of evidence, It follows that effective computerized support for and describe some of the entities and relationships that such an intelligence analysts must support processes of evidential ontology would represent. reasoning. For this purpose, computational representations are needed for categories and relationships related to evidential II. EVIDENCE AND ARGUMENT reasoning, such as hypotheses, evidence, sources, credibility, Schum [1] has written a systematic treatise on evidence and and the like. its role in constructing arguments. All evidence, according to Some have argued that computational representations of Schum, has three major credentials: relevance, credibility, and evidential categories and relationships, while necessary to inferential force or weight. Relevance concerns the degree to intelligence analysis, do not belong in an ontology. Ontology, which the evidence bears upon the hypothesis under the argument goes, is the systematic study of existence: the consideration. Credibility means the degree to which the evidence is believable; whether or not the evidence is Manuscript received November 1, 2008. trustworthy. Inferential force concerns the strength of the systems that allow them to catalog, organize, and explore the relationship between evidence and hypothesis – the degree to implications of large collections of reports and other evidence. which the evidence sways our belief in the hypothesis. Quantitative measures of the strength of evidence are useful as Evidence can come from diverse types of sources (e.g. a way to summarize and communicate the implications of physical sensors, human reports, direct tangible evidence such large bodies of evidence. A natural candidate for such as objects or documents), each with different degrees of summarization, with a long and respected intellectual tradition relevance, levels of credibility, and force. behind it, is probability. Systematic deviations of intuitive As examples of the factors bearing the credibility of a human reasoning from the tenets of probability theory (e.g., source, evidence coming from physical sensors needs to be [3]) have been cited as justification for heuristic approaches to evaluated with respect to environmental conditions, distance combining strength of evidence (e.g., [4]). Nevertheless, from observer, and physical characteristics of the respective naturalistic human reasoning can usefully be treated as a sensor. Human sensors, on the other hand, must be scrutinized computationally bounded approximation to a probabilistic with respect to opportunity, competence, and veridicality. norm (c.f., [5], [6]). There is a robust literature on the use of Opportunity concerns whether the person was in a position to probability and decision theory to support human inference have observed the event or verified the fact. Competence and decision making, and to protect against errors that can concerns whether the source was capable of making the occur in naïve human reasoning (e.g., [7], [8]). Furthermore, distinction in question. Veridicality concerns whether the heuristic techniques introduced as cognitively natural ways to source is telling the truth. Clearly, there may be complex overcome perceived disadvantages of probability theory have chains of inference involved in ascertaining any of these been shown to admit a probabilistic interpretation (e.g., [9]). factors influencing credibility. Approaches for dealing with When the independence conditions justifying the probabilistic the weight or strength of evidence include both qualitative and interpretation are met, such heuristic weighting factors can quantitative aspects of the reasoning process adopted to draw work well, but they can produce disastrous results when inferences from it (e.g. probability theory, logical reasoning, applied without regard to whether these conditions are met. etc). There is no match for probability theory in its generality, A vital (and too often overlooked) distinction to be made is logical coherence, and well-developed methodological base. the difference between an event and evidence that the event For this reason, we focus on probability theory as a logically occurred, or between a fact and evidence that the fact obtains. justified approach to combining numerical measures of Schum uses the notational device of an asterisk to make the evidential force. distinction between event or fact E and evidence E* relating to We provide several examples to illustrate how probability E. It is important to note that E* does not entail E; the can be used to represent and reason about credibility, to inference to E depends on the credibility of the source of E*. combine reports from different sources, and to handle We do not always have the luxury of a direct report E* on subtleties such as dependence relationships that can stymie an event or fact E of interest. We may need to reason naïve heuristic weighting schemes. Our examples are indirectly from a report R* to an event or proposition R whose deliberately kept simple to illustrate the key points. They are truth bears on the truth of E, and from there to E itself. not intended to represent the full complexity of the evidential Collections of interrelated propositions can be chained reasoning problems faced in real applications. Nevertheless, together into complex arguments. We often think of an they illustrate the building blocks from which a more argument as a linear chain from evidence through a collection sophisticated reasoning capability can be constructed. of intermediate conclusions to a final conclusion. However, Figure 1 shows a Bayesian network that illustrates the each link in such a chain must be justified. A judgment must combination of three independent pieces of evidence regarding be made that each antecedent in the chain is relevant to its the whereabouts of Osama bin Laden. Prior to receiving the consequent. The evidential force of each link must also be reports, the probability is 3% that he is in Kandahar. After established. These judgments often require evidential receiving the first report, the chance increases to 11%. After a reasoning in their own right. Schum uses the term ancillary evidence to refer to evidence about the nature and force of an evidential relationship. Intelligence analysts require support for keeping account of chains of argument and the ancillary evidence on which their force depends. III. PROBABILISTIC TREATMENTS OF EVIDENCE The past century has brought broad appreciation of the statistical regularities underlying the seeming complexity of physical, biological, psychological, and societal phenomena [2]. Computational advances are enabling automated and semi-automated support for many “knowledge tasks” once Figure 1: Three Independent Reports Increase Probability of thought to be the exclusive province of human cognition. Hypothesis from 3% to 69% Intelligence analysts increasingly rely upon computerized second report, the probability is 35%; the third report brings the probability to 69%. The figure shows the situation after the third report has been received. The top rectangle represents hypotheses about bin Laden’s location and their probabilities (Kandahar at 69%; Other at 31%). The three reports are shown below the location hypotheses. The gray color indicates that they have been specified as evidence, with 100% probability assigned to the actual reported location. Figure 2 extends this example to explicitly represent report credibility. The figure now shows credibility hypotheses (low, moderate and high) for the three reports. If we had specified no evidence about the a. Sources for Rep1 and Rep2 are Different credibility values, the results would have been the same as Figure 1. But if we specify that the credibility of the third report is low, then the probability decreases to 55% that bin Laden is in Kandahar. That is, lowering the credibility of a report decreases its evidential force, resulting in less change in belief when the report is received. Our final example illustrates an issue not easily accounted for by heuristic methods for assigning and combining evidential weights. Suppose we discover that two of the reports, which we had originally treated as independent, may b. Sources for Rep1 and Rep2 are the Same have actually come from the same informant. We can treat this Figure 3: Common Source Reduces Force of Report case by explicitly representing a hypothesis for whether the reports came from the same source. In Figure 3a, we indicate that the sources of the two reports are different. In this case, disciplinary subject. Practitioners from many disciplines can they can be treated as independent evidence items, and the profit from a formalization of the discipline of evidential resulting belief in bin Laden’s location is the same as in Figure reasoning. Due to its heavy dependence on evidence in almost 1. However, if we specify that the sources are the same every aspect of its operations, the domain of intelligence (Figure 3b), the probability that bin Laden is in Kandahar is analysis would be a prime beneficiary of an ontology of reduced to 35%, the same as if we had received only two evidence. Benefits of an ontology of evidence include a independent reports. The structural assumptions (the common, shared vocabulary for important features and independence relationships represented in the graphs) together relationships that occur across different applications of with the numerical probability values ensure that subtleties evidential reasoning, as well as the ability to share information such as source credibility and common sources are properly among diverse systems. accounted for in evidential reasoning. Despite considerable diversity and individual variation in Additional treatments of probabilistic representations of the conduct of investigation and analysis, there are relevance and credibility in evidential reasoning can be found fundamental common structures and processes. Examples in [10] and [11]. include assessing the credibility and relevance of individual items or of masses of evidence, or constructing reasoning IV. A PROBABILISTIC ONTOLOGY OF EVIDENCE AND chains to connect evidence to hypothesis. A formal representation of evidence and evidential relationships provides the obvious benefit of allowing analysts to query a knowledge base not just for conclusions (e.g., “Where is Osama bin Laden?”), but also for the evidence on which the conclusions are based (e.g., “What is the evidence that bin Laden is in Kandahar?”) Analysts can reason about the relevance of evidence to hypotheses, the credibility of sources, errors that may be common to several evidential reasoning chains, and other subtleties of evidential reasoning. There has been an increasing emphasis in recent years in Figure 2: Low Credibility Reduces Force of Report sharing knowledge among intelligence applications. An ontology of evidence and inferential reasoning is a first step in that direction. Ontologies provide shared representations of INFERENTIAL REASONING the entities and relationships characterizing a domain, into The above concepts pertain to the use of evidence as an which vocabularies of different systems can be mapped so to informational asset and to the inferential process that provide interoperability among them. Techniques for making transforms it into knowledge. This is clearly a multi- semantic information explicit and computationally accessible are key to effective exploitation of evidence from diverse languages were designed to handle. We have argued elsewhere sources, with distinct grades of credibility and relevance. (e.g. [5]) that for domains characterized by uncertainty, Shared formal semantics enables systems with different probabilistic ontologies ([13], [14]) provide a principled internal representations to exchange information, and provides means to represent the structural and numerical aspects of a means to enforce business rules such as access controls for evidential reasoning. Indeed, many researchers have pointed security. out the importance of structural information in probabilistic However, traditional ontologies do not provide a principled models (e.g. [15], [16]), and it is well known that some means to ensure semantic consistency with respect to issues of questions about evidence can be answered entirely in uncertainty related to credibility of sources, relevance of structural terms ([1], page 271). Shafer ([17], pages 5-9) evidence, and other aspects of the evidential reasoning argues that probability is more about structure than it is about process. Because uncertainty is a fundamental aspect of numbers. Numerical probabilities enable quantitative evidential reasoning, this is a serious deficiency. assessment of the force of evidence, which depends on the When faced with the challenge of representing uncertainty strength of relevance and credibility arguments. Exploring the in an ontology, the natural tendency is to introduce a means to details of probabilistic ontologies is not in the scope of this annotate property values with information regarding their level work, but the interested reader is referred to http://www.pr- of confidence. This approach addresses only part of the owl.org. information that needs to be represented in a full ontology of Finally, apart from the advantages of knowledge sharing evidence. To understand why more is needed, consider the tools to the Intelligence Analysis domain, it is important to example from Section II above, in which evidence from foresee the institutional and cultural implications of several sources is combined to draw an inference about the systematizing and standardizing vocabulary and semantics of current location of Osama bin Laden. We saw that the evidential reasoning. The very difficulties an effective inferential force of each report depended not only on that information-sharing scheme is meant to overcome can become report’s credibility, but also on whether the information from obstacles to its widespread adoption. Given the nature of the which it was derived overlapped with the information on field, with highly personalized approaches to analysis, a which another report was derived. In other words, we need to knowledge tool may encounter resistance if it is perceived as represent not just a single credibility number, but information threatening deeply ingrained processes. Yet, the increasing about how that credibility was derived. An assessment from demands within the Intelligence community for effective source x, in order to be used in conjunction with evidence exchange create an opportunity for developing standardized coming from other sources would not only state that (say) representations and approaches. This is an important and "with 75% probability, Osama bin Laden is in Kandahar." To difficult issue. A probabilistic ontology of evidence is a be part of a comprehensive probabilistic model capable of promising first step to provide a structure for knowledge performing sophisticated evidential reasoning, such a sharing that is sufficiently flexible to address the demands of statement would have to include the supporting evidence and the multiple approaches currently used to handle evidential how its credibility affects the overall assessment. A simple reasoning. example would be “with 75% probability, given reports that his physician was spotted in a local market (evidence E1) and V. SUMMARY AND CONCLUSIONS that a radio communication regarding his whereabouts was After identifying some concepts regarding the process of intercepted (evidence E2),” accompanied by information transforming masses of evidence into knowledge, we explored clarifying how this number changes as the credibility of E1 the need for formal representations of evidential processes as a and E2 varies. Further, as new evidence accrues, a means to provide cross-fertilization among domains that sophisticated evidential reasoning system must be capable of depend on processes of evidential reasoning. Among these, capturing the impact of additional evidence on the body of intelligence analysis is paradigmatic. We proposed a evidence being analyzed. As an example, if a source were probabilistic ontology of evidence as a key enabler of this found to be a double agent, the credibilities of all reports from vision. Implementation of this concept must be cognizant of that agent would need to be called into question. A system that institutional and cultural barriers. In conclusion, we argue that relies on or can represent only numerical weights of individual the benefits of effective evidential reasoning and knowledge arguments cannot cope with the complexity and dynamic sharing tools far outpace the difficulties in implementing aspect of real world multi-source evidential reasoning. them. In short, annotating a standard ontology with numerical probabilities is not sufficient, as too much information is lost ACKNOWLEDGEMENTS due to the lack of a good representational scheme that captures The authors gratefully acknowledge helpful comments from structural constraints and dependencies among probabilities. anonymous reviewers of an earlier draft of this paper. Over the past several decades, semantically rich and computationally efficient formalisms have emerged for REFERENCES representing and reasoning with probabilistic knowledge (e.g., [12]). A true probabilistic ontology must be capable of [1] D. A. Schum, Evidential Foundations of Probabilistic Reasoning. New York: John Wiley & Sons, Inc., 1994. properly representing the nuances these more expressive [2] G. Gigerenzer, Z. Swijtink, T. Porter, L. Daston, J. Beatty, and L. Kruger. The Empire of Chance: How Probability Changed Science and Everyday Life. Cambridge, Cambridge University Press, 1990. [3] D. Kahneman, P. Slovic, and A. Tversky, eds., Judgement Under Uncertainty: Heuristics and Biases. Cambridge: Cambridge Univ. Press, 1982. [4] S. Bringsjord, J. Taylor, A. Shilliday, M. Clark and K. Arkoudas. “Slate: An Argument-Centered Intelligent Assistant to Human Reasoners,” in F. Grasso, N. Green, R. Kibble and C. Reed (eds.) Proceedings of the 8th International Workshop on Computational Models of Natural Argument (CMNA 8), Patras, Greece, 2008. http://kryten.mm.rpi.edu/Bringsjord_etal_Slate_cmna_crc_061708.pdf. [5] L. Martignon and K. B. Laskey. “Taming Wilder Demons: Bayesian Benchmarks for Fast and Frugal Heuristics.” In Simple Heuristics that Make us Smart. The ABC Group, Oxford University Press, 1999. [6] J. R. Anderson and M. Matessa. “Explorations of an Incremental, Bayesian Algorithm for Categorization.” Machine Learning 9(4), 275- 308, 1992. [7] D. Von Winterfeld and W. Edwards. Decision Analysis and Behavioral Research. Cambridge, U.K.: University Press, 1986. [8] W. Edwards, R. F. Miles, Jr., and D. von Winterfeldt, Advances in Decision Analysis: From Foundations to Applications. Cambridge, U.K.: University Press, 2007. [9] D.E. Heckerman. “Probabilistic interpretations for MYCIN's certainty factors”. In J. Lemmer and L. Kanal, editors, Uncertainty in Artificial Intelligence Vol 1, pages 167-196, Amsterdam: Elsevier, 1986. [10] S. Mahoney, D. Buede, and J. Tatman. “Patterns of Report Relevance.” Proceedings of the Third Annual Bayesian Modeling Applications Workshop, 2005. http://www.intel.com/research/events/bayesian2005/docs/Mahoney- ReportRelevance.pdf. [11] E. Wright and K. Laskey. “Credibility Models for Multi-Source Fusion.” Proc. 9th International Conf. on Information Fusion, 2006. http://ite.gmu.edu/~klaskey/papers/Wright_Laskey_Credibility.pdf [12] K. B. Laskey, “MEBN: A Language for First-Order Bayesian Knowledge Bases” Artificial Intelligence, 172(2-3), 200 [13] K. B. Laskey, P. C. G. Costa, and T. Jensen, “Probabilistic Ontologies for Knowledge Fusion,” in Proc. 11th International Conf. on Information Fusion, Cologne, Germany, 2008. [14] P. C. G. Costa, “Bayesian Semantics for the Semantic Web,” PhD Dissertation, Dept. of Sys. Eng. and Op. Res., George Mason Univ. 315p, Fairfax, VA, USA, 2005. [15] J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. San Mateo, CA, USA: Morgan Kaufmann Publishers, 1988. [16] J. B. Kadane, and D. A. Schum, A Probabilistic Analysis of the Sacco and Vanzetti Evidence. New York: John Wiley & Sons, 1996. [17] G. Shafer, “Combining AI and OR,” University of Kansas School of Business, Working Paper No. 195, April 1988.