Substance-Blind Classification of Evidence for Intelligence Analysis David Schum, Gheorghe Tecuci, Mihai Boicu and Dorin Marcu  Abstract²Intelligence analysis requires the development of documents, images, and records of any kind, or testimony arguments that link evidence to hypotheses by establishing and from human sources or witnesses. fusing the relevance, believability and inferential force or weight Evidence may have any possible substance or content. of a wide variety of items of evidence of different types. T his pa- Therefore, attempts to categorize it in terms of its substance or per presents several substance-blind classifications of evidence content would be an endless and fruitless task. Why should which are based on these inferential characteristics and facilitate anyone wish to be able to categorize evidence? First, it is often the clarification of many uncertainties lur king in intelligence analysis. It also shows how the Disciple-L T A cognitive assistant necessary to compare the force or weight of different lines of uses these classifications to develop Wigmorean probabilistic argument based on different evidence in a particular analysis. inference networ ks for assessing the likelihood of hypotheses. For example, here is a line of argument based on HUMINT evidence; how does this argument compare with a different Index Terms²evidence classification, relevance, believability, line of argument based on IMINT or one based on MASINT? inferential force, Wigmorean networ ks, cognitive assistant, on- Second, there are different uncertainty issues that arise when tology, evidence-based hypothesis analysis, high-level fusion we have different kinds of evidence. Third, how does the strength of our conclusions in one analysis compare with I. WHY IS A SUBSTANCE-BLIND CLASSIFICATION those reached in another analysis, given the fact that these two OF EVIDENCE NEEDED? analyses are based on entirely different mixtures of evidence? 'Evidence' is word of relation used in the context of argumen- Fourth, how will we ever resolve differences a mong analysts tation: e.g. "A is evidence of B". In that context information themselves, or a mong analysts and their " customers" , regard- has a potential role as relevant evidence if it tends to support ing interpretations of evidence forming the basis for conclu- or tends to negate, directly or indirectly, some hypothesis sions reached in an analysis? Finally, how do we ever say about a contested matter. One draws inferences from evidence anything general about evidence given that its substance or in order to prove or disprove a hypothesis. The framework is content varies in a near infinite fashion? What is badly needed argument, the process is proof, and the engine is inferential in so many situations is an evidence categorization scheme for reasoning from information [1]. Thus evidence differs from allowing us to say what kinds of evidence we have without the words data or items of information, since data or items of resorting to discussions about its substance or content. information only become evidence when their relevance is In this paper we present a foundation for such an evidence established regarding some hypothesis at issue. The term evi- categorization scheme that will tell us what kinds and combi- dence must also be distinguished from the term fact. We may nations of evidence we have in any intelligence analysis re- all agree that it is a fact that we have evidence about event E. gardless of the substance or content of the evidence and the But whether it is a fact that event E did occur is another matter objectives of the analysis. First, we present a general approach since we have questions about the credibility of the source of to evidence-based hypothesis analysis which consists in de- this evidence. This makes it necessary to distinguish between veloping a Wigmorean probabilistic inference network that evidence for an event and the event itself. Evidence can be any shows how evidence is linked to a hypothesis through a poten- species of proof consisting of tangible items such as objects, tially very complex argument that establishes and fuses the relevance, the believability and the inferential force or weight of a wide variety of items of evidence of different types [2, 3]. This work was supported in part by several U.S. Government organiza- tions, including the Air Force Office of Scientific Research (AFOSR, Then we present three substance-blind classifications of evi- FA9550-07-1-0268), the National Science Foundation (NSF, 0750461), and dence, one based on believability, one on relevance and one the Department of Defense (DOD). The US Government is authorized to on inferential force, which support the development of Wig- reproduce and distribute reprints for Governmental purposes notwithstanding morean networks for hypotheses analysis. This approach to any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily hypothesis analysis and the substance-blind classifications of representing the official polices or endorsements, either expressed or implied, evidence are implemented in Disciple-/7$DQDQDO\VW¶VFRg- of AFOSR, NSF, DOD or the U.S. Government. nitive assistant that can learn complex analytic expertise di- Dr. David Schum is Professor in the Volgenau School of Information rectly from expert analysts, can support analysts in hypothesis Technology and Engineering (VS IT&E) and in the School of Law at George Mason University (GMU), Fairfax VA 22030, USA. (phone: 703-993-1694; analysis, collaboration and sharing of intelligence, and can fax: 703-993-9275; e-mail: dschum@gmu.edu). teach its analytic expertise to new analysts [4, 5, 6]. Dr. Gheorghe Tecuci is Professor of Computer Science in the VS IT&E and Director of the GMU Learning Agents Center (e-mail: tecuci@gmu.edu). II. WIGMOREAN NETWORKS Dr. Mihai Boicu is Assistant Professor of Applied IT and Associate Direc- tor of the GMU Learning Agents Center (e-mail: mboicu@gmu.edu). Disciple-LTA assists an analyst in assessing the likelihood Dr. Dorin Marcu is Research Assistant Professor in the GMU Learning of YDULRXV K\SRWKHVHV VXFK DV ³$O 4DHGD KDV QXFOHDU ZHa- Agents Center (dmarcu@gmu.edu). 1 SRQV´RU³The United States will be the world leader in non- Because there is very strong evidence favoring H11 and there is conventional energy sources withiQWKHQH[W\HDU´RU³Iran weak evidence disfavoring H11, Disciple-LTA concludes: is pursuing nuclear power for peaceful purposes´[4]. This is   It  is  almost  certain  that  H11  is  true.   accomplished by developing an argument in the form of an The sub-hypotheses H12 and H13 are assessed in a similar way: Wigmorean inference networks, through the use of a general   It  is  likely  that  H12  is  true.                                    It  is  likely  that  H13  is  true.   problem-reduction/solution-synthesis reasoning approach The solutions of H11, H12 and H13 are composed (e.g., through which is illustrated in Fig. 1 and discussed in the following. ³DYHUDJH´ LQWRWKHHYLGHQFH-based assessment of H1: A complex hypothesis is first reduced to simpler and simp-   It  is  likely  that  H1  is  true.   ler hypotheses and the simplest hypotheses are assessed A concrete example of such a Wigmorean network generated through evidence analysis. For example, in Fig. 1, the hypo- by Disciple-LTA is shown in Fig. 4. thesis H1 (or problem [P1]) is reduced to three simpler hypo- theses, H11, H12, and H13 (problems [P2], [P3] and [P4]). Each III. CLASSIFICATION OF EVIDENCE BASED ON BELIEVABILITY of these hypotheses is assessed by considering both favoring In the previous section we have discussed the process of evidence and disfavoring evidence (i.e., problems [P5] and evidence-based hypothesis assessment down to the level [P6]). Let us assume that there are two items of favoring evi- where one has to assess the relevance and the believability of dence for H11: E1 and E2. For each of them (e.g., E1) Disciple- an item of evidence. In this section we discuss how Disciple- LTA assesses the extent to which it favors the hypothesis H 11 LTA and its user assess the believability of an item of evi- (i.e., [P7]). This requires assessing both the relevance of E1 to dence by using a substance-blind classification of evidence. H11 (problem [P9]) and the believability of E1 (problem [P10]). Here is an important question we are asked to answer re- Let us assume that Disciple-LTA has obtained the following garding the individual kinds of evidence we have: How do solutions for these two last problems: you, the analyst, stand in relation to this item of evidence?   If  we  believe  E1  then  H11  is  almost  certain. Can you examine it for yourself to see what events it might   It  is  likely  that  E1  is  true. reveal? If you can, we say that the evidence is tangible in na- ,Q WKLV H[DPSOH ³DOPRVW FHUWDLQ´ DQG ³OLNHO\´ DUH V\PEROLF ture. But suppose instead you must rely upon other persons, SUREDELOLWLHV IRU OLNHOLKRRG EDVHG RQ WKH '1,¶V VWDQGDUG Hs- assets, or informants, to tell you about events of interest. Their timative language. By compositing the solutions [S9] and reports to you about these events are examples of testimonial [S10@ HJWKURXJKD ³PLQ´ IXQFWLRQ 'LVFLSOH-LTA assesses evidence. Fig. 2 shows a substance-blind classification of evi- the inferential force or weight of E1 on H11: dence based on its believability credentials.   Based  on  E1  it  is  likely  that  H11  is  true.   [S7]   Similarly Disciple-LTA assesses the inferential force or A. Tangible Evidence weight of E2 on H11: There is an assortment of tangible items we might encounter   Based  on  E2  it  is  almost  certain  that  H11  is  true.   [S8]   and that could be examined by an intelligence analyst. Both By composing the solutions [S7] and [S8] (e.g., through a IMINT and SIGINT provide various kinds of sensor records ³PD[´ IXQFWLRQ  'LVFLSOH-LTA assesses the inferential and images that can be examined. MASINT and TECHINT force/weight of the favoring evidence (i.e., E 1 and E2) on H11: provide various objects such as soil samples and weapons that   Based  on  the  favoring  evidence  it  is  almost  certain  that  H11  is  true.   can be examined. COMINT can provide audio recordings of Through a similar process Disciple-LTA assesses the disfavor- communications that can be overheard and translated if the ing evidence for H11: communication has occurred in a foreign language. Docu-   Based  on  the  disfavoring  evidence  it  is  unlikely  that  H11  is  false.   ments, tabled measurements, charts, maps [P1] Assess  H1 It  is  likely that  H  is  true [S1] and diagrams or plans of various kinds are also tangible evidence. [P2] [S2] [P3] [S3] [P4] [S4] There are two different kinds of tangible Assess   It  is  almost  certain   Assess   It  is  likely that   Assess   It  is  likely that   evidence: real tangible evidence and demon- H11 that  H11 is  true H12 H12 is  true H13 H13 is  true strative tangible evidence Real tangible evi- Inferential  force of  evidence  on  H1 dence is a thing itself and has only one major [P5] [S5] [P6] [S6] believability attribute: authenticity. Is this Assess  the   Based  on  the  favoring   Assess  the   Based  on  the  disfavoring   object what it is represented as being or is favoring  evidence   evidence  it  is  almost   disfavoring  evidence   evidence  it  is  unlikely claimed to be? There are as many ways of for  H11 certain that  H11 is  true for  H11 that  H11 is  false generating deceptive and inauthentic evi- Inferential  force dence as there are persons wishing to gener- of  favoring  evidence  on  H1 [P7] [S7] [P8] [S8] ate it. Documents or written communications Assess  the  extent  to   Based  on  E1 it  is   Assess  the  extent  to   Based  on  E2 it  is  almost   may be faked, captured weapons may have which  E1 favors  H11 likely that  H11 is  true which  E2 favors  H11 certain that  H11 is  true been altered, and photographs may have Inferential  force been altered in various ways. One problem is of  E1 on  H1 [P9] [S9] [P10] [S10] that it usually requires considerable expertise Assess  the  relevance If  we  believe  E1 then   Assess  the   It  is  likely that   to detect inauthentic evidence. of  E1 to  H11 H11 is  almost  certain believability of  E1 E1 is  true Demonstrative tangible evidence does not concern things themselves but only represen- Fig. 1. Wigmorean inference network for hypothesis assessment generated by Disciple-LTA. tations or illustrations of these things. Ex- 2 amples include diagrams, maps, scale models, statistical or This involves information about the source's relevant sensory other tabled measurements, and sensor images or records of capabilities and the conditions under which a relevant obser- various sorts such as IMINT, SIGINT, and COMINT. Demon- vation was made . strative tangible evidence has three believability attributes. As indicated in Fig. 2, there are several types of testimonial The first concerns its authenticity. For example, suppose we evidence. If the source does not hedge or equivocate about obtain a hand drawn map from a captured insurgent showing what he/she observed (i.e., the source reports that he/she is the locations of various groups in his insurgency organization. certain that the event did occur), then we have unequivocal Has this map been deliberately contrived to mislead our mili- testimonial evidence. If, however, the source hedges or equi- tary forces or is it a genuine representation of the location of vocate in any way (e.g., "I'm fairly sure that E occurred") then these insurgency groups? we have equivocal testimonial evidence . The first question we The second believability attribute is accuracy of the repre- would ask this source of unequivocal testimonial evidence is: sentation provided by the demonstrative tangible item. The How did you obtain information about what you have just re- accuracy question concerns the extent to which the device that ported? It seems that this source has three possible answers to produced the representation of the real tangible item had a this question. The first answer is: "I made a direct observation degree of sensitivity (resolving power or accuracy) that allows myself. In this case we have unequivocal testimonial evidence us to tell what events were observed. We would be as con- based upon direct observation. The second possible answer is: cerned about the accuracy of the hand-drawn map allegedly "I did not observe this event myself but heard about its occur- showing insurgent groups locations as we would about the rence (or nonoccurrence) from another person". Here we have accuracy of a sensor in detecting traces of some physical oc- a case of secondhand or hearsay evidence, called unequivocal currence. Different sensors have different resolving power that testimonial evidence obtained at second hand . A third answer also depends on various settings of their physical parameters is possible: "I did not observe event E myself nor did I hear (e.g., the settings of a camera). about it from another source. But I did observe events C and D The third major attribute, reliability, is especially relevant and inferred from them that event E definitely occurred". This to various forms of sensors that provide us with many forms of is called testi monial evidence based on opinion and it requires demonstrative tangible evidence. A system, sensor, or test of some very difficult questions. The first concerns the source's any kind is reliable to the extent that the results it provides are credibility as far as his/her observation of event C and D; the repeatable or consistent. You say that a sensing device is reli- second involves our examination of whether we ourselves able if it would provide the same image or report on succes- would infer E based on events C and D. This matter involves sive occasions on which this device is used. our assessment of the source's reasoning ability. It might well be the case that we do not question this source's credibility in B. Testimonial Evidence observing events C and D, but we question the conclusion that For testimonial evidence we have two basic sources of un- event E occurred the source has drawn from his observations. certainty: competence and credibility. This is one reason why We would also question the certainty with which the source it is more appropriate to talk about the believability of testi- KDVUHSRUWHGDQRSLQLRQWKDW(RFFXUUHG'HVSLWHWKHVRXUFH¶V monial evidence which is a broader concept that includes both conclusion thaW³HYHQW(GHILQLWHO\occurred", we should con- competence and credibility considerations. The first question sider that testimonial evidence based on opinion is a type of to ask related to competence is whether this source actually equivocal testimonial evidence . made the observation he claims to have made or had access to There are two other types of equivocal testimonial evidence. the information he reports. The second competence question The first we call completely equivocal testimonial evidence . concerns whether this source understood what was being ob- Asked whether event E occurred or did not, our source says: served well enough to provide us with an intelligible account "I don't know", or "I can't remember". of what was observed. Thus competence involves access and But there is another way a source of HUMINT can equivo- understandability. cate; the source can provide probabilistically equivocal testi- Assessments of human source credibility require considera- monial evidence in various ways: "I'm 60 percent sure that tion of entirely different attributes: veracity (or truthfulness), event E happened"; or "I'm fairly sure that E occurred´; or "It objectivity, and observational sensitivity under the conditions is very unlikely that E occurred". We could look upon this of observation. Here is an account of why these are the major evidence attributes of testimonial credibility. First, is this source telling us about an event he/she believes to have occurred? This tangible   testimonial   missing   authoritative   source would be untruthful if he/she did not believe the re- evidence evidence evidence record ported event actually occurred. So, this question involves the source's veracity. The second question involves the source's real   demonstrative   unequivocal   equivocal   objectivity. The question is: Did this source base a belief on tangible   tangible   testimonial   testimonial   sensory evidence received during an observation, or did this evidence evidence evidence evidence source believe the reported event occurred either because this source expected or wished it to occur? An objective observer unequivocal   unequivocal   testimonial   completely   probabilistically   is one who bases a belief on the basis of sensory evidence in- testimonial  evidence   testimonial  evidence   evidence   equivocal   equivocal   stead of desires or expectations. Finally, if the source did base based  upon  direct   obtained  at  second based  on   testimonial   testimonial   observation hand opinion evidence evidence a belief on sensory evidence, how good was this evidence? Fig. 2. Evidence classification based on believability. 3 particular probabilistic equivocation as an assessment by the al assertion based on other alleged tangible evidence. Thus source of his own observational sensitivity. these forms of evidence are not mutually exclusive; they can occur together in a single item of evidence. C. Missing Evidence To say that evidence is missing entails that we must have F . Believability Assessment with Disciple-LTA had some basis for expecting we could obtain it. There are Disciple-LTA knows about the types of evidence shown in some important sources of uncertainty as far as missing evi- Fig. 2 and how their believability should be evaluated. For dence is concerned. In certain situations missing evidence can example, Fig. 3 shows the reasoning tree automatically gener- itself be evidence. Consider some form of tangible evidence, ated by Disciple-LTA for solving the problem: ³$VVHVV WKH such as a document, that we have been unable to obtain. There extent  to  which  one  can  believe  Osama  bin  Laden  as  the  source  of   are several reasons for our inability to find it, some of which EVD-­Dawn-­Mir01-­F´ Notice that, in accordance with the are more important than others. First, it is possible that this above discussion, Disciple-LTA reduces the believability of tangible item never existed in the first place; our expectation this testimony of Osama bin Laden to two simpler problems, that it existed was wrong. Second, the tangible item exists but one for assessing the competence of Osama bin Laden, and the we have simply been looking in the wrong places for it. Third, other for assessing his credibility. This second problem is fur- the tangible item existed at one time but has been destroyed or WKHUUHGXFHGWRDVVHVVLQJELQ/DGHQ¶VYHUDFLW\REMHFWLYLW\DQG misplaced. Fourth, the tangible item exists but someone is observational sensitivity. keeping it from us. This fourth consideration has some very Disciple-LTA may have knowledge about these believabili- important inferential implications including denial and possi- ty characteristics of Osama bin Laden (e.g., that his veracity is bly deception. An adverse inference can be drawn from some- an even chance). Alternatively, the analyst may make assump- one's failure to produce evidence. tions with respect to the values of these characteristics. In any case, once the solutions of the simplest problems are obtained, D. Accepted F acts they are combined, from bottom up, to assess the believability There is one final category of evidence about which we of Osama bin Laden. For example, the probabilistic estimates would never be obliged to assess its believability. Tabled in- RIELQ/DGHQ¶VYHUDFLW\REMHFWLYLW\DQGREVHUYDWLRQDOVHQVLWLv- formation of various sorts such as tide table, celestial tables, ity (i.e., an even chance, almost certain, and almost certain, tables of physical or mathematical results such as probabilities respectively) are combined (through a min function) to obtain associated with statistical calculations, and many other tables a probabilistic estimate of his credibility (i.e., an even chance). of information we would accept as being believable provided 7KHQELQ /DGHQ¶VFUHGLELOLW\LVDXWRPDWLFDOO\FRPELQHG ZLWK that we used these tables correctly. For example, an analyst his competence (again through a min function), to estimate bin would not be obliged to prove that temperatures in Iraq can be /DGHQ¶VEHOLHYDELOLW\DVWKHVRXUFHRI(9'-Dawn-Mir01-01c. around 120 degrees Fahrenheit in summer months, or that the Disciple-LTA also allows the analysts to assess these belie- population of Baghdad is greater than that of Basra. vability characteristics by developing Wigmorean networks, as E. Mixed Evidence illustrated in Fig. 4 where Disciple-LTA reduces the problem of assessing the veracity of bin Laden to simpler problems, We have just considered a categorization of individual items and then assesses the simplest problems based on the available of evidence but there are situations in which individual items evidence [7]. As one can see, the Wigmorean network in Fig. can reveal various mixtures of these types of evidence. One 4 has the general structure shown in Fig. 1. example involves a tangible document containing a testimoni- EVD-­Dawn-­Mir-­01-­01c Fig. 3. Source believability assessment. 4 Fig. 4. Wigmorean network for veracity assessment. IV. CLASSIFICATION OF EVIDENCE BASED ON RELEVANCE V. CLASSIFICATION OF EVIDENCE BASED ON Here is an important relevance question we are asked to INFERENTIAL FORCE OR WEIGHT answer regarding the individual kinds of evidence we have: Here is an inferential force or weight question we are asked How does this item of evidence stand in relation to what you, to answer regarding the individual kinds of evidence we have: the analyst, are trying to prove or disprove from it? How does this item of evidence changes your belief in the There are two species of relevant evidence. Some evidence truthfulness in what you are trying to assess? If the item of may be directly relevant if you can form a defensible chain of evidence increases our belief in the truthfulness of the hypo- reasoning from this item of evidence to hypotheses you are thesis we are analyzing, we call it favoring evidence . Other- considering. For example, E1 and E2 in Fig. 1 are directly rele- wise, we call it disfavoring evidence . For example, both E1 vant items of evidence. and E2 in Fig.1 are examples of favoring evidence with respect Other evidence may be indirectly relevant, or ancillary evi- to the hypothesis H11. dence if it bears upon the strength or weakness in chains of As shown in Fig. 1, one also has to assess the inferential reasoning set up by directly relevant evidence. Consider, for force of a combination of two or more individual items of evi- H[DPSOH WKH SUREOHP ³Assess   the   believability   of   E1´ IURP WKH dence. These combinations of evidence are also recurrent and bottom right side of Fig. 1. Any item of evidence that might be do not involve the substance or content of the evidence. One used in solving this problem would be indirectly relevant evi- reason for carefully considering these combinations of evi- dence. Indirectly relevant evidence is also any evidence used dence is that they are often confused or incorrectly identified in solving the problem ³$VVHVV WKH   extent   to   which   one   can   be-­ leading to mistakes in how the evidence is described in an lieve  Osama  bin  Laden  as  the  source  of  EVD-­Dawn-­Mir01-­F´ from analysis. But perhaps the most important reason is that there Fig. 3, such as any evidence from the reasoning tree in Fig. 4. are very important sources of uncertainty lurking in these evi- The term meta-evidence is also appropriate since ancillary dential combinations. As shown in Fig. 6, there are three main evidence is evidence about other evidence. Fig. 5 shows this classes of evidence combinations. relevance-based classification of evidence. A. Harmonious Evidence evidence Two or more items of evidence are harmonious if they are directionally consistent in the sense that they all favor the directly  relevant   indirectly  relevant same hypothesis. There are two basic forms of harmonious evidence (ancillary  or  meta)  evidence evidence, corroborative evidence and convergent evidence. In Fig. 5. Evidence classification based on relevance. the case of corroborative evidence we have two or more 5 sources telling us that the same event E has occurred. This dictory evidence in the following way. A contradiction always form of corroboration often allows us to have greater confi- involves whether one event occurred or did not occur. But dence that the event in question did occur. In such cases we divergent evidence involves entirely different events; the di- would say that one source has verified what the other source rectional inconsistency here means that these events point us has told us. The exception involves instances where we have toward different hypotheses. In one case, suppose credible other evidence suggesting that two or more HUMINT sources evidence about event E would favor hypothesis H, but credible collaborated in deciding what to tell us, or that one source evidence about event F would favor hypothesis not-H. influenced or coerced another source to report the same event. C. Evidential Redundance In the case of convergent evidence we have two or more evidence items that concern different events all of which point We often encounter two or more items of evidence in which toward or favor the sa me hypothesis. Convergent evidence can the first item acts to reduce the force of subsequent items of exhibit evidential synergism. In many situations two or more evidence. Stated another way, the first item acts to make sub- evidence items, considered jointly, have greater inferential sequent items redundant to some degree. There are two ways force or weight than they would have if we considered them this can happen. The first form of evidential redundance in- separately or independently. Another way to characterize evi- volves the corroborative evidence we discussed above. In this dential synergism is to say that one item of evidence can have case we have repeated evidence of the same events. Although greater force if we consider it in light of other evidence. having corroborative evidence does add to our confidence that an event of interest did occur, each additional item adds less B. Dissonant Evidence and less to our confidence. We refer to this situation as corro- Dissonant evidence involves combinations of two or more borative redundance . items that are directionally inconsistent ; they can point us in The second form of redundancy involves different events in different inferential directions or toward different hypotheses. which evidence about one event, if credible, takes something There are two basic forms of evidential dissonance; the first off the inferential force of evidence about another event. We involves contradictory evidence . Contradictory evidence al- have called this cumulative redundance . The word "cumula- ways involves events that are mutually exclusive, they cannot tive" is an expression used in law to refer to evidence that does have occurred jointly. From one source we learn that event E not add anything to what we already know. occurred; but from another source we learn that this same It is very important to consider these two forms of eviden- event did not occur. The dissonance seems obvious in this case tial redundancy. In the case of corroborative redundance we since event E cannot have occurred and not have occurred at risk double counting evidence about the same event and as- the same time. Evidential contradictions are always resolved cribing additional weight the evidence does not always have. on credibility grounds. As an example, suppose we have three For cumulative redundance we risk getting more inferential HUMINT sources who tell us that event E occurred, and one mileage out of the evidence than can be justified. HUMINT source who tells us that event E did not occur. In the not so distant past, it was believed that we should always VI. CONCLUSIONS resolve the contradiction by counting heads; i.e. majority We have discussed several substance-blind forms and com- rules. So, on this basis we would side with the three sources binations of evidence, each raising uncertainty issues that can- who tell us that event E did occur. The trouble here is that not be ignored in any intelligence analysis. Disciple-LTA counting heads assumes that all of the four sources involved in knows about several of them and takes them into account for this episode of contradictory evidence have equal credibility. evidence-based hypothesis assessment, but much more work This may be a very bad assumption since, on ancillary evi- remains to be done, especially concerning the inferential force. dence about these four sources, we may well believe that the one source telling us that E did not occur has greater credibili- REFERENCES ty than does the aggregate credibility of the three sources who [1] Twining W., Evidence as a Multi-Disciplinary Subject. Law, Probability tell us that event E did occur. So, what matters in resolving & Risk: A Journal of Reasoning Under Uncertainty. Vol. 2, No. 2, June evidential contradictions is the aggregate credibility of the 2003. sources on either side of this contradiction. [2] Wigmore J.H., The Science of Judicial Proof, Boston: Little, Brown, 1937. There is another form of dissonant evidence called diver- [3] Schum D.A., The Evidential Foundations of Probabilistic Reasoning, gent evidence . This pattern of dissonance differs from contra- Northwestern University Press, 2001. evidence  combination [4] Tecuci G., Boicu M., Marcu D., Boicu C., Barbulescu M., Disciple- LTA: Learning, Tutoring and Analytic Assistance, Journal of Intelli- gence Community Research and Development, 2008. harmonious   redundant   dissonant   evidence evidence evidence [5] 6FKXP'$7HFXFL*%RLFX0³$QDO\]LQJ(YLGHQFHDQGLWV&KDLQ of Custody: A Mixed-,QLWLDWLYH&RPSXWDWLRQDO$SSURDFK´ International Journal of Intelligence and Counterintelligence, 22:2, 2009, pp.298-319. corroborative   convergent   corroborative   cumulative   contradictory   divergent   evidence evidence redundant   redundant   evidence evidence [6] Boicu M., Tecuci G., Schum D., Intelligence Analysis Ontology for evidence evidence Cognitive Assistants, in Proc OIC-08, George Mason University. synergistic   [7] Schum D.A. and Morris J., Assessing the Competence and Credibility of evidence Human Sources of Intelligence Evidence: Contributions from Law and Fig. 6. Recurrent substance-blind combinations of evidence. Probability, Law, Probability, & Risk, Vol. 6, pp. 247±274, 2007. 6