Recognizing and Countering Biases in Intelligence Analysis with TIACRITIS Gheorghe Tecuci, David Schum, Dorin Marcu, Mihai Boicu Learning Agents Center, Volgenau School of Engineering, George Mason University, Fairfax, VA 22030, USA Abstract— This paper discusses different biases which have been learning. Then, in Section III, we address the analysts’ biases identified in Intelligence Analysis and how TIACRITIS, a discussed by Heuer [2, pp.111-171]: biases in the evaluation of knowledge-based cognitive assistant for evidence-based evidence, in the perception of cause and effect, in the hypotheses analysis, can help recognize and partially counter estimation of probabilities, and in the retrospective evaluation them. After reviewing the architecture of TIACRITIS, the paper of intelligence reports. After that we address three other origins shows how it helps recognize and counter many of the analysts’ of bias that are rarely discussed, even though they may be at biases in the evaluation of evidence, in the perception of cause least as important on occasion as any analysts’ biases. and effect, in the estimation of probabilities, and in the retrospective evaluation of intelligence reports. Then the paper II. THE TIACRITIS COGNITIVE ASSISTANT introduces three other types of bias that are rarely discussed, biases of the sources of testimonial evidence, biases in the chain of TIACRITIS is a knowledge-based system that supports an custody of evidence, and biases of the consumers of intelligence, intelligence analyst in performing evidence-based hypothesis which can also be recognized and countered with TIACRITIS. analysis in the framework of the scientific method. It guides the analyst to view intelligence analysis as ceaseless discovery of Bias, cognitive assistant, intelligence analysis, evidence-based evidence, hypotheses, and arguments in a non-stationary world, reasoning, argumentation, symbolic probabilities. involving collaborative processes of evidence in search of hypotheses, hypotheses in search of evidence, and evidentiary I. INTRODUCTION testing of hypotheses [1, 3]. Fig.1 is an abstract illustration of Intelligence analysts face the difficult task of drawing this astonishingly complex process. First we search for possible defensible and persuasive conclusions from masses of hypotheses that would explain a surprising observation E* (see evidence, requiring the development of often stunningly the left side of Fig.1): It is possible that F might be true. complex arguments that establish and defend the three major Therefore G might be true. Therefore H, a hypothesis of high credentials of evidence: relevance, believability, and inferential interest, might be true. The problem with drawing this force [1]. This highly complex task is affected by various conclusion, however, is that there are other hypotheses that also biases which are inclinations or preferences that interfere with explain E*, such as F’, G’, and H’. To conclude H we would impartial judgment. Some of the biases are due to our need to assess all the competing hypotheses, showing that F, G, simplified information processing strategies that lead to and H are more likely than their competitors. consistent and predictable mental errors. These errors remain H H’ ••• H likely compelling even when one is fully aware of their nature, and likely are therefore exceedingly difficult to overcome [2, p.111-112]. min In this paper we propose an approach to the identification G G’ ••• G likely M very likely and countering of the biases in intelligence analysis. The likely no support min approach is based on the observation that the best protection against biases comes from the collaborative effort of teams of N almost Q very S very F F’ ••• certain likely likely analysts, who become skilled in the evidential and very likely likely max argumentational elements of their tasks, and who are willing to share their insights with colleagues, who are also willing to E* En* ••• listen. As we discuss in this paper, this could be achieved by employing an intelligent analytic tool like TIACRITIS [3] Evidence in search Hypotheses in Evidentiary tests of hypotheses search of evidence of hypotheses which helps the analyst perform a rigorous evidence-based hypothesis analysis that makes explicit all the reasoning steps, Fig. 1. Scientific method framework of TIACRITIS. probabilistic assessments, and assumptions, so that they can be critically analyzed and debated. The name TIACRITIS is an Let us assume that we have shown that F and G are more abbreviation of Teaching Intelligence Analysts Critical likely than their corresponding competing hypotheses. Next we Thinking Skills, which was the initial motivation of developing have to assess H, H’, … . To assess H we need additional this system. The system was later extended to also support its evidence which is obtained by successively decomposing H use for regular analysis. into simpler and simpler hypotheses, as shown by the blue tree in the right part of Fig.1. H would be true if G and M would be In the next section we introduce the architecture of the true. Then M would be true if N, Q, and S would be true. But if TIACRITIS cognitive assistant which is based on semantic N would be true, then we would need to observe evidence En*. technologies for knowledge representation, reasoning, and So we look for En* and we may or may not find it. This is the This research was partially supported by the Department of Defense and by George Mason University. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Department of Defense or the U.S. Government. STIDS 2013 Proceedings Page 25 process of hypotheses in search of evidence that guides the contains knowledge applicable to the evidence-based analysis evidence collection task. Now some of the newly discovered of any type of intelligence hypothesis, from any domain. Under items of evidence (e.g. En*) may trigger new hypotheses, or the it, and inheriting from it, are domain-specific knowledge bases. refinement of the current hypotheses. Therefore, as indicated at Each such Domain KB contains knowledge specific to a the bottom part of Fig.1, the processes of evidence in search of particular type of IA problems, such as predictive analysis hypotheses and hypotheses in search of evidence take place at related to energy sources, or assessments related to the current the same time, and in response to one another. production of weapons of mass destruction by various actors. Under each Domain KB there are several Scenario KBs, each Then we use all the collected evidence to assess the corresponding to an instance of a problem pattern from that hypothesis H. This assessment is probabilistic in nature domain, such as, “Assess whether the United States will be a world because the evidence is always incomplete, usually leader in wind power within the next decade.” This particular inconclusive, frequently ambiguous, commonly dissonant, and Scenario KB contains specific knowledge about the United has various degrees of believability [1]. In the computational States, as well as items of evidence to make the corresponding theory of intelligence analysis we have developed [3], analysis. The actual analysis is done by using this knowledge hypotheses assessment is based on a combination of ideas from as well as more general knowledge inherited from the the Baconian probability system [4] and the Fuzzy probability corresponding Domain KB and from the IA KB. system [5], and uses a symbolic probability scale. In particular, in the latest version of TIACRITIS, the likeliness of a Knowledge Base Transactional Access Knowledge Management hypothesis may have one of the following ordered values: Tutoring & Repository Graphical User Interface Testing Management Intelligence Analysis KB no support < likely < very likely < almost certain < certain governing body Scenario Ontology IF the problem to solve is P1g IF the problem to solve is P1g state government group governing body IF the problem to solve is P1g Condition IF the problem to solve is P1g Condition IF the problem to solve is P1g Except-When Condition Condition IF the problem to solve is P1g feudal god king … Except-When Condition In this scale, “no support” means that our evidence does not monarchy dictator deity figure Condition government … Except-When Condition Condition Except-When…Condition Except-WhenCondition Condition Except-When… ConditionExcept-When Condition totalitarian chief and Except-When… ConditionExcept-When Condition Except-When…Condition theocratic autocratic government tribal council government THEN solve its sub-problems Development leader Elicitation Except-When Condition 1 …solve THEN its sub-problems democratic government P1g P11ng solve THEN its sub-problems Except-When Condition P11ng solve P1g … THEN its sub-problems police state military theocratic religious democratic council P11ng solve P1g … THEN its sub-problems dictatorship democracy dictatorship or board P11ng solve P1g … THEN its sub-problems P1g … P11ng P1g … P1 ng support the conclusion that the hypothesis is true. This may, representative parliamentary democracy democracy Ontology Rules Problem Multistrategy however, change if new evidence favoring the hypothesis is Solving Learning later discovered. The likeliness of an upper-level hypothesis Domain KB Domain KB (e.g., H) is obtained from the likeliness of its sub-hypotheses Evidence- Mixed- Based Initiative (i.e., G and M) by using min or max Baconian and Fuzzy Reasoning Interaction combination functions, depending on whether the sub- Scenario KB Scenario KB hypotheses G and M represent necessary and sufficient Asynchronous Message-Based Interaction conditions for the hypothesis H, sufficient conditions, or just Fig. 2. Learning agent shell for intelligence anlaysis. indicators. Competing hypotheses (e.g., H’) are assessed in a similar way and the most likely hypothesis is selected. But if no Each of these knowledge bases is structured into an hypothesis is more likely than all its competitors, then the ontology of concepts and a set of general problem solving rules processes of hypotheses in search of evidence, and evidence in expressed with these concepts. The rules are learned from search of hypotheses have to be resumed. specific examples of reasoning steps, by using the ontology as TIACRITIS was developed by first customizing the a generalization hierarchy [7]. The learning agent shell for Disciple learning agent shell (a general agent building tool [6, intelligence analysis was obtained by training the Disciple 7]) into a learning agent shell for intelligence analysis, and then learning agent shell with general intelligence analysis know- by training it with analysis knowledge from several domains [8]. The overall architecture of the Disciple learning agent shell for intelligence analysis is shown in Fig. 2. It contains integrated modules for ontology development, rule learning, problem solving and evidence-based reasoning, mixed- initiative interaction, and tutoring, as well as a hierarchically organized repository of knowledge bases (KB). At the top level of this repository is the general knowledge base for intelligence analysis (IA KB) which Fig. 3. Ontology fragment showing various types of evidence. STIDS 2013 Proceedings Page 26 ledge resulting in the development of the IA KB. The IA KB The ontology and the rules from the knowledge repository contains both a general ontology and a set of general reasoning of TIACRITIS allow it to support the analyst in formulating rules which are necessary for any Disciple agent for hypotheses, developing arguments that reduce complex intelligence analysis, as we will briefly present in the hypotheses to simpler and simpler ones (as discussed above), following. For example, Fig. 3 shows a general ontology of collecting evidence relevant to the simplest hypotheses, and evidence. It includes both basic types (e.g., testimonial finally assessing the relevance, the believability, and the evidence and tangible evidence), as well as evidence mixtures inferential force of evidence, and the likeliness of the (e.g., testimonial evidence about tangible evidence). The hypotheses. Additionally, TIACRITIS continuously learns ontology language of Disciple is an extension of RDFS [9] from the performed analyses. with additional features to facilitate learning [6, 7, 10]. As discussed in the rest of this paper, TIACRITIS has one Learned general rules from the IA KB include those for additional important capability. It supports the analysts in directly assessing a hypothesis based on evidence. These rules recognizing and countering many of their biases. Because automatically reduce the assessment of a leaf hypothesis, such Heuer has made a detailed and very well-known analysis of as Q in Fig.1, to assessments based on favoring and disfavoring biases in intelligence analysis [2, pp.111-171], we follow his evidence and, further down, to the assessment of the relevance classification and identified characteristic of biases to show and the believability of each item of evidence with respect to how TIACRITIS helps recognizing and countering many of Q. Once these assessments are made, they are combined, from them. bottom-up, to obtain the inferential force of all the items of evidence on Q, which results in the likeliness of Q. III. BIASES OF THE ANALYST An example of a learned rule is shown in Fig. 4. It is an if- A. Biases in the Evaluation of Evidence then problem reduction rule that expresses how and under what Heuer first mentions vividness of evidence as a necessary conditions a generic hypothesis can be reduced to simpler criterion for establishing its force. Analysts, like other persons, generic hypotheses. The conditions are represented as first- have preferences for certain kinds of evidence and these order logical expressions [7]. In particular, this rule states that, preferences can induce biases. In particular, analysts can have a in order to assess the believability of unequivocal testimonial distinct preference for vivid or concrete evidence when less evidence obtained at second hand, one needs to assess both the vivid or concrete evidence may be more inferentially valuable. believability of our source, and the believability of the source In addition, their personal observations may be over-valued. of our source. It is by the application of such rules that an agent can generate the reduction part of the trees in Fig.1 and Fig.5. First, as discussed in the previous section, the hypothesis in search of evidence phase of the analysis helps identify a wide range of evidentiary needs. For example, the argumentation in Fig. 1 shows that we need evidence relevant to N, evidence relevant to Q, evidence relevant to S, etc. It is unlikely that we would have vivid evidence for each basic hypothesis. So we would be forced to use less vivid evidence as well. Second, as illustrated by the abstract analysis example in Fig. 5 and discussed in the following, TIACRITIS guides us to assess a simple hypothesis Q by performing a uniform, detailed, and systematic evaluation of each item of evidence, regardless of its “vividness”, helping us be more objective in the evaluation of the force of evidence. Let us first consider how to assess the probability of Q based only on one item of favoring evidence Ek* (see the bottom of Fig. 5). First notice that we call this likeliness of Q, and not likelihood, because in classic probability theory likelihood is P(Ek*|Q), while here we are interested in P(Q|Ek*), the posterior probability of Q given Ek*. With TIACRITIS, to assess Q based only on Ek*, we have three judgments to make by answering three questions: The relevance question is: How likely is Q, based only on Ek* and assuming that Ek* is true? If Ek* favors Q, then our answer should be one of the values from “likely” to “certain.” If Ek* is not relevant to Q then our answer should be “no support” because Ek* provides no support for the truthfulness of Q. If, however, Ek* disfavors Q, then it favors the negation (or complement) of Q, and it should be moved under Qc. The believability question is: How likely is it that Ek* is Fig. 4. Learned rule for believability analysis. true? Here the answer should be one of the values from “no STIDS 2013 Proceedings Page 27 support” to “certain.” “Certain” means that we are sure that the on evidence. This argumentation structure makes very clear event Ek reported in Ek* did indeed happen. “No support” that S is not supported by any evidence. Thus the analyst means that Ek* provides us no reason to believe that the event should lower her confidence in the final conclusion, countering Ek reported in Ek* did happen. For example, we believe that the absence of evidence bias. the source of Ek* has lied to us. The next source of bias mentioned by Heuer is a related The inferential force question is: How likely is Q based one: oversensitivity to evidence consistency, and not enough only on Ek*? TIACRITIS automatically computes this answer concern about the amount of evidence we have. This kind of as the minimum of the relevance and believability answers. bias can easily manifest when using an analytic tool like Indeed, to believe that Q is true based only on Ek*, Ek* should Heuer’s ACH [11] where the analyst judges alternative hypo- be both relevant to Q and believable. theses based on evidence, without building any argumentation. With TIACRITIS, the argumentation will reveal if most of the Q very likely evidence is only relevant to a small fraction of sub-hypotheses, on balance while many other sub-hypotheses have no evidentiary support. For example, the argumentation from Fig. 1 shows that most of Inferential force of evidence on Q the evidence is related to hypothesis Q. Q based only on almost Qc based only on According to Heuer [2, pp. 121-122]: “When working with likely favoring evidence certain disfavoring evidence a small but consistent body of evidence, analysts need to max consider how representative that evidence is of the total body Inferential force of favoring evidence on Q of potentially available information.” The argumentation from Q based almost Q based Fig. 1 makes very clear that the available evidence is not likely representative of all the potentially available information. We only on Ei* certain only on Ek* min have no evidence relevant to S. If we would later find such Inferential force of Ek* on Q evidence which would indicate “no support” for S, then the How likely is Q based only on Ek*? considered argumentation would provide “no support” for the top-level hypothesis H. When faced with sub-hypotheses for Relevance of Ek to Q likely Believability of Ek* very likely which there is no evidence (e.g., S in Fig. 1), TIACRITIS How likely is Q, based only on Ek* How likely is it that Ek* is true? allows the analyst to consider various what-if scenarios, and assuming that Ek* is true? making alternative assumptions with respect to the likeliness of S, and determining their influence on the likeliness of H. This Fig. 5. The relevance, believability, and inferential force of evidence. should inform the analyst on how to adjust her confidence in When we assess a hypothesis Q we may have several items the analytic conclusion, to counter the oversensitivity to of evidence, some favoring it and some disfavoring it. The evidence consistency bias. favoring evidence is used to assess the likeliness of Q and the Finally, Heuer lists the persistence of impressions based on disfavoring evidence to assess the likeliness of Qc. Because discredited evidence as an origin of bias. If Heuer had written disfavoring evidence for Q is favoring evidence for Qc, the his book in 2003, he might have used the case of Curveball as a assessment process for Qc is similar to the assessment for Q. very good example [12]. In this case, Curveball’s evidence was When we have several items of favoring evidence, we discredited on a number of grounds but was still believed and evaluate Q based on each of them (as was explained above), taken seriously by some analysts as well as many others. and then we compose the obtained results. This is illustrated in TIACRITIS helps countering this bias by incorporating in Fig.5 where the assessment of Q based only on Ei* (almost the argumentation an explicit analysis of the believability of certain) is composed with the assessment of Q based only on evidence, especially for key evidence that has a direct influence Ek* (likely), through the maximum function, to obtain the on the analytic conclusion. When such an evidence item is assessment of Q based only on favoring evidence (almost discredited, specific elements of its analysis are updated, and certain). In this case the use of the maximum function is this leads to the automatic updating of the likeliness of each justified because it is enough to have one item of evidence that hypothesis to which it is relevant. For example, as shown in the is both very relevant and very believable to make us believe left hand side of Fig. 6, the believability of the observations that the hypothesis is true. performed by a source (such as Curveball) depends on source’s Let us now assume that Qc based only on disfavoring competence and credibility. Moreover, competence depends on evidence is “likely.” How should we combine this with the access and understandability. Credibility depends on veracity, assessment of Q based only on favoring evidence? As shown at objectivity, and observational sensitivity under the conditions the top of Fig.5, TIACRITIS uses an on balance judgment: of observation. Thus, the bias that would result from the Because Q is “almost certain” and Qc is “likely,” it concludes persistence of impressions based on discredited evidence is that, based on all available evidence, Q is “very likely.” countered in TIACRITIS with a rigorous, detailed and explicit believability analysis. Heuer also mentions the absence of evidence as another origin of bias. The bias here concerns a failure to consider the But there are additional biases in the evaluation of evidence degree of completeness of available evidence. Consider again that Heuer does not mention, particularly with respect to the argumentation from Fig. 1 which decomposes complex establishing the credentials of evidence: relevance, hypotheses into simpler sub-hypotheses that are assessed based believability, and inferential force or weight. An analyst may STIDS 2013 Proceedings Page 28 confuse the competence of a HUMINT source with his/her Heuer assumes is the conventional view of probability which credibility. Or, the analyst may focus on the veracity of the might be best called the Kolmogorov view of probability since source and ignore source’s objectivity and observational the Russian mathematician was the first one to put this view of sensitivity. Analysts may fail to recognize possible synergisms probability on an axiomatic basis [13, 14]. This is also the only in convergent evidence, as happened in the 9/11/2001 disaster. view of probability considerd by Heuer’s sources of inspiration Analysts may even overlook evidence having significant on biases: Daniel Kahneman, Amos Tversky, and their many inferential force. colleagues in psychology [15, 16]. In his writings, Kolmogorov makes it abundantly clear that his axioms apply only to Believability of Ek* Source S Believability of Ei* instances in which we can determine probabilities by counting. min min But Heuer also notes that intelligence analysis usually deals Competence of S Credibility of S with one-of-a-kind situations for which there are never any Authenticity Reliability min of Ei* of Ei* statistics. In such cases, analysts resort to subjective or personal min Observational numerical probability expressions. He discusses several reasons Access of S Veracity of S sensitivity of S Accuracy why verbal assessments of probability are frequently criticized Understandability Objectivity of Ei* for their ambiguity and misunderstanding. In his discussion he of S of S recalls Sherman Kent’s advice that verbal assessments should Fig. 6. Believability of testimonial and tangible evidence. always be accompanied by numerical probabilities [17]. B. Biases in the Perception of Cause and Effect Since Heuer only considers numerical probabilities conforming to the Kolmogorov axioms, any biases associated As noted by Heuer, analysts seek explanations for the with them (e.g., using the availability rule, the anchoring occurrence of events and phenomena. These explanations strategy, expressions of uncertainty, assessing the probability involve assessments of causes and effects. But biases arise of a scenario) are either irrelevant or not directly applicable to a when analysts assign causal relations to those that are actually type of analysis that is based on different probability systems, accidental or random in nature. One related consequence is that such as the one performed with TIACRITIS, which is based on analysts often overestimate their ability to predict future events the Baconian and Fuzzy probability systems. Indeed, analysts from past events, because there is no causal association using TIACRITIS never assess any numerical probabilities. between them. One major reason for these biases is that analysts may not have the requisite level of understanding of Heuer [2, p.122] mentions coping with evidence of the kinds and amount of information necessary to infer a uncertain accuracy as an origin of bias: “The human mind has dependable causal relationship. difficulty coping with complicated probabilistic relationships, so people tend to employ simple rules of thumb that reduce the According to Heuer, when feasible, the “increased use of burden of processing such information. In processing scientific procedures in political, economic, and strategic information of uncertain accuracy or reliability, analysts tend to research is much to be encouraged”, to counter these biases [2, make a simple yes or no decision. If they reject the evidence, p.128]. Because TIACRITIS makes all the judgments explicit, they tend to reject it fully, so it plays no further role in their they can be examined by other analysts to determine whether mental calculations. If they accept the evidence, they tend to they contain any mistakes or are incomplete. Because different accept it wholly, ignoring the probabilistic nature of the people have different biases, comparing and debating analyses accuracy or reliability judgment.” He then further notes [2, of the same hypothesis made by different analysts can also help p.123]: “Analysts must consider many items of evidence with identify individual biases. Finally, as a learning system, different degrees of accuracy and reliability that are related in TIACRITIS can acquire correct reasoning patterns from expert complex ways with varying degrees of probability to several analysts which can then be used to analyze similar hypotheses. potential outcomes. Clearly, one cannot make neat Now, here is something that can occur in any analysis mathematical calculations that take all of these probabilistic concerning chains of reasoning. It is always possible that an relationships into account. In making intuitive judgments, we analyst’s judgment will be termed biased or fallacious, on unconsciously seek shortcuts for sorting through this maze, and structural grounds if it is observed that this analyst frequently these shortcuts involve some degree of ignoring the uncertainty leaves out important links in his/her chains of reasoning. This inherent in less-than-perfectly-reliable information. There is actually a common occurrence since, in fact, there is no such seems to be little an analyst can do about this, short of breaking thing as a uniquely correct or perfect argument. Someone can the analytical problem down in a way that permits assigning always find alternative arguments to the same hypothesis; what probabilities to individual items of information, and then using this says is that there may be entirely different inferential routes a mathematical formula to integrate these separate probability to the same hypothesis. Another possibility is that someone judgments.” may find arguments based on the same evidence that lead to First, as discussed in the previous section, concerning the different hypotheses. This is precisely why there are trials at believability of evidence, there is more than just its accuracy to law; the prosecution and defense will find different arguments, consider. Second, as discussed above, Heuer only considers the and tell different stories, from the same body of evidence. conventional view of probability which, indeed, involves C. Biases in Estimating Probabilities complex probability computations. With TIACRITIS, the There are different views among probabilists on how to analyst does precisely what Heuer imagined that could be done assess the force of evidence [1]. The view of probability that for countering this bias. It breaks a hypothesis into simpler hypotheses (see Fig.1), and assesses the simpler hypotheses STIDS 2013 Proceedings Page 29 based on evidence (see Fig.5). Also, TIACRITIS allows the their justifications, and what was the actual logic of our analyst to express probabilities in words rather than numbers, analytic conclusion. We can now add additional evidence and and to employ simple min/max strategies for assessing the use our hindsight knowledge to restructure the argumentation probability of interim and final hypotheses that do not involve and re-evaluate our hypotheses, and we can compare the any full-scale and precise Bayesian or other methods that hindsight analysis with the foresight one. But we will not would require very large numbers of probability assessments. confuse them. As indicated by Heuer [2, pp.166-167]: “A fundamental question posed in any postmortem investigation of There are many places to begin a defense of verbal or fuzzy intelligence failure is this: Given the information that was probability statements. The most obvious one is law. All of the available at the time, should analysts have been able to foresee forensic standards of proof are given verbally: “beyond what was going to happen? Unbiased evaluation of intelligence reasonable doubt”; “clear and convincing evidence”, “balance performance depends upon the ability to provide an unbiased of probabilities”; “sufficient evidence”, and “probable cause’. answer to this question.” We suggest that this may be Over the centuries attempts have been made to supply accomplished with a system like TIACRTIS. numerical probability values and ranges for each of these standards, but none of them have been successful. The reason, IV. SOME FREQUENTLY OVERLOOKED ORIGINS OF BIAS of course, is that every case is unique and rests upon many subjective and imprecise judgments. Wigmore [18] understood So much of the discussion of bias in intelligence analysis is completely that the catenated inferences in his Wigmorean directed at intelligence analysts themselves. But we have networks were probabilistic in nature. Each of the arrows in the identified three other origins of bias that are rarely discussed, chain of reasoning describe the force of one hypothesis on the even though they may be at least as important on occasion as next one, e.g., E Æ F. Wigmore graded the force of such any analysts’ alleged biases. The three other origins of bias we linkages verbally using such terms as “strong force”, “weak will consider are: (1) persons who provide testimonial evidence force”, “provisional force”, etc. Toulmin [19] also used fuzzy about events of interest (i.e. HUMINT sources); (2) other qualifiers in the probability statements of his system which intelligence professionals having varying capabilities who grounds Rationale [20]. There are many other examples of serve as links in what we term “chains of custody” linking the situations in which it is difficult or impossible for people to evidence itself, as well as it’s sources, with the users of find numerical equivalents for verbal probabilities they assess. evidence (i.e. the analysts); and (3) the “consumers” of Intelligence analysis so often supplies very good examples in intelligence analyses (government and military officials who spite of what Sherman Kent said some years ago. make policy and decisions regarding national security). We conclude this discussion by recalling what the well- A. HUMINT Sources known probabilist Professor Glenn Shafer said years ago [21]: Our concern here is with persons who supply us with Probability is more about structuring arguments than it is testimonial evidence consisting of reports of events about about numbers. All probabilities rest upon arguments. If the matters of interest to us. Heuer [2, p.122] does mention the arguments are faulty, the probabilities however determined, “bias on the part of the ultimate source,” but he does not will make no sense. In TIACRITIS, the structure of the bottom- analyze it. In our work on evidence in a variety of contexts, we up argument is given by the logical top-down decomposition, have always been concerned about establishing the and the conclusions are hedged by employing rigorous believability of its sources, particularly when they are human Baconian operations with fuzzy qualifiers, leading to a witnesses, sources, or informants [1]. In doing so, we have defensible and persuasive argument. made use of the 600 year-old legacy of experience and scholarship in the Anglo-American adversarial trial system D. Hindsight Biases in Evaluating Intelligence Reporting concerning witness believability assessments. We have As Heuer notes, analysts often overestimate the accuracy of identified the three major attributes of the credibility of their past judgments; customers often underestimate how much ordinary witnesses: veracity, objectivity, and observational they have learned from an intelligence report; and persons who sensitivity (see Fig. 6). We will show how there are distinct and conduct post-mortem analysis of an intelligence failure will important possible biases associated with each such judge that events were more readily foreseeable than was in believability attribute. These biases are recognized in the fact the case. “The analyst, consumer, and overseer evaluating MACE system (Method for Assessing the Credibility of analytical performance all have one thing in common. They are Evidence), developed for the IC [22]. This system incorporates exercising hindsight. They take their current state of knowledge both Baconian and Bayesian methods for combining evidence and compare it with what they or others did or could or should about our source. have known before the current knowledge was received. This is in sharp contrast with intelligence estimation, which is an As discussed above, assessing the credibility of a human exercise in foresight, and it is the difference between these two source S involves assessing S’s veracity, objectivity, and obser- modes of thought—hindsight and foresight—that seems to be a vational sensitivity. We have to consider that source S can be source of bias. … After a view has been restructured to biased concerning any of these attributes. On veracity, S might assimilate the new information, there is virtually no way to prefer to tell us that event E occurred, whether S believed E accurately reconstruct the pre-existing mental set.” [2, p.162] occurred or not. As an example, an analyst evaluating S’s evidence E* might have evidence about S suggesting that S Apparently Heuer did not envision the use of a system like would tell us that E occurred because S wishes to be the bearer TIACRITIS that keeps track of the performed analysis, what of what S believes we will regard as good news that event E evidence we had, what assumptions we made and what were occurred. On objectivity, S might choose to believe that E STIDS 2013 Proceedings Page 30 occurred because it would somehow be in S’s best interests if E Heuer [2, p.122] mentions the “distortion in the reporting did occur. On observational sensitivity, there are various ways chain from subsource through source, case officer, reports that S’s senses could be biased in favor of recording event E; officer, to analyst” but he does not analyze it. In criminal cases clever forms of deception supply examples. in law, there are persons identified as “evidence custodians”, who keep careful track of who discovered an item of evidence, These three species of bias possible for HUMINT sources who then had access to it and for how long, and what if must be considered by analysts attempting to assess the anything they did to the evidence when they had access to it. credibility of source S and how much weight or force S’s evidence E* should have in the analyst’s inference about These chains of custody add three major additional sources whether or not event E did happen. The existence of any of of uncertainty for intelligence analysts to consider, that are these three biases would have an effect on an analyst’s associated with the persons in chains of custody whose assessment of the weight or force of S’s report E*. As we competence and credibility need to be considered. The first and know, all assessments of the credibility of evidence rest upon most important question involves authenticity: Is the evidence available evidence about its sources. In the case of HUMINT received by an analyst exactly what the initial evidence said we need ancillary evidence about the veracity, objectivity, and and is it complete? The other questions involve assessing the observational sensitivity of its sources. In the process, we have reliability and accuracy of the processes used to produce the to see whether any such evidence reveals any of the three evidence if it is tangible in nature (see the right side of Fig. 6), biases just considered. TIACRITIS supports the analyst in this or also used to take various actions on the evidence in a chain determination by guiding her to answer specific questions of custody, whether the evidence is tangible or testimonial. As based on ancillary evidence. For instance, the veracity an illustration, consider an item of testimonial HUMINT questions considered are shown in Table 1. coming from a foreign national whose code name is Table 1. Questions concerning the veracity of human sources. “Wallflower”, who does not speak English [23]. Wallflower gives his report to case officer Bob. This report is recorded by 1. Goals of this source? Does what this source tells us support any of his Bob and then translated by Husam. Then, Wallflower’s or her goals? translated report is transmitted to a report’s officer Marsha who 2. Present influences on this source? Could this source have been edits it and transmits it to the analyst Clyde who evaluates it influenced in any way to provide us with this report? and assesses its weight or force. 3. Exploitation potential? Is this source subject to any significant exploi- tation by other persons or organizations to provide us this information? Now, here is where forms of bias can enter that can be 4. Any contradictory or divergent evidence? Is there any evidence that associated with the persons involved in these chains of custody. contradicts or conflicts with what the source has reported to us? The case officer Bob might have intentionally overlooked 5. Any corroborative or confirming evidence? Is there any other evidence details in his recording of Wallflower’s report. The translator that corroborates or confirms this source's report? Husam may have intentionally altered or deleted parts of this 6. Veracity concerning collateral details? Are there any contradictions or report. The report’s officer Marsha might have altered or conflicts in the collateral details provided by this source that reflect the possibility of this source's dishonesty? deleted parts of the translated report of Wallflower’s testimony 7. Source's character? What evidence do we have about this source's in her editing of it. The result of these actions is that the analyst character and honesty that bears upon this source's veracity? Clyde receiving this evidence almost certainly did not receive 8. Reporting record? What does the record show about the truthfulness of an authentic and complete account of it, nor did he receive a this source's previous reports to us? good account of its reliability and accuracy. What he received 9. Source expectations about us? Is there any evidence that this source was the transmitted, edited, translated, recorded testimony of may be reporting events he/she believes we will wish to hear or see? Wallflower. Fig. 7 shows how TIACRITIS may determine the 10. Interview behavior? If this source reported these events to us, what believability of the evidence received by the analyst. Although was this source's demeanor and bearing while giving us this report? the information to make such an analysis may not be available, the analyst should adjust the confidence in his conclusion, in B. Persons in Chains of Custody of Evidence recognition of these biases. Unfortunately, there are other persons, apart from Believability of transmitted, edited, tran- HUMINT sources, whose possible biases need to be carefully slated, recorded testimony of Wallflower considered. We know that analysts make use of an enormous min Believability of variety of evidence that is not testimonial or HUMINT, but is Believability of transmission by tangible in nature. Examples include objects, images, sensor edited translation Marsha records of various sorts, documents, maps, diagrams, charts, min Believability of Believability of and tabled information of various kinds. translated recording editing by Marsha But the intelligence analysts only rarely have immediate min Believability of Believability of and first access to HUMINT assets or informants. They may recorded testimony translation by Husam only rarely be the first ones to encounter an item of tangible min evidence. What happens is that there are several persons who Believability Believability of have access to evidence between the times the evidence is first of Wallflower recording by Bob acquired and when the analysts first receive it. These persons Fig. 7. Chain of custody of Wallflower’s testimony. may do a variety of different things to the initial evidence during the time they have access to it. In law, these persons C. Consumers of Intelligence Analyses constitute what is termed a “chain of custody” for evidence. The policy-making consumers or customers of intelligence STIDS 2013 Proceedings Page 31 analysts are also subject to a variety of inferential and structured analytic methods, in the debate on how to decisional biases that may influence the reported analytic significantly improve intelligence analysis [26]. conclusions. As is well known, the relationships between intelligence analysts and governmental policy makers are much REFERENCES discussed and involve considerable controversy [24, 25]. On [1] Schum D.A. (2001). The Evidential Foundations of Probabilistic the one hand we hear intelligence professionals say that they do Reasoning, Northwestern University Press. not make policies but only try to help policy makers be as [2] Heuer R.J. (1999). Psychology of Intelligence Analysis, Center for the informed as they can be when they do form policies and make Study of Intelligence, Central Intelligence Agency, Washington, DC. decisions in the nation’s best interests. But we also learn facts [3] Tecuci, G., Marcu, D., Boicu, M., Schum, D.A., Russell K. (2011). Computational Theory and Cognitive Assistant for Intelligence Analysis, about the intelligence process that complicate matters. An in Proc. 6th Int. Conf. on Semantic Technologies for Intelligence, intelligence analysis is usually a hierarchical process involving Defense, and Security, pp. 68-75, Fairfax, VA, 16-18 November. many intelligence officers, at various grade levels, who become [4] Cohen L.J. (1977). The Probable and the Provable, Clarendon Press, involved in producing an intelligence “product”. At the most Oxford. basic level of this hierarchy are the so-called “desk analysts” [5] Zadeh L. (1983). The Role of Fuzzy Logic in the Management of Uncer- who are known and respected experts in the specific subject tainty in Expert Systems, Fuzzy Sets and Systems, vol.11, pp.199-227. matter of the analysis at hand. An analysis produced by one or [6] Tecuci, G. (1998). Building Intelligent Agents: An Apprenticeship more desk analysts is then passed “upward” through many Multistrategy Learning Theory, Methodology, Tool and Case Studies, administrative levels, at each of which persons at these higher San Diego: Academic Press. levels can comment on the desk analysts’ report. It is often [7] Tecuci, G., Boicu, M., Boicu, C., Marcu, D., Stanescu, B., Barbulescu, M. (2005). The Disciple-RKF Learning and Reasoning Agent, recognized that the higher an editor is in this hierarchy, the Computational Intelligence, Vol.21, No.4, pp. 462-479. more political his/her views and actions become that may affect [8] Tecuci, G., Boicu, M., Marcu, D., Schum, D. (2013). How Learning the content and conclusions of the analysis at hand. As this Enables Intelligence Analysts to Rapidly Develop Practical Cognitive “upward” process continues, the analysis that results may be Assistants, in Proc. 12th International Conference on Machine Learning quite different from the one produced by the desk analysts, and Applications (ICMLA'13), Miami, Florida, December 4-7. reflecting the biases of those who have successively edited it. [9] W3C (2004). http://www.w3.org/TR/rdf-schema/, accessed 10/11/13. In some cases, these editing biases are the direct result of the [10] Boicu M., Tecuci G., Schum D. (2008). Intelligence Analysis Ontology consumer’s biases who may wish to receive a certain analytic for Cognitive Assistants, in Proc. of Conf. “Ontology for the Intelligence conclusion. Using a system like TIACRITIS that shows very Community”, Fairfax, VA, 3-4 December. clearly how the analytic conclusion is rooted in evidence would [11] Heuer R.J. (2008). Computer-Aided Analysis of Competing Hypotheses, in George R.Z., Bruce J.B., eds., Analyzing Intelligence: Origins, significantly help in reducing the above biases. Obstacles, and Innovations, Georgetown Univ. Press, Washington, DC. V. CONCLUSIONS [12] Drogin B. (2007). CURVEBALL: Spies, Lies, and the Con Man Who Caused a War. Random House, New York, NY. A wide variety of biases affect the correctness of [13] Kolmogorov A.N. (1933). Foundations of a Theory of Probability intelligence analyses. In this paper we have shown how the use (1933), 2nd English edition, Chelsea Publishing, New York, NY., 1956 of TIACRITIS, a knowledge-based cognitive assistant, helps [14] Kolmogorov A.N. (1969). The Theory of Probability. In: Aleksandrov, analysts recognize and counter many of them. TIACRITIS A. D., Kolmogorov, A. N., Lavrentiev, M. A. (eds) Mathematics: Its integrates several semantic technologies (knowledge Content, Methods, and Meaning. MIT Press, Cambridge, MA. representation through ontologies and rules, evidence-based [15] Kahneman, D., Tversky, A. (1974). Judgment under Uncertainty: Heuristics and Biases. Science, Vol, 185, 1124-1131. reasoning, machine learning and knowledge acquisition). It [16] Kahneman, D., Slovic, P., Tversky, A. (1982). Judgment under can run in a browser as a web-based system, or it can be Uncertainty: Heuristics and Biases. Cambridge University Press. installed locally, and has been used in many civilian, military, [17] Kent S. (1994). Words of Estimated Probability, in Steury D.P., ed., and intelligence organizations. Sherman Kent and the Board of National Estimates: Collected Essays, Center for the Study of Intelligence, CIA, Washington, DC. There are two complementary ways by which TIACRITIS helps mitigate biases. First, as a cognitive assistant, it helps [18] Wigmore J.H. (1937). The Science of Judicial Proof. Boston, MA: Little, Brown & Co. automate many parts of the analysis process, making this task [19] Toulmin S.E. (1963). The Uses of Argument. Cambridge Univ. Press. much easier for the analyst. Thus it alleviates one of the main [20] van Gelder T.J. (2007). The Rationale for Rationale, Law, Probability causes of biases, which is the employment of simplified and Risk, 6, pp. 23-42. information processing strategies on the part of the analyst. [21] Shafer G. (1988). Combining AI and OR. University of Kansas School of Second, TIACRITIS performs a rigorous evidence-based Business Working Paper No. 195. hypothesis analysis that makes explicit all the reasoning steps, [22] Schum D.A., and Morris J. (2007). Assessing the Competence and evidence, probabilistic assessments, and assumptions, so that Credibility of Human Sources of Evidence: Contributions from Law and they can be critically analyzed and debated. Indeed, the best Probability, Law, Probability and Risk, Vol. 6, pp. 247-274. protection against biases comes from the collaborative effort of [23] Schum D.A., Tecuci G, and Boicu M. (2009). Analyzing Evidence and teams of analysts, who become skilled in solving their analytic its Chain of Custody: A Mixed-Initiative Computational Approach, Int. tasks through the development of sound evidence-based Journal of Intelligence and Counterintelligence, Vol. 22, pp. 298-319. arguments, and who are willing to share their insights with [24] George, R., Bruce, J. (eds) (2008). Analyzing Intelligence: Origins, Obstacles, and Innovations. Georgetown Univ. Press, Washingon. DC. colleagues, who are also willing to listen. TIACRITIS makes [25] Johnston, R. (2005). Analytic Culture in the U.S. Intelligence all this possible. Community. Central Intelligence Agency, Washington, DC. Finally, this paper adds a strong argument in favor of using [26] Marrin, S. (2011). Improving Intelligence Analysis: Bridging the gap between scholarship and practice. Routlege, Lundon and New York. STIDS 2013 Proceedings Page 32