=Paper=
{{Paper
|id=Vol-1097/STIDS2013_T04
|storemode=property
|title=Recognizing and Countering Biases in Intelligence Analysis with TIACRITIS
|pdfUrl=https://ceur-ws.org/Vol-1097/STIDS2013_T04_TecuciEtAl.pdf
|volume=Vol-1097
|dblpUrl=https://dblp.org/rec/conf/stids/TecuciSMB13
}}
==Recognizing and Countering Biases in Intelligence Analysis with TIACRITIS ==
https://ceur-ws.org/Vol-1097/STIDS2013_T04_TecuciEtAl.pdf
Recognizing and Countering Biases
in Intelligence Analysis with TIACRITIS
Gheorghe Tecuci, David Schum, Dorin Marcu, Mihai Boicu
Learning Agents Center, Volgenau School of Engineering, George Mason University, Fairfax, VA 22030, USA
Abstract— This paper discusses different biases which have been learning. Then, in Section III, we address the analysts’ biases
identified in Intelligence Analysis and how TIACRITIS, a discussed by Heuer [2, pp.111-171]: biases in the evaluation of
knowledge-based cognitive assistant for evidence-based evidence, in the perception of cause and effect, in the
hypotheses analysis, can help recognize and partially counter estimation of probabilities, and in the retrospective evaluation
them. After reviewing the architecture of TIACRITIS, the paper of intelligence reports. After that we address three other origins
shows how it helps recognize and counter many of the analysts’ of bias that are rarely discussed, even though they may be at
biases in the evaluation of evidence, in the perception of cause least as important on occasion as any analysts’ biases.
and effect, in the estimation of probabilities, and in the
retrospective evaluation of intelligence reports. Then the paper II. THE TIACRITIS COGNITIVE ASSISTANT
introduces three other types of bias that are rarely discussed,
biases of the sources of testimonial evidence, biases in the chain of TIACRITIS is a knowledge-based system that supports an
custody of evidence, and biases of the consumers of intelligence, intelligence analyst in performing evidence-based hypothesis
which can also be recognized and countered with TIACRITIS. analysis in the framework of the scientific method. It guides the
analyst to view intelligence analysis as ceaseless discovery of
Bias, cognitive assistant, intelligence analysis, evidence-based evidence, hypotheses, and arguments in a non-stationary world,
reasoning, argumentation, symbolic probabilities. involving collaborative processes of evidence in search of
hypotheses, hypotheses in search of evidence, and evidentiary
I. INTRODUCTION testing of hypotheses [1, 3]. Fig.1 is an abstract illustration of
Intelligence analysts face the difficult task of drawing this astonishingly complex process. First we search for possible
defensible and persuasive conclusions from masses of hypotheses that would explain a surprising observation E* (see
evidence, requiring the development of often stunningly the left side of Fig.1): It is possible that F might be true.
complex arguments that establish and defend the three major Therefore G might be true. Therefore H, a hypothesis of high
credentials of evidence: relevance, believability, and inferential interest, might be true. The problem with drawing this
force [1]. This highly complex task is affected by various conclusion, however, is that there are other hypotheses that also
biases which are inclinations or preferences that interfere with explain E*, such as F’, G’, and H’. To conclude H we would
impartial judgment. Some of the biases are due to our need to assess all the competing hypotheses, showing that F, G,
simplified information processing strategies that lead to and H are more likely than their competitors.
consistent and predictable mental errors. These errors remain
H H’ ••• H likely
compelling even when one is fully aware of their nature, and likely
are therefore exceedingly difficult to overcome [2, p.111-112]. min
In this paper we propose an approach to the identification G G’ ••• G likely M very likely
and countering of the biases in intelligence analysis. The likely no
support min
approach is based on the observation that the best protection
against biases comes from the collaborative effort of teams of N almost Q very S very
F F’ •••
certain likely likely
analysts, who become skilled in the evidential and very likely
likely max
argumentational elements of their tasks, and who are willing to
share their insights with colleagues, who are also willing to E* En* •••
listen. As we discuss in this paper, this could be achieved by
employing an intelligent analytic tool like TIACRITIS [3] Evidence in search Hypotheses in Evidentiary tests
of hypotheses search of evidence of hypotheses
which helps the analyst perform a rigorous evidence-based
hypothesis analysis that makes explicit all the reasoning steps, Fig. 1. Scientific method framework of TIACRITIS.
probabilistic assessments, and assumptions, so that they can be
critically analyzed and debated. The name TIACRITIS is an Let us assume that we have shown that F and G are more
abbreviation of Teaching Intelligence Analysts Critical likely than their corresponding competing hypotheses. Next we
Thinking Skills, which was the initial motivation of developing have to assess H, H’, … . To assess H we need additional
this system. The system was later extended to also support its evidence which is obtained by successively decomposing H
use for regular analysis. into simpler and simpler hypotheses, as shown by the blue tree
in the right part of Fig.1. H would be true if G and M would be
In the next section we introduce the architecture of the true. Then M would be true if N, Q, and S would be true. But if
TIACRITIS cognitive assistant which is based on semantic N would be true, then we would need to observe evidence En*.
technologies for knowledge representation, reasoning, and So we look for En* and we may or may not find it. This is the
This research was partially supported by the Department of Defense and by George Mason University. The views and conclusions contained in this document are
those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the Department
of Defense or the U.S. Government.
STIDS 2013 Proceedings Page 25
process of hypotheses in search of evidence that guides the contains knowledge applicable to the evidence-based analysis
evidence collection task. Now some of the newly discovered of any type of intelligence hypothesis, from any domain. Under
items of evidence (e.g. En*) may trigger new hypotheses, or the it, and inheriting from it, are domain-specific knowledge bases.
refinement of the current hypotheses. Therefore, as indicated at Each such Domain KB contains knowledge specific to a
the bottom part of Fig.1, the processes of evidence in search of particular type of IA problems, such as predictive analysis
hypotheses and hypotheses in search of evidence take place at related to energy sources, or assessments related to the current
the same time, and in response to one another. production of weapons of mass destruction by various actors.
Under each Domain KB there are several Scenario KBs, each
Then we use all the collected evidence to assess the corresponding to an instance of a problem pattern from that
hypothesis H. This assessment is probabilistic in nature domain, such as, “Assess whether the United States will be a world
because the evidence is always incomplete, usually leader in wind power within the next decade.” This particular
inconclusive, frequently ambiguous, commonly dissonant, and Scenario KB contains specific knowledge about the United
has various degrees of believability [1]. In the computational States, as well as items of evidence to make the corresponding
theory of intelligence analysis we have developed [3], analysis. The actual analysis is done by using this knowledge
hypotheses assessment is based on a combination of ideas from as well as more general knowledge inherited from the
the Baconian probability system [4] and the Fuzzy probability corresponding Domain KB and from the IA KB.
system [5], and uses a symbolic probability scale. In particular,
in the latest version of TIACRITIS, the likeliness of a Knowledge Base Transactional Access
Knowledge Management
hypothesis may have one of the following ordered values: Tutoring & Repository
Graphical User Interface
Testing Management Intelligence Analysis KB
no support < likely < very likely < almost certain < certain governing body
Scenario Ontology
IF the problem to solve is P1g
IF the problem to solve is P1g
state government group governing body IF the problem to solve is P1g
Condition IF the problem to solve is P1g
Condition IF the problem to solve is P1g
Except-When Condition
Condition IF the problem to solve is P1g
feudal god king … Except-When Condition
In this scale, “no support” means that our evidence does not
monarchy dictator deity figure Condition
government
… Except-When Condition
Condition
Except-When…Condition Except-WhenCondition
Condition
Except-When… ConditionExcept-When Condition
totalitarian chief and Except-When… ConditionExcept-When Condition
Except-When…Condition
theocratic autocratic
government tribal council
government
THEN solve its sub-problems
Development
leader
Elicitation
Except-When Condition
1 …solve
THEN its sub-problems
democratic
government P1g P11ng solve
THEN its sub-problems Except-When Condition
P11ng solve
P1g … THEN its sub-problems
police state
military theocratic religious democratic council P11ng solve
P1g … THEN its sub-problems
dictatorship democracy dictatorship or board P11ng solve
P1g … THEN its sub-problems
P1g … P11ng
P1g … P1 ng
support the conclusion that the hypothesis is true. This may,
representative parliamentary
democracy democracy
Ontology Rules
Problem Multistrategy
however, change if new evidence favoring the hypothesis is Solving Learning
later discovered. The likeliness of an upper-level hypothesis Domain KB Domain KB
(e.g., H) is obtained from the likeliness of its sub-hypotheses Evidence- Mixed-
Based Initiative
(i.e., G and M) by using min or max Baconian and Fuzzy Reasoning Interaction
combination functions, depending on whether the sub- Scenario KB Scenario KB
hypotheses G and M represent necessary and sufficient Asynchronous Message-Based Interaction
conditions for the hypothesis H, sufficient conditions, or just
Fig. 2. Learning agent shell for intelligence anlaysis.
indicators. Competing hypotheses (e.g., H’) are assessed in a
similar way and the most likely hypothesis is selected. But if no Each of these knowledge bases is structured into an
hypothesis is more likely than all its competitors, then the ontology of concepts and a set of general problem solving rules
processes of hypotheses in search of evidence, and evidence in expressed with these concepts. The rules are learned from
search of hypotheses have to be resumed. specific examples of reasoning steps, by using the ontology as
TIACRITIS was developed by first customizing the a generalization hierarchy [7]. The learning agent shell for
Disciple learning agent shell (a general agent building tool [6, intelligence analysis was obtained by training the Disciple
7]) into a learning agent shell for intelligence analysis, and then learning agent shell with general intelligence analysis know-
by training it with analysis
knowledge from several
domains [8]. The overall
architecture of the
Disciple learning agent
shell for intelligence
analysis is shown in Fig.
2. It contains integrated
modules for ontology
development, rule
learning, problem solving
and evidence-based
reasoning, mixed-
initiative interaction, and
tutoring, as well as a
hierarchically organized
repository of knowledge
bases (KB). At the top
level of this repository is
the general knowledge
base for intelligence
analysis (IA KB) which Fig. 3. Ontology fragment showing various types of evidence.
STIDS 2013 Proceedings Page 26
ledge resulting in the development of the IA KB. The IA KB The ontology and the rules from the knowledge repository
contains both a general ontology and a set of general reasoning of TIACRITIS allow it to support the analyst in formulating
rules which are necessary for any Disciple agent for hypotheses, developing arguments that reduce complex
intelligence analysis, as we will briefly present in the hypotheses to simpler and simpler ones (as discussed above),
following. For example, Fig. 3 shows a general ontology of collecting evidence relevant to the simplest hypotheses, and
evidence. It includes both basic types (e.g., testimonial finally assessing the relevance, the believability, and the
evidence and tangible evidence), as well as evidence mixtures inferential force of evidence, and the likeliness of the
(e.g., testimonial evidence about tangible evidence). The hypotheses. Additionally, TIACRITIS continuously learns
ontology language of Disciple is an extension of RDFS [9] from the performed analyses.
with additional features to facilitate learning [6, 7, 10].
As discussed in the rest of this paper, TIACRITIS has one
Learned general rules from the IA KB include those for additional important capability. It supports the analysts in
directly assessing a hypothesis based on evidence. These rules recognizing and countering many of their biases. Because
automatically reduce the assessment of a leaf hypothesis, such Heuer has made a detailed and very well-known analysis of
as Q in Fig.1, to assessments based on favoring and disfavoring biases in intelligence analysis [2, pp.111-171], we follow his
evidence and, further down, to the assessment of the relevance classification and identified characteristic of biases to show
and the believability of each item of evidence with respect to how TIACRITIS helps recognizing and countering many of
Q. Once these assessments are made, they are combined, from them.
bottom-up, to obtain the inferential force of all the items of
evidence on Q, which results in the likeliness of Q. III. BIASES OF THE ANALYST
An example of a learned rule is shown in Fig. 4. It is an if- A. Biases in the Evaluation of Evidence
then problem reduction rule that expresses how and under what Heuer first mentions vividness of evidence as a necessary
conditions a generic hypothesis can be reduced to simpler criterion for establishing its force. Analysts, like other persons,
generic hypotheses. The conditions are represented as first- have preferences for certain kinds of evidence and these
order logical expressions [7]. In particular, this rule states that, preferences can induce biases. In particular, analysts can have a
in order to assess the believability of unequivocal testimonial distinct preference for vivid or concrete evidence when less
evidence obtained at second hand, one needs to assess both the vivid or concrete evidence may be more inferentially valuable.
believability of our source, and the believability of the source In addition, their personal observations may be over-valued.
of our source. It is by the application of such rules that an agent
can generate the reduction part of the trees in Fig.1 and Fig.5. First, as discussed in the previous section, the hypothesis in
search of evidence phase of the analysis helps identify a wide
range of evidentiary needs. For example, the argumentation in
Fig. 1 shows that we need evidence relevant to N, evidence
relevant to Q, evidence relevant to S, etc. It is unlikely that we
would have vivid evidence for each basic hypothesis. So we
would be forced to use less vivid evidence as well.
Second, as illustrated by the abstract analysis example in
Fig. 5 and discussed in the following, TIACRITIS guides us to
assess a simple hypothesis Q by performing a uniform,
detailed, and systematic evaluation of each item of evidence,
regardless of its “vividness”, helping us be more objective in
the evaluation of the force of evidence.
Let us first consider how to assess the probability of Q
based only on one item of favoring evidence Ek* (see the
bottom of Fig. 5). First notice that we call this likeliness of Q,
and not likelihood, because in classic probability theory
likelihood is P(Ek*|Q), while here we are interested in
P(Q|Ek*), the posterior probability of Q given Ek*. With
TIACRITIS, to assess Q based only on Ek*, we have three
judgments to make by answering three questions:
The relevance question is: How likely is Q, based only on
Ek* and assuming that Ek* is true? If Ek* favors Q, then our
answer should be one of the values from “likely” to “certain.”
If Ek* is not relevant to Q then our answer should be “no
support” because Ek* provides no support for the truthfulness
of Q. If, however, Ek* disfavors Q, then it favors the negation
(or complement) of Q, and it should be moved under Qc.
The believability question is: How likely is it that Ek* is
Fig. 4. Learned rule for believability analysis. true? Here the answer should be one of the values from “no
STIDS 2013 Proceedings Page 27
support” to “certain.” “Certain” means that we are sure that the on evidence. This argumentation structure makes very clear
event Ek reported in Ek* did indeed happen. “No support” that S is not supported by any evidence. Thus the analyst
means that Ek* provides us no reason to believe that the event should lower her confidence in the final conclusion, countering
Ek reported in Ek* did happen. For example, we believe that the absence of evidence bias.
the source of Ek* has lied to us.
The next source of bias mentioned by Heuer is a related
The inferential force question is: How likely is Q based one: oversensitivity to evidence consistency, and not enough
only on Ek*? TIACRITIS automatically computes this answer concern about the amount of evidence we have. This kind of
as the minimum of the relevance and believability answers. bias can easily manifest when using an analytic tool like
Indeed, to believe that Q is true based only on Ek*, Ek* should Heuer’s ACH [11] where the analyst judges alternative hypo-
be both relevant to Q and believable. theses based on evidence, without building any argumentation.
With TIACRITIS, the argumentation will reveal if most of the
Q very likely evidence is only relevant to a small fraction of sub-hypotheses,
on balance
while many other sub-hypotheses have no evidentiary support.
For example, the argumentation from Fig. 1 shows that most of
Inferential force of evidence on Q
the evidence is related to hypothesis Q.
Q based only on almost Qc based only on According to Heuer [2, pp. 121-122]: “When working with
likely
favoring evidence certain disfavoring evidence
a small but consistent body of evidence, analysts need to
max consider how representative that evidence is of the total body
Inferential force of favoring evidence on Q of potentially available information.” The argumentation from
Q based almost Q based
Fig. 1 makes very clear that the available evidence is not
likely representative of all the potentially available information. We
only on Ei* certain only on Ek*
min have no evidence relevant to S. If we would later find such
Inferential force of Ek* on Q
evidence which would indicate “no support” for S, then the
How likely is Q based only on Ek*?
considered argumentation would provide “no support” for the
top-level hypothesis H. When faced with sub-hypotheses for
Relevance of Ek to Q likely Believability of Ek* very likely which there is no evidence (e.g., S in Fig. 1), TIACRITIS
How likely is Q, based only on Ek* How likely is it that Ek* is true? allows the analyst to consider various what-if scenarios,
and assuming that Ek* is true? making alternative assumptions with respect to the likeliness of
S, and determining their influence on the likeliness of H. This
Fig. 5. The relevance, believability, and inferential force of evidence. should inform the analyst on how to adjust her confidence in
When we assess a hypothesis Q we may have several items the analytic conclusion, to counter the oversensitivity to
of evidence, some favoring it and some disfavoring it. The evidence consistency bias.
favoring evidence is used to assess the likeliness of Q and the Finally, Heuer lists the persistence of impressions based on
disfavoring evidence to assess the likeliness of Qc. Because discredited evidence as an origin of bias. If Heuer had written
disfavoring evidence for Q is favoring evidence for Qc, the his book in 2003, he might have used the case of Curveball as a
assessment process for Qc is similar to the assessment for Q. very good example [12]. In this case, Curveball’s evidence was
When we have several items of favoring evidence, we discredited on a number of grounds but was still believed and
evaluate Q based on each of them (as was explained above), taken seriously by some analysts as well as many others.
and then we compose the obtained results. This is illustrated in TIACRITIS helps countering this bias by incorporating in
Fig.5 where the assessment of Q based only on Ei* (almost the argumentation an explicit analysis of the believability of
certain) is composed with the assessment of Q based only on evidence, especially for key evidence that has a direct influence
Ek* (likely), through the maximum function, to obtain the on the analytic conclusion. When such an evidence item is
assessment of Q based only on favoring evidence (almost discredited, specific elements of its analysis are updated, and
certain). In this case the use of the maximum function is this leads to the automatic updating of the likeliness of each
justified because it is enough to have one item of evidence that hypothesis to which it is relevant. For example, as shown in the
is both very relevant and very believable to make us believe left hand side of Fig. 6, the believability of the observations
that the hypothesis is true. performed by a source (such as Curveball) depends on source’s
Let us now assume that Qc based only on disfavoring competence and credibility. Moreover, competence depends on
evidence is “likely.” How should we combine this with the access and understandability. Credibility depends on veracity,
assessment of Q based only on favoring evidence? As shown at objectivity, and observational sensitivity under the conditions
the top of Fig.5, TIACRITIS uses an on balance judgment: of observation. Thus, the bias that would result from the
Because Q is “almost certain” and Qc is “likely,” it concludes persistence of impressions based on discredited evidence is
that, based on all available evidence, Q is “very likely.” countered in TIACRITIS with a rigorous, detailed and explicit
believability analysis.
Heuer also mentions the absence of evidence as another
origin of bias. The bias here concerns a failure to consider the But there are additional biases in the evaluation of evidence
degree of completeness of available evidence. Consider again that Heuer does not mention, particularly with respect to
the argumentation from Fig. 1 which decomposes complex establishing the credentials of evidence: relevance,
hypotheses into simpler sub-hypotheses that are assessed based believability, and inferential force or weight. An analyst may
STIDS 2013 Proceedings Page 28
confuse the competence of a HUMINT source with his/her Heuer assumes is the conventional view of probability which
credibility. Or, the analyst may focus on the veracity of the might be best called the Kolmogorov view of probability since
source and ignore source’s objectivity and observational the Russian mathematician was the first one to put this view of
sensitivity. Analysts may fail to recognize possible synergisms probability on an axiomatic basis [13, 14]. This is also the only
in convergent evidence, as happened in the 9/11/2001 disaster. view of probability considerd by Heuer’s sources of inspiration
Analysts may even overlook evidence having significant on biases: Daniel Kahneman, Amos Tversky, and their many
inferential force. colleagues in psychology [15, 16]. In his writings, Kolmogorov
makes it abundantly clear that his axioms apply only to
Believability of Ek* Source S Believability of Ei* instances in which we can determine probabilities by counting.
min min But Heuer also notes that intelligence analysis usually deals
Competence of S Credibility of S with one-of-a-kind situations for which there are never any
Authenticity Reliability
min of Ei* of Ei*
statistics. In such cases, analysts resort to subjective or personal
min
Observational numerical probability expressions. He discusses several reasons
Access of S Veracity of S
sensitivity of S Accuracy why verbal assessments of probability are frequently criticized
Understandability Objectivity of Ei* for their ambiguity and misunderstanding. In his discussion he
of S of S recalls Sherman Kent’s advice that verbal assessments should
Fig. 6. Believability of testimonial and tangible evidence. always be accompanied by numerical probabilities [17].
B. Biases in the Perception of Cause and Effect Since Heuer only considers numerical probabilities
conforming to the Kolmogorov axioms, any biases associated
As noted by Heuer, analysts seek explanations for the
with them (e.g., using the availability rule, the anchoring
occurrence of events and phenomena. These explanations
strategy, expressions of uncertainty, assessing the probability
involve assessments of causes and effects. But biases arise
of a scenario) are either irrelevant or not directly applicable to a
when analysts assign causal relations to those that are actually
type of analysis that is based on different probability systems,
accidental or random in nature. One related consequence is that
such as the one performed with TIACRITIS, which is based on
analysts often overestimate their ability to predict future events
the Baconian and Fuzzy probability systems. Indeed, analysts
from past events, because there is no causal association
using TIACRITIS never assess any numerical probabilities.
between them. One major reason for these biases is that
analysts may not have the requisite level of understanding of Heuer [2, p.122] mentions coping with evidence of
the kinds and amount of information necessary to infer a uncertain accuracy as an origin of bias: “The human mind has
dependable causal relationship. difficulty coping with complicated probabilistic relationships,
so people tend to employ simple rules of thumb that reduce the
According to Heuer, when feasible, the “increased use of
burden of processing such information. In processing
scientific procedures in political, economic, and strategic
information of uncertain accuracy or reliability, analysts tend to
research is much to be encouraged”, to counter these biases [2,
make a simple yes or no decision. If they reject the evidence,
p.128]. Because TIACRITIS makes all the judgments explicit,
they tend to reject it fully, so it plays no further role in their
they can be examined by other analysts to determine whether
mental calculations. If they accept the evidence, they tend to
they contain any mistakes or are incomplete. Because different
accept it wholly, ignoring the probabilistic nature of the
people have different biases, comparing and debating analyses
accuracy or reliability judgment.” He then further notes [2,
of the same hypothesis made by different analysts can also help
p.123]: “Analysts must consider many items of evidence with
identify individual biases. Finally, as a learning system,
different degrees of accuracy and reliability that are related in
TIACRITIS can acquire correct reasoning patterns from expert
complex ways with varying degrees of probability to several
analysts which can then be used to analyze similar hypotheses.
potential outcomes. Clearly, one cannot make neat
Now, here is something that can occur in any analysis mathematical calculations that take all of these probabilistic
concerning chains of reasoning. It is always possible that an relationships into account. In making intuitive judgments, we
analyst’s judgment will be termed biased or fallacious, on unconsciously seek shortcuts for sorting through this maze, and
structural grounds if it is observed that this analyst frequently these shortcuts involve some degree of ignoring the uncertainty
leaves out important links in his/her chains of reasoning. This inherent in less-than-perfectly-reliable information. There
is actually a common occurrence since, in fact, there is no such seems to be little an analyst can do about this, short of breaking
thing as a uniquely correct or perfect argument. Someone can the analytical problem down in a way that permits assigning
always find alternative arguments to the same hypothesis; what probabilities to individual items of information, and then using
this says is that there may be entirely different inferential routes a mathematical formula to integrate these separate probability
to the same hypothesis. Another possibility is that someone judgments.”
may find arguments based on the same evidence that lead to
First, as discussed in the previous section, concerning the
different hypotheses. This is precisely why there are trials at
believability of evidence, there is more than just its accuracy to
law; the prosecution and defense will find different arguments,
consider. Second, as discussed above, Heuer only considers the
and tell different stories, from the same body of evidence.
conventional view of probability which, indeed, involves
C. Biases in Estimating Probabilities complex probability computations. With TIACRITIS, the
There are different views among probabilists on how to analyst does precisely what Heuer imagined that could be done
assess the force of evidence [1]. The view of probability that for countering this bias. It breaks a hypothesis into simpler
hypotheses (see Fig.1), and assesses the simpler hypotheses
STIDS 2013 Proceedings Page 29
based on evidence (see Fig.5). Also, TIACRITIS allows the their justifications, and what was the actual logic of our
analyst to express probabilities in words rather than numbers, analytic conclusion. We can now add additional evidence and
and to employ simple min/max strategies for assessing the use our hindsight knowledge to restructure the argumentation
probability of interim and final hypotheses that do not involve and re-evaluate our hypotheses, and we can compare the
any full-scale and precise Bayesian or other methods that hindsight analysis with the foresight one. But we will not
would require very large numbers of probability assessments. confuse them. As indicated by Heuer [2, pp.166-167]: “A
fundamental question posed in any postmortem investigation of
There are many places to begin a defense of verbal or fuzzy intelligence failure is this: Given the information that was
probability statements. The most obvious one is law. All of the available at the time, should analysts have been able to foresee
forensic standards of proof are given verbally: “beyond what was going to happen? Unbiased evaluation of intelligence
reasonable doubt”; “clear and convincing evidence”, “balance performance depends upon the ability to provide an unbiased
of probabilities”; “sufficient evidence”, and “probable cause’. answer to this question.” We suggest that this may be
Over the centuries attempts have been made to supply accomplished with a system like TIACRTIS.
numerical probability values and ranges for each of these
standards, but none of them have been successful. The reason, IV. SOME FREQUENTLY OVERLOOKED ORIGINS OF BIAS
of course, is that every case is unique and rests upon many
subjective and imprecise judgments. Wigmore [18] understood So much of the discussion of bias in intelligence analysis is
completely that the catenated inferences in his Wigmorean directed at intelligence analysts themselves. But we have
networks were probabilistic in nature. Each of the arrows in the identified three other origins of bias that are rarely discussed,
chain of reasoning describe the force of one hypothesis on the even though they may be at least as important on occasion as
next one, e.g., E Æ F. Wigmore graded the force of such any analysts’ alleged biases. The three other origins of bias we
linkages verbally using such terms as “strong force”, “weak will consider are: (1) persons who provide testimonial evidence
force”, “provisional force”, etc. Toulmin [19] also used fuzzy about events of interest (i.e. HUMINT sources); (2) other
qualifiers in the probability statements of his system which intelligence professionals having varying capabilities who
grounds Rationale [20]. There are many other examples of serve as links in what we term “chains of custody” linking the
situations in which it is difficult or impossible for people to evidence itself, as well as it’s sources, with the users of
find numerical equivalents for verbal probabilities they assess. evidence (i.e. the analysts); and (3) the “consumers” of
Intelligence analysis so often supplies very good examples in intelligence analyses (government and military officials who
spite of what Sherman Kent said some years ago. make policy and decisions regarding national security).
We conclude this discussion by recalling what the well- A. HUMINT Sources
known probabilist Professor Glenn Shafer said years ago [21]: Our concern here is with persons who supply us with
Probability is more about structuring arguments than it is testimonial evidence consisting of reports of events about
about numbers. All probabilities rest upon arguments. If the matters of interest to us. Heuer [2, p.122] does mention the
arguments are faulty, the probabilities however determined, “bias on the part of the ultimate source,” but he does not
will make no sense. In TIACRITIS, the structure of the bottom- analyze it. In our work on evidence in a variety of contexts, we
up argument is given by the logical top-down decomposition, have always been concerned about establishing the
and the conclusions are hedged by employing rigorous believability of its sources, particularly when they are human
Baconian operations with fuzzy qualifiers, leading to a witnesses, sources, or informants [1]. In doing so, we have
defensible and persuasive argument. made use of the 600 year-old legacy of experience and
scholarship in the Anglo-American adversarial trial system
D. Hindsight Biases in Evaluating Intelligence Reporting concerning witness believability assessments. We have
As Heuer notes, analysts often overestimate the accuracy of identified the three major attributes of the credibility of
their past judgments; customers often underestimate how much ordinary witnesses: veracity, objectivity, and observational
they have learned from an intelligence report; and persons who sensitivity (see Fig. 6). We will show how there are distinct and
conduct post-mortem analysis of an intelligence failure will important possible biases associated with each such
judge that events were more readily foreseeable than was in believability attribute. These biases are recognized in the
fact the case. “The analyst, consumer, and overseer evaluating MACE system (Method for Assessing the Credibility of
analytical performance all have one thing in common. They are Evidence), developed for the IC [22]. This system incorporates
exercising hindsight. They take their current state of knowledge both Baconian and Bayesian methods for combining evidence
and compare it with what they or others did or could or should about our source.
have known before the current knowledge was received. This is
in sharp contrast with intelligence estimation, which is an As discussed above, assessing the credibility of a human
exercise in foresight, and it is the difference between these two source S involves assessing S’s veracity, objectivity, and obser-
modes of thought—hindsight and foresight—that seems to be a vational sensitivity. We have to consider that source S can be
source of bias. … After a view has been restructured to biased concerning any of these attributes. On veracity, S might
assimilate the new information, there is virtually no way to prefer to tell us that event E occurred, whether S believed E
accurately reconstruct the pre-existing mental set.” [2, p.162] occurred or not. As an example, an analyst evaluating S’s
evidence E* might have evidence about S suggesting that S
Apparently Heuer did not envision the use of a system like would tell us that E occurred because S wishes to be the bearer
TIACRITIS that keeps track of the performed analysis, what of what S believes we will regard as good news that event E
evidence we had, what assumptions we made and what were occurred. On objectivity, S might choose to believe that E
STIDS 2013 Proceedings Page 30
occurred because it would somehow be in S’s best interests if E Heuer [2, p.122] mentions the “distortion in the reporting
did occur. On observational sensitivity, there are various ways chain from subsource through source, case officer, reports
that S’s senses could be biased in favor of recording event E; officer, to analyst” but he does not analyze it. In criminal cases
clever forms of deception supply examples. in law, there are persons identified as “evidence custodians”,
who keep careful track of who discovered an item of evidence,
These three species of bias possible for HUMINT sources who then had access to it and for how long, and what if
must be considered by analysts attempting to assess the anything they did to the evidence when they had access to it.
credibility of source S and how much weight or force S’s
evidence E* should have in the analyst’s inference about These chains of custody add three major additional sources
whether or not event E did happen. The existence of any of of uncertainty for intelligence analysts to consider, that are
these three biases would have an effect on an analyst’s associated with the persons in chains of custody whose
assessment of the weight or force of S’s report E*. As we competence and credibility need to be considered. The first and
know, all assessments of the credibility of evidence rest upon most important question involves authenticity: Is the evidence
available evidence about its sources. In the case of HUMINT received by an analyst exactly what the initial evidence said
we need ancillary evidence about the veracity, objectivity, and and is it complete? The other questions involve assessing the
observational sensitivity of its sources. In the process, we have reliability and accuracy of the processes used to produce the
to see whether any such evidence reveals any of the three evidence if it is tangible in nature (see the right side of Fig. 6),
biases just considered. TIACRITIS supports the analyst in this or also used to take various actions on the evidence in a chain
determination by guiding her to answer specific questions of custody, whether the evidence is tangible or testimonial. As
based on ancillary evidence. For instance, the veracity an illustration, consider an item of testimonial HUMINT
questions considered are shown in Table 1. coming from a foreign national whose code name is
Table 1. Questions concerning the veracity of human sources.
“Wallflower”, who does not speak English [23]. Wallflower
gives his report to case officer Bob. This report is recorded by
1. Goals of this source? Does what this source tells us support any of his Bob and then translated by Husam. Then, Wallflower’s
or her goals? translated report is transmitted to a report’s officer Marsha who
2. Present influences on this source? Could this source have been edits it and transmits it to the analyst Clyde who evaluates it
influenced in any way to provide us with this report?
and assesses its weight or force.
3. Exploitation potential? Is this source subject to any significant exploi-
tation by other persons or organizations to provide us this information? Now, here is where forms of bias can enter that can be
4. Any contradictory or divergent evidence? Is there any evidence that associated with the persons involved in these chains of custody.
contradicts or conflicts with what the source has reported to us? The case officer Bob might have intentionally overlooked
5. Any corroborative or confirming evidence? Is there any other evidence details in his recording of Wallflower’s report. The translator
that corroborates or confirms this source's report? Husam may have intentionally altered or deleted parts of this
6. Veracity concerning collateral details? Are there any contradictions or report. The report’s officer Marsha might have altered or
conflicts in the collateral details provided by this source that reflect the
possibility of this source's dishonesty? deleted parts of the translated report of Wallflower’s testimony
7. Source's character? What evidence do we have about this source's
in her editing of it. The result of these actions is that the analyst
character and honesty that bears upon this source's veracity? Clyde receiving this evidence almost certainly did not receive
8. Reporting record? What does the record show about the truthfulness of an authentic and complete account of it, nor did he receive a
this source's previous reports to us? good account of its reliability and accuracy. What he received
9. Source expectations about us? Is there any evidence that this source was the transmitted, edited, translated, recorded testimony of
may be reporting events he/she believes we will wish to hear or see? Wallflower. Fig. 7 shows how TIACRITIS may determine the
10. Interview behavior? If this source reported these events to us, what believability of the evidence received by the analyst. Although
was this source's demeanor and bearing while giving us this report? the information to make such an analysis may not be available,
the analyst should adjust the confidence in his conclusion, in
B. Persons in Chains of Custody of Evidence recognition of these biases.
Unfortunately, there are other persons, apart from
Believability of transmitted, edited, tran-
HUMINT sources, whose possible biases need to be carefully
slated, recorded testimony of Wallflower
considered. We know that analysts make use of an enormous min Believability of
variety of evidence that is not testimonial or HUMINT, but is Believability of transmission by
tangible in nature. Examples include objects, images, sensor edited translation Marsha
records of various sorts, documents, maps, diagrams, charts, min
Believability of Believability of
and tabled information of various kinds. translated recording editing by Marsha
But the intelligence analysts only rarely have immediate min
Believability of Believability of
and first access to HUMINT assets or informants. They may recorded testimony translation by Husam
only rarely be the first ones to encounter an item of tangible min
evidence. What happens is that there are several persons who Believability Believability of
have access to evidence between the times the evidence is first of Wallflower recording by Bob
acquired and when the analysts first receive it. These persons Fig. 7. Chain of custody of Wallflower’s testimony.
may do a variety of different things to the initial evidence
during the time they have access to it. In law, these persons C. Consumers of Intelligence Analyses
constitute what is termed a “chain of custody” for evidence. The policy-making consumers or customers of intelligence
STIDS 2013 Proceedings Page 31
analysts are also subject to a variety of inferential and structured analytic methods, in the debate on how to
decisional biases that may influence the reported analytic significantly improve intelligence analysis [26].
conclusions. As is well known, the relationships between
intelligence analysts and governmental policy makers are much REFERENCES
discussed and involve considerable controversy [24, 25]. On [1] Schum D.A. (2001). The Evidential Foundations of Probabilistic
the one hand we hear intelligence professionals say that they do Reasoning, Northwestern University Press.
not make policies but only try to help policy makers be as [2] Heuer R.J. (1999). Psychology of Intelligence Analysis, Center for the
informed as they can be when they do form policies and make Study of Intelligence, Central Intelligence Agency, Washington, DC.
decisions in the nation’s best interests. But we also learn facts [3] Tecuci, G., Marcu, D., Boicu, M., Schum, D.A., Russell K. (2011).
Computational Theory and Cognitive Assistant for Intelligence Analysis,
about the intelligence process that complicate matters. An in Proc. 6th Int. Conf. on Semantic Technologies for Intelligence,
intelligence analysis is usually a hierarchical process involving Defense, and Security, pp. 68-75, Fairfax, VA, 16-18 November.
many intelligence officers, at various grade levels, who become [4] Cohen L.J. (1977). The Probable and the Provable, Clarendon Press,
involved in producing an intelligence “product”. At the most Oxford.
basic level of this hierarchy are the so-called “desk analysts” [5] Zadeh L. (1983). The Role of Fuzzy Logic in the Management of Uncer-
who are known and respected experts in the specific subject tainty in Expert Systems, Fuzzy Sets and Systems, vol.11, pp.199-227.
matter of the analysis at hand. An analysis produced by one or [6] Tecuci, G. (1998). Building Intelligent Agents: An Apprenticeship
more desk analysts is then passed “upward” through many Multistrategy Learning Theory, Methodology, Tool and Case Studies,
administrative levels, at each of which persons at these higher San Diego: Academic Press.
levels can comment on the desk analysts’ report. It is often [7] Tecuci, G., Boicu, M., Boicu, C., Marcu, D., Stanescu, B., Barbulescu,
M. (2005). The Disciple-RKF Learning and Reasoning Agent,
recognized that the higher an editor is in this hierarchy, the Computational Intelligence, Vol.21, No.4, pp. 462-479.
more political his/her views and actions become that may affect
[8] Tecuci, G., Boicu, M., Marcu, D., Schum, D. (2013). How Learning
the content and conclusions of the analysis at hand. As this Enables Intelligence Analysts to Rapidly Develop Practical Cognitive
“upward” process continues, the analysis that results may be Assistants, in Proc. 12th International Conference on Machine Learning
quite different from the one produced by the desk analysts, and Applications (ICMLA'13), Miami, Florida, December 4-7.
reflecting the biases of those who have successively edited it. [9] W3C (2004). http://www.w3.org/TR/rdf-schema/, accessed 10/11/13.
In some cases, these editing biases are the direct result of the [10] Boicu M., Tecuci G., Schum D. (2008). Intelligence Analysis Ontology
consumer’s biases who may wish to receive a certain analytic for Cognitive Assistants, in Proc. of Conf. “Ontology for the Intelligence
conclusion. Using a system like TIACRITIS that shows very Community”, Fairfax, VA, 3-4 December.
clearly how the analytic conclusion is rooted in evidence would [11] Heuer R.J. (2008). Computer-Aided Analysis of Competing Hypotheses,
in George R.Z., Bruce J.B., eds., Analyzing Intelligence: Origins,
significantly help in reducing the above biases. Obstacles, and Innovations, Georgetown Univ. Press, Washington, DC.
V. CONCLUSIONS [12] Drogin B. (2007). CURVEBALL: Spies, Lies, and the Con Man Who
Caused a War. Random House, New York, NY.
A wide variety of biases affect the correctness of [13] Kolmogorov A.N. (1933). Foundations of a Theory of Probability
intelligence analyses. In this paper we have shown how the use (1933), 2nd English edition, Chelsea Publishing, New York, NY., 1956
of TIACRITIS, a knowledge-based cognitive assistant, helps [14] Kolmogorov A.N. (1969). The Theory of Probability. In: Aleksandrov,
analysts recognize and counter many of them. TIACRITIS A. D., Kolmogorov, A. N., Lavrentiev, M. A. (eds) Mathematics: Its
integrates several semantic technologies (knowledge Content, Methods, and Meaning. MIT Press, Cambridge, MA.
representation through ontologies and rules, evidence-based [15] Kahneman, D., Tversky, A. (1974). Judgment under Uncertainty:
Heuristics and Biases. Science, Vol, 185, 1124-1131.
reasoning, machine learning and knowledge acquisition). It
[16] Kahneman, D., Slovic, P., Tversky, A. (1982). Judgment under
can run in a browser as a web-based system, or it can be Uncertainty: Heuristics and Biases. Cambridge University Press.
installed locally, and has been used in many civilian, military,
[17] Kent S. (1994). Words of Estimated Probability, in Steury D.P., ed.,
and intelligence organizations. Sherman Kent and the Board of National Estimates: Collected Essays,
Center for the Study of Intelligence, CIA, Washington, DC.
There are two complementary ways by which TIACRITIS
helps mitigate biases. First, as a cognitive assistant, it helps [18] Wigmore J.H. (1937). The Science of Judicial Proof. Boston, MA: Little,
Brown & Co.
automate many parts of the analysis process, making this task
[19] Toulmin S.E. (1963). The Uses of Argument. Cambridge Univ. Press.
much easier for the analyst. Thus it alleviates one of the main
[20] van Gelder T.J. (2007). The Rationale for Rationale, Law, Probability
causes of biases, which is the employment of simplified and Risk, 6, pp. 23-42.
information processing strategies on the part of the analyst. [21] Shafer G. (1988). Combining AI and OR. University of Kansas School of
Second, TIACRITIS performs a rigorous evidence-based Business Working Paper No. 195.
hypothesis analysis that makes explicit all the reasoning steps, [22] Schum D.A., and Morris J. (2007). Assessing the Competence and
evidence, probabilistic assessments, and assumptions, so that Credibility of Human Sources of Evidence: Contributions from Law and
they can be critically analyzed and debated. Indeed, the best Probability, Law, Probability and Risk, Vol. 6, pp. 247-274.
protection against biases comes from the collaborative effort of [23] Schum D.A., Tecuci G, and Boicu M. (2009). Analyzing Evidence and
teams of analysts, who become skilled in solving their analytic its Chain of Custody: A Mixed-Initiative Computational Approach, Int.
tasks through the development of sound evidence-based Journal of Intelligence and Counterintelligence, Vol. 22, pp. 298-319.
arguments, and who are willing to share their insights with [24] George, R., Bruce, J. (eds) (2008). Analyzing Intelligence: Origins,
Obstacles, and Innovations. Georgetown Univ. Press, Washingon. DC.
colleagues, who are also willing to listen. TIACRITIS makes
[25] Johnston, R. (2005). Analytic Culture in the U.S. Intelligence
all this possible. Community. Central Intelligence Agency, Washington, DC.
Finally, this paper adds a strong argument in favor of using [26] Marrin, S. (2011). Improving Intelligence Analysis: Bridging the gap
between scholarship and practice. Routlege, Lundon and New York.
STIDS 2013 Proceedings Page 32