=Paper=
{{Paper
|id=Vol-1876/paper04
|storemode=property
|title=Probabilistic Argument Maps for Intelligence Analysis: Capabilities Underway
|pdfUrl=https://ceur-ws.org/Vol-1876/paper04.pdf
|volume=Vol-1876
|authors=Robert Schrag,Edward Wright,Robert Kerr,Robert Johnson,Bryan Ware,Joan McIntyre,Melonie Richey,Kathryn Laskey,Robert Hoffman
|dblpUrl=https://dblp.org/rec/conf/ijcai/SchragWKJWMRLH16
}}
==Probabilistic Argument Maps for Intelligence Analysis: Capabilities Underway==
https://ceur-ws.org/Vol-1876/paper04.pdf
Proceedings of CMNA 2016 - Floris Bex, Floriana Grasso, Nancy Green (eds)
Probabilistic Argument Maps for Intelligence Analysis: Capabilities Underway
Robert Schrag*, Edward Wright*, Robert Kerr*, Robert Johnson*, Bryan Ware*, Joan McIntyre+,
Melonie Richey$, Kathryn Laskey%, Robert Hoffman&
*
Haystax Technology, McLean, VA and Las Vegas, NV (Schrag)
$
Camber Corporation, Columbia, MD
+
Innovative Analytics and Training, Fairfax, VA
%
George Mason University, Fairfax, VA
&
Florida Institute for Human-Machine Cognition, Pensacola, FL
Abstract maps. Exploratory modeling in the domain of intelligence
analysis—where argument mapping [1] is useful but (until
We describe enhancements underway to our
FUSION [6]) has not been underpinned by mathematically
probabilistic argument mapping framework called
sound probabilistic reasoning—has highlighted
FUSION. Exploratory modeling in the domain of requirements for additional capabilities.
intelligence analysis has highlighted requirements
Forthcoming sections first present an early model
for additional knowledge representation and
motivating some of these capabilities, then describe our
reasoning capabilities, particularly regarding
technical approach to each. Appendices describe key
argument map nodes that are specified as
elements of our proposed intelligence domain-specific
propositional logic functions of other nodes. We
variant of FUSION called CRAFT.
also describe more flexible specifications for link
strengths and node prior probabilities. We expect
these enhancements to find general applicability 2 Motivating FUSION model
across problem domains. Figure 1 is a screenshot of a FUSION model addressing the
CIA’s Iraq retaliation scenario [4], where Iraq might
1 Introduction respond to US forces’ bombing of its intelligence
headquarters by conducting major, minor, or no terror
Haystax has developed the FUSION framework (see our
attacks. The model emphasizes Saddam’s incentives to act.
companion paper [7]) to facilitate creation of useful By setting a hard finding of false on the node SaddamWins,
Bayesian networks (BNs) by non-technical subject matter
we can examine computed beliefs under Saddam’s worst-
experts (SMEs). Because we use exclusively binary random
case scenario. See [7] for details regarding the scenario, our
variables (BN nodes) over the domain {true, false}, it is
different models addressing it, and analyst SME model
natural to construe FUSION models as probabilistic argument
feedback, as well as review of related work.
Downstream+!
Opposites
Opposite ! Upstream
Figure 1. Statement nodes are connected by positive (solid grey line) and negative (dashed grey line) indication links of
various strengths (per line thicknesses). Argument flow (from evidence to outcomes) is from right to left—e.g., SaddamWins
is strongly indicated by SaddamKeepsFace. Outcome hypothesis nodes are circled in yellow. SaddamWins (hard finding
false) captures Saddam’s incentives to act or not.
16
Proceedings of CMNA 2016 - Floris Bex, Floriana Grasso, Nancy Green (eds)
Node colors in Figure 1 capture whether Saddam’s •! TerrorAttacksFail (likewise TerrorAttacksSucceed) should
disposition (or attitude) regarding a given statement is be allowed to be true only when TerrorAttacks also is
favorable (blue) or unfavorable (red). FUSION computes true. We are correcting this by adding the capability to
these dispositions from a single directive—to propagate the assert Logic constraints, such as the one described for this
favorability of SaddamWins upstream (only), respecting model in section 3 to be used instead of the naïve
link polarities. One node (gray, bottom left) is not touched OppositeOf link now connecting TerrorAttacksFail and
by this propagation. Four nodes (purple, top right) have TerrorAttacksSucceed.
ambiguous status with respect to Saddam’s disposition.
Belief bars augment spatial tick marks with colors chosen to 3 Logic statements
reflect the degree of concern a given statement would pose
given a node’s disposition. Lower belief (redder bar) poses We are working to make FUSION support any standard
less concern regarding a statement viewed unfavorably, propositional logic expression using unary, binary, or higher
more regarding one viewed favorably, higher belief (bluer arity operators2. When a Logic statement has a hard true
bar) the reverse. Red/blue contrast thus draws attention to a finding3, we refer to it as a Logic constraint, otherwise as a
statement that should be of concern to Saddam. summarizing Logic statement.
FUSION models can include argument map link types per Figure 1’s model would be better if TerrorAttacksFail
Table 1. were allowed to be true only if TerrorAttacks also were true.
Table 1. FUSION supports probabilistic argument map model We know that an attempted action can succeed or fail only if
it occurs. By explicitly modeling (as Hypotheses) both
link types (center column). Only the final two link types
these potential action results and adding a Logic constraint4,
implementing propositional logic operators take an
we can force zero probability for every excluded truth value
arbitrary number of input statements. All other link types
combination, improving the model (in tradecraft terms,
are binary. correcting a “logic flaw”). See Figure 2.
IndicatedBy/ Figure 2’s graphics are from the COTS Netica BN
CounterIndicatedBy/ Application. Because all nodes in the generated BN
Downstream1 MitigatedBy/ Upstream underlying a FUSION model are binary, we also could render
statement RelevantIf/ statement(s) the full BN using more perspicuous single belief bars.
OppositeOf/ FUSION implements a target belief spec either
ImpliedByConjunction/ (depending on purpose) using a BN node like Figure 2’s
ImpliedByDisjunction/ ConstraintTBC or (equivalently) via a likelihood finding on
the subject BN node. The FUSION GUI does not ordinarily
In a FUSION model, every argument map statement is a
expose an auxiliary node like ConstraintTBC to an analyst.
Hypothesis. For the last two link types in Table 1, the
This example is for illustration. We can implement this
downstream statement also is a Logic statement.
particular BN pattern without target beliefs. We also could
In developing the model in Figure 1, we identified the
implement absolute-strength IndicatedBy links as simple
following representation and reasoning shortcomings for
implication Logic constraints. However, this would not
which we are now implementing responsive capabilities.
naturally accommodate one of these links’ key properties—
•! Beliefs computed for the scenario’s three outcomes do not
the ability to specify degree of belief in the link’s upstream
sum to 1.0. We can correct this by recasting
node when the downstream node is true—relevant because
IraqRetaliatesWithTerror as an exclusive-or (summary or
we can infer nothing about P given P ! Q and knowing Q
constraint) Logic statement (see section 3) over the
to be true. It also demands two target belief specs that tend
scenario’s outcomes, rather than as an absolutely
to compete. We are working to identify more Logic
supported Hypothesis, while also addressing the next
constraint patterns that can be implemented without target
issue.
beliefs and to generalize specification of belief degree for
•! In current FUSION, a given node cannot be both a Logic
any underdetermined entries in a summarizing Logic
statement and an indicator. Our FUSION spec-to-BN
statement’s CPT.
conversion software [6] treats these as distinct patterns of
parents. An indicated node acts as a BN parent to an
indicating one. A Logic statement’s input nodes act as
2
parents to the Logic node itself. To implement indication See, e.g., https://en.wikipedia.org/wiki/Truth_table.
3
by a Logic statement node L, we will create (under the A likelihood finding could be used to implement a soft
hood) an auxiliary node I to serve as the indicator and constraint.
4
assert a Logic constraint C (with parents L and I) This constraint can be rendered (abbreviating statement
establishing L = I. names) as (or/(and/occur/(xor/succeed/fail))/(and/(not/Occurs)/(nor/
Succeeds/ Fails))) or more compactly via an if-then-else logic
function (notated ite) as (ite/ Occurs/ (xor/ Succeeds/ Fails)/ (nor/
1
Per argument map convention, “downstream” is left, Succeeds/ Fails))—if an attack occurs, it either succeeds or fails,
“upstream” right in the left-flowing argument map of Figure 1. else it neither succeeds nor fails.
17
Proceedings of CMNA 2016 - Floris Bex, Floriana Grasso, Nancy Green (eds)
FailedAttack
true 25.0
false 75.0
If#Attack#Only#1#Outcome#Constraint Attacks#occur
FailedAttack true )100 true 50.0
false )))0 false 50.0
true 25.0
false 75.0
SuccessfullAttack
true 25.0
If#Attack#Only#1#Outcome#Constraint Attacks#occur false 75.0 ConstraintTBC
true 75.0 true 50.0 true )100
false 25.0 false 50.0 false )))0
SuccessfullAttack
true 25.0
false 75.0
Figure 2. We are implementing Logic/ constraints to enforce sound logical reasoning. The constraint node (left, in right
model fragment) ensures that the model will believe in attack success/failure only when an attack actually occurs. Setting the
hard true finding on this node turns the summarizing Logic statement (left, in the left fragment) into the Logic constraint—but
also distorts the model’s computed probabilities for the three Hypotheses. Presuming these probabilities have been
deliberately engineered by the modeler, FUSION must restore them. It does so by specifying (bottom fragment) a target belief
(implemented via the ConstraintTBC node) on one of the Hypotheses.
4' Target'beliefs published algorithms. Per the recent survey by Mrad et al
[6], who refer to this capability as “fixed probability
We can set a target belief for any FUSION node whose distribution” specification, the only published results are for
computed belief under a baseline situation (say, before the much smaller problems.
application of a set of evidence items or assumptions)
deviates from a modeler’s expectations or requirements. 5 Flexible belief and strength specifications
We have formerly done this to address unacceptably small
probabilities for far-upstream nodes—an issue we hope to A given Bayesian network requires the specification of
see less of with new flexible link strengths (section 5). A many individual, point probability values, but intelligence
target belief also can serve an exogenous variable that analysts generally work with probability ranges, as
should be informed by real-world data statistics. institutionalized in Intelligence Community Directive (ICD)
As with BN belief inference, there is no closed-form 203 [2] per Figure 3 below. Beyond perfunctorily
solution for target belief satisfaction, which requires a associating a uniform distribution with the range [45, 55]%
gradient descent optimization over individual BN inference (zero elsewhere) with the designation “roughly even
invocations, measuring in each step nodes’ differences chance,” a user may select a standard distribution (such as
between observed beliefs and targets. Our implementation our grey one) or specify his/her own mode, inflection points,
[8] computes differences on a log odds (vs. linear) scale and non-zero probability range—perhaps all by dragging a
(reflecting actual belief impacts and reducing gradient few handles on a control. Only the curves’ shapes matter
descent oscillation), initially adjusts all involved nodes in a here. FUSION normalizes height to agree with unit area (=
single optimization step (effectively, in parallel—saving 1.0).
steps compared to strictly serial optimization), and saves the While several of FUSION’s parameters are probability-
work from previous satisfaction processes over a given valued, the most prominent way probabilities enter a
model (e.g., under edit) for fast incremental operation. FUSION-generated BN is via under-the-hood encoding of on-
In Haystax’ primary risk assessment FUSION application the-dashboard-specified indication strengths. FUSION now
(called CARBON) including hundreds of BN nodes and encodes strengths using fixed odds ratios per Figure 4.
dozens of target beliefs, processing takes just a few seconds. More flexible specifications may be more appropriate for
Our results compare favorably with those of related many problem types.
18
Proceedings of CMNA 2016 - Floris Bex, Floriana Grasso, Nancy Green (eds)
Figure 3. FUSION will bridge the gap between analysts’ familiar probability ranges (shown here with preferred natural
language labels per ICD 203) and Bayesian networks’ exact probability requirements by supporting explicit probability
distributions (shown here in grey). Beyond perfunctorily associating a uniform distribution with the range [45, 55]% (zero
elsewhere) with the designation “roughly even chance,” a CRAFT user may select a standard distribution (like our grey one)
or a customized one, perhaps (in a future GUI) by dragging a few handles (small circles) on the distribution curve. Thus,
CRAFT supports analysts’ best intuitions about argument probabilities without forcing commitments. By sampling over input
distributions, develop (under the hood) and display (on the dashboard) output belief distributions for argument map
statements/nodes.
Figure 4. We are generalizing FUSION’S current fixed indication strengths (labeled vertical lines) as user-specifiable
probability distributions similar to those planned for probabilities (Figure 3), supporting any odds ratio or distribution
thereof. Absolutely is intended as logical implication. Fusion does not otherwise commit analysts to absolute certainty. An
odds or log odds scale may be more salient than standard probability. This approach develops finest warranted beliefs.
Robust statistical sampling of belief distributions could, probabilistic distinctions will enable more precise modeling
in large models, exceed GUI near-real-time thresholds for by individual SMEs and SME teams or crowds—values for
to-user feedback. Another attractive option, given numeric model parameters might be aggregated using
reasonably tight belief distributions, is to develop only crowdsourcing methods.
bounds for statement beliefs, exploiting FUSION’s per- Our proposed CRAFT variant designed for intelligence
statement actor disposition framework (described at Figure analysis will express all of FUSION’s capabilities in analyst-
1). Develop a statement’s favorable bound (i.e., the lower friendly vocabulary (appendix A) and develop mechanisms
bound for an unfavorable statement, upper bound for a and methods explicitly supporting the modeling of source
favorable one) by considering just other favorable bounds in credibility per Intelligence Community standards (appendix
an unambiguous-statement-only subgraph, and address B).
limited combinatorics over disconnected subgraphs. Belief
bars resulting would thus include just two tick marks, rather References
than a richer distribution shape. [1]! CIA Directorate of Intelligence, “A Tradecraft Primer: The
Use of Argument Mapping,” Tradecraft Review 3(1), Kent
6 Conclusion Center for Analytic Tradecraft, Sherman Kent School.
We are working to make FUSION, already a powerful [2]! Intelligence Community Directive (ICD) 203, “Analytic
framework for SMEs to author Bayesian network-based Standards,” January 2, 2015. http://fas.org/irp/dni/icd/icd-
models, more complete and versatile for probabilistic 203.pdf
argument map applications—in intelligence analysis and in [3]! Intelligence Community Directive (ICD) 206, “Sourcing Re-
quirements for Disseminated Analytic Products,” October 17,
other domains, generally. Allowing summary Logic
2007. http://fas.org/irp/dni/icd/icd-206.pdf
statements to serve as probabilistic indicators of hypotheses [4]! Richards J. Heuer, Jr., Psychology of Intelligence Analysis,
will afford uniform expressive power across a model’s Central Intelligence Agency Historical Document.
downstream and upstream levels. Supporting arbitrary Logic https://www.cia.gov/library/center-for-the-study-of-
formulas as constraints as well as summary statements will intelligence/csi-publications/books-and-
increase expressiveness and conciseness. Supporting more monographs/psychology-of-intelligence-analysis (Posted: Mar
flexible belief and strength specifications to capture finer 16, 2007 01:52 PM. Last Updated: Jun 26, 2013 08:05 AM.)
19
Proceedings of CMNA 2016 - Floris Bex, Floriana Grasso, Nancy Green (eds)
[5]! Frank Hughes, “A Tutorial on Credibility Assessment for Validity/ A statement characterizing an analyst’s self-
Various Forms of Intelligence Evidence,” Joint Military Intel- (of Assumption)/ assessed legitimacy of a posited Assumption.
ligence College, Defense Intelligence Agency, Revised 5 A propositional logic expression over other
April, 2003. Logic/ statement nodes, either summarizing or
[6]! Ali Ben Mrad, Veronique Delcroix, Sylvain Piechowiak, Phil- constraining them.
ip Leicester, and Mohamed Abid, “An explication of uncer-
tain evidence in Bayesian networks: likelihood evidence and Table 3. CRAFT will support probabilistic argument map
probabilistic evidence,” Applied Intelligence, published online model link types (center column). All link types are binary.
20 June 2015.
[7]! Robert Schrag, Joan McIntyre, Melonie Richey, Kathryn Las- SupportedBy/
key, Edward Wright, Robert Kerr, Robert Johnson, Bryan Downstream RefutedBy/ Upstream
Ware, Robert Hoffman, “Probabilistic Argument Maps for In- statement MitigatedBy/ statement
telligence Analysis: Completed Capabilities,” 16th Workshop RelevantIf/
on Computational Models of Natural Argument, 2016. LogicInput/
[8]! Robert Schrag, Edward Wright, Robert Kerr, and Robert
Johnson, “Target Beliefs for SME-oriented, Bayesian Net-
work-based Modeling,” 13th Annual Bayesian Modeling Ap- B CRAFT credibility reasoning
plications Workshop, 2016. CRAFT will apply to a HUMINT EvidenceReport
[9]! David A. Schum, Evidential Foundations of Probabilistic statement Schum’s framework [6] that assesses Credibility
Reasoning, Wiley and Sons, 1994.
with respect to four attributes of the reporting agent:
[10]!Edward Wright, Robert Schrag, Robert Kerr, and Bryan
Veracity, Objectivity, Competence, and Opportunity to
Ware, “Automating the Construction of Indicator-Hypothesis
Bayesian Networks from Qualitative Specifications,” Haystax observe what’s been reported. A reported statement is
Technology technical report, 2015, believed credible only if all four of its factor statements are
https://labs.haystax.com/wp- believed true, so we link the EvidenceReport as RelevantIf
content/uploads/2016/06/BMAW15-160303-update.pdf. this conjunctive Logic statement holds. Note that the factor
statements are themselves Hypotheses, subject to supporting
and refuting statements bearing potentially rich argument
A CRAFT node and link types
structure. Note that direct or indirect corroboration will
Table 2 and Table 3 describe the roles of different node and increase an EvidenceReport’s Credibility.
link types in our proposed intelligence domain-specific vari- Below we outline how CRAFT will map (in italics)
specific analytical tradecraft standards (in bold) regarding
ant of FUSION called CRAFT. We anticipate distinct GUI
items of evidence per ICD 203 [2] to Schum’s factors or, as
icons for these types.
appropriate, other model elements.
Factors from ICD 203 D.6.e.1:
Table 2. CRAFT will distinguish among model statement
types in ways that are salient to intelligence analysts. 1.!Accuracy (use Credibility) and completeness (explicitly
model omitted possibilities as Assumptions or other
A Hypothesis distinguished as a primary
Outcome/ intelligence analysis alternative—e.g., a
appropriate statements)
possible future (or present, or past) situation. 2.!Possible denial (use Opportunity) and deception (elabo-
A statement whose probability of truth rate a deceptive course of action as an Outcome or
Hypothesis/
depends on upstream-neighbor links. other Hypothesis—see SaddamMaintainsDiplomacy in
EvidenceJ A statement that is in principle knowable from Figure 1, e.g.)
Hypothesis/ evidence available in the problem domain. 3.!Age and continued currency of information (use tem-
A statement reported by some source. poral relevance)
Absolutely influences a like-content 4.!Technical elements of collection (apply true/false posi-
EvidenceHypothesis that is source- tive/negative sensor models, confusion matrices—
EvidenceJ/
independent and accommodates any number ancillary to Schum’s framework—see also [5], dis-
Report/
of SupportedBy or RefutedBy links from
cussed below)
EvidenceReports corresponding to different
reports.
5.!Source access (use Opportunity)
A statement characterizing the credibility of 6.!Validation (model as corroboration by other sources)
Credibility/ an EvidenceReport. Per modeler discretion, 7.!Motivation (model as in Figure 1, e.g.)
(of either a single node or a conjunctive Logic 8.!Possible bias (use Objectivity)
EvidenceReport)/ statement summarizing relevant credibility 9.!Expertise (use Competence)
factors.
A statement posited by an analyst to fill a gap More factors from ICD 206 [3] appendix A (glossary),
in available information. Plays a modeling “source descriptor:”
Assumption/
role similar to EvidenceReport, when
10.! Precision or technical quality (see 1, 4)
evidence is unavailable.
20
Proceedings of CMNA 2016 - Floris Bex, Floriana Grasso, Nancy Green (eds)
11.! Context (connotes scope or socio-cultural setting—
capture via other model statements, as appropriate)
12.! For human sources:
a.! Level of access (use Opportunity)
b.! Past reporting record (associate with base
rate/prior for any credibility factor—
including/especially Veracity)
c.! Potential biases—e.g., political, personal, profes-
sional, religious affiliations (counter-indicators for
Objectivity)
From ICD 206 appendix D.5:
13.! Potential strengths and limitations of available in-
formation (see 1)
14.! Notable inconsistencies in reporting (impugn CredibilC
ity, per factor as warranted)
15.! Important information gaps (see 1)
16.! Other factors deemed relevant (model as appropri-
ate)
HUMINT is testimonial evidence. Hughes [5] addresses
credibility also for tangible and sensor evidence and
presents considerations that may inform argument structure
affecting each of Schum’s testimonial evidence credibility
factors. He also tenders the factor “observational
sensitivity”—which may be as relevant to sensing by
humans as it is to sensing by devices—and addresses
authenticity, chain of custody, and primary vs. secondary (or
tertiary, ...) sources. CRAFT might be engineered to support
some of these refinements explicitly. CRAFT’s credibility
reasoning will remain explicitly probabilistic, however,
exploiting the FUSION foundation, rather than assessed ad
hoc as in the later sections of Hughes’ report.
21