Proceedings of CMNA 2016 - Floris Bex, Floriana Grasso, Nancy Green (eds) Probabilistic Argument Maps for Intelligence Analysis: Capabilities Underway Robert Schrag*, Edward Wright*, Robert Kerr*, Robert Johnson*, Bryan Ware*, Joan McIntyre+, Melonie Richey$, Kathryn Laskey%, Robert Hoffman& * Haystax Technology, McLean, VA and Las Vegas, NV (Schrag) $ Camber Corporation, Columbia, MD + Innovative Analytics and Training, Fairfax, VA % George Mason University, Fairfax, VA & Florida Institute for Human-Machine Cognition, Pensacola, FL Abstract maps. Exploratory modeling in the domain of intelligence analysis—where argument mapping [1] is useful but (until We describe enhancements underway to our FUSION [6]) has not been underpinned by mathematically probabilistic argument mapping framework called sound probabilistic reasoning—has highlighted FUSION. Exploratory modeling in the domain of requirements for additional capabilities. intelligence analysis has highlighted requirements Forthcoming sections first present an early model for additional knowledge representation and motivating some of these capabilities, then describe our reasoning capabilities, particularly regarding technical approach to each. Appendices describe key argument map nodes that are specified as elements of our proposed intelligence domain-specific propositional logic functions of other nodes. We variant of FUSION called CRAFT. also describe more flexible specifications for link strengths and node prior probabilities. We expect these enhancements to find general applicability 2 Motivating FUSION model across problem domains. Figure 1 is a screenshot of a FUSION model addressing the CIA’s Iraq retaliation scenario [4], where Iraq might 1 Introduction respond to US forces’ bombing of its intelligence headquarters by conducting major, minor, or no terror Haystax has developed the FUSION framework (see our attacks. The model emphasizes Saddam’s incentives to act. companion paper [7]) to facilitate creation of useful By setting a hard finding of false on the node SaddamWins, Bayesian networks (BNs) by non-technical subject matter we can examine computed beliefs under Saddam’s worst- experts (SMEs). Because we use exclusively binary random case scenario. See [7] for details regarding the scenario, our variables (BN nodes) over the domain {true, false}, it is different models addressing it, and analyst SME model natural to construe FUSION models as probabilistic argument feedback, as well as review of related work. Downstream+! Opposites Opposite ! Upstream Figure 1. Statement nodes are connected by positive (solid grey line) and negative (dashed grey line) indication links of various strengths (per line thicknesses). Argument flow (from evidence to outcomes) is from right to left—e.g., SaddamWins is strongly indicated by SaddamKeepsFace. Outcome hypothesis nodes are circled in yellow. SaddamWins (hard finding false) captures Saddam’s incentives to act or not. 16 Proceedings of CMNA 2016 - Floris Bex, Floriana Grasso, Nancy Green (eds) Node colors in Figure 1 capture whether Saddam’s •! TerrorAttacksFail (likewise TerrorAttacksSucceed) should disposition (or attitude) regarding a given statement is be allowed to be true only when TerrorAttacks also is favorable (blue) or unfavorable (red). FUSION computes true. We are correcting this by adding the capability to these dispositions from a single directive—to propagate the assert Logic constraints, such as the one described for this favorability of SaddamWins upstream (only), respecting model in section 3 to be used instead of the naïve link polarities. One node (gray, bottom left) is not touched OppositeOf link now connecting TerrorAttacksFail and by this propagation. Four nodes (purple, top right) have TerrorAttacksSucceed. ambiguous status with respect to Saddam’s disposition. Belief bars augment spatial tick marks with colors chosen to 3 Logic statements reflect the degree of concern a given statement would pose given a node’s disposition. Lower belief (redder bar) poses We are working to make FUSION support any standard less concern regarding a statement viewed unfavorably, propositional logic expression using unary, binary, or higher more regarding one viewed favorably, higher belief (bluer arity operators2. When a Logic statement has a hard true bar) the reverse. Red/blue contrast thus draws attention to a finding3, we refer to it as a Logic constraint, otherwise as a statement that should be of concern to Saddam. summarizing Logic statement. FUSION models can include argument map link types per Figure 1’s model would be better if TerrorAttacksFail Table 1. were allowed to be true only if TerrorAttacks also were true. Table 1. FUSION supports probabilistic argument map model We know that an attempted action can succeed or fail only if it occurs. By explicitly modeling (as Hypotheses) both link types (center column). Only the final two link types these potential action results and adding a Logic constraint4, implementing propositional logic operators take an we can force zero probability for every excluded truth value arbitrary number of input statements. All other link types combination, improving the model (in tradecraft terms, are binary. correcting a “logic flaw”). See Figure 2. IndicatedBy/ Figure 2’s graphics are from the COTS Netica BN CounterIndicatedBy/ Application. Because all nodes in the generated BN Downstream1 MitigatedBy/ Upstream underlying a FUSION model are binary, we also could render statement RelevantIf/ statement(s) the full BN using more perspicuous single belief bars. OppositeOf/ FUSION implements a target belief spec either ImpliedByConjunction/ (depending on purpose) using a BN node like Figure 2’s ImpliedByDisjunction/ ConstraintTBC or (equivalently) via a likelihood finding on the subject BN node. The FUSION GUI does not ordinarily In a FUSION model, every argument map statement is a expose an auxiliary node like ConstraintTBC to an analyst. Hypothesis. For the last two link types in Table 1, the This example is for illustration. We can implement this downstream statement also is a Logic statement. particular BN pattern without target beliefs. We also could In developing the model in Figure 1, we identified the implement absolute-strength IndicatedBy links as simple following representation and reasoning shortcomings for implication Logic constraints. However, this would not which we are now implementing responsive capabilities. naturally accommodate one of these links’ key properties— •! Beliefs computed for the scenario’s three outcomes do not the ability to specify degree of belief in the link’s upstream sum to 1.0. We can correct this by recasting node when the downstream node is true—relevant because IraqRetaliatesWithTerror as an exclusive-or (summary or we can infer nothing about P given P ! Q and knowing Q constraint) Logic statement (see section 3) over the to be true. It also demands two target belief specs that tend scenario’s outcomes, rather than as an absolutely to compete. We are working to identify more Logic supported Hypothesis, while also addressing the next constraint patterns that can be implemented without target issue. beliefs and to generalize specification of belief degree for •! In current FUSION, a given node cannot be both a Logic any underdetermined entries in a summarizing Logic statement and an indicator. Our FUSION spec-to-BN statement’s CPT. conversion software [6] treats these as distinct patterns of parents. An indicated node acts as a BN parent to an indicating one. A Logic statement’s input nodes act as 2 parents to the Logic node itself. To implement indication See, e.g., https://en.wikipedia.org/wiki/Truth_table. 3 by a Logic statement node L, we will create (under the A likelihood finding could be used to implement a soft hood) an auxiliary node I to serve as the indicator and constraint. 4 assert a Logic constraint C (with parents L and I) This constraint can be rendered (abbreviating statement establishing L = I. names) as (or/(and/occur/(xor/succeed/fail))/(and/(not/Occurs)/(nor/ Succeeds/ Fails))) or more compactly via an if-then-else logic function (notated ite) as (ite/ Occurs/ (xor/ Succeeds/ Fails)/ (nor/ 1 Per argument map convention, “downstream” is left, Succeeds/ Fails))—if an attack occurs, it either succeeds or fails, “upstream” right in the left-flowing argument map of Figure 1. else it neither succeeds nor fails. 17 Proceedings of CMNA 2016 - Floris Bex, Floriana Grasso, Nancy Green (eds) FailedAttack true 25.0 false 75.0 If#Attack#Only#1#Outcome#Constraint Attacks#occur FailedAttack true )100 true 50.0 false )))0 false 50.0 true 25.0 false 75.0 SuccessfullAttack true 25.0 If#Attack#Only#1#Outcome#Constraint Attacks#occur false 75.0 ConstraintTBC true 75.0 true 50.0 true )100 false 25.0 false 50.0 false )))0 SuccessfullAttack true 25.0 false 75.0 Figure 2. We are implementing Logic/ constraints to enforce sound logical reasoning. The constraint node (left, in right model fragment) ensures that the model will believe in attack success/failure only when an attack actually occurs. Setting the hard true finding on this node turns the summarizing Logic statement (left, in the left fragment) into the Logic constraint—but also distorts the model’s computed probabilities for the three Hypotheses. Presuming these probabilities have been deliberately engineered by the modeler, FUSION must restore them. It does so by specifying (bottom fragment) a target belief (implemented via the ConstraintTBC node) on one of the Hypotheses. 4' Target'beliefs published algorithms. Per the recent survey by Mrad et al [6], who refer to this capability as “fixed probability We can set a target belief for any FUSION node whose distribution” specification, the only published results are for computed belief under a baseline situation (say, before the much smaller problems. application of a set of evidence items or assumptions) deviates from a modeler’s expectations or requirements. 5 Flexible belief and strength specifications We have formerly done this to address unacceptably small probabilities for far-upstream nodes—an issue we hope to A given Bayesian network requires the specification of see less of with new flexible link strengths (section 5). A many individual, point probability values, but intelligence target belief also can serve an exogenous variable that analysts generally work with probability ranges, as should be informed by real-world data statistics. institutionalized in Intelligence Community Directive (ICD) As with BN belief inference, there is no closed-form 203 [2] per Figure 3 below. Beyond perfunctorily solution for target belief satisfaction, which requires a associating a uniform distribution with the range [45, 55]% gradient descent optimization over individual BN inference (zero elsewhere) with the designation “roughly even invocations, measuring in each step nodes’ differences chance,” a user may select a standard distribution (such as between observed beliefs and targets. Our implementation our grey one) or specify his/her own mode, inflection points, [8] computes differences on a log odds (vs. linear) scale and non-zero probability range—perhaps all by dragging a (reflecting actual belief impacts and reducing gradient few handles on a control. Only the curves’ shapes matter descent oscillation), initially adjusts all involved nodes in a here. FUSION normalizes height to agree with unit area (= single optimization step (effectively, in parallel—saving 1.0). steps compared to strictly serial optimization), and saves the While several of FUSION’s parameters are probability- work from previous satisfaction processes over a given valued, the most prominent way probabilities enter a model (e.g., under edit) for fast incremental operation. FUSION-generated BN is via under-the-hood encoding of on- In Haystax’ primary risk assessment FUSION application the-dashboard-specified indication strengths. FUSION now (called CARBON) including hundreds of BN nodes and encodes strengths using fixed odds ratios per Figure 4. dozens of target beliefs, processing takes just a few seconds. More flexible specifications may be more appropriate for Our results compare favorably with those of related many problem types. 18 Proceedings of CMNA 2016 - Floris Bex, Floriana Grasso, Nancy Green (eds) Figure 3. FUSION will bridge the gap between analysts’ familiar probability ranges (shown here with preferred natural language labels per ICD 203) and Bayesian networks’ exact probability requirements by supporting explicit probability distributions (shown here in grey). Beyond perfunctorily associating a uniform distribution with the range [45, 55]% (zero elsewhere) with the designation “roughly even chance,” a CRAFT user may select a standard distribution (like our grey one) or a customized one, perhaps (in a future GUI) by dragging a few handles (small circles) on the distribution curve. Thus, CRAFT supports analysts’ best intuitions about argument probabilities without forcing commitments. By sampling over input distributions, develop (under the hood) and display (on the dashboard) output belief distributions for argument map statements/nodes. Figure 4. We are generalizing FUSION’S current fixed indication strengths (labeled vertical lines) as user-specifiable probability distributions similar to those planned for probabilities (Figure 3), supporting any odds ratio or distribution thereof. Absolutely is intended as logical implication. Fusion does not otherwise commit analysts to absolute certainty. An odds or log odds scale may be more salient than standard probability. This approach develops finest warranted beliefs. Robust statistical sampling of belief distributions could, probabilistic distinctions will enable more precise modeling in large models, exceed GUI near-real-time thresholds for by individual SMEs and SME teams or crowds—values for to-user feedback. Another attractive option, given numeric model parameters might be aggregated using reasonably tight belief distributions, is to develop only crowdsourcing methods. bounds for statement beliefs, exploiting FUSION’s per- Our proposed CRAFT variant designed for intelligence statement actor disposition framework (described at Figure analysis will express all of FUSION’s capabilities in analyst- 1). Develop a statement’s favorable bound (i.e., the lower friendly vocabulary (appendix A) and develop mechanisms bound for an unfavorable statement, upper bound for a and methods explicitly supporting the modeling of source favorable one) by considering just other favorable bounds in credibility per Intelligence Community standards (appendix an unambiguous-statement-only subgraph, and address B). limited combinatorics over disconnected subgraphs. Belief bars resulting would thus include just two tick marks, rather References than a richer distribution shape. [1]! CIA Directorate of Intelligence, “A Tradecraft Primer: The Use of Argument Mapping,” Tradecraft Review 3(1), Kent 6 Conclusion Center for Analytic Tradecraft, Sherman Kent School. We are working to make FUSION, already a powerful [2]! Intelligence Community Directive (ICD) 203, “Analytic framework for SMEs to author Bayesian network-based Standards,” January 2, 2015. http://fas.org/irp/dni/icd/icd- models, more complete and versatile for probabilistic 203.pdf argument map applications—in intelligence analysis and in [3]! Intelligence Community Directive (ICD) 206, “Sourcing Re- quirements for Disseminated Analytic Products,” October 17, other domains, generally. Allowing summary Logic 2007. http://fas.org/irp/dni/icd/icd-206.pdf statements to serve as probabilistic indicators of hypotheses [4]! Richards J. Heuer, Jr., Psychology of Intelligence Analysis, will afford uniform expressive power across a model’s Central Intelligence Agency Historical Document. downstream and upstream levels. Supporting arbitrary Logic https://www.cia.gov/library/center-for-the-study-of- formulas as constraints as well as summary statements will intelligence/csi-publications/books-and- increase expressiveness and conciseness. Supporting more monographs/psychology-of-intelligence-analysis (Posted: Mar flexible belief and strength specifications to capture finer 16, 2007 01:52 PM. Last Updated: Jun 26, 2013 08:05 AM.) 19 Proceedings of CMNA 2016 - Floris Bex, Floriana Grasso, Nancy Green (eds) [5]! Frank Hughes, “A Tutorial on Credibility Assessment for Validity/ A statement characterizing an analyst’s self- Various Forms of Intelligence Evidence,” Joint Military Intel- (of Assumption)/ assessed legitimacy of a posited Assumption. ligence College, Defense Intelligence Agency, Revised 5 A propositional logic expression over other April, 2003. Logic/ statement nodes, either summarizing or [6]! Ali Ben Mrad, Veronique Delcroix, Sylvain Piechowiak, Phil- constraining them. ip Leicester, and Mohamed Abid, “An explication of uncer- tain evidence in Bayesian networks: likelihood evidence and Table 3. CRAFT will support probabilistic argument map probabilistic evidence,” Applied Intelligence, published online model link types (center column). All link types are binary. 20 June 2015. [7]! Robert Schrag, Joan McIntyre, Melonie Richey, Kathryn Las- SupportedBy/ key, Edward Wright, Robert Kerr, Robert Johnson, Bryan Downstream RefutedBy/ Upstream Ware, Robert Hoffman, “Probabilistic Argument Maps for In- statement MitigatedBy/ statement telligence Analysis: Completed Capabilities,” 16th Workshop RelevantIf/ on Computational Models of Natural Argument, 2016. LogicInput/ [8]! Robert Schrag, Edward Wright, Robert Kerr, and Robert Johnson, “Target Beliefs for SME-oriented, Bayesian Net- work-based Modeling,” 13th Annual Bayesian Modeling Ap- B CRAFT credibility reasoning plications Workshop, 2016. CRAFT will apply to a HUMINT EvidenceReport [9]! David A. Schum, Evidential Foundations of Probabilistic statement Schum’s framework [6] that assesses Credibility Reasoning, Wiley and Sons, 1994. with respect to four attributes of the reporting agent: [10]!Edward Wright, Robert Schrag, Robert Kerr, and Bryan Veracity, Objectivity, Competence, and Opportunity to Ware, “Automating the Construction of Indicator-Hypothesis Bayesian Networks from Qualitative Specifications,” Haystax observe what’s been reported. A reported statement is Technology technical report, 2015, believed credible only if all four of its factor statements are https://labs.haystax.com/wp- believed true, so we link the EvidenceReport as RelevantIf content/uploads/2016/06/BMAW15-160303-update.pdf. this conjunctive Logic statement holds. Note that the factor statements are themselves Hypotheses, subject to supporting and refuting statements bearing potentially rich argument A CRAFT node and link types structure. Note that direct or indirect corroboration will Table 2 and Table 3 describe the roles of different node and increase an EvidenceReport’s Credibility. link types in our proposed intelligence domain-specific vari- Below we outline how CRAFT will map (in italics) specific analytical tradecraft standards (in bold) regarding ant of FUSION called CRAFT. We anticipate distinct GUI items of evidence per ICD 203 [2] to Schum’s factors or, as icons for these types. appropriate, other model elements. Factors from ICD 203 D.6.e.1: Table 2. CRAFT will distinguish among model statement types in ways that are salient to intelligence analysts. 1.!Accuracy (use Credibility) and completeness (explicitly model omitted possibilities as Assumptions or other A Hypothesis distinguished as a primary Outcome/ intelligence analysis alternative—e.g., a appropriate statements) possible future (or present, or past) situation. 2.!Possible denial (use Opportunity) and deception (elabo- A statement whose probability of truth rate a deceptive course of action as an Outcome or Hypothesis/ depends on upstream-neighbor links. other Hypothesis—see SaddamMaintainsDiplomacy in EvidenceJ A statement that is in principle knowable from Figure 1, e.g.) Hypothesis/ evidence available in the problem domain. 3.!Age and continued currency of information (use tem- A statement reported by some source. poral relevance) Absolutely influences a like-content 4.!Technical elements of collection (apply true/false posi- EvidenceHypothesis that is source- tive/negative sensor models, confusion matrices— EvidenceJ/ independent and accommodates any number ancillary to Schum’s framework—see also [5], dis- Report/ of SupportedBy or RefutedBy links from cussed below) EvidenceReports corresponding to different reports. 5.!Source access (use Opportunity) A statement characterizing the credibility of 6.!Validation (model as corroboration by other sources) Credibility/ an EvidenceReport. Per modeler discretion, 7.!Motivation (model as in Figure 1, e.g.) (of either a single node or a conjunctive Logic 8.!Possible bias (use Objectivity) EvidenceReport)/ statement summarizing relevant credibility 9.!Expertise (use Competence) factors. A statement posited by an analyst to fill a gap More factors from ICD 206 [3] appendix A (glossary), in available information. Plays a modeling “source descriptor:” Assumption/ role similar to EvidenceReport, when 10.! Precision or technical quality (see 1, 4) evidence is unavailable. 20 Proceedings of CMNA 2016 - Floris Bex, Floriana Grasso, Nancy Green (eds) 11.! Context (connotes scope or socio-cultural setting— capture via other model statements, as appropriate) 12.! For human sources: a.! Level of access (use Opportunity) b.! Past reporting record (associate with base rate/prior for any credibility factor— including/especially Veracity) c.! Potential biases—e.g., political, personal, profes- sional, religious affiliations (counter-indicators for Objectivity) From ICD 206 appendix D.5: 13.! Potential strengths and limitations of available in- formation (see 1) 14.! Notable inconsistencies in reporting (impugn CredibilC ity, per factor as warranted) 15.! Important information gaps (see 1) 16.! Other factors deemed relevant (model as appropri- ate) HUMINT is testimonial evidence. Hughes [5] addresses credibility also for tangible and sensor evidence and presents considerations that may inform argument structure affecting each of Schum’s testimonial evidence credibility factors. He also tenders the factor “observational sensitivity”—which may be as relevant to sensing by humans as it is to sensing by devices—and addresses authenticity, chain of custody, and primary vs. secondary (or tertiary, ...) sources. CRAFT might be engineered to support some of these refinements explicitly. CRAFT’s credibility reasoning will remain explicitly probabilistic, however, exploiting the FUSION foundation, rather than assessed ad hoc as in the later sections of Hughes’ report. 21