=Paper=
{{Paper
|id=None
|storemode=property
|title=Requirements for Developing Strategic Decision Facilitation and Event Prediction Tools
|pdfUrl=https://ceur-ws.org/Vol-818/paper9.pdf
|volume=Vol-818
}}
==Requirements for Developing Strategic Decision Facilitation and Event Prediction Tools==
https://ceur-ws.org/Vol-818/paper9.pdf
Requirements for Developing Strategic Decision Facilitation and
Event Prediction Tools
Oscar Kipersztok
Boeing Research & Technology
P.O.Box 3707, MC: 7L-44
Seattle, WA 98124
oscar.kipersztok@boeing.com
Abstract forecast, will be able to better focus computational
resources and minimize (as far as possible) the quantified
This paper describes the requirements of a uncertainty over the most relevant aspects of the forecast.
strategic decision facilitation tool that relies on The goal of this paper is to clarify the requirements of a
forecasting to support critical decisions. A strategic decision facilitation tool that relies on
hypothesis-driven (data supported) system rather forecasting to support critical decisions.
than a purely data-driven methodology. It
further describes the importance of simple and Such system will offer users maximum flexibility and
natural human-computer interactions that provide quick turn-around through a decision facilitation
simplify the creation of complex domain models process that allows: a) easy capture and organization of
in a system that uses probabilistic reasoning knowledge, b) building complex models that can be
methods to facilitate high-quality decision readily queried about future events, c) applying advanced
making under uncertainty. Such a system helps algorithms, made transparent to users, to forecast
users create complex models, query them for predictions, d) searching and piecing together relevant
predictions, formulate hypotheses and validate and coherent argumentation in favor (or against) courses
their prediction with evidence retrieved from a of action; and e) making actionable recommendations to
corpus of text documents. The system must have facilitate significant strategic decisions.
a technology to automatically assemble and
explain the forecasts so that users--who should 2 KEY COMPONENTS
not be required to understand the mathematics
There are four key components to a forecasting system
behind the forecast--will be able to understand
that will be discussed to facilitate high-quality decision-
why certain predictions are being made.
making: 1. the forecasting algorithms should have access
to the context of decisions under consideration, not
1 INTRODUCTION simply the raw data--that is, they should be hypothesis-
A principal goal in any forecast of future events is to help driven; 2. the system should enable simple and natural
decision-makers deal with uncertainty. Our pictures of human-computer interactions to allow forecasting directly
the present and the past are always incomplete and noisy. over concepts of relevance and importance to the decision
So uncertainty is ubiquitous. A deeper concern is not just makers; 3. the simplicity of user interaction should not
the future, but even our theories and models of how prevent the use of advanced probabilistic reasoning
events will unfold that are underdetermined by our data. methods to quantify and minimize uncertainty over
Thus, forecasting from data alone is not sufficient and it forecasts; and, lastly, 4. the system should be capable of
can be improved by embedding the forecast technology automatically constructing explanations of forecasts
within a larger framework for decision support. Such a which can be understood without requiring users to
framework can supplement our raw data with information master the details of the forecasting algorithms. Together,
about the appropriate context in which to interpret the these components yield a complete decision-support
framework that allows for ongoing critical evaluation
and validation of high-quality forecasts created.
3 HYPOTHESIS DRIVEN
METHODOLOGY
3.1 HYPOTHESIS VERSUS DATA DRIVEN
There is a body of evidence in experimental psychology
suggesting different modalities in the way people make
decisions; some modalities result in more accurate
decisions than others (Heuer, 1999). In general, there is
the distinction between “data-driven” and “hypothesis-
driven” decision making. In the former, the emphasis is
on initial search and gathering of as much information as
possible before raising a hypothesis leading to an
informed decision. In the latter, the emphasis is on a more
selective and guided information search driven by a prior
hypothesis. An iterative process follows where the search
is aimed at specific information enabling validation or
rejection of the hypothesis. The hypothesis is either
accepted with sufficient evidence or re- formulated based
on insufficient evidence. Validated hypotheses with
sufficient evidence, in general, lead to more accurate
decisions. Our decision support system is architected to
direct users to follow a process that in practice has been
shown to result in more accurate decisions.
Figure 2a: Knowledge capture and representation
Figure 1: Sequential steps and feedback in hypothesis-
driven decision making
4 SIMPLE AND NATURAL HUMAN
COMPUTER INTERACTION
This section describes basic human computer interaction
principles used to facilitate the creation of complex
domain models while making transparent the complexity
of analytic methods.
Figure 2 shows the three types of analytic methods
required by the decision facilitation tool: 2a) knowledge
representation and capture, 2b) reasoning inference and
2c) text processing and search methods.
Figure 2b: Reasoning methods
of concept definitions and causal relations between them.
Most concepts affect or are affected by other concepts. In
most realistic domains there is feedback where a
particular concept starts a causal chain feeding back to
itself. Feedback loops can induce complex reinforcing or
inhibiting dynamic behavior. The “mental model” is
critical because it is used to make predictions and to
process and interpret outside information.
4.3 FREE LANGUAGE, ASSOCIATION AND
BRAINSTORMING
The use of free language in model creation enables more
flexibility in building models. Concepts are defined and
labeled with short phrases or using a few words. To
reduce ambiguity, users further expand concept definition
by providing added descriptions for more precise
meaning. Concepts are defined based on specific
assumptions that also need to be captured. Additional
documents and information (e.g. names, locations,
specific dates, events, etc.) are also associated with each
Figure 2c: NLP & Text Processing concept for further clarification. The use of free language
serves a dual purpose. Firstly, the words and phrases used
to define the concepts are also used in the creation of a
Three corresponding screens have been designed as user rule- based search engine to improve the recall and
interfaces to allow users to build models by defining precision of retrieved content needed to substantiate the
concepts and their associations, allowing to query the decisions. Secondly, the concept definitions and attached
model and make predictions by making assumptions descriptions are also used to create chains of rationale that
based on existing facts or beliefs, and searching for will provide explanations to subsequent predictions.
information through a corpus of documents in order to
validate the assumptions and predictions. 4.4 COMPLEXITY MANAGEMENT AND
SCALABILITY
These methods are used to provide maximum flexibility
and ease of use for rapid model creation, immediate query Automated knowledge capture should be made easy for
response and prediction, and fast document retrieval for the user. It should be just as easy to build models of high
forecast validation. complexity as it is very simple models. The user should
not be concerned with how the knowledge is being
4.1 TRANSPARENCY OF ANALYTIC captured, represented and organized. Users should be able
METHODOLOGY AND TERMINOLOGY to add or subtract information to and from the model with
ease and at will. Quantity of information should not be of
Advanced analytic methods often require familiarity with
concern to users. The information should be easily
complex methodology. In our approach to modeling
accessible at any time during the model building process,
complex domains, it is not necessary for users to learn
or later during the analysis phase. Providing flexibility to
and familiarize with analytic methods.
users during model creation in a free-associative,
By making the analytic methods transparent, users brainstorming fashion is important since it enables: a)
interact with the system by only using the familiar adequate coverage of the domain, leaving no stone
language of their domain. Domain concepts are defined unturned; b) seamless scalability to large, complex
using free language and users can add to those definitions domains; c) collaborative multiple-user participation with
to make their meaning more precise. This eliminates the access to second opinions and feedback; d) ease of model
need for knowledge engineers to acquire and convert user refinement and evolution at any future time; and e) speed
knowledge and expertise into computational models. - quick addition and deletion of ideas without concern
about performance or limits of scalability. Building
4.2 KNOWLEDGE AND RATIONALE models fast, with ease and with transparent complexity
CAPTURE management, enables users to build unconstrained models
of any size. The knowledge is represented using
The model creation process is the most critical step. Users constructs that are readily mapped into graphical
create a “mental model” of their domain, which consists
3
probabilistic networks for subsequent forecasting and nodes (Expected time to effect will be discussed in section
predictive analysis. 5.2).
Alternative methods for building the networks and their
5 PROBABILISTIC REASONING conditional probability tables require obtaining
METHODS probability estimates directly from domain experts for
each combination of child and parent nodes’ states. This
This section describes the creation behind-the-scenes,
tedious process can be aided by methods developed for
following the construction of the unconstrained mental
probability elicitation (Wang, 2004). A major goal of our
model, of a Bayesian network for making probabilistic
approach, however, is to circumvent this difficulty and
forecasts of events and trends.
make the process of building models readily accessible to
users without need of expertise in graphical probabilistic
5.1 FORECASTING PROBABILITY, IMPACT methods. In either case, once the models are built, their
AND TIMING OF EVENTS performance and robustness can be validated using
Analysis and prediction methods must be compatible with sensitivity analysis which can help identify the parameters
knowledge representation and acquisition constructs used that are most influential for any given query and
in the knowledge capture phase. In addition to defining prediction (Kipersztok and Wang, 2003).
concepts and relations, analysis methods require Analysts and decision makers require probability
additional quantitative input parameters that need to be estimates to guide strategic decisions. In order to maintain
obtained from the user during model creation. In the spirit the simple and natural human-computer interface, our
of devising easy ways to capture knowledge directly from approach limits the decision space to predictions of event
users, the number of requested variables is kept to a occurrences and trends. Decision makers need to know:
minimum. Methods were developed to map those a) how probable occurrences of events or emerging trends
variables to fit the requirements of the analysis and are; b) the magnitude of their impact; and c) the time
inference algorithms. Quantitative inputs should be when such events are expected to occur. Probabilistic
acquired from the user in an intuitive manner within the models, in particular graphical models, provide capability
context of the familiar user’s domain. to handle problems where data and information may be
sparse, noisy or incomplete. In addition to rapid
knowledge capture during model building, our methods
also provide quick turn-around forecasting during the
prediction phase.
As part of our on-going effort, we have built prototypes of
the system. (Kipersztok, 2007) describes in more detail
the implementation of various features of the system. The
system has been applied to several specific domain areas.
In (Seidler, et. al., 2010), the authors describe the DecAid
system which was used to model and predict the readiness
of a country to possess nuclear weapons capability. The
paper reviews the domain associations used to build an
unconstrained model. The model predictions were used to
retrieve textual documents with information on Iran’s
nuclear program and to compile the risk assessment
against the hypothesis that they are building a nuclear
Figure 3: Request minimal number of parameters weapon.
5.2 REASONING ABOUT EVENT TIME
The qualitative and quantitative inputs variables are New methods are being developed that allow for
shown in Figure 3. These are used to populate the probabilistic reasoning over systems evolving in
parameters of graphical probabilistic (Bayesian) continuous time (Nodelman, et. al., 2002). These
networks. The numbers represent the weight of causal techniques allow direct computation of distributions over
belief that the user associates with each relation between when events of interest may occur. Moreover, they allow
pairs of concepts in the model. The relations and the for automatic focusing of computational resources on
belief numbers are used to build the structure and the those portions of the domain that may undergo rapid
conditional probability tables of the Bayesian networks by change. Larger, unified models over domains which
combining the weights of causal belief of incoming parent include variables with widely divergent rates of change
4
can thus be made computationally tractable. Machine
learning algorithms can be used to help discover
underlying structure in context where connections within
the data are poorly understood (Nodelman, et. al., 2003).
There are already, in the literature, reviews of the
advantages of these methods over traditional discrete-time
probabilistic models--for instance, showing that the
discrete-time models are subject to artifacts from the fixed
time granularity and are less efficient to learn than the
continuous time models (Nodelman, 2007). Adoption of
these methods has been slow due to lack of exposure,
limited software support, and ongoing research and
development.
We define the expected-time-to-effect as one additional
quantity to be provided by the user for each concept-pair
relation defined in the model (Figure 3). Just as the weight
of causal belief is used to create conditional probability
tables in Bayesian networks; the addition of expected-
time-to-effect is used for the construction of transition
matrices in underlying continuous-time Bayesian
networks. In this manner the system also can forecast the
timing of events.
5.3 IDENTIFICATION OF CONCEPTS
RELEVANT TO A QUERY
Experimental psychology experiments show that a critical
number of variables is needed to make predictions at a
fixed level of accuracy. Adding more variables to the
Figure 4: Creation of a predictive model (DAG)
decision increases the expert’s confidence, without
necessarily improving the accuracy of the prediction
(Oskamp, 1965)(Shepard , 1964). This emphasizes the
importance of being able to determine the critical,
relevant concepts associated with a specific query. This
5.4 INDIVIDUALS VS. COLLECTIVE
JUDGMENT – CONSENSUS VS.
important feature of our decision facilitation system is
DISENTING OPINIONS
based on research done on relevance and feature selection
learning algorithms (Druzdzel and Suermondt., 1994)(Fu A system that offers such model building flexibility and
and Desmarais, 2008b). quick turn-around in decision-facilitation and forecasts
can be equally effective for use by a single analyst as well
Querying the model triggers a prediction. A query is a
as by a collective group of decision makers. Difficult and
request for predicting the future state of a ‘target’ concept
significant decisions are often arrived at by consensus in a
given the assumptions about the current state of a set of
group setting. Collective consensus is often built around a
‘source’ or ‘trigger’ concepts. The model can contain any
particular set of assumptions, a hypothesis, and a
number of concepts and associations, but for each query
prediction. Once this occurs, it becomes increasingly
the ‘source’, the ‘target’ and the set of ‘relevant’ concepts
difficult to deviate from the consensus opinion.
are the critical set of concepts that matter.
Consensus can often be dominated by a vocal minority
Once the model is complete the system is ready for within a group at the risk of ignoring dissenting but
inference and prediction, based on a specific query. At the equally, or more, valid alternative opinions.
time when the query is made, the system identifies the
In our system, various parallel hypotheses can be
variables and relations relevant to that query and that
formulated with ease and subject to different sets of
subset of the unconstrained model is converted into a
assumptions. With our proposed approach, a single team
predictive model (Figure 4).
member may be capable of quickly making predictions
and forecasting scenarios based on a dissenting
hypothesis, while at the same time compiling evidence
that can be used to steer the consensus opinion in a
5
different direction that may, convincingly, lead to a because it narrows the search for specific information as
different and possibly better decision. evidence for clearly stated assumptions; thus, lending
credibility and validity to the predictions. Precise concept
6 FORECAST VALIDATION AND definitions and rationale that explain concept relations
EXPLANATION together with gathered evidence from search make it
possible to support a hypothesis-driven prediction. In
In (Lacave and Díez, 2002), the authors review various arriving at critical decision, the facilitation methods
explanation methods for Bayesian networks and argue, on discussed can help users step through a process that helps
the one hand, that the normative approach for building capture knowledge and data, organize them, invoke
expert systems, based on probabilistic reasoning, leads to analysis methods to forecast predictions, piece together
more robust and accurate results. On the other hand, they evidence, and rationale for or against courses of action,
also require more explanation capability because the and make actionable recommendations. The final choice
methods are more foreign to human beings than in the of action must be ultimately made by humans. The system
heuristic approach. will compute and present the necessary trade-offs
between risk and cost for each recommended course of
Here, we are suggesting that much of the information to
action.
be presented as explanation during inference and
prediction can be captured upfront, during the model Retroactive historical analysis constitutes another
building phase. It is part of the contextual knowledge validation approach. It entails making predictions of past
imparted by the user as concepts and relations are defined. events and comparing the model forecast to actual
A very specific context and specific assumptions are outcomes. Predictions can also be compared among
made for every concept and causal relation defined in the different methods.
unconstrained model. The weight-of-causal- belief and the
expected-time-to-effect are quantities also defined subject 6.3 RAPID PROTOTYPING - ‘WHAT IF”
to very specific assumptions. The system allows for user SCENARIOS
to systematically add such context during the model
creation phase. The information is organized so that it is Being able to quickly build complex mental models, and
readily available for retrieval at the time that the having the underlying machinery to automatically convert
explanation is needed. the created entities and relations into analytic models to
make immediate predictions, provide single or multiple
users with great flexibility. A single user can in one
6.1 EXPLANATION AND CHAIN OF
sitting use their knowledge to build a complex model,
REASONING
define concepts and relations, document their rationale,
Predictions of highly probable events, of high impact and query the model to make predictions, and search for
possibly occurring in the near future will be of most evidence to validate a new hypothesis. A quick turn-
interest to users. Before courses of action are decided, around decision facilitation method like ours enables
decision makers require explanations that support a users to postulate various ‘what-if’ scenarios and test
particular prediction and its assumptions. In addition, they parallel hypotheses side by side.
also require convincing evidence that can back the
predictions with plausible and believable facts. 6.4 AUTOMATED DOCUMENTATION AND
Qualitative explanations are provided by showing the SUMMARIZATION
causal chains of reasoning from trigger assumptions to
predicted target outcomes where the entire relevant Our system automatically compiles and packages all the
context that was captured during model building is information needed for a strategic decision by
organized and presented in every step along various paths summarizing the hypothesis and its assumptions, together
of reasoning. Rationale captured and documented by the with the associated evidence, the forecasts and the chain
users, together with evidence retrieved from document of reasoning explaining the prediction in the context of
search, constitute the basis for the explanation given to the specific concept definitions and the assumptions made
decision makers associated with forecasts and predictions when causal relations were defined. This capability to
by the system. provide a summary documentation of the prediction, the
assumptions and its explanation can be made available to
second parties for critique and revision before actions of
6.2 INFORMATION RETRIEVAL-
significant consequence are taken.
VALIDATION THROUGH SEARCH
In political, cultural and socio- economic domains,
validation often comes from validated evidential facts.
Having a hypothesis makes the search more efficient
6
7 SUMMARY
Lacave, C. and Díez, F. J. (2002). "A review of
Requirements for a decision facilitation system are explanation methods for Bayesian networks", Knowl.
presented that describe human- machine interface Eng. Rev., vol. 17, p.107 , 2002.
concepts that simplify for users the creation of complex
domain models, while making transparent the analytic Nodelman, U., Shelton, C. R., and Koller, D. (2002).
methodology that requires additional, specialized Continuous Time Bayesian Networks. Proceedings of the
expertise. Those simplifying features are built into the Eighteenth Conference on Uncertainty in Artificial
user-interface to help users step through the creation of a Intelligence, pp. 378-387, 2002.
model, query the model to make predictions, formulate
hypotheses and validate the prediction from searched Nodelman, U., Shelton, C.R., and Koller, D. (2003).
evidence (for or against) retrieved from a large corpus of Learning Continuous Time Bayesian
documents. Explanation to predictions combines the Networks. Proceedings of the Nineteenth Conference on
rationale captured from the user during model Uncertainty in Artificial Intelligence, pp. 451-458, 2003.
development and the evidence gathered in support of a
hypothesis; and it is presented to decision makers in Nodelman, U. (2007). Continuous Time Bayesian
context along the various paths of causal inference. Networks. Ph.D. Dissertation, Stanford University. 2007.
Acknowledgments Oskamp, S., (1965). Overconfidence in Case-Study
Judgments, Journal of Consulting Psychology, 29, 1965,
Very special thanks to Uri Nodelman for his insightful pp. 261-265, 1965.
comments and feedback; and for his valuable contribution
in the area of temporal reasoning. Shepard, R. N. (1964). On Subjectively Optimum
Selection Among Multi-attribute Alternatives, in M. W.
References Shelly, II and G. L. Bryan, eds., Human Judgments and
Optimality, New York: Wiley, 1964, p. 166, 1964.
Druzdzel, M., Suermondt, H. (1994). Relevance in
Probabilistic Models: “Backyards” in a “Small World”. In Seidler, W., Kipersztok, O., and Wright, K. (2010).
Working Notes of the AAAI 1994 Fall Symposium Series: "Tools to Identify Nuclear Terrorist," Journal of
Relevance, pp 60-63, New Orleans, Louisiana, Nov 4-6, Radiation Effects, Research and Engineering, Vol 28,
1994. No.1, pp 122 - 133, February 2010.
Fu, S. K. and Desmarais, M. C. (2008b). Tradeoff Wang H. (2004). Building Bayesian Networks:
Analysis of Different Markov Blanket Local Learning Elicitation, Evaluation, and Learning. Ph.D. Dissertation,
Approaches. In Proceedings of Pacific Asian Conference University of Pittsburgh, 2004.
on KDD (PAKDD), 562-571, 2008b.
Heuer, R. J. Jr. (1999). Psychology of Intelligence
Analysis. Center for the Study of Intelligence, Central
Intelligence Agency, 1999.
Kipersztok, O. (2007). “A Tool that Uses Human Factors
to Simplify Model Building and Facilitate More Accurate
Strategic Decisions”. Fourth Bayesian Modeling
Applications Workshop at the Uncertainty in AI
Conference, Vancouver BC, Canada, July, 2007.
Kipersztok, O. and Wang, H. (2003). Validation of
Diagnostic Models Using Graphical Belief Networks. In
Intelligent Systems for Information Processing: From
Representation to Applications. B. Bouchon-Meunier, L.
Foulloy, R.R. Yager. Elsevier, 2003.
Koller, D., and Sahami, M. (2006). Toward Optimal
Feature Selection. In Proceedings of International
Conference in Machine Learning (ICML), 284-292, 2006.
7