=Paper=
{{Paper
|id=Vol-268/paper-2
|storemode=property
|title=A Tool that Uses Human Factors to Simplify Model Building and Facilitate More Accurate Strategic Decisions
|pdfUrl=https://ceur-ws.org/Vol-268/paper2.pdf
|volume=Vol-268
|dblpUrl=https://dblp.org/rec/conf/uai/Kipersztok07
}}
==A Tool that Uses Human Factors to Simplify Model Building and Facilitate More Accurate Strategic Decisions==
https://ceur-ws.org/Vol-268/paper2.pdf
A Tool that Uses Human Factors to Simplify Model Building and
Facilitate More Accurate Strategic Decisions
Oscar Kipersztok,
Mathematics & Computing Technology
Boeing Phantom Works
P.O.Box 3707, MC: 7L-44
Seattle, WA 98124
oscar.kipersztok@boeing.com
Abstract summarizes those human factors and
processes and investigates how those can be
This paper describes a tool to validate used for building systems or computational
hypotheses in strategic decision making. The
system builds on experimental evidence of
tools that facilitate better decision making.
human factors that lead to more accurate There has been a recent surge in technology
decisions. The paper shows how those
used to support decisions. Graphical
factors are translated into specifications for
a user interface that simplifies the capture of probabilistic networks (Bayesian networks),
expert knowledge to create complex models for example, are used to model large
of strategic domains of interest. The tool complex domains from the aggregation of
uses graphical probabilistic reasoning, smaller, local, probabilistic dependencies
combined with text classification methods, to
between the variables of a domain [Pearl,
expedite the search for relevant information
in a large number of documents to help 1988; Lauritzen and Spiegelhalter, 1988].
validate hypotheses. The methodology uses efficient probability-
update algorithms that guarantee consistent
1 INTRODUCTION and correct propagation of uncertainty when
This paper describes a tool that facilitates the models are queried. In Data Mining
strategic decision making, i.e., decisions [Cabena, et. al., 1998], algorithms are used
that, if wrong, can have significantly to search for information in very large
negative or even catastrophic consequences. databases to discover patterns and trends
The questions addressed are what type of that can explain new phenomena. For
information decision makers need to know, example, for strategic decisions,
and how they process such information to classification algorithms [Langley, 1996] are
arrive at decisions? Are there processes that contributing to the ability to retrieve
lead to better decisions? What specific pertinent information in support of critical
quantities do decision makers need to be decisions. In many instances, not having the
able to make decisions? right information available at the right time
can have serious consequences. The large
There is evidence from experimental amounts of information that can be accessed
psychology that suggest trends and today make it difficult for decision makers
behaviors that, on the average, result in
more accurate decisions. This work
to search for specific information that may into a decision process, increases the
be needed for time-critical decisions. confidence of the decision maker without
necessarily increasing the accuracy of the
In this paper, a prototype tool is presented
decision. People tend to believe that they use
that uses human factors found to lead to
considerably more information in their
more accurate decisions. The system uses
decisions than they actually do [Summers et
graphical probabilistic reasoning, combined
al., 1970; Slovic and Lichtenstein, 1971].
with text classification methods, to facilitate
People compile vast amounts of information,
the search through a large number of
over a life time of making decisions.
documents. It allows the capture of user’s
Although it may appear that all that
knowledge in a straight forward manner to
information is brought to bear in each new
create complex models of domains of
decision; actually, only a relatively small
interest. These are also referred to as
amount is relevant for each individual
“mental models” of the domain.
decision [Shepard, 1964].
As the user builds a model, a graphical
Over time, people develop “mental models”
probabilistic network is automatically
of the world that are comprised of ideas and
created that allows the user to raise
concepts learned, and the relationships
hypotheses by making queries to the model.
between them [Heuer, 1999]. Such mental
At the same time a text classifier is also
models become the “filter” through which
created for search and retrieval from a large
people interpret the world around them.
volume of documents. Once built, the user
There is a continuous refinement of the
queries the model and the system responds
mental model based on new experiences
by predicting the likelihood of a hypothesis.
gained that reinforce certain relationships
In prediction-driven mode, the classifier will
and weaken others.
search for relevant documents that can be
used to substantiate or negate a given People tend to utilize two distinct modalities
hypothesis. In evidence-driven mode, the for decision making. One modality is
evidence is retrieved ahead of formulating referred to as the “mosaic” model [Heuer,
the hypothesis. In both cases the search is 1999] where pieces of information are
driven by the most likely hypotheses or the accumulated, in no particular pattern, on the
most likely evidence. The tool creates a belief that a unified picture will, eventually,
summary report describing the formulated emerge revealing the solution to a problem
hypothesis and the evidence gathered to query. The emphasis is in accumulating as
substantiate or refute it. much information on the subject of interest
as possible.
This paper shows that it is possible to embed
in the GUI design features based on the The second modality is using the mental
human factors mentioned to guide the model to create a hypothesis for the query
operational use of the tool to flow in a and then use the hypothesis to guide the
manner that facilitates more expedient and retrieval of information. The search for
effective strategic decision making. information is driven by the model and the
information retrieved is used to validate,
2 HUMAN FACTORS FROM refute, or reformulate the hypothesis.
EXPERIMENTAL PSYCHOLOGY Experimental psychology provides evidence
Evidence from experimental psychology that the latter is the modality that tends to
[Oskamp, 1965; Goldberg, 1968; Heuer, conduce to more accurate decisions [Elstein,
1999] shows that factoring more information et. al., 1978]. The process of scientific
2
discovery, for example, follows the second knowledge engineer. The communication
modality where theories are created about language of the system with the user is
unexplained phenomena and experiments limited exclusively to the language of the
are designed to validate or refute the user’s domain of interest. All technical
theories. In turn, the data are also used to terminology is hidden from the user. The
improve theories and the process continues user defines the domain as a list of concepts
where experiment and theory affect each with appropriate labels and defines
other. relationships between them with a weight of
causal belief. The model is captured, as a
The described system uses these results to
directed graph and hidden from the user.
guide the user in the process of making
decisions. Firstly, it allows the user to build The second principle is to allow the user to
a mental model of their domain of interest in attach specific qualitative and quantitative
a fairly simple and unrestricted manner. information to nodes and links in the graph.
Secondly, as the user queries the model, a The intent is to associate the same type of
hypothesis is proposed, and the system information uniformly across all nodes
automatically identifies the essential keeping the amount of added information to
parameters that are relevant to it. The system a minimum. The new information, although
also conducts a search to retrieve of the same type, contains different semantic
information for use in validating or negating meaning for each individual concept or
the hypothesis. The gathered evidence is relation.
used to redefine the query and to propose a
better and more likely hypothesis. This cycle 4 DESCRIPTION OF THE SYSTEM
of operation patterns itself after the human AND ITS USE
factor trends described in this section that
4.1 WHAT STRATEGIC DECISION
can lead to better decisions.
MAKERS WHANT TO KNOW
3 A KNOWLEDGE ACQUISITION Based on documents of strategic decisions
PERSPECTIVE FOR BUILDING and discussions during meetings of strategic
COMPLEX MODELS nature, the observation was postulated that a
common set of prediction parameters that
From a knowledge acquisition perspective,
decision makers are interested are the
the translation of the aforementioned human
occurrence of events and the emergence of
factors into system design specifications
trends. Moreover, for each predicted event
must conform to proper practices of user
or trend, decision makers are interested in
interface design. Domain models that
knowing their likelihood of occurrence, their
facilitate strategic decisions can become
magnitude or impact, and the time when
quite complex. How can such complexity be
they expected to occur.
managed during the knowledge acquisition
stage, was an important question that needed 4.2 BUILDING THE “MENTAL MODEL”
to be addressed.
Specifying the use of mental models in
Two basic principles are defined and applied computing architectures of decision support
to the user interface design to manage the system requires defining methodologies of
building of complex models. The first how such models are to be built. The type of
principle is driven by the goal to create a model suggested here is a cognitive causal
system that will allow direct and iterative model in the form of a directed graph made
interaction with the user, and will not up of nodes representing concepts and links
necessitate the presence of an intermediary
3
representing relationships. Concepts are model enables the system to acquire as
ideas represented by descriptive labels or much information as possible.
phrases that convey a specific meaning or
Figure 1, shows the model building screen
thought that is relevant to the domain of
for building the mental model. The model is
interest. Such model can be built by making
built by defining concepts and their
a list of concepts and identifying, for each
relations. As an example, concepts are
concept, all other concepts that may affect it.
shown for the “Aviation Safety” domain. In
This type of model is referred to as an Figure 2, “Traffic Control Errors” and
“unconstrained model” because there are no “Public concern” represent state transition
restrictions as to the number of concepts or concepts that affect the “Occurrence of
directional relations that can be defined in Accidents”, a discrete event concept.
the graph. Any number of cycles is allowed.
Since not all relations have the same degree
of belief, a “weight of causal belief” is
introduced and attached to each parent-child
relation. Such weight represents the degree
of belief that if the parent is true, it will
influence the child to become true as well.
An arbitrary scale between [0, 1] is used to
assign such weight. In order to make Figure 2- Examples of concepts, relations
inference about proposed hypotheses and and weight of causal belief
make predictions about the domain, the The unconstrained model can be as large
unconstrained model is converted into a and complex as needed. The model is
graphical probabilistic network by reducing captured as a directed graph. Figure 3 shows
the directed graph into an acyclic graph. a simple version of the Aviation Safety
model where the negative signs at the end of
the arrows representing negative weights of
belief.
Figure 3- Simplified graph of Aviation
Figure 1 – Model building screen Safety domain
During model creation the system allows as 4.3 INFERENCE AND PREDICTION
much flexibility for free association, without
To be able to make predictions that decision
much concern about its ability perform
makers are interested, such as the likelihood,
inference. The use of the unconstrained
magnitude and time of expected event and
4
trend occurrences, Bayesian networks are 5 MENTAL MODEL GUIDING
used. The tool computes likelihoods and EVIDENCE SEARCH
probabilities in response to queries. The user As mentioned in Section 2, the iterative
can add information to concepts to define process of using mental models to raise
them in term of dimensional quantities. hypotheses and the subsequent search for
Characteristic time quantities are also added information to help validate them has been
for use in temporal reasoning [Nodelman, et. shown by experiments to result in more
al., 2002]. To transform the directed graph accurate decision making.
into an acyclic Bayesian network, two steps
are necessary. The first step is to eliminate
cycles in the directed graph. This is done by
minimizing information loss as a trade-off
between the full expressiveness of the
unconstrained model and the ability to make
predictive inference. The second step is to
utilize the weights of causal belief to create
conditional probability tables between each
node in the acyclic graph and its parent
nodes. Similar approaches are found in
[Rosen and Smith, 1996]
4.4 QUERY AND HYPOTHESIS CREATION
A query presents a question of how evidence Figure 4- Query response with predicted
or beliefs about current events (or trends) hypothesis
can predict the occurrence of future events
of interest. The current evidence is the state While the user builds the model, the tool
of occurring events and on-going trends. also provides added utilities to facilitate the
Experimental psychology factors described automated building of a text classifier. The
in Section 2, suggests that only a few creation of the classifier is based mostly on
parameters are relevant in addressing a the concept labels and the text descriptions
specific query in support of a decision. To that are attached to the nodes and links of
answer a query the system identifies the sub the model graph.
model from the unconstrained model The tool’s prediction capability using the
containing only those parameters that are mental model allows the user to discover the
relevant to the query [Druzdzel and most likely hypotheses that justify the search
Suermondt, 1994]. The screen in Figure 4 for validating information. The capability to
shows the results of such query with the identify the parameters most relevant to the
corresponding prediction. hypotheses further narrows the space of
The response to the query is in the form of a possible search. And the text classifier built
hypothesis that predicts the likelihood of an using the domain language, introduced by
event or trend given current evidence or the user, helps conduct the search in an
beliefs. The user can simulate various ‘what efficient manner as is shown next. The
if’ scenarios to gain insight into which of the search requires the presence of a large
postulated hypotheses will result in the most corpus of text documents. The text classifier
likely future events. built by the tool is used to rank text content
relevant to concepts and relations associated
5
with the hypotheses that resulted from the hypothesis. In this manner, the user can
query. search through the documents and build a
case by selecting the pertinent document
Figure 5 shows the screen with the results of
paragraphs that help support or reject the
the classification process. Every document
hypothesis.
has been classified into the relevant concepts
for the query. A score is computed by the
classifier for each relevant concept. The
documents are ranked according to content
most relevant to the concepts in the
hypothesis.
Figure 6 - Query response with predicted
hypothesis
Figure 7 shows a screen with the ranked
paragraphs for each specific document. The
Figure 5- Query response with predicted search process is driven, from the top down,
hypothesis by the most relevant content. Tens of
The user can inspect each concept and thousands of documents can be processed in
identify the specific documents that have this manner, considerably reducing the
been associated with it. The screen in Figure search time.
6 shows the specific documents that have
been assigned to a concept. A second score
is also computed to rank the documents with
the most relevant content. Shown on the
right pane of Figure 6 is information about
author, source, title of each document, and a
level of reliability that the user selects for
each source. Such reliability factor is used to
re compute the ranking of each document.
Each document associated with a concept is
further broken down into ranked paragraphs,
as shown in Figure 7.
The highest ranked paragraph contains the Figure 7 - Query response with predicted
content most closely associated with the hypothesis
concept. The user can, quickly, inspect the The last step is summarization of the
most relevant paragraphs of each document evidence gathered from the selected
and select the paragraphs that contain documents and paragraphs which content
evidence to help substantiate or reject the helps to substantiate the hypotheses. The
6
paragraphs selected by the user are compiled Applications to Expert Systems. Journal of the Royal
automatically and captured in a file with the Statistical Society, B, Vol. 50, No. 2, 1988.
postulated hypothesis and the conclusions [Nodelman, et. al., 2002] Nodelman U., Shelton, C.
drawn by the user. R., and Koller, D. Continuous Time Bayesian
Networks. Proceedings of the Eighteenth Conference
on Uncertainty in Artificial Intelligence, pp. 378-387,
6 SUMMARY
[Oskamp, 1965] Stuart Oskamp, Overconfidence in
Experimental evidence from human factors
Case-Study Judgments, Journal of Consulting
studies describe behaviors that lead to more Psychology, 29, 1965, pp. 261-265.
accurate decisions. A prototype system is
[Pearl, 1988] Pearl. J. Probabilistic Reasoning in
presented that utilizes those factors in the Intelligent Systems, Morgan Kaufmann, San Mateo,
design of the user-interface to help the user California, 1988.
step through the creation of a model,
[Rosen and Smith, 1996] Julie A. Rosen and Wayne
presentation of a query, postulation of a L. Smith. Influence Net Modeling with Causal
hypothesis and, lastly, validation of the Strengths: An Evolutionary Approach, Proceedings
hypothesis by expediting the search through of the Command and Control Research and
a large corpus of documents in the context Technology Symposium, Naval Post Graduate
of the most likely evidence. School, Monterey, CA, June 25-28, 1996.
[Shepard, 1964] R. N. Shepard, On Subjectively
References Optimum Selection Among Multiattribute
[Allison, 1971] Graham Allison, Essence of Alternatives, in M. W. Shelly, II and G. L. Bryan,
Decision, 1971, The Case Study Anthology, Editor eds., Human Judgments and Optimality, New York:
Robert K. Yin, Sage Publications, 2004. Wiley,1964, p. 166.
[Cabena, et. al., 1998] Cabena P., P. Hadjinian, R. [Slovic and Lichtenstein, 1971] Paul Slovic and
Stadler, J. Verhees, and A. Zanasi. Discovering Data Sarah Lichtenstein, Comparison of Bayesian and
Mining: From Concept to Implementation. Upper Regression Approaches to the Study of Information
Saddle River, NJ: Prentice Hall, 1998. Processing in Judgment, Organizational Behavior
and Human Performance, 6 ,1971, p. 684.
[M. Druzdzel and H. Suermondt, 1994] Relevance in
Probabilistic Models: “Backyards” in a “Small [Summers et al., 1970] David A. Summers, J. Dale
World”. In Working Notes of the AAAI 1994 Fall Taliaferro, and Donna J. Fletcher, Subjective vs.
Symposium Series: Relevance, pp 60-63, New Objective
Orleans, Louisiana, Nov 4-6, 1994. Description of Judgment Policy, Psychonomic
[Elstein, et. al., 1978] Arthur S. Elstein et.al, Science, 18 , 1970, pp. 249-250.
Medical Problem Solving: An Analysis of Clinical
Reasoning. Cambridge, MA and London: Harvard
University Press, 1978, pp. 270 and 295.
[Goldberg, 1968] Lewis R. Goldberg, Simple
Models or Simple Processes? Some Research on
Clinical Judgments, American Psychologist, 23,
1968, pp. 261-265.
[Heuer, 1999] Richards J. Heuer, Jr., Psychology of
Intelligence Analysis. Center for the Study of
Intelligence, Central Intelligence Agency, 1999.
[Langley, 1996] Langley P. Elements of Machine
Learning. San Francisco: Morgan Kaufmann, 1996.
[Lauritzen and Spiegelhalter, 1988] Lauritzen S. L.,
and D.J. Spiegelhalter. Local Computation with
Probabilities in Graphical Structures and Their
7