=Paper=
{{Paper
|id=Vol-268/paper-5
|storemode=property
|title=User-Centered Methods for Rapid Creation and Validation of Bayesian Belief Networks
|pdfUrl=https://ceur-ws.org/Vol-268/paper5.pdf
|volume=Vol-268
|dblpUrl=https://dblp.org/rec/conf/uai/PfautzCCKCR07
}}
==User-Centered Methods for Rapid Creation and Validation of Bayesian Belief Networks==
https://ceur-ws.org/Vol-268/paper5.pdf
User-Centered Methods for Rapid Creation and Validation of
Bayesian Belief Networks
Jonathan Pfautz, Zach Cox, Geoffrey Catto, David Koelle,
Joseph Campolongo, Emilie Roth
Charles River Analytics
625 Mt. Auburn St.
Cambridge, MA 02138
Abstract However, the expression of a model (i.e., creating a “view
of that model”) by either a knowledge engineer or a
domain expert using most current tools is relatively
Bayesian networks (BN) are particularly well cumbersome, since the full representation of Bayesian
suited to capturing vague and uncertain network computation is beyond the mathematical
knowledge. However, the capture of this sophistication of many domain experts and some
knowledge and associated reasoning from human knowledge engineers (who, in some cases, may have
domain experts often requires specialized backgrounds in Cognitive Science and Psychology, rather
knowledge engineers and computational than Mathematics or Computer Science). Even with an
modelers responsible for creating BN-based effort to reduce the complexity and power of a BN
models. Through our experiences in applying representation, there is still the problem of constructing
BN modeling techniques across application conditional probability tables (CPTs). Since the number
domains, we have analyzed how these models of distributions that must be expressed in a CPT grows
are constructed, refined, and validated with exponentially in the number of parents and the number of
domain experts. From this analysis, we have states per parent, a knowledge engineer or domain expert
identified potential simplifying assumptions and can end up responsible for expressing a vast number of
used these to guide the design of computational what may end up to be meaningless separate distributions
and user interface methods that support the rapid (e.g., a child with 3 states and 7 parents that have 4 states
creation and validation of BN models. each would require the entry of 49,152 probabilities!).
Clearly, this exponential explosion, combined with the
underlying sophistication of the representation, presents a
1. INTRODUCTION & MOTIVATION challenge.
A Bayesian network (BN) (Jensen, 2001; Pearl, 1988) is a To some degree, the problem has been recognized by the
probabilistic model used to reason under uncertainty. BNs research community and the subsequent development of
have been used by the authors, their colleagues, and “canonical models” that reduce the number of parameters
numerous other researchers to reason about a wide variety needed to specify a CPT. A canonical model (Diez &
of phenomena (e.g., computer vision (Rimey & Brown, Druzdzel, 2006) makes a specific assumption about the
1994), social networks (Koelle et al., 2006), human type of relationship between a node and its parents. This
cognition (Guarino et al., 2006; Glymour, 2001), and assumption results in many fewer parameters being
disease detection (Pang et al., 2004)) with notable needed to specify an entire CPT. There are many types of
success. These successes have led to increased interest in canonical models used in practice and each assumes a
BNs as computational methods for domains where the different relationship. Noisy-OR (Henrion, 1989; Pearl,
representation of expert reasoning and knowledge is 1988), Noisy-MAX (Diez & Galan, 2003), and Influence
paramount. For example, a knowledge engineer may Networks (IN) (Rosen & Smith, 1996a; Rosen & Smith,
wish to capture an expert’s reasoning about how incoming 1996b) are three commonly used canonical models.
information should be classified based on the source of However, the assumptions underpinning these canonical
the information and associated environment factors (prior models may make them less generalizeable, or may
work has discussed issues with this elicitation introduce other inconsistencies that decrease their utility.
(Tabachneck-Schijf & Greenen, 2006; Mathias et al., Furthermore, it is unclear what advances in user interface
2006)). Or, an expert may want to directly externalize his designs have been made as a result of these models.
or her reasoning so that it could be encoded and
automated. In this paper, we describe our efforts to understand how
knowledge engineers and domain experts express their
reasoning and knowledge with the goal of using this is focused on diagnostic tools for printers, it can be used
understanding to select simplifying assumptions for the in other domains that follow similar constraints. By
construction and execution of Bayesian networks. From asking the domain expert natural language questions,
these assumptions, we have constructed both an imposing strict constaints on the resulting BN structure,
underlying computational representation as well as a and eliciting reverse parameters (the probability of a
prototype user interface to further explore the degree to cause given the effect), they greatly simplify the
which users can more rapidly express their reasoning, as elicitation of both structure and parameters. Kraaijeveld
well as the degree to which those assumptions may be et al., (2005) also present their GeNIeRate system that
violated. This approach varies from that of Tabachneck- aids in eliciting diagnostic BNs from experts. They
Schijf et al., (2006), who proposed a more formal user- simplify the elicitation by assuming the BN has only three
centered process for knowledge engineering, in that our levels of variables and that all nodes use the Noisy-MAX
end goal is to create both computational and user interface canonical model. This approach removes the value of
methods to allow for both knowledge engineers and showing an expert an interactive graphical model where
domain experts to rapidly create, validate, refine, and they can visualize the reasoning captured as they express
share their own and others’ reasoning. We recognize that it.
from a particular set of requirements, many possible user Neil, Fenton, & Neilsen (1999) build on recent research in
interface designs are possible; in this paper, we present
Object-Oriented software design and Object-Oriented
our overall approach to design and development, and BNs to come up with a process similar to the spiral
describe the resulting methods only to support our process of software engineering for building complex
approach. In Section 2, we describe relevant background
BNs from smaller BN parts. They define five of these
material. In Section 3, we introduce our approach to parts, called idioms, in detail and describe how to use
analysis of user cognition and decision-making across them in practice to build large BNs. A process-based
different application domains. In Section 4, we outline
approach to simplifying BN construction is indeed useful,
our results from these many analyses and their and contrasts with our goal of enabling a domain expert to
implications for model construction and validation. In create their own model.
Section 5, we describe the design of underlying
algorithms and supporting user interfaces to address our The Weighted Sum Algorithm (WSA) (Das, 2004)
understanding of how to best support model generation, reduces the number of parameters to specify a CPT from
testing, and refinement. Finally, in Section 6, we discuss being exponential in the number of parents to linear in the
implications for future research and development efforts. number of parents. In WSA, the domain expert need only
specify one probability distribution over the child node’s
states for each state of each parent (instead of one such
2. BACKGROUND distribution for each combination of parent states). This
A review of Bayesian network literature indicates a clear distribution is conditional on the state of the parent and
bias towards mathematical techniques for representation, the most compatible state of each other parent. Therefore,
inference, and learning instead of techniques for eliciting joint effects of parents can still be taken into account
Bayesian networks from experts. The research that does while specifying a small number of parameters. After
exist related to Bayesian network elicitation focuses on specifying these parameters for compatible parent
obtaining the structure (random variables, their states, and configurations and a relative weight for each parent, the
the causal connections between them) of the Bayesian WSA is used to combine them into the full CPT. This
network, the conditional probability distribution method represents yet another approach to solving the
parameters that quantify the network, or both. underpinning mathematical issues with making a causal
network easier to express. However, this work is focused
Nadkarni & Shenoy (2000) demonstrate an approach for on computational efficiency, not the ease with with the
converting a causal map into a Bayesian network. A model can be expressed by a user.
causal map is similar to the structure of a BN, where an
expert draws directed links between concepts to represent Helsper et al. (2005) describe a method for eliciting
causal relationships. Their approach focuses on elicitation qualitative information about the probabilistic relations in
techniques for the causal map and proper conversion a BN from an expert using a dedicated eliciation
techniques for tranlating it into a BN. They recommend technique, like {Tabachneck-Schijf, 2006 9201 /id}. This
using canonical models (such as Noisy-OR and Noisy- qualitative infromation is then used to constrain the
AND) to simplify parameter elicitation for nodes in the probabilities learned from a small set of data that
resulting BN with multiple parents. Without easy otherwise would be too small to provide accurate learned
conversion between the causal map and the BN (and, probabilities. Again, these approaches are important in
indeed, bi-directional conversion), then the ability of an cases where dedicated knowledge elicitation is the correct
expert to rapidly test and validate a BN may be hindered. approach, but may be less useful when encouraging
domain experts to express their own reasoning.
Skaanning (2000) presents an automated tool for eliciting
diagnostic BNs from domain experts. While the approach Helsper, van der Gaag, & Groenendaal (2004) list the
three types of effects that parent nodes have on child
nodes: qualitative influence (QI), additive synergy (AS),
and product synergy (PS). The specific values of these
three effects in a BN constrain the actual values of entries
in the CPT. To determine QI, AS, and PS, they propose
using “case cards” which ask the domain expert to order
the parent configurations by causal effect on the child
node. Therefore, an domain expert need only specify an
ordering of parent configurations without any conditional
probabilities. This approach is currently constrained to
nodes with only two states and nodes with only two
parents. This approach also does not generate actual
CPTs; it only constrains the values the CPTs can have
given the higher-level qualitative concepts of QI, AS, and
PS. Wiegmann (2005) provides a review of several
methods for eliciting probabilities from domain experts as
well as combining the probabilities elicited from multiple
experts. Their approach in their fielded system is to use
multiple elicitation techniques to improve the accuracy of
the probabilities that the experts specify. These
approaches represent a step towards our stated goal –
understanding the way the user may want to express
causal relations to provide them with a method for
directly expressing those relations into a computational
model. This work, more generally, may help to illustrate
cases where models developed by domain-experts may be
subject to biases or incorrect assumptions about the
underlying comptuation.
3. APPROACH
Our goal across projects has been to identify and study the
users and experts and their approaches to decision-making
in different application domains, with the express purpose
of aiding in that decision making with techniques such as
BN modeling. To accomplish this goal in each project,
Figure 3-1: Cognitive Systems Engineering Process
we use a particular set of analytic methods generally
referred to as Cognitive Systems Engineering (CSE).
While there are many different approaches to the analysis
The CSE community emerged as experience with the components of CSE, such as Cognitive Task Analysis
introduction of new technology demonstrated that (Schraagen, Chipman, & Shalin, 2000), Cognitive Work
increased computerization does not guarantee improved Analysis (Vicente, 1999), Work Centered Support
human-computer system performance (Woods & Dekker, Systems (Eggleston, Roth, & Scott, 2003), and Applied
2000; Roth, Malin, & Schreckenghost, 1997; Woods, Cognitive Task Analysis (Militello & Hutton, 1998),
Sarter, & Billings, 1997). Poor use of technology can these methods share a commitment to analyzing the
result in systems that are difficult to learn or use, can cognitive and environmental demands imposed by the
create additional workload for system users, or in the domain of practice and identifying implications for
extreme, can result in systems that are more likely to lead information, visualization, and decision-support
to catastrophic errors (e.g., confusions that lead to requirements. CSE methods generally entail a multi-
casualties from friendly fire). CSE attempts to prevent phase, iterative design approach that includes an analysis
these types of failures in the design and development of phase, a concept development and prototyping phase, and
complex system by addressing design issues through a user evaluation phase. The cognitive analysis phase
careful analysis of the problem domain, the tasks to be typically employs knowledge elicitation methods such as
performed by a human-computer system, and the interviews of domain practitioners and observations of
limitations of both the human and the machine. work in context. These methods uncover the reasoning
processes involved in making decisions and performing
tasks in the domain and the challenges that arise (Potter,
Roth, Woods and Elm, 2000; Roth and Patterson, 2005).
The analysis phase supports the development of system
requirements that can be used to prototype computational
support tools and user interfaces, including the knowledge the form of the modeling tool itself. That is, the domain
elicitation needed to engineer formalized representations expert could conceivably develop the model herself, if the
of user knowledge and expertise. The development of interface to the model were sufficiently simple (and
requirements and capture of expert knowledge is followed presuming a knowledge engineer was present to provide
by subjective evaluations of a prototype system. These some guidance). This concept of supporting domain
subjective evaluations lead to the development of more experts in expressing their own models has additional
robust prototypes, which are then more rigorously benefits, in that a domain expert has an external record of
evaluated. The results of the evaluation then aid in the their own reasoning, which in turn can be used in a
refinement of the system requirements. This spiral collaborative decision-making process, shared and used as
development process will eventually converge on an the basis for future reasoning, or independently validated
optimal human-computer system (although time/cost by other experts.
constraints will influence the number of possible These main findings were accompanied by more specific
iterations) (Pfautz & Roth, 2005c). This development results about the actual construction of Bayesian networks
process is illustrated in Figure 3-1.
by both our knowledge engineers and the domain experts
with whom we work:
4. RESULTS (1) The exponential growth of CPTs with the number of
In applying this process across numerous projects, we parents and parent states was cited most frequently as
have spent thousands of hours with hundreds of domain a cause for frustration. Few domain experts or
experts across a wide variety of applications, such as knowledge engineers were willing to tackle the entry
supporting the analysis of weather impacts (Lefevre, of extremely large CPTs. Domain experts facing
Pfautz, & Jones, 2005), military intelligence (Pfautz et al., this challenge often resorted to saying, “they’d just be
2006b) and command and control (Pfautz et al., 2006a). making it up” at that level of detail to avoid the task.
In these projects, we have observed some consistent (2) The lack of ability with many BN software packages
patterns in how knowledge is (and can be) elicited by a to develop a network and immediately update beliefs
knowledge engineer to develop models. Similarly, we (as a function of new nodes, states, CPTs, or causal
have observed how experts tend to express their reasoning links) slowed model development.
about the domain. In addition, we have identified ways in
(3) “Evidence” and “belief” were commonly confused
which we could improve our efforts to encode domain
when shown simultaneously. These terms were also
experts’ knowledge and reasoning in BN models, and to
commonly confused in discussions about what a BN
work with domain experts to validate those models. This
was representing.
meta-analysis of our own development process led us to
postulate some key features that may be aided by (4) The expression of vague or uncertain knowledge or
improved computational methods and associated user reasoning varied significantly between users. A
interfaces. great deal of speculation was used in the creation of
the CPTs (more so than the relationships among
The first and foremost finding is that the power of a
variables)
computational representation is seemingly proportionate
to the ease with which expert knowledge can be both (5) The selection of variables was a primary challenge.
encoded and validated. That is, the ability to quickly We discovered that expression could be guided
create a model with domain experts, then to work through towards primarily Boolean and/or Ordinal ranges,
a set of cases within that model allows for a very rapid although in some cases, Categorical variables were
cycle of elicitation, representation, and validation. used to simplify the network (“Categorical” refers to
Rather than the model being reliant on the knowledge non-Boolean, non-Ordinal groups such as {apples,
engineer’s ability to ex post facto recall and extrapolate oranges, jet skis}). It was possible to create networks
from the statements of the domain expert, the model can that represented identical reasoning, but that used a
be more comprehensively and reliably based on expert different selection of variables and variable types.
knowledge and reasoning. An additional benefit to this (6) Many networks were constructed with the
ability to rapidly cycle through model development is that assumption that parents were independent, or when
time with domain experts is typically constrained, and parents had dependent influences, variable names and
that multiple sessions would be otherwise required to states could be relatively easily re-defined to preserve
develop, then refine, then validate a model. parental independence. Working towards networks
A second finding was that while domain experts may not with parental independence may also have helped to
be conversant in computationally sophisticated more explicitly represent un-stated or assumed
technologies, they can be systematically guided to express influences among variables.
their own knowledge and reasoning. This systematic (7) In some cases, variables were selected such that they
guidance can come in the form of a knowledge engineer caused wholesale disregard for other variables (e.g.,
performing a structured interview, but it may also come in
“if A, then I don’t care about anything else, the methods that leveraged the results of our analysis and the
answer is True”.) power of the simplified computational model. Both of
(8) The underlying reasoning, while mathematically these components are described in more detail below.
correct, could be opaque and non-intuitive
(particularly when certain non-linearities were 5.1 CAUSAL INFLUENCE MODELS
exposed). This led to issues of trust in the model and
the modeling technique. One of the key findings from our analysis was that experts
tend to express the degree to which certain factors
These findings are the result of our analysis and influence the likelihood of other factors independently
experience, not a comprehensive, empirical set of (and, where they do not consider these factors
evaluations. However, we would assert that they are independently, the act of explicitly expressing these
highly valuable observations to be used as part of an interdependencies leads to a model where new factors are
effort to build better BN-based modeling tools and will, created to represent the dependency) (see (6) above).
as part of our iterative approach to development, be Therefore, we started with the simplifying assumptions
subjected to more formal evaluation to test the limits of that each parent node influences the child node (causes it
their application in BN modeling. to be more or less likely) and these parents act
independently. These assumptions, along with a
procedure for combining parent influences into a full
5. APPLICATION OF RESULTS CPT, lead to the Causal Influence Model (described in
The results of this analysis were used in the development full detail in (Cox & Pfautz, 2007).
of Charles River Analytics’ BNet™ suite of products In our CIM, the user first specifies a baseline probability
(note: other BN tools (e.g., Elvira, GeNIe) may also distribution over the states of the child node. These
perform a subset of features we developed – our goal in baseline probabilities represent the a priori likelihood of
this paper is not to perform a product comparison, but the child states, without the effects of any of its parents.
simply to show how an analysis of common problems Next, the user specifies the influence that each parent
should systematically lead to user interface design state has on each child state. This influence is a number
elements). Each of the findings has resulted in a in the range [-1, +1] and represents the amount that the
particular feature or set of features. For example, our parent state increases or decreases the baseline probability
finding that users typically wanted to see updates to of the child state.
beliefs whenever any type of change was made to the
network led to the development of a “no-compile” or Once the user specifies the baseline probabilities and
“mode-less” user interface (see (2) above). Other user influences, they are used to calculate all of the
interface methods were used to simplify, where possible, probabilities in the CPT. For each row and child state in
the completion of CPTs (e.g., supporting entry of multiple the CPT, we combine the influence of each parent state in
rows simultaneously, supporting cut and paste from that row on the child state into the overall parent
spreadsheet applications (see (7) above)). However, while influence. While many combination functions are
these improvements to BN modeling clearly improve the possible, we typically use the mean of the parent
user experience for an experienced knowledge engineer, influences since it is a linear function causing positive and
they fail to address the broader goal of making BN model negative influences to balance each other out and
construction a more interactive process with domain therefore produces CPTs that represent linear relations
experts to whom the subtleties of BN representations are (see (8) above). This overall parent influence is then used
not as immediately important as the construction and to either increase or decrease the baseline probability of
validation of the model. the child state, and this result is used as the conditional
probability of the child state given the parent states in that
Therefore, we focused on two key components of a BN row.
modeling tool that would support a more rapid model
creation and validation cycle with domain experts and The CIM requires a number of parameters that is only
knowledge engineers, with the greater goal of working linear in the number of parents, as opposed to exponential
towards methods that would allow domain experts to for a CPT (addressing (1) above). The baseline
comfortably and intuitively externalize their knowledge probabilities and influences are also easily specified by
and reasoning. The first component consists of an domain experts since they do not involve the joint effects
underlying mathematical representation (Causal Influence of numerous parent states. In our experience, a domain
Models, or CIMs) that is based entirely on BNs, but uses expert is much more likely to accurately specify a small
simplifying assumptions much like the canonical models number of simple parameters. This is the primary benefit
described in Section 1. Compared to these canonical of using the CIM instead of a full CPT. Of course, the
models, these simplifications are based primarily on the user can always further refine the actual CPT later, by
desire to support improvements to the user interface, hand or with data, meaning that the full representational
rather than computational efficiency. The second capabilities of the BN model are accessible as needed.
component of our effort was to develop user interface
While the IN canonical model (Rosen & Smith, 1996c) network after evidence has been posted), although others
also provides these same benefits, the CIM overcomes the have developed similar user interface designs to achieve
IN’s two primary drawbacks. First, the CIM can be used this capability (e.g., Elvira: http://www.ia.uned.es/~elvira)
with any discrete node, regardless of the number or Next, we addressed the need to develop methods that
meaning of its states, while the IN is restricted to Boolean
render nodes differently as a function of the type of
nodes only. This capability was based on our observation variable they represent (see (5) above). The following
that, while experts can be guided to formulate their figures show the prototype node renderers for Boolean
reasoning in purely Boolean terms, it is often more
(Figure 5-1 and Figure 5-2), Ordinal (Figure 5-3 and
representative or concise to support Ordinal and Figure 5-4), and Categorical (Figure 5-5 and Figure 5-6)
Categorical terms. Second, the IN model uses a non- node types:
linear function to combine parent influences which has
been known to produce unintuitive CPT entries in certain
cases (e.g., where beliefs become overly sensitive to
evidence from a particular parent). The CIM uses a
simple linear function to combine parent influences, thus
avoiding this problem.
For each parent, the user must specify a number of Figure 5-1: A Boolean node
influences equal to the product of the number of parent
states and child states for the CIM. While this is certainly
better than the CPT, it can still result in too many
parameters for a user to specify. We simplify the CIM
even further by making assumptions about the types of
the parent and child nodes and the type of influence that
the parent has on the child. These assumptions allow us
to completely specify the CIM using only a single
parameter for edges between Boolean and Ordinal nodes, Figure 5-2: A Boolean node expanded to show the
and for edges connected to a Categorical node only a evidence/baseline slider
single parameter for each state of the Categorical node is
needed. For example, in a network with only Ordinal or
Boolean variables, a child with 7 parents would require
the entry of only 7 values. In a full BN representation, the
user would need to specify a minimum (assuming only 2
states per variable) of 128 values.
5.2 USER INTERFACES Figure 5-3: An Ordinal node
The CIM, and the underpinning assumptions that it uses,
is based on the goal of improving interfaces to BN models
to the level where they can be dynamically and
interactively created with or by domain experts. This goal
has led to the development of a number of specific user
interface methods that have been incorporated into our
BNet™ product suite for further evaluation and
refinement. These methods are described in more detail
below.
First, we addressed the need to allow for dynamic Figure 5-4: An Ordinal node expanded to show the
updating of the model as new links, variables, states, and evidence/baseline slider
evidence are created (see (1) above). This is supported by
the underlying computation, but results in a network that
actively animates as it changes. For example, a user
adjusting an evidence slider will see all of the beliefs in
the networks move to indicate the change. This
animation, in particular, supports current efforts to
understand how best to visualize causality (Ware, 2000).
We have also developed methods to help draw attention Figure 5-5: A Categorical node
to changes in a network, so that many variables may be
simultaneously monitored for change (e.g., through
“snapshots” of the beliefs in the network compared to the
environments, but may be used in the refinement of initial
models developed with an interface permitting more
limited expressivity.
Figure 5-6: A Categorical node expanded to show the
evidence/baseline sliders
Figure 5-7: Methods for expressing confidence intervals
We attempted to address the confusion about “beliefs” on posted evidence and baselines for individual states
versus “evidence” that we observed with many of our
domain experts by visually separating these two elements
In addition to using our experience with how experts may
(belief being represented by the vertical bar, evidence via
tend to express their knowledge and reasoning to guide
the horizontal slider) (see (3) above) and only expanding
the design of node visualization and user interface
the node to show the evidence slider(s) when the user
methods, we also developed methods for exploiting the
hovers the mouse over the node. This also provided
CIM’s ability to more rapidly quantify causal relations.
additional affordances (or lack thereof) to show that
Because the CIM allows the influence between Boolean
evidence is something that could be entered by the user.
and Ordinal nodes to be expressed as a single number, we
The expanded nodes in Figure 5-2, Figure 5-4, and Figure
developed the interface shown in Figure 5-8.
5-6 show an evidence knob above the slider track and also
a baseline knob below the track. The baseline knob can
be removed to simplify the user interface, if necessary.
The Boolean node simply displays the belief for the true
state in a vertical bar and provides a slider that allows the
user to (optionally) set the evidence between true and
false. The Ordinal node (shown with 4 states to represent
the intermediate steps from “none” to “full”) necessarily
breaks the vertical belief bar into segments to represent
the distribution across the states. A straightforward
defuzzification of these results has also been developed to
represent this belief as a non-segmented bar. The user
can (optionally) set the evidence using a single slider that Figure 5-8: Interface for manipulating the strength of a
is fuzzified into values for each state. The Categorical causal relationship between variables
node displays the belief for each state in its own vertical
bar and provides a separate evidence slider for each state. In this case, the user has moused-over the link between
We are continuing to prototype additional visualization the two nodes, and buttons have appeared allowing the
and user interface methods for all of these nodes. user to change the excitatory or inhibitory nature of the
We have also developed methods for expressing link. By adjusting this to various ends of a discrete range
probability ranges and baseline probabilities in an attempt (or a continuous range using a slider rather than the two
to address (4) above (Figure 5-7), although we believe buttons shown) for each parent, the user completely
these methods may add a level of complexity that may specifies the CPT for the child. In the cases where a child
inhibit usability and have developed versions of the or parent variable is Categorical, additional buttons
interface that omit these methods. Given that experts may appear for each state. This interface becomes clumsy for
struggle to express in any numerical form their Categorical-to-Categorical variable relationships, as the
uncertainty about evidence, we developed an interface relationship between each child state and each parent state
that allows the user to “grow” a confidence interval must be specified. In practice, however, we have found
around any evidence that is posted. Similarly, we that experts are able to reformulate their reasoning in a
provided a “dual slider” interface where the baseline form that uses primarily Ordinal or Boolean types. In
probability can be expressed. We believe these two addition, we have also developed prototype methods for
additional capabilities (expressing a confidence interval collapsing certain structures of Boolean nodes into
and a baseline) may represent a higher level of Categorical structures (and vice versa).
complexity in the user interface that may be too
sophisticated for some very rapid model development
Although the CIM allows parent influences to be 6. CONCLUSIONS & FUTURE WORK
specified in the range [-1, 1], our experience has
suggested ±5 steps to be a reasonable level of granularity All of the above interfaces have been implemented and
so we actually elicit an integer in the range [-5, 5] for developed as part of a future release of our BNet™
each parent’s influence on the child (but note that product suite, and as part of many of the custom
establishing the optimal level of granularity with applications we develop for our clients. As a result, we
empirical evaluation remains a goal of future work). In have used these interfaces and will assert that domain
Figure 5-6, we show these levels of link strength and that experts without a significant understanding of the
we adjust the link’s hue, saturation, and width to indicate underlying computational representation can rapidly
its influence (providing multiple, redundant visual cues). externalize and validate their own reasoning more easily
Alternatively, we could allow the user to adjust a than with current off-the-shelf BN modeling packages.
continuous value between [-1, 1] with a slider and We have observed experts, particularly working
continuously vary the link’s visual parameters. collaboratively, use our prototype user interface to
develop sophisticated models with very little intervention
from either knowledge engineers or computational
modelers. While this empowers the domain experts to
create their own models, it also introduces an opportunity
for representational errors that would not otherwise occur
with an experienced knowledge engineer “in the loop”
Figure 5-9: Visualization of link strength and along the lines suggested by (Tabachneck-Schijf et al.,
inhibitory/excitatory influence 2006).
With these encouraging experiences, but without any
By providing this additional visualization, we allow the comprehensive empirical evaluation, we hope to pursue
user to see, at a glance, not only the relationships among additional efforts to resolve some of the user interface and
the variables in the network, but also the proportional computational challenges that have arisen from this effort
strength of those relationships. This imparts, to a degree, (e.g., semantic issues with node names and the natural
the sensitivity of the variables to each other, meaning that mappings of certain colors to certain meanings). We also
differences in sensitivities can be rapidly adjusted to hope to pursue experimental efforts to more formally test
match domain expert expectations (e.g., “I realized I the simplifying assumptions in the CIM, and other
made this a much bigger influence than I should have observation- and analysis-based design choices. In
when I looked at all of these other variables”). We have addition, the user interface designs presented, while
discovered potentially challenging interactions between capturing the desired functionality, will still require
the names of variables and the implications of the red and additional refinement to ensure consistency across
green hues used in the visualization. There may be some variable types (and, ideally, additional customizability).
natural mappings of the color to what the variable is Finally, we are also in the process of developing methods
intended to represent (“Happy” is more green than red, for using the simplified interface to aid in the expression
and “Being Happy” has an inhibitory influence on of cases that can be used within existing BN learning
“Moping”, but the red link may be perceived as causing algorithms.
something to be “less happy”). Similarly, the semantics
of the variable names can interact with the visualization
of causal links (e.g., “Not Happy” has an excitatory Acknowledgements
influence on “Moping”). We have, in cases where the We would like to express our appreciation for the
variable semantics can be more structured, provided invaluable testing and feedback provided by Karen
mechanisms for “flipping” the terminology and associated Harper, Chen Ling, Sam Mahoney, Sean Guarino, and
link colors. Eric Carlson. In addition, we would extend our deepest
An additional challenge faced in a network where experts gratitude to Greg Zacharias for his continued funding and
can easily express their own reasoning is the need for support of our work with Bayesian networks.
tools to support validation. In the application of these
methods across domains, we have discovered that invalid
References
causal chains may be formed simply because it was easy
to do so. As a result, we have developed methods that do Cox, Z. & Pfautz, J. (2007). Causal Influence Models: A
relatively simple translation of the causal paths in the Method for Simplifying Construction of Bayesian
network into text (e.g., variable A with a +5 link to
Networks. (Rep. No. R-BN07-01). Cambridge, MA:
variable B would result in “A has a strong positive
influence on B”). This method appears to be effective for Charles River Analytics Inc.
simple models or models with relatively short causal Das, B. (2004). Generating Conditional Probabilities for
pathways (or for models that follow particular structural Bayesian Networks: Easing the Knowledge
patterns). Acquisition Problem. http://www.arxiv.org [On-line].
Available: http://www.citebase.org/cgi- Meteorology, Oceanography, and Hydrology. San
bin/citations?id=oai:arXiv.org:cs/0411034 Diego, CA.
Diez, F. J. & Druzdzel, M. (2006). Canonical Mathias, K., Isenhour, C., Dekhtyar, A., Goldsmith, J., &
Probabilistic Models for Knowledge Engineering. Goldstein, B. (2006). When Domains Require
(Rep. No. CISIAD-06-01). Madrid: UNED. Modeling Adaptations. In Proceedings of 4th Annual
Diez, F. J. & Galan, S. F. (2003). An Efficient Bayesian Modeling Applications Workshop at UAI
Factorization for the Noisy MAX. International '06. Cambridge, MA.
Journal o Intelligent Systems, 18165-177. Nadkarni, S. & Shenoy, P. P. (2000). A Causal Mapping
Glymour, C. (2001). The Mind's Arrows: Bayes Nets and Approach to Constructing Bayesian Networks.
Graphical Causal Models in Psychology. Cambridge, School of Business Working Paper NO.289.
MA: The MIT Press. Neil, M., Fenton, N., & Nielsen, L. (1999). Building
Guarino, S., Pfautz, J., Cox, Z., & Roth, E. (2006). Large-Scale Bayesian Networks. submitted
Modeling Human Reasoning About Meta-Information. Knowledge Engineering Review .
In Proceedings of 4th Bayesian Modeling Applications Ref Type: Generic
Workshop at the 22nd Annual Conference on Pang, B., Zhang, D., Li, N., & Wang, K. (2004).
Uncertainty in AI: UAI '06. Cambridge, Computerized Tongue Diagnosis Based on Bayesian
Massachusetts. Networks. IEEE Transactions on Biomedical
Helsper, E. M., van der Gaag, L. C., Feelders, A. J., Engineering, 51(10), 1803-1810.
Loeffen, W. L. A., Geenen, P. L., & Albers, A. R. W. Pearl, J. (1988). Probabilistic Reasoning in Intelligent
(2005). Bringing Order into Bayesian-Network Systems: Networks of Plausible Inference. San Mateo,
Construction. In Proceedings of K-CAP'05. Banff, CA: Morgan Kaufmann.
Alberta, Canada. Pfautz, J., Roth, E., Bisantz, A., Llinas, J., & Fouse, A.
Helsper, E. M., van der Gaag, L. C., & Groenendaal, F. (2006a). The Role of Meta-Information in C2
(2004). Designing a Procedure for the Acquisition of Decision-Support Systems. In Proceedings of
Probability Constraints for Bayesian Networks. In Command and Control Research and Technology
Engineering Knowledge in the Age of the Semantic Symposium. San Diego, CA.
Web (pp. 280-292). Berlin: Springer-Verlag Berlin Pfautz, J., Roth, E., Powell, G., Fichtl, T., Guarino, S., &
Heidelberg. Carlson, E. (2006b). Cognitive Complexities
Henrion, M. (1989). Some Practical Issues in Impacting Army Intelligence Analysis. In Proceedings
Constructing Belief Networks. In L. Kanal, T. Levitt, of Human Factors and Ergonomics Society 50th
& J. Lemmer (Eds.), Uncertainty in Artificial Annual Meeting. San Francisco, California.
Intelligence 3 (pp. 161-173). North Holland: Elsevier Rimey, R. & Brown, C. (1994). Control of Selective
Science Publishers. Perception Using Bayes Nets and Decision Theory.
Jensen, F. V. (2001). Bayesian Networks and Decision International Journal of Computer Vision, 12(2-3),
Graphs. New York: Springer-Verlag. 173-207.
Koelle, D., Pfautz, J., Farry, M., Cox, Z., Catto, G., & Rosen, J. & Smith, W. (1996a). Influence Net Modeling
Campolongo, J. (2006). Applications of Bayesian With Causal Strengths: An Evolutionary Approach. In
Belief Networks in Social Network Analysis. In Proceedings of Command and Control Research and
Proceedings of 4th Bayesian Modeling Applications Technology Symposium.
Workshop at the 22nd Annual Conference on Rosen, J. A. & Smith, W. L. (1996b). Influencing Global
Uncertainty in AI: UAI '06. Cambridge, Situations: A Collaborative Approach. US Air Force
Massachusetts. Air Chronicles.
Kraaijeveld, P., Druzdzel, M., Onisko, A., & Wasyluk, H. Rosen, J. A. & Smith, W. L. (1996c). Influence Net
(2005). GeNIeRate: An Interactive Generator of Modeling With Causal Strengths: An Evolutionary
Diagnostic Bayesian Network Models. In Proceedings Approach. In Proceedings of the 1996 Command and
of Working Notes of the 16th International Workshop Control Research and Technology Symposium.
on Principles of Diagnosis (DX-05), (pp. 175-180). Skaanning, C. (2000). A Knowledge Acquisition Tool for
Lefevre, R., Pfautz, J., & Jones, K. (2005). Weather Bayesian-Network Troubleshooters. In Proceedings of
Forecast Uncertainty Management and Display. In The Sixteenth Conference on Uncertainty in Artificial
Proceedings of 21st Int'l Conf. on Interactive Intelligence. Stanford, CA: Stanford University.
Information Processing Systems (IIPS) for
Tabachneck-Schijf, H. & Greenen, P. (2006). Preventing
Knowledge Transfer Errors: Probabilistic Decision
Support Systems Through The Users' Eyes. In
Proceedings of 4th Annual Bayesian Modeling
Applications Workshop at Uncertainty in AI '06.
Cambridge, MA.
Ware, C. (2000). Information Visualization: Perception
for Design. New York: Morgan-Kauffman.
Wiegmann, D. A. (2005). Developing a Methodology for
Eliciting Subjective Probability Estimates During
Expert Evaluations of Safety Interventions:
Application for Bayesian Belief Networks. (Rep. No.
AHFD-05-13/NASA-05-4). Aviation Human Factors
Division Institute of Aviation, University of Illinois.