Bayesian networks (BNs) are excellent tools for reasoning about uncertainty and capturing detailed domain knowledge. However, the complexity of BN structures can pose a challenge to domain experts without a background in artificial intelligence or probability when they construct or analyze BN models. Several canonical models have been developed to reduce the complexity of BN structures, but there is little research on the accessibility and usability of these canonical models, their associated user interfaces, and the contents of the models, including their probabilistic relationships. In this paper, we present an experimental procedure to evaluate our novel Causal Influence Model structure by measuring users' ability to construct new models from scratch, and their ability to comprehend previously constructed models. [Results of our experiment will be presented at the workshop.]
A Bayesian network (BN)
There have been few user-centered evaluation efforts to assess how (and if) canonical models help domain experts elicit their knowledge and understanding of models presented to them, or how graphical interfaces and their features and properties impact the way people create, interpret, reason with, or base actions on Bayesian networks. The purpose of our study is to provide baseline information on how people construct and describe CIMs presented and created within a graphical user interface.
A canonical model
A review of canonical models sheds light on the advantages and drawbacks of each model. The Influence Network (IN) model can only be used with Boolean nodes. It assumes that the child node has a baseline probability of occurring independently of any parent effects and that each parent independently influences the child to be more or less likely to be true. Since a single baseline probability for the child and a single change in probability for each parent are simple parameters for users to specify, the IN represents a powerful mechanism for capturing domain knowledge. However, since only Boolean nodes are allowed in the IN model, model flexibility is significantly reduced. BNs commonly contain nodes that represent concepts other than the occurrence or non-occurrence of events, and INs cannot be used to simplify these BNs without considerably rearchitecting the model.
The Noisy-OR model is also used only with Boolean
nodes and assumes that a true state in any parent can
cause the child to be true independently of the other
parents, with some uncertainty. Similar to INs, the main
drawback of the Noisy-OR is its limitation to only
Boolean nodes. The Noisy-MAX model generalizes the
Noisy-OR and allows ordinal nodes at the expense of
increasing the complexity of parameters. Although
NoisyMAX does work with ordinal nodes, it cannot be used
with more general discrete nodes that do not have ordered
states. These nodes, referred to as categorical nodes, have
an arbitrary number of unordered states and usually
represent the category or type of something. Qualitative
Probabilistic Networks (QPNs) allow for the construction
of purely qualitative relationships between nodes in a
network, to abstract from the highly quantitative and
numerical nature of typical Bayesian models. QPNs
consider the “signs” inherent in probabilistic relationships
between nodes, and consider the additive synergies
between nodes to capture more complicated probabilistic
relationships between them (i.e., if A and B both have a
positive influence on node C, their influences may be
synergistic in nature: if A and B are both true, their
cumulative influence upon C may be greater than just the
sum of their individual influences.) QPNs allow for more
qualitative model elicitation and may therefore be
appropriate for interactions with non-technical experts,
but they are limited in their ability to provide hard,
numerical estimates of the likelihood of events.
The Causal Influence Model (CIM) is a canonical model
that retains the desirable properties of the IN while
providing solutions to its problems. The CIM assumes
that each node is discrete and has an arbitrary number of
states with arbitrary meaning. Each node has a baseline
probability distribution, independent of any parent effects.
Each parent independently influences these baseline
probabilities to be more or less likely. The CIM also
introduces simplifications that govern the generation of
conditional probability relationships, enabling Boolean,
ordinal, and categorical nodes to be included. A full
description of the mathematical formulas that govern
CIMs, including formulas to translate CIM link strengths
into conditional probability tables, is provided in
Studies have been conducted to analyze and mitigate
complexities that arise in the construction of Bayesian
models as a result of knowledge elicitation
Up to twenty participants are recruited from the university community to perform the study. After providing informed consent, participants are given the Ishihara Test for color blindness. Participants who pass this screening continue with the study.
We have developed an CIM-enabled version of our
BNet.Builder product to allow us to experiment with
graphical interfaces for Bayesian network modeling
To simplify model construction for this particular experiment, the CIM interface has been constrained so that all nodes are Boolean; initial beliefs are set to 0.5 for all nodes and cannot be changed directly by the user (but can change based on evidence or link strengths); and only “hard” evidence can be posted (e.g., evidence that the node was either fully true, or fully false). This represents a set of simplifications we have found useful in other work, particularly among users less familiar with Bayesian modeling techniques. Our main goal in this study is to determine whether participants can reason about previously constructed CIMs and construct models to match a given situation. Since these are specific, novel, and fundamental questions with little previous research behind them, we have started with a simple case. The inclusion of additional node types, in particular, is useful for future work in comparing CIMs to other canonical models such as INs, Noisy-OR, and Noisy-MAX.
Participants will be asked to provide descriptions of and answer questions about a series of CIMs shown in the BNet.Builder interface. In the first task, participants will be shown a model and asked questions about the structure and nature of relationships in the model (specifically, questions asking them to describe elements of the model, and questions related to abductive and deductive reasoning using the model). For instance, given the following example model (Figure 1), participants would be asked: • • •
Description: This picture shows a model of part of a car. Describe what causes headlights to be dim, or not dim.
Abductive Reasoning: If the headlights are dim, what does that mean about the other parts of the car? Deductive Reasoning: The alternator is working. What does that suggest about the headlights? The battery is old. What does that suggest about the headlights? What if the battery is new and the alternator is failing?
In the second task, participants can manipulate the causal links and post evidence to see how changing the strength and directionality of the links between the nodes, and evidence about the state of the nodes, affects beliefs about whether the nodes are true or false. They will respond to similar sets of questions as provided in the first task. Finally, in the third task, participants will be asked to construct models from scratch using the interface based on several different vignettes, such as the following: The headlight system on a car is dependent on two components: a battery, which stores energy to power the lights, and an alternator, which converts mechanical energy from the car’s engine into stored energy in the battery. When the car is running, the alternator “recharges” the battery. This process only works if the alternator is working, and the battery is new.
Four models/vignettes have been constructed for each task (a total of 12). Each model has the following relationships: 1 child/1 parent, 2 children/1 parent, 1 child/2 parents, 2 children/2 parents. In all cases, all children are linked to all parents. Also, in all but the 1 child/1 parent case, one parent-child link is negative. This simplification provides the basis for the initial study. We expect to expand upon this simple representation with later empirical work.
Two stimuli sets are created based on the 12 models. Either the nodes in the models (or phrases in the vignette) are phrased positively, or they include at least one node that uses negative phrasing (e.g., “battery is not new”). This difference allows us to investigate how semantic properties of the model or situation affect task performance. This condition has been inspired by our experience in domain expert interaction with CIM modeling interfaces, where we observed the articulation of variable names as a source of common confusion. The use of negatives in the variable name (e.g., “not raining”) or logical antonyms (e.g., “happy” and “sad”) tends to lead to later confusion in expressing causal relationships (e.g., “if it is not not-raining, then it is unlikely that Rakesh will not bring his umbrella”). By including this specific independent variable, we will be able to assess which specific patterns of reasoning are most difficult for users. Participants are randomly assigned to one of the two stimuli sets (up to 10 participants per condition). This sample size is consistent with those used in usability type tests, and will allow us to analyze verbal protocols of participants to look for patterns across conditions.
Throughout all three tasks, participants are asked to “talk aloud” while performing the task to describe how they are thinking about or creating the models. Screen capture software is used to record participants’ interaction with and construction of models. Participants are also fitted with a view point eye tracker (lightweight glasses that have an attached camera that tracks the corneal movements of the participant’s eye to assess gaze relative to the computer screen they are working on). The eye tracking system is used to record aspects of gaze position and dwell time at a screen location. Time to complete the tasks is also being recorded.
Data from the audio, eye track, and screen capture
processes is combined to create a “process trace” of each
participant’s behavior describing and creating CIMs
The purpose of this study is to provide baseline information regarding how people construct and describe CIM models presented and created within the BNet.Builder interface. There is continued interest in simplifying the manner in which domain expertise is elicited, and the creation and presentation of Bayesian network models through direct manipulation and visualization. However, information on how these tools are used by practitioners, how they affect the models that people produce, and how they affect the way that people interpret models or predict outcomes is missing. We anticipate that users will have more difficulty explaining and constructing models with more parent-child connections. We also anticipate users having more difficulty explaining and constructing models when there are more nodes with negative causal links because of the increase in complexity of the models.
In this study, we intend to measure reasoning patterns involving negative quantities that give users the most trouble. We anticipate that users will have the most difficulty interpreting and creating models when nodes are presented with “negatively phrased” labels (e.g., assessing the influence of a node labeled “battery is not new” on a node labeled “headlights are dim”). If this is the case, it suggests a need for developers of CIMs (and BNs in general) to encourage users to employ certain modeling patterns, possibly by constraining the description of nodes. These constraints, in turn, can be accomplished through prior training or interface wizards, or through intelligent, automatic processing of user entries, and provision of suggested alternatives (e.g., popup suggestions). These interventions could be tested in further studies.
The primary contribution of this paper will be processand product-oriented descriptions of how this graphical tool is used to interpret and create CIMs. Future research could compare how models created within the CIM framework compare to those using more traditional BN structures, from the point of view of the user. This study used simple Bayesian models, with constrained parameters and interaction capabilities, and used only Boolean nodes. Future studies, guided by these initial findings, can be conducted using more complex models, a greater variety of node types (e.g., categorical, ordinal), and allow subjects greater flexibility in manipulating CPTs and posting evidence. Other issues for investigation include measuring and mitigating user tendencies to confuse “evidence” and “belief” (both as terms, and in the values these terms represent), measuring tendencies to disregard parental independence when constructing CIMs, and further observation of user reaction to non-intuitive but correct behavior (e.g., becoming confused when particular variables appear overly sensitive or insensitive to posted evidence.) The CIM interface provides a user-friendly way to express causal influences between nodes, vastly decreasing the number of parameters needed to construct causal models and providing the capability for a much broader base of users to perform Bayesian modeling. Within the experimental interface, participants express relative degrees of influence over a range of 11 steps (from positive to negative 5, with a neutral intermediate value). Additional studies are necessary to clarify the appropriate level of granularity of influence assignment (e.g., 3 steps? 11 steps? 51 steps?) as well whether other methods of assigning strengths across sets of links (e.g., normalized strengths, rank ordered strengths) have merit. Finally, detailed studies with real-world models, situations, and domain experts are required.
We would like to thank David Koelle, Geoffrey Catto, Joseph Campolongo, Sam Mahoney, Sean Guarino, and Eric Carlson for their contributions in the development of the CIM and identifying hypotheses to investigate. We also extend our deepest gratitude to Greg Zacharias for his continued funding and support of our work with Bayesian networks.