This paper explores the e↵ ects of parameter sharing on Bayesian network (BN) parameter learning when there is incomplete data. Using the Expectation Maximization (EM) algorithm, we investigate how varying degrees of parameter sharing, varying number of hidden nodes, and di↵ erent dataset sizes impact EM performance. The specific metrics of EM performance examined are: likelihood, error, and the number of iterations required for convergence. These metrics are important in a number of applications, and we emphasize learning of BNs for diagnosis of electrical power systems. One main point, which we investigate both analytically and empirically, is how parameter sharing impacts the error associated with EM's parameter estimates.
INTRODUCTION
Bayesian network (BN) conditional probability tables
(CPTs) can be learned when the BN structure is
known, for either complete or incomplete data.
Different algorithms have been explored in the case of
incomplete data, including: Expectation
Maximization [
We focus in this paper on EM [
Even when EM converges to a high-likelihood MLE,
the error can be large and vary depending on initial
conditions. This is a fundamental limitation of EM
using MLE; even a BN with high likelihood may be far
from the true distribution and thus have a large error.
Error as a metric for the EM algorithm for BN
parameter learning has not been discussed extensively in the
existing literature. The analysis and experiments in
this paper provide new insights in this area.
Our main application is electrical power systems, and
in particular NASA’s Advanced Diagnostics and
Prognostics Testbed (ADAPT) [
Why are BNs and arithmetic circuits useful for
electrical power system diagnostics? First, power systems
exhibit multi-variate uncertainty, for example
regarding component and sensor health (are they working
or failing?) as well as noisy sensor readings.
Second, there is substantial local structure, as reflected
in an EPS schematic, that can be taken advantage
of when constructing a BN automatically or
semiautomatically [
The rest of this paper is structured as follows. In Section 2, we introduce BNs, parameter learning for incomplete data using EM, and related research. Section 3 presents our main application area, electrical power systems. In Section 4, we define the sharing concept, discuss sharing in EM for BN parameter learning, and provide analytical results. In Section 5 we present experimental results for parameter sharing in BNs when using EM, emphasizing electrical power system fault diagnosis using BNs. Finally, we conclude and outline future research opportunities in Section 6. 2
BACKGROUND This section presents preliminaries including notation (see also Table 1). 2.1
BAYESIAN NETWORKS Consider a BN = (X, W , ✓ ), where X are discrete nodes, W are edges, and ✓ are CPT parameters. Let E ✓ X be evidence nodes, and e the evidence. A Notation X W ✓ ✓ ˆ ✓ ⇤ O H S P U Y TP E R tmin tmax t0 ✏ err(✓ ˆ) nA = |A| ` ``
= (X, W , ✓ )
E(Z)
V (Z)
r
✓ 2 [
Explanation BN nodes BN edges BN CPTs estimated CPTs true CPTs observable nodes hidden nodes (actually) shared nodes (potentially) shared nodes unshared nodes set partition of X number of wrong CPTs evidence nodes non-evidence nodes min # of EM iterations max # of EM iterations iteration # at EM convergence tolerance for EM error of ✓ ˆ relative to ✓ ⇤ cardinality of the set A likelihood log-likelihood Bayesian network (BN) expectation of r.v. Z variance of r.v. Z Pearson’s corr. coe↵ .
CPT parameter estimated CPT parameter true CPT parameter error bound for ✓
BN factors a joint distribution Pr(X), enabling different probabilistic queries to be answered by e cient algorithms; they assume that nodes E are clamped to values e. One query of interest is to compute a most probable explanation (MPE) over the remaining nodes R = X \ E, or MPE(e). Computation of marginals (or beliefs) amounts to inferring the posterior probabilities over one or more query nodes Q ✓ R, specifically BEL(Q, e), where Q 2 Q.
In this paper, we focus on situations where ✓ needs to be estimated but the BN structure (X and W ) is known. Data is complete or incomplete; in other words there may be hidden nodes H where H = X \ O and O are observed. A dataset is defined as (x1, . . . , xm) with m samples (observations), where xi is a vector of instantiations of nodes X in the complete data case. When the data is complete, the BN parameters ✓ are often estimated to maximize the data likelihood (MLE). In this paper, for a given dataset, a variable X 2 X is either observable (X 2 O) or hidden (X 2 H); it is not hidden for just a strict subset of the samples.2 Let H > 0. For each hidden node 1Further information can be found here: https:// sites.google.com/site/dxcompetition/.
2In other words, a variable that is completely hidden in the training data is a latent variable. Consequently, its H 2 H there is then a “?” or “N/A” in each sample. Learning from incomplete data also relies on a likelihood function, similar to the complete data case. However, for incomplete data several properties of the complete data likelihood function–such as unimodality, a closed-form representation, and decomposition into a product form–are lost. As a consequence, the computational issues associated with BN parameter learning are more complex, as we now discuss. 2.2
For the problem of optimizing such multi-dimensional, highly non-linear, and multimodal functions, several algorithms have been developed. They include EM, our focus in this paper. EM performs a type of hillclimbing in which an estimate in the form of an expected likelihood function ` is used in place of the true likelihood `.
Specifically, we examine the EM approach to learn BN parameters ✓ from incomplete data sets.3 The traditional EM algorithm, without sharing, initializes parameters to ✓ (0). Then, EM alternates between an E-step and an M-step. In the t-th E-step, using parameters ✓ (t) and observables from the dataset, EM generates the likelihood `(t) taking into account the hidden nodes H. In the M-step, EM modifies the parameters to ✓ (t+1) to maximize the data likelihood. While |`(t) `(t 1)| ✏, where ✏ is a tolerance, EM repeats from the E-step.
EM monotonically increases the likelihood function `
or the log-likelihood function ``, thus EM converges
to a point ✓ ˆ or a set of points (a region). Since ``
is bounded, EM is guaranteed to converge. Typically,
EM converges to a local maximum [
One topic that has been discussed is the initialization
phase of the EM algorithm [
3Using EM, learning from complete data is a special case of learning from incomplete data.
When data is incomplete, the BN parameter
estimation problem is in general non-identifiable. There may
be several parameter estimates ✓ ˆ1, ..., ✓ ˆm that have
the same likelihood, given the dataset [
ELECTRICAL POWER SYSTEMS Electrical Power Systems (EPSs) are critical in today’s society, for instance they are essential for the safe operation of aircraft and spacecraft. The terrestrial power grid’s transition into a smart grid is also very important, and the emergence of electrical power in hybrid and all-electric cars is a striking trend in the automotive industry.
ADAPT (Advanced Diagnostics and Prognostics
Testbed) is an EPS testbed developed at NASA [
ADAPT has been used in di↵ erent configurations and
represented in several fault detection and diagnosis
BNs [
In BN instances where parameters are not shared, the
CPT for a node in the BN is treated as separate from
the other CPTs. The assumption is not always
reasonable, however. ADAPT BNs may benefit from
shared parameters, because there are typically several
repeated nodes or fragments in these BNs [
For parameter sharing as investigated in this paper,
the CPTs of some nodes are assumed to be
approximately equal to the CPTs of di↵ erent nodes elsewhere
in the BN.4 Data for one set of nodes can be used
elsewhere in the BN if the corresponding nodes are shared
during EM learning. This is a case of parameter
sharing involving the global structure of the BN, where
di↵ erent CPTs are shared, as opposed to parameter
sharing within a single CPT [
In some ADAPT BNs, one sub-network is essentially
duplicated three times, reflecting triple redundancy in
ADAPT’s power storage and distribution network [
4It is unrealistic to assume that several engineered
physical objects, even when it is desired that they are exactly
the same, in fact turn out to have exactly the same
behavior. A similar argument has been made for object-oriented
BNs [
Consider a complex engineered system, such as an electrical power system. After construction, but before it is put into production, it is typically tested extensively. The sensors used during testing lead to one set of observation nodes in the BN, OT . The sensors used during production lead to another set of observations in the BN, OP . For reasons including cost, fewer sensors are typically used during production than during testing, thus we assume OP ✓ OT .
As an example, in Figure 1 we denote OP = {CB, S} as production observation nodes, and OT = {CB, S, HB, HS } as testing observation nodes. In all our experiments, shared nodes are also hidden, or S ✓ H. Typically, there are hidden nodes that are not necessarily shared, or S ⇢ H. This is, for example, the case for the mini-ADAPT BN as reflected in the sub-network in Figure 1. 4 4.1
EM
WITH SHARING
SHARING IN BAYESIAN NETWORKS Consider a BN = (X, W , ✓ ). A sharing set partition for nodes X is a set partition Y of X with subsets Y 1, ..., Y k, with Y i ✓ X and k 1. For each Y i with k i 1 the nodes X 2 Y i share a CPT during EM learning as discussed in Section 4.2. We assume that the nodes in Y i have exactly the same number of states. The same applies to their respective parents in , leading to each Y 2 Y i having the same number of parent instantiations and exactly the same CPTs. Traditional non-sharing is a special case of sharing in the following way. We assign each BN node to a separate set partition such that for X = {X1, ..., Xn} we have Y 1, ..., Y n with Y i = {Xi}.
One key research goal is to better understand the behavior of EM as sharing nodes S and observable nodes O vary. We examine three cases: complete data OC (no hidden nodes); a sensor-rich testing setting with observations OT ; and a sensor-poor production setting with observations OP . Understanding the impact of varying observations O is important due to cost and e↵ ort associated with observation or sensing. 4.2
Similar to traditional EM (see Section 2.2), the
sharing EM algorithm also takes as input a dataset and
estimates a vector of BN parameters ✓ ˆ by iteratively
improving `` until convergence. The main di↵ erence
of sharing EM compared to traditional EM is that we
are now setting some nodes as shared S, according to
Y . To arrive at the sharing EM algorithm from the
traditional EM algorithm, we modify the likelihood
function to introduce parameter sharing and combine
parameters (see also [
EM’S BEHAVIOR UNDER SHARING We now provide a simple analysis of certain aspects of traditional EM (TEM) and sharing EM (SEM). For simplicity, we only consider EM runs that converge and exclude runs that time out.7 For a node Xi 2 X, TEM will converge to one among potentially several convergence regions. Suppose that the CPT of node Xi has (Xi) convergence regions. Then the actual number of convergence regions ( U ) for a non-shared BN U with nodes X = {X1, ..., Xn} is upper bounded by ¯( U ) as follows: ( U ) ¯( U ) = n Y (Xi). i=1 5For the relevant LibDAI source code, please see here: https://github.com/erikreed/HadoopBNEM/blob/ master/src/emalg.cpp#L167. In words, it is a modification on the collection of su cient statistics during the maximization step.
6We assume that (1) is better than likelihood in this regard, if the original BN was manually constructed. The BNs experimented with in Section 5 were manually constructed, for example.
7In practice, runs that time out are very rare with the parameter settings we use in experiments. (2)
Due to the sharing, SEM intuitively has fewer convergence regions than TEM. This is due to SEM’s slightly modified M-step that aggregates the shared CPTs and parameters. Consider a BN S with exactly the same nodes and edges as U , but with sharing, specifically with sharing set partitions Y 1, ..., Y k and k < n. Without loss of generality, assume that Xi 2 Y i. Then the actual number of convergence regions ( S ) is upper bounded by ¯( S ) as follows: (3) (4) ( S ) ¯( S ) = k Y (Xi), i=1 assuming that the (Xi) convergence regions used for Xi in (2) carry over to Y i.
A special case of (3) is when there exists exactly one Y 0 2 Y such that |Y 0| 2 while for any Z 2 Y \Y 0 we have |Z| = 1. The experiments performed in Section 5 are all for this special case. Specifically, but without loss of generality, let Y i = {Xi} for 1 i < k and Y k = {Xk, ..., Xn} with nS = n k + 1 (i.e., S = Y k has nS sharing nodes). It is illustrative to consider the ratio: ¯( U ) ¯( S ) =
QQikn=1 (Xi) = Yn (Xi).
i=1 (Xi) i=k+1 Here, we assume that Xi has (Xi) convergence regions in both U and S and take into account that for shared nodes Y k = S, CPTs are tied together. The simple analysis above suggests a non-trivial impact of sharing, given the multiplicative e↵ ect of the (Xi)’s for k + 1 i n in (4). However, since upper bounds are the focus in this analysis, only a partial and conservative picture is painted. The experiments in Section 5—see for example Figure 3, Figure 5, and Figure 6—provide further details. 4.4
ANALYSIS OF ERROR We now consider the number of erroneous CPTs as estimated by SEM when sharing is varied. Clearly, a CPT parameter is continuous and its EM estimate is extremely unlikely to be equal to the original parameter. Thus we consider here a discrete variable, based on forming an interval in the one-parameter case. Generally, let a discrete BN node X 2 X have k states such that xi 2 {x1, ..., xk}. Consider ✓ xi|z = Pr(X = xi | Z = z) for a parent instantiation z. We now have an original CPT parameter ✓ i 2 {✓ x1|z, ..., ✓ xk 1|z} and its EM estimate ✓ˆi 2 {✓ˆx1|z, ..., ✓ˆxk 1|z}. Let us jointly consider the original CPT parameter ✓ i⇤ and its estimate ✓ˆi. If ✓ˆi 2 [✓ i⇤ i, ✓ i⇤ + i] we count ✓ˆi as correct, and say ✓ˆi = ✓ i⇤ ; else it is incorrect or wrong, and we say ✓ˆi 6= ✓ i⇤ . This analysis clearly carries over to multiple CPT parameters, parent instantiations, and BN nodes. This shows how we go from a continuous (estimated CPT parameters ✓ˆ) to a discrete value (number of wrong or incorrect CPT estimates), where the latter is used in this analysis.
Suppose that up to nP nodes can be shared. Further suppose that nS nodes are actually shared while nU nodes are unshared, with nU + nS = nP . Let TP be a random variable representing the total number of wrong or incorrect CPTs, TS the total for shared nodes, and TU the total for unshared nodes. Clearly, we have TP = TS +TU .
Let us first consider the expectation E(TP ). Due to linearity, we have E(TP ) = E(TS) + E(TU ). In the non-shared case, assume for simplicity that errors are iid and follow a Bernoulli distribution, with probability p of error and (1 p) = q of no error.8 This gives E(TU ) = nU p, using the fact that a sum of Bernoulli random variables follows a Binomial distribution.9 In the shared case, all shared nodes either have an incorrect CPT ✓ˆ 6= ✓ ⇤ or the correct CPT ✓ˆ = ✓ ⇤ . Assuming again probabilities p of error10 and (1 p) = q of no error, and by using the definition of expectation of Binomials we obtain E(TS) = nSp.
Substituting into E(TP ) we get
E(TP ) = nU p + nSp = nP p.
Let us next consider the variance V (TP ). While variance in general is not linear, we assume linearity for simplicity, and obtain
V (TP ) = V (TU ) + V (TS).
In the non-shared case we have again a Binomial distribution, with well-known variance
V (TU ) = nU p(1 p).
In the shared case we use the definition of variance, put p1 = (1 p), p2 = p, and µ = nSp, and obtain after some simple manipulations:
8This is a simplification, since our use of the Bernoulli assumes that each CPT is either “correct” or “incorrect.” When learned from data, the estimated parameters are clearly almost never exactly correct, but close to or far from their respective original values.
9If X is Binomial with parameters n and p, it is wellknown that the expected value is E(X) = np.
10The error probabilities of TS and TU are assumed to be the same as a simplifying assumption. (5) (6) (7) and by substituting (7) and (8) into (6) we get V (TP ) = p(1 p)((nP nS) + n2S).
(9) In words, (9) tells us that as the number nS of shared nodes increases at the expense of the number of unshared nodes nU , variance due to non-shared nodes decreases linearly, but variance due to sharing increases quadratically. The net e↵ ect shown in (9) is that variance V (TP ) of the error increases with the number of shared nodes, according to our analysis above. Expectation, on the other hand, remains constant (5) regardless of how many nodes are shared. These analytical results have empirical counterparts as discussed in Section 5, see for example the error sub-plot at the bottom of Figure 2. 5
EXPERIMENTS for tmax and tmin = 3.
We now report on EM experiments for several di↵ erent BNs, using varying degrees of sharing. We also vary the number of hidden nodes and dataset size. We used ✏ = 1e 3 as an EM convergence criterion, meaning that EM stopped at iteration t0 when the ``-score changed by a value ✏ between iterations t0 1 and t0 t0 tmin. In these experiments, tmax = 100 5.1
METHODS AND DATA Bayesian networks. Table 2 presents BNs used in the experiments.11 Except for the BN Pigs, these BNs all represent (parts of) the ADAPT electrical power system (see Section 3). The BN Pigs has the largest number of nodes that can be shared (nP = 296), comprising 67% of the entire BN. The largest BN used, in terms of node count, edges, and total CPT size, is ADAPT T2.
Datasets. Data for EM learning of parameters for these BNs were generated using forward sampling.12 Each sample in a dataset is a vector x (see Section 2.1). The larger BNs were tested with increasing numbers of samples ranging from 25 to 400, while mini-ADAPT was tested with 25 to 2000 samples.
Sharing. Each BN has a di↵ erent number of parameters that can be shared, where a set of nodes Y i 2 11ADAPT BNs can be found here: http://works. bepress.com/ole_mengshoel/.
12Our experiments are limited in that we are only learning the parameters of BNs, using data generated from those BNs. Clearly, in most applications, data is not generated from a BN and the true distribution does not conform exactly to some BN structure. However, our analytical and experimental investigation of error would not have been possible without this simplifying assumption.
V (TS) =
2 X pi(Xi i=1 µ)2 = n2Sp(1 p), (8)
Y with equal CPTs are deemed sharable. In most cases, there were multiple sets of nodes Y 1, ..., Y k with |Y i| 2 for k i 1. When multiple sets were available, the largest set was selected for experimentation, as shown in Section 5.2’s pseudo-code. Metrics. After an EM trial converged to an estimate ✓ ˆ, we collected the following three metrics: 1. number of iterations t0 needed to converge to ✓ ˆ, 2. log-likelihood `` of ✓ ˆ, and 3. error: distance between ✓ ˆ and the original ✓ ⇤ (see (1) for the definition).
To provide reliable statistics on mean and standard deviation, many randomly initialized EM trials were run.
Software. Among the available software
implementations of the EM algorithm for BNs, we have based our
work on LibDAI [
VARYING NUMBER OF SHARED
NODES Here we investigate how varying the number of shared nodes impacts EM. A set of hidden nodes H was created for each BN by selecting
H = arg max |Y i|,
Y i2 Y where Y is a sharing set partition for BN nodes X (see Section 4.1). In other words, each experimental BN had its largest set of shareable nodes hidden, giving nH = 12 nodes for ADAPT T1, nH = 66 nodes for ADAPT T2, nH = 32 nodes for ADAPT P1, and nH = 145 nodes for Pigs.
The following gradual sharing method is used to vary sharing. Given a fixed set of hidden nodes H and an initially empty set of shared nodes S: 1. Randomly add nS 1 hidden nodes that are not yet shared to the set of shared nodes S. Since we only have a single sharing set, this means moving nS nodes from the set H \ S to the set S. 2. Perform m sharing EM trials in this configuration, and record the three metrics for each trial. 3. Repeat until all hidden nodes are shared; that is,
S = H.
Using the gradual sharing method above, BN nodes were picked as hidden and then gradually shared. When increasing the number of shared nodes, the new set of shared nodes was a superset of the previous set, and a certain number of EM trials was performed for each set. 5.2.1
One Network For ADAPT T2, m = 200 samples were generated and nH = 66 nodes were hidden. We used, in the gradual sharing method, nS = 4 from nS = 2 to nS = 66 (every hidden node was eventually set as shared). Figure 2 summarizes the results from this experiment. Here, nS is varied along the x-axis while the y-axis shows statistics for di↵ erent metrics in each of the three sub-plots. For example, in the top plot of Figure 2, each marker is the mean number of iterations (µ), and the error bar is +/- one standard deviation ( ). The main trends14 in this figure are: parameter sharing increased the mean number of iterations required for EM and slowly decreased the mean ``. Increasing the number of shared nodes resulted in a corresponding increase in standard deviation for the number of iterations, ``, and error of the BN ADAPT T2. For standard deviation of error, this is in line with our analysis in Section 4.4. 5.2.2
Multiple Networks We now investigate how varying the number of shared nodes impacts EM for several BNs, specifically the correlation between the number of parameters shared and the mean µ and standard deviation for our three metrics. To measure correlation, we use Pearson’s 14We say “main trends” because the curves for the metrics mean number of iterations and `` are in fact reversing their respective trends and dropping close to the maximum of 66 shared nodes. sample correlation coe cient r: m
P (xi r(X, Y ) = i=1 (m x¯)(yi 1)sxsy y¯) , (10) where x¯ and y¯ are the sample means of two random variables X and Y , and sx and sy are the sample standard deviations of X and Y respectively. Here, (10) measures the correlation between m samples from X and Y .15 The number of samples, m, refers to the number of (X, Y ) sharing samples we have for this correlation analysis (and not the number of samples used to learn BN parameters). In all these experiments, we do a trial for a number of shared nodes, giving several (X, Y ) pairs. Consequently, each number of shared nodes tested would be an X, and the metric measured would be a Y . For example, if we use nS 2 {2, 4, 6, 8, 10} shared nodes, then m = 5.
We now tie r(X, Y ) in (10) to the r(µ) and r( ) used in Table 3. In Table 3, µ and show the mean and standard deviation, respectively, for a metric. Thus, r(µ) is the correlation of the number of shared nodes and the mean likelihood of a metric, while r( ) is the correlation of the number of shared nodes and the standard deviation of likelihood of a metric.
Figure 2 helps in understanding exactly what is being correlated, as µ and for all three metrics are shown for the BN ADAPT T2. In the top plot, r(µ) is the correlation between the number of shared nodes (xaxis) and the mean number of iterations (y-axis). In other words, the mean yi = µi is for a batch of 50 trials of EM. The mean y¯ used in Pearson’s r is, in this case, a mean of means, namely the mean over 50-EM-trialsmeans over di↵ erent numbers of shared nodes. Table 3 summarizes the experimental results for four BNs. In this table, a positive correlation implies that parameter sharing increased the corresponding metric statistic. For example, the highest correlation between number of shared nodes and mean likelihood is for ADAPT T1 at 25 samples, where r(µ) = 0.997. This suggests that increasing the number of shared nodes was highly correlated with an increase in the likelihood of EM. Negative coe cients show that increasing the number of shared nodes resulted in a decrease of the corresponding metric statistic.
A prominent trend in Table 3 is the consistently positive correlation between the number of shared nodes 15In this case, X is the independent variable, specifically the number of shared nodes. We treat the metric Y as a function of X. When X is highly correlated with Y , this is expressed in r through extreme (positive or negative) correlation values. nS and the standard deviation of error, r( ), for all 4 BNs. This is in line with the analytical result involving nS in (9).
The number of samples was shown to have a significant impact on these correlations. The Pigs network showed a highly correlated increase in the mean number of iterations for 25 and 50 samples. However, for 100, 200, and 400 samples there was a decrease in the mean number of iterations. The opposite behavior is observed in ADAPT T1, where fewer samples resulted in better performance for parameter sharing (reducing the mean number of iterations), while for 200 and 400 samples we found that parameter sharing increased the mean number of iterations. Further experimentation and analysis may improve the understanding of the interaction between sharing and the number of samples. 5.3
First, we will show how sharing influences EM parameter interactions for the mini-ADAPT BN shown in Figure 1 and demonstrate how shared parameters jointly converge.
Earlier we introduced OP as observable nodes in a production system and OT as observable nodes in a testing system. Complementing OP , hidden nodes are HP = {HB, HS , VB, SB}. Complementing OT , hidden nodes are HT = {VB, SB}. When a node is hidden, the EM algorithm will converge to one among its potentially many convergence regions. For OP , EM had much less observed data to work with than for OT (see Figure 1). For OP , the health breaker node HB was, for instance, not observed or even connected to any nodes that were observed. In contrast, OT was designed to allow better observation of the components’ behaviors, and HB 2 OT . From mini-ADAPT, 500 samples were generated. Depending on the observable set used, either nH = |HT | = 2 or nH = |HP | = 4 nodes were hidden, and 600 random EM trials were
Table 4 shows results, in terms of means µ and standard deviations , for these EM trials. For OP , with nH = 4, the means µ of the metrics error, likelihood, and number of iterations showed minor di↵ erences when parameter sharing was introduced. The largest change due to sharing was an increase in of likelihood. For OT , where nH = 2, di↵ erences were greater. The µ of likelihood for sharing was lower with over a 2x increase in . The µ for error demonstrated only a minor change, but nearly a 2x increase in . This is consistent with our analysis in Section 4.4. Figure 3a and Figure 3c show how log-likelihood or `` (x-axis) and error (y-axis) changed during 15 EM trials for OP and OT respectively. These EM trials were selected randomly among the trials reported on in Table 4. Parameter sharing is introduced in Figure 3b and Figure 3d. For OP , the progress of the EM trials is similar for sharing (Figure 3b) and non-sharing (Figure 3a), although for a few trials in the sharing condition the error is more extreme (and mostly smaller!). This is also displayed in Table 4, where the di↵ erence in number of iterations, error, and likelihood was minor (relative to OT ). On the other hand, there is a clear di↵ erence in the regions of convergence for OT when parameter sharing is introduced, consistent with the analysis in Section 4.3. Figure 3d shows how the (c) OT – No Sharing
EM trials typically followed a path heading to optima far above or far below the mean error, with two of the EM trials plotted converging in the middle region of the error.
Histograms for the 600 EM trials used in Table 4 are shown in Figure 4. The 20-bin histograms show error err(✓ ˆ) at convergence. The OP and OT sets are shown without parameter sharing in Figure 4a and Figure 4c, respectively. Parameter sharing is introduced in Figure 4b and Figure 4d. There is an increased of error due to parameter sharing for OT . When comparing Figure 4c (No Sharing) and Figure 4d (Sharing), we notice di↵ erent regions of error for EM convergence due to sharing. Figure 4c appears to show four main error regions, with the middle two being greatest in frequency, while Figure 4d appears to show three regions of error, with the outer two being most frequent. The outer two regions in Figure 4d are further apart than their non-sharing counterparts, showing that parameter sharing yielded a larger range of error for OT , see Section 4.4. 5.3.2
Large Bayesian Networks Next, large BNs are used to investigate the e↵ ects of parameter sharing, using a varying number of shared nodes. The larger ADAPT networks and Pigs were run with 50 EM trials16 for each configuration of observ16The decrease in number of EM trials performed relative to mini-ADAPT was due to the substantial increase in (a) No Sharing (b) Sharing able nodes, number of samples, and number of shared nodes. Some of the results are reported here. Figure 5a shows results for ADAPT T2 without parameter sharing during 15 EM trials using nH = 66 hidden nodes and 200 samples. Figure 5b shows a substantial change in error when the nH = 66 hidden nodes were shared. The range of the error for EM is much larger in Figure 5b, while the upper and lower error curves have a symmetric quality. Four regions for the converged error are visible in Figure 5b, with the inner two terminating at a lower `` than the outer two. The lowest error region of Figure 5b is also lower than the lowest error of Figure 5a, while retaining a similar ``.
Figure 6 uses a smaller ADAPT BN, containing 172 nodes instead of 671 nodes (see Table 2). Here, nH = 33 nodes were hidden and 200 samples were used. In several respects, the results are similar to those obtained for ADAPT T2 and mini-ADAPT. However, CPU time required (days to weeks).
Figure 6a shows that EM terminates on di↵ erent likelihoods, which is not observed in Figure 5a. The error also appears to generally fluctuate more in Figure 6a, whereas the error changes the most during later iterations in Figure 5a. Figure 6b applies parameter sharing to the nH = 33 hidden nodes. A symmetric e↵ ect is visible between high and low error, reflecting the analysis in Section 4. Of the 15 trials shown in Figure 6b, two attained `` > 1.2e 4, while the rest converged at `` ⇡ 1.21e 4. Additionally, the ``s of these two trials were greater than any of the non-sharing ``s shown in Figure 6a. 6
CONCLUSION
Bayesian networks have proven themselves as very
suitable for electrical power system diagnostics [
Our results suggest complex interactions between varying degrees of parameter sharing, varying number of hidden nodes, and di↵ erent dataset sizes when it comes to impact on EM performance, specifically likelihood, error, and the number of iterations required for convergence. One main point, which we investigated both analytically and empirically, is how parameter sharing impacts the error associated with EM’s parameter estimates. In particular, we have found analytically that the error variance increases with the number of shared parameters. Experiments with several BNs, mostly for fault diagnosis of electrical power systems, are in line with the analysis. The good news here is that there is, in the sharing case, smaller error some of the time.
Further theoretical research to better understand parameter sharing is required. Since parameter sharing was demonstrated to perform poorly in certain cases, further investigations appear promising. Parameter sharing sometimes reduced the number of EM iterations required for parameter learning, while at other times the number of EM iterations increases. Improving the understanding of the joint impact of parameter sharing and the number of samples on the number of EM iterations would be useful, for example. Finally, it would be interesting to investigate the connection to object-oriented and relational BNs in future work.