In the work we have implemented an ensemble of Naive Bayes classi ers using committee of bootstraps and monte carlo splits. We have conducted 50 iterations of learning in each tested model. Fixed percentage of random objects from the original training system was used. New training decision systems that were considered consisted of 10 to 100 percent of random objects from original training decision system. Two main variants were checked, rst with objects returning after the drawn (bootstraps) - and without returning (as monte carlo splits). We have presented how Naive Bayes classi er works in mentioned models on selected data from UCI repository.
The ensemble scheme of classi cation is really e ective in many contexts, for
instance in rough set methods the exemplary successful applications can be found
in [
Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
The Bayes classi er, for a general perspective, cf., Mitchell [
Naive Bayes classi er owes its naivety epithet to the fact that one assumes the independence of attributes which condition in reality is not often met. Its working in the realm of decision systems can be described concisely as follows. For a given training decision system (Utrn; A; d) and a test system (Utst; A; d), where U = Utrn [ Utst is the set of objects, A = fa1; a2; :::; ang is the conditional attribute set, and d is the decision attribute.
The classi cation of a test object v 2 Utst described by means of its information set (a1(v); a2(v); :::; an(v)) consists of computing for all decision classes the value of the parameter
P (d = dijb1 = a1(v); b2 = a2(v); :::; bn = an(v)) and the decision on v is the decision value with the maximal value of the parameter.
The Bayes theorem along with the frequency interpretation of probability allows to express this probability as
P (b1 = a1(v); b2 = a2(v); :::; bn = an(v)jd = di) P (d = di)
P (b1 = a1(v); b2 = a2(v); :::; bn = an(v)) (1) One usually dispenses with the denominator of equation 1, because it is constant for all decision classes. Assuming independence of attributes, the numerator of equation 1 can be computed as
P (d = di) n Y P (bm = am(v)jd = di): m=1
In practice, we can use partial estimation P (bm = am(v)jd = di) = number of test instances bm = am(v) in training class di cardinality of class di : Each decision class is voting by submitting the value of the parameter In this approach, we could encounter a problem of zero frequency of a descriptor bm = am(v) in a class di, i.e., P (bm = am(v)jd = di) = 0. One of the methods to avoid the problem of zero{valued decisive parameters, is to search among the remaining classes for the smallest non{zero value of P (bm = am(v)jd = dj ). The found value is additionally slightly lowered, and assigned instead of the zero value. In case of more than one class with the zero frequency of the descriptor bm = am(v), we could assign such reduced value to all of them. Another method to avoid this problem is to consider the remaining decision classes, which contain the value bm = am(v). In case of the zero frequency of the descriptor bm = am(v) in all training classes, this descriptor can be disregarded. In order to help ourselves with the task of computing with small numbers, we can use logarithms of probabilities. In practice, it is also acceptable to use sums instead of products in which case decision classes vote by the parameter This classi er is t for symbolic attributes. In case of numerical data, assuming the normal distribution, the probability P (bm = am(v)jd = di) can be estimated based on the Gaussian function f (x) = p(2 1 2) c
(x c)2 e 2 c2 :
To compute this value, the estimates of mean values and variances in decision classes are necessary: c =
Pcardinality of class c a(vi) ; i=1 cardinality of class c c2 =
1 cardinality of class c cardinality of class c
X i=1 (a(vi) c)2: 2.1
In this section we show an exemplary classi cation on the lines of equation (2). The test decision system is given as We have P (c = 2) = 36 = 12 , P (c = 4) = 21 .
We start with classi cation of the test object x1 whose information set is (2; 4; 2; 1) and the decision c = 4.
According to the formula in (??), we obtain P (a1 = 2jc = 2) = 13 .
P (a2 = 4jc = 2) = 03 we cannot handle it, there is no descriptor a2 = 4 in all classes. Next, P (a3 = 2jc = 2) = 31 .
P (a4 = 1jc = 2) = 33 : Finally, P aramc=2 = 12 ( 31 + 03 + 13 + 33 ) = 56 .
Continuing, we obtain P (a1 = 2jc = 4) = 13 .
P (a2 = 4jc = 4) = 03 we cannot handle it, there is no descriptor a2 = 4 in all classes.
P (a3 = 2jc = 4) = 31 .
P (a4 = 1jc = 4) = 23 .
Finally, P aramc=4 = 12 ( 31 + 03 + 13 + 23 ) = 23 .
As P aramc=2 > P aramc=4, the object x1 is assigned the decision value of 2. This decision is inconsistent with the expert decision, this object is incorrectly classi ed.
For the second test object x2, with the information set (1; 2; 1; 1) and the decision value of 4, we obtain in the analogous manner: P (a1 = 1jc = 2) = 13 , we increase counter by 1 because P (a1jc = 4) = 0 so nally P (a1 = 1jc = 2) = 23 .
P (a2 = 2jc = 2) = 30 we cannot handle it because the descriptor a2 = 2 is missing in all classes.
P (a3 = 1jc = 2) = 31 . sPo(aP4a=ra1mjcc==2 2=) =12 33( 23 + 30 + 13 + 33 ) = 1.
, P(a1 = 1jc = 4) = 30, in this case we have to increase counter of P(a1 = 1jc = 2) by one to account for the class, which contains at least one count of the descriptor a1 = 1.
P(a2 = 2jc = 4) = 30, we cannot handle it, a2 = 2 is missing in all classes. P(a3 = 1jc = 4) = 13.
P(a4 = 1jc = 4) = 23, so, nally, Paramc=2 = 12 (30 + 03 + 13 + 23) = 12. As Paramc=2 > Paramc=4, the object x2 is assigned the decision value of 2; this decision is consistent with the expert decision so the object is correctly classi ed. The next test object is x3 with the information set (9;7;10;7;4). We have Paramc=2 = P(c = 2) Pi4=1 P(ai = vijc = 2), and, P(a1 = 9jc = 2) = 03.
P(a2 = 7jc = 2) = 03.
P(a3 = 10jc = 2) = 03.
P(a4 = 7jc = 2) = 30, so, nally, Paramc=2 = 21 (03 + 03P+i4=031+P(03a)i ==0v.ijc = 4), we have Also, for Paramc=4 = P(c = 4) P(a1 = 9jc = 4) = 03.
P(a2 = 7jc = 4) = 03.
P(a3 = 10jc = 4) = 03.
P(a4 = 7jc = 4) = 30, and, nally, Paramc=2 = 12 (30 + 03 + 03 + 03) = 0.
As Paramc=2 == Paramc=4, the random decision random(2;4) = 4 is assigned to x3, so the object is correctly classi ed.
For the last test object, x4 with the information set (4;4;10;10;4), we compute Paramc=2 = P(c = 2) Pi4=1 P(ai = vijc = 2): P(a1 = 4jc = 2) = 03.
P(a2 = 4jc = 2) = 03.
P(a3 = 10jc = 2) = 03.
P(a4 = 10jc = 2) = 30, and, nally, Paramc=2 = 21 (30 + 30 + 03 + 03) = 0.
A random decision assignment random(2; 4) = 4 causes x4 to be incorrectly classi ed.
We now compute parameters: Global Accuracy = number of tst objects correctly classi ed in whole test system number of classi ed objects in whole testsystem ; Balanced Accuracy = In our exemplary case, these values are
Pnumber of classes number of test objects correctly classi ed in class ci i=1 number of objects classi ed in class ci number of classes : Global Accuracy =
= ; There are many techniques in the family of Ensemble models. One of the most popular are Random Forests, Bagging and Boosting - see [25]. Short description of used models is to be found below.
Bootstrap Ensembles - Pure Bagging: It is the random committee of bootstraps [28]. It is a method in which the original decision system - the basic knowledge - is split into (T RN ) training data set, and (T ST valid) validation test data set. And from the TRN system, for a xed number of iterations, we form a new Training systems (N ewT RN ) by random choice with returning of cardfT RN g objects. In all iterations we classify the TRNvalid system in two ways: the rst based on the actual N ewT RN system and the second based on the committee of all performed classi cations. In the committee majority voting is performed and the ties are resolved randomly.
Committee of Monte Carlo splits: Classi cation method used in this algorithm
is similar to the previously described with the di erence that the NewTST are
formed in a di erent way - see [
In the next subsections we present information on how the models described above are used in our experiments. 4.1
Committee of Monte Carlo splits for NB Classi er
We have carried out a series of experiments using Australian credit data set
form University of Irvine repository [
Committee of Bootstraps with NB classi er The method [28] works in the same way as described above, but the only di erence is that the objects are returned after the draw and it is possible to see the copies in the training systems. The other experimental settings are identical.
As the base classi er we used the NB classi er from Sect. 2 for symbolic data. E ectiveness is assessed by the accuracy of the classi cation - expressed as a percentage of correctly classi ed objects. 5
In Figs. 1 to 10 we have the classi cation results based on the training systems formed from 10 to 100 percent random objects of the original training system. There are two variants presented - on the left-hand side a draw without a return - on the right-hand side a draw with a return. Additionally, in Tables 3 and 4 we present the average result of 50 iterations of learning with additional parameters of assessment of the quality of classi cation. We used our own implementations to carry out the tests. 5.1
From the results we can conclude that single classi ers may work unstable when they are based on a small part of the original training system. That is, when sets of objects from individual iterations do not overlap. Another reason for their instability of single classi cations may be the appearance of copies of objects for appropriately larger training systems in the return variant. The classi cation committee, starting from 20 iterations, starts to work steadily even for the only 10 percent of the drawn objects. The classi cation committee seems to be slightly more stable for the monte carlo split but its in the range of standard deviation of results. Individual classi cations from individual iterations with the help of larger training systems are much more stable in the case of the monte carlo technique. But this does not have a major impact on the nal e ectiveness of the classi cation committee. In this experimental work we checked the performance of the Nave Bayes classi er in the context of the classi cation committees based on a xed percentage of objects drawn from the training system. We used two techniques to create training systems in particular iterations - the rst one is based on the monte carlo split, where objects are drawn without returning them - the second one is based on the bootstrap model, where objects are returned. It turned out that the stability of individual classi ers increases in the case of the monte carlo method (compared to the bootstrap method) with the increase in the size of random training systems. In the case of bootstraps, increasing the training system causes more and more copies of objects, which apparently disturbs the NB classi cation. We observed that for the examined system, up to 10 percent of random objects used in the committee are nally starting to work steadily and give classi cation results comparable to those of the whole original training system. In each of the tests the classi cation committee starting from about the twentieth iteration begins to give good, stable results.
In future works, we plan to check the detected irregularities in other decisionmaking systems. We will try to check the Bayes classi cation under similar conditions in the context of other ensemble methods. And we plan to test the behavior of other selected classi ers. 7
The research has been supported by grant 23.610.007-300 from Ministry of Science and Higher Education of the Republic of Poland. 25. Yang, P., Yang, Y., H., Zhou, B., B.; Zomaya, A., Y.: A review of ensemble methods in bioinformatics: Including stability of feature selection and ensemble feature selection methods. In Current Bioinformatics, 5, (4):296-308, 2010 (updated on 28 Sep. 2016) 26. Yao, Yiyu, Zhou, B.: Naive Bayesian rough sets. In Proceedings of the Int. Conference on Rough Sets and Knowledge Technology RSKT 2010, Lecture Notes in Computer Science vol. 6401, pp 719{726 (2010) 27. Zhang, H., Zhou, J., Miao, D. and Gao, C.: Bayesian rough set model : A further investigation. International Journal of Approximate Reasoning 53, pp 541{557 (2012) 28. Zhou, Z.-H.: Ensemble Methods: Foundations and Algorithms. Chapman and Hall/CRC. p. 23. ISBN 978-1439830031. The term boosting refers to a family of algorithms that are able to convert weak learners to strong learners (2012) 29. Zhou, Z.-H.: Boosting 25 years, CCL 2014 Keynote (2014)