The paper dedicates to development of an information technology for the design of Mamdani-type fuzzy systems derived from experimental datasets. The proposed information technology provides synthesis of accurate, compact and interpretable fuzzy rule bases. A distinctive feature of the information technology is the following four-stage operation scheme: 1) generating a list of adequate candidate-rules; 2) selection of the rules; 3) reduction of the antecedents of the selected rules; 4) parametric tuning of a fuzzy rule base. The criteria of accuracy and interpretability are involved in the first and the fourth stages, and the criteria of accuracy and compactness are used in the second and the third stages. New models for interpretability protection of fuzzy rule bases during the tuning are also proposed. Examples of the application of this information technology for solving 7 identification problems with real experimental data are given.
Fuzzy rule-based systems are a popular tool for modeling complex dependencies in engineering, agronomy, economics, ecology, medicine, biology, politics, sports and in other fields [1, 2]. The semantic part of a fuzzy system is a rule base, i.e. a set of if– then rules that describes the relationship between inputs and outputs with usage of terms such as Low, Average, High etc. Those linguistic terms are formalized by fuzzy sets [3]. There are several types of fuzzy rule-based systems: Mamdani, Sugeno, Tsukamoto etc. Among them, the most popular and human-friendly is Mamdani-type system.
The paper dedicates to the design of Mamdani-type fuzzy systems based on utilizing experimental dataset. Such design usually is performed in two stages: fuzzy structural identification and fuzzy parametrical identification [4]. The structural identification consists of determining the input and output variables of models, the formation of term-sets for linguistic variables and describing the dependence by fuzzy production rules. The first two procedures are common to any identification method. The last two procedures are specific for fuzzy identification. As a result of fuzzy structural identification, we obtain a rough model, which outlines the target dependence. At the stage of fuzzy parametric identification, the model parameters are tuned, generally, the membership functions and weights of the rules are changed.
The quality of the fuzzy rule bases is evaluated according to the criteria of accuracy, compactness, and interpretability [4–9]. There are a lot of research relating to extraction fuzzy rule base from datasets [4, 7, 9–14], but a question is still open how to guarantee obtaining an accurate, compact and reasonable (transparent) fuzzy rule base in expert-free regime.
The aim of the paper is to develop an information technology that provides synthesis from experimental data of accurate, compact and interpretable fuzzy rule bases. A distinctive feature of the technology is the following four-stage operation scheme: 1) generating a list of adequate candidate-rules; 2) selection of rules; 3) reduction of the antecedents of the selected rules; 4) parametric tuning of a fuzzy rule base. The first 3 stages correspond to structural identification, and the fourth stage is parametric one. The criteria of accuracy and interpretability are involved in the first and the fourth stages, and the criteria of accuracy and compactness are used in the second and the third stages. New models for interpretability protection of fuzzy rule bases during the tuning are also proposed. The proposed information technology is tested on typical tasks from UCI Machine Learning Repository. 2
Antecedent and consequent in a Mamdani-type rule are represented by fuzzy sets. Each Mamdani-type rule can be interpreted as a zone with fuzzy boundaries in some factor space. For each fuzzy zone, the output variable takes a fuzzy constant value.
Let us write the Mamdani-type fuzzy rule base as follows [10]:
if ( x1 = ai1 and x2 = ai2 and ... and xn = ain ) , then y = di , with weight wi
(
N denotes a number of the rules; di denotes a consequent of the i-th rule in the form of a fuzzy term; wi [0; 1] denotes the weight of the i-th rule.
Each fuzzy term aij belongs to term-set l j1, l j2 , ..., l jk j , j = 1, n , k j N .
Some rules in the base (
The logical inference for the input vector X = ( x1 , x2 , , xn ) is carried out
according to Fig. 1. Firstly, membership degrees ij (x*j ) of the input values x*j to fuzzy
terms aij from rule base (
Let us denote the membership function of terms aij as ij (x j ) , x j x j ; x j , and
of terms di as di ( y) , y y; y . We use the Gaussian membership function:
(
Mapping the output bi-fuzzy set (
di* = imp (di , di ( X )) , i = 1, N ,
where imp denotes the implication, which is realized by the operation of the
minimum. The geometric interpretation of this implication is a cut of the graph of the
membership function di ( y) at the level di ( X ) , i = 1, N (Fig. 2).
(
The resulting fuzzy set is obtained by aggregation of fuzzy sets (
A crisp output value y * corresponding to the input vector X is found by
defuzzifying the fuzzy set (
. y y ( y) dy y
To prevent the effect of narrowing the output range through centroid defuzzification, we expand the support of fuzzy terms di , i = 1, N as in [10]. 3
It is assumed that the dataset reflecting relation between features X = ( x1, x2 , ..., xn )
and output y is presented as follows:
( X r , yr ) , r = 1, M ,
(
The task of fuzzy identification is to extract the rule base (
The accuracy of fuzzy model F ( X ) is assessed by the root mean squared error RMSE = 1
( yr − F ( X r ))2 or normalized error NRMSE = M r=1, M RMSE y − y
To assess the compactness of the model, we use the following indicators: N – the total number of rules and A – the total length of all antecedents, i.e., the number of terms in all the antecedents.
Interpretability refers to the possibility of a meaningful explanation of the structure and parameters of the model. A fuzzy model is interpretable if the following conditions are satisfied [10]:
• the rule base is not contradictory or redundant, that is, it does not contain rules with the same antecedents;
• the rule base is consistent with the number of terms, that is, each term appears in at least one fuzzy rule; • an arbitrary input vector produces a non-empty output fuzzy set; • each term is represented by a normal and convex fuzzy set; • each term-set is meaningfully interpreted.
A term-set is interpretable if the placement of fuzzy sets on the support is reasonable. For example, the term Average is between the terms Low and High. At the same time, the height of Low-High intersection is lower than height of Low-Average intersection and also lower than height of Average-High intersection. As an example, Fig. 3 shows an uninterpreted term-set with the following problems:
A) strong similarity of membership functions of neighboring fuzzy sets Low and Below Average, which may contradict the rule base;
B) a loss of linear ordering of the term-set through different curve of membership functions, for example, on the interval (65; 82) fuzzy set Average is larger than fuzzy set Above Average, and on interval (0; 3) fuzzy set Below average is larger than fuzzy set Low, although it should be the other way around;
C) a bias of the extreme term core – a decrease in the value of the variable x from 8 to 0 reduces the membership grade to the term Low, although it should be the other way around;
D) a cover spot – value x (84;88) does not belong to any fuzzy set, hence, for any rule base, the result of inference for x (84;88) will be an empty fuzzy set.
Let us define a fuzzy term-set interpretability in the following linguistic form: Low lower Average, Average lower High, High lower Very High etc. In a more formalized form, we tie the term-set interpretability with the following three conditions. 1. The number of terms should not be too large so that the expert can match each fuzzy set with a reasonable linguistic assessment. In [4], the term-set cardinality is bounded above by a “magic” number 7 2 [17]. Based on our experience in the design of fuzzy systems, it is more expedient to use term-sets with cardinality up to 7.
2. Fuzzy sets of different terms should not be equivalent or almost equivalent. Hence, the graphs of the membership functions of neighboring terms, for example, Low and Average should definitely differ by an eye.
3. The-term set must be linearly ordered. Let us denote by li a term with number i, i = 1, K . This term is used as a linguistic assessment of variable x over an interval x; x . Let term li corresponds to fuzzy set li with membership function i (x) . Linear ordering condition for term-set l1, l2 , ..., lK is written as follows: x x, sup (core(li )) : i (x) i+1(x) x x, sup (core(li )) : i (x) i+1 (x) x inf (core(li+1)) , x : i (x) i+1 (x) x inf (core(li+1)) , x : i (x) i+1 (x) , i = 1, K −1 (7)
To preserve the linear ordering of term-set l1, l2 , ..., lK of the variable x over the
interval x; x , let us introduce the following constrains on the parameters of
membership functions (
si denotes an abscissa of crossing point of the graphs of membership functions of neighboring fuzzy sets li and li+1 , i = 1, K −1 .
The first two lines in (8) determine that the cores of extreme terms are fixed at the variable boundaries. These conditions protect against C-type problem with interpretability. The third line provides the linear ordering of the cores of fuzzy sets l1, l2 , ..., lK . The first three lines (8) together hold the core of any fuzzy set into range [x; x] . The fourth line (8) demands that any inside-interval crossing point of the neighboring membership functions locates between the cores of these fuzzy sets. The fourth line protects against B-type problem with interpretability.
The crossing points of neighboring fuzzy sets are calculated by the following simple formulas: si = bici+1 + bi+1ci
ci+1 + ci si = bici+1 − bi+1ci
ci+1 − ci
One point of pair (9)–(10) locates into the range (bi ; bi+1 ) . Let us denote it as qi , and another point of the pair as vi . It allows to simplify line 4 in (8) into the following two lines: b1 q1 b2 q2 ... qk −1 bK ; vi (x; x) .
To protect against D-type problem with interpretability, we fix the minimal level of
the height hi of the intersection of neighboring fuzzy sets li and li+1 , i = 1, K −1 .
This height is calculated by formula (
2 ci ci+1 As a result, system (8) is transformed to the following form: where h* denotes the minimum permissible height of neighboring fuzzy sets intersection. 4
One of the most important tasks of structural identification is the selection of fuzzy rules from some pre-formed set of candidates. Candidate rules can be formed by an expert or obtained by processing relevant experimental data. Ideally, a fuzzy rule base should be interpretable, compact, and adequate. It is impossible to achieve this in real problems, because in practice it is rational to choose an interpretable rule base with the right balance between compactness and accuracy. A necessary condition for such balance is getting the interpretable rule base from the Pareto front in the coordinates “model compactness – model accuracy”.
Let us introduce the following notations: R is a candidate-list of rules and y = F (R ', X ) is a fuzzy model that ties inputs X with the output y with usage of R R fuzzy rules ( R R ).
A typical approach to the rules selection [15, 18] is a choice of R R that provides: or RMSE(R) → min N (R) N * N (R) → min RMSE(R) RMSE * , where N * and RMSE * denotes the permissible levels of compactness and accuracy.
The typical approach forms a large region of feasible solutions, a significant part of which is located far from the Pareto front (Fig. 4a and 4b). This slows down the search for optimal solution that locates at the Pareto front. To reduce the searching space, we proposed a method [19] for choosing the rule base in the vicinity of the Pareto front. We mark this vicinity by the following linear constraint that describes the compactness-accuracy trade-off:
RMSE(R) k0 + k1 N (R) , where k0 0 and k1 0 denotes the parameters, choosing which one can form the feasible region in the vicinity of the Pareto front.
Taking into account (15), problems (13)–(14) are transformed to the following form: (13) (14) (15) RMSE(R) → min RMSE(R) k0 + k1 N (R) N (R) → min RMSE(R) k0 + k1 N (R) ; . (17) The reduction of the search space for problems (16)–(17) is illustrated on Fig. 4c.
To match the coefficients in the linear constrains in (16)–(17) it is possible to use the endpoints of the Pareto front. They correspond to almost empty and to almost full rule bases. Also, it is easy to find out the upper estimation of the Pareto front (Fig. 5) using a greedy algorithm based on the ideas of the approximate Sahni method for knapsack problem [20]. The computational complexity of this greedy algorithm is quadratic.
The search for optimal solutions is performed by a genetic algorithm with
chromosome encoding according to the Pittsburgh scheme [21]. Each chromosome represents
a fuzzy base with its own set of rules R . Each gene of this chromosome corresponds
to one candidate rule. A gene has the value 1, if the relative rule is picked up. If the
rule is not selected, then the gene has the value 0. For example, the chromosome
(
After obtaining the optimal set of rules, we carry out the reduction of antecedents. The goal of this procedure is to replace one or more terms in the antecedent with the term Don’t care. Such a replacement is equivalent to deleting the corresponding terms from the rule antecedents, i.e. transition from long rules to short rules. For example, the transition from rule
If ( x1 = Low and x2 = Average and x3 = High ) , then y = Low to rule
If x2 = Average, then y = Low .
During this procedure the compactness of the rule base is improving by the criterion A – the total length of all the antecedents. Sometimes this leads to a reduction of the cardinality of term-sets, as well as to the merging the several rules into one.
Antecedents’ reduction is also carried by the genetic algorithm. The gene takes the value 1, if the rule uses the term from the best fuzzy knowledge base, and the value 0 if Don’t care term is used. The larger the number of Don’t care terms in a rule, the shorter and more compact the fuzzy rule base. Optimization is carried out according to a criterion RMSE with a ban on conflicting rules. Conflicting rules mean rules with the same antecedents. After optimization, a special procedure cleans the rule base – it deletes the rules with only Don’t care terms and compresses the term-sets with inactive terms. 5
A parametric identification is tuning the weights of the rules and parameters of membership functions that provide the minimum RMSE on the test sample. From mathematical point of view, the parametric identification is a kind of continuous optimization task. Parametric identification does not change the structure of the model, therefore, the criteria of compactness is not used during optimization. However, unconstrained optimization can lead to an uninterpretable rule base.
We form the vector of controlled variables P = (W , P1, P2 , ..., Pn , Py ) , where W denotes weights of the rules, ( P1, P2 , ..., Pn ) denotes the parameters of membership functions of terms for the input variables ( x1, x2 , ..., xn ) and Py denotes the parameters of membership functions of terms for the output variable y .
For a term-set t l1, l2 , ..., lK of a variable x on an interval x; x , the following parameters are configurable: (b2 , b3 , ..., bK −1 ) – the cores of non-extreme terms l2 , l3 , ..., lK −1 ; (q1 , q2 , ..., qK −1 ) – the crossing points of adjacent fuzzy sets on the interval x; x ; c1 – the concentration coefficient of the membership function for term l1 .
Thus, for a variable x , the vector of tuning parameters is set up as follows: Px = (q1, b2 , q2 , b3 , ..., bK −1, qK −1, c1 ) . This allows to use the constraint for interpretability protection in a convenient form (12), in contrast to more cumbersome and subjective expression in [5, 10].
Based on Px , the concentration coefficients for the membership functions of li are qi−1 − bi , i = 2, K . calculated as follows: ci = ci−1 qi−1 − bi−1 6
The proposed models and methods are implemented in the form of fuzzy identification information technology. Fuzzy identification is carried out in 4 stages, according to the concept of Generation – Selection – Reduction – Tuning.
At the first stage Fuzzy Rules Generation, fuzzy rules are generated from experimental data by the direct pass method. This method is based on the ideas of Wang– Mendel method [22], so we only distinguish that the term chosen is not with the maximum grade for any one row of the data sample, but the term with the maximum average membership for all data from the corresponding zones of factor space. If the analogue of the Wang–Mendel method is the fuzzy classification algorithm with a single winner rule, then the analogue of the propose method is the fuzzy classification algorithm with voting rules [23]. Usually fuzzy classifiers with voting rule scheme provide better accuracy, therefore, a similar scheme is chosen to rule generation.
The second stage is Fuzzy Rules Selection. It is implemented using a binary genetic algorithm with conditions (16)–(17). Optionally, constraint parameters can be estimated from the learning curves that produced by the greedy algorithm.
The third stage is Antecedents Reduction. It is implemented by the same binary genetic algorithm using alternative coding of each rule with Don’t care terms.
The last stage is Tuning. It changes rule weights and membership function using gradient and quasi-Newton optimization methods. For interpretability protection the optimization is carried out with constrains (12). 7
Fuzzy identification experiments were carried out for 7 real tasks (Table 1). Each dataset was divided into training and test samples. The training set includes odd lines of source data, and the test set includes even lines.
0.9 . Quasi-Newton method is used as optimization routine for tunlowing scaled number of rules Rules =
Amax ing the rule base. The optimization lasted 150 iterations, and two cases are inspected – with and without interpretability protection.
At first, large rule bases were generated; the number of rules for some tasks exceeded several hundred (Fig. 6). After selection, the number of rules was significantly reduced to a level comparable to FCM bases.
During antecedents reduction the number of rules didn’t change, but some rules became short.
To show the effect of antecedents reduction the red bars on Fig. 6 present the folA N
. In some cases it was possible to halve
Amax the length of antecedents. For SkillCraft task, FCM was not used due to missing some data in the data set. In general, after tuning, the fuzzy rule bases that synthesized by proposed method are a little more accurate than in the case of the FCM method (Fig. 6). Moreover, the proposed method produces not only accurate rule bases, but also interpretable. In addition, the rule bases that were synthesized by the proposed method, have a lot of short rules. For example, for CMB task, the length of the antecedents is reduced by 4 times. Selection of rules and reduction of antecedents often reduce the total number of terms also (Fig. 7). The less terms are in the rule base, the simpler the task of membership functions tuning. A significant reduction in the number of terms took place for tasks with a large number of inputs: SkillCraft, Housing and CMB.
The experiments showed that the proposed constraints for interpretability protection do not reduce the accuracy (Fig. 8). The difference between accuracy in the case of tuning with the constraints and without them is only a few hundredths for six tasks. This confirms that for a correctly synthesized fuzzy rule base, interpretability does not harm accuracy.
The proposed information technology of fuzzy identification provides the synthesis of accurate, compact and interpretable rule bases. Information technology includes 4 stages: the generation of reliable fuzzy candidate rules, selection of rules, reduction of antecedents and tuning the membership functions and rule weights. Computer experiments on 7 tasks from the UCI Machine Learning Repository showed, that the proposed information technology synthesizes fuzzy rule bases with accuracy and compactness at the level of competitive technologies. At the same time, unlike competitors, the proposed technology provides the interpretability of fuzzy rule bases. 7. Cordon O.: A historical review of evolutionary learning methods for Mamdani-type fuzzy rule-based systems: Designing interpretable genetic fuzzy systems. International Journal of Approximate Reasoning. 52, 894–913 (2011). doi: 10.1016/j.ijar.2011.03.004. 8. Gacto, M. J., Alcalá, R., Herrera, F.: Interpretability of linguistic fuzzy rule-based systems: An overview of interpretability measures. Information Sciences. 181, 4340–4360 (2011). doi: 10.1016/j.ins.2011.02.021. 9. Guillaume, S., Charnomordic, B.: Learning interpretable fuzzy inference systems with
FisPro. Information Sciences. 181, 4409–4427 (2011). doi:10.1016/j.ins.2011.03.025. 10. Shtovba S.: Ensuring Accuracy and transparency of Mamdani fuzzy model in learning by experimental data. Journal of Automation and Information Sciences. 39, 39-52 (2007). doi: 10.1615/JAutomatInfScien.v39.i8.50. 11. Jin, Y., Cao, W., Wu, M., & Yuan, Y.: Accurate fuzzy predictive models through complexity reduction based on decision of needed fuzzy rules. Neurocomputing, 323, 344-351 (2019). doi: 10.1016/j.neucom.2018.10.010. 12. Tsakiridis, N. L., Theocharis, J.B., Zalidis, G.C.: DECO3RUM: A differential evolution learning approach for generating compact Mamdani fuzzy rule-based models. Expert Systems with Applications. 83, 257-272 (2017). doi: 10.1016/j.eswa.2017.04.026. 13. Ojha, V., Abraham, A., Snášel, V.: Heuristic design of fuzzy inference systems: A review of three decades of research. Engineering Applications of Artificial Intelligence. 85, 845864. (2019). doi: 10.1016/j.engappai.2019.08.010. 14. Mamaghani, A.S., Pedrycz, W. Structural optimization of fuzzy rule-based models: Towards efficient complexity management. Expert Systems with Applications. 152, 1-12 (2020). doi: 10.1016/j.eswa.2020.113362. 15. Ishibuchi, H., Nozaki, K., Yamamoto, N., Tanaka, H.: Selecting fuzzy if-then rules for classification problems using genetic algorithms. IEEE Transactions on Fuzzy Systems. 3, 260–270 (1995). doi: 10.1109/91.413232. 16. Liu, B.: Theory and practice of uncertain programming. Studies in Fuzziness and Soft
Computing, Vol. 103, Physica-Verlag, (2002). doi: 10.1007/978-3-7908-1781-2. 17. Miller G.A.: The magic number seven plus or minus two: some limits on our capacity for processing information. Psychological Review. 63, 81–97 (1956). doi: 10.1037/h0043158. 18. Ishibuchi, H., Murata, T., Turksen, I. B.: Single-objective and two-objective genetic algorithms for selecting linguistic rules for pattern classification problems. Fuzzy Sets and Systems. 89, 135–150 (1997). doi: 10.1016/S0165-0114(96)00098-X. 19. Shtovba, S., Mazurenko, V., Savchuk, D.: Genetic algorithm for selecting rules of the fuzzy knowledge base balanced according to the accuracy and compactness criteria. Scientific Works of Vinnytsia National Technical University. 3 (2012). 20. Martello, S., Toth, P.: Knapsack problems: algorithms and computer implementations.
John Wiley & Sons, New York, (1990). 21. Cordon, O., Gomide, E., Herrera, E., Homannc, E., Magdalena, L.: Ten years of genetic fuzzy systems: current framework and new trends. Fuzzy Sets and Systems. 141, 5–31 (2004). doi: 10.1016/S0165-0114(03)00111-8. 22. Wang, L.X., Mendel, J.M.: Generating fuzzy rules by learning from examples. IEEE Transactions on Systems, Man and Cybernetics. 22, 1414-1427 (1992). doi: 10.1109/21.199466. 23. Ishibuchi, H., Nakashima, T., Morisawa, T.: Voting in fuzzy rule-based systems for pattern classification problems. Fuzzy Sets and Systems. 103, 223–238 (1999). doi: 10.1016/S0165-0114(98)00223-1.