<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta>
      <journal-title-group>
        <journal-title>First Workshop on Online Learning from Uncertain Data Streams, July</journal-title>
      </journal-title-group>
    </journal-meta>
    <article-meta>
      <title-group>
        <article-title>An explainable evolving fuzzy neural network in position identification of basketball players</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Paulo Vitor de Campos Souza</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Edwin Lughofer</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Johannes Kepler university Linz. Institute for Mathematical Methods in Medicine and Data Based Modeling.</institution>
          <addr-line>Altenberger Strasse 69, 4040, Linz</addr-line>
          ,
          <country country="AT">Austria</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2022</year>
      </pub-date>
      <volume>18</volume>
      <issue>2022</issue>
      <fpage>0000</fpage>
      <lpage>0002</lpage>
      <abstract>
        <p>Evolving fuzzy neural networks have the adaptive capacity to solve complex problems. They can facilitate understanding the behavior of the analyzed problem as they can extract knowledge from an analyzed data set. Thus, this work proposes applying an evolving fuzzy neural network capable of solving pattern classification problems with considerable interpretability in the position identification of basketball players. The models used in these tests were compared to the state-of-the-art on the subject, and their results were superior (87.50 % of accuracy) and interpretable. In addition to being accurate in solving the problems, the model presented relevant information on data stream processing, allowing a complete evaluation of the data behavior during its evaluation.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;Evolving fuzzy neural networks</kwd>
        <kwd>Evolving fuzzy systems</kwd>
        <kwd>Interpretability</kwd>
        <kwd>Data streams</kwd>
        <kwd>Position identification of basketball players</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        Evolving Fuzzy neural networks are hybrid methods combining the notions of artificial neural
networks and fuzzy logic [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. Consequently, these models can solve complex problems while
extracting knowledge from the data, making them expert systems [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ]. The association of a fuzzy
inference system and artificial intelligence training methodologies streamlines the performance
of these models [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ], making them applicable, for example, to dynamical systems.
      </p>
      <p>
        Evolving fuzzy neural networks are models with a high capacity for solving complex problems,
adding interpretability to the results. However, many proposed models do not present alternative
ways of interpreting the results, especially when confronted with the evaluation of stream data
or surveying the model’s behavior over time [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]. To bridge this interpretability gap, models
were proposed in the literature with the ability to extract knowledge about the analyzed data,
in addition to exploring the joint interpretability of the functioning of these models as they
evaluate new data [
        <xref ref-type="bibr" rid="ref5 ref6 ref7 ref8">5, 6, 7, 8</xref>
        ]. These interpretation factors in the evaluation of streaming data
can identify outliers in the data, garner new insight, and new situations that can add knowledge
to the users of these systems.
      </p>
      <p>
        This paper aims to present the functioning of an evolving fuzzy neural network (ENFS-Uni0)
[
        <xref ref-type="bibr" rid="ref7">7</xref>
        ] in solving pattern classification problems. In this work, we intend to explain how the
interpretability of the results with the interpretability of the model’s behavior can help in the
complex analyses to be carried out in a data set. ENFS-Uni0 is an evolving model with three
layers, with the first one being composed of the data fuzzification process. This procedure
is performed by an evolving fuzzification technique based on data density and is responsible
for constructing Gaussian neurons to represent the data of a problem. The second layer is
composed of fuzzy logic neurons, which aggregate the neurons formed in the first layer with
their respective weights. Finally, the third layer of the model is represented by an aggregation
artificial neural network, responsible for the defuzzification process. Aspects of interpretability
about the evolution of Gaussian neurons, evaluation of the features of the problem, and training
that defines consequents of interpretable rules are present in the model to reach an advanced
level of interpretation of the results. The interpretability of evolving fuzzy systems models
meets diferent criteria from traditional models of artificial neural networks. Therefore, it is
intended to present practical examples of action in solving problems in an interpretable way
following the interpretability criteria proposed by Lughofer [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ] for this category of models.
These examples will be provided by analyzing the model’s manners as it performs the task
of pattern classification problems. The interpretability obtained by the model through fuzzy
rules will be compared with the approach of fuzzy systems that are also capable of extracting
knowledge about an analyzed data set. Thus, it may be possible to measure the degrees of
interpretability achieved by the ENFS-Uni0.
      </p>
      <p>This article proposes to work with the resolution of a multiclass problem of basketball players’
position profiles on the court by comparing the results obtained with fuzzy solutions that also can
extract knowledge. The highlights of this paper are to present the main features of interpretation
of the problems that the evolving fuzzy neural networks have. In addition to this element, it also
proposes solving a complex classification problem, that has been published recently, along with
the interpretation of its results. Thus, the reader can appreciate the characteristics of accuracy
and interpretability, reach an understanding of evolving fuzzy systems and identify how hybrid
methods could be combined to exploit their benefits for online (evolving) learning.</p>
      <p>In addition to the introduction, the paper presents a theoretical reference section (section 2)
related to the concept of evolving fuzzy models and their respective interpretability capabilities.
Section 3 shows the reader the layers and training of the model. Experiments and their
discussions are highlighted in the Sec. 4 and section 5 sections respectively. Finally, in the 6 section,
conclusions about the activities carried out for this paper are given.</p>
    </sec>
    <sec id="sec-2">
      <title>2. Evolving fuzzy systems and interpretability</title>
      <sec id="sec-2-1">
        <title>2.1. Evolving Fuzzy Systems</title>
        <p>
          Evolving fuzzy systems (EFS) are to bet set apart from traditional neural network models mainly
by their ability to combine the advantages of fuzzy inference systems with the training of
artificial neural network devices, operating to adapt the parameters according to the dynamics
of the data. Thus, the knowledge extraction can be helpful to others while the model has an
advanced and assertive problem-solving capacity [
          <xref ref-type="bibr" rid="ref10">10</xref>
          ].
        </p>
        <p>
          They are models which build upon the concept of fuzzy logic (truth values are fuzzified in
[
          <xref ref-type="bibr" rid="ref1">0,1</xref>
          ] by membership functions [
          <xref ref-type="bibr" rid="ref11">11</xref>
          ]). They have a tradeof between universal approximation
(models with a high-handed degree of non-linearity by piece-wise local approximation) and
interpretability (understandable linguistic terms and rules) [
          <xref ref-type="bibr" rid="ref10">10</xref>
          ].
        </p>
        <p>
          The main concepts about evolving fuzzy systems are their ability to process samples or data
blocks step by step for model building, omitting time-consuming retraining (incrementality),
performing recursive parameter adaptation (adaptability), and adding structural elements
(evolving) as the system deals with newly loaded samples or has new states. EFS also is a fuzzy system
in a single-pass incremental and evolving manner with online recordings/data streams with
similarities between specific architectures with certain types of neural networks. Evolving fuzzy
systems are considered (dark) gray-box models with knowledge-based input and data-driven
learning [
          <xref ref-type="bibr" rid="ref10">10</xref>
          ]. Its main requirements for real applications are linked to quick online
identification of models from scratch, updating and extending existing models, bringing reliability and
security to the process, avoiding extrapolation, and working with model adjustments based
on the knowledge extracted from the data [
          <xref ref-type="bibr" rid="ref12">12</xref>
          ]. EFS can also work on extracting models from
massive databases since, in most fundamental problems [
          <xref ref-type="bibr" rid="ref12">12</xref>
          ], it is not possible to load all the data
at once. Its ability to improve the human-machine interaction by monitoring deviations/changes
in data flow (gradual forgetting, smoothing over time) is also highlighted [
          <xref ref-type="bibr" rid="ref10">10</xref>
          ].
        </p>
        <p>
          Evolving fuzzy neural networks (EFNN) are EFS and have an input and an output layer as the
main element. The hidden layers of these models vary, depending on the number of features in
the fuzzy inference systems and the aggregations performed by the neural networks [
          <xref ref-type="bibr" rid="ref1">1</xref>
          ]. This
ability provided by the fuzzy inference system, that makes up the evolving fuzzy neural network,
enables it to build fuzzy rules and demonstrate knowledge over time. Thus, it is possible to
analyze the results of knowledge evolution as new samples are presented for the training and
evaluation of the model [
          <xref ref-type="bibr" rid="ref3">3</xref>
          ].
        </p>
      </sec>
      <sec id="sec-2-2">
        <title>2.2. On-line assurance of interpretability</title>
        <p>
          Evaluating the interpretability of intelligent models has been exhaustively addressed in the
literature, mainly in the XAI (Explainable AI) [
          <xref ref-type="bibr" rid="ref13">13</xref>
          ] research area, where humans can understand
the solution results. A general definition of interpretability is that it can be seen as how a
human being can understand the cause of a decision or the degree to which a human being
can consistently predict the model’s outcome. It is an established factor that, the greater the
interpretability of a machine learning model, the easier it is for users to understand why certain
decisions or predictions were made [
          <xref ref-type="bibr" rid="ref14">14</xref>
          ].
        </p>
        <p>
          Interpretable machine learning is an embracing term that captures extracting relevant
knowledge from a machine learning model about relationships contained in data or learned by the
model. In the context of evolving learning, some characteristics are necessary to be adapted
so that the evaluation of these models is done in a way that humans understand them, mainly
because of their adaptive behavior [
          <xref ref-type="bibr" rid="ref14">14</xref>
          ]. Lughofer [
          <xref ref-type="bibr" rid="ref9">9</xref>
          ] addressed the criteria of transparency,
readability, and interpretability of the EFS fuzzy rule bases (which, consequently, also cover
the EFNN). For this purpose, some essential criteria for evaluating these models were defined,
which are listed below [
          <xref ref-type="bibr" rid="ref9">9</xref>
          ]:
• Distinguishability and simplicity: Simplicity demands models with a tradeof between
low complexity and high precision, while distinguishability requires using structural
components (rules, fuzzy sets) in a separable way (non-overlapping and non-redundant).
• Consistency: Consistency in a rule base is given when two rules do not overlap in
antecedents and consequents. Such occurrences lead to a more remarkable case of conflict
within an evolving classification context as classes overlap within the same local region.
A fuzzy rule is consistent with another fuzzy rule if the similarity of its antecedents is
less than the similarity of its consequents.
• Coverage and completeness: Coverage refers to the specific characteristics of fuzzy
partitions and rules that do not allow any holes in the resource space, hence undefined
input states. Completeness is the evaluation of the contribution of rules with a significant
distance to the sample. It can be seen as a generalization of coverage.
• Feature importance levels: This ability assesses the importance of features in the final
output of the model, allowing an assessment of their influence to bring interpretability to
the process and reduce the rule length.
• Rule importance levels: Criteria for evaluating importance levels of rules defined by
numerical values (weights or rule consequents) to assess the relevance of the rule for the
analyzed context.
• Interpretation of consequents: Possibility to evaluate the consequents of the model
through distinguishability, simplicity, and completeness for fuzzy Mamdani systems and
rule confidence for single-model fuzzy classifiers that have the form of a rule.
• Knowledge expansion: Evaluation of results beyond just accuracy. Criteria for
incorporating new knowledge, evaluations, and operations with the rules serve as parameters for
this criterion. Rule evolution criteria also allow the identification of an assessment of the
knowledge acquired by the model.
        </p>
        <p>
          Studies on the interpretability of models based on fuzzy rules have found a conflict between
the precision of the model and its interpretability- two objectives that are at odds with each
other. Preserving the interpretation of a fuzzy system during adaptation is a dificult task that
has received much attention in the fuzzy system modeling community [
          <xref ref-type="bibr" rid="ref15">15</xref>
          ]. Therefore, to be
considered interpretable, an EFNN model must meet high accuracy criteria (to guarantee the
eficiency of the results) and must also meet the criteria listed above. These listed criteria will be
used to evaluate the evolving fuzzy neural network model results in this paper. The following
section will present the model architecture, training, and interpretability techniques.
        </p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>3. Evolving Fuzzy neural network</title>
      <p>
        This section presents the architectural and training features of the ENFS-Uni0 [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. The aspects
of its operation help in the interpretability of online problems, allowing the behavior of the
evaluation of the model as it analyzes the data set samples. The architecture of ENFS-Uni0 is
composed of three layers. The first two are a fuzzy inference system (responsible for extracting
knowledge from the data set through IF-THEN rules). A neural network represents the last
layer capable of aggregating all the consequents of fuzzy rules and transforming them into the
expected output (defuzzification process) [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ].
      </p>
      <p>
        The model parameters are established during a fuzzification process, which determines the
number of fuzzy rules that the system can extract from the data. This process results in the
Gaussian neurons formed with the centers and the standard deviation of the clusters found by
means of an autonomous data clustering technique [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ]. These clusters are defined through data
density concepts and empirical data operators [17]. Therefore, these Gaussian neurons generated
by the fuzzification process are responsible for the composition of the antecedent terms of
the fuzzy rules [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. Each of these neurons has a weight determined by an online technique
for determining the class separability criterion of the problem (feature weight separability
criteria (FWSC)) [18]. This approach brings benefits of interpretability to the dimensions of the
problem (identifying those with greater relevance to find the analyzed classes) and allowing
the rules’ reducibility, generating a compact knowledge about the problem in question [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ].
These neurons and their respective weights are aggregated in the second layer of the models
through fuzzy logic neurons. These neurons use fuzzy aggregators to aggregate the weights
and Gaussian neurons and transform them into a single value [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. For this, the EFNS-Uni0 uses
the uni-nullneuron [19], composed of uni-nullnorms [20]. These special fuzzy operators allow
that there are diferent connectives of the antecedents in a group of fuzzy rules formed by the
models. For instance, when only one t-norm is used to aggregate neurons, all the generated
connectives are of the AND type. When using a t-conorm, the whole set of generated rules
uses the OR connective [21]. The use of uni-nullnorm derives from the concepts of n-uninorms
[20], which allow fuzzy operators to vary between uninorms [22, 23] or nullnorms [24] (two
particular types of fuzzy operators that allow the use of t-norms and t-conorms together) [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ].
      </p>
      <p>
        Thus, a set of rules that expresses knowledge about a data set may contain rules either
with AND or with OR connectors. This skill facilitates the dynamic resolution of complex
problems, allowing for knowledge extraction with flexibility. This process performed by the
uni-nullneuron uses three parameters to determine what type of operator the neuron will use to
perform a given aggregation [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. The rule consequents are determined diferently according to
the stage in which the model is executing its activities to complete the formation of fuzzy rules.
The Extreme Learning Machine concept [25] (using pseudo-inverse of the Moore-Penrose matrix
[26]) is applied in the ofline stage. In the evolving phase of the model, the rule consequents are
updated by a technique inspired by a version of recursive weighted least squares [27] called
indicator-based recursive weighted least squares (I-RWLS) for each class [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ], where there is a
value for each of the analyzed classes. The process of obtaining the model output is performed
by a neural aggregation network that uses all the consequents of the rules as weights of the
artificial neuron (Singleton concept). This neuron, which has a linear activation function, is
responsible for the model result [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ].
      </p>
      <p>
        The original EFNS-Uni0 technique also uses a pruning technique to select the most relevant
neurons for the model. However, this approach will not be used in this paper, as we want to
see the impacts and evolution of all the knowledge acquired during the tests. The model has
abilities to extract knowledge and interpret the results. The fuzzy rules formed by EFNS-Uni0
can be represented by [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]:
where  is the number of classes in the problem, A are the Gaussian neurons (where the
membership functions of fuzzy sets, formed by the input data density through an evolving
clustering method, are the activation functions of the corresponding neurons i.e.,  =  for j
= 1... N and l = 1 . . . L, where N is the number of inputs and L is the number of fuzzy sets for
each input) and w is its respective weight ( (for i = 1... N and l = 1... L). ⃗ = [1, ...,  ] is
calculated in two diferent ways. The ofline phase is based on Eq.
      </p>
      <sec id="sec-3-1">
        <title>2 and the evolving step uses Eq. 5 related below.</title>
        <p>⃗ = +⃗ ∀ = 1, ..., 
 = ⃗−1 (︀  + (⃗) −1⃗)︀ −1
 = (I − ⃗)−1−1
⃗ = ⃗
−1 +  (−⃗⃗</p>
        <p>−1)
1 :  1  11 ℎ  11...
/(,,) 2  12 ℎ  21...</p>
        <p>2 :  1  21 ℎ  12...
/(,,) 2  22 ℎ  22...</p>
        <p>ℎ 1  [11...1 ]
 ℎ 2  [21...2 ]
.... :  1  1 ℎ  1...</p>
        <p>/(,,) 2  2 ℎ  2...</p>
        <p>ℎ   [1... ]
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
+ =   is the pseudo-inverse of the Moore-Penrose matrix [26] of  (uninull-neurons (Eq.
6)) and  denotes the column indicator vector containing 1s at the row positions for samples
belonging to class k and 0 for all samples belonging not to class .  is the current kalman
The  is a uni-nullneuron vector, and this fuzzy neuron can be represented by:
gain (row) vector,  is an identity matrix based on the number of neurons in the second layer,
 × ;  ∈]0, 1] denotes a possible forgetting factor, but is to 1 per default (no forgetting). 
denotes the inverse Hessian matrix  = ( )−1 and is set initially as  , where =1000.
 =   (, , , , ) =  =1 (, , , , )
(, , , , ) =
︃{</p>
        <p>+ ¯  ,  1
 + ¯ 1−− ,  2
 (, , , , ) =
︃{</p>
        <p>
          1(  ,  ),  x, y ∈ [0, ]
 + (1 − ) 2 ( 1−
− , 1−− ),  x, y ∈ (, 1]
where p is a conditional transformation of the values of the uni-nullnorm (Eq. (8)- commutative
binary function  : [
          <xref ref-type="bibr" rid="ref1">0, 1</xref>
          ]2 → [
          <xref ref-type="bibr" rid="ref1">0, 1</xref>
          ], with , ,  ∈ [
          <xref ref-type="bibr" rid="ref1">0,1</xref>
          ] with 0 ≤  ≤  ≤  ≤ 1 and
0 &lt;  &lt; 1 )). 1 is a uninorm with a neutral element (identity) =  and 2 is a uninorm using
( 1−− ) like as neutral element.
        </p>
        <p>
          The assessment of the changes of Gaussian neurons in the first layer can measure how the
rule antecedents change over time [28]. Other interpretable factors that the model can also
measure are identifying the evolution of fuzzy rules and their respective consequents. This
overview allows the model’s user to identify moments of change in the model’s eficiency or
even to measure moments in which the technique acquired new knowledge [
          <xref ref-type="bibr" rid="ref7">7</xref>
          ]. Details on how
the techniques work can be seen in depth at [
          <xref ref-type="bibr" rid="ref7">7</xref>
          ].
        </p>
        <p>The process carried out in the third layer is also seen as defuzzification process with an
aggregation neural network composed of a single neuron.</p>
        <p>= Ω
⎛
⎝
︁∑</p>
        <p>=0
Γ( ,  )⎠
⎞
outputs, ∑︀</p>
        <p>is given by:
where 0 = 1, 0 is the bias, and  and  , j = 1, ..., l are the output of each fuzzy neuron of
the second layer and their corresponding weight, respectively and Γ represents the neuron
activation function. When the model acts in solving problems with multiple classification
the th neuron), where each entry is the overall certainty (among all neurons/rules) that the
sample belongs to the corresponding class. Therefore, when the model has  class outputs, Ω
=0 Γ( , ⃗ ) delivers an output vector ⃗ (as  turns into a vector of outputs ⃗ for
Ω = =1,..., 
(9)
(10)
The architecture of the model can be seen in Fig. 1 and the flow of its operation can be visited
in Fig. 2. A pseudo-code of ENFS-Uni0 is presented in Algorithm 1.</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Experiments</title>
      <p>This paper aspires to demonstrate the ability of evolving fuzzy neural networks in the
interpretability of pattern classification problems. To this end, some experiments and comparisons</p>
      <sec id="sec-4-1">
        <title>Algorithm 1 ENFS-Uni0 Training and Update Algorithm</title>
        <p>Initial Batch Learning Phase (Input: data matrix  with 
samples):
(1) Extract L clouds in the first layer using the ADPA approach (  is automatically estimated
therein).
(2) Estimate center values ⃗ and widths ⃗ for the  clouds derived from ADPA.
(3) Calculate the combination (feature) weights ⃗ for neuron construction using FWSC.
(4) Construct L logic neurons on the second layer of the network by welding the L fuzzy
neurons of the first layer, using uni-nullnorms concept and the centers ⃗ and widths ⃗.
(5)
for  = 1, ...,  do
(5.1) Calculate the regression vector ().</p>
        <p>(5.2) Store it as one row entry into the activation level matrix .
end for
(6) Extract reduced activation level matrix  according to the  neurons.
(7) Estimate the weights of the output layer for all classes  = 1, ...,  by ELM approach
using  and indicator vectors ⃗ .</p>
        <p>Update Phase (Input: single data sample ⃗):
(8) Update  clouds and evolving new one on demand (due to rule evolution conditions) in
the first layer using extended evolving ADPA approach (→ , clouds).
(9) Update the feature weights ⃗ by updating the within- and between-class scatter matrix
and recalculating FWSC.
(10) Perform Steps (2) and (4).
(11) Calculate the degree of change of all neurons.
(12) Calculate the regression vector (⃗).</p>
        <p>(13) Update the weights of the output layer by I-RWLS.
with state-of-the-art models will be carried out and discussed. The quality measure evaluated
in this paper is the accuracy (ACC):</p>
        <p>+  
 =   +   +   +   * 100.
where   = true positive,   = true negative,   = false negative and   = false positive.</p>
        <p>Another method of analyzing accuracy using EFNN is combined with the trend-line method
of the stream mining case. In these cases, the accuracy is cumulatively updated by:
( + 1) = () *  + ^= , (12)
 + 1
Where  represents the indicator function, if the prediction is correct, it is 1, that is, ˆ = ;
otherwise, it is 0 ((0) = 0); after updating the accuracy, the model will be updated, so that
produces an interleaved test-then-train protocol that is widely used in the data stream mining
community [29] to exploit the predictive power of incremental adaptive (and evolving) models.</p>
        <p>All tests were run on a computer with the following settings: Intel(R) Core(TM) i7-6700 CPU
3.40GHz, 16GB RAM.</p>
        <sec id="sec-4-1-1">
          <title>4.1. data set</title>
          <p>The data set 1 used in the experiments is the Spanish Basketball League ACB data set: The
data set consists of 80 samples (perfectly balanced with 20 samples per class) corresponding to
the four classes (point guard, shooting guard, small forward, center) that are associated with
13 attributes (height, blocks (Fig. 3 -a), rebounds, assists, points, personal fouls committed,
personal fouls received, free throw percentage (Fig. 3 -b), 2-point field goal percentage, 3-point
ifeld goal percentage, turnover, steals, and global assessment). The numerical values associated
with each sample correspond to the statistics available online at the website of the Spanish
Basketball League ACB. The authors have collected statistics from each player’s 2017-2018
season [30].</p>
          <p>This data set has already been addressed in another academic research [31] with a maximum
accuracy of 63%.</p>
        </sec>
        <sec id="sec-4-1-2">
          <title>4.2. Models used in the experiments</title>
          <p>The models used in the test are presented below. Except for the models provided by the online
solution, which already have predefined parameters, the other models previously underwent a
10-fold procedure with cross-validation of 70%-30% for the definition of parameters.
1https://citius.usc.es/investigacion/datasets/basketballplayers</p>
          <p>Evolving Neuro-Fuzzy System based on Uni-Nullneurons (ENFS-Uni0) –The evolving
fuzzy neural network is used as a reference in this study. It uses uni-nullneurons, and the
non-regularized approach was adopted to analyze all rules.</p>
          <p>Autonomous Learning Multimodel (ALMMo) — A model is a neuro-fuzzy technique
for autonomous zero-order multiple learning with pre-processing that improves the classifier
and the approximation model accuracy by creating stable models. The parameter is radius =
2-cos(30∘ ) [32].</p>
          <p>Fuzzy Hoefding Decision Tree (FHDT)— Approach for incremental learning of multi-way
in classification tasks with uniform fuzzy partitions for each input attribute: selecting the
best input attribute to be used for the splitting at each node is performed by using the fuzzy
information gain defined [33]. The configuration is based on the online solution 2</p>
          <p>Fuzzy Unordered Rule Induction Algorithm -(FURIA) — Fuzzy rule-based classification
is a method for classifying items using fuzzy rules with fuzzy sets of trapezoidal shapes in the
antecedent of each rule. FURIA uses fuzzy logic to learn how to operate instead of relying on
rigid rules and ordered rule sets. The configuration is based on the online solution [34]. 3</p>
          <p>Self-Adaptive Fuzzy learning (SAFL) — A novel self-adaptive fuzzy learning (SAFL) system
is proposed for streaming data prediction with a set of prototype-based fuzzy rules. The
parameters are 0=0.5, Ω0 = 1000, 0 = 0.054</p>
        </sec>
        <sec id="sec-4-1-3">
          <title>4.3. Results</title>
          <p>The results of classification experiments is presented in Table 1. As for the pattern-classification
tests were 50% for training and 50% for testing. In the evaluation of stream data, 20% of the
samples were destined for training and the rest for testing. This facilitates the evaluation of the
data in trendlines presented in the Fig. 4. In addition to the results in trendlines of the model,
the evolution criteria of the first layer Gaussian neurons are also presented (Fig. 5), and the
evolution and changes over time of the relevance of the features to the problem impact directly
on the weights of Gaussian neurons (Fig. 7) and the behavior of fuzzy neurons relative to their
evolution during the experiment (Fig. 6).</p>
        </sec>
      </sec>
      <sec id="sec-4-2">
        <title>2https://demos.citius.usc.es/ExpliClas 3https://demos.citius.usc.es/ExpliClas 4defined according to the experiments performed in [35].</title>
      </sec>
    </sec>
    <sec id="sec-5">
      <title>5. Discussions</title>
      <p>The discussions carried out in this paper seek to bring a relationship between the main
interpretability characteristics of an evolving fuzzy system model and the results obtained in the
experiments performed.</p>
      <sec id="sec-5-1">
        <title>5.1. Distinguishability and simplicity</title>
        <p>The simplicity of the ENFS-Uni0 model in solving the problem of identifying the position of
basketball players showed in the form of obtaining the best results in the evaluation of trendlines
(Fig. 4) when compared to state-of-the-art models with a rule evolution that goes from 4 initial
rules to a total of 10 fuzzy rules for solving the problem (Fig. 6). The distinguishability aspects
can be seen in Fig. 5, where it is possible to identify during the analysis when the Gaussian
neurons obtained changes as the model evaluated new samples. This behavior facilitates the
understanding of how the architecture changed as new significant samples were evaluated by
the model and, at the same time, identifies how the distinction provided by the Gaussian neurons
originated in the fuzzification process helped the adaptation of the model in the identification
of basketball player profiles. The centers of the clusters initially formed by the algorithm with
the two dimensions most relevant to the problem according to Fig. 7 can be seen in Fig. 8. The
ifnal version with the first layer of Gaussian neurons can be seen in Fig. 9. In both cases, no
overlapping was identified, despite the centers of rules 2 and 5 being very close.</p>
      </sec>
      <sec id="sec-5-2">
        <title>5.2. Consistency</title>
        <p>The analysis of the antecedents of fuzzy rules in this context varies according to the value of
the Gaussian membership functions elaborated by the fuzzification method and the respective
weight generated by the feature weight technique. In the context of evolving classification, it
is essential to analyze whether, after training, the rules received the impact of the evaluation
of a new sample. Fig. 7 presents a dynamic variation of the weight values for the neurons of
the first layer, thus altering the similarity of the antecedents when compared to the rule in its
previous format. The dynamics of evaluating the similarity of Gaussian neurons can be seen in
Fig. 5. A factor that should also be highlighted in this figure is that rules 3, 7, and 10 maintain
a high degree of similarity in their antecedents throughout the evaluation. Therefore, these
rules can be evaluated on their consistency. Table 2 presents an evaluation obtained during
the experiment comparing rule 3 and Table 3 for the rule 7 in the first moment of evolution
with regard to their respective changes when a new class appears (for rule 3 when the shooting
guard class appears and for rule 7 when the small forward class is presented to the model). It is
noted that membership functions change, but the similarity of antecedents does not change. At
the same time, there is a change in the consequents of the rules.</p>
        <p>Therefore, the aforementioned rules are considered consistent because even with the similarity
of the Gaussian neurons equal 1, the weights strongly impact the construction of the rule
antecedents, generating assertive results at the end of the evaluation.</p>
      </sec>
      <sec id="sec-5-3">
        <title>5.3. Coverage and completeness</title>
        <p>In evaluating this criterion, rule consequents allow an assessment that, for all samples, at least
one rule is triggered to a significant degree. The model proved to be able to activate at least one
rule for each group formed by the model. The initial evaluation situation, where only one rule
is activated (because only one class was presented to the model), is presented in Table 4. The
other rules always activated evaluations of a class of the problem throughout the experiment.
The interpretability of the rule consequents at the end of the experiment is presented in Table5,
demonstrating the relevance of each rule for a particular class of problem. For these tables, a
rule that has no relation to the class in question is considered very small (0% probability). The
one with the highest probability of belonging to that class is considered high (between 85% and
100%). This evaluation is completed by the values small (between 1 and 15% of probability) and
medium (between 16 and 84% of probability).</p>
      </sec>
      <sec id="sec-5-4">
        <title>5.4. Feature importance levels</title>
        <p>The importance of the features of the problem brings several evaluations of interpretability to
the results. First, as can be seen in Fig. 7, it is possible to evaluate the evolution of features as
new data is analyzed. This allows a dynamic assessment of how specific samples change the
separability of classes as they are analyzed. A key factor in the interpretability of this model
was the identification of the most relevant features to correctly identify the problem classes,
namely height, blocks, rebounds, and 3P Fields goals percentage. This evaluation demonstrates
that the model can correctly diferentiate the classes and is consistent with accurate evaluations
of a basketball game. A player’s height is a determining factor for his position on the court. His
ability to block, rebound and shoot three-pointers can also facilitate his on-court functionality.
These characteristics found by the model are also highlighted in studies in the literature on the
positions of players on the court [36].</p>
        <p>Another point to be highlighted in this context is connected to the reducibility of fuzzy rules
generated according to the features of the problem. The features that contribute the least to the
problem can be eliminated, thus creating more compact rules. The following is an example of a
fuzzy rule generated by the model:</p>
        <p>IF height is mf5 with impact 1.00 OR blocks is mf8 with impact 0.82 OR rebounds is mf7
with impact 0.81 OR assists is mf6 with impact 0.86 OR points is mf5 with impact 0.68 OR
personal fouls committed is mf10 with impact 0.69 OR personal fouls received is mf10
with impact 0.69 OR free throw percentage is mf10 with impact 0.72 OR 2-point field goal
percentage is mf9 with impact 0.79 OR 3-point field goal percentage is mf8 with impact 0.86
OR turnover is mf7 with impact 0.72 OR steals is mf7 with impact 0.74 OR global assessment
is mf5 with impact 0.69 THEN position is point guard: very small probability, shooting
guard: high probability, small forward: small probability, center: high probability.</p>
        <p>Let us consider a relevance criterion (for example, weights below 0.70 are considered
irrelevant) and replace the membership functions with linguistic terms. We can build a more compact
and interpretable fuzzy rule, as shown below 5.</p>
        <p>IF height is medium with impact 1.00 OR blocks is large with impact 0.82 OR rebounds
is large with impact 0.81 OR assists is medium with impact 0.86 OR free throw percentage
is very large with impact 0.72 OR 2-point field goal percentage is very large with impact
0.79 OR 3-point field goal percentage is large with impact 0.86 OR turnover is large with
impact 0.72 OR steals is large with impact 0.74 THEN position is point guard: very small
probability, shooting guard: high probability, small forward: small probability, center:
high probability.</p>
      </sec>
      <sec id="sec-5-5">
        <title>5.5. Rule importance levels</title>
        <p>The criteria for evaluating the relevance of rules are given in this experiment by evaluating
the weights of their consequents concerning each of the classes. Some rules have greater
relevance to determining one class over others. It is even possible to identify if there is duality in
5the linguistic terms were converted following the following criteria mf1 and mf2 = very small (short), mf3 and
mf4=small, mf5 and mf6=medium, mf7 and mf8=large(tall), mf9 and mf10= very large(tall).</p>
        <p>Point guard
large
small
very small
very small
shooting guard
small
small
medium
medium
identifying the rule if it has similar values for the identification of a class. The values presented
in Table 6 exemplify this context.</p>
        <p>Table 7 presents the changing behavior of the relevance of a first rule throughout the
evaluation of the model’s behavior as new samples are submitted to the model.</p>
        <p>As seen in Table7, as new samples are evaluated, the relevance of a rule for the determination
of specific class changes. Rule 1 corresponded to identifying the point guard profile in the
ifrst evolving iteration. In the second iteration, this rule started to have a small probability of
identifying the classes involved. In the third and fourth iterations, the rule started to act in
identifying the shooting guard class.</p>
      </sec>
      <sec id="sec-5-6">
        <title>5.6. Interpretation of consequents</title>
        <p>The evaluation of the consequents of this model allows us to identify the rules that best
collaborate in identifying a basketball player’s position. In this contextual evaluation, it is
possible to identify (based on the Tables 5 and 6) that for the point guard position, for example,
the rules that most contribute to identifying it are 2, 4, 7, 8 and 9. As for the shooting guard
position, rules 1, 3, 5, and 6 best classify this class in the model. In evaluating a small forward,
only rule 3 allows a high identification of this profile. Finally, the fuzzy rules of classifying
correctly for the center position are 1, 3, 5, 6, and 10. We can also infer the relevance of a rule to
ifnd a specific class on the evaluation of consequents. The rules that clearly define a basketball
player’s position are 4 and 8 for the point guard and 10 for the center position. This is due to
the probability of the other classes involved in the problem for these rules to be null. Rules 2
and 7 are also considered in identifying a point guard with a high probability for this position.
A study can also be carried out on the shooting guard position. All rules that can identify this
profile with high precision are also linked to identifying the center position. Rule 3 can belong
to characteristics of 3 groups.</p>
        <p>This evaluation even difers from the FURIA and FHDT models, which concluded that there
is confusion in the classifier between types of point guard and shooting guard positions (models
with 69.44% and 75.00% accuracy, respectively). One way of evaluating the consequents can
also be seen as an indirect pruning of neurons. When a rule has all its consequents come out as
zero, it can no longer represent a class. As the neural network works with the aggregation of all
fuzzy rules to perform the defuzzification process, a rule with zero weights will not contribute
to obtaining the answers.</p>
      </sec>
      <sec id="sec-5-7">
        <title>5.7. Knowledge expansion</title>
        <p>The assessment of the knowledge acquired by the model comprises some factors listed in
previous subtopics. The evolution of the rules (Fig. 6) indicating that the model acquired
knowledge about new data strongly evidences that this model was able to learn from the
samples. The interpretability of rule antecedents, features weights, and rule consequents allow
for a complete analysis of the problem, thus enabling a rule generated on a problem to provide
the most diverse information about a context. Another factor that evidences the expansion
of knowledge of the model is how several operational elements can compose the complete
extraction of knowledge about a data set. The explanation of the FURIA (Fig. 10) and FHDT
(Fig.11) models present graphical information about the problem, but they fail to analyze changes
sample by sample. Another diference in knowledge gain compared to these two models is
the final number of rules extracted: the FURIA model obtained 69.44% accuracy with seven
rules, the FHDT achieved 75.00% accuracy with only five, and the ENFS -Uni0 achieved 87.50%
accuracy with ten fuzzy rules. The models used in the trend line test only presented an accuracy
of their results, not identifying any interpretable evaluation of them. The use of ENFS-Uni0
allows a comprehensive analysis, including behavioral data. Thus, researchers in the field can
even understand how a specific sample can facilitate or complicate the classifier’s conclusions.
A solid interpretability criterion is the ability to translate data into fuzzy rules, which show
logical relationships between rule antecedents.</p>
        <p>By solving the same problem addressed in this paper, the FURIA model generated a graphical
(Fig. 10) and contextual analysis of the assessments of the position of basketball players. The
classifier was quite confusing as a global assessment because correctly classified instances
represent a 69.44%. As explained earlier, there may be confusion related to some classes.
Regarding the interpretability over some samples, the model concluded, for example, that an
analyzed sample was small forward because height is medium. However, this does not seem
right because the type should be a shooting guard instead of a small forward, according to
the information in the data set. Another action was classifying a sample as a center because
height is tall. Already the FHDT model (Fig. 11), on the other hand, presented the same global
description (except for the accuracy of 75.00%) and an evaluation of the interpretability of
samples where the model classified a player as being small forward. However, it highlighted
the medium chance probability that it is a point guard. Another evaluation concluded that the
player was a small forward because his height is medium, following rule 3 of the FHDT model.
However, this classification is also wrong because the type should be Shooting guard instead of
Small forward, according to the information in the data set.</p>
      </sec>
    </sec>
    <sec id="sec-6">
      <title>6. Conclusion</title>
      <p>This paper presented the functioning of an evolving fuzzy neural network acting in an
interpretable way in the identification of basketball players’ positions on the court. The model
obtained expressive results compared to state-of-the-art evolving fuzzy systems models and
presented a complete explanation of the model’s functioning and its results. The ENFS-Uni0
presented an interpretable evaluation that ranged from the evaluation of antecedents, feature
weights, and evaluation of rule consequents. Such an approach facilitates the understanding of
researchers on the topic addressed, providing insights and evaluations that other interpretable
fuzzy models were not able to do. The use of models with a high degree of interpretability helps
respond to the challenges of emerging research on how explainability can be handled in online
learning. In future work, this paper’s extension is left to measure better the impact of weights
on the similarity of antecedents regarding the weight of the respective neurons.</p>
    </sec>
    <sec id="sec-7">
      <title>Acknowledgments</title>
      <p>The author acknowledges the support by the Austrian Science Fund (FWF): contract number
P32272-N38, acronym IL-EFS.
[17] P. Angelov, X. Gu, D. Kangin, Empirical data analytics, International Journal of Intelligent</p>
      <p>Systems 32 (2017) 1261–1284.
[18] E. Lughofer, On-line incremental feature weighting in evolving fuzzy classifiers, Fuzzy</p>
      <p>Sets and Systems 163 (2011) 1–23.
[19] P. V. de Campos Souza, E. Lughofer, An advanced interpretable fuzzy neural network
model based on uni-nullneuron constructed from n-uninorms, Fuzzy Sets and Systems
426 (2022) 1–26. Fuzzy and Neurofuzzy Systems.
[20] P. Akella, Structure of n-uninorms, Fuzzy Sets and Systems 158 (2007) 1631–1651.
[21] K. Hirota, W. Pedrycz, Or/and neuron in modeling fuzzy set connectives, IEEE Transactions
on Fuzzy Systems 2 (1994) 151–161.
[22] R. R. Yager, A. Rybalov, Uninorm aggregation operators, Fuzzy sets and systems 80 (1996)
111–120.
[23] J. C. Fodor, R. R. Yager, A. Rybalov, Structure of uninorms, International Journal of</p>
      <p>Uncertainty, Fuzziness and Knowledge-Based Systems 5 (1997) 411–427.
[24] T. Calvo, B. De Baets, J. Fodor, The functional equations of frank and alsina for uninorms
and nullnorms, Fuzzy Sets and Systems 120 (2001) 385–394.
[25] G.-B. Huang, Q.-Y. Zhu, C.-K. Siew, Extreme learning machine: theory and applications,</p>
      <p>Neurocomputing 70 (2006) 489–501.
[26] A. Albert, Regression and the Moore-Penrose pseudoinverse, Elsevier, 1972.
[27] G.-B. Huang, L. Chen, C. K. Siew, et al., Universal approximation using incremental
constructive feedforward networks with random hidden nodes, IEEE Trans. Neural
Networks 17 (2006) 879–892.
[28] E. Lughofer, J.-L. Bouchot, A. Shaker, On-line elimination of local redundancies in evolving
fuzzy systems, Evolving Systems 2 (2011) 165–187.
[29] A. Bifet, G. Holmes, R. Kirkby, B. Pfahringer, MOA: Massive online analysis, Journal of</p>
      <p>Machine Learning Research 11 (2010) 1601–1604.
[30] J. M. Alonso, Explainable artificial intelligence for kids, in: Proceedings of the 11th
Conference of the European Society for Fuzzy Logic and Technology (EUSFLAT 2019),
Atlantis Press, 2019/08, pp. 134–141.
[31] J. M. Alonso, Teaching explainable artificial intelligence to high school students,
International Journal of Computational Intelligence Systems 13 (2020) 974–987.
[32] P. P. Angelov, X. Gu, J. C. Príncipe, Autonomous learning multimodel systems from data
streams, IEEE Transactions on Fuzzy Systems 26 (2018) 2213–2224.
[33] R. Pecori, P. Ducange, F. Marcelloni, Incremental learning of fuzzy decision trees for
streaming data classification, in: Proceedings of the 11th Conference of the European
Society for Fuzzy Logic and Technology (EUSFLAT 2019), Atlantis Press, 2019/08, pp.
748–755.
[34] J. Hühn, E. Hüllermeier, Furia: an algorithm for unordered fuzzy rule induction, Data</p>
      <p>Mining and Knowledge Discovery 19 (2009) 293–319.
[35] X. Gu, Q. Shen, A self-adaptive fuzzy learning system for streaming data prediction,</p>
      <p>Information Sciences 579 (2021) 623–647.
[36] J. Sampaio, S. J. Ibañez Godoy, M. Á. Gómez Ruano, A. Lorenzo Calvo, E. Ortega Toro, Game
location influences basketball players performance across playing positions., International
Journal of Sport Psychology 39 (2008) 43–50.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <surname>C.-T. Lin</surname>
            ,
            <given-names>C. G.</given-names>
          </string-name>
          <string-name>
            <surname>Lee</surname>
            ,
            <given-names>C.-T.</given-names>
          </string-name>
          <string-name>
            <surname>Lin</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          <string-name>
            <surname>Lin</surname>
          </string-name>
          ,
          <article-title>Neural fuzzy systems: a neuro-fuzzy synergism to intelligent systems</article-title>
          , volume
          <volume>205</volume>
          ,
          <string-name>
            <surname>Prentice hall PTR Upper Saddle River</surname>
            <given-names>NJ</given-names>
          </string-name>
          ,
          <year>1996</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>P.</given-names>
            <surname>V. de Campos Souza</surname>
          </string-name>
          ,
          <article-title>Fuzzy neural networks and neuro-fuzzy networks: A review the main techniques and applications used in the literature</article-title>
          ,
          <source>Applied Soft Computing</source>
          <volume>92</volume>
          (
          <year>2020</year>
          )
          <fpage>106275</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>W.</given-names>
            <surname>Pedrycz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Gomide</surname>
          </string-name>
          ,
          <article-title>Fuzzy systems engineering: toward human-centric computing</article-title>
          , John Wiley &amp; Sons,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>I.</given-names>
            <surname>Škrjanc</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. A.</given-names>
            <surname>Iglesias</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Sanchis</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            <surname>Leite</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            <surname>Lughofer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Gomide</surname>
          </string-name>
          ,
          <article-title>Evolving fuzzy and neuro-fuzzy approaches in clustering, regression, identification, and classification: A survey</article-title>
          ,
          <source>Information Sciences 490</source>
          (
          <year>2019</year>
          )
          <fpage>344</fpage>
          -
          <lpage>368</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>P.</given-names>
            <surname>V. de Campos Souza</surname>
          </string-name>
          , E. Lughofer,
          <article-title>Efnn-nulluni: An evolving fuzzy neural network based on null-uninorm, Fuzzy Sets and Systems (</article-title>
          <year>2022</year>
          ).
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>P.</given-names>
            <surname>V. de Campos Souza</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            <surname>Lughofer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. J.</given-names>
            <surname>Guimaraes</surname>
          </string-name>
          ,
          <article-title>An interpretable evolving fuzzy neural network based on self-organized direction-aware data partitioning and fuzzy logic neurons</article-title>
          ,
          <source>Applied Soft Computing</source>
          <volume>112</volume>
          (
          <year>2021</year>
          )
          <fpage>107829</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>P.</given-names>
            <surname>V. de Campos Souza</surname>
          </string-name>
          ,
          <string-name>
            <surname>E. Lughofer,</surname>
          </string-name>
          <article-title>An evolving neuro-fuzzy system based on uninullneurons with advanced interpretability capabilities</article-title>
          ,
          <source>Neurocomputing</source>
          <volume>451</volume>
          (
          <year>2021</year>
          )
          <fpage>231</fpage>
          -
          <lpage>251</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>P.</given-names>
            <surname>Souza</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Ponce</surname>
          </string-name>
          , E. Lughofer,
          <article-title>Evolving fuzzy neural hydrocarbon networks: A model based on organic compounds, Knowledge-Based Systems 203 (</article-title>
          <year>2020</year>
          )
          <fpage>106099</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>E.</given-names>
            <surname>Lughofer</surname>
          </string-name>
          ,
          <article-title>On-line assurance of interpretability criteria in evolving fuzzy systems - achievements, new concepts and open issues</article-title>
          ,
          <source>Information Sciences 251</source>
          (
          <year>2013</year>
          )
          <fpage>22</fpage>
          -
          <lpage>46</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>E.</given-names>
            <surname>Lughofer</surname>
          </string-name>
          ,
          <source>Evolving Fuzzy Systems - Methodologies, Advanced Concepts and Applications</source>
          , Springer, Berlin Heidelberg,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>L. A.</given-names>
            <surname>Zadeh</surname>
          </string-name>
          , Fuzzy sets,
          <source>Information and control 8</source>
          (
          <year>1965</year>
          )
          <fpage>338</fpage>
          -
          <lpage>353</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>E.</given-names>
            <surname>Lughofer</surname>
          </string-name>
          ,
          <article-title>Evolving multi-user fuzzy classifier systems integrating human uncertainty and expert knowledge</article-title>
          ,
          <source>Information Sciences 596</source>
          (
          <year>2022</year>
          )
          <fpage>30</fpage>
          -
          <lpage>52</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>D.</given-names>
            <surname>Gunning</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Stefik</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Choi</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Miller</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Stumpf</surname>
          </string-name>
          , G.-
          <string-name>
            <given-names>Z.</given-names>
            <surname>Yang</surname>
          </string-name>
          ,
          <article-title>Xai-explainable artificial intelligence</article-title>
          ,
          <source>Science Robotics</source>
          <volume>4</volume>
          (
          <year>2019</year>
          )
          <article-title>eaay7120</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <surname>C. Molnar,</surname>
          </string-name>
          <article-title>Interpretable machine learning</article-title>
          ,
          <source>Lulu. com</source>
          ,
          <year>2020</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <surname>S.-M. Zhou</surname>
            ,
            <given-names>J. Q.</given-names>
          </string-name>
          <string-name>
            <surname>Gan</surname>
          </string-name>
          ,
          <article-title>Low-level interpretability and high-level interpretability: a unified view of data-driven interpretable fuzzy system modelling</article-title>
          ,
          <source>Fuzzy Sets and Systems</source>
          <volume>159</volume>
          (
          <year>2008</year>
          )
          <fpage>3091</fpage>
          -
          <lpage>3131</lpage>
          . Theme: Modeling.
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>X.</given-names>
            <surname>Gu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. P.</given-names>
            <surname>Angelov</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. C.</given-names>
            <surname>Príncipe</surname>
          </string-name>
          ,
          <article-title>A method for autonomous data partitioning</article-title>
          ,
          <source>Information Sciences 460</source>
          (
          <year>2018</year>
          )
          <fpage>65</fpage>
          -
          <lpage>82</lpage>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>