Gated Recurrent Units (GRU) are neural network gated architectures that simplify other ones (such as, LSTM) by joining gates mainly. For this, instead of using two gates, if is the first gate, standard operation 1 − is used to generate the second one, optimizing the number of parameters. In this work, we interpret this information as a fuzzy set, and we generalize the standard operation using fuzzy negations, and improving the accuracy obtained with the standard one.
and a value near 1 means that is going to update almost fully. Therefore, the 1 − operation can be understood as a negation or the complement of the fuzzy set in question. In this way, we generalize the expression 1 − of the GRU equations by using fuzzy negations [8, 9, 10], and generating the complementary fuzzy set from these negations. Experimentally diferent fuzzy negations are considered, where both fixed expressions and values that are learned by the Gated Recurrent Unit itself are used. We test our results with a text classification dataset, and we show that our approach using diferent expressions [ 8] improves the performance of the GRU. The structure of this work is as follows. In Section 2 the fuzzy and GRU preliminaries are reminded. In Section 3 the GRU architecture modification is explained. In Section 4 the experimental framework and results are presented. Finally, some conclusions and future research are described in Section 5.
In the present section we present the definitions and constructions of fuzzy negations and we also explain the main concepts about the GRU.
From now on, we denote by a non-empty and finite universe.
Definition 2.1. [7] A fuzzy set on is given by = {(, ()) | ∈ } where, by abuse
of notation denotes a map : → [
Definition 2.2. [11] A function : [
Definition 2.4. [11] A fuzzy negation is called strong if it is an involution, i.e., (N5) ( ()) =
for all ∈ [
Example 2.5. (i) The standard strong fuzzy negation is defined as () = 1 − known as the standard or Zadeh’s Negation. (ii) [11] Another examples of fuzzy negations are shown on Table 1 and represented on Figure 2.2.
Definition 2.6. [8] A function : [
Example 2.8. (i) If we use () = as automorphism (Theorem 2.7), we obtain Yager class
of negations (Table 1).
(ii) If we use () = 2 ( () = √) as automorphism (Theorem 2.7), we obtain concrete
examples of Yager class of negations (2)() = √1 − 2 ( ( 21 )() = (1 − √)2), which
is the same by evaluating Yager expression for = 2 and = 12 , respectively.
Definition 2.9. The complement of a fuzzy set on with respect to a fuzzy negation is the
fuzzy set : → [
In this subsection we explain the operation of the GRU [
The input values x()∈ R and h(− 1)∈ R enter to the update (Eq. 1) and reset (Eq. 2) gates.
In each of them, the value of x() is multiplied by each of the input weight matrices (W,
1−
×
h(t−1)
×
+
cell (h˜(t))
tanh
h(t)
x(t)
h(t−1)
update (z(t))
reset (r(t))
σ
σ
W). The same occurs with the values of h(− 1) and the recurrent weight matrices (Wℎ,
Wℎ). The -dimensional vectors obtained from these multiplications are fused summing with
the corresponding bias b, b. As activation function non-linear sigmoid logistic function is
1
used coordenate-wise ( : R → [
z() = (Wx() + Wℎh(− 1) + b)
r() = (Wx() + Wℎh(− 1) + b)
(update gate)
(reset gate)
For the calculation of the candidate activation (Eq. 3), input value x() is multiplied by Wℎ
matrix. Input value h(− 1) is weighted with r() by multiplying element-wise and the resultant
vector is multiplied by Wℎℎ matrix. As well as in the previous step, both -dimensional
structures are summed with bℎ. As activation function of the candidate activation the hyperbolic
tangent tanh : R → [
h˜() = tanh(Wℎx() + Wℎ(r() ∘ h(− 1)) + bℎ) (candidate activation) (3) The previous timestep unit vector (h(− 1)) and the candidate activation (h˜()) are combined in this step. The Hadamard or element-wise product (∘ ) is calculated between the values of the update gate (z()) and the complement of the update gate respect 1 (1 − z()) respectively (Eq. 4). Both values are added obtaining the current timestep value of the unit output vector, h(). The equations that describe the explained process are the following h() = (1 − z()) ∘ h(− 1) + z() ∘ h˜() (output)
Let be the hidden size of the GRU (Section 2). In the GRU learning process, z() vector represents the part of the current state is going to be retained and 1 − z() represents the part (1) (2) (4) of the previous time step memory is forgotten. In this work, we generalize the second one, because the operation does not have the necessity to be a -dimensional convex combination. Being z() = (1(), . . . , ()) the update vector of the GRU, and having the non-empty finite universe = (1, . . . , ) we can interpret z() as a fuzzy set on , where each vector element () is the membership of an element, hence, () = () for all ∈ {1, . . . , } having the following fuzzy set:
= {(, ()) | ∈ }
If the membership of a element ∈ to the fuzzy set is 0, this element is not updating
(h() = h(− 1)), whereas if the membership is 1, is going to have a full update (h() = h˜()).
Between 0 and 1 the membership and consequently the update measure is modelled by the
fuzzy set , that is, the element updates a part, weighted by his membership to the fuzzy set.
We can obtain the complementary set of with respect to a fuzzy negation : [
= {(, (())) | ∈ } = {(, ()) | ∈ } Here, we usually consider the standard negation to calculate the complement, although we also use diferent expressions (Table 1 and Figure 2.2). As in the case of the construction of , we can obtain a vector from as follows:
() = () for all ∈ {1, . . . , } obtaining z() = (1() , . . . ,
() ) This way, we modify the GRU Equation 4 replacing 1 − z() by the vector generated from the fuzzy complementarity with respect to fuzzy negations (Equation 5): h() = z() ∘ h(− 1) + z() ∘ h˜() (5)
In the present section, on one hand we explain the used framework (the dataset, the used neural network architecture, ) and on the other hand we present the obtained results.
4.1.1. Dataset As the Gated Recurrent Unit improves other recurrent models in reduced datasets we have selected a small one. The dataset we use is Text REtrieval Conference (TREC) [12], which is a dataset for question classification. It contains 5500 questions in the training test and another 500 in the test one. The dataset is distributed in 6 classes.
x(1) x(2) x( )
Embedding Embedding Embedding GRU
...
GRU GRU
...
GRU
Linear Linear Linear y(1) y(2) y( ) 4.1.2. Architecture The used architecture (Figure 4.1.2) is separated in four layers: • Embedding layer. It consists in an algorithm designed to reduce the input dimensionality into a fixed one (in this case, 50) encoding the input words by means of vectors. Words with close representations have a greater relation. • Double stacked GRU layers. Two Gated Recurrent Units with a hidden size of 64 each one. • Linear fully connected layer. The second GRU is fully connected like a multilayer perceptron with a 6-node layer, which is a 6-dimensional probability vector. We classify as member of the class the vector position value that corresponds with the maximum. 4.1.3. Training hyperparameters In this experiment, for each negation function, 10 independent runs of 30 epoch each are performed. The used optimization algorithm is Adam [13] and its fixed learning rate, = 1 × 10− 3. The selected loss function is the Cross Entropy Loss. 4.1.4. Metrics Once the architecture has been presented, we will go on to explain the metrics we will use to evaluate the experimental results. For each experiment , the metric to be used is the accuracy on test set (), calculated as follows: =
Total number of test dataset for 1 ≤ ≤ 10 (number of experiments). For the evaluation of the experiments for each negation function, mean (Eq. 7) and standard deviation (Eq. 8) of accuracies of 10 experiments are calculated as follows: 10 = 110 ∑︁
=1 ⎯ = ⎷⎸⎸ 19 ∑10︁( − )2 =1 (6) (7) (8)
The results are presented in Table 2. The table is divided in two parts, regarding to the used negations: in the first part negations with fixed values are used and in the second part values are learned by the GRU. For each fuzzy negation expression, 10 independent runs have been executed and after the mean accuracy (Eq. 7) and its standard deviation (Eq. 8) are measured (Table 2).
According to the first part of the table, we can see that the best accuracy value is obtained when the expression of the circular negation (2)() = √1 − 2 is used, gaining 1.68 points of average accuracy with respect to the standard negation. Also better results than are obtained with () = 1 − 2. Taking account the first part of the table, we can resume that the best results are obtained when we use a fuzzy negation fulfilling () > 1 − for all ∈ (0, 1) (Table 2.2).
Regarding to the second part of the table, we have used the Sugeno class and the Yager class fuzzy negation expressions, each one depending by the parameters ∈ (− 1, ∞) and ∈ (0, ∞) respectively. These parameters are learnt by the recurrent neural network. As we can see in Table 2, both learnt expressions improve the ones selected by a fixed number. Concretely, the diference between the means of accuracy of the standard negation and the best learnt expressions is of 2.93 percentage points. This diference reflects the improvement in the use of other expressions and specifically those learned by the neural network itself. Regarding the average values learned by the network, for the Yager expression we obtain = 1.417 for the ifrst GRU and = 1.508 for the second one. The standard deviation for each one is 0.016 and 0.028, respectively. For the Sugeno expression, we obtain = − 0.265 and = − 0.384, with the standard deviations 0.014 and 0.023, respectively. In both cases the standard deviations show that for the 10 independent runs, the obtained values have had very small diferences. These learned numbers also show that better results are obtained when we use a fuzzy negation such that () > 1 − for all ∈ (0, 1). Regarding to a overall conclusion about the properties, we can also remark that the best 3 results are obtained using strong negations (Definition 2.4).
In this work we have interpreted as a fuzzy set a part of a GRU architecture and we have proposed the use of diferent negations to perform it. We have observed that better results are obtained using fuzzy negations for which () > () for all ∈ (0, 1). Regarding future lines of research, in the theoretical aspect our intention is to continue investigating about new ways to generalize and interpret recurrent neural networks operators, such as using -dimensional fuzzy sets or extending the concept of fuzzy negation. On the applied side, future lines go on modifying other architectures, as well as using these architectures to other specific problems, such as language modelling.
Grant PID2019-108392GB-I00 funded by MCIN/AEI/10.13039/501100011033 and by Tracasa Instrumental and the Immigration Policy and Justice Department of the Government of Navarre. gio, Learning phrase representations using rnn encoder-decoder for statistical machine translation, 2014. arXiv:1406.1078. [7] L. Zadeh, Fuzzy sets, Information and Control 8 (1965) 338–353. doi:https://doi.org/ 10.1016/S0019-9958(65)90241-X. [8] H. Bustince, P. Burillo, F. Soria, Automorphisms, negations and implication operators, Fuzzy Sets and Systems 134 (2003) 209–229. doi:https://doi.org/10.1016/ S0165-0114(02)00214-2. [9] H. Zapata, H. Bustince, L. D. Miguel, C. Guerra, Some properties of implications via aggregation functions and overlap functions, International Journal of Computational Intelligence Systems 7 (2014) 993–1001. doi:https://doi.org/10.1080/18756891. 2014.967005. [10] E. Trillas, Sobre funciones de negación en la teoría de conjuntos difusos., Stochastica 3 (1979) 47–60. URL: http://eudml.org/doc/38807. [11] M. Baczyński, B. Jayaram, Fuzzy implications, in: Studies in Fuzziness and Soft Computing, 2008. [12] X. Li, D. Roth, Learning question classifiers, in: Proceedings of the 19th International Conference on Computational Linguistics - Volume 1, COLING ’02, Association for Computational Linguistics, USA, 2002, p. 1–7. URL: https://doi.org/10.3115/1072228.1072378. doi:10.3115/1072228.1072378. [13] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, CoRR abs/1412.6980 (2015).