In this paper, we jointly use a Transductive Ensemble Learning with a Graph Convolutional Network to perform a task of Speech Emotion Recognition (SER). Additionally, we propose solving this task using ensemble learning methods with simple base classifiers such as Multilayer Perceptrons and k-Nearest Neighborhood. We extracted features using Wav2Vec and prosodic. The performance reaches 0.5248 in The following sections describe the techniques of Ensemble Learning and Graph Neural Network.
The speech can express more than words; it can express feelings. With this objective in mind,
we would like to identify sentiment, intention, opinion, genre, and humor using audio and
language features [
Discover the feeling present in the audio content is known as Speech Emotion Recognition (SER). Among the most known dataset for this task, the Portuguese are not among them. For this reason, the Internation Conference on the Computer Processing of Portuguese (PROPOR 2022) proposed a workshop to bring new researchers and enthusiasts to processing speech in Portuguese. One of the challenges of this conference is the SER for Portuguese.
Traditional methods of machine learning and deep learning techniques employ several
solutions for SER with strong performances [
All of them were inserted in our method Transductive Ensemble Learning with Graph Neural Network. After these techniques section, we introduced our proposal.
An ensemble combines multiple base classifiers to obtain final predictions. A simple ensemble
can be obtained performing majority voting or weighting the voting from base classifiers.
Another type of ensemble is the technique that includes label ranking in the process of voting
rules [
In general, the combination of the base classifiers is a crucial step and is a non-trivial process
in the presence of several base classifiers. It is common in the usage of ensemble methods to
perform neural machine translation (NMT) [
Neural Networks with support for graph representation are known as Graph Neural Network
(GNN). Originally proposed by [
Node classification is another way to work with graphs and neural networks. This idea
introduced by [
In this work, we propose Ensemble Learning with Transductive Learning for SER problem. Transductive learning [14] is similar to semi-supervised learning, but it considers a search space of hypothesis where all testing data is known beforehand. Our proposal of ensemble learning runs over a Graph Convolutional Network (GCN). We configure a graph with the training set that carries the relation between samples, labels, and models.
In a GCN, we build the graph that connects all data samples with all the labels for the respective models of classifiers. Figure 1 shows the representation of this bipartite graph. One set of bipartite nodes represents the samples. The _0 is a node representing the first example in the training dataset, and nodes below until _ are the − ℎ last one in the training set. These nodes receive a label from training. The _0 is a node for the first sample in the testing set. The following nodes till _ belong to the − ℎ example in the test dataset. We set nodes from the testing set with an invalid label not used in the training set.
The model nodes are another set of bipartite nodes. The nodes 10 , 11 e 12 , represent the model 1 for each valid label, and this setting repeat for the others model nodes 20 until the 2 , the − ℎ model. The edges between model and samples nodes are the probability of the output from models to respective samples.
The representation of the Graph Convolution Network is an adjacency matrix. Rows and
columns represent nodes of the graph. If a node has a connection, the value in the matrix cell
represents the probability that the node of instance belongs to the node class. The matrix of
adjacency is normalized and becomes a piece of the function of propagation of data [
The Figure 2 shows a diagram of a GCN with 4 layers, with respective sizes 512, 256, 128 and + 1 , where is the number of classes. Note that the adjacency matrix is shared with all layers. The network has + 1 outputs, are for the class of the problem, and one more for the nodes that do not have a class. The proposal avoids examples without defined classes using masks to not participate in the CNN backpropagation algorithm.
The following steps of the algorithm are the same as the CNN. The problem is a classification with the CNN, where it propagates labels from training set to unlabeled set. The training uses the function and applies it to the backpropagation to adjust the weights with the respective derivatives.
We used the dataset available by the competition for the SER task for the experiments. They are audios from the dataset called CORAA (Corpus of Annotated Audios). The distribution of examples and classes is shown in Table 1. We used two types of audio features as input to the base models. Both types were made available by the competition. The first type is the Wav2Vec, an unsupervised pre-training model for speech recognition [15]. The second is prosodic, which extracts features from the intonations and energy curves of audios [16]. The base models chosen for this task are: • Multilayer Perceptron (MLP): is an artificial neural network with few layers. It is a supervised learner, able to build a universal function to approximate [17, 18]. • k-Nearest Neighborhood (kNN): a classifier that performs the predictions based on voting across the metric of distance between train and test samples [19].
neutral non-neutral-female non-neutral-male Total
491 89 45 625 308 equal to 1000 steps, and layers with size 512, 256, 128 e 4 number of neurons from beginning to the end of GCN. A graphical representation is shown in Figure 2.
Method tSriazienoinfg set vSaizleidoaftion set vAaclciduaraticoynosnet oFn1-vscaolirdeamtioancrsoet Transductive Ensemble Learning with GCN 615 10 0.8000 0.2962 Transductive Ensemble Learning with GCN 500 125 0.7952 0.4532 Transductive Ensemble Learning with GCN 625 0 - Baseline Wav2Vec 625 0 - Baseline prosodic 625 0 - F1-score macro on real test set 0.5248 0.5356 0.5335
The Transductive Ensemble Learning with GCN is a competitive method for the task of SER. The results do not exceed baseline metrics of Wav2Vec or prosodic, but we do not use any new feature for the network. We plan to use features from other models with fine-tuning for this dataset for future works. Apply new features into nodes as a piece of the learning data to improve the results of the method. We can combine other models of neural networks with attention mechanism.
This work has been supported by the following Brazilian research agencies: Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) and Fundação de Apoio ao Desenvolvimento de Ensino, Ciência e Tecnologia do MS (FUNDECT).
URL: https://openreview.net/forum?id=rytstxWAW. [14] A. Gammerman, V. Vovk, V. Vapnik, Learning by transduction, in: Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, UAI’98, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1998, p. 148–155. [15] S. Schneider, A. Baevski, R. Collobert, M. Auli, wav2vec: Unsupervised pre-training for speech recognition, arXiv preprint arXiv:1904.05862 (2019). [16] I. Luengo, E. Navas, I. Hernáez, J. Sánchez, Automatic emotion recognition using prosodic parameters, in: Ninth European conference on speech communication and technology, Citeseer, 2005. [17] G. E. Hinton, Connectionist learning procedures, in: Machine learning, Elsevier, 1990, pp.
555–610. [18] X. Glorot, Y. Bengio, Understanding the dificulty of training deep feedforward neural networks, in: Proceedings of the thirteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, 2010, pp. 249–256. [19] E. Fix, J. L. Hodges, Discriminatory analysis. nonparametric discrimination: Consistency properties, International Statistical Review/Revue Internationale de Statistique 57 (1989) 238–247.