Automatic emotion recognition plays a signi cant role in the process of human computer interaction and the design of Internet of Things (IOT) technologies. Yet, a common problem in emotion recognition systems lies in the scarcity of reliable labels. By modelling pairwise di erences between samples of interest, a Siamese network can help to mitigate this challenge since it requires fewer samples than traditional deep learning methods. In this paper, we propose a distance loss, which can be applied on the Siamese network ne-tuning, by optimizing the model based on the relevant distance between same and di erent class pairs. Our system uses samples from the source data to pre-train the weights of proposed Siamese neural network, which are ne-tuned based on the target data. We present an emotion recognition task that uses speech, since it is one of the most ubiquitous and frequently used biobehavioral signals. Our target data comes from the RAVDESS dataset, while the CREMA-D and eNTERFACE'05 are used as source data, respectively. Our results indicate that the proposed distance loss is able to greatly bene t the ne-tuning process of Siamese network. Also, the selection of source data has more e ect on the Siamese network performance compared to the number of frozen layers. These suggest the great potential of applying the Siamese network and modelling pairwise di erences in the eld of transfer learning for automatic emotion recognition.
FineAutomatic emotion recognition refers to identifying emotions using various
humanrelated signals such as facial expression, physiological signals, and speech [
A great number of machine learning approaches have been proposed to
address this challenge, and transfer learning was shown to be one of the most
promising directions. Transfer learning methods such as ne-tuning [
Modelling the pairwise di erences between samples of interest (e.g., through
Siamese networks [
The Siamese network is a type of neural network which takes a pair of data
samples as an input, and decides whether the corresponding samples belong
to the same or di erent classes. This network structure was rst introduced by
Bromley et al. for the task of signature veri cation by comparing whether the two
signatures are from the same person [
Researchers further optimized the initially proposed structure of the Siamese
network by considering the data distribution within the classes of interest [
The Siamese network has been applied to emotion-related tasks with a
promising performance [
The target data comes from the RAVDESS dataset [
Speech samples depicting four common emotions (anger, happiness, sadness,
and fear) across all datasets were selected and processed using openSMILE
toolkit [
In this section, we plan to discuss our three baseline methods: in-domain training, out of domain training, and Siamese network ne-tuning. Then we will introduce the proposed distance loss, and how this loss is applied for the Siamese network. All the methods are based on an optimized Siamese network structure, which contains 64, 32, and 16 nodes with ReLU activation for the rst three feature extractor layers, followed by a 16-node decision-based layer. During the training process, the cross entropy loss is calculated and used to update the weights of the model. After the Siamese network is trained, the test data will be compared with all the available labeled target data, and a nal classi cation decision will be generated based on the similarity of log-sum with each class. The unweighted average recall (UAR) will be used to evaluate the performance for each method. 4.1
Baseline In order to assess whether the proposed loss will bene t the knowledge transfer e ciency when limited data sample is available, we proposed three baseline methods. The rst baseline performs an out-of-domain training (OODT). A Siamese network is trained on all the source data, and tested on the target dataset without any adaptation. For this baseline, the data samples from the
Data sample 1
Input layer
Cross Entropy
Loss Distance
Loss
Hidden layer Data sample 2
Input layer
Hidden layer
Feature extractor out
Feature extractor out
Feature vector 1 Feature vector 2
L1-norm
Decision making layer
Same or different pair source are used to determine the nal classes, since no labeled target data is available.
The second baseline is an in-domain training (IDT), which is trained using su cient amount of target data. A leave-one-subject-out (LOSO) crossvalidation is performed. More speci cally, samples from a given speaker are used for testing, and all other data are used for training of the Siamese network. This process is repeated until all the speakers have been used for testing. This baseline serves as an upper limit of potential knowledge transfer.
The third baseline is the traditional ne-tuning method in the eld of transfer learning, to further assess the bene t provided by our proposed distance loss. The models trained in OODT are used for ne-tuning. One random data sample for each emotion from random 2 speakers are selected as labeled data. As a result, a total number of 8 samples (2 speakers 4 emotions 1 sample/emotion/speaker) are used for ne-tuning process and the remaining data are used as testing set. This process is repeated 10 times to increase the robustness of the results. A di erent number of frozen layers (none, rst layer, or rst two layers), as well as the di erent number of speakers in target data (2, 4, : : : , 18, 20) are tested to evaluate in detail the knowledge transfer e ciency. Inspired by the fact that the traditional ne-tuning method fails to e ciently leverage the knowledge when limited target data is available, we designed a distance loss to maximize the distance di erence between pairs from the same class and pairs from the di erent class. Let Xs be the set of s pairs with the same class in a batch, and Xd is the set of d pairs with the di erent class. Let gW(x) be the function parameterized by the weights W of the Siamese network that performs the transformation between the original input x and the extracted feature vector. The parameter W is learned by minimizing the average relative distance between pairs Xs of the same class and maximizing the distance between pairs Xd of di erent classes:
As indicated in the previous description, this loss is only applied to the feature extraction layers at the end of a batch in the Siamese network netuning process to minimize the possible in uence to the decision making process (Figure 1). Besides this loss, the other part for this method remains the same compared to the ne-tuning baseline method. 5
In an e ort to discuss the domain di erence between the di erent speech emotion datasets, we rst examine the unweighted average recall (UAR) for four emotion classi cation task on OODT baseline. This yields a 32.8% UAR on the RAVDESS dataset when using eNTERFACE'05 as source and 29.3% when using CREMA-D as source. We then examines the upper limit of classi cation performance using the proposed feature and model structure, and obtained a UAR at 50.0% with in-domain training.
The ne-tuning of the Siamese neural network without the proposed loss is used to illustrated the knowledge transfer e ciency. This approach resulted in a minor improvement when using eNTERFACE'05 as source data at 32.9%, and a relatively large improvement when using CREMA-D as source data at 37.8%. We nally added our proposed distance loss in the ne-tuning process, and obtained a signi cant improvement of 39.9% using eNTERFACE'05 and 43.3% using CREMA-D. A comparison of the performance of di erent methods can be found at Table 1. Our results also indicate that compared with di erent number of frozen layers, di erent source data may has a more important role for Siamese network ne-tuning (Figure 2). 6
Our results indicate that including 3-5 speakers in the Siamese NN ne-tuning with modi ed loss may result in the best trade-o between performance and the number of data samples used for training. This can be potentially explained by the fact that this number of speakers might not be enough to capture the distribution of the target dataset. Performance is degraded when including a smaller number of speakers in the target data. Also, if the data distribution includes a lot of variability, Siamese NN with modi ed loss will be greatly restricted, since pairwise di erences are less likely to express the class information.
There has not been a lot of research relevant to few-shot emotion
recognition with speech signals, where data from only a few number of speakers is
included in the target data. The most comparable results are from Gideon et
al., in which the proposed approach with progressive neural networks achieved a
limited improvement (around 3%) compared to traditional ne-tuning ([
We propose a distance loss which is based on the relative distance between same and di erent class pairs. Such loss can increase the upper limit and increase knowledge transfer e ciency when very limited target data is available. Our results also indicate that the selection of source data plays a more important role than the number of frozen layers. Findings of this work can be applied on other tasks using Siamese network, ne-tuning, or few-shot learning. The application of distance between di erent emotion classes can be a fundamental step in understanding the di erence between emotion categories.
As part of our future work, we plan to explore the usage of pairs in addressing the domain mismatch, and mitigate the in uence of di erent source data in the process of knowledge transfer. We will further perform the multi-source transfer learning by using pairwise information to select proper source data from each dataset. Finally, we plan to understand the distance between di erent emotions with the help of the pairing information.