Facial emotion recognition (FER) is a quite new and fast growing area of computer vision. Its main task is to identify what kind of emotion a person feels, using his/her facial expression. As an area of computer vision, the use of neural networks looks quite promising for this task. Because convolutional networks show, good results in other tasks. Many of the public networks are pretrained. Therefore, the question of using transfer learning for FER task arises. This will reduce the uncertainty of researchers when choosing a machine learning model and significantly speed up and increase the effectiveness of experiments in the field of FER. Therefore, the issue of transferring skills from other problem areas is rather little studied and promising.
The aim of the work is to research the possibility of neural networks and transfer learning technology applying to FER problems.
The goals of the work are to choose a dataset, based on it, plan and perform experiments, the results of which will allow:
• to formulate an effective algorithm for neural network identification and usage within the framework of the FER task; • to determine which architecture of neural networks is better to use as a backbone for FER tasks in different situations; • to compare the effectiveness of face recognition based backbones with standard solutions for transfer learning.
COLINS-2022: 6th International Conference on Computational Linguistics and Intelligent Systems, May 12-13, 2022, Gliwice, Poland EMAIL: kyrylo.smelyakov@nure.ua (K. Smelyakov); oleksandr.bohomolov@nure.ua (O. Bohomolov); maksym.kizitskyi@nure.ua (M. Kizitskyi); anastasiya.chupryna@nure.ua (A. Chupryna) ORCID: 0000-0001-9938-5489 (K. Smelyakov); 0000-0002-9539-8888 (O. Bohomolov); 0000-0001-9771-5771 (M. Kizitskyi); 0000-0003-0394-9900 (A. Chupryna)
Researches in recent years focus on facial emotion recognition (FER) task [1][2][3]. Such systems often supplement to face recognition systems (Azure Face API, Face, FaceReader, etc.) [4][5][6] and can be used in many situations, from customer satisfaction analysis, service at the checkout, to tracking emotions at a psychologist's appointment [7,8], in perspective drone vision services [9], etc.
The most efficient approaches that use such networks as ResNet, AffectNet, MobileNet, etc. on facial emotion recognition (FER) task are described by researchers. To simplify the access to this information they organized special list [10].
On the other hand, it includes various forms of ensembling and stacking of neural networks. It gives a win in the quality of the classification of emotions, but this approach also has disadvantages. Firstly, the model itself becomes quite huge and heavy, and a lot of time is spent on predictions. Because of this, the application of models of this kind is very complicated on mobile devices or in real-time systems. Secondly, due to the presence of several neural networks, the process of maintaining them within the production system becomes more complicated, and the task of updating models while maintaining the logic of work becomes more difficult compared to a solution in the form of an end-toend model. Therefore, the issue of developing a model, perhaps not as effective, but much more compact and easy to maintain for use in face recognition systems, remains relevant and open.
At the same time, a wide variety of machine learning models and algorithms, as well as a high degree of uncertainty in the application conditions, often create great difficulties in choosing an appropriate network architecture and tuning its parameters effectively [11][12][13].
Why are neural networks and transfer learning considered to solve FER problems?
In recent years, neural networks have become the standard tool in the area of computer vision [14,15]. A large number of diverse architectural solutions (EfficientNet, ResNet, Yolov5, etc.) and machine learning methods have been proposed to solve the problems of image classification object detection, and recognition. Their performance is affected by the quality of images [16,17], result of image segmentation [18,19], the architecture and hyperparameter settings of neural networks [20]. Moreover, researches on the application of convolutions are carried out to improve the effectiveness of CNN application in the case of the optimization of convolution mask parameters, the number of layers and a number of other parameters [21,22].
For the purposes of identifying the parameters of a neural network, a wide range of machine learning algorithms is currently used. One of the most effective is transfer learning [23]. Transfer learning (finetuning a neural network with pre-trained weights on a huge data set (for example ImageNet) to solve a specific problem) is widely used in all areas of computer vision and increases the quality of solving different kinds of problems [24,25].
The main advantage of this approach is that, thanks to the pre-trained weights, the model transforms the input image into a smaller set of meaningful features. Because of this, the relief of the loss function is smoothed out and the models converge faster to its minimum. Also recently, in such a field as face recognition, SOTA technique is often used, where the model is trained to compress the image into a feature vector by which a person's face can be identified [26]. Which in turn is very similar to what transfer learning is used for. That's why we decided to compare classical transfer learning models with face recognition models in more detail. Besides, this domain was selected because this is quite a popular area and many pre-trained models are in the public access [27].
For models to benefit from pre-trained weights, the task must be related to the domain on which the models were trained.
The research results are important not only for FER services, but also for solving a great number of related tasks, including the development of effective integrated E-learning services, AI solutions [28], the development of ICT solutions, network solutions and security services [12,29,30]. In addition, if face recognition based models show advantages over standard approaches, it means that the use of face recognition learning approaches can improve the quality of transfer learning models in other areas, increase learning speed and allow using less data for training. And it will allow specialists to conduct more experiments and reduce the outlay of cloud learning services.
First of all, consider the data that will be used in further experiments, some other materials and methods proposed to solve the problem under consideration.
In order to test our approach, we chose a quite well known data set FER2013 [31]. The 2013 Facial Expression Recognition dataset (FER2013) is a Kaggle dataset, introduced by Pierre-Luc Carrier and Aaron Courvill at the International Conference on Machine Learning (ICML) in 2013.
This dataset was chosen because it is in the public access. It also contains photographs of people of different age, gender, race, nationality, with different background and accessories (such as glasses, masks). It allows a better evaluation of the generalization abilities in emotions recognizing.
This dataset contains grayscale images of faces. Their size is 48x48 pixels. These images have been created using an automatic face registration so that the faces on them are centered and occupy nearly the same amount of space in each image. So when making a comparison, we assume that the images have already been preprocessed in advance, therefore we will not consider this issue within the framework of our paper. Each image is labeled with one of seven emotions from the following list: Angry, Disgust, Fear, Happy, Sad, Surprise, Neutral.
The Disgust expression has the minimal number of images -547, while other labels have nearly 5,000 samples each. More detailed information is presented in Table 1. Figure 1 shows examples of randomly selected pictures from the data set. As we can see, both men and women of different ages (from babies to old people), of different nationalities and races are represented in the data set. [31] In general, this data set provides a wide variety of face images, which will favorably affect the generalization ability of the model. However, it also has an imbalance of classes that is why the accuracy of determining the emotion of disgust will probably be lower in comparison with others.
To split the data set, a standard function from the sklearn package, train_test_split, was used. Training dataset -70% (25,121 images). Validation dataset -10% (3,589 image). Test dataset -20% (7,177 images). The partition was stratified by emotion in the image with random_state = 42.
We have chosen as key metrics:
• accuracy on the training set;
• loss on the training set;
• accuracy on the validation set;
• accuracy on the validation set;
• mean convergence rate (MCR)
where nnumber of epochs; 𝑀𝑒𝑡𝑟𝑖𝑐 𝑖performance metric on train data set during i`s epoch;
where nnumber of epochs; 𝑀𝑒𝑡𝑟𝑖𝑐_𝑡𝑟𝑎𝑖𝑛 𝑖performance metric on train data set during i`s epoch; 𝑀𝑒𝑡𝑟𝑖𝑐_𝑣𝑎𝑙 𝑖performance metric on validating data set during i`s epoch;
• initial accuracyaccuracy after training for 1 epoch. We chose this metric because it shows how well the pre-trained weights of the model fit the domain; • initial lossloss after training for 1 epoch. In our experiment Metric will be accuracy and loss (categorical cross entropy).
This section presents the plan of the experiment. In order to evaluate the effectiveness of transfer learning, we will compare several popular architectures such as VGG-Face (Figure 2), OpenFace (Figure 3) which are neural networks trained for face recognition. Our hypothesis is that since the task of face recognition is in some way similar to FER, therefore, the weights of the networks will already contain the necessary features that will increase the learning performance. We also chose ResNet-50, MobileNet (Figure 4) pretrained on ImageNet dataset because they are the standard choice as a backbone in transfer learning. In these networks, the last layer was excluded, and all layers except the last 4 were frozen. The model structure of VGG-Face and OpenFace were loaded using deepface library [35]. The pretrained weights are available on [36][37][38]. ResNet-50, MobileNet were loaded using keras framfork [39]. Each model will be trained with a fixed set of hyperparameters such as the learning rate (10-4), the number of epochs is 20. Also, key metrics will be measured every 5 epochs. As a loss function we chose categorical cross entropy.
To compare the efficiency of transfer learning, we will train neural networks in 2 versions: with pretrained weights and with randomly initialized weights. This approach will allow us to determine how and at what stages the pre-trained weights affect the efficiency of the model.
After the experiment we will find out in which model the pre-trained weights give the greatest value compared to random initialization, determine which model converges faster than others, is more resistant to overfitting and shows the highest accuracy.
Training will be carried out in the Google Colaboratory environment.
The results of the experiments are presented in Figures 5 -Figure 8 and in Tables 2 -Table 5. The high resolution versions of all images are presented here [40]. Table 2 shows MCR (1) for each model based on accuracy. Table 3 shows MCR for each model based on loss.
Table 4 shows MOFR (2) for each model based on accuracy. Table 5 shows MOFR for each model based on loss. It shows how much the difference between the metrics on the validation and training sets increases on average over the epoch, that is, how quickly the model over fits.
As a result of the experiment, it was revealed that pre-trained models showed better performance than randomly initialized ones in the FER task. Also, the pre-trained models had a higher average convergence rate at the first epochs (1-10), but then values became the same, in some cases, at epochs 15-20, the randomly initialized model converged faster. This is mainly due to the fact that the pretrained model at that moment reached an accuracy of more than 0.8 and, accordingly, the quality increase slowed down. On the other hand, pre-trained models are more prone to overfitting, therefore, when using them, it is desirable to apply various regularization methods or data augmentation.
The best model in terms of initial and final accuracy on the validation set is VGGFace_pretrained. Therefore, its weights are initially best suited for the FER task. But in our experiment, this model had the worst performance in terms of convergence rate. That is why, for its training, other hyperparameters should be used, for example, to increase the learning rate or add more dense classification layers.
The second model for face detection -OpenFace -shows a level of accuracy comparable to the standard solutions in transfer learning -ResNet-50. But at the same time it has fewer parameters, therefore it fits and predicts faster. OpenFace has 3,743,280 parameters and ResNet-50 has 23,587,712 parameters. MobileNet has the fewest parameters (3,228,864), but it`s performance is lower than in OpenFace. Also, OpenFace has the highest convergence rate and overfitting rate in comparison with other models.
Thus, the face recognition based models proved to be at a fairly high level, in some cases even surpassing standard models like ResNet-50 and MobileNet.
As can be seen from Figures 8-14, such emotions as happiness, anger, fear, surprise are best recognized, and disgust is worst of all recognized. This is because this class is the least represented in the dataset. In addition, some pictures are rather controversially labeled (for example, pictures 12-13).
In these examples neural networks show low confidence in the image class.
Based on the results of the experiment, the final learning algorithm was developed, which can be suggested to use in FER systems:
Preprocessing: 1) apply the face detection model to the image. You can use one of the pre-trained models, or train your own;
2) apply various augmentations to images. This will balance the classes (if the original dataset is unbalanced) and also increase the stability of the model on new data.
Training: 1) select a backbone model. If speed is more important within the task and there is enough data for training, we recommend choosing OpenFace. If the quality of recognition is more important and there are no enough resources for full model training, choose VGGFace;
2) freeze all layers of the neural network for training and add fully connected layers on top of them; 3) select hyper parameters and start the learning process with them.
As a result of the research the aim and goals of the work were reached. We formulated an effective algorithm for neural network identification and usage within the framework of the FER task; determined which architecture of neural networks was better to use as a backbone for FER tasks in different situations; compared the effectiveness of face recognition based backbones with standard solutions for transfer learning.
We found one of the most popular datasets on FER task -FER-2013. While analyzing its structure we found out that it was quite unbalanced. On the one hand it's a drawback, because models will learn how to distinguish minor class worse. But on the other hand it will show how models will work with real-world datasets that are often unbalanced.
Then we defined key metrics for analysis of networks performance during learning. Proposed metrics showed the efficiency of transfer learning for each architecture and determined what pre trained weights are most suitable for FER task and lead to faster convergence and less overfitting speed.
As part of this work, we organized an experiment and conducted a comparative analysis of the quality of the most popular neural network architectures for transfer learning (ResNet-50, MobileNet) with networks for face recognition (OpenFace, VGG-Face) within the FER task using various metrics. The obtained results show only general performance of the networks because they were all trained under the same conditions, and the best set of hyperparameters was not selected.
Based on the analysis of the experimental results, we recommend using the algorithm proposed in this article with a pretrained VGGFace. Also, under the condition of limited resources and the use of regularization methods, we recommend OpenFace as an alternative. But we also recommend specifically setting up the classifier for each specific task separately, because this will give a gain in quality.
For a deeper analysis of the effectiveness of neural networks, it is necessary to perform a deeper study, which is not the purpose of this work. It includes testing a larger class of architectures on a larger number of data sets, using various types of classifiers for embedding (including those not based on neural networks).