Facial expressions play a fundamental role in human communication, and their study, which represents a multidisciplinary subject, embraces a great variety of research fields, e.g., from psychology to computer science, among others. Concerning Deep Learning, the recognition of facial expressions is a task named Facial Expression Recognition (FER). With such an objective, the goal of a learning model is to classify human emotions starting from a facial image of a given subject. Typically, face images are acquired by cameras that have, by nature, diferent characteristics, such as the output resolution. Moreover, other circumstances might involve cameras placed far from the observed scene, thus obtaining faces with very low resolutions. Therefore, since the FER task might involve analyzing face images that can be acquired with heterogeneous sources, it is plausible to expect that resolution plays a vital role. In such a context, we propose a multi-resolution training approach to solve the FER task. We ground our intuition on the observation that, often, face images are acquired at diferent resolutions. Thus, directly considering such property while training a model can help achieve higher performance on recognizing facial expressions. To our aim, we use a ResNet-like architecture, equipped with Squeeze-and-Excitation blocks, trained on the Afect-in-the-Wild 2 dataset. Not being available a test set, we conduct tests and model selection by employing the validation set only on which we achieve more than 90% accuracy on classifying the seven expressions that the dataset comprises.
Facial expressions play a fundamental role in human communication. Indeed, they typically
reveal the actual emotional status of people beyond the spoken language. Moreover, the
comprehension of human afect based on visual patterns is a crucial ingredient for any
humanmachine interaction [
Nowadays, the most promising approaches to the FER task are based on the use of Deep Convolutional Neural Networks (DCNN). A typical approach consists of a pre-processing phase, where the images are subject to various transformations, and a training phase where these images are iteratively given as input to a DCNN model for feature extraction and expression classification.
In [
Usually, face images come from heterogeneous sources [12], e.g., cameras with diferent resolutions or diferent distances from the scene. Such characteristics directly impact DL models’ performance on tasks such as Face Recognition (FR) by dramatically lowering their performance [13]. Based on such an observation, we propose our approach grounded on the hypothesis that the images’ resolution has a non-negligible impact on DL models’ behavior when tested against the FER task. Precisely, we move our steps from [13] in which the authors explicitly take care of the multi-resolution nature of face images by designing a training technique to accommodate for such an issue adequately.
In our work, we take inspiration from the author’s training procedure, and we adapted it to our case. Specifically, we experimentally notice that we do not need any Teacher-supervised signal nor curriculum learning. Thus, we simplify the training procedure by only exploiting the double random extraction to set the final image resolution. To train the models and perform model selection, we employ the Af-Wild2 [ 14] dataset. We refer the reader to Section 4 for a brief description of the dataset.
Our base model is a ResNet-50 architecture [15], equipped with Squeeze-and-Excitation blocks [16], that has been pre-trained on the VGGFace2 dataset [17]. To train our models, we use the Adam [18] optimizer. We set the weight decay of 1.− 4 and the learning at 1.− 3 for the last fully connected layer and 1.− 4 for all the others. Moreover, we set the batch size to 128, and we use data augmentation techniques to avoid overfitting. Specifically, we first resize the images to have the shortest side of 256 pixels (while keeping the original aspect ratio), then we random crop a square of 224x224 pixels, and finally, we normalize the input channels. Moreover, we apply a random grayscale conversion with a probability of 0.2. We substitute the random crop with the center one, and we remove the grayscale operation to test the model on the validation set.
Concerning the random resolution extractions to train the models, we perform several experiments considering diferent ranges for the final image size concerning the multi-resolution training, with the minimum and maximum considered values being 8 and 256 pixels, respectively.
The Af-Wild2 [ 14] dataset is the first-ever database annotated for all three main behavior tasks: Valence Arousal (VA), Action Units (AU), and Expression (EX) classification. Concerning the last one, the dataset consists of 547 videos (collected from YouTube) that account for ∼ 2.6M of frames labeled considering seven basic expressions: neutral, anger, disgust, fear, happiness, sadness, and surprise. The annotation is made frame-by-frame by a team of seven experts. The dataset is shipped with a protocol that divides it into three non-overlapping subsets for training, validation, and test purposes. Specifically, the three partitions consist of 253, 71, and 223 videos, respectively. The cropped-aligned version of the dataset is made of images preprocessed to have a fixed resolution of 112x112 pixels. Among the ∼ 2.6M available images, ∼ 1.2M are available for training and validation on the FER task. We report in Figure 1 an example of training images in the Af-Wild2 [14] dataset.
Happiness
As we mentioned previously, the dataset comprises seven diferent types of expressions with a very diferent cardinality. In Table 1, we report the number of images for each class, both for the training and validation sets, while in Table 2, we report the classes’ weight concerning the training images only.
Training (%) Validation (%)
Neutral
As one can notice from Table 1, the classes are not balanced. For that reason, we leveraged a balanced cross-entropy loss to account for the class unbalance. To such an aim we use the weights reported in Table 2.
In this section, we report the experimental results we obtained on the Af-Wild2 [ 14] dataset. Since the dataset is currently employed in the Afect-in-the-Wild Challenge [ 19], the test set’s ground truth labels are not available. For such a reason, we quote the performance of our model on the validation set. Before the training, we took a small subsample of the validation set and used it for model selection purposes to avoid bias. Subsequently, we tested the best model on the entire validation set. To quote our results, we use diferent metrics. First, we evaluate the F1-score on each class, then we summarize the overall performance of our best model across all the seven expressions by quoting the F1-score (macro-average) and the overall accuracy. Finally, we evaluate the same score as required by the Afect-in-the-Wild Challenge [ 19], which is equal to: = 0.33 · accuracy + 0.67 · f1 score; (1) where the accuracy and the f1 score are relative to the whole dataset.
We report the results in Table 3 and Table 4 concerning single classes and the whole dataset, respectively.
F1 Score
Neutral 0.978
Anger 0.960
Disgust
0.965
From the previous tables, we can notice that our model shows promising performance on the FER task. Moreover, we acknowledge the stability of the scores among diferent classes even though the dataset is highly unbalanced as reported in Table 1
In this work, we report our first experimental campaign focused FER task. We tackle such a
problem by giving more representational power to our models, assuming a multi-resolution
context, and we observe promising results. As a next step, we will extend our experimental
campaign to test our approach on diferent publicly available datasets such as FER2013 [
We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan V GPU used for this research. This work was partially supported by WAC@Lucca funded by Fondazione Cassa di Risparmio di Lucca, AI4EU - an EC H2020 project (Contract n. 825619), and upon work from COST Action 16101 “Action MULTI-modal Imaging of FOREnsic SciEnce Evidence (MULTI-FORESEE)”, supported by COST (European Cooperation in Science and Technology). [9] M. Yu, H. Zheng, Z. Peng, J. Dong, H. Du, Facial expression recognition based on a multi-task global-local network, Pattern Recognition Letters 131 (2020) 166–171. [10] G. Zhao, X. Huang, M. Taini, S. Z. Li, M. PietikäInen, Facial expression recognition from near-infrared videos, Image and Vision Computing 29 (2011) 607–619. [11] P. Luan, V. Huynh, T. Tuan Anh, Facial expression recognition using residual masking network, in: IEEE 25th International Conference on Pattern Recognition, 2020, pp. 4513– 4519. [12] F. V. Massoli, F. Falchi, C. Gennaro, G. Amato, Cross-resolution deep features based image search, in: International Conference on Similarity Search and Applications, Springer, 2020, pp. 352–360. [13] F. V. Massoli, G. Amato, F. Falchi, Cross-resolution learning for face recognition, Image and Vision Computing 99 (2020) 103927. [14] D. Kollias, S. Zafeiriou, Af-wild2: Extending the af-wild database for afect recognition, arXiv:1811.07770 (2018). [15] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition. corr abs/1512.03385 (2015), 2015. [16] J. Hu, L. Shen, G. Sun, Squeeze-and-excitation networks, in: Proceedings of the IEEE
CVPR, 2018, pp. 7132–7141. [17] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, A. Zisserman, Vggface2: A dataset for recognising faces across pose and age. corr abs/1710.08092 (2017), arXiv:1710.08092 (2017). [18] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv:1412.6980 (2014). [19] D. Kollias, S. Zafeiriou, First Afect-in-the-Wild Challenge, https://ibug.doc.ic.ac.uk/ resources/first-afect-wild-challenge/, 2020. [20] S. Li, W. Deng, J. Du, Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild, in: Proceedings of the IEEE CVPR, 2017, pp. 2852–2861.