Facial expression recognition (FER) has been studied widely up to the present because FER is highly applicable. However, the COVID-19 pandemic forces people to wear face masks daily. A face mask removes information necessary for FER from a face and contains unnecessary information such as color and material. This paper aims to prevent the influence of meaningless regions for recognition, such as masks and backgrounds, due to the use of the attention mechanism. To obtain the optimal attention map, this paper has prepared two-stage networks. In stage 1, the network learns a simple task like masked/unmasked classification to generate an attention map. In stage 2, we train the network and use an attention map to guide the network to focus on the meaningful region to FER. As a result, the accuracy of our proposal achieved 86.8% and is 2.5% higher than VGG. The experimental results have proved the efectiveness of the proposed method.
Facial expression recognition (FER) is one of the major recognition themes because this is
expected to be widely used in many fields, such as healthcare and smartphone applications.
Various deep-learning models have been proposed to achieve these goals until now[
The 5th International Symposium on Advanced Technologies and Applications in the Internet of Things (ATAIT 2023) , ∗These authors contributed equally.
LGOBE http://http://www.ihpc.se.ritsumei.ac.jp/ (L. MENG) CEUR
To resolve this problem, this paper proposes the FER system based on CNN that introduces an attention mechanism divided into two stages. In stage 1, the network classifies whether the input image contains a mask and generates an attention map from this learning. This attention map has information about where to focus attention in the spatial direction of the input image or feature map. Moreover, this stage also aims to stabilize the generation of the attention map by learning a simple classification. In stage 2, the network performs classification by synthesizing the feature map and the attention map obtained in the previous stage. Due to synthesizing them, the region necessary for recognition is given attention, and the rest is ignored. In short, the major contribution of the paper has two aspects: • By generating and using the attention mechanism, our proposal prevents influence from extraneous information such as masks and backgrounds. Our proposal also has improved accuracy compared to other CNNs. • Our method has dynamically achieved stable attention map generation without human power like annotation and image processing such as landmark detection.
The remaining parts of this paper are organized as follows. Section 2 introduces the related works. In Section 3, we propose our FER system. Section 4 shows the experimental method and dataset. Section 5 reports experimental results. Finally, this paper is concluded in Section 6.
Deep learning is one of the popular techniques of machine learning[
FER is approximately divided into a static image-based FER network and dynamic-based FER
networks[
One of the major problems in facial expression recognition is the occlusion problem. This
problem is that the face is partially hidden by obstacles such as glasses or hands. Wearing a face
mask falls into the category of this issue. To solve this, various attention mechanisms have been
proposed. Attention mechanisms’ approaches to occlusion problems are broadly divided into
two categories, such as patch-base-CNN[
As shown in Figure 1, our FER system consists of two-stages such as Masked/Unmasked Classification and Facial Expression Classification.
In stage 1, the Masked/Unmasked Classification Network (MCN) is intended to dynamically generate the appropriate attention map from simple classification such as masked/unmasked binary classification. MCN, designed based on ABN as shown in Figure 2, incorporates the traditional recognition network structure and the Attention branch. The attention branch compresses the feature map into a 1 channel by inputting the 1 × 1 convolution layer and outputs it as attention maps. To ensure that the attention branch learns optimally, This branch also makes a class prediction and calculates the loss. Accordingly, the overall loss function of this network is determined by Eq1.
( ) = ( ) + ( ) (1) The output attention map is normalized from 0 to 1 by the sigmoid function before the final output. The attention map output from the attention branch has diferent features depending on the prediction class of the attention branch because of the diferent points of the GPA and 1×1 convolution layer response. If the network containing the Attention branch directly learns to classify facial expressions, the appearance of the output attention map varies in accordance with expression classes. To avoid this, MCN classifies whether the input image is masked or not. When masked facial images are inputs, return a common attention map regardless of facial expression. Moreover, expression recognition is a dificult task because it requires looking at every detail of the face. It makes the generation of attention maps tricky. Thus, this network replaces this with a simple classification like the masked/unmasked binary classification to stabilize. When learning of this classification is complete, unnecessary nodes to output the attention map are disconnected from the network which fixes the weight parameters of this network. Therefore, MCN becomes a generator that returns an appropriate attention map for the input images as shown in Figure 2 (b).
In Stage 2, Facial Expression Classification Network (FECN) uses the attention map to guide the network to pay more attention to the region that is essential to FER when this network classifies facial expressions. FECN combines feature maps generated from input images with attention maps by hadamard product and outputs the final probability of each class as shown in Figure 4. The attention map obtained from MFC is ( ), and a feature map is ( ). Weighted feature map is defined ′( ), i.e., Eq.2.
′( ) = ( ) ⊙ ( ) (2) Due to using this weighted feature map, this network is trained while ignoring the unnecessary regions, including face mask and background.
Large amounts of data are required to train deep learning adequately. Unfortunately, it is dificult to collect facial expression data, including masks. Moreover, these data are not widely available. Therefore, in this paper, we create a new dataset by pasting a mask image on the facial expression dataset called RAF-DB[20]. RAF-DB is a large-scale facial expression database with around 30K great-diverse facial images downloaded from the Internet and includes subjects of various ethnicities, ages, and genders. By 40 experts, these data were categorized into 7 classes such as angry, fearful, disgusted, happy, sad, surprised, and normal. We paste the face mask image into this dataset following the method[21]. It detects facial contour points (landmarks) from a face image and synthesizes a mask image according to the coordinates. In reality, people wear face masks of various textures and colors, so in this experiment, white non-woven cloth, black cloth, and gray cloth are equally included in the dataset as shown in Figure 5. We removed data that cannot be visually classified as facial expressions and failed to paste a face mask. This experiment has prepared two datasets as shown in Table1. The first dataset is the masked/unmasked dataset (MU-dataset), including unprocessed face images and mask-wearing images to train the masked/unmasked binary classifier. MU-dataset has about 500 images for training and 100 images for testing and includes multiple emotions. The second dataset is the Masked FER dataset (M-FER-dataset) containing expression classes. However, the mouth has meaningful information in helping to distinguish between the emotions of fear and surprise[22]. By wearing a face mask, it is dificult for both humans and computers to classify these expressions. Therefore, this FER focuses on four target classes such as angry, surprised, normal, and happy. M-FER-dataset has about 4000 images for training and 400 images for testing. These datasets are performed parallel shifts and random horizontal flips as data augmentation.
This paper compares the accuracy of our proposal with the VGGNet and simple ABN to elucidate our superiority. Table2 indicates the dataset used by each network as well as whether each network outputs the attention map. ABN and VGGNet are trained on M-FER-dataset to classify facial expressions. In our proposal, the MCN is firstly trained on MU-dataset to generate the attention map, and FECN is nextly trained on M-FER-dataset. This experiment used a GPU device called GeForce RTX 3080 and Pytorch as the Python framework.
This experiment prepares ABN which directly is trained in FER to compare with our proposal, which is divided into two-stage. When a masked face image is input to ABN, the output is the unique attention map that depends on the input facial expression, as shown in Figure 6 (c). In addition, dificult classifications such as FER make it hard to generate attention maps, and Figure 6 (c) shows that the angry class is one example. When these unstable attention maps are used as the attention mechanism, Table 3 indicates that recognition performance decreases on the contrary. To avoid this situation, we replaced the learning to generate attention maps with a simple classification. As a result, an attention map was generated focused on the area around the eyes for all classes, as shown in Figure 6 (a). These attention maps make FECN pay attention to the region that is meaningful to FER, such as the area around the eyes, and remove extraneous information, including background and mask patterns. Table 3 shows our proposal has achieved improved recognition accuracy on Masked FER.
FER requires real-time performance. The network must also be lightweight so that it runs on small devices. In this paper, our FER system is divided into two stages and trains diferent networks at each stage. Thus, the number of parameters for the entire network considerably has increased. Both MCN and FECN have their own feature extractors. Therefore, we aim to solve this problem by having these feature extractors share weights.
As COVID-19 is spreading worldwide, people are forced to wear masks when going out. Therefore, the demand for FER while wearing a face mask has increased. However, masks hide most of the face as well as contain extraneous information such as the pattern and color of the mask. To prevent these negative efects, our FER system has incorporated an Attention mechanism and consists of two stages such as classification and facial expression classification. By dividing our FER system into two stages, our proposal has succeeded in generating an attention map that is closer to the ideal for this task than other attention models. As a result, our proposal achieved 86.7% recognition accuracy. In the future, the challenge is to reduce the size of the network to decrease the number of parameters and amount of calculations. [14] Y. Li, J. Zeng, S. Shan, X. Chen, Occlusion aware facial expression recognition using cnn with attention mechanism, IEEE Transactions on Image Processing 28 (2019) 2439–2450. doi:1 0 . 1 1 0 9 / T I P . 2 0 1 8 . 2 8 8 6 7 6 7 . [15] S. Minaee, M. Minaei, A. Abdolrashidi, Deep-emotion: Facial expression recognition using attentional convolutional network, Sensors 21 (2021) 3046. [16] H. Fukui, T. Hirakawa, T. Yamashita, H. Fujiyoshi, Attention branch network: Learning of attention mechanism for visual explanation, 2019. a r X i v : 1 8 1 2 . 1 0 0 2 5 . [17] Y. Zhang, C. Wang, X. Ling, W. Deng, Learn from all: Erasing attention consistency for noisy label facial expression recognition, 2022. a r X i v : 2 2 0 7 . 1 0 2 9 9 . [18] P. D. M. Fernandez, F. A. G. Peña, T. I. Ren, A. Cunha, Feratt: Facial expression recognition with attention net, 2019. a r X i v : 1 9 0 2 . 0 3 2 8 4 . [19] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba, Learning deep features for discriminative localization, 2015. a r X i v : 1 5 1 2 . 0 4 1 5 0 . [20] S. Li, W. Deng, J. Du, Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 2017, pp. 2584–2593. [21] A. Anwar, A. Raychowdhury, Masked face recognition for secure authentication, 2020.
a r X i v : 2 0 0 8 . 1 1 1 0 4 . [22] M. W. Schurgin, J. Nelson, S. Iida, H. Ohira, J. Y. Chiao, S. L. Franconeri, Eye movements during emotion recognition in faces, Journal of Vision 14 (2014) 14–14. URL: https://doi.org/10.1167/14.13.14. doi:1 0 . 1 1 6 7 / 1 4 . 1 3 . 1 4 . a r X i v : h t t p s : / / a r v o j o u r n a l s . o r g / a r v o / c o n t e n t ¥ / / /933685/1534 − 7362 − 14 − 13 − 14. .