In this study, we examine the relationship between surface electromyography (sEMG) and facial expressions using a novel Virtual Reality multi-sensor mask insert - emteqPROtm, equipped with seven sEMG sensors. We designed a dataset collection scenario to analyze the efects of expression intensity, expression duration, and head movements. Using data from 30 participants, we developed a machine learning pipeline that included preprocessing of the sensor data, de-noising, filtering, segmentation, feature engineering, and training a classification model. The experimental results indicate that the mask is suitable for recognizing five posed facial expressions (smile, frown, eyebrows raise, squeezed eyes, and neutral expression). The best-performing model achieved an F1-Macro score of 0.86. Head movement decreased the results to an F1-macro score of 0.82. The facial expressions that activate the same muscles were the most challenging to diferentiate. We also present results on the influence of diferent scaling and oversampling techniques. Finally, expression duration, intensity, and head movements influence the performance of the models for expression recognition and should be considered in the development of recognition algorithms.
build resilience and establish supportive environments
[
The book “The Expression of the Emotions in Man and Virtual reality (VR) has been a growing trend in the
Animals” by Charles Darwin reports on the first studies past decade. It enables the simulation of ecologically
valion human emotions [
Afective states can lead to diferent physiological and cent studies have considered EMG sensing for facial
exbehavioral responses [
© 2023 Copyright for this paper by its authors. Use permitted under Creative
CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmmUoRns LWiceonsrekAstthribouptionP4r.0oIncteerenadtiionnagl s(CC(CBYE4U.0)R.-WS.org)
identifying happy, angry, and neutral faces. They
concluded that the outputs from both systems were highly
correlated, showing that EMG-based model can identify
facial expressions produce results comparable to an
image processing-based model. Chen et al. [
Our study aims to explore the usage of a novel VR facial mask equipped with seven surface electromyography (sEMG) sensors to monitor facial muscle activity and classify five diferent facial expressions. Our approach is based on signal-processing and machine learning (ML) techniques to detect smiles, frowns, eyebrows raise, squeezed eyes, and neutral facial expressions with diferent intensities (high and low) and duration (short and long). We chose these five facial expressions because of their relation to specific afective states: smiles are related to positive afect and happiness; frowns are related to negative afect, depression, and anxiety; eyebrows raise is related to surprise, which can be positive and negative in terms of afective valence; and squeezed eyes is a facial expression generally related to negative afective states like fear and disgust.
The main diference was the inclusion of head movement in a specific direction (left, right, up, down) while doing the expressions. Also, as a diference from task A, the expressions in task B were only of high intensity and long duration. The data collection process was uninterrupted. The participants had a neutral expression on their faces between the posed expressions, making the neutral class the most common in the dataset, with 55.6%, while the rest of the classes comprised 11.1% each.
2. Data were continuously recorded at a fixed rate of 1000 Hz.
These data underwent a data preparation process,
includThe experiment was done on 30 participants aged 16 - 23 ing data filtering, segmentation, and feature engineering.
(20.8 ± 1.4), eighteen males and twelve females. All par- To increase the data quality, we performed signal
deticipants were healthy and had no family history of facial noising and filtering. The EMG signals were initially
neuromuscular and nervous disorders or heart problems. filtered with a Hampel filter to remove sudden peaks in
The data were recorded using the emteqPROtm mask the signals that appear because of rapid movements.
Ad[
The participants were asked to perform two tasks (Task coeficients, cepstral coeficients, frequency-based feaA and Task B) that included five posed expressions: smile, tures (e.g., main frequency), and statistical features (e.g., frown, eyebrows raise, squeezed eyes, and neutral expres- statistical moments). sion. Task A contains the five posed expressions with diferent durations (short and long) and intensities (low and high), with three repetitions of each expression. Task
diferent data resampling techniques to achieve a balanced class distribution. These were: (i) Random Undersampling – instances from the majority class are randomly chosen and removed from the training dataset; (ii) Synthetic Minority Oversampling Technique (SMOTE) – an oversampling technique that creates a synthetic example of the minority class based on the features of K-nearest neighbors; and (iii) One-Sided Selection Undersampling (OSS) – an undersampling technique that combines Tomek Links and the Condensed Nearest Neighbor (CNN) Rule to remove ambiguous points on the class boundary and to eliminate redundant examples from the majority class that are far from the decision boundary. For feature scaling, we implemented standardization and normalization. Both techniques were done participantwise, i.e., for each participant data separately.
The data resampling and feature scaling techniques
were combined, and such processed data were used as
input to several ML algorithms, including Decision Tree
Classifier [
The dataset was divided into three disjoint subsets: a validation set (five randomly selected participants), a test set (five randomly selected participants), and a training set consisting of the remaining 20 participants’ data. The validation set was used for tunning, and all the models were evaluated on the test set. As performance metrics, accuracy and F1-Macro scores were used.
diferent intensity.
The results obtained when data from Task A were used for training and testing are shown in Table 1. All the results presented in the table were achieved with the
efective one out of the ML algorithms used in our experiments (on the validation set). Depending on the scaling and the over/undersampling technique, the accuracy values range from 84.2% to 89.48%, while F1-Macro score values are between 0.75 and 0.86.
Regarding the feature scaling, both the standardization and the normalization improved the model’s performance compared to the default (no scaling and no additional data sampling). The improvement was greater in the case where feature standardization was used. We believe there are two reasons for the improvement: (i) the scaling was performed for each participant’s data separately, thus it acts as an unsupervised personalization technique reducing the inter-participants diferences; (ii) besides scaling the feature ranges (e.g., in the range -3 to 3), the standardization also shifts the data distribution for each participant separately. Whereas the normalization only scales the feature ranges (e.g., 0 to 1). Thus, the additional distribution shift that the standardization causes for each feature may be why standardization is better than normalization.
Regarding the data subsampling technique, both OSS undersampling and SMOTE oversampling performed better than the random undersampling. The SMOTE oversampling technique was the best performing one, achieving an accuracy of 86.34% and an F1-score of 0.8. By combining feature scaling (standardization) and SMOTE, we achieved the highest results (an F1-score of 0.84). Finally, Hidden Markov Method was applied in combination with Standardization and SMOTE, achieving an F1Macro score of 0.86 and an accuracy of 89.48%.
To further inspect the best-performing model, we present the confusion matrix in Figure 2. It indicates that the model struggles to correctly predict the smiling expressions. We speculate that this may be the case because smiling activates only the zygomatic face muscles.
A high intense smile is easily distinguishable from neutral expressions as the zygomatic muscle activity is high.
However, the low-intensity smile leads to low activation of the zygomatic muscles. A low-intensity smile resem
, bles a neutral expression, and it doesn’t afect the sEMG sensors enough to notice the diference between these two expressions.
B) Task B – Long-duration expressions with high intensity and head movements.
The results obtained when data from Task B were used for training and testing are shown in Table 2. All the results presented in the table were achieved with Extreme Gradient Boost (XGBoost) with gbtree, which proved to be the most efective one on the validation set.
The accuracy values range from 82.6% to 86.4%, and the F1-Macro scores are between 0.76 and 0.82. Feature scaling and resampling are not as critical preprocessing steps as in Task A. The method with the highest accuracy is the one where only participant-wise standardization was performed. It achieved 86.46% accuracy and an F1Macro score of 0.82. However, the method that combines standardization and random undersampling has the highest F1-Macro score of 0.82 and 85.88% accuracy. Although this method’s accuracy is lower, we consider it the best performing one since the F1-Macro score is more suitable for evaluations on an unbalanced dataset.
Figure 3 presents the confusion matrix for the bestperforming model. We can see from the confusion matrix that the model can diferentiate between all the classes with the neutral class, which was not the case in task A.
This is because, in Task B, only high-intensity expressions with a long duration were examined. In this case, the muscles were highly activated, making the expressions more distinguishable from the neutral expression.
The problem with this model is that it struggles to detect frowns and eyes squeezed. Both expressions activate the same facial muscles: mainly the frontalis and corrugator muscles are activated, and this leads to wrongly predicting these expressions.
Overall, it seems that the head movements had a minor influence, i.e., the best- performing model achieved an F1score of 0.82, which is on par with the best-performing model from Task A (Table 1), where the method without HMM achieved and F1-score of 0.84. We excluded the HMM-based method from this analysis and the following one because it adds a layer of complexity to the training process.
C) Task A and Task B combined – Long and shortduration expressions with head movements.
Table 3 shows the performance of the methods for
six train-test combinations: (i) training on Task A data
and testing on Task A data; (ii) training on Task B data
and testing on Task A data; (iii) training on both Task A pants, the best-performing model achieved an accuracy
and Task B data and testing on Task A data; (iv) training of 89.48% and an F1-Macro score of 0.86. The approach
on Task B data and testing on Task B data; (v) training is based on Random Forest in combination with
stanon Task A data and testing on Task B data; (vi) training dardization and oversampling (SMOTE) steps, and the
on both Task A and Task B data and testing on Task B Hidden Markov Method as a prediction-smoothing
techdata. With this, we want to examine whether mixing no- nique [
From Table 3, we can see that for Task A, the best frown and squeezed eyes. Both expressions activate foreresults are achieved when only no-movement data is used head muscles closely placed to each other (corrugator and in the training set (i.e., Task A is used), and the inclusion frontalis muscles). In the future, we plan to investigate of movement data (Task B data) reduces the method’s feature selection, model personalization, and end-to-end accuracy by 3 percentage points and the F1-score drops deep learning to overcome this weakness. by 0.06. On the other hand, when Task B data are used for It should also be noted that in some cases, the difertesting, the inclusion of no-movement data (Task A data) ences in the results may have been due to the diferent in the training set has a lower influence on the results random steps in the processing steps and in the learning for Task B, as the F1-score drops by 0.02. By mixing the ensembles that have built-in random steps. Nonetheless, training sets, we did not observe any improvements in this does not diminish the main findings of the study:(i) the results for both Task A and Task B data. These results The novel VR-mask equipped with sEMG sensors in comindicate that scenario specificity is important for the bination with ML is suitable for recognizing facial expresmodel’s accuracy, i.e., if we expect movement during the sions (smile, frown, eyebrows raise, squeezed eyes, and usage of the models, then it is better to include training neutral); (ii) facial expressions that activate the same musdata that involves movement. cles are the most challenging to diferentiate; (iii) feature scaling is an important step which enables for minimizing inter-participant feature diferences; (iv) the results 5. Conclusion regarding data oversampling or undersampling were inconclusive, as it improved the results in some cases (Table This study examined the relationship between sEMG 1), but not in other cases (and Table 2); Finally, (v) expressensor data from facial muscles and posed facial expres- sion duration, intensity, and head movements influence sions using the novel emteqPROtm VR multi-sensor facial the performance of the models for expression recognimask. We analyzed sEMG data from 30 participants that tion and should be taken into account in the development performed five facial expressions while wearing the de- of facial expression recognition algorithms. These convice. The data collection scenario was specifically de- clusions contribute to afect sensing in VR, which has signed to inspect several aspects of facial expression potential in symptom monitoring during VR-delivered recognition - duration (short vs. long), intensity (low vs. therapy for mental health disorders. high), and head movements. The collected data was then used to develop models that recognize smiles, frowns, eyebrows raise, squeezed eyes, and neutral facial expres- Acknowledgments sions. We explicitly inspected the influence of normalization techniques and data oversampling and undersam- This study was partially supported by the WideHealth pling techniques. On the test data of five unseen partici- project (European Horizon 2020) under grant agreement