In recent years, extensive research has emerged in a ective computing on topics like automatic emotion recognition and determining the signals that characterize individual emotions. Much less studied, however, is expressiveness|the extent to which someone shows any feeling or emotion. Expressiveness is related to personality and mental health and plays a crucial role in social interaction. As such, the ability to automatically detect or predict expressiveness can facilitate signi cant advancements in areas ranging from psychiatric care to arti cial social intelligence. Motivated by these potential applications, we present an extension of the BP4D+ dataset [27] with human ratings of expressiveness and develop methods for (1) automatically predicting expressiveness from visual data and (2) de ning relationships between interpretable visual signals and expressiveness. In addition, we study the emotional context in which expressiveness occurs and hypothesize that di erent sets of signals are indicative of expressiveness in di erent contexts (e.g., in response to surprise or in response to pain). Analysis of our statistical models con rms our hypothesis. Consequently, by looking at expressiveness separately in distinct emotional contexts, our predictive models show signi cant improvements over baselines and achieve comparable results to human performance in terms of correlation with the ground truth.
Although humans constantly experience internal reactions to the stimuli around
them, they do not always externally display or communicate those reactions. We
refer to the degree to which a person does show his or her thoughts, feelings,
or responses at a given point in time as expressiveness. That is, a person
being highly expressive at a given moment can be said to be passionate or even
dramatic, whereas a person being low in expressiveness can be said to be stoic
or impassive. In addition to varying moment-to-moment, a person's tendency
toward high or low expressiveness in general can also be considered a trait or
disposition [
In this paper, we study momentary expressiveness, or expressiveness at a
given moment in time. This quantity has not been previously explored in detail.
We have two primary goals: (1) to automatically predict momentary
expressiveness from visual data and (2) to learn and understand interpretable signals of
expressiveness and how they vary in di erent emotional contexts. In the following
subsections, we motivate the need for research on these two topics.
Prediction of Expressiveness The ability to automatically sense and
predict a person's expressiveness is important for applications in arti cial social
intelligence and especially healthcare. For an example of how expressiveness
might be useful in arti cal social intelligence, as many customer-facing areas
become increasingly automated, the computers, robots, and virtual agents who
now interact with humans must be aware of expressiveness in order to
interact with humans in appropriate ways (e.g., a highly expressive display might
need to be a orded more attention than a less expressive one). With regard to
healthcare, expressiveness holds promise as an indicator of mental health
conditions like depression, mania, and schizophrenia, which have all been linked to
distinct changes in expressiveness. Depression is associated with reduced
expressiveness of positive emotions and increased expressiveness of certain negative
emotions [
We aim to formalize the relationship between interpretable visual signals and expressiveness through statistical analysis. Furthermore, we hypothesize that the speci c signals that contribute to expressiveness vary somewhat under di erent contexts and seek to con rm this hypothesis by modeling expressiveness in different emotional states. Contributions To realize our goals, we must collect data about how expressiveness is perceived in spontaneous (i.e., not acted) behavior and develop techniques to analyze, model, and predict it. As such, we address the gap in the literature through the following contributions.
We introduce an extension of the BP4D+ emotion elicitation dataset [
We present statistical and deep learning models that are able to predict expressiveness from visual data. We perform experiments on a test set of the BP4D+ extended dataset, establish baselines, and show that our models are able to signi cantly outperform those baselines and for some metrics even approach human performance, particularly when taking context into consideration.
We present context-speci c and context-agnostic statistical models that reveal interpretable relationships between visual signals and expressiveness. We conduct an analysis of these relationships over three emotional contexts| startle, pain, and disgust|that supports our hypothesis that the set of visual signals that are important to expressiveness varies depending on the emotional context. 2
Although little prior work has been conducted on direct prediction of expressiveness, advances have been made in the adjacent eld of emotion recognition. Likewise, within the scope of psychology, there exists a substantial body of literature dedicated to determining the visual features that characterize di erent emotions; however, to our knowledge, little to no similar work has been conducted on the visual features that characterize how strongly those emotions are shown (i.e., expressiveness). We describe the current state of these areas of research, as we draw from this related work to de ne our own approaches to predicting and characterizing expressiveness.
Emotion Recognition Because the task derives from similar visual features|
facial landmarks and movement, for example|advancements in deep learning
for the eld of emotion recognition are highly informative and provide much of
the guiding direction for our predictive deep learning models. A number of
architectures have achieved high accuracy for multiclass emotion classi cation in
a variety of settings, including still images, videos, and small datasets. [
Interpretable Signals of Emotion The three emotional contexts of startle,
pain, and disgust all have well-studied behavioral responses that could serve as
visual signals of emotion and therefore expressiveness. Previous observational
research has found that the human startle response is characterized by blinking,
hunching the shoulders, pushing the head forward, grimacing, baring the teeth,
raising the arms, tightening the abdomen, and bending the knees [
We describe the data collection pipeline and engineering process for the dataset
we used to perform our modeling and analysis of expressiveness.
Video Data The BP4D+ dataset contains video and metadata of 140
participants performing ten tasks meant to elicit ten di erent emotional states [
In this study, we focus on the tasks meant to elicit startle, pain, and disgust. Example frames from each of these tasks can be found in Figure 1. These tasks were selected because they did not involve the participant talking; we wanted to avoid tasks involving talking because the audio recordings are not available as part of the released dataset. In the startle task, participants unexpectedly heard a loud noise behind them; in the pain task, participants submerged their hands in ice water for as long as possible; and in the disgust task, participants smelled an unpleasant odor similar to rotten eggs. Startle
Pain
Disgust High expressiveness
Low to moderate
expressiveness
Because a person's expressiveness may change moment-to-moment and we wanted to have a ne-grained analysis, we segmented each task video into multiple 3-second clips. Because task duration varied between tasks and participants, and we did not want examples with longer durations to dominate those with shorter durations, we decided to focus on a standardized subset of video clips from each task. For the startle task, we focused on the ve clips ranging from second 3 to second 18 as this range would capture time before, during, and after the loud noise. For the pain task, we focused on the rst three clips when pain was relatively low and the nal four clips when pain was relatively high. Finally, for the disgust task, we focused on the four clips ranging from second 3 to second 15 as this range would capture time before, during, and after the unpleasant odor was introduced. In a few cases, missing or dropped video frames were replaced with empty black images to ensure a consistent length of 3 seconds per clip. Human Annotation We de ned expressiveness as the degree to which others would perceive a person to be feeling and expressing emotion. Thus, we needed to have human annotators watch each video clip and judge how expressive the person in it appeared to be. To accomplish this goal, we recruited six crowdworkers from Amazon's Mechanical Turk platform to watch and rate each video clip. We required that raters be based in the United States and have approval ratings of 99% or greater on all previous tasks. Raters were compensated at a rate approximately equal to $7:25 per hour.
Because raters may have di erent understandings of the word \expressiveness," we did not want to simply ask them to rate how expressive each clip was. Instead, we generated three questions intended to directly capture important aspects of expressiveness. Speci cally, we asked: (1) How strong is the emotional response of the person in this video clip to [the stimulus] compared to how strongly a typical person would respond? (2) How much of any emotion does the person show in this video clip? (3) How much does the person move any part of their body/head/face in this video clip? Each question was answered using a ve-point ordered scale from 0 to 4 (see the appendix for details).
To assess the inter-rater reliability of the ratings (i.e., their consistency across
raters), we calculated intraclass correlation coe cients (ICC) for each question
in each task and across all tasks. Because each video clip was rated by a
potentially di erent group of raters, and we ultimately analyzed the average of
all raters' responses (as described in the next subsection), the appropriate ICC
formulation is the one-way average score model [
x1 ε1 0 λ1 η λ2 x2 ε2 λ3 1 x3 ε3
In Figure 2, the observed question variables are depicted as squares (x) and
the aforementioned latent variable is depicted as a circle ( ) with zero mean and
unit variance. The factor loadings ( ) represent how much each question variable
was composed of shared variance, and the residuals (") represent how much each
question variable was composed of non-shared variance (including measurement
error). We t this same CFA model for each task separately and across all tasks
using the lavaan package [
The resulting estimates are provided in Table 2. Three patterns in the results are notable. First, all the standardized loadings were higher than 0.85 (and most were higher than 0.95), which suggests that there is a great deal of shared variance between these questions and they are all measuring the same thing (e.g., expressiveness). Second, there were some factor loading di erences within tasks, which suggests that there is value in aggregating the question responses using CFA rather than averaging them. Third, there were some factor loading di erences between tasks, especially for the motion question, which suggests that the relationship between motion and expressiveness depends upon context.
Finally, we estimated each video clip's standing on the latent variable (i.e., as
a continuous, real-valued number) by extracting factor score estimates from the
CFA model; this was done using the Bartlett method, which produces unbiased
estimates of the true factor scores [
We selected our models with our two primary goals in mind: we wanted to nd a model that would perform well in predicting expressiveness, and we wanted at least one interpretable model so that we could understand the relationships between the behavioral signals and the expressiveness scores. We experimented with three primary architectures|ElasticNet, LSTM, and 3D-CNN|and describe our approaches in greater detail below.
ElasticNet We chose ElasticNet [
We engineered visual features from the raw video data to use as input for
our ElasticNet model. For each clip, we used the OpenFace [
From this data, we computed frame-to-frame displacement (i.e., distance travelled) and velocity (i.e., the derivative of displacement) for each facial landmark. We also calculated frame-to-frame changes in gaze angle and head position with regard to translation and scale (\head"); pitch; yaw; and roll. For each clip, we used the averages over all frames of these quantities as our features. We also counted the total number of action units and calculated the mean intensity of action units occurring in the clip. We selected these features to represent both amount and speed of facial, head, and gaze movement.
We used an out-of-the-box implementation of ElasticNet from sklearn and
tuned the hyperparameters by searching over 2 f0:01; 0:05; 0:1; 0:5; 1:0g for
the penalty term and over 2 f0:0; 0:1; : : : ; 1:0g for the L1 prior ratio. For the
nal models on the startle task, pain task, disgust task, and all tasks, was
0:01, 0:1, 0:1, and 0:05, respectively; was 0:0, 0:0, 0:7, and 0:7, respectively.
When = 0:0, ElasticNet corresponds to Ridge regression, and when = 1:0,
ElasticNet corresponds to Lasso regression [
OpenFace-LSTM We also explored several deep learning approaches to
determine whether we could achieve better predictive performance by sacri cing some
interpretability. Due to the small size of the training dataset and the need to
capture the temporal component of the data, we proposed the use of a relatively
simple deep architecture suitable for modeling sequences of data, LSTM [
In this section, we describe the evaluation metrics, data partitions, and baselines that we used to evaluate the performance of our models and to conduct our analysis of the interpretable visual features relevant to expressiveness. Code for our evaluation and analyses is available at https://osf.io/bp7df/?view_ only=70e91114627742d7888fbdd36a314ee9.
Evaluation Metrics and Dataset We selected RMSE and correlation of
model predictions with the ground truth expressiveness scores as the evaluation
metrics for our model performance. For ease of interpretability and comparison,
we report normalized RMSE [
To determine whether di erences in performance between models and
baselines were statistically signi cant, we used the cluster bootstrap [
Because we suspected that expressiveness might manifest di erently in di erent emotions, we wanted to see whether training separate models for each emotion elicitation task would produce better predictive performance than training a single model over all tasks. Furthermore, tting separate ElasticNet models for each task would allow us to understand whether the feature set relevant to expressiveness is di erent depending on the emotional context, which would test our hypothesis. Therefore, we separated the BP4D+ dataset by task and created 60=20=20 train/validation/test splits for each of these task-speci c datasets and a separate split in the same proportions over the entire dataset. This partitioning was done such that no subject appeared in multiple splits. For each model, we report results from training and evaluating on each task-speci c dataset and on the entire dataset.
Baselines We de ned several baselines against which to compare our models' performance:
Uniform baseline: This baseline samples randomly from a uniform distribution over the theoretical range of the expressiveness scores (i.e., 3:5 to 3:5).
Normal baseline: This baseline samples randomly from a standard normal distribution with mean and variance equal to the theoretical mean and variance of the expressiveness scores (i.e., mean 0 and variance 1). Human baseline: This baseline represents the performance of a single randomly selected human crowdworker. We calculated an estimated factor score for each rater by weighting their answers to each question by that question's factor loading and summing the weighted values. These weighted sums were then standardized and compared to the average of the remaining 5 raters' 1 Software to conduct this procedure is available at https://github.com/jmgirard/ mlboot. estimated factor scores to assess each rater's solitary performance. Finally, these performance scores were averaged over all crowdworkers to capture the performance of a randomly selected crowdworker. 6
In the following subsections, we present the results of our experiments, rst comparing our model approaches and baselines and then visualizing and interpreting the feature weights of the ElasticNet model.
Despite achieving NRMSEs well below those of the normal baseline (Figure 3, Table 3), the proposed deep learning had relatively poor performance in most tasks according to the correlation metric. For example, OpenFace-LSTM attained a reasonable correlation compared to the human baseline on the startle and disgust tasks but produced essentially no correlation with the ground truth on the pain task. Likewise, pretrained 3D-CNN and 3D-CNN trained from scratch yielded little and no correlation, respectively, of their predictions with the ground truth. We suspect that such results may be the product of the small dataset on which the models were trained, as the data quantity may be insufcient to allow the models to generalize and learn the appropriate predictive signals from complex data without human intervention.
As such, of the proposed models, we consider ElasticNet to demonstrate the best performance overall. Its NRMSEs were consistently lower than those of the other proposed models, and its correlations were much higher than those of any other proposed model and come close to (and in the case of the disgust task, slightly exceed) those of the human baseline. Statistical analyses of the di erences in performance between ElasticNet and all other models and baselines, the results of which are shown in Table 4, support our intuition. Speci cally, when trained across all tasks, ElasticNet attains signi cantly lower NRMSE and signi cantly higher correlation of its predictions with the ground truth compared to all other models and baselines except the human baseline. However, the same comparison also shows that ElasticNet has signi cantly higher NRMSE and signi cantly lower correlation of its predictions with the ground truth compared to the human baseline, indicating that there is still room for improvement. Understanding Signals of Expressiveness Because our best-performing model, ElasticNet, is an interpretable linear model, we were able to determine the relationship between the visual features in our dataset and overall expressiveness by examining the feature weights of the model trained over all tasks. Furthermore, by doing the same for the feature weights of models trained over individual tasks, we were able to explore the hypothesis that the set of signals indicative of expressiveness varies from context to context. These visualizations are shown in Figure 4. We directly interpret those features with a standardized weight close to or greater than 0:2 in absolute value.
From the weights of the model trained over all tasks, we can see that three primary features contribute to predicting overall expressiveness: action unit count, action unit intensity, and point displacement (i.e., the distance traveled by all fa
Action Unit Count Action Unit Intensity Points Displacement
Points Velocity Head Displacement ●
Head Velocity Pitch Displacement
Pitch Velocity Yaw Displacement Roll Displacement
Yaw Velocity Roll Velocity
Startle Task
● ● ● ● ● ● ● ● ●
● ● ●
Pain Task ● ● ● ● ● ● ● ● ● ● ● ● ● ● Disgust Task
● ● ● ● ● ● ● ● All Tasks
● ● ● cial landmark points). This suggests that there are some behavioral signals that index expressiveness across emotional contexts, and these are generally related to the amount and intensity of facial motion. Notably, features related to head motion and the velocity of motion did not have high feature weights for overall expressiveness.
We also observe that each individual task had its own unique set of features that were important to predicting expressiveness within that context. These features make intuitive sense when considering the nature of the tasks and are consistent with the psychological literature we reviewed.
In the startle task, higher expressiveness was associated with more points displacement, higher points velocity, less head displacement, and higher action unit count. These features are consistent with components of the hypothesized startle response, including blinking, hunching the shoulders, grimacing, and baring the teeth. The negative weight for head displacement was somewhat surprising, but we think this observation may be related to subjects freezing in response to being startled.
In the pain task, higher expressiveness was associated with higher action unit count, higher action unit intensity, and less points velocity. These features are consistent with components of the hypothesized pain response, including grimacing, frowning, wincing, and eye closure. Although the existing literature hypothesizes that body motion increases in response to pain, we found that points velocity has a negative weight. However, we think this nding may be related to increased muscle tension and/or the nature of this speci c pain elicitation task (e.g., decreased velocity may be related to the regulation of pain in particular).
Finally, in the disgust task, higher expressiveness was associated with higher action unit count, higher action unit intensity, higher points displacement, and higher head displacement. These features are consistent with components of the hypothesized disgust response, including furrowed brows, eye closure, nose wrinkling, upper lip retraction, upward movement of the lower lip and chin, and drawing the corners of the mouth down and back. We believe that the observed head displacement weight may be related to subjects recoiling from the source of the unpleasant odor, which would produce changes in head scale and translation. 7
In this paper, we de ne expressiveness as the extent to which an individual shows his or her feelings, thoughts, or reactions in a given moment. Following this de nition, we present a dataset that can be used to model or analyze expressiveness in di erent emotional contexts using human labels of attributes relevant to visual expressiveness. We propose and test a series of deep learning and statistical models to predict expressiveness from visual data; we also use the latter to understand the relationship between intepretable visual features derived from OpenFace and expressiveness. We nd that training models for speci c emotional contexts results in better predictive performance that training across contexts. We also nd support for our hypothesis that expressiveness is associated with unique features in each context, although several features are also important across all contexts (e.g., the amount and intensity of facial movement). Future work would bene t from attending to the similarities and di erences in signals of expressiveness across emotional contexts to construct a more robust predictive model. 8
This material is based upon work partially supported by the National Science Foundation (Awards #1734868 #1722822) and National Institutes of Health. We are grateful to Je rey F. Cohn and Lijun Yin for the use of the BP4D+ dataset. Any opinions, ndings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily re ect the views of National Science Foundation or National Institutes of Health, and no o cial endorsement should be inferred.
Amazon Mechanical Turk Questions Four questions were proposed to capture aspects of expressiveness: 1. How strong is the emotional response of the person in this video to [the stimulus] compared to how strongly a typical person would respond? 2. How much of any emotion does the person show in this video clip? 3. How much does the person move any part of their body/head/face in this video clip? 4. How much does any part of the person's face become or stay tense in this video clip? Amazon Mechanical Turk Ratings For the rst question, the Likert scale was anchored for raters as follows: 0 - No emotional response / Nothing to respond to 1 - Weak response 2 - Typical strength response 3 - Strong response 4 - Extreme response For the remaining questions, the Likert scale was anchored: 0 - A little / None 1 2 - Some 3 4 - A lot Video Segmentation For each task, the following segments were sampled from each full subject/task video combination. Timestamps are in SS (seconds) format. The notation -SS refers to a timestamp SS seconds from the end of the video. Frames do not overlap between segments (that is, the last frame of a segments ending at 03 is the frame prior to the rst frame of a segment starting at 03).
Sadness: [00, 03], [03, 06], [30, 33], [33, 36], [{12, {09], [{09, {06], [{06, {03],
[{03, {00]
Startle: [03, 06], [06, 09], [
Pilot Studies on Human Rating Reliability To determine which tasks and questions could be annotated with adequate inter-rater reliability, we conducted a pilot study with 3 crowdworkers rating the video clips from 5 subjects on 4 questions. The results of this study are provided in Table 5. The ICC scores for the sadness and fear tasks looked poor overall, and these tasks were excluded. The ICC scores looked good for the disgust task, and we thought that increasing the number of raters to 6 might increase the reliability of the startle and pain tasks to adequate levels. The results of a follow-up study with 6 crowdworkers are provided in Table 6. The ICC scores indicate that the rst three questions could be annotated with adequate reliability, but the fourth question had poor reliability and was excluded. As such, the nal study included 6 raters of the startle, pain, and disgust tasks with the rst three questions only. 1 2 3 4 1 2 3 4 0.632 [0.290, 0.825] 0.616 [0.248, 0.821] 0.749 [0.515, 0.861] 0.280 [{0.391, 0.658] 0.197 0.391 0.368 0.086 [{0.403, 0.567] [{0.168, 0.639] [{0.103, 0.659] [{0.596, 0.507]
Task