The relationship between emotions elicited by film clips and spontaneous dynamic facial expressions was investigated. Participants (n = 10) watched 13 emotional film clips, and their facial responses were recorded using a motion capture system. We extracted 3-sec length intervals in which facial events occurred from motion sequences. The participants were asked to self-assess their felt emotional arousal and positive and negative affect for each interval. To find the spatiotemporal components of dynamic facial expressions, we employed the multiway decomposition method, PARAFAC, on a time sequence of facial landmark coordinates standardized via methodologies of geometric morphometrics. The second component was related to facial movement that appears slowly and then maintains a stable state over a long term. Finally, the third component was linked to movement that appears and rapidly returns to the initial state. Local regression analysis was performed to obtain the distribution of the component scores on a two-dimensional plane: pleasure-displeasure and arousal-sleepiness. The third component was negatively correlated with arousal level.
Many studies have investigated morphological features of
facial expressions. Most researches on facial expressions
have relied on the use of photographed images or “static”
information. However, affective facial expressions are
dynamic in nature. Several researchers have claimed that
dynamically changing facial configuration plays an
important role in the perception of facial patterns and
attribution of emotional category labels
Although the presence of a motion signal is important for
emotion recognition, the benefit of dynamic display does
not seem to be solely attributed to an increase in the amount
of static information. Natural facial events would not occur
at a constant speed as in morphing animations; however,
they occur in a nonlinear manner. Such nonlinear facial
motion leads to more accurate emotion recognition
compared to linear motion animation
In many studies of facial expressions, portraits of actors’
stereotypical emotion expressions or facial actions of
specific predefined patterns (e.g., FACS
Facial motion capture systems have been widely used in
the film industry. Recently, such landmark-based
approaches have been applied to research on spontaneous
dynamic facial expressions
Japanese undergraduate and graduate students (n = 10: 8 men and 2 women; age: 18 to 24 years, mean age = 21.4, SD = 1.96) participated on a voluntary basis. The participants gave written consent to participate in the study, and Osaka Electro-Communication University ethics committee approved the study.
The participants watched thirteen clips intended to elicit differentiated emotional states, such as positive/negative mood, anger, sadness, fear, or disgust. The clips were taken from a database of emotion-eliciting films (Schaefer, Nils, Sanchez, & Philippot, 2010) (Table 1). Each of the clips was shown in a Japanese-dubbed version and was approximately 2 min in length.
Participants were instructed to watch all the clips attentively, without diverting their attention from the monitor. When participants were ready to begin the experiment, lighting in the room was dimmed. Each trial started with a countdown leader (3s). It was followed by a presentation of explanatory text of the outline of the narrative of the film (30s), and a presentation of the clip. This procedure was repeated for each film excerpt. The order of items within each set was randomized. Each session lasted approximately 40 min. The film clips were displayed on a 12-inch monitor placed about 1m in front of the participant. The audio of the films was played through speakers placed in front of the participant.
The sequences of 30 facial landmark coordinates were recorded using a facial motion capture system (OptiTrackFLEX: V100R2) at 100Hz (Figure 1). The landmark locations for each participant were decided based on a front facial image that was photographed before the experiment. We also set up a video camera to capture the entire scene and speakers for monitoring and as a reference for synchronization between facial motion sequence and stimuli.
To extract intervals in which facial events occurred from
the motion data, we applied principal component analysis
(PCA) based on automatic motion segmentation technique
to the facial motion sequences
2 12 24
Each face in each frame differed in location, size, and
orientation. To standardize them, we performed a
Generalized Procrustes analysis (GPA) on the facial
landmarks of all frames of all faces
This motion data of the participants can be regarded as a three-mode data = !"# = 1,2, … , , = 1,2, … , , = 1,2, … , (i corresponds to the time sequence, j corresponds to the landmark, and k corresponds to each extracted interval). To find the components of facial expressions from the motion data, we performed PARAFAC analysis on the three-mode data. PARAFAC is a generalization of PCA to higher order arrays, given by ! !"# =
!" !"!" + !"# !!! In the above, !", !", !" correspond to the mode of time sequence, and landmark and interval, respectively.
We calculated the principal component scores and loadings up to the third factor with PARAFAC. Figure 3 illustrates the changes of each factor loading concerning time sequence. The first component was found to be a “static” factor, indicating that it was not linked to any facial motion but to static morphological features that are a geometric arrangement of facial parts. However, the second and the third component were related to facial movement. In this study, we examined the second and third components that were linked to facial movement.
The second component score started increasing at around 60 msec and peaked at around 150 msec, after which the score becomes stable. Thus, the second component is considered to be related to a facial movement that appears slowly and maintains a stable state. The third component was linked to a movement that appears at around 50 msec, peaks at 100 msec, and then rapidly returns to the initial state.
To visualize the facial changes along the second and third component, facial landmark movements were reconstructed based on the loadings of both time sequence and landmarks for each component (Figure 4). The results of the reconstruction indicated that a higher second component score of extracted interval was related to eyebrow raising and mouth opening, whereas a lower score was related to eyebrow lowering and mouth closing (Figure 4-A). A higher third component score was related to horizontal movement, including mouth opening in a horizontal direction, and a CO M PO N EN T1
0 G .01 DIN .000 A
LO .−105 0 CO M PO N EN T2
0 G .01 DIN .000 A
LO .−105 0 CO M PO N EN T3
0 G .01 DIN .000 A LO .−105 0 50 50 50 001 05 0 −05 −010 001 50 0 −05 −010 A B 100 100 100
TIM 1E（50m s） TIM 1E（50m s）
TIM 1E（50m s） CO M PO N EN T2 200 250
300 200 250
300 200 250 300 lower score was related to a movement of contracting lips into a rounded shape and knitting eyebrows (Figure 4-B). −100 −50 0 50 100
Subjective emotional states during observation of emotional
film clips were examined using the same film clips as used
in the facial motion capture session. The participants were
also the same as those in the facial motion capture session.
For each participant, we presented each film clip and
paused the clip at the moment the facial expression had
occurred, which marked the segmentation point. Participants
were asked to rate their mood using the Affect Grid
technique
The relationship between facial motion components and
subjective emotional state was investigated. It was
considered inappropriate to assume an a priori model
regarding the relationship between facial expressions and
emotion. Therefore, we adopted LOESS, a robust leveling
technique based on local polynomial regression
The relationship between Affect Grid scores and smoothed component scores of both the second component and the third component is shown in Figure 5 (span, kernel function range was set as 3.0). Figure 5-A shows that both negative and low arousal affect lead to higher scores for the second component, suggesting that the component of facial movement that appears slowly is related to expressions both of emotional valence and arousal. For this component, both positive and high arousal expressions are similarly correlated to eyebrow raising and mouth opening. On the other hand, it can be seen from Figure 5-B that arousal level was basically linked to the third component that reflects short-term facial deformations, except for the cases of strong unpleasant affect. As for this component, a higher arousal level corresponds to the movement of contracting lips into a rounded shape and knitting eyebrows.
This study suggests that dynamic facial expressions consist of plural components differing in spatiotemporal characteristics (i.e., long-term facial deformations and shortterm facial deformations). One component was found to be connected to facial deformations that appear slowly and maintains a stable state over a long term, and the other was connected to rapid appearance and disappearance of facial deformations. Each component corresponds to movements in different directions and of different facial parts. The findings of the present study indicate that dynamically changing facial expressions can be described by synthesizing a few components of facial movement that differ in spatiotemporal characteristics.
Moreover, we show the connection between the components of expressions and emotional valence and arousal. The results suggested that long-term facial deformation was related both to valence and arousal whereas short-term deformation was related solely to arousal.
The results of the study also suggest that the combination of motion segmentation technique, methodology of geometric morphometrics, and modified principal component analysis is a valid method for finding components of dynamic facial expressions.
A C O M PO N EN T2 −2
Because of the small number of participants in this study, it is difficult to consider the facial components found were stable. Future studies should examine the accuracy of PARAFAC-based expression model in detail with a larger number of participants.
A part of this study was supported by a Grant-in-Aid for Scientific Research (No. 22730591) from the Japan Society for the Promotion of Science. and
Vision