How does the relationship between personality traits and the basic emotions vary across the modalities of self-report and facial expression analysis? This article presents the results of an exploratory study that quanti es consistencies and di erences in personality-emotion mappings across these two modalities. Twenty-four participants answered a personality questionnaire before watching twelve emotionally provocative videos. Participant's self-reported their emotional reactions per video, while their facial expressions were being recorded for automated emotional analysis. The results indicated that overall there was greater consistency than di erences in personality-emotion mappings across the two modalities. The robustness of this relationship enables direct applications of emotional-state-to-personality-trait in academic and industrial domains.
Cognitive science has shown that the functioning of both personality and emotion
is necessary for positive well-being [
There exists a wealth of research that has investigated the phenomena of
personality and emotion [
However, whilst the potential for applying personality-emotion mappings in
practical domains is exciting, there is a need to assess the robustness and
generalisability of this relationship. If emotions are a reliable indicator of personality
(or vice versa), then it needs to be demonstrated that this relationship is robust
across important factors [
In terms of generalisability, an important factor is the modality of emotional
expression, e.g. subjective self-report and facial expressions. Research
methodology for assessing personality, emotion, and their relationship is largely reliant on
self-report-based questionnaires. From a researcher's point of view,
self-reportbased questionnaires are cost-e ective and quick to administer. However, this
reliance on questionnaires can weaken the validity of results in cases where
repeated self-report is required per participant (e.g. \retest artifact" e ects [
How consistent are emotional-state-to-personality trait mappings converge
across multiple modalities? This paper describes an exploratory research
experiment that investigated this question. The experiment investigated the level
of consistency of personality-to-emotion mappings across the modalities of
selfreport and facial expressions. If it can be shown that there exists a large degree of
consistency with self-reported and automatically extracted emotions, then this
forti es the concept of \state-to-trait" mappings as a usable tool [
Personality was conceptualised as personality traits, which are the typical
expressions of cognition, behaviour, a ect, and motivation across time [
The research study utilised a machine learning-based detection platform,
Emotion Viewer [
The paper is structured in the following manner. Section 2 describes the experiment's methodology in terms of the design, participant pool, materials used, and procedure. Section 3 presents the results for descriptive and inferential statistical analyses. Section 4 discusses the results in the context of the research question. Section 5 concludes the paper and provides recommendations for future research in this area. 2
The experimental method adopted generated a range of interesting results
worthy of publication. Experimental results quantifying the links between
personality traits and basic emotions via self-reported emotions were published in [
The sample consisted of 24 participants (n = 24, females = 16, males = 8, Mage = 31.96, SD = 13.73) from a subset of 38 participants. The age range of the sample (range = 19-63) is larger than in most social science research, which tends to primarily consist of 18-23 undergraduate students. 2.2
Emotions Scale - A Likert-scale was created for the purposes of this study. Participants were asked to answer the question \While watching the previous video, to what extent did you experience these following emotions? Please use the following scale in your self-assessment: 1 = Not at All; 2 = A little bit; 3 = Moderately; 4 = A lot; 5 = A great deal/an extreme amount". This scale was designed to capture the experience of each emotion, but not the level of valence or arousal. Participants completed the scale 12 times each, consisting of 84 questions overall (with alpha = .95). Each individual emotion was assessed 12 times throughout the duration of the study: Anger (alpha = .80), Disgust (alpha =.83), Fear (alpha = .85), Sadness (alpha = .76), Joy (alpha =.66), Surprised (alpha =.88).
Personality Questionnaire - The Big Five Aspects Scale is a reliable
measure for Big Five traits and their associated sub-traits [
Technology - Participant's reactions were recorded with the camera of a 2015 MacBook Pro, which included a 1080p web-camera. The software, Screen-Cast-O-Matic, was used in order to both record the participant and the MacBook screen simultaneously. Participants were also given a pair of Bose QuietComfort Noise-Cancelling Headphones to wear whilst watching the videos.
Emotional Stimuli - Previous research by independent groups have
demonstrated that video clips from movies and TV shows are a reliable
method for evoking emotional reactions [
Twelve video clips were used in this experimental design. Nine of those video clips were chosen based on prior research demonstrating their ability to evoke emotional reactions. Three new video clips were also selected - the rationale being that the nine tested video clips provided a solid foundation to empirically evaluate new stimuli. The overall list for the videos is presented in Table 2, along with the clip's length and the expected emotional reaction it would elicit.
Emotion Viewer - The Emotion Viewer analyses real-time video to
automatically classify emotions using a machine learning supported support
vector machine. Figure 1 depicts a demonstration of the tool in operation. The
tool comes with three options: track face, track expressions, and to set the
voting count. The rst two are required to enable facial expression analysis.
The voting count refers to the amount of consecutive classi ed emotions on a
frame-by-frame basis required to register a particular emotion. The voting
count for this research study was set to 10, which is the highest supported by
the Emotion Viewer. This means that for the Emotion Viewer to output an
emotion classi cation (for this example, Anger) it would require 10 consecutive
frames where it detected the emotion Anger. This represented a conservative
approach and was chosen to reduce the risk for Type 1 errors. More
information on the design of the Emotion Viewer is available from [
Participants initially lled out a demographic information form online, which included questions regarding their age, gender, nationality, and previous experience with psychometric tests. Participants were then invited to the laboratory stage of the experiment, provided they t inclusion criteria for the study (over 18 years of age and have not been diagnosed with an a ective disorder).
In the rst part of the laboratory stage, participants completed the Big Five Aspects scale questionnaire, which on average took about 15 minutes to complete. In the second part of the laboratory stage, the researcher set up the video recording on the MacBook Pro. Given that participants varied in height, there had to be a manual check to ensure that each participant's face occupied the camera frame. Once this was settled, the researcher would leave the room, and the participants watched 12 video clips always in the same order (see Table 2). Participants were alone when viewing videos to elicit a more natural reaction and to prevent the participant from feeling self-conscious about their response. Participants were given a pair of noise-cancelling headphones whilst watching the videos, to help immerse themselves in the video. After each video, participants completed a short emotion questionnaire, asking how they felt whilst watching the video clip. Overall, the study took a participant 1 hour to complete. 3
This section presents the key descriptive and inferential statistics from the study in relation to the level of di erentiation between mappings between personality and emotions across modalities. 3.1
Emotions - Participants did not seem to experience a dominant emotion throughout the study. The mean and standard deviations for both self-report and automatically classi ed emotions are presented in Table 1.
standard deviation for self-reported emotional reactions per video clip. Personality Self-Reported Scores - The descriptive statistics for self-reported personality scores are presented in Table 3.
Emotions M
SD Emotions
M SD A-Anger
and Automatically Extracted Emotions Table 4 presents the correlated mapping only for sub-sample (n = 24). These results are presented as a means for comparison with the mapping between personality traits, self-reported emotions, and automatically classi ed emotions.
Emotions Conscientiousness positively correlated with recognition of Anger with a large e ect size (df = 23, p 0:01, r = 0.58). The sub-traits of Conscientiousness, Industriousness and Orderliness also positively correlated with Anger with large e ect sizes (df = 23, p 0:05, r = 0.51; df = 23, p 0:05, r = 0.50). Orderliness negatively correlated with recognition of Sadness with a medium e ect size (d f = 23, p = .059, r = -0.39).
Neuroticism negatively correlated with recognition of Fear with a medium-tolarge e ect size (df = 23, p = .04, r = -0.41). The sub-trait Volatility negatively correlated with recognition of Fear with a medium-to-large e ect size (df = 23, p = .04, r = -0.42). Volatility positively correlated with recognition of Disgust with a medium e ect size (df = 23, p = .05, r = 0.39)
tions A Pearson correlation was conducted on the matrix of self-reported emotions and A-emotions. A matrix depicting the level of consistency and di erences between self-reported emotions and automatically classi ed emotional expressions,is presented in Figure 2 along with the e ect size for each comparison. In terms of consistency/di erences for same-emotions across modalities, then 5 of the 7 emotions are positively correlated: Fear (df = 23, p = 0.10), Sadness (df = 23, p = 0.65), and Surprise (df = 23, p = 0.08). The emotion Disgust (df = 23, p = 0.72) and Anger (df = 23, p = 0.88) diverge with negative correlations with their modality counterpart. 4
This study aimed to investigate whether the relationship between personality traits and emotions is robust enough to generalise across the modalities of selfreport and facial expression analysis. The results showed that there existed more consistency than di erences in personality-emotion mappings. This section discusses both (i) which personality-emotion mappings showed greater consistency than di erence and (ii) which personality-emotion mappings showed greater differences than consistency. Potential reasons for (i) and (ii) are also discussed. Additionally, an evaluation of the video-clips used is conducted and recommendations for future researchers interested in employing video clips in emotionelicitation research is provided. 4.1
A consistent mapping is de ned here as similarity in the direction of correlation (e.g. both positive, both negative) and the existence of non-trivial e ect sizes (both must be greater than r = :10) across both self-reported and automatically classi ed emotions. Overall, 11 of the 15 personality traits showed greater consistency than di erences across both modalities.
Openness to Experience mapped consistently for Joy and Disgust. Openness
mapped consistently for Joy and Surprise; Intellect mapped consistently for Joy.
The largest consistency found across both emotion modalities for these traits
was the emotion Joy, which positively correlated with small-to-medium e ect
sizes. This is consistent with prior research linking Openness to Experience with
positive emotion systems in the brain [
Conscientiousness mapped consistently for emotions Anger, Surprise, Disgust. Industriousness mapped consistently for Anger and Surprise; Orderliness mapped consistently for Anger and Disgust. The positive relationship found between Conscientiousness and Anger was the strongest across the entire data-set for both self-reported and automatically classi ed emotions.
Extraversion mapped consistently for Joy. Enthusiasm mapped consistently
for Joy and Sadness; Assertiveness did not map consistently across any emotion.
The consistent relationship found between Extraversion and Joy is consistent
with past research ndings showing that people high in Extraversion experience
more joy on a daily basis [
Agreeableness mapped consistently for Fear and Surprise. Compassion also mapped consistently for Fear and Surprise; Politeness mapped consistently for Joy. As detailed out in the next sub-section, there was a considerable amount of di erences between the modalities.
Neuroticism mapped consistently for Anger, Joy, and Disgust. Withdrawal
mapped consistently for Sadness and Surprise; Volatility mapped consistently
for Anger, Joy, and Disgust. The positive correlation found between Joy and the
personality traits associated with Neuroticism is inconsistent with past research.
Neuroticism has been consistently considered a personality trait that is
negatively valenced and it has been repeatedly linked to the experience of mental
illnesses [
An inconsistent mapping (di erence) is de ned here as when the direction of correlation for both modalities is the opposite of one another (e.g. one positive, one negative) and the existence of non-trivial e ect sizes (both must be greater than r = :10). Overall, 2 of the 15 personality traits showed greater inconsistency (di erence) than consistency across both modalities.
Openness to Experience mapped di erently for Sadness. Openness also mapped di erently for Sadness; Intellect mapped di erently for Surprise and Disgust. For both Openness to Experience and Openness, there were more consistencies than di erences across modalities. This was the opposite for the sub-trait Intellect.
Conscientiousness mapped di erently for Sadness. Industriousness and Orderliness also mapped di erently for Sadness. Overall, there were more consistencies than di erences for Conscientiousness and its two sub-traits.
Extraversion mapped di erently for Anger. Enthusiasm also mapped di erently for Anger; Assertiveness did not map di erently across both modalities. It should be stated although the di erences between self-reported and automatically classi ed Fear were still striking, it did not hit our threshold.
Agreeableness mapped di erently for Anger. Compassion also mapped di
erently for Anger. Politeness mapped di erently for Anger and Sadness. Anger was
the major source of divergence for mappings related to Agreeableness.
Agreeableness, and in particular Politeness, has been linked to reduced experience of
Anger [
Neuroticism mapped di erently for Fear. Withdrawal also mapped di erently for Fear; Volatility did not map di erently. There existed several mappings with Neuroticism that were stronger in a given modality (e.g. Sadness), however, these mappings did not have a large enough e ect size in both modalities to meaningfully compare and contrast. 4.3
Twelve video clips were used as emotional stimuli in this research study. Three of those videos had not been previously used in prior experimental research. Two of those new videos, Annabelle and Who Dunnit? Test Your Awareness, were successfully elicited high levels of self-reported emotion. Annabelle evoked the strongest mean experience of Fear across the participant group and Who Dunnit? Test Your Awareness evoked the strongest mean experience of Surprise across the participant group. Both videos are recommended as reliable emotional stimuli for future research. However, the third inclusion, Peep Show, is not recommended for future usage. The results of this study also cast suspicion on the reliability of the video clip, Sea of Love, which has been used in prior research experiments. but only a minimal amount of its targeted emotion, Surprise. Researchers requiring a video to reliably produce Surprise are recommended to use the video clip Who Dunnit? Test Your Awareness instead.
Personality traits and basic emotions are signi cant predictors of human
behaviour and the functioning of both phenomena is necessary for positive
wellbeing. Previous research has also found quanti able links between these two
phenomena that can enable state-trait inferences, i.e. personality-emotion
mappings [
This paper described an empirical research study that tested the generalisability of personality-emotion mappings across a self-reported based approach and an automatically classi ed emotion-based approach via video. If personalityemotion mappings were robust across both modalities, then this would be an indicator that technological-based approaches can directly analyse personality, emotion, and their relationship. Technological-based approaches can enable intelligent and automatic observations of these phenomena with minimal input from participants.
The results showed greater consistency than di erences in personality-emotion mappings across the two modalities. For the 15 personality traits captured in this study, the results showed that: (i) 11 personality traits showed greater consistency than di erences personality-emotion mappings across modalities; (ii) 2 traits showed an equal amount of consistency and di erences in personalityemotion mappings across modalities; (iii) 2 traits showed greater di erences than consistency in personality-emotion mappings across modalities.
The criteria for assessing a personality-emotion mapping was two-fold, a comparison between the direction of the correlations and the existence of non-trivial e ect sizes. However, there is scope for future research to assess the degree of consistency/di erence between the two modalities. Quantifying the degree of consistency/di erence would enable assessments about the strengths and weaknesses of self-report and automatically classi ed approaches.
Overall, the results are a promising indication that the relationship between personality traits and basic emotion is robust enough to enable research and commercial applications. However, given the small sample size of the research study, caution is required in generalising the results. Future research in this area that can (a) scale the number of participants and (b) incorporate other modalities of emotional expression (e.g. language, speech) and (c) quantify the degree of consistency/di erence across modalities would represent a signi cant next step in this research area.