Generate Context-Sensitive Backchannel Smiles for Embodied AI Agents with Applications in Mental Health Dialogues

Abstract

Addressing the critical shortage of mental health resources for efective screening, diagnosis, and treatment remains a significant challenge. This scarcity underscores the need for innovative solutions, particularly in enhancing the accessibility and eficacy of therapeutic support. Embodied agents with advanced interactive capabilities emerge as a promising and cost-efective supplement to traditional caregiving methods. Crucial to these agents' efectiveness is their ability to simulate non-verbal behaviors, like backchannels, that are pivotal in establishing rapport and understanding in therapeutic contexts but remain under-explored. To improve the rapport-building capabilities of embodied agents we annotated backchannel smiles in videos of intimate face-to-face conversations over topics such as mental health, illness, and relationships. We hypothesized that both speaker and listener behaviors afect the duration and intensity of backchannel smiles. Using cues from speech prosody and language along with the demographics of the speaker and listener, we found them to contain significant predictors of the intensity of backchannel smiles. Based on our findings, we introduce backchannel smile production in embodied agents as a generation problem. Our attention-based generative model suggests that listener information ofers performance improvements over the baseline speaker-centric generation approach. Conditioned generation using the significant predictors of smile intensity provides statistically significant improvements in empirical measures of generation quality. Our user study by transferring generated smiles to an embodied agent suggests that agent with backchannel smiles is perceived to be more Workshop Proceedings

-

human-like and is an attractive alternative for non-personal conversations over agent without backchannel smiles. CEUR

ceur-ws.org

1. Introduction

Fewer than a third of the US population has suficient access to mental health professionals [ 1 ]. This highlights the need for additional resources to help mental health professionals meet the community’s demands. Problems like symptom detection and evaluating treatment eficacy have made great strides with AI [ 2, 3, 4 ] and the mental health community can greatly benefit from this AI intervention. Embodied agent-based systems due to their multimodal behavioral capabilities are a promising solution to support such mental health needs. However, the development of such systems presents numerous challenges. These include the scarcity of mental health-related datasets, limited access to domain experts for designing reliable and robust systems, and the ethical considerations crucial to their design and adaptation.

Among such challenges, one aspect that stands out is the agent’s ability to establish a common ground with users. Addressing this is particularly crucial when the agent functions as a listener. Efective grounding in such

Machine Learning for Cognitive and Mental Health Workshop (ML4CMH), AAAI 2024, Vancouver, BC, Canada. ∗Corresponding author. nEvelop-O in an embodied agent in a human-agent interaction: Speaker and listener (agent) turns are used to generate the listener’s response facial expression as landmarks. The landmarks are then integrated with the embodied agent and added to the conversation flow represented as a dotted arrow. scenarios relies heavily on multimodal non-verbal behaviors like backchannels. These subtle yet impactful cues are pivotal in building rapport and understanding between the user and the agent. Hence, understanding and incorporating these behaviors into embodied agents is not only challenging but also essential for creating a supportive and empathetic environment for individuals and their physical realization by emulating the seeking mental health support. Addressing these chal- generated behavior with an embodied agent. lenges can pave the way for more efective, accessible, 5. Show that our BC smile generation yields approand empathetic digital mental health interventions. priate and natural-looking smiles through a user

In dyadic conversations, at any given time one person study involving the embodied agent. may have the floor (i.e., is speaking) while the other is listening. Backchannels (BC) refer to behaviors of the Results suggest speaker sex, their use of negations, listener that do not interrupt the speaker. BCs signal loudness, word count in the listener’s turn, their usage of attention, agreement, and emotional response to what is comparisons, and mean pitch are significant predictors said. Inappropriate BC smiles such as ones that appear of BC smile intensity. Our generative approach shows too short or too long or for which the timing appears that taking listeners’ behavior into account improves “of” can disrupt the conversational rapport and result in performance, and adding the conditioning vector ofers unsuccessful or disrupted conversations. Our objective significant improvements in terms of empirical metrics is to understand appropriate BC smiles from dyadic con- such as Average Pose Error (APE) and Probability of versations and how an embodied agent can employ them Correct Keypoints (PCK). when interacting with a human.

Conversational agents typically realize BC smiles us- 2. Related Work ing rule-based systems, discriminative approaches, or sometimes simply mimicking the smiles of the speaker. Existing works have validated the eficacy of an agentMimicking, however, fails to generalize to situations that driven conversation in mental health dialogue and counrequire a contextually relevant smile. And rule-based seling situations. DeVault et al. [ 6 ], through their agentand discriminative approaches ofer limited coverage due based interviews for distress and trauma symptoms, to the diversity of smiles [ 5 ]. found that participants were comfortable interacting with

We present a generative approach for BC smiles in the agent as well as sharing intimate information. Utami listeners to address these limitations and enable contextu- and Bickmore [ 7 ] used embodied agents for couples counally relevant BC smiles in embodied agents. An overview seling. Participants reported significantly improved afof the approach is presented in Figure 1. Unlike existing fect and intimacy with their partner and generally enworks that solely depend on speaker behavior for BC pro- joyed the agent-driven counseling session. Our work duction (see related work section), we use both speaker builds on this line of research to improve the BC capabiland listener behaviors to study how they afect the in- ities of agents. tensity and duration of the BC smile. We use cues from Backchannel behaviors were traditionally produced prosody, language, and the demographics of dyads to using a set of predefined rules based on prosodic or linidentify statistically significant predictors (referred to as guistic cues of the speaker. Both Ward and Tsukahara a conditioning vector) of smiles. In addition to the audio [ 8 ], Benus et al. [ 9 ] have found prosodic cues (particufeatures from both interaction participants, we leverage larly pitch and its changes) to be reliable predictors for the conditioning vector in generating the BC smiles. In vocal BC occurrence. In contrast, we use prosody and this paper, we: linguistic cues from both speaker and listener to identify significant predictors of BC smiles. 1. Annotate backchannel smiles in a face-to-face In the multimodal context, Bertrand et al. [ 10 ] studinteraction dataset1 of dyads that difer in their ied prosodic, morphological, and discourse markers for composition of biological sex and type of relation- their efect on vocal and gestural backchannels (hand gesship. tures, smiles, eyebrows), and Truong et al. [ 11 ] explored 2. Present our statistical analysis to identify vari- visual BCs by often limiting them to head nods and, at ous speaker and listener-specific cues that sig- times, grouping diferent BCs into the same category [ 12 ] nificantly predict the duration and intensity of without accounting for their intrinsic diferences. They backchannel smiles. depended on the speaker’s behavior to identify the occur3. Generate backchannel smiles using an attention- rence and ignored the listener. In addition to leveraging based generative model that uses the listener and the listener behavior, we specifically study smiles because speaker turn features with the identified signifi- of their diversity and include both unimodal (visual) and cant predictors. bimodal (visual together with vocal activity) BC smiles. 4. Bridge the gap between the model-based genera- Wang et al. [ 13 ] introduced diversity in generated tion of non-verbal behaviors (as facial landmarks) smiles by conditioning on a specific class and sampling using a variational autoencoder. Learn2Smile [ 14 ] used 1Data and code: https://github.com/bmaneesh/Generating-Context- the facial landmarks of the speaker to generate comSensitive-Backchannel-Smiles/ plete listener behavior by separately predicting the lowfrequency (nods) and high-frequency (blinks) components of facial motion. Ng et al. [ 15 ] leverage the speaker and listener’s motion and speech features to predict the listener’s future motion information. Unlike earlier works that have been limited to facial expression generation using landmarks, their usage of 3D Morphable Models to define facial expressions ofers a flexible solution to generate realistic facial expressions in the presence of diverse head orientations. These solutions focus on Figure 2: Distribution of speaker and listener sex across diferthe entire listener’s behavior and ofer no insights about ent interpersonal relationships in annotated RealTalk dataset. specific BC behaviors. Their integrations are also limited Relationships are color-coded: siblings (pink), friends (orange), to 3D Morphable Models. paternal (green), and romantic couple (grey).

The BC smiles produced in this work not only leverage the speaker and listener activity but also condition the generation on salient factors that were found to be signif- the 191 annotated smiles had an A-level or higher inicant predictors of smile attributes – duration (the time tensity. One outlier smile was dropped because of the elapsed between the onset of a smile and its ofset) and extremely long duration. The resultant 157 smiles, along intensity (maximum amplitude of a smile). Using an em- with their predicted intensity, were used in this work. bodied agent, we also bridge the gap between generated In addition to the video recordings at 25 fps and 720p landmarks and their physical realization. resolution, the dataset also contains speaker-identified turn-level text obtained through automatic transcription [ 18 ]. The individuals in the dyadic interaction occupied 3. Dataset ifxed positions (left and right) in the videos. In this work, the biological sex of the participants was inferred from the videos. Videos where sex could not be established with confidence were discarded.

3.2. Efect of Sex and Relationship on

Smile Attributes

Given various interpersonal relationships in the dataset

of individuals of both sexes, we compared the mean duration of backchannel smiles across the factors using ANOVA (Table 1) with type-III sum of squares to account for imbalance between males and females. Two-way interactions between sex, and sex and relationship were also included. The ANOVA analysis suggests that the duration of backchannel smiles difers significantly by listener sex and the interaction efect of the listener sex and relationship. A post hoc Tukey revealed that male listeners, when interacting with their siblings (regardless of speaker sex), express longer BC smiles (p<0.05).

Similarly, the intensity of smiles marginally difered by the speaker’s sex. The post hoc Tukey revealed that the smiles as a response to a male speaker are less intense than a female speaker (p<0.1). ANOVA analysis is presented in the appendix as Table 4.

3.3. Efect of Context Cues

Our contextual cues were extracted from prosody and

speech features independently derived from the turns of both the speaker and the listener just before the smile onset. Since the speaker’s turn continues while the listener backchannels, speaker activity till the onset of the smiles One of the primary challenges in studying non-verbal behavior in mental health interactions is access to an appropriate dataset. Patient-therapist interactions or interactions with mental health professionals are accessrestricted to protect the identifiable information of the individuals. As a result, we use a YouTube-based largescale dataset of face-to-face dyadic interactions–RealTalk [ 16 ]. The RealTalk dataset consists of individuals taking turns asking predefined, intimate questions about family, dreams, relationships, illness, and mental health2. We believe intimate conversations are among the closest accessible alternatives to studying BC behaviors for mental health applications. In this section, we elaborate on our contributions in terms of the annotations for BC smiles and discuss how they difer by the demographics of the dyads and features from the speaker and listener turn preceding it.

3.1. Annotating Backchannel Smiles

We manually annotated 191 BC smiles from 48 (out of 692) dyadic interactions in the RealTalk dataset. The dyads comprised male and female participants from diferent ethnicities, and social relationships such as siblings, paternal, romantic, and fraternal. The smiles were nearly balanced across the diferent interpersonal relationships (see Figure 2). An automated facial expression prediction framework [ 17 ] was used to evaluate the reliability of the manual annotations. About 83% (i.e., 158 smiles) of

2The original videos can be accessed from https://www.youtube.

com/c/TheSkinDeep

Prosody cues: Our prosodic features consisted of some

of the fundamental characteristics of speech, such as mean pitch during the turn, range of the pitch, and Root Mean Square (RMS) energy of the audio signal. These features were chosen because of their relevance (see related work) in BC behavior and also due to the ease of interpretation as well as their ability to convey various behavioral traits. For example, RMS energy conveys traits such as confidence, doubtfulness, and enthusiasm [ 20 ]. Lastly, using the OpenSMILE [21] software, prosodic features were obtained.

4. Modeling Smiles

To automatically generate BC smile and non-smile activity in listeners, we use the audio from the speaker’s current turn and the listener’s last turn as input. 15 smiles were dropped due to dificulties in the preprocessing steps with MFA. The remaining 142 annotated smile instances were augmented with an equal number of nonsmile instances. The non-smile instances were identified so that they were at least two seconds away from the onset of the closest smile instance, a strategy adopted from [23] for turn-taking prediction. The mean duration of smiling and non-smiling instances was ensured to be the same.

Attention-based generative model: The generative model (Figure 4) for facial landmark prediction primarily consisted of an encoder and a decoder with a one-layer GRU each. Inputs to the model were embeddings from speaker and listener turns extracted using the pretrained vggish model [24]. We limited the input context length to use turn durations of 60 seconds. The output context was limited to predicting one second of facial activity.

The speaker vggish embeddings were used as input to the encoder. The hidden state of the GRU was initialized as the mean of the listener’s turn embeddings. The final hidden state of the encoder was concatenated with the Mean Squared Error (MSE) between predictions and the conditioning vector, and a linear layer with ReLU the ground truth. The learning rate was halved when activation was used to match the dimensionality of the validation loss plateaued for 20 consecutive epochs. Data decoder’s hidden state. At each decoding step, attention was partitioned into 75 (train), 15 (validation), and 15 [25] was applied between the encoder output and the de- (test) split in terms of the number of dyads. Models were coder’s last hidden state (Equation 1) to use as the input trained for 250 epochs, and validation loss was used to deto the next step. termine the best model for testing. This was repeated 10 times to evaluate the statistical significance of diferences (1) against baseline speaker-based BC generation setting.

ℎ + −1 ) where ( −1 , ℎ ) is the attention between decoder last hidden state ( −1 ) and encoder output (ℎ ). s and are linear layers.

Metrics: Objective measures of performance from ges

ture generation approaches, including Average Pose Error (APE) and Probability of Correct Keypoints (PCK), were adopted to quantify the generated landmarks 4.1. Implementation details against the ground truth from the AFAR toolbox. APE The videos were split into two vertical halves, one cor- (Equation 2) is equivalent to the mean squared error beresponding to each individual in the dyadic interaction. tween predicted facial expression and ground truth facial These were used for facial landmark extraction using the expression. PCK (Equation 3) is a proximity-based metric AFARtoolbox [ 17 ]. To account for various facial shapes, that considers the landmark to be correctly predicted if we normalized landmarks to the mean face of the dataset the diference with ground truth falls below a margin. using the approach described in [26]. Because of the We report mean PCK for = 0.1 and 0.2. high degree of correlation between successive frames, fervaemryesthwiredrefrdamowe.nDsaimspplalecdembeynat wfaacstotrheonf tcharlceuel,attoedusaes = 1 ∑=1 ‖( (̂) − ()) ‖2 (2) the diference between the landmarks from successive frames. These were further subjected to a min-max nor- where is the number of landmarks, (̂) is the premalization to allow for individual diferences in smiling diction and () is the groundtruth. dynamics. The normalized displacements were predicted 1 using the attention-based generative model. The pre- = ∑ ( ‖( (̂) − ()) ‖2 ≤ ) (3) dicted frame-level displacements were incorporated into =1 the last known listener facial expression to generate the where is an indicator function and is the margin. sequence of facial landmarks recursively.

We enforced teacher-forcing with simulated annealing during training and linearly decreased the likelihood of 4.2. Results using ground truth at every 20 epochs. Stochastic Gra- Using listener behavior and conditioning vector together dient Descent with a learning rate initialized at 1 − 4 with the speaker behavior resulted in improved perforweight decay and 0.99 momentum were used to minimize mance compared to the baseline speaker behavior-based prediction. As shown in Table 2, APE decreased by 0.273 points while PCK increased by 0.004; these gains were statistically significant. When listener behavior was added to the speaker behavior, marginally significant improvements were observed. APE reduced by 0.206 points while PCK increased by 0.001 points. These reiterate our hypothesis that both speaker and listener contribute to BC behaviors. When speaker behavior was augmented with the conditioning vector, only nominal diferences were observed against the baseline. APE increased by 0.063 points, and PCK decreased by 0.001.

To understand how the performance varies with different smiles, we predicted APE (and PCK) as a linear combination of duration, intensity, and the model configuration using a regression model. Results from Figure 5 show that duration significantly afects the PCK. Interestingly, the positive slope suggests that longer smiles are generated better over shorter smiles. Only a marginally Figure 5: Efect of duration and intensity of smile along with significant efect of duration can be observed for APE. ablation of inputs on generative model performance measured With the increase in the intensity of the smile, the gen- using APE (top) and PCK (bottom). S & C-speaker and condieration performance decreases. This is significant for tainodnilnisgtveencetroar,nSd&cLo-nsdpietaiokneirnagnvdelcisttoernaesr, iannpduSts,Lto&tCh-espmeoadkeerl. D-level and E-level smiles. Using listener features and ‘⋅’, ‘*’ and ‘***’ indicate significance with p <0.1, p <0.05 and p the conditioning vector along with the speaker features <0.001 respectively. improves the performance (negative and positive slopes for APE and PCK, respectively) compared to the baseline speaker-based generation. However, this efect is not utterance. However, the model fails to capture this vertistatistically significant. cal motion.

Qualitative evaluation of ground truth landmarks from Metrics like APE and PCK provide an objective meaFigure 6 suggest the deficiencies of the existing facial sure of the prediction. However, evaluating concepts landmark prediction approaches [ 17 ] to accurately track such as realism and contextual relevance of the BC prediclip corners both in the presence and absence of non- tion requires subjective ratings from human evaluation. frontal head pose. While a visually noticeable diference A convention in evaluating landmark or keypoint-based can be observed as the smile evolves, the ground truth generative approaches is the human comparison of prelandmarks fail to capture the subtle lip corner motion. dicted keypoints against the ground truth [ 14, 27 ]. While This limitation in the ground truth has resulted in nom- this might work for problems such as gesture generainal motion in the predicted landmarks. We also found tion that involve a strong motion component, evaluating that BC smiles that co-occur with vocal activity are chal- subtle behaviors like facial expressions using a similar lenging to predict. Figure 7 shows one example where strategy could be challenging. To address this concern, the vertical distance between the upper and lower lips we leverage the emulated version of an embodied agent: increases and decreases because of the simultaneous yeah Furhat [28].

5. Smiles on an Embodied Agent

So far, we have shown modeling smiles by generating

facial landmarks. However, users in real-world scenarios do not expect to see such abstract representations of faces. Aligning these facial landmarks with embodied agents is key for an interactable conversational agent. To achieve this, we describe the procedure to transfer generated landmarks to an embodied robotic simulation system called Furhat. We then conduct a user study for subjective perceived diferences in Furhat’s behavior due to BC smile.

5.1. Emulation Setup

Furhat allows users to control facial expressions using

a set of facial parameters called BasicParams3 (ex. MOUTH_SMILE_LEFT and MOUTH_SMILE_RIGHT to control the left and the right lip corners; BROW_UP_LEFT, BROW_UP_RIGHT to control the left and right eyebrows, etc.). Our setup uses these parameters to enable the embodied agent’s smile and express associated eyebrow actions. The landmarks from a generated smile expression were used to calculate the displacement between successive frames and normalized to the [ 0, 1 ] range. For eyebrows, only vertical displacement was used. Our inputs to the Furhat API consisted of the lip corner and eyebrow displacements corresponding to the frame with the widest smile (maximum horizontal displacement between the lip corners). The duration of the Furhat smile was set to the duration of the generated smile. Figure 8 shows an example of the resultant expression. The user study was conducted using the Furhat Desktop SDK. However, we do not foresee dificulties transferring the emulation setup to a physically embodied Furhat.

5.2. User Study Procedure

We conducted a small-scale user study of participants

watching two pre-recorded videos of the Furhat interacting with an individual. They difer only in terms of Furhat expressing a BC smile. In both interactions, Furhat starts with a brief introduction of itself, followed by a short question–“How have you been feeling over the last two weeks?”. As the user responds, a smile is generated at the appropriate location (see Figure 8). We refer to this that the brightness of the BC smile was appropriate while scenario as the backchannel setting. Another video of two found that the duration of BC smile was longer or the same individual interacting with Furhat with no BC shorter than expected. While no diference was observed (non-backchannel) serves as our baseline. Seven gradu- in terms of users’ preference for Furhat for personal conate students then rated each video recording separately. versations based on the presence of the BC smile, more Note that raters were not primed on the study’s outcome, users (3/7) responded that they would use Furhat with and no explicit instructions about smiles were given. BC smiles for non-personal conversations over Furhat

To quantify the user’s perception of Furhat interacting without BC smiles (2/7). with an individual, the influence of BC smile in addition to the efect of its intensity and duration, and their will- 6. Discussion ingness to interact with one was quantified through the following questions on a 5-point Likert scale (1: strongly agree, 5: strongly disagree).

Our quantitative results suggest that both speaker and lis

tener behavior are important in generating BC behavior.

Using listener behavior together with the conditioning 1. The Furhat’s smiles looked human-like. vector ofered statistically significant improvements in 2. The Furhat’s smiles looked natural and friendly. performance when compared to the baseline speaker3. I would talk to this agent frequently. only model. This efect was observed both in terms of 4. I felt the brightness of Furhat’s smiles was appro- APE and PCK. We also found that our attention-based priate. generative model can predict low-intensity smiles better 5. The Furhat was smiling for longer or shorter du- than high-intensity smiles. Our user study shows that ration than it was expected. more people find our agent human-like when it was able 6. I would feel comfortable talking to this agent to express BC smiles. Participants prefer to interact with about non-personal topics. it over the agent with no BC smile capabilities for non7. I would feel comfortable talking to this agent personal conversations. However, for intimate personal about personal topics. conversations, the presence of a BC smile did not sway their decision.

In addition, open-ended feedback was also a part of the Some limitations of this work include the following. questionnaire. We believe these questions help identify We employed an afordable measure of reliability for BC some user-facing challenges in generating BC behav- smile annotations using a prediction model over a huiors and how they influence users’ attitudes to embodied man rater. A robust approach would involve at least one agent-based dialogue systems for conversations related more human annotator to perform reliability annotations to mental health. on a portion of the dataset. The statistical analysis also assumes that the smiles were independent of the individ5.3. Results uals and dyads. However, a given individual typically produces multiple smiles. Grouping of smiles by factors such as individuals and dyads can be better modelled using a mixed-efects model. Our user study was designed to demonstrate the feasibility of transferring generated facial landmarks to an embodied agent together with understanding perceived diferences between interactions with and without BC smiles. An appropriate evaluation framework would include the user interacting with the romantic relationships, and lack of age and ethnicity inagent. Followed by a comparison of qualitative subjec- formation in the dataset might have resulted in biased tive ratings of user experience and quantified parameters generations. We also acknowledge that using embodied (such as diference in turn duration, language usage, etc.) agents in such sensitive applications should undergo rigof the interaction with and without BC smiles. We believe orous evaluations by technical and domain experts and such approaches provide a holistic evaluation to identify regulatory bodies. In our work, we do not interpret emcritical instances in the interaction. Lastly, we focused on bodied agents as a substitute for professionals in mental BC smiles leaving out other conventional signals such as health or allied areas of healthcare but to provide tools vocal and headpose-based BCs, and how they are afected for them to better serve the community’s demands. We by the cues from the speaker and listener. believe that the advantages and limitations of embodied agents in mental health should be presented to the users and the healthcare experts to provide maximum 7. Conclusion benefits. The information used in this work is identified from a publicly available dataset. Also, special attention has been paid to privacy and copyright requirements for relevant images showing individual faces. The user study raters were voluntary participants, and the University of Pittsburgh IRB approved the data collection.

To enable BCs in embodied agents for mental health applications, we proposed an annotated dataset of faceto-face conversations including topics related to mental health. Our statistical analysis showed that speaker gender together with prosodic and linguistic cues from both speaker and listener turns are significant predictors of the BC smile intensity. Using the significant predictors 9. Acknowledgments together with the speaker and listener behaviors to generate BC smiles ofers significant improvements in terms Bilalpur and Cohn were supported by the U.S. National Inof empirical metrics over the baseline speaker-centric stitutes of Health through award MH R01-096951. Zeinali generation. was supported through the Khoury Distinguished Fel

We bridge the gap between conventional non-verbal lowship at Northeastern University. behavior generation approaches such as landmarks and poses and their realization by showing that generated landmarks can be transferred to an embodied agent. Thus References creating the opportunity for evaluation with a humanlike manifestation over a traditional evaluation by comparing generated landmark (or keypoint) outputs. Our small-scale user study suggests our Furhat agent that backchannels is more human-like and are more likely to attract users for non-personal interactions. In addition to these contributions, we also discussed some limitations in existing technology towards generating accurate ground truth landmarks through examples such as failure to capture mouth movement in bimodal BCs and how they afect the generated outputs. We believe these limitations also serve as directions for future research. Our work serves as a baseline for computer scientists interested in behavior generation, and an attractive source of BC smiles for behavioral scientists to study the efect of context cues on BC smiles in intimate conversations.

8. Ethical Statement

We proposed a generative approach for backchannel smile production to enable naturalistic interactions with embodied AI agents for mental health dialogue. While our dataset ofers diverse smiles from people in diferent interpersonal relationships, like many existing generative approaches, the choice of pretrained embeddings, imbalance between males and females, lack of male-male

Note that the intensity of the smile difers marginally by the speaker sex. It is not afected by other factors such as relationship, listener sex and their interaction.

Mean Sq F value ℎ ℎ ℎ ∗ ∗ ∗ 0.60 3.31 1.22 0.4417 0.0710 ⋅ 0.3055

References

[1] H. Modi , K. Orgera , A. Grover , Exploring barriers to mental health care in the u .s. ( 2022 ). doi: 10 .15766/ rai_a3ewcf9p.

[2] S. Song , S. Jaiswal , L. Shen , M. Valstar , Spectral representation of behaviour primitives for depression analysis , IEEE Transactions on Afective Computing 13 ( 2020 ) 829 - 844 .

[3] F. Ceccarelli , M. Mahmoud , Multimodal temporal machine learning for bipolar disorder and depression recognition , Pattern Analysis and Applications 25 ( 2022 ) 493 - 504 .

[4] Y. Yang , C. Fairbairn , J. F. Cohn , Detecting depression severity from vocal prosody , IEEE transactions on afective computing 4 ( 2012 ) 142 - 150 .

[5] Z. Ambadar , J. F. Cohn , L. I. Reed , All smiles are not created equal: Morphology and timing of smiles perceived as amused, polite, and embarrassed/nervous , Journal of nonverbal behavior 33 ( 2009 ) 17 - 34 .

[6] D. DeVault , R. Artstein, G. Benn, T. Dey , E. Fast , A. Gainer , K. Georgila , J. Gratch , A. Hartholt , M. Lhommet , et al., Simsensei kiosk: A virtual human interviewer for healthcare decision support, in: Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems , 2014 , pp. 1061 - 1068 .

[7] D. Utami , T. Bickmore, Collaborative user responses (2020). in multiparty interaction with a couples counselor [21] F. Eyben , M. Wöllmer , B. Schuller , Opensmile: the robot , in: 2019 14th ACM/IEEE International Con - munich versatile and fast open-source audio feaference on Human-Robot Interaction (HRI), IEEE, ture extractor , in: Proceedings of the 18th ACM 2019 , pp. 294 - 303 . international conference on Multimedia, 2010 , pp.

[8] N. Ward , W. Tsukahara, Prosodic features which 1459- 1462 . cue back-channel responses in english and japanese , [22] J. W. Pennebaker , R. L. Boyd , K. Jordan , K. BlackJournal of pragmatics 32 ( 2000 ) 1177 - 1207 . burn, The development and psychometric proper-

[9] S. Benus , A. Gravano , J. B. Hirschberg , The prosody ties of LIWC2015 , Technical Report , 2015 . of backchannels in american english ( 2007 ). [23] E. Ekstedt , G. Skantze, Voice activity projec-

[10] R. Bertrand , G. Ferré, P. Blache , R. Espesser, tion: Self-supervised learning of turn-taking events, S. Rauzy, Backchannels revisited from a multimodal arXiv preprint arXiv: 2205 .09812 ( 2022 ). perspective, in: Auditory-visual Speech Processing, [24] S. Hershey , S. Chaudhuri , D. P. Ellis , J. F. Gemmeke , 2007 , pp. 1 - 5 . A. Jansen , R. C. Moore , M. Plakal , D. Platt , R. A.

[11] K. P. Truong , R. Poppe , I. de Kok, D. Heylen , A mul- Saurous , B. Seybold , et al., Cnn architectures for timodal analysis of vocal and visual backchannels large-scale audio classification, in: 2017 ieee interin spontaneous dialogs ., in: INTERSPEECH , 2011 , national conference on acoustics, speech and signal pp. 2973 - 2976 . processing (icassp), IEEE, 2017 , pp. 131 - 135 .

[12] A. Gravano , J. Hirschberg , Backchannel-inviting [25] D. Bahdanau , K. Cho , Y. Bengio, Neural machine cues in task-oriented dialogue, in: Tenth Annual translation by jointly learning to align and translate , Conference of the International Speech Communi- arXiv preprint arXiv:1409.0473 ( 2014 ). cation Association , 2009 . [26] S. Stoll , N. C. Camgöz , S. Hadfield , R. Bowden , Sign

[13] W. Wang , X. Alameda-Pineda , D. Xu , P. Fua , E. Ricci, language production using neural machine translaN . Sebe, Every smile is unique: Landmark-guided tion and generative adversarial networks, in: Prodiverse smile generation , in: Proceedings of the ceedings of the 29th British Machine Vision ConIEEE Conference on Computer Vision and Pattern ference (BMVC 2018 ), British Machine Vision AssoRecognition , 2018 , pp. 7083 - 7092 . ciation, 2018 .

[14] W. Feng , A. Kannan , G. Gkioxari, C. L. Zit- [27] C. Ahuja , D. W. Lee , R. Ishii , L.-P. Morency , No gesnick, Learn2smile: Learning non-verbal interac- tures left behind: Learning relationships between tion through observation, in: 2017 IEEE/RSJ In- spoken language and freeform gestures, in: Findternational Conference on Intelligent Robots and ings of the Association for Computational LinguisSystems (IROS) , IEEE, 2017 , pp. 4131 - 4138 . tics: EMNLP 2020 , 2020 , pp. 1884 - 1895 .

[15] E. Ng , H. Joo , L. Hu , H. Li , T. Darrell , A. Kanazawa , [28] S. Al Moubayed , J. Beskow , G. Skantze, S. Ginosar , Learning to listen: Modeling non- B. Granström, Furhat: a back-projected humandeterministic dyadic facial motion, in: Proceedings like robot head for multiparty human-machine of the IEEE/CVF Conference on Computer Vision interaction , in: Cognitive Behavioural Systems: and Pattern Recognition , 2022 , pp. 20395 - 20405 . COST 2102 International Training School , Dresden,

[16] S. Geng , R. Teotia , P. Tendulkar , S. Menon , C. Von- Germany, February 21- 26 , 2011 , Revised Selected drick, Afective faces for goal-driven dyadic com- Papers , Springer, 2012 , pp. 114 - 130 . munication, arXiv preprint arXiv:2301.10939 ( 2023 ).

[17] I. O. Ertugrul , L. A. Jeni , W. Ding , J. F. Cohn , Afar: A deep learning based tool for automated facial afect recognition , in: 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019 ), IEEE, 2019 , pp. 1 - 1 .

[18] S. Schneider , A. Baevski , R. Collobert , M. Auli , wav2vec: Unsupervised pre-training for speech recognition , arXiv preprint arXiv: 1904 . 05862 ( 2019 ).

[19] M. McAulife , M. Socolof , S. Mihuc , M. Wagner , M. Sonderegger , Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi , in: Proc. Interspeech 2017 , 2017 , pp. 498 - 502 . doi: 10 .21437/ Interspeech.2017- 1386 .

[20] S. A. Memon , Acoustic correlates of the voice qualifiers: A survey , arXiv preprint arXiv:2010.15869 10. Appendix 10.1. Distribution of Intensity and Duration of Smiles 10.2. Efect of Sex and Relationship on Smile Intensity 0.67 0.10 1.05 0.89 0.75 0.11 1.19 0.5225 0.7424 0 . 3176