Modeling dyadic conversation between speaker and listener is the technology that involves spatial/temporal facial motion and language understanding. This has been a key area of interest for building complete conversational artificial intelligence systems. In this work, we propose a generic approach to learn listening styles from multiple listeners and enable facial animations to mimic the listening behavior of conversational avatars. Unlike existing methods which learns one model per person, and cannot generate the listening style of new speakers, our work allows designers to generate new listening styles without requiring any listener data. Furthermore, it is able to generate diferent listening styles and gives a unique facial expressions and head movements to the listener. Instead of deploying diferent models for diferent listeners at runtime, our approach deploys a single model that can generalize to new listeners to generate nonverbal facial responses.
learn listening styles from multiple listeners and enable Conversation with Artificial intelligence is still a long facial animations to mimic listening behavior of converway away from how a regular conversation between two sational avatars. Unlike existing methods that learn one humans looks like. Art of expressing and listening is model per person, and cannot generalize to new listenwhat makes or breaks a high quality conversation, and ing styles, our work allows designers to generate new in a multi-turn human-to-human conversation, listener listening styles without needing any listener data. Adbehavior is crucial. In most use cases, the avatar acting as ditionally, it is able to generate diferent listening styles listener can interact with the users both verbally as well and produce unique facial expressions and head motions. as nonverbally. During the verbal response, the avatar We first develop a style representation (embedding) for talks and its facial expression can be synthesized using each listener, and once the style embeddings are availconversational face and gesture models [1, 2, 3], as well able, we sample new listening styles from this embedded as the audio-driven expression models [4, 5]. The non- space to generate facial expressions and head motions verbal response is nondeterministic and hard to model. for diferent listeners. The key highlights of our work There have been a few approaches [6, 7] that have at- are as follows: tempted to model dyadic conversations. Conversations between people for a similar dialogue can be very diferent for diferent listeners just because every person has a unique style (e.g. expressions, head motions) to listen to a conversation. Hence, we need to develop models that can express this back-and-forth of nonverbal facial expressions, eye gaze and head motions during dyadic conversation. Only then, we can enable more natural interactions, increase engagement and build a harmonic user experience for the product that deploys this interactive avatar. • Instead of deploying diferent models for diferent listeners at runtime, our approach deploys a single model that can be generalized to new listeners to generate listener facial response; • Our approach allows users to choose their own listening style or sample the new listening style for conversational avatars, enabling the fine-tuning of the avatars’ behavior for specific applications; • We also propose a novel style transfer aware training techniques to improve the performance of the dyadic facial motion generation pipeline, where the style representation, and the resulting facial motion generation model are trained simultaneously.
Previous works focus on data-driven methods that can predict the 2D motion of a person as a function of the motion of the other person(s) he/she is conversing with [8, 9]. There are some other works that simplify the task of motion generation to predicting head nods [10] or estimating head pose [11]. In contrast, in addition to [12, 13], some recent works [6, 7] capture the natural complexity of interactions by considering the full range of facial expressions and head rotations. While [6] uses a Glow-based model and ingests the full temporal context of listener audio to predict the listener facial motion, [7] proposes a transformer-based predictor and an autoregressive vector-quantized variational auto-encoder to predict listener facial motion, given past motion and the current speaker audio and facial motion. While [7] does not require the listener audio which can facilitate realtime and synchronized listener motion, it generates a single model per listener ID, and hence, is not generalizable to new/diferent listeners during runtime. We propose a novel listening style-aware dyadic facial motion generation framework, that can encode the listening style of any person, and generate facial motion corresponding to the encoded style for a particular listener audio and motion. We use the large-scale dyadic facial dataset released by [7] to evaluate our approach. 4. One Model Per Style: Existing works require diferent models to generate the facial motion corresponding to each style, which is impractical to generalize to new styles during runtime. It is also not feasible to deploy a large number of models to generate a wide range of listening styles in resource-constrained edge devices. 5. Quantification of Motion Realism : Quantifying the facial motion realism is a hard problem, as there is no well-defined ground truth (The ground truth listener 3. Key Challenges response from the dataset is unique to a particular style, and might not be the optimal response.) unlike image We identify the main challenges in a robust dyadic facial classification/detection tasks. Naive metrics, such as L2 motion generation pipeline as follows: diference between the predicted and ground truth ex1. Indeterministic Listener Response: Modeling non- pression and pose parameters, might not capture realistic verbal feedback during dyadic interaction is a dificult listener-speaker interaction in the wild. This motivates problem, as listener responses are nondeterministic in the use of other metrics, such as Frechet Inception Disnature. This requires the use of probabilistic/generative tance [14]. models, such as GAN, generative model with VQ-VAE, etc., which have been shown to not generalize well to a wide range of diverse, realistic, and indeterministic lis- 4. Methodology tening responses during runtime. 2. Multimodal Problem: In a dyadic conversational To address these challenges and generate a nonverbal setting, speakers are inherently multimodal, as they com- response with facial motion and head movement, we promunicate both verbally via speech, and nonverbally via pose a novel framework that extracts two diferent emface and body motion. This requires the near-perfect beddings: the listening style embedding and motion emalignment of these two modalities to accurately capture bedding. Our framework utilizes the popular encoder and the speaker-listener interaction. decoder architecture and includes a multi-modality pre3. Training Complexity & Instability: The nondeter- dictor that can generate preliminary results and retrieve ministic listener motion generation requires the use of the final result from pre-trained latent embeddings. By generative models such as VQ-VAE, while the multimodal decoding these embeddings, we can generate the avatar’s speaker inputs require the use of deep feature extractors, responses that reflect the inherence of the training space. such as cross-modal transformers. This requires the embeddings obtained from the two models to be aligned and 4.1. Problem Formulation fused accurately for realistic facial motion generation, which complicates the training process.
4.3. Proposed Model generate the nonverbal response. The facial expression and head movement can be extracted with DECA [15] from the speaker’s images. We denote the facial expression input as () and head movement as ℎ() at time . The face motion is the combination of the expression and head movement () = {(), ℎ()}. The input audio sequence is denoted as () and the listening style is denoted as (). () can be trained ofline or together with the predictor. If () is trained ofline, the () consists of the specific style. If the listener’s past steps are considered, the predictor can predict the avatar’s response at time ′ with the input audio, motion and avatar’s previous motion:
design a model to generate the natural and realistic nonverbal response to the users (speakers). Therefore, we propose a listening style-aware dyadic facial motion generation framework with a predictor, a style extractor and a motion extractor, as illustrated in Figure 2.
The predictor consists of an encoder and decoder pair, which can be either a Transformer [16], or U-Net [17] based network. The predictor generates the avatar’s nonverbal response to the input audio and facial motions.
The style extractor generates the listening style embedding for the predictor. The style embeddings can be (′) = ((′), (′), (′ − : ′), (′)) (1) fused to the predictor in two ways. We can either extract the listening style embeddings ofline, and fuse the embedding corresponding to the desired listening style in 4.2. Listening Style Analysis the model during runtime. Alternatively, we can train the The dataset used in [7] extracts facial features and motion style extractor model end-to-end with motion prediction for 72 hours of video from 6 diferent Youtube channels pipeline with similar loss, where the style embeddings are by using DECA [15], where each channel features a par- fused directly with the multi-modality encoder output. ticular host and several interviewees from a variety of The first approach can generate new listening styles durbackgrounds. Our goal is to extract the listening style ing runtime, and the second approach can better capture (facial expressions and head pose parameters) of the 6 the diferent listening styles with a single model. hosts. In order to visualize the listening style of each host, The motion extractor is designed to generate the mowe generate their t-SNE embeddings from the training tion latent embedding while reducing the dimensionality. data, as illustrated in Figure 1. The embeddings indicate VQ-VAE [18] is firstly used in the image generation model that the listening styles can probably be distinguished and is being used in [7, 19] for motion generation. The using first order moments, such as mean of the facial ex- VQ-VAE learns a discrete codebook with multiple vectors pressions and head pose for each listener ID in the entire to quantize the latent space. Each of the vectors can be dataset. However, we see significant overlaps in some looked as an embedding of the input motion. listening styles, such as listener 2 and 4 in Figure 1, and hence, we need to design a model to extract an accurate listener representation style. 4.3.1. Style Extractor of the weights connecting each listening style in the output layer is determined to represent the listening style We design a 6-layer 1-D temporal convolutional neural embedding corresponding to that style. network, where each convolutional layer has a stride of With the style extractor, we can generate diverse lis2 in the temporal dimension. Given 64 frames per input, tening styles by sampling from the embedding space. For the model progressively reduces the temporal dimension example, we can assume the listening styles to be Gausto 1 in 6 layers. We increase the number of channels from sian, independently and identically distributed, where the 56 (53 expression parameters and 3 head pose parameters) mean and standard deviation of the expression/pose pain the input layer to 256, followed by a fully-connected rameters are computed empirically from the style embed(FC) classifier layer consisting of 6 neurons to predict the ding space. Though we have limited number of listening listen ID. Our proposed architecture, and the correspond- styles, they are quite diverse as can be visualized from the ing style embedding output is illustrated in the Figure 3. Figure 1, and with more listener data, our approach can We employ cross-entropy loss ℒℰ on the ground truth be further improved to generate more listening styles. listener identity distribution ().
ℒℰ = ∑︁ − 1
∑︁ ∈ | ()| ∈ ())
ℒℰ = E∼ [− ()]
In addition, we employ the supervised contrastive loss [20, 21] to achieve better performance with limited amount of data: (2) , ∑︀
( · / ) ∈() ( · / )
(3) where () = { ∈ () : ̃︀ = ̃︀} is the set of indices of all positives in the multiview batch distinct from , and | ()| is its cardinality. is a scalar temperature parameter. , and are the embeddings, which are generated by representation learning, of anchor, positive andWneetgraatiinvethsaemcpolnetsraresstipveectliovseslyw.ith a meta-learning ℒ = ℒ + ℒℰ (4) [22, 23] setup, for which we emulate by leaving out a At inference step, the predictor predicts multinomial disparticular listening style, while training with the − tribution of future facial motion, and by which, we re1 listening style data, where is the total number of trieve the closest quantized embeddings from the codeavailable listener identities in the dataset. book generated by motion extractor, and send them to
The contrastive supervision aims to learn the style the motion decoder to generate the avatar’s facial motion. embedding corresponding to the new listener identity better than the traditional cross-entropy loss, when the 4.3.3. Motion Extractor number of available samples is limited.
To obtain the listening style embedding, we extract The motion extractor is implemented with VQ-VAE the weights of the fully connected (FC) layer. The subset to learn a disrcete codebook = (1, · · · , ) ∈ The motion predictor is an encoder-decoder architecture, and we use a Transformer network to better capture the multi-modality feature of the users. The style embedding, previous motion and the multi-modality embedding are concatenated and fed to the decoder to predict the facial motion. When training this model with style extractor end-to-end, we follow the teacher-forcing scheme and train the model with both motion predictor loss ℒ (same as Equ. 2) on codebook index and the style extractor loss ℒℰ (Equ. 2 and 3) on style ID. The overall loss for the end-to-end training will be: R256× to quantize the latent space of the motion. Given an input of sequence with length of facial motion 1: ∈ R56× , the encoder will convert it into an embedding 1: = (1, · · · , ) ∈ R256× , where = , and is the temporal window size. Then the embedding will be mapped to the nearest code in the corresponding codebook: ′ = m∈in ‖ − ‖ ∈ R256 (5) layers of kernel size 5, stride 1, padding 2. Each convolutional layer is followed by a max pooling operation. We pass this bottlenecked sequence through a Transformer of 512 hidden layers, 8 attention heads, 12 attention layers. We train the VQ-VAE on sequences of length 32 for 1000 epochs ( 1 day on 4 V100 GPUs) with a learning rate of 2.0 with 4,000 warm-up steps. We optimize using Adam with a batch size of 32. Train/val/test split is 70/20/10. We then use the frozen model downstream to quantize the listener inputs to the Predictor.
And we get the quantized features 1: = (1, ..., ) ∈ R256× .
The decoder takes these quantized features to recon- coder consist of linear layers and a Transformer encoder struct the input motion. The encoder and decoder can be takes the raw motion representation as input [16]. We trained simultaneously with the loss function: feed the audio and the motion independently through a linear layer for each modality to obtain their respecℒ = ℒ(, ̂︁) tive projected embeddings, then send them to the Trans+ ‖[] − ‖ (6) former encoder for cross attention. The Transformer encoder is composed of 1024 hidden layers , 8 attention + ‖ − []‖ , heads and 12 attention layers. Following the transformer where ℒ is the MSE loss for the reconstruction, [· ] there are 3 convolutional layers of kernel size 5, stride is a stop gradient operation [18] to calculate the codebook 1, padding 2. Each convolutional layer is followed by a loss, and ‖ − []‖ is the “commitment" loss with the max pooling operation that temporally downsamples the tradeof coeficient [24, 19]. speaker embedding of length 32 to match the size of the listener embedding.
We conduct extensive experiments to demonstrate the capability of our proposed framework. We implement the framework based on the learning to listen model [7] and evaluate our model on the same dataset. We employ the same metrics to compare with the person agnostic model in [7], which is taken as the baseline. Our method shows significant improvement with both the L2 and the Frechet Inception Distance (FID) [14] metrics. 5.1. Experiment Details Multi-Modality Decoder The multi-modality decoder is composed of a Transformer decoder with hidden size 200, number of heads 10, and number of layers 5. We concatenate the output of the multi-modality embedding with the listener embedding and previous motion to get a sequence, which serves as input to the multi-modality decoder. During training we take the first 4 indices of the output and discard the remainder. We train the predictor for 1000 epochs ( 12 hours on 8 GPUs) with a learning rate of 0.01 with 4,000 warm-up steps.
Baseline (L2L) Data We use the dataset released by [7], which con- FID L2 tains facial features and head motion for 72 hours of (*1e3) E47rr.0o4r video from 6 diferent Youtube channels by using DECA 47.28 [15], where each channel features a particular host and 33.5 4477..2313 22.2 20.3 several interviewees from a variety of backgrounds. The 47.13 corresponding audio melspectrogram features are also Table 1 generated by audio processing library, librosa [25]. The Experiment results. Comparison to the baseline[7]. Ours-1 original videos contain the views of both the host and (Train the style representation separately with cross entropy the guest in a split-screen format. The irrelevant seg- loss) improves the FID by 25% to 24. Ours-2 (Train the style ments have been removed and only the segments that representation separately with contrastive loss) improves the contain the hosts’ nonverbal response are kept to extract FID by 30% to 22. Ours-3 (Train the style representation model the pseudo-ground truth. end-to-end) further improves the FID to 20.3. Style-Aware
(Ours-1) FID L2 (*1e3) Error 43.22 43.43 24.7 43.45 43.37 43.20
Style-Aware
(Ours-2) FID L2 (*1e3) Error 42.36 42.44 42.28 42.04 42.19
Style-Aware
(Ours-3) FID L2 (*1e3) Error 40.28 40.18 40.50 40.53 40.41
22.2
FID 28.4 25.1 24.2
where (· ) is the trace of the matrix, i.e. the sum of elements on the main diagonal.
(denoted as Ours-2), improves the FID of the listeneragnostic model by 25% to 24, and 30% to 22 respec• L2: L2 distance between ground truth and the tively. Training the style representation model end-togenerated facial parameter ˆ. Here, the ground end where it is co-optimized with the main model (detruth is extracted from listener in the video. noted as Ours-3) further improves the FID to 20.3. Since our model leads to probabilistic output for realistic lis2 = ‖ − ˆ‖ (7) tener facial motion, we sample the L2 error of each model
5 times. • Frechet Inception Distance: FID is a standard met- We also render the prediction result of diferent listenric for assessing the quality of generative models. ing style from the ours-2 method to visually compare the For two multidimensional Gaussian distributions result as shown in Figure 4. To facilitate the visual com (, Σ) and ( ′, Σ ′), it is explicitly solvable parison, we sample images from the generated sequences as: at intervals of 40 frames. Each row of the resulting image ( (, Σ) , ( ′, Σ ′))2 = ⃦⃦ − ′⃦⃦ 22 sequence represents a distinct listening style, with ac+ (Σ + Σ ′ − 2(Σ 21 · Σ ′ · Σ 21 ) 21 ), (8) tcoo mthpeasnayminegtfaalckiianlgexvpidreeoss.iFornoamndthheecaodmppoaserisroesnpwonedcinang see that the proposed method is capable of generating diverse nonverbal response with diferent listening style. 5.3. Exploratory Study 5.2. Experiment Results We also perform exploratory studies to better demonstrate the generalizability of our model with the conWe conduct extensive experiment and generate predic- trastive loss to learn the listening style, especially when tions for diferent listening styles. The metrics compar- the data is not suficient. Specifically, we hold out the lisison is shown in Table 1. Our style-aware model that tener data of one host to train our style extractor model, trains the style representation separately with cross en- and extract the mean of the final convolutional layer outtropy loss (denoted as Ours-1) and with contrastive loss put of the trained model for the particular listener data in the test set, and use this mean as the listener embedding input to our predictor model. And we train our motion extraction model without the particular listener data. We observed a drop of 80% to 39.8 in the FID by removing the complete data as shown in Table 2 for the listening data of that host. However, we notice that even by using only 5% of that host’s data, we were able to recover the FID to 28.4. This was mainly because of the supervised contrastive loss instead of the traditional cross-entropy loss that helps to generalize our model.
In this paper, we propose a novel framework and training technique to develop a dyadic facial motion generation pipeline. The goal is to generate the accurate, realistic, and diverse avatar responses for a particular speaker audio and facial motion. We use the open-source, inthe-wild, dyadic conversational dataset released in [7] to evaluate our approach. We conduct extensive experiment to generate the facial response, compare the metrics and visualize the responses. Our method significantly improves the L2 error and FID between the predicted and ground truth facial motion, compared to the listeneragnostic model proposed in [7]. Unlike any other existing approach, our approach can also generate listening response corresponding to a wide range of styles during runtime by sampling in the style embedding space. We will experiment with more modalities and deploy our work to the conversational AI system. tation learning, in: CVPR, 2020. [25] M. Brian, C. Rafel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, O. Nietou, librosa: Audio and music signal analysis in python, in: Proc. Python in Science Conference, 2015.