CNNs and Fisher Vectors for No-Audio Multimodal Speech Detection Jose Vargas1 , Hayley Hung1 1 Delft University of Technology, Netherlands j.d.vargasquiros@tudelft.nl,h.hung@tudelft.nl ABSTRACT salient manifestation of speech behavior in our dataset. Although This paper presents the algorithms that the organisers deployed gesture recognition can certainly be treated as an action recognition for the automatic Behavior Analysis (HBA) task in MediaEval 2019, or localization problem, it has received some attention in studies consisting on the detection of speech in social interaction from that focus specifically on this task [3, 10, 15, 16]. The datasets used, body-worn acceleration and video only. For acceleration-based pre- however, differ in that they normally offer a clear frontal view of a diction, a CNN with access to a window of 3s around and including single person. the one-second prediction window is shown to perform remarkably. For video-based prediction, a Fisher vector pipeline with access 3 APPROACH only to the prediction window of 1s was found to perform signifi- The task was approached using a traditional dense trajectories cantly worse, while the late fusion of both approaches resulted in a pipeline for video-based detection. For acceleration-based detection, small improvement. a one-dimensional convolutional neural network with access to context outside of the prediction window was used. Multimodal 1 INTRODUCTION detection was approached via late fusion of classification scores. The No-Audio Multimodal Speech Detection task [5] of MediaEval 2019 aims to study the problem of determining the speaking status 3.1 Estimation from video: Dense Trajectories of standing subjects in crowded mingling scenarios. The non-verbal and Fisher Vectors input consists in accelerometer readings from a wearable devices The method for video classification was based on dense trajectories worn around the neck of the subjects, and video recorded from [13, 14] due to their relative simplicity and competitive performance overhead cameras. even when compared with more recent deep learning approaches The problem is of interest because the automatic detection of for action recognition. speech from the visual modality allows for more detailed computa- Fisher vectors [12], and specially their improved variant , [11] tional analyses of social behavior when audio of conversations is were found to perform remarkably well in comparisson with Multi- not available. The importance of the acceleration modality is two- ple Instance Learning [2] in experiments with 3-second windows, fold. First, the use of accelerometers in wearable devices poses little and were therefore chosen as classification algorithm. privacy concerns and such devices have therefore become common Fisher vectors provide a way to obtain a compact feature vector in social interaction datasets, providing limited but exploitable in- from an arbitrary number of local features by making use of the formation about the body movement of subjects. Second, being a additive property of log-likelihood in a generative model (see figure proxy for body movement, insights about how to best detect social 2). Let X = {x t , t = 1...T } be the set of T local descriptors of actions from acceleration information could potentially transfer to dimensionality D extracted from an image and u λ be the probability other modalities like video. density function with parameters λ. The fisher score is defined as the gradient of the log-likehood over X , with respect to the model 2 RELATED WORK parameters: Using the same dataset, a previous submission for the same task [1] 1 makes use of PSD feature extraction and a Transductive Parameter G λX = ∇ log u λ (X ) (1) Transfer method for classifying based on acceleration and dense T λ trajectories and a Multiple Instance Learning method for classifying where λ denotes the model parameters. The fisher vector is a the video modality. Late fusion is also used and results in an increase normalized version of the Fisher score: in performance. Both methods were proposed in separate papers for the speech detection task [2, 6]. GλX = L λ G λX (2) Research in psychology and computer science has investigated the synchrony between speech and gesture [4] and the role that where normalization by L λ corresponds to whitening of the di- gestures play in complementing or being redundant to speech [8, 9]. mensions. Any generative model can be used as u λ . We chose a Very little literature is concerned with the specific task of recog- Gaussian mixture model (GMM) with K components with diagonal nizing speaking status without access to audio. Much more concern covariance matrices, in line with previous work [11]. The param- has received the automatic detection of gestures, possibly the most eters λ of a GMM are λ = {w i , µ i , σi2 , i = 1, . . . , K }, where w i , µ i and σi2 are the mixture weight, mean vector and diagonal of the Copyright 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution covariance matrix of Gaussian i. Mean and standard deviation are 4.0 International (CC BY 4.0). the only parameters considered because mixture weights add little MediaEval’19, 27-29 October 2019, Sophia Antipolis, France MediaEval’19, 27-29 October 2019, Sophia Antipolis, France J. Vargas, E. Gedik, H. Hung additional information [11]. Under the assumption of independence 60x3 of local descriptors: T 1Õ G λX = ∇ log u λ (x t ) (3) T t =1 λ 0/1 Let γt (i) be the soft assignment of descriptor x t to Gaussian i: w i ui (x t ) 56x24 27x64 13x96 13x96 13x64 384 384 192 γt (i) = ÍK (4) j=1 w j u j (x t ) Derivation of the gradients leads to: Figure 2: Architecture of the 1D-CNN used. Input data has 3 channels corresponding to axes X, Y and X of the accelerom- T xt − µi   X 1 Õ eter. Filter sizes are 5 for the first convolutional layer and 3 Gµ,i√= γt (i) (5) T w i t =1 σi for the rest of the layers, with unit padding. As with AlexNet, first, second and last layers are followed by a max-pooling T " # X 1 Õ (x t − µ i )2 layer kernel size 3 and stride of 2. Gσ,i = √ γt (i) −1 (6) T 2w i t =1 σi2 where the division between vectors is term-by-term. The Fisher Vector aggregates all gradients into a vector of 2KD dimensions. 3.3 Multimodal estimation: late fusion Finally Fisher vectors are normalized by dividingpby their L2 norm Late fusion of the scores of both modalities was used to obtain multi- and then power-normalized with f (z) = siдn(z) |z|. modal scores, by training a logistic regressor with no regularization For the task, person videos were resized to 100x100px. A set of on the output scores of both modalities. 200 one-second windows were sampled per person, reducing the size of the training set to 10800 examples, due to the large size of 4 RESULTS AND ANALYSIS the represenation. A GMM with 256 components was used. Fisher Table 1 presents the results on the provided test set. vectors were fed into a linear SVM classifier. 4-fold cross validation at the subject level was used to determine the optimal regularization parameter. Submission Method AUC 1D CNN 0.692 This submission Bags of dense trajectories GMM Fisher Vectors SVM Fisher vectors 0.552 Fusion 0.693 TPT 0.656 i+2 Past submission [1] MILES 0.549 i+1 Fusion 0.658 i Table 1: Test results. Figure 1: Fisher vectors pipeline. 5 DISCUSSION AND OUTLOOK 3.2 Estimation from acceleration: 1-D Although the submitted results indicate much better performance Convolutional Neural Network (CNN) from the acceleration-based method, our experiments using predic- For the classification of one-second windows using acceleration, a tion windows of 3s for both methods have resulted in very similar one-dimensional CNN was chosen. The architecture was based on performance, indicating that the larger context fed into the CNN is the two-dimensional AlexNet [7]. The ratios between number of useful for prediction. The experiments made for the submission sug- channels was preserved but the number of channels was reduced gested multiple areas for possible future work. One of them relates due to the reduced complexity of the input (see figure 2). Because to how to give dense trajectory methods context in an equivalent experiments have revealed that 3-second windows are more infor- way. Giving dense-trajectory-based methods access to context for mative for the detection of speaking status, the network was fed high-resolution prediction is not straightforward given that aggre- 3-second windows to give it access to a wider context, but only the gation methods like Fisher vectors are time-agnostic, unlike a CNN middle second is predicted. The data was padded with zeros at both which only compresses its time dimension. ends. The comparison with the results of our past submission indi- A sliding window of 3s with stride of 1s was used to produce cates that Fisher Vectors are capable of outperforming MILES. Our the training examples. Data was pre-processed by z-score stan- experiments also showed that personalisation using TPT does not dardization on each axis, to reduce the effect of gravity and device deliver better results for this dataset, even when compared with a miscalibration. more simple Logistic Regressor. No-audio Multimodal Speech Detection MediaEval’19, 27-29 October 2019, Sophia Antipolis, France ACKNOWLEDGMENTS Action Recognition with Improved Trajectories. ICCV - IEEE Interna- A special acknowledgement goes to Ekin Gedik and Laura Cabrera- tional Conference on Computer Vision December (2013), 3551–3558. [15] Huogen Wang, Pichao Wang, Zhanjie Song, and Wanqing Li. 2018. Quiros for their support and input during the making of the paper. Large-scale multimodal gesture recognition using heterogeneous net- This task is supported by the Netherlands Organization for Scientific works. Proceedings - 2017 IEEE International Conference on Com- Research (NWO) under project number 639.022.606. puter Vision Workshops, ICCVW 2017 2018-Janua (2018), 3129–3137. https://doi.org/10.1109/ICCVW.2017.370 REFERENCES [16] X Zabulis, H Baltzakis, and a Argyros. 2009. Vision-based hand gesture recognition for human-computer interaction. The Universal Access . . . [1] Laura Cabrera-Quiros, Andrew Demetriou, Ekin Gedik, Leander (2009), 1–56. http://users.ics.forth.gr/ van der Meij, and Hayley Hung. 2018. The MatchNMingle dataset: a novel multi-sensor resource for the analysis of social interactions and group dynamics in-the-wild during free-standing conversations and speed dates. IEEE Transactions on Affective Computing (2018), 1–17. https://doi.org/10.1109/TAFFC.2018.2848914 [2] Laura Cabrera-Quiros, David M.J. Tax, and Hayley Hung. 2018. Ges- tures in-the-wild : detecting conversational hand gestures in crowded scenes using a multimodal fusion of bags of video trajectories and body worn acceleration. (2018), 1–10. [3] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, and Richard Bowden. 2016. Using Convolutional 3D Neural Networks for User- independent continuous gesture recognition. Proceedings - Inter- national Conference on Pattern Recognition 0 (2016), 49–54. https: //doi.org/10.1109/ICPR.2016.7899606 [4] Anna Esposito and Antonietta M. Esposito. 2011. On speech and gestures synchrony. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 6800 LNCS (2011), 252–272. https://doi.org/10.1007/ 978-3-642-25775-9_25 [5] Ekin Gedik, Laura Cabrera-Quiros, and Hayley Hung. 2019. No-Audio Multimodal Speech Detection task at MediaEval 2019. (2019). [6] Ekin Gedik and Hayley Hung. 2017. Personalised models for speech detection from body movements using transductive parameter transfer. Personal and Ubiquitous Computing 21, 4 (2017), 723–737. https://doi. org/10.1007/s00779-017-1006-4 [7] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Ima- geNet Classification with Deep Convolutional Neural Networks. NIPS Proceedings (2012). https://doi.org/10.1201/9781420010749 [8] Alex Lascarides and Matthew Stone. 2009. A formal semantic analysis of gesture. Journal of Semantics (2009). http://citeseerx.ist.psu.edu/ viewdoc/summary?doi=10.1.1.48.3741 [9] David McNeill. 1994. Hand and Mind: What Gestures Reveal About Thought. University of Chicago Press (1994). https://doi.org/10.1177/ 002383099403700208 [10] Qiguang Miao, Yunan Li, Wanli Ouyang, Zhenxin Ma, Xin Xu, Weikang Shi, and Xiaochun Cao. 2018. Multimodal Gesture Recognition Based on the ResC3D Network. Proceedings - 2017 IEEE International Confer- ence on Computer Vision Workshops, ICCVW 2017 2018-Janua (2018), 3047–3055. https://doi.org/10.1109/ICCVW.2017.360 [11] Florent Perronnin, Jorge Sanchez, and Thomas Mensink. 2010. Im- proving the Fisher Kernel for Large-Scale Image Classificatio. ECCV 2010 (2010). [12] Jorge Sánchez, Florent Perronnin, Thomas Mensink, and Jakob Ver- beek. 2013. Image classification with the fisher vector: Theory and practice. International Journal of Computer Vision 105, 3 (2013), 222– 245. https://doi.org/10.1007/s11263-013-0636-x [13] Heng Wang, Alexander Kläser, Cordelia Schmid, and Cheng Lin Liu. 2011. Action recognition by dense trajectories. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recog- nition (2011), 3169–3176. https://doi.org/10.1109/CVPR.2011.5995407 arXiv:1505.04868 [14] Heng Wang, Cordelia Schmid, Heng Wang, Cordelia Schmid, Action Recognition, Trajectories Iccv, Heng Wang, and Cordelia Schmid. 2013.