=Paper=
{{Paper
|id=Vol-2670/MediaEval_19_paper_23
|storemode=property
|title=CNNs
and Fisher Vectors for No-Audio Multimodal Speech Detection
|pdfUrl=https://ceur-ws.org/Vol-2670/MediaEval_19_paper_23.pdf
|volume=Vol-2670
|authors=Jose
Vargas,Hayley Hung
|dblpUrl=https://dblp.org/rec/conf/mediaeval/QuirosH19
}}
==CNNs
and Fisher Vectors for No-Audio Multimodal Speech Detection==
https://ceur-ws.org/Vol-2670/MediaEval_19_paper_23.pdf
CNNs and Fisher Vectors for No-Audio Multimodal Speech
Detection
Jose Vargas1 , Hayley Hung1
1 Delft University of Technology, Netherlands
j.d.vargasquiros@tudelft.nl,h.hung@tudelft.nl
ABSTRACT salient manifestation of speech behavior in our dataset. Although
This paper presents the algorithms that the organisers deployed gesture recognition can certainly be treated as an action recognition
for the automatic Behavior Analysis (HBA) task in MediaEval 2019, or localization problem, it has received some attention in studies
consisting on the detection of speech in social interaction from that focus specifically on this task [3, 10, 15, 16]. The datasets used,
body-worn acceleration and video only. For acceleration-based pre- however, differ in that they normally offer a clear frontal view of a
diction, a CNN with access to a window of 3s around and including single person.
the one-second prediction window is shown to perform remarkably.
For video-based prediction, a Fisher vector pipeline with access 3 APPROACH
only to the prediction window of 1s was found to perform signifi- The task was approached using a traditional dense trajectories
cantly worse, while the late fusion of both approaches resulted in a pipeline for video-based detection. For acceleration-based detection,
small improvement. a one-dimensional convolutional neural network with access to
context outside of the prediction window was used. Multimodal
1 INTRODUCTION detection was approached via late fusion of classification scores.
The No-Audio Multimodal Speech Detection task [5] of MediaEval
2019 aims to study the problem of determining the speaking status 3.1 Estimation from video: Dense Trajectories
of standing subjects in crowded mingling scenarios. The non-verbal and Fisher Vectors
input consists in accelerometer readings from a wearable devices The method for video classification was based on dense trajectories
worn around the neck of the subjects, and video recorded from [13, 14] due to their relative simplicity and competitive performance
overhead cameras. even when compared with more recent deep learning approaches
The problem is of interest because the automatic detection of for action recognition.
speech from the visual modality allows for more detailed computa- Fisher vectors [12], and specially their improved variant , [11]
tional analyses of social behavior when audio of conversations is were found to perform remarkably well in comparisson with Multi-
not available. The importance of the acceleration modality is two- ple Instance Learning [2] in experiments with 3-second windows,
fold. First, the use of accelerometers in wearable devices poses little and were therefore chosen as classification algorithm.
privacy concerns and such devices have therefore become common Fisher vectors provide a way to obtain a compact feature vector
in social interaction datasets, providing limited but exploitable in- from an arbitrary number of local features by making use of the
formation about the body movement of subjects. Second, being a additive property of log-likelihood in a generative model (see figure
proxy for body movement, insights about how to best detect social 2). Let X = {x t , t = 1...T } be the set of T local descriptors of
actions from acceleration information could potentially transfer to dimensionality D extracted from an image and u λ be the probability
other modalities like video. density function with parameters λ. The fisher score is defined as
the gradient of the log-likehood over X , with respect to the model
2 RELATED WORK parameters:
Using the same dataset, a previous submission for the same task [1]
1
makes use of PSD feature extraction and a Transductive Parameter G λX =
∇ log u λ (X ) (1)
Transfer method for classifying based on acceleration and dense T λ
trajectories and a Multiple Instance Learning method for classifying where λ denotes the model parameters. The fisher vector is a
the video modality. Late fusion is also used and results in an increase normalized version of the Fisher score:
in performance. Both methods were proposed in separate papers
for the speech detection task [2, 6]. GλX = L λ G λX (2)
Research in psychology and computer science has investigated
the synchrony between speech and gesture [4] and the role that where normalization by L λ corresponds to whitening of the di-
gestures play in complementing or being redundant to speech [8, 9]. mensions. Any generative model can be used as u λ . We chose a
Very little literature is concerned with the specific task of recog- Gaussian mixture model (GMM) with K components with diagonal
nizing speaking status without access to audio. Much more concern covariance matrices, in line with previous work [11]. The param-
has received the automatic detection of gestures, possibly the most eters λ of a GMM are λ = {w i , µ i , σi2 , i = 1, . . . , K }, where w i , µ i
and σi2 are the mixture weight, mean vector and diagonal of the
Copyright 2019 for this paper by its authors. Use
permitted under Creative Commons License Attribution covariance matrix of Gaussian i. Mean and standard deviation are
4.0 International (CC BY 4.0). the only parameters considered because mixture weights add little
MediaEval’19, 27-29 October 2019, Sophia Antipolis, France
MediaEval’19, 27-29 October 2019, Sophia Antipolis, France J. Vargas, E. Gedik, H. Hung
additional information [11]. Under the assumption of independence 60x3
of local descriptors:
T
1Õ
G λX = ∇ log u λ (x t ) (3)
T t =1 λ 0/1
Let γt (i) be the soft assignment of descriptor x t to Gaussian i:
w i ui (x t ) 56x24 27x64 13x96 13x96 13x64 384 384 192
γt (i) = ÍK (4)
j=1 w j u j (x t )
Derivation of the gradients leads to: Figure 2: Architecture of the 1D-CNN used. Input data has 3
channels corresponding to axes X, Y and X of the accelerom-
T
xt − µi
X 1 Õ eter. Filter sizes are 5 for the first convolutional layer and 3
Gµ,i√= γt (i) (5)
T w i t =1 σi for the rest of the layers, with unit padding. As with AlexNet,
first, second and last layers are followed by a max-pooling
T
" #
X 1 Õ (x t − µ i )2 layer kernel size 3 and stride of 2.
Gσ,i = √ γt (i) −1 (6)
T 2w i t =1 σi2
where the division between vectors is term-by-term. The Fisher
Vector aggregates all gradients into a vector of 2KD dimensions. 3.3 Multimodal estimation: late fusion
Finally Fisher vectors are normalized by dividingpby their L2 norm Late fusion of the scores of both modalities was used to obtain multi-
and then power-normalized with f (z) = siдn(z) |z|. modal scores, by training a logistic regressor with no regularization
For the task, person videos were resized to 100x100px. A set of on the output scores of both modalities.
200 one-second windows were sampled per person, reducing the
size of the training set to 10800 examples, due to the large size of 4 RESULTS AND ANALYSIS
the represenation. A GMM with 256 components was used. Fisher
Table 1 presents the results on the provided test set.
vectors were fed into a linear SVM classifier. 4-fold cross validation
at the subject level was used to determine the optimal regularization
parameter. Submission Method AUC
1D CNN 0.692
This submission
Bags of dense trajectories GMM Fisher Vectors SVM Fisher vectors 0.552
Fusion 0.693
TPT 0.656
i+2 Past submission [1]
MILES 0.549
i+1 Fusion 0.658
i Table 1: Test results.
Figure 1: Fisher vectors pipeline.
5 DISCUSSION AND OUTLOOK
3.2 Estimation from acceleration: 1-D Although the submitted results indicate much better performance
Convolutional Neural Network (CNN) from the acceleration-based method, our experiments using predic-
For the classification of one-second windows using acceleration, a tion windows of 3s for both methods have resulted in very similar
one-dimensional CNN was chosen. The architecture was based on performance, indicating that the larger context fed into the CNN is
the two-dimensional AlexNet [7]. The ratios between number of useful for prediction. The experiments made for the submission sug-
channels was preserved but the number of channels was reduced gested multiple areas for possible future work. One of them relates
due to the reduced complexity of the input (see figure 2). Because to how to give dense trajectory methods context in an equivalent
experiments have revealed that 3-second windows are more infor- way. Giving dense-trajectory-based methods access to context for
mative for the detection of speaking status, the network was fed high-resolution prediction is not straightforward given that aggre-
3-second windows to give it access to a wider context, but only the gation methods like Fisher vectors are time-agnostic, unlike a CNN
middle second is predicted. The data was padded with zeros at both which only compresses its time dimension.
ends. The comparison with the results of our past submission indi-
A sliding window of 3s with stride of 1s was used to produce cates that Fisher Vectors are capable of outperforming MILES. Our
the training examples. Data was pre-processed by z-score stan- experiments also showed that personalisation using TPT does not
dardization on each axis, to reduce the effect of gravity and device deliver better results for this dataset, even when compared with a
miscalibration. more simple Logistic Regressor.
No-audio Multimodal Speech Detection MediaEval’19, 27-29 October 2019, Sophia Antipolis, France
ACKNOWLEDGMENTS Action Recognition with Improved Trajectories. ICCV - IEEE Interna-
A special acknowledgement goes to Ekin Gedik and Laura Cabrera- tional Conference on Computer Vision December (2013), 3551–3558.
[15] Huogen Wang, Pichao Wang, Zhanjie Song, and Wanqing Li. 2018.
Quiros for their support and input during the making of the paper.
Large-scale multimodal gesture recognition using heterogeneous net-
This task is supported by the Netherlands Organization for Scientific works. Proceedings - 2017 IEEE International Conference on Com-
Research (NWO) under project number 639.022.606. puter Vision Workshops, ICCVW 2017 2018-Janua (2018), 3129–3137.
https://doi.org/10.1109/ICCVW.2017.370
REFERENCES [16] X Zabulis, H Baltzakis, and a Argyros. 2009. Vision-based hand gesture
recognition for human-computer interaction. The Universal Access . . .
[1] Laura Cabrera-Quiros, Andrew Demetriou, Ekin Gedik, Leander
(2009), 1–56. http://users.ics.forth.gr/
van der Meij, and Hayley Hung. 2018. The MatchNMingle dataset: a
novel multi-sensor resource for the analysis of social interactions and
group dynamics in-the-wild during free-standing conversations and
speed dates. IEEE Transactions on Affective Computing (2018), 1–17.
https://doi.org/10.1109/TAFFC.2018.2848914
[2] Laura Cabrera-Quiros, David M.J. Tax, and Hayley Hung. 2018. Ges-
tures in-the-wild : detecting conversational hand gestures in crowded
scenes using a multimodal fusion of bags of video trajectories and
body worn acceleration. (2018), 1–10.
[3] Necati Cihan Camgoz, Simon Hadfield, Oscar Koller, and Richard
Bowden. 2016. Using Convolutional 3D Neural Networks for User-
independent continuous gesture recognition. Proceedings - Inter-
national Conference on Pattern Recognition 0 (2016), 49–54. https:
//doi.org/10.1109/ICPR.2016.7899606
[4] Anna Esposito and Antonietta M. Esposito. 2011. On speech and
gestures synchrony. Lecture Notes in Computer Science (including
subseries Lecture Notes in Artificial Intelligence and Lecture Notes in
Bioinformatics) 6800 LNCS (2011), 252–272. https://doi.org/10.1007/
978-3-642-25775-9_25
[5] Ekin Gedik, Laura Cabrera-Quiros, and Hayley Hung. 2019. No-Audio
Multimodal Speech Detection task at MediaEval 2019. (2019).
[6] Ekin Gedik and Hayley Hung. 2017. Personalised models for speech
detection from body movements using transductive parameter transfer.
Personal and Ubiquitous Computing 21, 4 (2017), 723–737. https://doi.
org/10.1007/s00779-017-1006-4
[7] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Ima-
geNet Classification with Deep Convolutional Neural Networks. NIPS
Proceedings (2012). https://doi.org/10.1201/9781420010749
[8] Alex Lascarides and Matthew Stone. 2009. A formal semantic analysis
of gesture. Journal of Semantics (2009). http://citeseerx.ist.psu.edu/
viewdoc/summary?doi=10.1.1.48.3741
[9] David McNeill. 1994. Hand and Mind: What Gestures Reveal About
Thought. University of Chicago Press (1994). https://doi.org/10.1177/
002383099403700208
[10] Qiguang Miao, Yunan Li, Wanli Ouyang, Zhenxin Ma, Xin Xu, Weikang
Shi, and Xiaochun Cao. 2018. Multimodal Gesture Recognition Based
on the ResC3D Network. Proceedings - 2017 IEEE International Confer-
ence on Computer Vision Workshops, ICCVW 2017 2018-Janua (2018),
3047–3055. https://doi.org/10.1109/ICCVW.2017.360
[11] Florent Perronnin, Jorge Sanchez, and Thomas Mensink. 2010. Im-
proving the Fisher Kernel for Large-Scale Image Classificatio. ECCV
2010 (2010).
[12] Jorge Sánchez, Florent Perronnin, Thomas Mensink, and Jakob Ver-
beek. 2013. Image classification with the fisher vector: Theory and
practice. International Journal of Computer Vision 105, 3 (2013), 222–
245. https://doi.org/10.1007/s11263-013-0636-x
[13] Heng Wang, Alexander Kläser, Cordelia Schmid, and Cheng Lin Liu.
2011. Action recognition by dense trajectories. Proceedings of the IEEE
Computer Society Conference on Computer Vision and Pattern Recog-
nition (2011), 3169–3176. https://doi.org/10.1109/CVPR.2011.5995407
arXiv:1505.04868
[14] Heng Wang, Cordelia Schmid, Heng Wang, Cordelia Schmid, Action
Recognition, Trajectories Iccv, Heng Wang, and Cordelia Schmid. 2013.