INTRODUCTION

The No-Audio Multimodal Speech Detection task [5] of MediaEval 2019 aims to study the problem of determining the speaking status of standing subjects in crowded mingling scenarios. The non-verbal input consists in accelerometer readings from a wearable devices worn around the neck of the subjects, and video recorded from overhead cameras.

The problem is of interest because the automatic detection of speech from the visual modality allows for more detailed computational analyses of social behavior when audio of conversations is not available. The importance of the acceleration modality is twofold. First, the use of accelerometers in wearable devices poses little privacy concerns and such devices have therefore become common in social interaction datasets, providing limited but exploitable information about the body movement of subjects. Second, being a proxy for body movement, insights about how to best detect social actions from acceleration information could potentially transfer to other modalities like video.

RELATED WORK

Using the same dataset, a previous submission for the same task [1] makes use of PSD feature extraction and a Transductive Parameter Transfer method for classifying based on acceleration and dense trajectories and a Multiple Instance Learning method for classifying the video modality. Late fusion is also used and results in an increase in performance. Both methods were proposed in separate papers for the speech detection task [2,6].

Research in psychology and computer science has investigated the synchrony between speech and gesture [4] and the role that gestures play in complementing or being redundant to speech [8,9].

Very little literature is concerned with the specific task of recognizing speaking status without access to audio. Much more concern has received the automatic detection of gestures, possibly the most

APPROACH

The task was approached using a traditional dense trajectories pipeline for video-based detection. For acceleration-based detection, a one-dimensional convolutional neural network with access to context outside of the prediction window was used. Multimodal detection was approached via late fusion of classification scores.

Estimation from video: Dense Trajectories and Fisher Vectors

The method for video classification was based on dense trajectories [13,14] due to their relative simplicity and competitive performance even when compared with more recent deep learning approaches for action recognition.

Fisher vectors [12], and specially their improved variant , [11] were found to perform remarkably well in comparisson with Multiple Instance Learning [2] in experiments with 3-second windows, and were therefore chosen as classification algorithm.

Fisher vectors provide a way to obtain a compact feature vector from an arbitrary number of local features by making use of the additive property of log-likelihood in a generative model (see figure 2). Let X = {x t , t = 1...T } be the set of T local descriptors of dimensionality D extracted from an image and u λ be the probability density function with parameters λ. The fisher score is defined as the gradient of the log-likehood over X , with respect to the model parameters:

G X λ = 1 T ∇ λ log u λ (X )(1)

where λ denotes the model parameters. The fisher vector is a normalized version of the Fisher score:

G X λ = L λ G X λ (2)

where normalization by L λ corresponds to whitening of the dimensions. Any generative model can be used as u λ . We chose a Gaussian mixture model (GMM) with K components with diagonal covariance matrices, in line with previous work [11]. The parameters λ of a GMM are λ = {w i , µ i , σ 2 i , i = 1, . . . , K }, where w i , µ i and σ 2 i are the mixture weight, mean vector and diagonal of the covariance matrix of Gaussian i. Mean and standard deviation are the only parameters considered because mixture weights add little additional information [11]. Under the assumption of independence of local descriptors:

G X λ = 1 T T t =1 ∇ λ log u λ (x t )(3)

Let γ t (i) be the soft assignment of descriptor x t to Gaussian i:

γ t (i) = w i u i (x t ) K j=1 w j u j (x t )(4)

Derivation of the gradients leads to:

G X µ,i = 1 T √ w i T t =1 γ t (i) x t − µ i σ i (5) G X σ,i = 1 T √ 2w i T t =1 γ t (i) (x t − µ i ) 2 σ 2 i − 1 (6)

where the division between vectors is term-by-term. The Fisher Vector aggregates all gradients into a vector of 2KD dimensions. Finally Fisher vectors are normalized by dividing by their L2 norm and then power-normalized with f (z) = siдn(z) |z|.

For the task, person videos were resized to 100x100px. A set of 200 one-second windows were sampled per person, reducing the size of the training set to 10800 examples, due to the large size of the represenation. A GMM with 256 components was used. Fisher vectors were fed into a linear SVM classifier. 4-fold cross validation at the subject level was used to determine the optimal regularization parameter.

GMM SVM Bags of dense trajectories

Fisher Vectors

Estimation from acceleration: 1-D Convolutional Neural Network (CNN)

For the classification of one-second windows using acceleration, a one-dimensional CNN was chosen. The architecture was based on the two-dimensional AlexNet [7]. The ratios between number of channels was preserved but the number of channels was reduced due to the reduced complexity of the input (see figure 2). Because experiments have revealed that 3-second windows are more informative for the detection of speaking status, the network was fed 3-second windows to give it access to a wider context, but only the middle second is predicted. The data was padded with zeros at both ends.

A sliding window of 3s with stride of 1s was used to produce the training examples. Data was pre-processed by z-score standardization on each axis, to reduce the effect of gravity and device miscalibration. for the rest of the layers, with unit padding. As with AlexNet, first, second and last layers are followed by a max-pooling layer kernel size 3 and stride of 2.

Multimodal estimation: late fusion

Late fusion of the scores of both modalities was used to obtain multimodal scores, by training a logistic regressor with no regularization on the output scores of both modalities.

RESULTS AND ANALYSIS

DISCUSSION AND OUTLOOK

Although the submitted results indicate much better performance from the acceleration-based method, our experiments using prediction windows of 3s for both methods have resulted in very similar performance, indicating that the larger context fed into the CNN is useful for prediction. The experiments made for the submission suggested multiple areas for possible future work. One of them relates to how to give dense trajectory methods context in an equivalent way. Giving dense-trajectory-based methods access to context for high-resolution prediction is not straightforward given that aggregation methods like Fisher vectors are time-agnostic, unlike a CNN which only compresses its time dimension.

The comparison with the results of our past submission indicates that Fisher Vectors are capable of outperforming MILES. Our experiments also showed that personalisation using TPT does not deliver better results for this dataset, even when compared with a more simple Logistic Regressor.