This paper presents the algorithms that the organisers deployed for the automatic Behavior Analysis (HBA) task in MediaEval 2019, consisting on the detection of speech in social interaction from body-worn acceleration and video only. For acceleration-based prediction, a CNN with access to a window of 3s around and including the one-second prediction window is shown to perform remarkably. For video-based prediction, a Fisher vector pipeline with access only to the prediction window of 1s was found to perform significantly worse, while the late fusion of both approaches resulted in a small improvement.
The No-Audio Multimodal Speech Detection task [
The problem is of interest because the automatic detection of speech from the visual modality allows for more detailed computational analyses of social behavior when audio of conversations is not available. The importance of the acceleration modality is twofold. First, the use of accelerometers in wearable devices poses little privacy concerns and such devices have therefore become common in social interaction datasets, providing limited but exploitable information about the body movement of subjects. Second, being a proxy for body movement, insights about how to best detect social actions from acceleration information could potentially transfer to other modalities like video.
Using the same dataset, a previous submission for the same task [
Research in psychology and computer science has investigated
the synchrony between speech and gesture [
Very little literature is concerned with the specific task of
recognizing speaking status without access to audio. Much more concern
has received the automatic detection of gestures, possibly the most
salient manifestation of speech behavior in our dataset. Although
gesture recognition can certainly be treated as an action recognition
or localization problem, it has received some attention in studies
that focus specifically on this task [
The task was approached using a traditional dense trajectories pipeline for video-based detection. For acceleration-based detection, a one-dimensional convolutional neural network with access to context outside of the prediction window was used. Multimodal detection was approached via late fusion of classification scores. 3.1
The method for video classification was based on dense trajectories
[
Fisher vectors [
Fisher vectors provide a way to obtain a compact feature vector from an arbitrary number of local features by making use of the additive property of log-likelihood in a generative model (see figure 2). Let X = {xt , t = 1...T } be the set of T local descriptors of dimensionality D extracted from an image and uλ be the probability density function with parameters λ. The fisher score is defined as the gradient of the log-likehood over X , with respect to the model parameters: (1) (2) 1 GλX = T ∇λ log uλ (X )
GλX = LλGλX where λ denotes the model parameters. The fisher vector is a normalized version of the Fisher score:
where normalization by Lλ corresponds to whitening of the
dimensions. Any generative model can be used as uλ . We chose a
Gaussian mixture model (GMM) with K components with diagonal
covariance matrices, in line with previous work [
∇λ log uλ (xt )
T t =1 Let γt (i) be the soft assignment of descriptor xt to Gaussian i: γt (i) = ÍK
j=1 wjuj (xt ) Derivation of the gradients leads to:
wiui (xt )
X Gµ ,i = 1
T Õ T √wi t =1 γt (i) " xt − µ i σi #
GσX,i = T √12wi ÕtT=1 γt (i) (xt σ−i2µ i )2 − 1 (6) where the division between vectors is term-by-term. The Fisher Vector aggregates all gradients into a vector of 2KD dimensions. Finally Fisher vectors are normalized by dividing by their L2 norm and then power-normalized with f (z) = siдn(z)p|z |.
For the task, person videos were resized to 100x100px. A set of
200 one-second windows were sampled per person, reducing the
size of the training set to 10800 examples, due to the large size of
the represenation. A GMM with 256 components was used. Fisher
vectors were fed into a linear SVM classifier. 4-fold cross validation
at the subject level was used to determine the optimal regularization
parameter.
(3)
(4)
(5)
For the classification of one-second windows using acceleration, a
one-dimensional CNN was chosen. The architecture was based on
the two-dimensional AlexNet [
A sliding window of 3s with stride of 1s was used to produce the training examples. Data was pre-processed by z-score standardization on each axis, to reduce the efect of gravity and device miscalibration. Although the submitted results indicate much better performance from the acceleration-based method, our experiments using prediction windows of 3s for both methods have resulted in very similar performance, indicating that the larger context fed into the CNN is useful for prediction. The experiments made for the submission suggested multiple areas for possible future work. One of them relates to how to give dense trajectory methods context in an equivalent way. Giving dense-trajectory-based methods access to context for high-resolution prediction is not straightforward given that aggregation methods like Fisher vectors are time-agnostic, unlike a CNN which only compresses its time dimension.
The comparison with the results of our past submission indicates that Fisher Vectors are capable of outperforming MILES. Our experiments also showed that personalisation using TPT does not deliver better results for this dataset, even when compared with a more simple Logistic Regressor.
A special acknowledgement goes to Ekin Gedik and Laura CabreraQuiros for their support and input during the making of the paper. This task is supported by the Netherlands Organization for Scientific Research (NWO) under project number 639.022.606.