=Paper=
{{Paper
|id=Vol-2670/MediaEval_19_paper_5
|storemode=property
|title=No-Audio
Multimodal Speech Detection Task at MediaEval 2019
|pdfUrl=https://ceur-ws.org/Vol-2670/MediaEval_19_paper_5.pdf
|volume=Vol-2670
|authors=Ekin Gedik,Laura Cabrera-Quiros,Hayley Hung
|dblpUrl=https://dblp.org/rec/conf/mediaeval/GedikQH19
}}
==No-Audio
Multimodal Speech Detection Task at MediaEval 2019==
https://ceur-ws.org/Vol-2670/MediaEval_19_paper_5.pdf
No-Audio Multimodal Speech Detection task at MediaEval 2019
Ekin Gedik1 , Laura Cabrera-Quiros3,1,2 , Hayley Hung1
1 Delft University of Technology, Netherlands
2 Instituto Tecnológico de Costa Rica, Costa Rica.
3 Eindhoven University of Technology, Netherlands
{l.c.cabreraquiros,e.gedik,h.hung}@tudelft.nl
ABSTRACT which is hung around the neck. These modalities are easy to use and
This overview paper provides a description of the No-Audio multi- replicate for crowded environments such as conferences, network-
modal speech detection task for the MediaEval 2019. Same as the ing events, or organisational settings. This approach also enables
first edition that was held in 2018, the task again focuses on the esti- a more privacy-preserving method of extracting socially relevant
mation of speaking status from multimodal data. Task participants information.
are provided with cropped videos of individuals interacting freely The presence of body movements such as gesturing while speak-
during a crowded mingle event, captured by an overhead camera. ing has been well-documented by social scientists [9]. Thus, an
Each individuals tri-axial acceleration throughout the event, cap- automatic estimation system should exploit the natural human
tured with a single badge-like device hung around the neck, is also movements that accompany speech. Past work which estimated
provided. The goal of this task is to automatically estimate if a speaking status from a single body worn tri-axial accelerometer
person is speaking or not using these two alternative modalities. In [5, 6] and other work that used video to estimate speaking status
contrast to conventional speech detection approaches, no audio is during standing conversations [4] motivated this task.
used for this task. Instead, the automatic estimation system must Despite these efforts, one of the major challenges of these al-
exploit the natural human movements that accompany speech. The ternative approaches has been achieving competitive estimation
task seeks to achieve competitive estimation performance compared performance against audio-based systems. As of 2019, exploiting
to audio-based systems by exploiting the multi-modal aspects of the multi-modal aspects of the problem is still under-explored and
the problem. this is the main focus of this challenge.
1 INTRODUCTION 2 TASK DETAILS
Speaking status is one of the most essential elements of social 2.1 Unimodal estimation of speaking status
behaviour since it is one of the key behavioural cues that is used For this subtask, participants are expected to design and implement
for studying conversational dynamics in face to face settings [10]. separate speaking status estimators for each modality. If partici-
This task focuses on the automatic detection of speaking status. pants prefer to focus on developing an estimator for only one of
Previous work has shown the benefit of deriving features from the modalities, they can use the provided baseline approach for
speaking turns (which can be obtained from the speaking status the other modality. For the video modality, the algorithm will have
of different people) for estimating many different social constructs a video of a person interacting freely in a social gathering (see
such as dominance [8], or cohesion [7]. Figure 1) as input and should provide a estimation of that per-
However, automated analysis of conversational dynamics in sons’ speaking status (speaking/non-speaking) estimation every
large unstructured social gatherings is an under-explored problem second. Similarly, for the wearable modality, the method will have
despite the fact that attendance of these type of events have shown the wearable tri-axial acceleration signal of a person as input and
to be contributing factors for career and personal success [11]. The must return a speaking status estimation every second. We provide
majority of speaking status detection work focuses on utilising baseline codes for each modality. The baseline using acceleration
the audio signal mainly captured through microphones. However, implements the logistic regression approach in [5] and the video
most unstructured social gatherings such as parties or cocktail baseline employs dense trajectories and multiple instance learning,
events tend to have inherent background noise due to the nature as explained in [3].
of these events. Because of this restriction, recording audio in such
cases is challenging. For example, to collect good quality audio 2.2 Multimodal estimation of speaking status
signals, participants need to wear personal headset microphones. For this subtask teams must provide an estimation of speaking
This requires uncomfortable and intrusive equipment to be worn. status every second by exploiting both modalities together. Teams
Recording audio can also have certain negative connotations as can use any type of fusion method they see fit [1]. The goal is
it can be perceived as an invasion of privacy to have the precise to leverage the complementary nature of the modalities to better
verbal contents of a conversation to be recorded. estimate the speaking status. Thus, teams are encouraged to go
Estimating a person’s speaking status using the provided video beyond basic fusion and really think about the impact of each
and wearable acceleration data instead of audio is the main goal of modality on the estimation.
this task. The accelerometer is embedded inside a smart ID badge
3 DATA
Copyright 2019 for this paper by its authors. Use
permitted under Creative Commons License Attribution The data for this task is a subset of the MatchNMingle dataset
4.0 International (CC BY 4.0). [2], which is open to the research community. This dataset was
MediaEval’19, 27-30 October 2019, Sophia Antipolis, France
MediaEval’19, 27-30 October 2019, France E. Gedik et al.
SUBJETCS
54 16
0min
TIME
Figure 1: Alternative modalities to audio used for the task.
Train set Test set
Left: Individual video of each participant while interacting 22min
freely. Right: Wearable triaxial acceleration recorded by a 25min
device hung around the neck. Test set (optional-
subject specific)
created as a resource to analyse unstructured mingle scenarios and 30min
Entire data
seated speed dates.1 The subset for this task contains data for 70 Figure 2: Separation of train and test set for the task.
people who attended one of three separate mingle events for over 45
minutes. To eliminate the effects of acclimatisation, only 30 minutes manner. This means that the training set will not include any
in the middle of the event are used. Subjects were separated using data from the participants in the test set.
stratified sampling to create the train and test sets (see Figure 2). Optional evaluation. As an optional task, teams can also sub-
Stratification was done with various criteria to ensure balanced mit up to 5 runs (per person) using a person dependent training
distributions in both sets for speaking status, gender, event day, scheme. To do so, a separate 5 minutes interval for all people in the
and level of occlusion in the video.2 An additional segment of training set is provided, as shown by the orange section in Figure 2.
the data (orange in Figure 2) is left for the optional subject specific In this setting, samples originating from the same subject (which
evaluation (see more in Section 4). Train and test sets provided to the are temporally non-adjacent) can be used in the training in addition
participants this year are entirely same with the one used in last to data from the other subjects. This evaluation can be a sanity
year’s iteration, making comparisons possible between solutions of check as the performance of the method, in theory, should perform
different years. better when trained on a specific person rather than other people.
Task participants are provided with videos of individuals recorded
at 20FPS, captured by an overhead camera. Due to the crowded
5 DISCUSSION AND OUTLOOK
nature of the events, there can be strong occlusions between par- This task aims to investigate the use of alternative modalities for
ticipants in the video. The video for each person has been cropped the detection of speaking status. With the information gained from
from the entire frame and provided in separated videos. Note that this task, we aim to learn more about the nature of the connec-
due to the crowded nature of social gatherings, the cropped scenes tion between speaking and body movements, providing valuable
do not just capture the behaviour of the person of interest, as cross insights for both social science and the multimedia communities.
contamination between bounding boxes does occur. Moreover, we expect these insights will pave the way for solutions
Each individual is also wearing a badge-like device, recording tri- that are privacy-preserving and scalable.
axial acceleration at 20Hz. Task participants have access to the raw In its first edition in 2018, we saw that the task was received
tri-axial acceleration, for which only the effect of gravity was com- as untypical and challenging. The participation for the task was
pensated by subtracting the mean of each axis and normalising with limited and no participant managed to provide a better performance
the variance of each respective axis. All the data is synchronised. or explanation than the baseline method provided by the organ-
Finally, binary speaking status (speaking/non-speaking) was isers. Various properties make the task challenging. The chosen
annotated every frame by 3 different annotators. Inter-annotator modalities are not for directly sensing the physical manifestation of
agreement for a 2 minute segment of the data reported a Fleiss’ the task (audio). Acceleration and video provides an indirect way of
kappa coefficient of 0.55. sensing speaking and requires carefully designed approaches that
can exploit the connection between body movements and speech.
4 EVALUATION Secondly, the connection between speech and body movements
Due to class imbalance, we use the Area Under the ROC Curve has been found to be person-specific [5], further complicating the
(ROC-AUC) as the evaluation metric. Participants need to submit task. In its current edition, we aimed to increase participation by
non-binary prediction scores (posterior probabilities, distances to providing the baseline codes for each modality.
the separating hyperplane, etc.). In addition, we are reaching out to different communities (af-
The task will be evaluated using a subset of the data left as a test fective computing, multimedia, computer vision, and speech). We
set (as shown by the red section of Figure 2). All the samples of this believe each of these communities can bring their own expertise
test set will be for subjects who are not present in the training set, to the task. In the following years as well as augmenting the data,
as can be seen in Figure 2. we aim to focus on the person dependent task and its fundamental
Required evaluation. For each subtask, each team must provide differences from a person independent training setting.
up to 5 runs with their non-binary estimations for a persons’ speak-
ing status. The evaluation will be made in a person independent ACKNOWLEDGMENTS
1 MatchNMingle is openly available for research purposes under an EULA at
This task is partially supported by the Instituto Tecnológico de
http://matchmakers.ewi.tudelft.nl/matchnmingle/pmwiki/ Costa Rica and the Netherlands Organization for Scientific Research
2 Occlusion levels can be requested if needed for training set. (NWO) under project number 639.022.606.
Human Behavior Analysis Task: No-Audio Multi-Modal Speech
Detection in Crowded Social Settings MediaEval’19, 27-30 October 2019, France
REFERENCES
[1] Pradeep K Atrey, M Anwar Hossain, Abdulmotaleb El Saddik, and Mo-
han S Kankanhalli. 2010. Multimodal fusion for multimedia analysis:
a survey. Multimedia Systems 16, 6 (2010), 345–379.
[2] Laura Cabrera-Quiros, Andrew Demetriou, Ekin Gedik, Leander
van der Meij, and Hayley Hung. 2018. The MatchNMingle dataset: a
novel multi-sensor resource for the analysis of social interactions and
group dynamics in-the-wild during free-standing conversations and
speed dates. IEEE Transactions on Affective Computing (2018).
[3] Laura Cabrera-Quiros, David MJ Tax, and Hayley Hung. 2019. Gestures
in-the-wild: detecting conversational hand gestures in crowded scenes
using a multimodal fusion of bags of video trajectories and body worn
acceleration. IEEE Transactions on Multimedia (2019).
[4] Marco Cristani, Anna Pesarin, Alessandro Vinciarelli, Marco Crocco,
and Vittorio Murino. 2011. Look at whoâĂŹs talking: Voice activ-
ity detection by automated gesture analysis. In International Joint
Conference on Ambient Intelligence. Springer, 72–80.
[5] Ekin Gedik and Hayley Hung. 2017. Personalised models for speech
detection from body movements using transductive parameter transfer.
Personal and Ubiquitous Computing 21, 4 (2017), 723–737.
[6] Hayley Hung, Gwenn Englebienne, and Jeroen Kools. 2013. Classify-
ing social actions with a single accelerometer. In Proceedings of the
2013 ACM international joint conference on Pervasive and ubiquitous
computing. ACM, 207–210.
[7] Hayley Hung and Daniel Gatica-Perez. 2010. Estimating cohesion in
small groups using audio-visual nonverbal behavior. IEEE Transactions
on Multimedia 12, 6 (2010), 563–575.
[8] Dinesh Babu Jayagopi, Hayley Hung, Chuohao Yeo, and Daniel Gatica-
Perez. 2009. Modeling Dominance in Group Conversations Using
Nonverbal Activity Cues. IEEE Transactions on Audio, Speech, and
Language Processing 17, 3 (2009), 501–513.
[9] David McNeill. 2000. Language and gesture. Vol. 2. Cambridge Univer-
sity Press.
[10] Alessandro Vinciarelli, Maja Pantic, Dirk Heylen, Catherine Pelachaud,
Isabella Poggi, Francesca D’Errico, and Marc Schroeder. 2012. Bridging
the gap between social animal and unsocial machine: A survey of social
signal processing. IEEE Transactions on Affective Computing 3, 1 (2012),
69–87.
[11] Hans-Georg Wolff and Klaus Moser. 2009. Effects of networking on
career success: a longitudinal study. Journal of Applied Psychology 94,
1 (2009), 196.