This paper describes the algorithm tested by the LIMSI team in the MediaEval 2015 Person Discovery in Broadcast TV Task. For this task we used an audio/video diarization process constrained by names written on screen. These names are used to both identify clusters and prevent the fusion of two clusters with di erent co-occurring names. This method obtained 83.1% of EwMAP tuned on the out-domain development corpus.
We present the approach of the LIMSI team to the Person
Discovery in Broadcast TV Task at MediaEval 2015. To
address this task we had to return the names of people who
can be both seen as well as heard in a selection of shots in a
collection of videos. The list of people is not known a priori
and their names must be discovered in an unsupervised way
from media content using text overlay or speech transcripts.
For further details about the task, dataset and metrics the
reader can refer to the task description [
We rst detail the fusion system baseline provided to all participants (we are both organizer and participant). Then, we describe the constrained multi-modal clustering. Finally, we compare the results obtained.
We propose two di erent approaches to address the task. They only rely on metadata provided to all participants (see Table 1). Only written names are used as source of identity. In addition to speech turn segmentation and face detection and tracking, the baseline relies on the provided speaker diarization and speaking face mapping. The constrained clustering relies on the similarity matrices (for speaker and face) to process its own clustering.
From the written names and the speaker diarization, we
used the \Direct Speech Turn Tagging" method described
in [
Mapping x
Source of names
Written names [
x x x x x we propagate the speaker identities on the co-occurring face tracks based on the speech turns/face tracks mapping. 2.2
Figure 2 shows a global overview of our method. We rst combined the mono-modal similarity matrix and the speaking face mapping into a large multi-modal matrix using weights and to give more or less importance to a given modality. In parallel, written names are used to identify co-occurring face tracks and speech turns.
Then, we perform an agglomerative clustering on the multimodal matrix to merge all face tracks and speech turns of a same person into a unique cluster. This process is constrained by avoiding the fusion of clusters named di erently. The two parameters and advance or delay the merge of components of a modality relatively to others during the agglomerative clustering process, while the stopping criterion is chosen to maximize the target metrics (here the EwMAP). Face tracks Speech turns Face tracks Similarity
×α Written names Name co-occurring with face tracks and speech turns
Speech turns Face tracks mapping ×β
Speech turns Similarity
×(1- α) Multimodal Similarity Matrix
Multi-modal Constrained clustering
Named clusters
A complete description of this method can be found in [
The last part is common for the two fusions. For each person who speaks and appears in a shot (following the shot segmentation provided to all participants), we compute a condence score. This score is based on the temporal distance between the speaking face and its closest written name. This con dence equals to: con dence = 8 1 + d if the speaking face co-occurs < with the written name : 1= otherwise where d is the co-occurrence duration and is the duration of the gap between the face track (or speech turn) and the written name.
In Table 2, we report the EwMAP, the MAP and the Correctness (denoted by C ) obtained by the baseline and the constrained clustering tuned on an out-domain corpus (for the rst deadline: 01-jul-15) and on an in-domain corpus (second deadline: 08-jul-15).
The baseline does not take into account the similarity between face and does not bene t from the knowledge of written names during the diarization process. In addition to these 2 additional information, our second method optimizes the stopping criterion of the clustering based on the target metric (EwMAP) while the diarization of the baseline is tuned to maximize the classical DER.
For the rst deadline (July 1st) we tuned the parameters and and the stopping criterion of the clustering process Run Baseline Const. clus. 01-jul-15 Const. clus. 08-jul-15 Oracle propagation mono-show Oracle propagation cross-show on the out-domain development set. For the second deadline (July 8th), we tuned these parameters with the evaluation proposed via the leaderboard (computed every six hours on a subset of the test set). We can see only a little improvement between them, showing that our method generalizes well.
To determine the scope for further progress we used an oracle capable of recognizing a speaking face as soon as his/her written name is correctly extracted by the OCR module. In the mono-show case, the name must be written in the same video. In the cross-show case, the name can be written in any video of the corpus. Since our own approach only uses mono-show propagation, these oracle experiments show it is possible to earn up to 1% of MAP using cross-show propagation approaches.
In Table 3 we report the mean precision and recall over all queries. Compared to the baseline, the constraints on the clustering process allows to have a lower stopping criterion (therefore to have bigger clusters and hence to improve the recall), while keeping very good clusters purity (see the precision in Table 3). The high precision of our constraint clustering made the choice of the con dence score (used to rank shots in the computation of the MAP) not really important. The tuning of the three parameters on an in-domain corpus improves recall by 1.3% and decreases precision by 0.8%. In practice, was reduced for the July 8th (in-domain tuning), therefore speech turns clustering was delayed (with respect to face tracks clustering) between July 1st (out-domain) and July 8th (in-domain tuning).
This paper presented our approach and results at the MediaEval Person Discovery in Broadcast TV task. The process used an audio/video diarization constrained by written names on screen. This source of identities is used to both identify clusters and avoid wrong merges during the agglomerative clustering process.
For future works we will improve the distance between speech turns, try other clustering methods and cross-show propagation.
Acknowledgment. This work was supported by the French National Agency for Research under grant ANR-12-CHRI0006-01 (CAMOMILE project).