We describe the \Multimodal Person Discovery in Broadcast TV" task of MediaEval 2015 benchmarking initiative. Participants were asked to return the names of people who can be both seen as well as heard in every shot of a collection of videos. The list of people was not known a priori and their names had to be discovered in an unsupervised way from media content using text overlay or speech transcripts. The task was evaluated using information retrieval metrics, based on a posteriori collaborative annotation of the test corpus.
TV archives maintained by national institutions such as the French INA, the Netherlands Institute for Sound & Vision, or the British Broadcasting Corporation are rapidly growing in size. The need for applications that make these archives searchable has led researchers to devote concerted e ort to developing technologies that create indexes.
Indexes that represent the location and identity of people in the archive are indispensable for searching archives. Human nature leads people to be very interested in other people. However, when the content is created or broadcast, it is not always possible to predict which people will be the most important to nd in the future. For this reason, it is not possible to assume that biometric models will always be available at indexing time. For some people, such a model may not be available in advance, simply because they are not (yet) famous. In such cases, it is also possible that archivists annotating content by hand do not even know the name of the person. The goal of this task is to address the challenge of indexing people in the archive, under real-world conditions (i.e. when there is no pre-set list of people to index).
Canseco et al. [
Started in 2011, the REPERE challenge aimed at
supporting research on multimodal person recognition [
Participants were provided with a collection of TV broadcast recordings pre-segmented into shots. Each shot s 2 S had to be automatically tagged with the names of people both speaking and appearing at the same time during the shot: this tagging algorithm is denoted by L : S 7! P(N ) in the rest of the paper. The main novelty of the task is that the list of persons was not provided a priori, and person biometric models (neither voice nor face) could not be trained on external data. The only way to identify a person was by nding their name n 2 N in the audio (e.g. using speech transcription { ASR) or visual (e.g. using optical character recognition { OCR) streams and associating them to the correct person. This made the task completely unsupervised (i.e. using algorithms not relying on pre-existing labels or biometric models).
Because person names were detected and transcribed automatically, they could contain transcription errors to a certain extent (more on that later in Section 5). In the following, we denote by N the set of all possible person names in the universe, correctly formatted as firstname_lastname { while N is the set of hypothesized names.
A Mr A
A A
Hello Mrs B
B B blah blah blah blah blah blah A
B B A
INPUT OUTPUT
LEGEND speech transcript text overlay speaking face evidence shot #1 shot #2 shot #3 shot #4
To ensure that participants followed this strict \no biometric supervision" constraint, each hypothesized name n 2 N had to be backed up by a carefully selected and unique shot prooving that the person actually holds this name n: we call this an evidence and denote it by E : N 7! S. In realworld conditions, this evidence would help a human annotator double-check the automatically-generated index, even for people they did not know beforehand.
Two types of evidence were allowed: an image evidence is a shot during which a person is visible, and their name is written on screen; an audio evidence is a shot during which a person is visible, and their name is pronounced at least once during a [shot start time 5s; shot end time + 5s] neighborhood. For instance, in Figure 1, shot #1 is an image evidence for Mr A (because his name and his face are visible simultaneously on screen) while shot #3 is an audio evidence for Mrs B (because her name is pronounced less than 5 seconds before or after her face is visble on screen). 3.
The REPERE corpus { distributed by ELDA { served as development set. It is composed of various TV shows (around news, politics and people) from two French TV channels, for a total of 137 hours. A subset of 50 hours is manually annotated. Audio annotations are dense and provide speech transcripts and identity-labeled speech turns. Video annotations are sparse (one image every 10 seconds) and provide overlaid text transcripts and identity-labeled face segmentation. Both speech and overlaid text transcripts are tagged with named entities. The test set { distributed by INA { contains 106 hours of video, corresponding to 172 editions of evening broadcast news \Le 20 heures" of French public channel \France 2", from January 1st 2007 to June 30st 2007.
As the test set came completely free of any annotation, it was annotated a posteriori based on participants' submissions. In the following, task groundtruths are denoted by function L : S 7! P(N) that maps each shot s to the set of names of every speaking face it contains, and function E : S 7! P(N) that maps each shot s to the set of person names for which it actually is an evidence.
This task targeted researchers from several communities including multimedia, computer vision, speech and natural language processing. Though the task was multimodal by design and necessitated expertise in various domains, the technological barriers to entry was lowered by the provision of a baseline system described in Figure 2 and available as open-source software1. For instance, a researcher from the speech processing community could focus its research efforts on improving speaker diarization and automatic speech transcription, while still being able to rely on provided face detection and tracking results to participate to the task.
The audio stream was segmented into speech turns, while
faces were detected and tracked in the visual stream. Speech
turns (resp. face tracks) were then compared and
clustered based on MFCC and the Bayesian Information
Criterion [
This information retrieval task was evaluated using a variant of Mean Average Precision (MAP), that took the quality of evidences into account. For each query q 2 Q N (firstname_lastname), the hypothesized person name nq with the highest Levenshtein ratio to the query q is selected ( : N N 7! [0; 1]) { allowing approximate name transcription: nq = arg max (q; n) and q = (q; nq)
n2N Average precision AP(q) is then computed classically based on relevant and returned shots: relevant(q) = fs 2 S j q 2 L(s)g returned(q) = fs 2 S j nq 2 L(s)gsorted by con dence Proposed evidence is Correct if name nq is close enough to the query q and if shot E(nq) actually is an evidence for q: C(q) = (1 if q > 0:95 and q 2 E(E(nq))
0 otherwise To ensure participants do provide correct evidences for every hypothesized name n 2 N , standard MAP is altered into EwMAP (Evidence-weighted Mean Average Precision), the o cial metric for the task:
EwMAP = 1
X C(q) AP(q) jQj q2Q Acknowledgment. This work was supported by the French National Agency for Research under grant ANR-12-CHRI0006-01. The open source CAMOMILE collaborative annotation platform2 was used extensively throughout the progress of the task: from the run submission script to the automated leaderboard, including a posteriori collaborative annotation of the test corpus. We thank ELDA and INA for supporting the task by distributing development and test datasets. 1http://github.com/MediaEvalPersonDiscoveryTask 2http://github.com/camomile-project