We describe the \Multimodal Person Discovery in Broadcast TV" task of MediaEval 2016 benchmarking initiative. Participants are asked to return the names of people who can be both seen as well as heard in every shot of a collection of videos. The list of people is not known a priori and their names has to be discovered in an unsupervised way from media content using text overlay or speech transcripts for the primary runs. The task is evaluated using information retrieval metrics, based on a posteriori collaborative annotation of the test corpus.
TV archives maintained by national institutions such as the French INA, the Netherlands Institute for Sound & Vision, or the British Broadcasting Corporation are rapidly growing in size. The need for applications that make these archives searchable has led researchers to devote concerted e ort to developing technologies that create indexes.
Indexes that represent the location and identity of people in the archive are indispensable for searching archives. Human nature leads people to be very interested in other people. However, when the content is created or broadcasted, it is not always possible to predict which people will be the most important to nd in the future and biometric models may not yet be available at indexing time The goal of this task is thus to address the challenge of indexing people in the archive under real-world conditions, i.e. when there is no pre-set list of people to index.
Started in 2011, the REPERE challenge aimed at
supporting research on multimodal person recognition [
Participants are provided with a collection of TV broadcast recordings pre-segmented into shots. Each shot s 2 S has to be automatically tagged with the names of people both speaking and appearing at the same time during the shot.
As last year, the list of persons is not provided a priori, and person biometric models (neither voice nor face) can not be trained on external data in the primary runs. The only way to identify a person is by nding their name n 2 N in the audio (e.g., using speech transcription { ASR) or visual (e.g., using optical character recognition { OCR) streams and associating them to the correct person. This makes the task completely unsupervised (i.e. using algorithms not relying on pre-existing labels or biometric models). The main novelty of this year task is that participants may use their contrastive run to try brave new ideas that may rely on any external data, including textual metadata provided with the test set.
Because person names are detected and transcribed automatically, they may contain transcription errors to a certain extent (more on that later in Section 5). In the following, we denote by N the set of all possible person names in the universe, correctly formatted as firstname_lastname { while N is the set of hypothesized names.
A Mr A
A A
Hello Mrs B
B B blah blah blah blah blah blah A
B B A
INPUT OUTPUT
LEGEND speech transcript text overlay speaking face evidence shot #1 shot #2 shot #3 shot #4 3.
The 2015 test corpus serves as development set for this year's task. It contains 106 hours of video, corresponding to 172 editions of evening broadcast news \Le 20 heures" of the French public channel \France 2", from January 1st 2007 to June 30st 2007. This development set is associated with a posteriori annotations based on last year participants' submissions.
The test set is divided into three datasets: INA, DW and
3-24. The INA dataset contains a full week of broadcast
for 3 TV channels and 3 radio channels in French. Only a
subset (made of 2 TV video channels for a total duration
of 90 hours) needs to be processed. However, participants
can process the rest of it if they think it might lead to
improved results. Moreover, this dataset is associated with
manual metadata provided by INA in the shape of CSV les.
The DW dataset [
As the test set comes completely free of any annotation, it will be annotated a posteriori based on participants' submissions. In order to ease this annotation process, participants are asked to justify their assertion. To this end, each hypothesized name n 2 N has to be backed up by a carefully selected and unique shot prooving that the person actually holds this name n: we call this an evidence. In real-world conditions, this evidence would help a human annotator double-check the automatically-generated index, even for people they did not know beforehand.
Two types of evidence are allowed: an image evidence is a time in a video when a person is visible, while his/her name is written on screen; an audio evidence is the time when the name of a person is pronounced, provided that this person is visible in a [time 5s; time+5s] neighborhood. For instance, in Figure 1, shot #1 contains an image evidence for Mr A (because his name and his face are visible simultaneously on screen) while shot #3 contains an audio evidence for Mrs B (because her name is pronounced less than 5 seconds before or after her face is visible on screen).
This task targets researchers from several communities including multimedia, computer vision, speech and natural language processing. Though the task is multimodal by design and necessitates expertise in various domains, the technological barriers to entry is lowered by the provision of a baseline system available partially as open-source software.
For instance, a researcher from the speech processing
community can focus its research e orts on improving speaker
diarization and automatic speech transcription, while still
being able to rely on provided face detection and tracking
results to participate to the task. Figure 2 summarizes the
available modules.
and the correlation tracker proposed by Danelljan et al. [
Optical character recognition followed by name detection
is contributed by IDIAP [
Three variants of the name propagation technique
proposed in [
Because of limited resources dedicated to collaborative annotation, the test set cannot be fully annotated. Therefore, the task is evaluated indirectly as an information retrieval task, using the folllowing principle.
For each query q 2 Q N (firstname_lastname), returned shots are rst sorted by the edit distance between the hypothesized person name and the query q and then by condence scores. Average precision AP(q) is then computed classically based on the list of relevant shots (according to the groundtruth) and the sorted list of shots. Finally, Mean Average Precision is computed as follows:
MAP = 1
X AP(q) jQj q2Q
This work was supported by the French National Agency for Research under grants ANR-12-CHRI-0006-01 and ANR14-CE24-0024. The open source CAMOMILE collaborative annotation platform3 was used extensively throughout the progress of the task: from the run submission script to the automated leaderboard, including a posteriori collaborative annotation of the test corpus. The task builds on Johann Poignant involvement in 2015 task organization. Xavier Trimolet helped design and develop the 2016 annotation interface. We also thank INA, LIUM, UPC and IDIAP for providing datasets and baseline modules. 4.1
Face tracking-by-detection is applied within each shot
using a detector based on histogram of oriented gradients [