This paper describes a system to identify people in broadcast TV shows in a purely unsupervised manner. The system outputs the identity of people that appear, talk and can be identi ed by using information appearing in the show (in our case, text with person names). Three types of monomodal technologies are used: speech diarization, video diarization and text detection / named entity recognition. These technologies are combined using a linear programming approach where some restrictions are imposed.
The 2015 Multimodal Person Discovery in Broadcast TV
[
Speaker information was extracted using an
Agglomerative Hierarchical Clustering diarization system based in
Hidden Markov Models [
Speaker clusters are modeled with Gaussian Mixture
Models (GMM). The complexity selection of the models is based
on the amount of data per cluster and the cluster complexity
ratio which xes the amount of speech per Gaussian.
Hidden Markov Model (HMM) training and cluster realignment
by Viterbi decoding is based on maximum likelihood. In the
decoding stage, a minimum speaker segment duration of 3
seconds is imposed to deal with too short segments. For
the cluster merging, the most likely pair of clusters are
selected in each iteration. This likelihood is calculated using a
modi ed Bayesian information criterion (BIC) [
This system has been used with two di erent kind of inputs for each show. In one hand, diarization is run with each audio le without any constraint. In the other hand, using a face-tracking system, segments without tracked faces are discarded. The purpose of this second method is to run the diarization only in those parts where we assume that someone in the video must be speaking. 3.
For face tracking, the baseline code was used (tracking
by detection using the Kanade-Lucas-Tomasi algorithm [
We used agglomerative hierarchical clustering. A binary
hierarchical tree is created by fusing tracks according to the
minimum distance between track vectors. The number of
clusters may vary between videos and has to be determined.
It is estimated by evaluating the CalinskiHarabasz [
To improve the diarization, spatio-temporal restrictions were introduced. We assume that a person can not appear twice in a frame so tracks with temporal overlapping should represent di erent persons and are prevented to merge into the same cluster. Also, as we use a multi-vector representation for each track, vectors in the same track must be part of the same cluster. Restrictions are modeled using a matrix expressing the relationship between the feature vectors. Entries for vectors in di erent tracks were assigned a value of 1, entries for vectors in the same track were assigned a value 0 < v 1, and entries for vectors on temporally cooccurring tracks received a very large value v 1. This matrix is used to point-wise multiply the vector-to-vector distance matrix used for clustering. 4.
We used the person names provided in the baseline [
Our system combines the previous information sources to obtain the nal person recognition labelling. Speaker diarization and video diarization are performed rst in an independent manner. In order to fuse this information to create a nal labelling, the development database was analyzed. Some assumptions were made:
Speaker is not always related with who is shown in the screen. So it is important to weigh accurately the temporal overlaps between each speaker with its di erent possible face identity assignments.
Some speakers do not come into view any time in the show and there are other people who are shown in the screen but do not speak. Both should be discarded. Text identities are more related with who is shown rather than with who is speaking. So text is better combined with video than with speech.
According to these assumptions, an algorithm was designed based in weighting temporal overlaps between tracks (Figure 1). This algorithm considers two di erent fusion modalities (Video/Text and Video/Audio) and combines both to obtain a nal track le. Firstly, text and video are fused. Their overlapped tracks are selected, and the temporary overlaps of their identities are weigthed to set the constraints of an ILP system (IBM CPLEX).
max(X X ij
i j X ij j ij ij ) 1 (1) (2) ( ij : assignment between i text identity with a j video identity; ij : weight of assignment). Equation 2 establishes that each text identity must only have one face identity assigned. The next step is to combine the speech diarization tracks with the face tracks that have a text identity assigned. The same method based on ILP is used. Finally, using the relation between text, face and speaker identities and the overlapped tracks in the second fusion, the nal labeling output was obtained. A second algorithm was implemented changing the order of the fusions. In this case, audio were fused with the video and the result was combined with the text identities. Thus, only the face identities with a speaker assigned were considered.
Five di erent experiments were performed, which are shown in Table 1. These experiments were evaluated with the training database and evaluated using the mean average precision metric (MAP). In the experiments we tested several variations: the order of the fusions, the input of the audio diarization and the text system used. In Table 1, System 1 refers to the architecture shown in Figure 1 where the rst fusion combines text and video, and System 2 refers to rst combining video and audio and later fusing text. f acetrack indicates that the audio diarization is performed using only audio tracks where there are faces detected. The null case means performing the diarization using the whole audio input. While the rst four experiments use the baseline names, in the fth one the system described in section 4 was used.
The best performance was achieved in experiment 4 by the System 1, without ltering the audio input for the diarization and using the Baseline person names. There is a clear evidence that the system works better when the diarization is run with the whole audio input. Referring to the fusion order in the algorithm, results indicate that mixing video and text tracks rst, provides a better performance.
The ve experiments were run on the test data. Experiments 1-4 were submitted on July 1st and experiment 5 on July 8th. The best set-up in the training data (Exp.4 in Table 1) was uploaded as our primary submission. After evaluating this primary submission with the nal set of annotations, the following results were obtained: EwMAP = 54.1%, MAP = 54.36% and C = 69.71%. Experiment 5 was submitted on July 8th. It is similar to experiment 4 but using our own technology to obtain person names. We had low performance with the OCR and NER and thus the results were worse than expected. 7.
Speaker diarization, face recognition, and text detection with named entity recogniton have been combined using the integer linear programming approach. Our idea was to rst perform monomodal speech and video diarizations, using as much restrictions as possible to improve the results and then use ILP to combine these diarizations along with the persons name information. Several architectures for this combination and several constrains of the integer linear programming algorithm were considered. The architecture which combines video and audio modalities after the fusion with the text stream has provided the best results.
This work has been developed in the framework of the projects TEC2013-43935-R, TEC2012-38939-C03-02 and PCIN2013-067. It has been nanced by the Spanish Ministerio de Econom a y Competitividad and the European Regional Development Fund (ERDF).