The UPC system works by extracting monomodal signal segments (face tracks, speech segments) that overlap with the person names overlaid in the video signal. These segments are assigned directly with the name of the person and used as a reference to compare against the non-overlapping (unassigned) signal segments. This process is performed independently both on the speech and video signals. A simple fusion scheme is used to combine both monomodal annotations into a single one.
This paper describes the UPC system for the 2016
Multimodal Person Discovery in Broadcast TV task [
The signal intervals that overlap with an overlaid person name are extracted and used for unsupervised enrollment, de ning a model for each detected name. This way, a set of classes corresponding to the di erent persons in the detected names is de ned. These intervals are directly labeled by assigning the identity corresponding to the overlaid name.
For each modality, a joint identi cation veri cation algorithm is used to assign each unlabeled signal interval (FT or This work has been developed in the framework of the projects TEC2013-43935-R, TEC2012-38939-C03-02 and PCIN- 2013-067. It has been nanced by the Spanish Ministerio de Econom a y Competitividad and the European Regional Development Fund (ERDF). SS not overlapping with an overlaid name) to one of the previous classes. For each unlabeled interval, the signal is compared against all models and the one with better likelihood is selected. An additional 'Unknown' class is implicitly considered, corresponding to the cases where the face track or speech segment correspond to a person that is never named (i.e. none of the appearances of this person in the video do overlap with a detected name).
At the end of this process we have two di erent sets of annotations, one for speech and one for video. The two results are fused, as described in section 5 to obtain the nal annotation. 2.
We have used the two baseline detections with some
additional post-processing. The rst one (TB1) was generated by
our team and distributed to all participants. The LOOV [
The second set of annotations, TB2 was provided by the
organizers [
For face tracking, the 2015 baseline code [
The VGG-face [
A face veri cation algorithm was used to compare and classify the tracks. First, the tracks that were overlapped with a detected name were named by assigning that identity. To reduce wrong assignations, the name was only assigned if it overlapped with a single track. Then, using the set of named tracks from the full video corpus, a Gaussian Naive Bayes (GNB) binary classi er model was trained, using the euclidean distance between pairs of samples from the named tracks. Then, for each speci c video, each unnamed track was compared with all the named tracks of the video, computing the euclidean distance between the respective feature vectors of the tracks (see Figure 1). This value was classi ed using the GNB to either being a intra-class distance (both tracks belong to the same identity) or an inter-class distance (the tracks are not from the same person). The probability of the distance being intra-class was used as the con dence score. The unnamed track was assigned the identity of the most similar named track. A threshold on the con dence score (0.75) was used to discard tracks not corresponding to any named track.
Speaker information was extracted using an i-vector based speaker tracking system. Assuming that text names are temporarily overlapped with their speaker and face identities, speaker models were created using the data of those text tracks. Speaker tracking was performed evaluating the cosine distance between model i-vectors and i-vectors extracted for each frame of the signal.
Speaker modelling was implemented using i-vectors [
The speaker tracking system has been implemented as a speaker identi cation system with a segmentation by classi System
Baseline 1 Spk Tracking
Baseline 2 Face Tracking
Baseline 3 Intersection
Union
cation method. For the feature extraction, 20 Mel Frequency
Cepstral Coe cients (MFCC) plus and coe cients
were extracted. Using the Alize toolkit[
Starting o with the speaker and face tracking shot labeling, two fusion methods were implemented. The rst method was the intersection of the shots of both tracking systems, averaging the con dence of the intersected shots. In the second method, the union strategy was implemented relying on the intersected shots of both modalities and reducing the con dence of those not intersected. The shots of both video and speaker systems were merged, averaging the con dence score if both systems detect the same identity in a shot, or reducing the con dence by a 0.5 factor if only one of the systems detected a query.
Four di erent experiments were performed which are shown in Table 1. Baseline 1 refers to the fusion between the baseline speaker diarization and OCR, Baseline 2 refers to the fusion between the face detection and the OCR and Baseline 3 is the intersection of the both previous baselines. Initially, speaker and face tracking have been evaluated separately. The intersection and the union of both tracking systems were implemented as fusion strategies.
As is shown in Table 1, both monomodal systems improve the baseline performances by a great margin. The union strategy has shown a better performance than the intersection strategy but this fusion does not show a signi cative performance increase over the individual modalities.
By analysing the results, we believe that failures at text detection was the main factor impacting the nal scores. 6.
Speaker and face tracking have been combined using different fusion strategies. This year, our idea was to focus only on the overlaid names to develop tracking systems instead of performing diarization systems merged with text dectection. Tracking systems have shown a better performance than the diarization based ones of the baseline. For fusion, the union strategy has shown higher results than the intersection method.