In this paper, we present the system developed by GTMUVigo team for the Multimedia Person Discovery in Broadcast TV task at MediaEval 2016. The proposed approach consists in a novel strategy for person discovery which is not based on speaker and face diarisation as in previous works. In this system, the task is approached as a person recognition problem: there is an enrolment stage, where the voice and face of each discovered person are detected and, for each shot, the most suitable voice and face are assigned using the i-vector paradigm. These two biometric modalities are combined by decision fusion.
The Person Discovery in Broadcast TV task at MediaEval
2016 aims at nding out the names of people who can be
both seen as well as heard in every shot of a collection of
videos [
The proposed system can be divided in an enrolment and a search stage. For each person name detected by optical character recognition (OCR), the most likely interval of speech and face presence are detected and used for enrolment. Once the detected people are enrolled, speaker and face recognition are performed for each shot in order to assign labels to that shot. A decision fusion strategy is implemented in order to combine the speech and video labels. The details of the system are described below.
The person names were obtained from the video using
the baseline system provided by the organisers. Speci cally,
the UPC OCR approach using LOOV was used [
First, features were extracted from the waveform;
specifically, 19 Mel-frequency cepstral coe cients (MFCCs)
including energy were extracted every 10 ms using a 25 ms
sliding window. A dynamic normalisation of the cepstral
mean was applied using a sliding window of 300 ms. These
features were extracted using the Kaldi toolkit [
The time interval (tstart; tend) in which the name of the speaker spk appears is taken as a starting point. A strategy to enlarge this time interval in order to obtain more data to enrol the speaker is applied: given the time intervals Sleft = (tstart 10; tend) and Sright = (tstart; tend+10), a change point is searched within each of these intervals using the Bayesian information criterion algorithm (BIC) for speaker segmentation, having the restriction that the change point has to be in the intervals (tstart 10; tstart) and (tend; tend + 10), respectively. If no change point was found within the interval Sleft then tleft is set to tstart 10 and, similarly, if no change point was found within the interval Sright then tright is set to tend + 10. Then, speaker spk is assumed to be speaking in the interval Sspk = (tleft; tright). In case speaker spk appears several times in the OCR output, a segment is computed for each occurrence. Speech activity detection (SAD) was performed in order to remove the non-speech parts. To do so, the energy-based SAD approach implemented in the Kaldi toolkit was applied.
An i-vector [
When dealing with faces, the rst step consisted in
performing face tracking using the baseline approach based on
histogram of oriented gradients [
The faces detected by the face tracker in the interval (tstart; tend) in which the name of the speaker spk appears are considered. Given that only one face was detected, the whole presence interval of that face is taken. In case more than one face was detected, the p c1 c2 b one that appeared in more frames was assigned to the speaker, assuming that was the dominant face in the given time interval.
Features were extracted in the time interval obtained
in the previous step. To do so, rst face detection was
performed, and a geometric normalisation was done.
After that, photometric enhancement of the image
using the Tan&Triggs algorithm [
Once the features were obtained, an i-vector representing that face was obtained using the Kaldi toolkit. As done when dealing with speech, if there were several time intervals where the face of the speaker was present, the features obtained in all the segments were concatenated. 2.4
The procedure to decide which speaker was present in each shot consisted in, for each shot:
In order to detect whether the shot includes speech, speech detection was performed: perceptual linear prediction coe cients plus pitch features were extracted from the time interval de ned by the shot, an i-vector was extracted and a logistic regression approach was used to classify the segment as speech or non-speech. Non-speech segments were straightforwardly discarded. In case speech was present in the shot, SAD was performed, an i-vector was extracted, and this shot ivector was compared with the enrolment i-vectors computing the dot scoring. The speaker that achieved the highest score was assigned to the shot.
The faces detected by the face tracker within the shot were identi ed, and the one that appeared in more frames was chosen as the most representative face of the shot. An i-vector was extracted and the same decision procedure described for the speech data was performed.
Once a decision was made for both speech and face data, the following fusion approach was implemented: given a shot, it is assigned to a speaker if the person detected by the face and speech detectors had the same name and if the sum of their scores was greater than a threshold.
Table 1 shows the results achieved with the audio+video fusion system (p), the audio system only (c1), the video system only (c2) and the baseline provided by the organisers (b). The main conclusions that can be extracted from the Table are: (1) the audio and video systems are complementary, since their combination leads to an improvement of the individual results; (2) the audio results are better than the video results, especially in the DW database; and (3) the worst results were obtained in the DW database, while the best ones were achieved in the 3-24 database. The reason why 3-24 results are, in general, better, might be caused by the small number of queries in the evaluation data corresponding to this database (only 15 queries out of 693), which leads to results that are not signi cative. In the case of DW database, 606 queries were evaluated; this, combined with the fact that the OCR approach used in this system did not nd person names in 612 out of 757 les in the database, led to poor results in DW data.
The aim of this system was to assess a novel approach for person discovery that is not based on speaker and face diarisation as in most state-of-art strategies. The achieved results are promising, and the experiments performed in this evaluation allowed the detection of the main weak points of the system that will be improved in the future:
The quality of the OCR output had a huge impact on the results, since this is the starting point of the whole enrolment stage, which leads to a degradation of performance on the whole system. A simple approach, based on natural language processing, for ltering the OCR output in order to remove everything that were not person names was assessed in this framework with no success, but further experiments on this topic will be done in the future.
All face-based steps relied on the baseline approach for face tracking, and its output was fed to the feature extraction module; however, only the information about presence was used, but not the bounding boxes where the faces appeared. This probably led to inconsistencies in the feature extraction stage and, therefore, on the face enrolment procedure. This issue will be addressed in order to improve the quality of the face-based approach.
This research was funded by the Spanish Government under the project TEC2015-65345-P, the Galician Government through the research contract GRC2014/024 (Modalidade: Grupos de Referencia Competitiva 2014) and `AtlantTIC Project' CN2012/160, and by the European Regional Development Fund (ERDF).