This paper describes our diarization system for the Multimodal Benchmark evaluation campaign [1]. The goal of this task is naming speakers, who are appearing and speaking simultaneously in the video, without prior knowledge. Our diarization system is based on multimodal approach to combine audio and visual informations. We extract features from a face in each shot to make visual i-vectors [2], and introduce them to the provided baseline system. In the case of faces are extracted performance becomes better, but based on the test run, clear improvement could not be observed.
The Multimodal Person Discovery in Broadcast TV task can be split into following three subtasks: speaker diarization, face detection and tracking, and name detection. We focused on the speaker diarization.
For speaker diarization, combining audio and visual features
are often effective. For example, in previous work, visual features
extracted from faces [
In the field of speaker recognition and diarization, using
ivector [
Here, we assume that a visual i-vector represents face- and channel-variabilities. We expect it works as face recognition and is easy to introduce to a diarization system.
Figure 1 and Figure 2 show the overview of visual i-vector
extraction, and the whole system, respectively. First, we extract
HOG features from a face in each shot to make a visual i-vector.
Second, we estimate the distance between each pair of visual
ivectors. Third, we combine audio score and visual i-vector’s score
to apply hierarchical clustering. Finally, we combine diarization
results and other results using the baseline system [
Let M be a GMM super-vector, which is the concatenation of
normalized mean vectors of an estimated GMM for a targeted
video shot. An i-vector w is extracted from it, by assuming that M
is modeled as
= +
a low rank matrix representing total variability. The EM algorithm
is used to estimate the total variability as proposed in [
= ( + ( ) )−1 −1 ( ), where N(u), and F(u) are the zero, and first order Baum-Welch statistics on the UBM for the current utterance u, and Σ is the covariance matrix of the UBM.
After visual i-vectors are extracted, we calculate distance between each pair of vectors in cosine distance. The distance Dij between i-vectors wi and wj is calculated by = 0.5 − || ||2 || ||2
To combine our system and the baseline diarization system, we use late fusion. The final score F is given by =
+ (1 − ) (0 ≤ ≤ 1), where A is a score from the baseline system (normalized BIC between each utterance), V is a score from the visual i-vector, and a is a weighting parameter.
After two scores combined, we apply hierarchal clustering according to fusion score F.
HOG Features are extracted from the face in each shot detected by using the baseline speaking-face system. If there is no candidate, the previous shot’s candidate is used instead of it. A HOG feature is 34 dimensions, which consists of 32 dimensional histograms of oriented gradients and its x-y coordinates. The
run #1 visual i-vector (a = 0.996)
Dev.: Test2
In the development and test videos, direction of speaker face is frontal for the most part. This is good condition for our system. However, the results do not show significant improvement compared with the baseline system. One of the reasons is the insufficient face detection accuracy. We observed that 63% of detected bounding boxes capture a face or part of a face in our analysis on randomly sampled 10 videos in the test set. However, since only 82% of them capture an entire face precisely, our system did not improve the overall performance significantly. We conclude that normalization of the face area for more sophisticated face detection is needed.
We presented an audio-visual based speaker diarization system, which uses visual i-vectors. The results did not show obvious advantage of our method due to the accuracy of face detection. In our future work, normalizing the face area and combining with other visual features extracted from whole image would help the improvement of the accuracy.