In this paper, we investigate the possibility of using visual information to improve the text based ranking. Both a structure based representation (using the similarity matrix of the frames of one video) as well as a key-frame based representation (using visual words) is evaluated. It appears that only in some queries the visual information can improve the performance by reranking. The presented experiments on reranking show the limitations and also the potential, for these structural and visual representations
In the MediaEval 2011 Genre Tagging Task, internet videos have
to be ranked according to their relevance for a set of genre tags [
In this paper, a Bayesian reranking approach is performed based on two kinds of video representations: the first one is a structure based feature and the second one is a key frame based feature.
Most video measurements are based on comparing visual similarity between videos directly, using color, shapes and movement. However, these measurements would not always work due to the high variance of video content. In particular, video from the same genre are not expected to have the same visual Copyright is held by the author/owner(s). content. Motivated by the limitations of visual similarity, we proposed a method for video structure representation and measurement.
In this method, a video self-similarity matrix is used to represent the structure. This matrix is generated by calculating the pairwise similarity between frames from one video, that are sampled with a fixed sampling rate. These similarities are calculated from the HSV histogram of each frame. This representation exploits the fact that one video tends to have consistent quality and editing conditions. A reliable similarity can be achieved without complicated low level visual features or additional domain knowledge Each video is now represented by a square similarity matrix, which can be considered as a square gray-level image. Next, three types of multi-scale statistical image features can be extracted from the self-similarity matrix: a) GLCM based features (30 components). The Gray Level Co-occurrence Matrix (GLCM) is constructed from the similarity matrix for 2 directions (0, 45) and 3 offsets (3, 6, 12). For each GLCM, energy, entropy, correlation, contrast and Homogeneity are computed; b) 3-scale Gabor texture features (14 components). c) Intensity Coherence Vectors (16 components). Each pixel within a given intensity bin is classified as either coherent or in coherent, based on whether or not it is part of a large similarly-colored region. The size of the region is determined by a fix threshold. (1/15 of the size of image is used.)
Next to the structure, a Visual Word representation based on
keyframes is used for measuring the visual similarity between videos
[
The attractiveness of reranking is that it is naturally unsupervised. Given an initial ranking list, an improved one can be achieved by grouping the visually similar videos into the nearby positions. However, in practice, the designing of re-ranking methods and the setting of parameters are highly depended on the quality and characteristics of the base line ranking list.
Bayesian video re-ranking method is used in this paper, because it
requires less assumptions of the original ranking list and it is less
sensitive to particular parameter settings [
E (r) r r2 ) i, j wij (ri rj )2 c (i, j)Sr (1 1 r1 r2 Here r [r1,r2,,rN ] is the ranking list after reranking, r [r1,r2,,rN] is the initial ranking list, w expx x /σ the visual similarity between two items in the refined ranking list. The first term measures the visual consistency of the ranking list, while the second term refers to the ranking distance between the reranking list and initial list. c is a trade-off parameter to the two terms, which can be optimized on the development set.
Reranking is performed based on two baseline ranking lists which
are generated by text retrieval. The first one uses information of
automatic speech transcript (ASR), the second one uses metadata
of the video. The second one is expected to outperform the first
one. Details of generating these two baselines can be seen at [
Furthermore, many videos in this dataset are presented in series. 1390 videos in the test set have more than 2 episodes belong to the same show. Videos of the same show tend to share certain visual similarities. Through the analysis of the reranking performance on each query, it can be observed that this property has a strong effect on the performance. (The reranking results for some selected queries are presented in Figure 1.) Generally speaking, if most of the true positive videos for a certain query are from one or several shows, the reranking results can be reasonable, such as the query ‘1016 politics’.
In particular, the most significant improvement appeared in the query ‘1001 autos_and_vehicles’. It can be seen from the ground truth that all the 6 videos in this genre are episodes of a same show. The reranking process takes advantage of the high visual similarity between the 6 videos. Especially, for this query, keyframe based features achieved higher performance than the structure based features. This is because videos in the same series Exploring the similarity within the same show is not enough for genre tagging. Videos of the same show have certain chances to be of different genres. Moreover, there are some queries of which videos are from many different shows. (For example, the 64 videos of the query ‘1019 sports’ are from 16 different shows.) In these cases, the visual similarities between true positive videos are not obvious. Therefore, the structure based features outperform key-frame based ones. This indicates that the videos of same genre may share similarities in structure even though they are not consistent in visual.
Although the visual reranking made no improvement for the initial ranking list on MAP, it does not necessarily mean that visual information is useless for detecting video genre. It is the special characteristics of this dataset that make it difficult to utilize information in visual channel. In particular, compared with conventional understanding of video ‘genres’, the genre tags given in this task are more related to the ‘topics’ of videos. The proposed structure based video representation provided a possibility for an inexact matching for video similarity. The characteristics of this representation can be observed through analyzing the reranking performance of certain queries. It is still not clear what is the most suitable way of representing the frame similarity matrix. A more attractive direction may be discovering a set of tags which could reflect the visual consistency of videos.