In this paper we propose a new approach for the genre tagging task of videos, using only their ASR transcripts and associated metadata. This new approach is based on calculating the semantic similarity between the nouns detected in the video transcripts and a bag of nouns generated from WordNet, for each category proposed to classify the videos. Specifically, we have used the Lin measure based on WordNet, which calculates the semantic distance between two synsets. Obviously, this approach has been only applied on the English test videos due to the use of WordNet, an English lexical resource. As base case, we have applied an information retrieval system as a classifier, using the generated bag of nouns for each category as index data and the ASR transcripts from each test video as query. Several experiments have been submitted, one of them combining both approaches (information retrieval and semantic similarity). As main conclusion we have shown that, using this combination of semantic similarity and information retrieval, we can improve the results obtained using the information retrieval approach only.
H.3.1 [Information Storage and Retrieval]: Content Analysis and Indexing - Indexing methods
Multimedia data are usually tagged with some relevant
information in order to make the retrieval easier. In fact,
the efficient use of textual data associated to other types of
information such as images can improve multimedia IR
systems [
The Genre Tagging task in MediaEval 2011 attempts to
automatically generate genre labels to organize videos [
In the Genre Tagging task of MediaEval 2011, participants
are required to automatically assign thematic subject labels
to videos using features derived from speech, metadata,
audio or visual content. It is important to note that this is
not a multilabel tagging task, so a given video can only be
assigned to one label. The data set provided are the same as
those used in MediaEval 2010 Wild Wild Web Tagging Task
(ME10WWW) [
Our main approach is based on using an Information Retrieval (IR) system as a classifier. On the one hand, we have generated a XML document or bag of words for each category proposed, making use of an external lexical resource like WordNet1. Specifically, we have included synonyms, hyponyms and domain terms related to the category. For example, for the “educational ” category we have generated a XML document including terms such as instruction, teaching, pedagogy, didactics, training, etc. On the other hand, the preprocessed ASR transcripts (stemming and stop word removal) from test videos have been used as queries, without any expansion. Finally, the Terrier2 IR system has been used to obtain a measure of relatedness (RSV, Retrieval Status Value) between each video and the generated bags of words. 1http://wordnet.princeton.edu 2http://terrier.org
As a second approach, we have used the formula proposed
by Lin [
Several experiments were carried out under both approaches. As baseline, we have considered the use of the preprocessed ASR transcripts from test videos as query (experiment IR ASR). Then, we have tried to evaluate the addition of the metadata provided, carrying out an expansion of the ASR transcripts using such metadata (experiments IR ASR+MD and IR ASR+MD+TAGS ). Regarding the second approach, we have submitted the experiment “SIMSEM-ASR”, in which we only calculate the semantic similarity between each video and each category (its bag of words), without using the IR approach. Finally, we have combined the IR and the semantic similarity approaches (experiment “SIMSEM+IR-ASR”), merging both lists of results. First, we have normalized the RSV score from the baseline. Then, for each test video, we have added their normalized RSV and semantic similarity scores. The results obtained are shown in Table 1, using the Mean Average Precision (MAP) measure. We also show the MAP obtained considering only the English test videos.
Run name
IR ASR
IR ASR+MD IR ASR+MD+TAGS
SIMSEM-ASR SIMSEM+IR-ASR
0.1031 0.1073 0.1115 0.0547 0.1266
0.1044 0.1088 0.1129 0.0559 0.1288
Analyzing the official results we can observe that the expansion of the ASR transcripts using the provided metadata improves the result obtained when metadata is not used (+4% and +8.15% better for the experiments IR ASR+MD and IR ASR+MD+TAGS, respectively), as it was expected. On the other hand, the combination of the semantic similarity and the IR approaches seems to be interesting because it improves the MAP value obtained for the baseline using the IR approach only (+22.79%). Taking into account the test groundtruth file provided by the MediaEval organizers, 185 videos of the 1,673 English videos (11.06%) belong to the default category, while our best experiment assigned only 18 videos (1.08%) to such category. This was motivated by the low threshold used to assign a video to the default category (0.25), which allowed to classify videos in categories that really did not correspond due to its low similarity score. Nevertheless, for some categories (art, politics, religion and sports), we obtained good results, achieving high MAP individual scores (e.g. 0.6176 for the politics category). This is due to such categories are more general concepts or genres than others (business, comedy, documentary, etc.), so it was easier to find more nouns semantically related, increasing the size of the XML document generated for such categories and, therefore, the probability of success. 5.
In this paper we propose the use of the semantic similarity based on WordNet combined with the IR approach in order to solve the genre tagging task of videos. Because our research field of interest is Natural Language Processing (NLP), we have only worked with the ASR transcripts from videos and their metadata. It was shown that combining the semantic similarity score with the RSV score obtained from the IR approach, we obtained a significant improvement. Nevertheless, it seems clear that working only with the ASR transcripts generally get poor results. For future work, we will study other resources in order to increase the size of the bag of words generated for each category, adding more terms semantically related with such categories.