We describe our approach and results towards the genre tagging task of MediaEval 2011. We approached this as an Information Retrieval task and applied a pseudo relevance feedback (PRF) approach for query expansion. Query expansion was also done using WordNet1 and Wikipedia2. Our results show improvement in the tagging task using these query expansion techniques over the original query retrieval results.
Video indexing can be improved by tagging the videos with
keywords. Genre tags communicate about the content of the
videos which can assist users or viewers to decide if the video will
interest them or not. In this paper we present our approach and
results towards the task of automatically tagging videos with
genre labels for MediaEval 2011. Participants were given videos,
including speech transcripts, metadata and user tags. The task is to
assign one genre tag to each of the videos [
Genre tags (26 unique) are considered as the query terms for this task. We use synonyms from the first synset of the query terms produced by the WordNet, as their expansion. We also retrieve Wikipedia article for each query term and extract useful terms from them. To extract keywords for expanding the query terms using Wikipedia articles, we first compute TF-IDF for each term in each article and then considered the top 40 terms with the highest TF-IDF values as the keywords of those articles. We also expanded the queried using PRF. In this technique the top 50 documents retrieved by using the cosine similarity between query 3 http://www.lextek.com/manuals/onix/stopwords1.html 4 http://snowball.tartarus.org/algorithms/porter/stemmer.html documents and the video document vectors are considered as the relevant documents and used for further query term expansion. All the words from these documents are joined with the original expanded query and TF-IDF values for these combined query documents are calculated. These expanded queries are again used to retrieve the remaining video documents using cosine similarity between the query vector and the video document vector. We finally take the top 50 retrieved documents from the second phase, i.e., after PRF, and add them to our earlier list of retrieved document. For Run 2 and Run3 we consider an additional clue based on video titles, i.e., after performing pseudo relevance feedback, we match the show names of the remaining nonretrieved videos (not among the first phase top 50 and also second phase top 50) with the already retrieved videos. If the names match we categorize the new videos in the same genre as the matched videos, given the case that their cosine score is non-zero for that genre. The videos that do not match any genre by doing any of these steps fall in the “default category 1006” genre.
Table 1 shows the official results of our 3 runs. They differ from each other in that the input video document for Run1 includes only Speech Transcripts, whereas Run2 consists of speech transcripts as well as metadata including the title. Input for Run3 includes everything for Run2 as well as user defined tags for each video. MAP 0.0621 0.0934 0.094 From Table 1, we can see that the Mean Average Precision (MAP) scores of Run 2 and Run 3 are higher than Run 1 and this can be attributed to a possible reason that we have metadata and extra user assigned tags in them. By looking at the individual query MAP scores, we can see that the queries with maximum MAP scores are Sports and Religion across all three runs. The genre Personal_or_Auto-biographical had 0 MAP score for all the three runs. Sports and Religion genres have average number of videos whereas there are only 8 videos in Personal_or_Autobiographical genre. Very few videos in this genre may have an impact on the retrieval result, especially after doing PRF the chances of getting a lot of false negative query terms increases. But, since this is not a classification task, it cannot be interpreted from the above statement that the more videos we have the better chances of their retrieval, because the genre with maximum number of videos, i.e., Politics with 552 videos does not have the maximum MAP score. As the main aim in this paper is to study the effect of query expansion in genre detection task, we performed further unofficial tests/runs to analyze the effect of these query expansion techniques used. All of these 6 additional runs use the input for official Run 2. We implement another change in these additional runs as compared to our officially submitted runs, such that we took all the video documents retrieved, i.e., the documents with non-zero cosine scores after the second phase and do not just use the top 50 documents. Table 2 explains the difference between each of these 6 runs. Run 4 is our baseline run in which we do not perform any kind of query expansion and match the video document vectors with single term query documents. The results shown in Table 3 are MAP scores for each of these unofficial runs. By examining the results in Table 3, we can see that there are improvements in MAP scores as we combine various query expansion techniques.
Runs Run4 Run5 Run6 Run7 Run8 Run9
Table 3. Comparison of various query expansion techniques Run Run5 Run6 Run7 Run8 Run9 4 MAP scores 0.07 0.07 2 0.083 9 0.061 9 0.083 6 0.098 6 In Run7 the MAP score drop as compared to Run4 which can be attributed to the fact that we are using PRF. By analyzing Run7 we saw that only 18% of the videos were correctly identified in the first round whereas, in PRF technique we consider all 100% of them to be correct and used them for further query expansion. As we can interpret this leads to a lot of false positive query terms. There is some improvement in the MAP score (32%), when the query expansion by WordNet and Wikipedia was combined along with PRF (Run9). By expanding the queries using WordNet and Wikipedia, 22% of the videos were correctly identified in the first phase, which aids in increase of MAP score. Hence, we can conclude that PRF alone is not a good technique to use, but combined with other kinds of query expansion, can help in retrieval. However, further analysis is needed to determine if the costs of performing PRF is worth the increase in retrieval results.
This work is the beginning for the genre detection task that we would like to undertake in near future. A detail work has to be done to increase the accuracy of this task, such as giving different weights to the different inputs. The most obvious analysis that is needed in this work is inspecting the PRF technique applied such as evaluating the top document cutoff or the combination of documents and query after PRF. Further improvements in the results can be done by using other modalities such as visual along with the textual metadata and speech transcripts. We would also like to approach the genre detection task as a classification task and apply all the modalities to learn different genres.