UAB at MediaEval 2011: Genre Tagging Task Richa Tiwari Chengcui Zhang Manuel Montes The University of Alabama at The University of Alabama at National Institute of Astrophysics, Birmingham Birmingham Optics and Electronics Birmingham, Alabama, USA Birmingham, Alabama, USA Puebla, Mexico rtiwari@cis.uab.edu zhang@cis.uab.edu mmontesg@inaoep.mx ABSTRACT We describe our approach and results towards the genre tagging 2. SYSTEM OVERVIEW We approach the task of automatic genre tagging of the videos in task of MediaEval 2011. We approached this as an Information MediaEval 2011 as an IR task in which the videos are considered Retrieval task and applied a pseudo relevance feedback (PRF) as the documents to be retrieved, and the genre tags are our approach for query expansion. Query expansion was also done queries. We present the results of three official runs in Table 1. using WordNet 1 and Wikipedia 2. Our results show improvement All the runs include the same preprocessing of the dataset and the in the tagging task using these query expansion techniques over query terms and differ in only the amount of input being used at the original query retrieval results. each run, i.e., only speech transcripts for first run, speech transcripts and metadata for second run and finally also the tags Categories and Subject Descriptors for third run. We also perform some unofficial runs for further H.3.3 [Information Storage and Retrieval]: Information Search result analysis and show their results in Table 3. and Retrieval – query formulation, relevance feedback, retrieval model, search process. 2.1 Video Document Indexing Preprocessing of video documents involves stop word removal Keywords and stemming. We use a list of stop words freely available on the Genre video tagging, Information retrieval, query expansion, web 3and remove them from the data. Next we use a Porter categorization, pseudo relevance feedback. stemmer 4 to stem the text. We did not do any preprocessing on Non-English text. Once the texts are pre-processed, we calculate TF-IDF for each document and keep the words with non-zero TF- 1. INTRODUCTION IDF values as the index words for those documents. For Runs 2 Video indexing can be improved by tagging the videos with and 3 in which we consider metadata containing the title of the keywords. Genre tags communicate about the content of the video, a similarity bias is given to the video title. For example, videos which can assist users or viewers to decide if the video will there are several episodes of the same show in the dataset and interest them or not. In this paper we present our approach and which have similar name. We can assume that different episodes results towards the task of automatically tagging videos with of the same show would most likely belong to the same genre. For genre labels for MediaEval 2011. Participants were given videos, example, EconomyInCrisis-CaliforniasGrowingDeficit561 and including speech transcripts, metadata and user tags. The task is to EconomyInCrisis-FewSignsOfRecovery990 are two different assign one genre tag to each of the videos [2]. This task of genre episode videos of the same show (Economy In Crisis) with the detection is approached as an Information Retrieval task. There same starting name and belong to the same genre. Consequently, are two main phases in any IR approach; the first is indexing of we believe that different episodes of a show with a genre such as documents and the second is actual retrieval step. One of the most Politics may talk about politics of different countries or different popular approaches towards indexing the documents is the vector political agendas but they would all still have the Politics genre. space model [3]. Using the term frequency and inverse document Based on this assumption, if one episode video of a show is frequency (TF-IDF), we can produce a weight for each term in a ranked at the top for any of the genres by our approach, we give document and determine the important terms that can describe it. the other episode videos of the same show the same genre. The second step of retrieval involves query expansion. There are several ways in which query expansion can be approached [4]. 2.2 Query Expansion WordNet and Wikipedia are often used for query expansion and Genre tags (26 unique) are considered as the query terms for this have shown to produce improvement in retrieval results. Another task. We use synonyms from the first synset of the query terms methodology known as pseudo relevance feedback (PRF) can help produced by the WordNet, as their expansion. We also retrieve in improving the search results by using additional terms from the Wikipedia article for each query term and extract useful terms highly ranked documents for each query, as expansion of those from them. To extract keywords for expanding the query terms queries [5]. We have used this approach along with expansion using Wikipedia articles, we first compute TF-IDF for each term using WordNet and Wikipedia in this task. in each article and then considered the top 40 terms with the highest TF-IDF values as the keywords of those articles. We also expanded the queried using PRF. In this technique the top 50 documents retrieved by using the cosine similarity between query 1 3 http://wordnet.princeton.edu http://www.lextek.com/manuals/onix/stopwords1.html 2 4 http://www.wikipedia.org http://snowball.tartarus.org/algorithms/porter/stemmer.html Copyright is held by the author/owner(s). MediaEval 2011 Workshop, September 1-2, 2011, Pisa, Italy documents and the video document vectors are considered as the Table 2. Unofficial Run description relevant documents and used for further query term expansion. All Runs Description the words from these documents are joined with the original Run4 Baseline expanded query and TF-IDF values for these combined query documents are calculated. These expanded queries are again used Run5 Query expansion using Wordnet to retrieve the remaining video documents using cosine similarity Run6 Query expansion using Wikipedia between the query vector and the video document vector. We finally take the top 50 retrieved documents from the second phase, Run7 Query expansion using PRF i.e., after PRF, and add them to our earlier list of retrieved Run8 Query expansion using WordNet + Wikipedia document. For Run 2 and Run3 we consider an additional clue Run9 Query expansion with PRF + WordNet + Wikipedia based on video titles, i.e., after performing pseudo relevance feedback, we match the show names of the remaining non- Table 3. Comparison of various query expansion techniques retrieved videos (not among the first phase top 50 and also second Run Run5 Run6 Run7 Run8 Run9 phase top 50) with the already retrieved videos. If the names 4 match we categorize the new videos in the same genre as the MAP 0.07 0.07 0.083 0.061 0.083 0.098 matched videos, given the case that their cosine score is non-zero scores 2 9 9 6 6 for that genre. The videos that do not match any genre by doing any of these steps fall in the “default category 1006” genre. In Run7 the MAP score drop as compared to Run4 which can be attributed to the fact that we are using PRF. By analyzing Run7 3. EXPERIMENTAL RESULTS ANALYSIS we saw that only 18% of the videos were correctly identified in Table 1 shows the official results of our 3 runs. They differ from the first round whereas, in PRF technique we consider all 100% each other in that the input video document for Run1 includes of them to be correct and used them for further query expansion. only Speech Transcripts, whereas Run2 consists of speech As we can interpret this leads to a lot of false positive query transcripts as well as metadata including the title. Input for Run3 terms. There is some improvement in the MAP score (32%), when includes everything for Run2 as well as user defined tags for each the query expansion by WordNet and Wikipedia was combined video. along with PRF (Run9). By expanding the queries using WordNet and Wikipedia, 22% of the videos were correctly identified in the Table 1. Official results (MAP scores) first phase, which aids in increase of MAP score. Hence, we can Run1 Run2 Run3 conclude that PRF alone is not a good technique to use, but MAP 0.0621 0.0934 0.094 combined with other kinds of query expansion, can help in retrieval. However, further analysis is needed to determine if the From Table 1, we can see that the Mean Average Precision costs of performing PRF is worth the increase in retrieval results. (MAP) scores of Run 2 and Run 3 are higher than Run 1 and this can be attributed to a possible reason that we have metadata and extra user assigned tags in them. By looking at the individual 4. FUTURE WORK query MAP scores, we can see that the queries with maximum This work is the beginning for the genre detection task that we MAP scores are Sports and Religion across all three runs. The would like to undertake in near future. A detail work has to be genre Personal_or_Auto-biographical had 0 MAP score for all done to increase the accuracy of this task, such as giving different the three runs. Sports and Religion genres have average number weights to the different inputs. The most obvious analysis that is of videos whereas there are only 8 videos in Personal_or_Auto- needed in this work is inspecting the PRF technique applied such biographical genre. Very few videos in this genre may have an as evaluating the top document cutoff or the combination of impact on the retrieval result, especially after doing PRF the documents and query after PRF. Further improvements in the chances of getting a lot of false negative query terms increases. results can be done by using other modalities such as visual along But, since this is not a classification task, it cannot be interpreted with the textual metadata and speech transcripts. We would also from the above statement that the more videos we have the better like to approach the genre detection task as a classification task chances of their retrieval, because the genre with maximum and apply all the modalities to learn different genres. number of videos, i.e., Politics with 552 videos does not have the maximum MAP score. As the main aim in this paper is to study 5. REFERENCES the effect of query expansion in genre detection task, we [1] Larson, M., Eskevich, M., Ordelman, R., Kofler, C., performed further unofficial tests/runs to analyze the effect of Schmiedeke, S. and Jones, G.J.F. Overview of MediaEval these query expansion techniques used. All of these 6 additional 2011 Rich Speech Retrieval Task and Genre Tagging Task, runs use the input for official Run 2. We implement another MediaEval 2011 Workshop, 1-2 September 2011, Pisa, Italy. change in these additional runs as compared to our officially [2] Manning, C.D., Raghavan, P., and Schtze, H., 2008. submitted runs, such that we took all the video documents Introduction to Information Retrieval. Cambridge University retrieved, i.e., the documents with non-zero cosine scores after the Press, New York, NY, USA. second phase and do not just use the top 50 documents. Table 2 explains the difference between each of these 6 runs. Run 4 is our [3] Efthimiadis, N.E., 1996. Query Expansion, In Annual baseline run in which we do not perform any kind of query Review of Information Systems and Technology, Vol. 31, expansion and match the video document vectors with single term 1996, 121—187. query documents. The results shown in Table 3 are MAP scores [4] Yan, R., Hauptmann, A., and Jin, R., 2003. Multimedia for each of these unofficial runs. By examining the results in search with pseudo-relevance feedback. In International Table 3, we can see that there are improvements in MAP scores as Conference on Image and Video Retrieval, 229–238. Lecture we combine various query expansion techniques. Notes in Computer Science, vol. 2728, Springer.