=Paper=
{{Paper
|id=None
|storemode=property
|title=UNED at MediaEval 2011: Can Delicious help us to improve automatic video tagging?
|pdfUrl=https://ceur-ws.org/Vol-807/Cigarran_UNED2011_Genre_me11wn.pdf
|volume=Vol-807
|dblpUrl=https://dblp.org/rec/conf/mediaeval/RecueroFGAG11
}}
==UNED at MediaEval 2011: Can Delicious help us to improve automatic video tagging?==
https://ceur-ws.org/Vol-807/Cigarran_UNED2011_Genre_me11wn.pdf
UNED at MediaEval 2011: Can Delicious help us to improve
automatic video tagging?
J. Cigarran, V. Fresno, A. García-Serrano, D. Hernández-Aranda, R. Granados,
{juanci, vfresno, agarcia, daherar, rgranados}@lsi.uned.es
NLP & IR Group, UNED, Madrid
ABSTRACT usefulness of social tags for resource classification and infor-
In this paper we present the second participation of the mation retrieval tasks. [1] presented a study of the charac-
NLP&IR group at UNED in the MediaEval Genre Tagging teristics of social annotations provided by end users, in order
Task. This categorization task was carried out applying an to determine their usefulness for web page classification. In
Information Retrieval (IR) approach considering the video [6] they studied the usefulness of social tags as a comple-
collection’s textual data and query expansion techniques. mentary source for improving the classification of academic
The results show that the combination of social tags and conferences into corresponding topics. With regard to the
language models is useful to perform query expansion. classification of resources other than web pages, in [4] they
present a comparison of tags annotated on books and their
Library of Congress subject headings and in [7] the authors
Categories and Subject Descriptors present a method to classify products from Amazon into
H.3.3 [Information Storage and Retrieval]: Information their corresponding categories using social tags.
Search and Retrieval—Search Process
3. SYSTEM OVERVIEW
Keywords Our tagging system applies an IR approach which uses the
tag prediction, video indexing, multimedia information re- candidate labels as queries. The IR module applies a BM25F
trieval, query expansion ranking function [5] to calculate the similarities. This func-
tion has shown a god performance on those scenarios where
data is represented and indexed on different fields. In our
1. INTRODUCTION particular case, we have indexed the MediaEval collection
In this paper we introduce a new approach for genre tag- considering the different textual data (i.e. trancripts, meta-
ging task within the MediaEval initiative [3]. We are only fo- data and tags) as separate fields. As the use of standalone
cused on textual information and our contribution is mainly labels to query the system could drastically reduce the recall
based on exploiting information extracted from social tag- performance we decided to expand them using three different
ging sites like Delicious. Until now, most of the automated approaches: a) Query expansion using Delicious. This
classifiers rely on the content of the videos, both textual, approach uses Delicious tags to expand each label. We used
audio and video information. Nonetheless, the lack of repre- the DeliciousT140 Dataset 1 which is made up by 144.574
sentative data within many videos makes this classification unique URLs, all of them with their corresponding social
complex. In some cases, it may not be feasible to obtain an tags retrieved from Delicious on June 2008. This set of doc-
accurate representation for video resources in a genre tag- uments is annotated with 67.104 different social tags and
ging task. Transcripts are not usually representative with it has also useful frequency information. Firstly, we pro-
respect to specific tags as, e.g., ”comedy” or ”tutorial”. As cessed the dataset to find those tags co-occurring with each
a mean to solve these issues, social systems can provide an targeted label. For each label we generated a set of Deli-
easier and cheaper way to obtain useful information. cious tags ordered by use frequency. The size of these sets
First, we will introduce the related work, then we will was between 400 and 2000 co-occurring tags and they were
present the experimental setup and we will focus on the characterized by a long tail where only a few of them had
query expansion techniques applied. Then, we will describe consensus between the most Delicious users (i.e. highest
the submited runs and, finally, we will discuss the results. frequencies). Thus, the query expansion process needed to
filter and weight the obtained set in order to reduce the fi-
nal recall. For each Delicious tag set we selected only those
2. RELATED WORK with a frequency over the 10% of the highest tag frequency.
Social annotations have been widely used for the sake of Finally, and to provide more value to higher frequency tags
information management tasks. Before [8], there was lit- and also to the targeted label, we weighted the expanded
tle work dealing with the analysis of the applicability and query terms. b) Query expansion using KLD. As the
labels set was the same for the development and also for the
test phases, this approach expands each targeted label with
Copyright is held by the author/owner(s). 1
MediaEval 2011 Workshop, September 1-2, 2011, Pisa, Italy http://nlp.uned.es/social-tagging/delicioust140/
specific vocabulary extracted from the development set. We RUN MAP
used the Kullback-Leibler divergence (KLD) [2] to rank the 1 0.1103
terms occurring on the relevant video items for each label 2 0.1111
with respect to the whole video collection. To do this pro- 3 0.1850
cess we used the transcripts, the metadata and the user tags 4 0.1836
and we considered the subset of relevant videos as a large 5 0.2070
document to compute the divergence. As in the previous
approach, a filtering and weighting process was needed, we Table 1: Results of the submitted runs
selected only those terms with a weight over the 33% of the
highest term weight due to there were no significant differ-
ences (i.e., there was no long tail). Finally, we also weighted 6. CONCLUSIONS
the query terms in order to provide more value to the tar-
geted label and higher weighted terms. c) Combination In this paper we have presented an IR based approach
of Delicious and KLD. The idea behind is to filter those for automatic genre tagging. We have considered only the
Delicious tags that, although they could be adequate for a textual data associated with the video collection and we
big collection such as Delicious, they are out of the scope of have described several techniques to perform query expan-
the MediaEval collection. Thus, we used KLD to drive the sion based on Delicious social tags and language models.
query term expansion. More specifically, we intersected both According to the results, the use of social tags and also its
sets of terms and Delicious tags obtained from the previous filtering using specific vocabulary from the collection im-
approaches and then we applied a frequency based selection proves the final retrieval results.
and query terms weighting to build the query.
Acknowledgments
4. SUBMITTED RUNS This work has been partially supported by the BUSCAME-
DIA Project (CEN-20091026).
We submitted five different runs combining the different
approaches presented in section 3:
Run 1. We queried the retrieval system using a subset of 7. REFERENCES
Delicious tags. The video collection was represented consid- [1] D. Godoy and A. Amandi. Exploiting the social capital
ering only the video transcripts. of folksonomies for web page classification. In Software
Run 2. We also used Delicious as the query expansion Services for E-World, volume 341 of IFIP Advances in
method but in this case, the video collection was represented Information and Communication Technology, pages
and indexed using the metadata and also the video tran- 151–160. Springer, 2010.
scripts. [2] S. Kullback and R. A. Leibler. On information and
Run 3. We evaluated the same query expansion approach sufficiency. Annals of Mathematical Statistics, 22(1),
but considering only the video metadata and the provided 1951.
social tags in the representation and indexing phase. [3] M. Larson, M. Eskevich, R. Ordelman, C. Kofler,
Run 4. We queried the retrieval system using the KLD S. Schmiedeke, and G. Jones. Overview of MediaEval
expansion approach. The video collection was represented 2011 Rich Speech Retrieval Task and Genre Tagging
using the video metadata and the provided social tags. Task. In MediaEval 2011 Workshop, Pisa, Italy,
Run 5. We combined both query expansion approaches, September 1-2 2011.
KLD and Delicious tags . Again, in this approach the video [4] C. Lu, J.-r. Park, and X. Hu. User tags versus
collection was represented using the video metadata and the expert-assigned subject terms: A comparison of
social tags. librarything tags and library of congress subject
headings. Journal of Information Science,
5. ANALYSIS OF RESULTS 36(6):763–779, 2010.
Table 1 shows the results obtained by our runs according [5] J. Pérez-Iglesias, J. R. Pérez-Agüera, V. Fresno, and
to Mean Average Precission (MAP), which was the official Y. Z. Feinstein. Integrating the Probabilistic Models
measure for this task. Runs 1, 2 and 3 show how the repre- BM25/BM25F into Lucene. CoRR, abs/0911.5046,
sentation of the video collection using combinations of differ- 2009.
ent data impacts in the retrieval process. In these cases, the [6] J. Xia, K. Wen, R. Li, and X. Gu. Optimizing academic
representation of the video collection using its transcripts conference classification using social tags.
(runs 1 and 2) get the worst results and it is not suitable Computational Science and Engineering, IEEE
for retrieval using Delicious query expansion. On the other International Conference on, 0:289–294, 2010.
hand, runs 3 and 4 show a better performance, which in- [7] Z. Yin, R. Li, Q. Mei, and J. Han. Exploring social
dicates that the representation of the video collection using tagging graph for web object classification. In
semantic and social data such as video metadata and tags is Proceedings of the 15th ACM SIGKDD international
more adequate to the proposed query expansion approaches. conference on Knowledge discovery and data mining,
Finally, run 5 shows that the combination of Delicious social KDD ’09, pages 957–966, New York, NY, USA, 2009.
tags and specific vocabulary extracted using KLD performs ACM.
better than any of the previous runs. This combination helps [8] A. Zubiaga. Harnessing Folksonomies for Resource
to adjusts the candidate set of query terms to the language Classification. PhD thesis, Madrid, Spain, 2011.
model of each targeted label and improves the final retrieval Adviser-Victor Fresno and Adviser-Raquel Martinez.
performance.