Towards Boosting Video Popularity via Tag Selection Elizeu Santos-Neto Tatiana Pontes University of British Columbia Univ. Federal de Minas Gerais Vancouver, BC, Canada Belo Horizonte, MG, Brazil elizeus@ece.ubc.ca tpontes@dcc.ufmg.br Jussara Almeida Matei Ripeanu Univ. Federal de Minas Gerais University of British Columbia Belo Horizonte, MG, Brazil Vancouver, BC, Canada jussara@dcc.ufmg.br matei@ece.ubc.ca 1 Introduction Given the sheer volume content owners generate (e.g., Abstract YouTube receives 100 hours of video every minute1 ), it is common they offload online publication and moneti- zation tasks to specialized content management com- Video content abounds on the Web. Although panies. Content managers publish, monitor, and pro- viewers may reach items via referrals, a large mote the owner’s content, and revenues are generally portion of the audience comes from keyword- shared with the owner. As revenue is directly re- based search. Consequently, the textual fea- lated to the number of ad prints each piece of content tures of multimedia content (e.g., title, de- receives, this incentivizes managers to boost content scription, tags) will directly impact the view popularity. count of a particular item, and ultimately the Although viewers may reach a content item start- advertisement-generated revenue. ing from many leads (e.g., an e-mail from a friend or This study makes progress on the problem a promotion campaign in an online social network), a of automating tag selection for online videos large portion of viewers relies on keyword-based search with the goal of increasing viewership. It and/or tag-based navigation to find videos. An argu- brings two major insights: first, it describes ment supporting this assertion is the fact that, as of a methodology to construct a ground truth 2/Dec/2013, 14% of the unique visitors on YouTube to evaluate methods that aim to improve so- come from Google.com searches 2 . The integration of cial content popularity; second, it provides Google and YouTube search will likely increase the vol- evidence that the tags on existing YouTube ume of search traffic that leads to views on YouTube. videos can be improved by an automated tag Moreover, YouTube itself is the third most popular site recommendation process even for a sample on the web. of well curated videos; finally, it suggests a Consequently, the textual features of a video (e.g., roadmap to explore low-cost techniques either title, description, tags, and comments) have a ma- based on crowdsourcing or on tag recommen- jor impact on the view count of each particular item dation algorithms to improve the quality of and, ultimately, on the advertisement-generated rev- tags for online video content. enues [5, 11]. Similarly, in other contexts, it has been shown that even simple textual features produce pos- itive results: for example, title suggestions on eBay Copyright c by the paper’s authors. Copying permitted only have benefitted both sellers, who increased revenue, for private and academic purposes. and buyers, who found relevant products faster [5]. In: S. Papadopoulos, P. Cesar, D. A. Shamma, A. Kelliher, R. Jain (eds.): Proceedings of the SoMuS ICMR 2014 Workshop, 1 See: http://www.youtube.com/yt/press/statistics.html Glasgow, Scotland, 01-04-2014, published at http://ceur-ws.org 2 http://www.alexa.com/siteinfo/youtube.com#keywords Experts can produce the textual features associated different context than most previous studies: our goal with video content via manual inspection (and our is not to design novel and more efficient tag recom- industry contacts confirm this is a still current prac- mendation algorithms, but to study whether textual tice 3 ). This solution, however, is manpower intensive features of social content can be further optimize to and limits the scale at which content managers can op- improve the value of alternative data sources in pro- erate. Therefore, mechanisms to support this process viding tags to boost video popularity. In this sense, (e.g., automating tag suggestion) are desirable. the mainstream recommender algorithms we use here This study starts from the observation that, with provide a lower bound on the achievable quality. More the ever-increasing volume of user-generated textual complex algorithms, e.g., as proposed in [1, 3, 6, 8], can content available on the Web, there is a plethora of be tested and tuned using the methodology we propose sources from which an automated mechanism that sug- here to further improve tag quality. gests textual features, in general, and tags, in par- In particular, this study concentrates on the chal- ticular, could extract candidate terms that can im- lenges related to constructing a ground truth to en- prove multimedia content popularity. For example, able the evaluation information sources. Therefore, Wikipedia (a peer-produced encyclopedia), MovieLens we adopt the following two-part methodology: and Rotten Tomatoes (social networks where movie enthusiasts collaboratively catalog, rate, and anno- • Construct a ground truth by recruiting turkers tate movies), New York Times movie review section from Amazon Mechanical Turk, asking them to (which includes over 28,000 movies) or even YouTube watch YouTube videos, and to provide the key- comments are potential sources of candidate keywords words they would use to search for each of them, to annotate multimedia content such as videos. It as opposed to simply describe the video (see Sec- is important to note that techniques to suggest tex- tion 3); tual features to improve multimedia content popular- ity are not restricted to movies. In fact, other types • Prototype an automated tag recommendation of content such as superbowl ads could benefit from pipeline, incorporate various recommender algo- a combination of information sources such as humans rithms, and couple it with different input data from a crowdsourcing service (e.g., Amazon Mechani- sources (see Sections 2 and 4.1); cal Turk). To make progress on understanding whether textual • Evaluate the tag quality of existing YouTube information, tags in particular, associated with video videos by comparing them with the ground truth content can be improved through an automated pro- (see Section 5); cess, and on understanding what information sources In summary, the contributions of this work are: provide the most valuable textual features (i.e., terms that can potentially improve videos popularity), this • The production of a ground truth released to the work focuses on the following research questions: community. Q1 What are the challenges in building a ground truth to evaluate popularity boosting of videos on so- • Evidence that the tags associated with a sample cial media via textual features optimization such of trailers of popular movies currently available as tagging with terms that can potentially improve on YouTube can be further optimized by an au- the discoverability of the video via search? How tomated low-cost process. This process can ei- can one leverage crowdsourcing channels such as ther incorporate human computing engines (e.g., Amazon Mechanical Turk for such purpose? recruited through Amazon Mechanical Turk) at a much lower cost than using dedicated channel Q2 To what extent the tags currently associated with managers (the current industry practice), or, at existing video content on social video-sharing web- an even lower cost, can use tag recommender al- sites, such as YouTube, are optimized to attract gorithms to harness textual information from a search traffic? Is there room for improvement pos- multitude of data sources that are related to the sibly using automated tag recommendation solu- video content. tions? 2 Context of Our Assessment It is worth highlighting that, to tackle these ques- tions, this work uses tag recommender algorithms in a This section describes the context for our investiga- 3 This work is motivated by our collaboration with a company tion. Our main assumption is that annotating a video specialized in promoting video content. An NDA prevents the with the tags that match the terms users would use disclosure of details. to search for it increases the chance that users view single word frequency, word co-occurrence frequency). Note that there are many ways of defining scoring func- tions; and, it is not our goal to advocate a specific one, as we focus on the value provided by various informa- tion sources. We discuss the recommenders used in this work in Section 4.2. Figure 1: The recommendation pipeline. Knapsack Solver. Finally, after ranking candidate the video. This view is supported by previous stud- keywords, the final step is selecting the ones that best ies [11] and by the observation that a large portion fit the budget. In this paper, the budget is expressed of the traffic landing on a video comes from textual in terms of the number of characters, as done in video searches. As a result, textual sources that are related sharing systems such as YouTube, where one can use a to the video and whose content can be automatically limited number of characters (500) for tags. This step retrieved (e.g., movie reviews, comments, wiki pages, is formulated as a 0-1-knapsack problem. news items, blogs) can be used as inputs for recom- Let v be a video and C =< ki >, i = 1, ..., n, menders to suggest tags for these video content items. be a list of candidate keywords provided by a data A recommendation pipeline that implements this source when used as input to a tag recommendation idea is schematically presented in Figure 1: data algorithm. Additionally, let us denote the length of a sources feed the pipeline with textual input data. keyword ki as wi in bytes. Therefore, the problem of Next, the textual data is pre-processed by filters to selecting the best tags to improve viewership of v is both clean and augment it (e.g., remove stopwords, equivalent to solving the following optimization [2]: detect named entities). This first processing step n provides candidate keywords for the recommenders. X maximize f (ki , v)xi The recommendation step uses the candidate keywords i=1 (and their related statistics, such as frequency and co- Xn occurrence) to produce a ranked list according to a subject to wi xi ≤ B scoring function implemented by a given recommender i=1 algorithm. Finally, as the space available for tags pro- where B is the budget in terms of number of characters vided by video sharing websites, such as YouTube or allowed in the tags field, xi ∈ {0, 1} is an indicator Vimeo, is limited, the selection of most valuable can- variable, and f (ki , v) is a scoring function provided by didate keywords is constrained by a budget, often de- the recommender for the keyword ki with respect to fined by the number of words or characters. There- video v. fore, the final step consists of solving an instance of Considering that the cost5 (i.e., the keyword length) the 0-1-knapsack problem [2] that selects a set of rec- and the scores are both nonnegative, we use a well- ommended tags from the ranked-list produced by the known dynamic programming algorithm [2] to solve recommender. this optimization problem. In summary, the recommendation pipeline is com- Our goal is primarily to understand whether posed of four main elements: data sources, filters, videos currently published on a popular social video recommenders, and knapsack solver. The next para- sharing website have their tags optimized to attract graphs discuss each of these elements. search traffic. If tags can still be further optimized, Data sources. This component provides the input one could evaluate how the choice of the data source textual data used by the tag recommenders. In partic- used as input for a recommendation pipeline impacts ular, we are interested in peer-produced data sources the quality of the recommended tag-set. Next, we dis- such as Wikipedia and social tagging systems like cuss how to build a ground truth that enables test- MovieLens, as well as expert-produced data sources ing whether the tags assigned to a sample of videos such as NYTimes movie reviews. We discuss in detail available on a popular video sharing website are op- each of the data sources used in Section 4.1. timized. Additionally, such ground truth, in a future Filters. The raw textual data extracted from a data work, could enable the evaluation of potential data source is filtered to both clean and augment the input sources that provide candidate tags. data, minimizing noise. We consider simple filters: stopword and punctuation removal, lowercasing, and 3 Building the Ground Truth named entity detection4 . Our main assumption is that annotating a video with Recommenders. A recommender scores the candi- the tags that match the terms users would use to date keywords based on their relevant statistics (e.g., 5 Our study can be easily extended to consider the budget as 4 We use OpenCalais.com for entity detection. the number of tags (as in Vimeo). search for it increases the chance that users watch the video. As a result, textual sources that are related to the video and whose content can be automatically retrieved (e.g., movie reviews, comments, wiki-pages, news items, blogs) can be used as inputs for recom- menders to suggest tags for these video content items. The ideal method to collect the ground truth would consist of experiments that vary the set of tags as- sociated to videos, and capture their impact on the number of views attracted. However, collecting this re- quires the publishing rights for the videos and implies executing experiments over a considerable duration. Figure 2: Histogram with number of evaluations per- We decided for an alternative solution: we built a formed by turkers ground truth by setting up a survey using the Ama- zon Mechanical Turk (AMT)6 . The survey asks par- ticipants to watch a video and answer the question: What query terms would you use to search for this video? The rationale is that these terms would, if used as tags to annotate the video, maximize its retrieval by the YouTube search engine, and indirectly maximize viewership. Content Selection. Our study focuses on movie trailers7 . The reason is that they are often short (about 5 minutes or less), and this makes the evalua- tion process more dynamic, encouraging turkers (i.e., the AMT workers who accept to participate in the sur- Figure 3: Number of distinct keywords assigned by vey) to watch more trailers and associate more key- turkers to each video. words to them. The dataset consists of 382 videos selected to meet the following constraints: they must 6 minutes (total cost to conduct the survey: $345). be publicly available on YouTube and have available This leads to $3 per hour, which is much cheaper than content in the data sources used to extract candidate the wage paid to dedicated channel managers. keywords (e.g., a page on Wikipedia, a NYTimes re- We also performed simple quality control by in- view page). specting each answer to avoid accepting spam (which is Survey. First, we conducted a pilot survey by re- expected to be rare, due to the reputation mechanism cruiting participants via our internal mailing lists and adopted by AMT). In fact, only one submission was online social networks. Their task was to watch the rejected as unrelated URLs were assigned as answers trailers and provide terms they would use to search instead of keywords. for those videos. This pilot highlighted two major is- A brief characterization of the ground truth. In to- sues: (i) relying only on volunteerism to mobilize par- tal, 33 turkers submitted solutions. Figure 2 shows the ticipants was insufficient; and, (ii) quality control of number of videos evaluated per turker: as we can ob- answers (e.g., typos in the keywords) is much more serve, 19 turkers (58%) evaluated more than 5 videos, difficult - all videos in the survey are in English and with the maximum reaching 333 videos. Figure 3 there was no automatic way to recruit only partici- shows a histogram presenting the number of different pants that are fluent in English. keywords each video received. Even though we asked Thus, we published an AMT task8 requiring the the turkers to associate at least 3 keywords to each turkers to watch trailers, and provide the query terms video, 82% of the evaluations provided more than the (3 to 10 keywords to each video, as queries are typi- required minimum, which resulted in 96% of the videos cally of that length [4]) they would use to search for with 10 or more different keywords. the videos they had watched. Following AMT pay Figure 4 presents the total number of characters in guidelines, each participant was paid $0.30 per task the set of unique keywords associated to each video. assignment, which had an average completion time of The length of the ground truth varies from 51 (min) 6 http://www.mturk.com to 264 (max) characters; in fact, 32% of the videos 7 As long as there are data sources to extract candidate tags have tags summing up to 100 characters. These values from, other content types can benefit from our methodology. guided the budget parameter in our experiments, as 8 Links to data and code can be provided upon request. we explain in Section 4.3. movies a user may like to watch. For our evaluation, we use some of the data available in MovieLens: only the tags users produce while collaboratively annotat- ing and bookmarking movies. This data is a publicly available trace of tag assignments10 . Wikipedia is a peer-produced encyclopedia where users collaboratively write articles about a multitude of topics. Users in Wikipedia also edit and maintain pages for specific movies11 . We leverage these pages Figure 4: Histogram with the number of characters in as the sources of candidate keywords for recommend- bytes to each video. ing tags for their respective movie trailers from our sample. To gain an understanding on what types of key- NYTimes reviews are written by movie critics con- words would drive search traffic to these videos, we sidered experts on the subject. Similar to the data look at the set of most popular terms (overall) in provided by Wikipedia, we leverage the review page of the ground truth. Among the top 10% most fre- a movie as the source of candidate keywords for the quent search terms provided by turkers, 68% of them tag recommendation task. The reviews are collected are named entities (e.g., actor, director, and producer via the query interfaces12 provided by the NYTimes names). Another category of terms with strong pres- API. ence is genre-describing terms. This suggests that a Rotten Tomatoes is a portal where users can rate strategy that aim to boost popularity of videos by op- and review movies. Moreover, users have access to timizing the tags associated with the content should critics’ reviews and all credits information: actors and use sources that provide named entities related to the roles, directors, soundtrack, synopsis, etc. The portal video. links to critics’ reviews as well. The information about It is important to note that we found some evi- the credits of a movie and the critics’ reviews can be dence that this happens to videos other than movie considered as produced by experts (likely the film cred- trailer. In a smaller sample of Super Bowl ads videos, its are obtained from the movie producers, while the we observed that terms users provide as keywords they critics’ reviews are similar to those from NYTimes). would use to search for the ads are also dominated by While users can review the movies as well (and this named entities like the brand names. qualifies as peer-produced information), these reviews are available on the website, but not accessible via the 4 Experimental Setup API at the time of our investigation. The rest of the information about the movies together with links to This section presents the instances of data sources the experts’ reviews is available via the Rotten Toma- and tag recommenders, as well as the success metrics toes API13 . used in our evaluation on whether the existing tags on YouTube is a video-sharing website, here used to YouTube videos are optimized. test whether the tags already assigned to videos can be further optimized. To this end, we collect the tags 4.1 Data Sources assigned to the YouTube videos in our sample from the To understand whether the tags assigned to existing HTML source of each video’s page (API requests in online video content are optimized to attract search this sense are only accessible by the video publisher). traffic, it is necessary to compare the current tags to a The reason for using page scraping rather than API set of tags recommended by using other data sources requests is that videos’ tags are accessible via the API as input. only to the video publisher, even though these tags are Therefore, we use a combination of data sources as still used by the search engine to match queries and are inputs to recommender algorithms to produce a com- available in the HTML of the video page. YouTube parison basis for the existing tags on the videos in our data source figures in the expert-produced end of the sample. Next, we describe each of these data sources: spectrum since only the publisher can assign tags to MovieLens9 is a web system where users collabo- the video (it is reasonable to assume that a publisher ratively build and maintain a catalog of movies and is an expert on the own video and aims to optimize its their ratings. Users can create and update movie en- textual features to attract more views). tries, annotate movies with tags, review and rate them. 10 http://www.grouplens.org/taxonomy/term/14 Based on previous user activities, MovieLens suggests 11 http://en.wikipedia.org/wiki/Pulp Fiction (Film) 12 http://developers.nytimes.com 9 http://www.movielens.org 13 http://developer.rottentomatoes.com/ 4.2 Tag Recommenders The experiments use two tag recommendation algo- rithms that process the input provided by the data sources: Frequency and RandomWalk. We se- lected them primarily because they harness some fun- damental aspects of the tag recommendation problem that more sophisticated methods (e.g., [1, 6, 9]) also use: tag frequency, and tag co-occurrence patterns. Moreover, our goal is to understand the relative in- fluence of the data sources on the quality of the rec- ommended tags. We note that the methodology we describe and the ground truth can be used to evaluate other, more sophisticated, recommender algorithms as Figure 5: CCDF of F3-measure for YouTube tags and well. recommended tags. The Frequency recommender scores the candi- tag set, respectively, for video v. The metric is defined date keywords based on how often each keyword ap- as follows: pears in the data source. Given the movie title, our F3-metric. F3 (v) = 9·P 10·P (v)R(v) (v)+R(v) , where P (v) = pipeline finds the documents in the data source that |Tv ∩Sv | match the title and extract a list of candidate key- Sv and R(v) = TvT∩S are the precision and recall v v words. For example, in Wikipedia, the candidate key- of tag recommendation for video v, respectively. words for recommendation to a given movie are ex- tracted from the Wikipedia page about the movie, and 5 Experimental Results its frequency are the number of times each one appears This section presents our experimental results to ad- in that page. Similarly, in MovieLens, the frequency dress the following research questions: is the number of times a tag is assigned to a movie. To what extent the tags currently associated with ex- The RandomWalk recommender harnesses both isting YouTube content are optimized to attract search the frequency and the co-occurrence between key- traffic? Is there room for improvement using auto- words. The co-occurrence is detected differently de- mated tag recommendation solutions? pending on the data source. In MovieLens, two key- words co-occur if they are assigned to the movie by To address these questions, we perform an experi- the same user, while in NYTimes, Rotten Tomatoes, ment to assess the quality of tags already assigned to and Wikipedia two keywords co-occur if they appear existing YouTube videos and whether there is room in the same page related to the movie (i.e., review, for improvement. By improvement we mean extend- movie record, and movie page, respectively). The ing/modifying the tags to better match the ground RandomWalk recommender builds a graph based on truth. To this end, we compare the tags to the ground keyword co-occurrence, where each keyword is a node. truth for each video and observe a wide gap. The dotted (blue) line on the left in Figure 5 presents 4.3 Budget Adjustment the Complementary Cumulative Distribution Function (CCDF) for the F3-measure. A point in the curve in- To make the comparison fair, for each movie trailer, we dicates the percentage of videos (on y-axis) for which adjust the budget to the size of the tag set of that video the F3-measure is larger than the corresponding value in the ground truth. The knapsack solver uses this on x-axis, thus, the closer the line is to the top-right budget to select the recommended tags for a particular corner, the better. video. The reason for setting a budget per video is To explore whether the gap than can be covered, at that a number of recommended tags greater than the least partially, by automated tag recommendations, we ground truth size penalizes some evaluation metrics, explore the performance of the tag recommendations such as the F3-measure (see definition below). using as inputs all data sources combined (MovieLens, Rotten Tomatoes, Wikipedia, and NY Times). The 4.4 Success Metrics results are presented in Figure 5 as the solid (red) line. The final step in the experiment is to estimate, for each The Kolmogorov-Smirnov test of significance indi- video and for various input data sources and recom- cates that the performance of using All data sources is mender algorithms, the quality of the recommended significantly higher than that achieved by the YouTube tag-set against the ground truth. To this end, we use tags (Frequency: D− = 0.44, p-value = 3.9 × 1016 ; F3-measure. Let Tv and Sv be the set of distinct key- RandomWalk: D− = 0.43, p-value = 5.5 × 10−15 ) words in the ground truth and in the recommended implying that the tags recommended by both meth- ods are better than those currently assigned to the of a particular content item, and ultimately the adver- videos on YouTube. Therefore, the tags currently as- tisement generated revenue. Therefore, understanding signed to the YouTube videos can still be improved by the performance of automatic tag recommenders is im- automated methods to attract more search traffic, and, portant to optimize the view count of content items. hence, boost video popularity. First, we discuss the challenges on building a ground truth to evaluate data sources and techniques that aim 6 Related Work to boost the popularity of multimedia content on the web. Next, this study provides evidence that tags cur- The quest to improve visibility of one’s content (e.g., rently assigned to a sample of YouTube videos can be a website, a video) is not new – the whole Search En- further improved to attract more search traffic. To gine Optimization segment has seen uninterrupted at- this end, we show an experiment that compares how tention. Multiple avenues are available, ranging from close the tags currently assigned to the videos in the some that are viewed as abusive (e.g., link-farms) to sample and tags harnessed from a combination of data perfectly legitimate ones (e.g., better content organi- sources are to the ground truth. The results show that zation, good summaries in the titlebar of web pages). using simple recommenders and a combination of data Our exploration falls into this latter category. sources can improve the tags. The related literature falls into two broad cate- These preliminary results suggest a few directions gories: automated content annotation and tag value of future research. Initially, one may perform com- assessment. The majority of related work on auto- parisons between data sources individually and/or mated content annotation (or tag recommendation) grouped by type (peer- and expert-produced, struc- focuses on suggesting tags to annotate content items tured vs. unstructured) with the goal of understand- such that they maximize the relevance of the tag given ing their relative value as inputs for tag recommenders. the content [1, 5, 7, 9], with a few exceptions where For example, are combinations of peer-produced data authors propose to leverage other aspects such as di- sources relatively more valuable than expert-based versity [1]. ones in the context of boosting multimedia content Although finding relevant tags to a given content popularity? item is an important component of improving the tags Additionally, more experiments could provide assigned to this item, previous studies fail to account deeper explanations on the performance of peer- for the potential improvement on the view count of produced data sources. For instance, does the value of the annotated content – an aspect which is valuable tags extracted from peer-produced sources (for boost- to content managers and publishers, as they monetize ing content popularity), such as Wikipedia or Movie- based on the audience that is able to find their content. Lens, increase with the number of contributors? All The study presented by Zhou et al. [10] is, to these questions are part of our future efforts. the best of our knowledge, the closest to our work. However, contrary to our study that focus on testing whether tags can be further optimized to attract traf- References fic, Zhou et al. propose to boost video popularity by [1] F. Belém, E. Martins, J. Almeida, and suggesting ways to connect a video to other influen- M. Gonçalves. Exploiting Novelty and Diver- tial videos (e.g., making title and description similar sity in Tag Recommendation. In P. Serdyukov, to those of influential videos) as a way to leverage the P. Braslavski, S. Kuznetsov, J. Kamps, S. Rüger, related video recommendations. E. Agichtein, I. Segalovich, and E. Yilmaz, edi- Our study is different from these previous efforts, as tors, Advances in Information Retrieval SE - 32, it focuses on testing the hypothesis that textual fea- volume 7814 of Lecture Notes in Computer Sci- tures of social content, such as online videos, can be ence, pages 380–391. Springer Berlin Heidelberg, further optimized to potentially attract search traf- 2013. fic. This motivates our future work on evaluating the impact of data source choice to produce recommenda- [2] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and tions. C. Stein. Introduction to Algorithms. The MIT Press, third edit edition, July 2009. 7 Summary and Future Work [3] Z. Guan, J. Bu, Q. Mei, C. Chen, and C. Wang. A large portion of traffic received by video content on Personalized tag recommendation using graph- the web originates from keyword-based search and/or based ranking on multi-type interrelated objects. tag-based navigation. Consequently, the textual fea- SIGIR ’09, pages 540–547, Boston, MA, USA, tures of this content will directly impact the popularity 2009. ACM. [4] B. He and I. Ounis. Query performance predic- recommendation. In Proceedings of the third ACM tion. Information Systems, 31(7):585–594, Nov. international conference on Web search and data 2006. mining - WSDM ’10, page 81, New York, New York, USA, 2010. ACM Press. [5] S. Huang, X. Wu, and A. Bolivar. The effect of title term suggestion on e-commerce sites. WIDM [9] C. Wang, F. Jing, L. Zhang, and H. J. Zhang. ’08, pages 31–38, Napa Valley, California, USA, Image annotation refinement using random walk 2008. ACM. with restarts. MULTIMEDIA ’06, pages 647–650, Santa Barbara, CA, USA, 2006. ACM. [6] R. Krestel and P. Fankhauser. Language Mod- els and Topic Models for Personalizing Tag Rec- [10] D. Zhou, J. Bian, S. Zheng, H. Zha, and C. L. ommendation. In 2010 IEEE/WIC/ACM Inter- Giles. Exploring social annotations for informa- national Conference on Web Intelligence and In- tion retrieval. In 17th International World Wide telligent Agent Technology, pages 82–89, Toronto, Web Conference, pages 715–724, Beijing, China, AB, Canada, Aug. 2010. IEEE. 2008. ACM. [7] D. Liu, X. S. Hua, L. Yang, M. Wang, and H. J. [11] R. Zhou, S. Khemmarat, L. Gao, and H. Wang. Zhang. Tag ranking. WWW ’09, pages 351–360, Boosting video popularity through recommenda- Madrid, Spain, 2009. ACM. tion systems. In Databases and Social Networks on - DBSocial ’11, pages 13–18, New York, New [8] S. Rendle and L. Schmidt-Thieme. Pairwise in- York, USA, June 2011. ACM Press. teraction tensor factorization for personalized tag