In this paper, we present a social media based approach to nding anchors in video archives. We use social activity on Twitter to nd topics on which people have questions about in order to select suitable anchors. The experiments were carried out on the MediaEval Search and Anchoring in Video Archives Task (SAVA) data set, consisting of 68 hours of BBC video content broadcasted in 2008. The performance of our relatively simple, but straightforward method seems su ciently promising to pursue further research.
One of the research questions of the SAVA task, and the
one that is being addressed in this paper, is how to
automatically identify anchors for a given set of videos, where
anchors are media fragments for which users could require
additional information [
Microblog platforms, such as Twitter,1 re ect social
activity that takes place around a TV show at the time that it
is broadcast. Our approach is based on the idea that users
will want to learn further information on segments that
discuss topics that trigger questions. We use activity on
Twitter to nd which topics trigger user questions. The more
Twitter questions associated to topics discussed in a certain
shot, the greater we consider the likelihood that the
corresponding part of the video represents a viable anchor. Our
approach is further based on keyphrase mining. We
understand keyphrases in the sense of [
Our approach to anchor generation in broadcast videos exploits social chatter about topics on the microblogging platform Twitter. The method requires that subtitles and shot boundary information for a video are available. Anchors for a given video are then generated as follows.
For each subtitle s = (l; ts; te) 2 S consisting of a line of
text l, a start time ts, and an end time te, we extract set of
keyphrases Ks using nltk [
For each shot de ned by its boundaries b = (bs; be) 2 B, consisting of a start time bs and end time be, we introduce the notion of a subset Sb S representing all subtitles that start in that shot: Sb = fs j s = (l; ts; te) 2 S ^ ts 2 [bs; be)g. With these de nitions set in place, we can now de ne all keyphrases that occur in a shot b:
Kb = fk j s 2 Sb ^ k 2 Ksg :
To determine the importance of a keyphrase term k 2 K, we retrieve tweets containing a question about the keyphrase by sending the following query to Twitter: \k ? since:ds until:de". In this paper, we use a xed date range corresponding to the given set of videos for all keyphrases, regardless of the airing date of the video. This means we retrieve questions of social relevance during that general time period, not \the issues of today" in the past. Let q : K ! N denote the function to count the number of tweets retrieved for a keyphrase k 2 K. The weight of each keyphrase k is then determined by a weighing function w : K ! R. This function w is of the form w(k) = f (q(k)) where f is implementation-dependent and its purpose is to scale the tweet count returned by q.
We then rank shots by the summed weight of all keyphrases appearing in each shot. Let W : B ! R denote this summation: W (b) = Pk2Kb w(k). Then r : B ! N denotes the function that assigns a rank to a shot de ned by its boundaries b 2 B: r(b) = 1 +
b0 j b0 2 B ^ W (b0) > W (b) :
After ranking the shots, we simply generate anchors of an arbitrarily chosen minimum xed length of T = 30 seconds, using shot boundaries for alignment. The underlying assumption is that cutting at shot boundaries should result in clean media fragments. Let a : B ! R2 denote the function that computes the start and time of the anchor derived from a shot b 2 B. The start time is equal to the start time of the shot itself, i.e., bs, and the end time is equal to end time of the rst shot that ends at least T = 30 seconds later, or the end time of the whole video: a(b) = (bs; min fb0e j b0 2 B ^ bs + T 2 [b0s; b0e)g ! [ max fb0e j b0 2 Bg ): 3.1
The dataset used in the SAVA task is a subset of
collection of 4021 hours of video broadcasted by the BBC [
In this paper, we have tested two weighing functions, w1 and w2, each being submitted as a separate run in the benchmark. The rst function takes the popularity of each keyphrase in a shot into account, while the second function only considers the number of di erent keyphrases. They are dened as follows: w1(k) = w2(k) = (1
0 ((ln q)(k) if q(k) > 0 0 otherwise if q(k) > 0 otherwise
Furthermore, the keyphrase extraction used in our method only considered words of length 2 to 40 characters and ignored stopwords (using nltk's default stopwords list) and words that were written in all capital letters. The latter lter was used to ignore words appearing in descriptive subtitles for the hearing impaired, such as \APPLAUSE". This resulted in 37,154 nounphrase candidates for both dev and test set. To reduce the number of queries to be crawled, we pruned the list of keyphrases using the following heuristics: The phrase should contain at least one capital letter; The phrase may not start or end with a stopword; The phrase does not start with a quote;
The phrase does not contain periods or commas. The last two heuristics were put in place to deal with tokenization mistakes. Applying these pruning heuristics, we reduced the number of phrases from 37,154 to 5,663.
Querying Twitter for these 5,663 phrases for the period between 2008-04-01 to 2008-07-31 (corresponding to the original broadcast dates of the dataset) resulted in 66,934 tweets. We did not impose any language or geographic restrictions and issued queries as speci ed in Section 2. In this querying process we used a cut-o point to speed up the crawling process. After collecting more than 150 tweets for a query, the process was stopped. This means that in this experiment 0 q(k) 150. The software we used to unrestrictedly crawl Twitter is available on GitHub.2 3.3
Submitted anchors from all submissions task were pooled
and assessed by crowdsourcing workers on Amazon
Mechanical Turk [
From the results, we can see that run 1, which uses the w1 weighting function that assigns a higher weight to more popular keyphrases, appears to have a better precision and recall than run 2, which uses the w2 weighing function that treats all keyphrases equally. Run 2 on the other hand achieves a higher MRR. However, in both runs, the MRR never drops below 0.5, indicating that the rst relevant anchor is always amongst the rst two results. 3.4
It is of most interest to see when our proposed method performs best and when it does not. For this analysis we will look at the correlation between di erent performance metrics and video features.
Our rst observation is that P@10 and recall appear to be positively correlated, with Pearson's r being 0.42 (with p < 0:02) and 0.76 (with p 0:01) for run 1 and run 2, respectively. This suggests that some videos are easier and others are harder for our method to get right. We found that video length correlate positively with P@10 (0.56 and 0.54 for the respective runs) and that this correlation is signi cant (p < 0:01, for both runs). No conclusions could be drawn for the correlation between video length and recall, however. The answer to why our method seem to be in favor of longer videos could be found in the following two correlations. First, our method tends to perform better (P@10) when the unpruned list of di erent extracted keyphrases is longer (correlation of 0.58 (p < 0:001) and 0.51 (p < 0:01) for run 1 and 2, respectively). Second, we can extract almost undoubtedly more keyphrases from longer videos, as the correlation between these two features is signi cant (0.83 with p 0:001 for both runs). Our nal observation is that, interestingly, P@10 also seems to correlate with to something related to how we score individual segments, namely the sum of the summed keyphrases weights (W (b)) of the rst 10 results: 0.44 (p < 0:02) and 0.48 (p < 0:01) for run 1 and 2, respectively. 4.
We have explored a method for predicting anchors in television broadcasts by measuring interrogative activity on microblogs. Using the simplifying assumption that a shot is important if its subtitles contain keyphrases that appear in questions asked on a microblog platform, our method is able to achieve promising performance. Suggestions for future work include the following. We believe that netuning the keyphrase extraction process and looking at incorporating tf-idf could help with dealing with generic keyphrases that sometimes are extracted (e.g., \Good evening"). Furthermore, investigating the impact of narrowing and broadening a query's date range seems interesting to see what type questions are important and how to strike the balance between questions which are ephemeral and questions which are evergreen. Last but not least, we could look into personalization of the method, such as localizing the query to a geographic region.