The MediaEval 2011 Rich Speech Retrieval Tasks and
Genre Tagging Tasks are two new tasks o ered in MediaEval
2011 that are designed to explore the development of
techniques for semi-professional user generated content (SPUG).
They both use the same data set: the MediaEval 2010 Wild
Wild Web Tagging Task (ME10WWW). The ME10WWW
data set contains Creative Commons licensed video collected
from blip.tv in 2009. It was created by the PetaMedia
Network of Excellence (http://www.petamedia.eu) in order to
test retrieval algorithms for video content as it occurs `in the
wild' on the Internet and, in particular, for user contributed
multimedia that is embedded within a social network. The
data set was initially described in [
The ME10WWW data set consists of video episodes from blip.tv and a social network comprised of Twitter users tweeting about them. Videos were collected for shows for which the link to one of their episodes had been tweeted. Topsy (http://topsy.com) was used to collect blip.tv links from tweets. Their licenses were checked to con rm that they were Creative Commons and then the videos were downloaded from blip.tv. Topsy was then searched again to gather all users mentioning any one of the videos. We crawled the tweets and social network of these users for two steps (i.e., collected their interlocutors and their interlocutors's interlocutors). The data set contains 1974 episodes (247 development and 1727 test) comprising a total of ca. 350 hours of data. The development set is small with respect to the test set and is not intended for training, but rather for parameter tuning. The episodes were chosen from 460 di erent shows|shows with less than four episodes were not considered for inclusion in the data set. Each video is associated with metadata record including uploader assigned information (title, description, license, tags, uploaded ID/series ID).
The data set is accompanied by automatic speech
recognition (ASR) transcripts [
The data set is also accompanied by the output of a shot
detection system developed at the Technische Universitat
Berlin [
The Rich Speech Retrieval task is a known-item retrieval task and requires participants to return a ranked list of results in response to a query. The features used can be derived from speech, audio, visual content or metadata. The task can be considered to be a `new generation' spoken content retrieval task in two respects. First, instead of requiring the identi cation of spoken documents or speech segments, the task requires the return of jump-in points, time points in the video at which users must start watching to view material relevant to the query. Second, the queries used express the user information need along three dimensions: the topical content, the speech act and the visual content.
The speech act dimension is, to our knowledge,
investigated for the rst time with this data set. When speakers
speak, they are, on the one hand, pronouncing words, but
on the other hand they are also actually `doing' something.
Treating spoken content in terms of `illocutionary speech
acts' (http://en.wikipedia.org/wiki/Speech acts) emphasizes
what speakers are accomplishing by speaking. The ve speech
acts used are `apology', `de nition', `opinion', `promise' and
`warning'. These categories are similar to the `speech-act
like' units that have ben used in dialogue act modeling [
The data set includes 30 queries associated with the development set and 50 queries associated with the test set. The form of queries includes a long form (<title>), a short form (<short_title>) and a label indicating the speech act type (act). An example of an apology is the following. Long form: `How does Peter Busch, sta member of the Morning Swim Show, save face after the faux pas he made during his interview with Terry Denison?'. Short form: `Peter Busch president chairman Denison morning Swim Show'. Although some queries in the data set make reference to visual characteristics (i.e., the speaker's clothing), most resemble this example and do not clearly need a visual contribution. Participants are required to make one submission for all test topics that uses only the provided ASR transcript (2010), and a second one using any combination of techniques which they believe will give the best overall performance. In the required run the participants must use the full queries.
The query set was created by having human annotators locate portions of the videos that they would be interested in sharing online and then formulating a comment on what the video portion was about (long form) and a query that would allow them to re- nd jump-in points corresponding to those portions (short form). Online-sharing was chosen as the scenario, since it seemed to be the most natural, widely-understood reason for which users would be attempting to re- nd previously seen video segments (i.e., known items). Access to a su cient number of human annotators was secured by using a crowdsourcing platform, Amazon's Mechanical Turk (http://www.mturk.com). The task was carefully designed, the context of online video-sharing was clearly described and the language kept simple, i.e., \When you come across something interesting you might want to share on Facebook, Twitter or your favorite social network." Examples were given of quotes falling into each of the speech act categories. We note that `opinions' were much more frequently located in the corpus than other categories such as `warning' or `promise'. The pairs of query+jump-in-point returned by the crowdsourcing workers were subjected to a standard quality control procedure (which eliminated ca. 40% of the results) and then by a screening for suitability and completeness (which eliminated a further ca. 40%).
The o cial evaluation metric of the task is mGAP, cf. [
Genre information in the form of genre tags can provide valuable support for users searching and browsing the Internet for video. However, much video|and especially SPUG video|is not accurately or adequately tagged. This task attempts to automatically generate genre labels such as they are used to organize videos on video platforms such as blip.tv. Genre tagging is related to the genre classi cation task set out by Google as an ACM Multimedia Grand Challenge task in 2009 and 2010.
The Genre Tagging task requires participants to automatically assign genre tags to videos using features derived from speech, audio, visual content or associated textual or social information (Twitter social network). The tag set users are required to predict contains 26 genre tags.1 Each video is associated with only one genre. Genre tags were collected for each episode using the API provided by blip.tv. A genre tag is represented by the eld categoryName in the JSON output provided by the API. Whitespace in the genre tags was replaced by the underscore character (` '), ampersands (`&') by the word `and'.
Participants submit up to ve runs in total representing ve di erent approaches to the task (i.e., experimental conditions). They are required to complete one run using ASR transcripts only and one run including metadata. Within the metadata, we assume that user-assigned tags might already contain explicit genre information. Since we are interested in algorithms that work under the (likely) event that no tags are available, one run is also required with metadata, but no tags. Upon request, groups with a particular focus were excused from speci c required runs. The o cial evaluation metric is MAP (Mean Average Precision). 5.
Thank you to Isabelle Ferrane (IRIT) who served as organizer of the Genre Tagging Task on the behalf of Quaero (http://www.quaero.org) and gave the task overview at the MediaEval 2011 workshop. Thank you to Thijs Verschoor (Twente) for his contribution to the RSR Task. This MediaEval e ort received funding from EC FP7 NoE PetaMedia and Science Foundation Ireland RFP 2008.
1art, autos and vehicles, business, citizen journalism, comedy, conferences and other events, default category, documentary, educational food and drink, gaming, health, literature, movies and television, music and entertainment, personal or auto-biographical, politics, religion, school and education, sports, technology, the environment, the mainstream media, travel, videoblogging, web development and sites