The Search and Hyperlinking Task at MediaEval 2014 is the third edition of this task. As in previous versions, it consisted of two sub-tasks: (i) answering search queries from a collection of roughly 2700 hours of BBC broadcast TV material, and (ii) linking anchor segments from within the videos to other target segments within the video collection. For MediaEval 2014, both sub-tasks were based on an ad-hoc retrieval scenario, and were evaluated using a pooling procedure across participants submissions with crowdsourcing relevance assessment using Amazon Mechanical Turk.
The full value of the rapidly growing archives of newly produced digital multimedia content and digitalisation of previously created analog audio and video material will only be realized with the development of technologies that allow users to explore them through search and retrieval of potentially interesting content.
The Search and Hyperlinking Task at MediaEval 2014 envisioned the following scenario: a user is searching for relevant segments within a video collection that address a certain topic of interest expressed in a query. If the user nds a segment which is relevant to their initial information need expressed through the query, they may wish to nd additional information about some aspect of this segment.
The task framework asks participants to create systems
that support the search and linking aspects of the task. The
use scenario is the same as in the Search and Hyperlinking
task 2013 [
The dataset for both subtasks was a collection of 4021 hours of videos provided by the BBC, which we split into a development set of 1335 hours, which coincided with the test collection used in the 2013 edition of this task, and a test set of 2686 hours. The average length of a video was 2.1
The audio was extracted from the video stream using the mpeg software toolbox (sample rate = 16,000Hz, no. of channels = 1). Based on this data, the transcripts were created using the following ASR approaches and provided to participants:
(i) LIMSI-CNRS/Vocapia1, which uses the VoxSigma vrbs trans
system (version eng-usa 4.0) [
(ii) The LIUM system2 [
(iii) The NST/She eld system3 is trained on multi-genre
sets of BBC data that does not overlap with the collection
used for the task, and uses deep neural networks [
Additionally, prosodic features were extracted using the
OpenSMILE tool version 2.0 rc1 [
The computer vision groups at University of Leuven (KUL)
and University of Oxford (OXU) provided the output of
concept detectors for 1537 concepts from ImageNet5 using
different training approaches. The approach by KUL uses
examples from ImageNet as positive examples [
In order to create realistic queries and anchors for our
test set, we conducted a study with 28 users between aged
between 18 and 30 from the general public around London,
U.K. The study was similar to our previous study carried out
for MediaEval 2013 [
Users were then instructed to de ne queries that they expected to have more than one relevant video segment in the collection. These queries consisted of several terms, and were used as input to a standard online search engine, e.g. \sightseeing london". The study resulted in 36 ad-hoc search queries for the test set. The development set for the task consisted of 50 known-item queries from the MediaEval 2013 Search and Hyperlinking task.
Subsequently, as in the 2013 studies, we asked the
participants to mark so-called anchors, or segments they would like
to see links to, within some the segments that are relevant
to the issued search queries. The reader can nd a more
elaborate description of this user study design in [
For the 2014 task, as well as ad hoc search, we were interested in cross-comparison of methods being applied across all four provided transcripts: one manual and 3 ASR. Thus, we allowed participants to submit up to 5 di erent approaches or their combinations, each being tested on all four transcripts, for both sub-tasks. In case any of the groups based their methods on video features only, they could submit this type of run in addition as well.
To evaluate the submissions of the search and linking
subtasks a pooling method was used to select submitted
segments and link targets for relevance assessment. The top-N
ranks of all submitted runs were evaluated using
crowdsourcing technologies. We report precision oriented metrics, such
as precision at various cuto s and mean average precision
(MAP), using di erent approaches to take into account
segment overlap, as described in [
This work was supported by Science Foundation Ireland (Grants 08/RFP/CMS1677 and 12/CE/I2267) and Research Frontiers Programme 2008 (Grant 07/CE/I1142) as part of the Centre for Next Generation Localisation (CNGL) project at DCU, and by the funding from the European Commission's 7th Framework Programme (FP7) under AXES ICT-269980. The user studies were executed in collaboration with Jana Eggink and Andy O'Dwyer from BBC Research, to whom the authors are grateful. 6.