Individual tasks carried out within benchmarking initiatives, or campaigns, enable direct comparison of alternative approaches to tackling shared research challenges and ideally promote new research ideas and foster communities of researchers interested in common or related scienti c topics. When a task has a clear prede ned use case, it might straightforwardly adopt a well established framework and methodology. For example, an ad hoc information retrieval task adopting the standard Cran eld paradigm. On the other hand, in cases of new and emerging tasks which pose more complex challenges in terms of use scenarios or dataset design, the development of a new task is far from a straightforward process. This letter summarises our re ections on our experiences as task organisers of the Search and Hyperlinking task from its origins as a Brave New Task at the MediaEval benchmarking campaign (2011{2014) to its current instantiation as a task at the NIST TRECVid benchmark (since 2015). We highlight the challenges encountered in the development of the task over a number of annual iterations, the solutions found so far, and our process for maintaining a vision for the ongoing advancement of the task's ambition.
Benchmark evaluation campaigns have become a key
activity within a broad range of information processing
disciplines, having demonstrated their critical impact on the
elds' scienti c progress especially for the information
retrieval research community [
Our idea to de ne and shape an exploration of Search
and Hyperlinking (S&H) through a benchmarking activity
initially emerged from a diverse combination of reasons. A
number of varied and challenging large scale multimedia
data archives relevant to such a task were already becoming
available, while the constantly increasing and diverse
deluge of new multimedia content being produced, stored and
shared by both non-, semi- and professionals meant that
there was a compelling motivation to explore methods to
search and manage this content. At the same time,
scienti c advances had reached the stage where algorithms with
the potential to address more creative tasks that could
encompass known-item and ad hoc retrieval of speci c parts of
content, as well as personalised collection exploration, were
becoming available. Embarking on this adventure was also
appealing since various aspects of the overall S&H task had
already been investigated or tested in smaller scale tasks,
e.g. the MediaEval 2011 Rich Speech Retrieval (RSR) Task
[
From a starting point of a use case for a new task, the development of an actual benchmark activity often appears straightforward. However, this is often not the case, and once the task organisers begin to operationalize their ideas technical and practical challenges begin to emerge. This means that the task released to the participants is generally a technical and practical compromise, often containing hidden questions that the task organisers are unable to answer based on their current understanding of the user behaviour model or technical issues of the task. Thus the current instance of a task can itself be designed to answer these questions in order to move the task forward towards its ultimate research goals by exploiting better use case de nition and representation in a subsequent version of the task.
The S&H tasks were a classic example of this situation.
Once we began to examine the scope of what the task
required in terms of speci cation and implementation, we
realised that there were many questions to be addressed in
order to fully understand the task itself and how it should
best be implemented to benchmark the usability of its
outputs and the algorithmic contributions of the participants'
solutions. The activity thus began as a relatively small scale
Brave New Task at MediaEval 2011 [
In subsequent years the task received the status of a Main Task, meaning that we were able to gather a group of core participants (at least ve) who expressed their interest in participating each year. Being a Main Task did not mean that the task de nition and evaluation were set in stone, and thus, we kept experimenting with the collection, the type of users and their requests, evaluation metrics each year.
In 2014, we felt that the innovative Video Hyperlinking
subtask within the S&H task had reached a good level of
maturity in terms of task infrastructure, i.e., task de nition
[
Although we took the task to another venue, where most of the evaluation is usually done by NIST experts, we adhered to the crowdsourcing anchor creation and evaluation procedures that were established within our MediaEval activities. This approach preserved our exibility in terms of the creativity of the task de nition, and we kept our commitment to have users involved at all stages of benchmarking.
Traditionally, well established tasks with a straightforward scenario follow a pattern of gradually growing their dataset with each year iteration, using the same evaluation metric or a set of metrics, sometimes running the same software on the revised dataset, in order to be able to carry out direct comparison between the technology performance over the years. In the case of a more exploratory and innovative task, that is being developed through collaboration and feedback with participants, the same broad user scenario can be tested under di erent conditions, e.g. diverse target users of the potentially developed approaches, di erent data sets and evolving evaluation metrics that cover aspects of the task that could not have been foreseen beforehand.
When the task is de ned by a clear use case scenario existing within an industrial set up, the task can be promoted by these industrial partners via data provision and help with the on-site evaluation. As our research focus is on large video archives that are not always created and gathered with a clear monetization strategy in mind, often aiming at cultural heritage preservation (without prede ned usage scenarios), we were more free in de ning the framework. The feedback from the crowdworkers helped us to test algorithms addressing the task in a fast iterative way.
Another aspect that has to be taken into account when
setting up a task with a large data collection in mind is
the copyright question. When the task is in its initial early
development stage, it is easier to use a Creative Commons
dataset to initially test the task feasibility. This proof of
concept of the task viability allows the organisers to
demonstrate the soundness of the overall framework, and thus to
engage potential industry partners. This was the case for
the S&H task that started with the BlipTV collection [
On the other hand, crowdsourcing of the task de nition and results evaluation keeps the focus of the task on the user, and allows us to relate the scienti c methods under test to the current users technology expectations. This brings a practical insight into the impact of the performance improvements in algorithms on user experience. In a way, the workers become part of the organisers team, i.e., the task, although being envisaged by the scientists, is nally shaped and vetted by the real users. 5.
Having run the task already for two years at the TRECVid benchmark, we can compare our experiences and outline the di erences. At both venues at the initial stage of the yearly cycle, the tasks get feedback from the overall benchmark organisers committee in terms of task feasibility and interest within the targeted scienti c community. However, during the yearly cycle of actually running the task, within the MediaEval campaign the organisers of all the tasks are aware of task progress, raising issues and sharing their solutions via bi-weekly conference calls. This is especially helpful, when tasks are sharing the datasets, or when they are being run for the rst time and the organisers lack experience.
As organisers of a creative novel task, we found that interaction within the community of task organisers and with the actual task participants proved to be very useful to enable us to react quickly to any issues arising with the task, from the data release to submissions and evaluation release. However, TRECVid allows organisers to delegate some organisational activities to the NIST, thus saving time.
Running the benchmarking task requires a lot of commitment from the organisers, and an interest and engagement of the scienti c community. In our experience, the growing cycle in terms of interest and participation in the S&H task coincided with a number of related projects funded at a time which also meant that the ending of the funding cycle a ected the number of participants, while the actual scienti c ndings and discussions were still on an upwards path. The move to the TRECVid allowed us to involve large labs and companies that often participate in this venue.
We have presented the evolution of the S&H task to date.
Despite operating at two benchmarking venues, future
challenges remain, with the most critical issue being
sustainability [
This work has been partially supported by: ESF Research Networking Programme ELIAS; BpiFrance within the NexGenTV project, grant no. F1504054U; Science Foundation Ireland (SFI) as a part of the ADAPT Centre at DCU (13/RC/2106); EC FP7 project FP7-ICT 269980 (AXES).