The CLEF 2016 conference is the seventeenth edition of the popular CLEF campaign and workshop series which has run since 2000 contributing to the systematic evaluation of multilingual and multimodal information access systems, primarily through experimentation on shared tasks. In 2010 CLEF was launched in a new format, as a conference with research presentations, panels, poster and demo sessions and laboratory evaluation workshops. These are proposed and operated by groups of organizers volunteering their time and e ort to de ne, promote, administrate and run an evaluation activity.
tasks are: handover information extraction; multilingual Information extraction; and, patient-centred information retrieval.
ImageCLEF3 organizes three main tasks with a global objective of benchmarking automatic annotation, indexing and retrieval of images. The tasks tackle di erent aspects of the annotation and retrieval problem and are aimed at supporting and promoting cutting-edge research addressing the key challenges in the eld. A wide range of source images and objectives are considered, such as general multi-domain images for object or concept detection, as well as domain-speci c tasks such as labelling and separation of compound gures from biomedical literature and scanned pages from historical documents.
LifeCLEF4 proposes three data-oriented challenges related to this vision, in the continuity of the two previous editions of the lab, but with several consistent novelties intended to push the boundaries of the state-of-the-art in several research directions at the frontier of information retrieval, machine learning and knowledge engineering including: an audio record-based bird identi cation task (BirdCLEF); an image-based plant identi cation task (PlantCLEF); and, a sh video surveillance task (FishCLEF).
Living Labs for IR (LL4IR)5 inprovides a benchmarking platform for researchers to evaluate their ranking systems in a live setting with real users in their natural task environments. The lab acts as a proxy between commercial organizations (live environments) and lab participants (experimental systems), facilitates data exchange, and makes comparison between the participating systems. The task focuses on on-line product search.
News Recommendation Evaluation Lab (NEWSREEL)6 provides two tasks designed to address the challenge of real-time news recommendation. Participants can: a) develop news recommendation algorithms and b) have them tested by millions of users over the period of a few weeks in a living lab. The tasks are: benchmark news recommendations in a living lab; benchmarking news recommendations in a simulated environment.
Uncovering Plagiarism, Authorship and Social Software Misuse (PAN)7 provides evaluation of uncovering plagiarism, authorship, and social software misuse. PAN o ers three tasks at CLEF 2016 with new evaluation resources consisting of large-scale corpora, performance measures, and web services that allow for meaningful evaluations. The main goal is to provide for sustainable and reproducible evaluations, to get a clear view of the capabilities of state-ofthe-art-algorithms. The tasks are: author identi cation; author pro ling; and, author obfuscation.
Social Book Search (SBS)8 provides evaluation of real-world information needs which are generally complex, yet almost all research focuses instead on 3 http://www.imageclef.org/2016 4 http://www.imageclef.org/node/197 5 http://living-labs.net/clef-ll4ir-2016/ 6 http://www.clef-newsreel.org/ 7 http://pan.webis.de/ 8 http://social-book-search.humanities.uva.nl/ either relatively simple search based on queries or recommendation based on pro les. The goal of the Social Book Search Lab is to investigate techniques to support users in complex book search tasks that involve more than just a query and results list. The tasks are: a user-oriented interactive task investigating systems that support users in each of multiple stages of a complex search tasks; a system-oriented task for systems to suggest books based on rich search requests combining several topical and contextual relevance signals, as well as user pro les and real-world relevance judgements; and, an NLP/Text Mining track focusing on detecting and linking book titles in online book discussion forums, as well as detecting book search research in forum posts for automatic book recommendation.
Cultural Microblog Contextualization (CMC) Workshop9 aims at developing processing methods for social media mining. The focus is around festivals that are organized or that have a large presence on social media. For its rst edition, this workShop gives access to a massive collection of microblogs and urls and allows researchers in IR and NLP to experiment a broad variety of multilingual microblog search techniques (WikiPedia entity search, automatic summarization, and more).
Acknowledgments We would like to thank the members of CLEF-LOC (the CLEF Lab Organization Committee) for their thoughtful and elaborate contributions to assessing the proposals during the selection process: Martin Braschler, Zurich University of Applied Sciences, Switzerland Donna Harman, National Institute for Standards and Technology (NIST), USA Marteen de Rijke, University of Amsterdam, The Netherlands
Last but not least without the important and tireless e ort of the enthusiastic and creative proposal authors, the organizers of the selected labs and workshops, the colleagues and friends involved in running them, and the participants who contribute their time to making the labs and workshops a success, the CLEF labs would not be possible.
Thank you all very much!
Krisztian Balog Linda Cappellato
Nicola Ferro Craig Macdonald