=Paper=
{{Paper
|id=Vol-2441/keynote1
|storemode=property
|title=None
|pdfUrl=https://ceur-ws.org/Vol-2441/keynote1.pdf
|volume=Vol-2441
|dblpUrl=https://dblp.org/rec/conf/iir/Harman19
}}
==None==
https://ceur-ws.org/Vol-2441/keynote1.pdf
The Evaluation Campaigns: Past, Present and Future
(Invited Talk)
Donna Harman
donna.harman@nist.gov
Scientist Emeritus, National Institute of Standards and Technology, USA
1 ABSTRACT with evaluations in 2019 tackling problems of tracking emergency
Evaluation has always been a critical component of information situations by following tweet streams, identifying birds by their
retrieval, and there have been some type of shared evaluations since calls, or working with Lifelogs.
the 1960s. The Cranfield test collection was used by multiple groups,
starting with Gerard Salton in the 60s, and then by researchers at 2 BIOGRAPHY
the University of Cambridge during the 1970s. But different versions Donna Harman graduated from Cornell University with a degree
of the collection were used and there was little attempt to compare in electrical engineering, and having worked with Professor Ger-
results across systems. The creation of the large TIPSTER collection ard Salton, has been involved with research in new search engine
in 1990, followed by the first Text REtrieval Conference (TREC) in techniques for many years. She retired from the National Institute
1992 reframed the shared concept to mean not only using the same of Standards and Technology in 2005 after leading a group that
test collection, but also having a specific shared task, which in 1992 worked in the area of natural language access to full text. In 1992
was an adhoc search tasks for 50 topics. Researchers could compare she started the Text REtrieval Conference (TREC), a still-ongoing
systems, and then incorporate what was jointly learned into their forum that brings together researchers from industry and academia
own systems. This paradigm grew in TREC to encompass new to test their search engines against common corpora. She received
community information retrieval tasks, such as question answering the 1999 Strix Award from the UK Institute of Information Scientists
and working with web data. It also branched into other new areas, for this effort. She is currently a scientist emeritus at NIST and is
such as video retrieval (which was spun off into TRECvid), and cross- the author of two textbooks: Information Retrieval Evaluation and a
language retrieval, which led to the formation of the European CLEF new history book, Information Retrieval: the Early Years.
in 2000. Other shared evaluations like NTCIR in Japan and FIRE in Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons
India were organized, each targeting retrieval tasks most pertinent License Attribution 4.0 International (CC BY 4.0).
IIR 2019, September 16–18, 2019, Padova, Italy
to their research communities. All of these evaluations have evolved
over the years as the interests of the research groups have changed,
1