=Paper=
{{Paper
|id=Vol-2167/keynote3
|storemode=property
|title=Balancing Efficiency and Effectiveness Trade-offs in Large Scale Multi-Stage Search Engines
|pdfUrl=https://ceur-ws.org/Vol-2167/keynote3.pdf
|volume=Vol-2167
|authors=J. Shane Culpepper
|dblpUrl=https://dblp.org/rec/conf/desires/Culpepper18
}}
==Balancing Efficiency and Effectiveness Trade-offs in Large Scale Multi-Stage Search Engines==
https://ceur-ws.org/Vol-2167/keynote3.pdf
Balancing Efficiency and Effectiveness Trade-offs in Large Scale
Multi-Stage Search Engines
J. Shane Culpepper
RMIT University
Melbourne, Australia
1 ABSTRACT Acknowledgements. This work was supported by the Australian
In this talk, we will discuss recent work on managing tradeoffs be- Research Council’s Discovery Projects Scheme (DP170102231) and a
tween efficiency and effectiveness in modern multi-stage ranking grant from the Mozilla Foundation.
architectures which are comprised of a candidate generation stage
followed by one or more reranking stages. In such an architecture, REFERENCES
[1] R.-C. Chen, L. Gallagher, R. Blanco, and J. S. Culpepper. 2017. Efficient cost-aware
the quality of the final ranked list is often sensitive to the quality of cascade ranking in multi-stage retrieval. In Proc. SIGIR. 445–454.
initial candidate pool. We will briefly discuss a few recent related [2] C. L. A. Clarke, J. S. Culpepper, and A. Moffat. 2016. Assessing efficiency–
papers from my group, and then discuss future directions. First, effectiveness tradeoffs in multi-stage retrieval systems without using relevance
judgments. Inf. Retr. 19, 4 (2016), 351–377.
we will explore dynamic cutoff prediction in early stage retrieval
[3] J. S. Culpepper, C. L. A. Clarke, and J. Lin. 2016. Dynamic cutoff prediction in
using query difficulty pre-retrieval features. We will then turn our multi-stage retrieval systems. In Proc. ADCS. 17–24.
attention to efficiency and effectiveness trade-offs in the later stage [4] D. X. de Sousa, S. D. Canuto, T. C. Rosa, W. S. Martins, and M. A. Gonçalves. 2016.
cascaded learning-to-rank algorithms. Specifically, we reexamine Incorporating risk-sensitiveness into feature selection for learning to rank. In
Proc. CIKM. 257–266.
the importance of tightly integrating feature costs into multi-stage [5] B. T. Dinçer, C. Macdonald, and I. Ounis. 2014. Hypothesis testing for the risk-
learning-to-rank (LTR) IR systems, and we present a novel approach sensitive evaluation of retrieval systems. In Proc. SIGIR. 23–32.
to optimizing cascaded ranking models which can directly leverage [6] C. Macdonald, R. L. T. Santos, and I. Ounis. 2013. The whens and hows of learning
to rank for web search. Inf. Retr. 16, 5 (2013), 584–628.
a variety of different state-of-the-art LTR rankers such as Lamb- [7] J. Mackenzie, F. M. Choudhury, and J. S. Culpepper. 2015. Efficient location-aware
daMART and Gradient Boosted Decision Trees. Finally, we discuss web search. In Proc. ADCS. 4.1–4.8.
interesting future research directions in multi-stage retrieval sys- [8] J. Mackenzie, J. S. Culpepper, R. Blanco, M. Crane, C. L. A. Clarke, and J. Lin.
2018. Query Driven Algorithm Selection in Early Stage Retrieval.. In Proc. WSDM.
tems as modern retrieval tasks continue to evolve towards more 396–404.
complex interactive search systems. [9] H. R. Mohammad, K. Xu, J. Callan, and J. S. Culpepper. 2018. Dynamic shard
cutoff prediction for selective search. In Proc. SIGIR. 85–94.
Biography. Associate Professor Shane Culpepper completed his [10] J. Pedersen. 2010. Query understanding at Bing. Invited talk, SIGIR (2010).
PhD in Computer Science at The University of Melbourne in 2008. [11] S. Peter, F. Diego, F. A. Hamprecht, and B. Nadler. 2017. Cost efficient gradient
He is currently a Vice-Chancellor’s Principal Research Fellow and boosting. In Proc. NIPS. 1551–1561.
[12] L. Wang, P. N. Bennett, and K. Collins-Thompson. 2012. Robust ranking models
Director for the Centre for Information Discovery and Data An- via risk-sensitive optimization. In Proc. SIGIR. 761–770.
alytics at RMIT University in Melbourne, Australia. His current [13] L. Wang, J. Lin, and D. Metzler. 2011. A cascade ranking model for efficient
research focuses on building search systems to effectively and effi- ranked retrieval. In Proc. SIGIR. 105–114.
ciently search web-scale data collections, and understanding how
to measure the quality of the answers found. Research interests
include efficient and scalable algorithm design, machine learning in
information retrieval, and system evaluation. For more information
about his research, visit his website at https://www.culpepper.io.
DESIRES 2018, August 2018, Bertinoro, Italy
© 2018 Copyright held by the author(s).