<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta>
      <journal-title-group>
        <journal-title>IIR</journal-title>
      </journal-title-group>
    </journal-meta>
    <article-meta>
      <title-group>
        <article-title>Does LambdaMART Do What You Expect?</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Discussion Paper</string-name>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Claudio Lucchese</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Federico Marcuzzi</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Salvatore Orlando</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Università Ca' Foscari Venezia</institution>
          ,
          <addr-line>Via Torino, 155, 30170 Mestre, Venezia VE</addr-line>
          ,
          <country country="IT">Italy</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2023</year>
      </pub-date>
      <volume>13</volume>
      <abstract>
        <p>Optimising information retrieval metrics is crucial in Learning-to-Rank tasks. One of the most efective approaches to do this is the well-known LambdaMART [ 1] algorithm. LambdaMART smoothly approximates for each document a gradient with respect to the IR metric to be optimised, the so-called lambdas. Intuitively, each lambda describes, with some degree of approximation, how much a document score should be pushed up or down to improve the ranking. In this work, we show that lambdas can be incoherent with respect to document relevance: a document with high relevance can receive a downward push larger than a document with lower relevance. This behaviour goes far beyond the expected degree of approximation. In addition, despite the gain in training eficiency, directly optimising truncated evaluation metrics can exacerbate this discrepancy due to less number of lambdas contributions each document gradient receives. As a consequence each document gradient is only partially computed and this lead to worse model learning. We analyse the idiosyncrasies of LambdaMART gradients and we introduce some strategies to remove or reduce gradient incoherencies. Specifically, we designed three selection strategies to compute the full gradient for only those documents that should be ranked in the top- positions of the ranking. We empirically demonstrate on publicly available datasets that the proposed approach leads to models that can achieve statistically significant improvements in terms of NDCG [2] while maintaining the same training eficiently as optimising truncated metrics. [1] C. J. Burges, From ranknet to lambdarank to lambdamart: An overview, Learning 11 (2010). [2] K. Järvelin, J. Kekäläinen, IR evaluation methods for retrieving highly relevant documents, in: E. J. Yannakoudakis, N. J. Belkin, P. Ingwersen, M. Leong (Eds.), SIGIR 2000: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July 24-28, 2000, Athens, Greece, ACM, 2000, pp. 41-48.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;learning to rank</kwd>
        <kwd>information retrieval</kwd>
        <kwd>machine learning</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body />
  <back>
    <ref-list />
  </back>
</article>