<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta>
      <journal-title-group>
        <journal-title>May</journal-title>
      </journal-title-group>
    </journal-meta>
    <article-meta>
      <title-group>
        <article-title>Instance Spaces for Objective Assessment of Algorithms and Benchmark Test Suites</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Kate Smith-Miles</string-name>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>University of Melbourne</institution>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2019</year>
      </pub-date>
      <volume>4</volume>
      <issue>2019</issue>
      <abstract>
        <p>of Invited Presentation Objective assessment of algorithm performance is notoriously di cult, with conclusions often inadvertently biased towards the chosen test instances. Rather than reporting average performance of algorithms across a set of chosen instances, we discuss a new methodology to enable the strengths and weaknesses of di erent algorithms to be compared across a broader generalised instance space. Initially developed for combinatorial optimisation, the methodology has recently been extended for machine learning classi cation, and to ask whether the UCI repository and OpenML are su cient as benchmark test suites. Results will be presented to demonstrate: 1. how pockets of the instance space can be found where algorithm performance varies signi cantly from the average performance of an algorithm; 2. how the properties of the instances can be used to predict algorithm performance on previously unseen instances with high accuracy; 3. how the relative strengths and weaknesses of each algorithm can be visualized and measured objectively; and 4. how new test instances can be generated to ll the instance space and o er greater insights into algorithmic power.</p>
      </abstract>
    </article-meta>
  </front>
  <body />
  <back>
    <ref-list />
  </back>
</article>