=Paper=
{{Paper
|id=Vol-2436/invited_2
|storemode=property
|title=None
|pdfUrl=https://ceur-ws.org/Vol-2436/invited_2.pdf
|volume=Vol-2436
|dblpUrl=https://dblp.org/rec/conf/sdm/Smith-Miles19
}}
==None==
<pdf width="1500px">https://ceur-ws.org/Vol-2436/invited_2.pdf</pdf>
<pre>
                              Instance Spaces for Objective Assessment of
                                Algorithms and Benchmark Test Suites

                                              Kate Smith-Miles∗
                                                  May 4, 2019



Abstract of Invited Presentation
Objective assessment of algorithm performance is notoriously difficult, with conclusions often inadvertently biased
towards the chosen test instances. Rather than reporting average performance of algorithms across a set of
chosen instances, we discuss a new methodology to enable the strengths and weaknesses of different algorithms
to be compared across a broader generalised instance space. Initially developed for combinatorial optimisation,
the methodology has recently been extended for machine learning classification, and to ask whether the UCI
repository and OpenML are sufficient as benchmark test suites.
    Results will be presented to demonstrate:

  1. how pockets of the instance space can be found where algorithm performance varies significantly from the
     average performance of an algorithm;
  2. how the properties of the instances can be used to predict algorithm performance on previously unseen
     instances with high accuracy;
  3. how the relative strengths and weaknesses of each algorithm can be visualized and measured objectively;
     and
  4. how new test instances can be generated to fill the instance space and offer greater insights into algorithmic
     power.




  ∗ University of Melbourne

</pre>