Experiment tracking systems log hundreds of runs with large numbers of (a) configuration parameters for the fine-tuning of the experiment and (b) a large number of metrics that assess the behavior of each configuration. However, the large numbers of runs makes it dificult to discover the trade-ofs of each configuration to its behavior, towards deciding the right configuration that satisfies an analyst's intent. We address this problem with an automated experiment analytics approach that (i) groups runs into behaviorally consistent clusters based on their observed metrics and (ii) generates compact, interpretable cluster-level summaries that connect characteristic metric outcomes to the configuration choices that tend to produce them. For each discovered behavior, the method produces actionable highlights, including representative metrics, salient metric relationships, and concise configuration rules that describe where the behavior occurs in the configuration space. Visualizations with radar charts and parallel coordinates provide an interactive means to highlight the pros and cons of each behavior with respect to its representative metrics and configuration. Experiments on several ML pipelines, show that the method yields strong cluster structure (Silhouette Score > 0.9) and stable descriptors using only 3-6 representative metrics per cluster, while remaining computationally practical with end-to-end runtimes on the order of seconds. An ablation study further shows that removing key components degrades at least one aspect of performance or interpretability, underscoring the importance of the overall approach.
In MLOps, machine learning pipelines are repeatedly executed under diferent configuration
settings, producing large collections of experiment run data annotated with configuration
parameters, metrics, and artifacts [
To support this process, practitioners rely on experiment tracking tools to log results,
maintain execution history, and compare runs. Systems such as MLflow [
In practice, a model is often evaluated under many combinations of configuration parameters. For example, some configurations may achieve high predictive performance but exhibit unfavorable fairness properties, while others ofer more balanced trade-ofs at the expense of peak accuracy. Figure 1 illustrates this situation: from many runs with configurations and heterogeneous metrics, distinct recurring behaviors emerge with characteristic trade-ofs (e.g., detection-oriented vs. fairness-oriented outcomes). Being able to explicitly identify such recurring configuration behaviors, and the regions of the configuration space in which they arise (e.g., simple conditions over max depth and estimators), together with their characteristic metric profiles in informative visualizations, (e.g., radar charts), would not only support optimization in the current experiment but also provide reusable insights to guide the design of subsequent experiments.
In this work, we introduce a clustering-based experiment analytics approach that derives interpretable summaries of ML experiment results at the level of configuration behavior. By making explicit how diferent regions of the configuration space are associated with characteristic metric outcomes and trade-ofs, the proposed method supports scalable reasoning over large experiment collections. The resulting summaries are compact and directly consumable through textual and visual representations, enabling comparison, exploration, and informed refinement of experiment designs.
This paper makes three main contributions. (1) We formalize the problem of analyzing large collections of ML experiment runs in terms of metric and configuration-level behavior, exposing the limitations of existing run-level analysis practices. (2) We introduce an automated experiment analytics approach that clusters runs in metric space and produces compact, interpretable summaries that relate characteristic metric outcomes to the configuration regions in which they arise, enabling scalable reasoning over complex experiment spaces. (3) We demonstrate, through an experimental evaluation on multiple realistic ML pipelines, that the proposed approach produces stable, discriminative, and interpretable summaries that support efective comparison and refinement of experiment designs.
The remainder of the paper is structured as follows: Section2 discusses related work. Section 3 defines the problem. Section 4 details the methodology we follow and the overall pipeline. Section 5 presents the experimental evaluation, and section6 the conclusions.
Complex scientific and ML experiments are supported by tools for managing workflows, tracking runs, and analyzing performance across many executions. Related work spans automated highlight extraction and exploratory data analysis, workflow profiling and analysis, interpretable clustering, and experiment tracking and workflow management systems. We outline the most relevant lines of work and clarify how our approach compares to them.
Automated Highlight Extraction. The current work is related to the field of automated
highlight extraction, where data sets are evaluated for ’hidden gems’, i.e., the existence of
interesting properties (trends, outliers, correlations, etc) hidden in subsets of a data set. Frequently,
the term used is insight. A general model as well as a survey of related work is found in5][.
Several tools have been proposed in the both the database and the visualization literature for
helping analysts understand the hidden patterns in the data as well as assessing the importance
of such findings with dedicated scores. The literature can be traced all the way to the distant
past (most notably [
Workflow Profiling and Analysis. A substantial body of work analyzes workflows to
understand performance and resource usage, by monitoring runtime, memory, I/O behavior
and provenance across many executions and using these data to detect bottlenecks or
anomalies [
Interpretable & Explainable Clustering. Interpretable clustering methods aim to couple
unsupervised grouping with human-understandable descriptions of clusters, for example by
learning rule sets, decision trees, or prototype-based characterizations that explain why points
are assigned to a given cluster [
Experiment Tracking and ML Workflow Management Systems.
ML-native workflow
systems (e.g., ZenML, Kubeflow) orchestrate end-to-end ML pipelines and difer from
generalpurpose engines (e.g., Airflow) by providing ML-specific metadata and experiment
management [
define the configuration and metric spaces as the Cartesian products Data Representation. Let 1, … , denote the configuration parameters (e.g., max_depth, n_estimators, algorithm choices), and le t 1, … , denote the logged metrics (e.g., accuracy, recall, fairness scores, CPU usage, memory consumption). Each parameter ℓ and metric has an associated domain of admissible values, denoted dom( ℓ) and dom( ), respectively. We X = dom( 1) × ⋯ × dom( ),
M = dom( 1) × ⋯ × dom( ), and each metric vector m ∈ M assigns one value to every metric. so that each configuration vector x ∈ X assigns one value to every configuration parameter, We consider a collection of experiment executions (runs), E = {(x , m )}=1 ⊆ X × M, where x ∈ X and m
∈ M are the configuration and metric vectors of run , respectively.
Stacking configurations and metrics row-wise yields the data matrices X and M, which we refer to as the configuration matrix and the metric matrix, respectively, that may contain both continuous and categorical columns.
Clustering. A clustering solution overE is defined as a complete and disjoint partition C = { 1, … , } of the run indices {1, … , } , with an assignment function ∶ {1, … , } → {1, … , } .
Given the logged experiment runsE , we extract their configuration parameter and metric matrices X and M respectively. Our goal is to construct a structured summary of the experiment space in the form of a clusteringC over the runs, together with associated visual highlights that can be presented to practitioners within existing experiment tracking tools.
Problem Decomposition. Addressing this objective involves two main tasks: 1. Experiment Partitioning. The first task is to compute a clustering solution C. Functionally, each cluster is intended to represent a distinct operating behavior of the experiment, grouping runs that exhibit similar metric profiles (e.g., high-accuracy/high-cost vs. low-accuracy/low-cost) to facilitate comparison. In this work, clustering is performed purely on the metricsM, while configuration parameters X are used downstream to construct interpretable descriptors that explain cluster membership. 2. Descriptor and Highlight Extraction. For each cluster , we derive a set of interpretable descriptors and visual artifacts to explain its behavior: • Representative Metrics ( ): A subset of metrics ⊆ { 1, … , } that most strongly distinguish from other clusters. • Metric Relationships ( ): A set of salient dependencies (e.g., trade-ofs or correlations) between metrics within . • Configuration Rules ( ): A small set of human-readable logical rules over the configuration variables 1, … , that approximate membership in (e.g., simple conjunctions of threshold and equality predicates). • Auxiliary Statistics ( ): Quantitative properties of the cluster, such as relative size, stability, and discriminativeness scores. • Visual Highlights ( ): Concrete visual artifacts (e.g., radar charts, parallel coordinate plots) that instantiate the above descriptors for user inspection.
Desiderata. A satisfactory solution should meet three complementary criteria:
1. Cluster Validity: The partition C should be structurally sound, maximizing internal
cohesion and external separation (e.g., as measured by the Silhouette Coeficient) to ensure
operating behaviors are distinct [
Metric Filtering outliers
ENnocrmodailnizga&tion z-norm RDeimdu.ction task: remove metrics out: subset of metrics M
This section describes our method for organizing experiment runs into clusters and deriving
cluster-level highlights over configuration parameters and metrics. Following3[
Figure 2 provides an overview of the analysis pipeline and an illustrative example with two resulting clusters. The top-left panel shows the input runsE , including their logged configurations X (light red) and outcome metricsM (green), while the bottom-left panel presents the resulting cluster summaries as textual and visual highlights. The right side outlines the two main components of our method: a clustering component, which performs metric filtering, normalization, dimensionality reduction, and -means clustering in metric space, and a highlightextraction component, which derives representative metrics, relationships, and configurationlevel rules for each cluster. In this toy example, runs 1–3 form a detection-oriented cluster, characterized by higher recall and Area Under the Curve (AUC) but worse fairness metrics, whereas runs 4–6 form a fairness‑oriented cluster exhibiting the opposite pattern.
The clustering component transforms the logged configuration and metric data into a numerical representation suitable for clustering and explanation, and then applies-means clustering to obtain a partition C of the runs.
Variance-based metric filtering. To focus the analysis on informative metrics, we apply
two filters based on the coeficient of variation (CV). For every metric , we compute its
sample mean and standard deviation over{1, … , } and derive CV( ) as their ratio. We remove
metrics with CV( ) < 0.05 to eliminate efectively constant features that lack discriminative
power [
Encoding and normalization. Both X and M may contain continuous and categorical
columns. We decomposeM = [Mcont ∣ Mcat] and X = [Xcont ∣ Xcat], where the subscripts
denote continuous and categorical parts, respectively. The continuous matricesXcont and Mcont
are standardized column-wise to zero mean and unit variance, while the categorical matrices
Xcat and Mcat remain nominal and are handled via correspondence analysis in the subsequent
metric-space embedding. For the model-based analyses in Section4.2.1 and Section 4.2.2, we
additionally construct encoded matricesM′ and X′ by one-hot encoding all categorical columns
in Mcat and Xcat and concatenating them with their standardized continuous counterparts.
Metric-space embedding. We perform dimensionality reduction separately on the distinct
metric types. Following [
Z using -means, exploring a range of cluster counts ∈ [ min, max] (in our experiments,
min = 2 and max = 9) and set the number of clusters to the that maximizes the silhouette
score [
For each cluster in the fixed partition C, we derive descriptors and assemble experiment highlights by selecting representative metric s and their relationships , extracting configuration rules that approximate cluster membership, and compiling textual and visual summari e s .
For each cluster , our goal is to select a small, non-redundant set of metric s that best distinguish runs in from the rest of E . We operate on the encoded metric matrix M′ and treat the problem as a series of binary classification tasks.
Selection of representative metrics. For each cluster , we iteratively select
representative metrics from the encoded matrix M′. In each of ⌊ /2⌋ iterations (with the number of
logged metrics), we set up a binary classification task ( vs. E ∖ ), labeling runs in as
positives and all others as negatives, and train a random forest classifier to separate the two classes.
We then compute the SHAP scores [
To avoid selecting redundant metrics, we group together strongly correlated candidates and
keep only one representative per group. For metrics and within , we define a type-aware
dependence measure (, ) ∈ [
using Kendall correlation for continuous–continuous
pairs, the square root of partial 2 for categorical–continuous pairs, and normalized mutual
information for categorical–categorical pairs [
For each metric that is ultimately selected as representative, we additionally record how its typical value in compares to the corresponding typical values in the other clusters by computing a cluster-leve l -score: metrics with a -score ≤ −1 are labeledlow, those with a -score ≥ 1 are labeledhigh, and those in between are labeledmid. In Figure 2, this procedure identifies
recall and calibration as representative metrics for the two clusters: for Cluster 1, recall is labeledhigh and calibrationlow, while Cluster 2 exhibits the opposite pattern.
To quantify how well the selected metrics separate from the stored as part of the auxiliary statistics .
remaining runs, we train an XGBoost classifier on the reduced encoded metric matrix M′ for the
same binary task ( vs. E ∖ ). We use an 80/20 stratified train–test split, perform 5-fold
crossvalidation on the training set, and evaluate on the held-out test set. The discriminativeness of
cluster is summarized by a scalar score = 0.6⋅AUC +0.4⋅F1 , defined by the classifier’s area
under the ROC curve (AUC) and F1-score. This scalar score
∈ [
We summarize relationships between the metrics in using the dependence measure (⋅, ⋅). The groups of previously computed correlated companions of each representative metric in , are used directly as part of the cluster summary, providing alternative but closely related views on the same underlying behavior. In addition, for each cluste r we compute pairwise relationships between the metrics in to detect strong negative correlations that indicate potential trade-ofs. Metric pairs with (, ) ≤ − high (e.g., high = 0.75) are recorded as cluster-specific trade-ofs between competing objectives such as performance and resource usage. The resulting collection of correlated companion groups and trade-ofs constitutes the metric-relationship summary for cluster . In the illustrative example, Cluster 1 exhibits a strong positive correlation between recall and AUC, at the expense of disparate impact ratio and calibration, whereas Cluster 2 exhibits the opposite correlation profile.
While metric-level summaries describe how runs in a cluster behave, practitioners also need to understand which configuration patterns give rise to that behavior. To this end, we derive human-readable rules over the set of parameters, approximating membership in each cluster. Rule extraction and scoring. Similar to our method in Section 4.2.1, we define a binary classification problem on the configuration space for each cluster (runs in vs. E ∖ ) using the encoded configuration matrix X′. We then learn an interpretable rule-based classifier by training an ensemble of shallow decision trees for this task and extracting those decision paths that attain suficiently high precision and recall for the positive class.
Each selected path is converted into a human-readable logical rule, that is, a conjunction of simple predicates on , that dictate membership in . For each candidate rule , we compute its F1-score F1 ( ) for predicting membership in together with its relative coverage ( ) = ( ) / | |, where ( ) is the number of runs in that satisfy . We then define the rule-quality score as ( ) = ( ) ⋅ F1 ( ), and retain as configuration-level descriptors only those rules with ( ) ≥ 0.75 . The configuration-level descriptor for cluster is the resulting small set of top-scoring rules, denote d , which serves as an interpretable summary of the configuration behaviors associated with the cluster’s metrics. In the illustrative example, the top-scoring rule for Cluster 1 ismax_depth ≤ 5, whilemax_depth = 20 defines Cluster 2.
Given the metric- and configuration-level descriptors for each cluster , we assemble a compact set of visual highlights . Each highlight card pairs a textual summary with coordinated visualizations, providing both an overview and detailed views of the cluster’s behavior and conifguration patterns. An example for Cluster 1 is shown in Figure 2, where the card highlights high recall and low calibration, the rule omnax_depth, and the representative run 2.
The textual summary reports the cluster size| |, proportion , and score , which is discretized into a quality label (e.g.e,xcellent, good, moderate, poor). It then summarizes the representative metrics in , enumerates key relationships in , and lists the top-scoring configuration rule in . Finally, the card highlights a single representative run fro m (e.g., the cluster medoid inZ) as a concrete example of a typical configuration and outcome profile. To complement the summary, we generate three coordinated views per cluster. We presen t (and companions ) as an interactive radar chart, allowing the user to toggle between views that distinctively show metrics where the cluster attains notablyhigh, mid, or low values. A graph view visualizes correlations in , scaling edges by (, ) , while a parallel-coordinates plot maps parameters to metrics , highlighting runs matching the top rule in . In all views, non-selected clusters (or runs) are rendered as a desaturated, low-opacity gray backdrop, while the focused cluster is overlaid in color with higher opacity and thicker strokes, preserving global context while emphasizing cluster-specific patterns.
The goal of our evaluation is to assess the proposed method’s performance across three key dimensions: the structural quality of discovered clusters, the stability of representative metrics, and the interpretability of the generated rule-based explanations.
Three representative ML pipelines were executed to generate experiment runs for evaluating
our method. We first tested an income classification pipeline on the Adult dataset [ 46].
Following [
We evaluate our pipeline across four dimensions: (i)Cluster Quality, using Silhouette Score
(SiS) and Davies–Bouldin Index (DBI)—the most commonly adopted internal validation
metrics throughout the literature [
This work was implemented in Python 3.11. The experiments were executed on a MacBook Pro M3 1. Source code and reproducibility instructions can be found on Github2. 1https://support.apple.com/en-us/117735 2https://github.com/billstam12/workflow-insight-extraction
To evaluate our algorithmic pipeline and quantify the contribution of each component, we perform an ablation study. We consider four configurations per pipeline: the full method (Full); No Dim. Reduction (NDR), which omits PCA/MCA and clusters directly in the original metric space; No Variance Filter (NVF), which keeps all metrics before clustering regardless of variance; and No Iterative Selection (NIS), which uses the same clustering steps as FULL but replaces iterative representative selection with a single-pass SHAP ranking. Table2 reports, for each pipeline and ablation setting, the number of cluster|sC |, the average number of representative metrics | |, internal clustering indices (SiS and DBI), representative-metric variability (CV), rule-based explanation quality (QSE), and total execution time; rows for FULL are shaded in blue, and best values within each pipeline are typeset in bold.
Across all pipelines, FULL yields the strongest cluster structure: Inc and Wine achieve average SiS above 0.90 with the lowest DBI (around0.14), while Taxi attains SiS≈ 0.80 with DBI ≈ 0.23. In contrast, NDR consistently collapses cluster quality (SiS close 0toand DBI up to 8.07 for Wine), and NVF either inflates the number of weak clusters (e.g., Taxi with = 17 ) or worsens SiS/DBI. The NIS variant operates on the same cluster assignments as FULL and therefore matches its SiS and DBI, but—as discussed next—difers substantially in representative quality.
FULL provides the most stable and compact set of representatives, especially on Inc, where it uses 5 representative metrics and attains the smallest CV-ratio values (average rati≈o 0.03 with a very narrow range), indicating that representatives vary far less than non-selected metrics within each cluster. For the rest of the pipelines, FULL also balances the number of representatives (5 on average) with low or moderate CV-ratios, whereas ablations either fail to define meaningful representatives (NDR), substantially increase the CV-ratio (NIS), or require more representatives to describe the same clusters making the process cognitively unmanageable.
Rule-based explanations are consistently strongest under FULL, which achieves the highest QSE across pipelines. Ablations either sharply reduce QSE (especially NDR and NVF) or, in the case of NIS, match FULL’s QSE as they use the same cluster assignments; proving that dimensionality reduction and variance filtering are crucial in producing interpretable clusters. 5.2.4. Runtime
FULL remains computationally practical, completing in around 10sec. across pipelines. In contrast, NVF is consistently the slowest variant (Inc: 17.30s, Taxi: 30.30s, Wine: 13.45s), indicating that retaining all metrics substantially increases end-to-end cost. NIS and NDR are the fastest, but FULL is within roughly 2 seconds while delivering substantially better representative stability (lower CV ratios) and strong explanation quality.
Overall, the full configuration is the only one that consistently balances (i) well-formed clusters, (ii) stable and compact representative metrics, and (iii) high-quality rule explanations across all three pipelines. The ablations confirm that dimensionality reduction and variance filtering are important for maintaining meaningful structure and interpretability, while replacing iterative selection degrades the faithfulness/compactness of representatives even when cluster assignments are unchanged. End-to-end runtime remains practical for Full, and retaining all metrics (NVF) is consistently the most expensive setting.
This paper presented an approach that organizes large collections of experiment runs into metric-space clusters and, for each cluster, derives compact textual and visual highlights over representative metrics, metric relationships, and configuration-level rules. The pipeline first iflters and embeds metrics and applies -means clustering to obtain compact, well-separated operating regimes. It then identifies representative metrics using iterative SHAP-based feature extraction, learns configuration rules from shallow decision trees, and assembles chart- and text-based highlights. Experiments on three ML pipelines showed that the proposed method attains high clustering quality (Silhouette Score> 0.9) while using only a handful (3–6) of representative metrics per cluster and short rules with few predicates in reasonable time; and that removing dimensionality reduction, variance filtering, or iterative selection degrades at least one of these properties. Future work includes extending our method with more clustering algorithms, and cluster quality metrics like Dunn Index and WCSS50[, 51].
This research is implemented in the framework of H.F.R.I call “3rd Call for H.F.R.I.’s Research Projects to Support Faculty Members & Researchers” (H.F.R.I. Project Number: 23640).
The author(s) have not employed any Generative AI tools. [45] D. A. Skoog, F. J. Holler, S. R. Crouch, Principles of Instrumental Analysis, Cengage Learning, 2017. Discusses Signal-to-Noise Ratio and precision limits. [46] B. Becker, R. Kohavi, Adult, UCI Machine Learning Repository, 1996. DOI: https://doi.org/10.24432/C5XW20. [47] S. Aeberhard, M. Forina, Wine, UCI Machine Learning Repository, 1992. DOI: https://doi.org/10.24432/C5PC7J. [48] New York City Taxi and Limousine Commission, TLC Trip Record Data, https://www.nyc.
gov/site/tlc/about/tlc-trip-record-data.page,2024. Accessed: 2025-15-12. [49] D. L. Davies, D. W. Bouldin, A cluster separation measure, IEEE Transactions on Pattern
Analysis and Machine Intelligence PAMI-1 (1979) 224–227. [50] J. C. Dunn, Well-separated clusters and optimal fuzzy partitions, Journal of Cybernetics 4 (1974) 95–104. [51] B. S. Everitt, S. Landau, M. Leese, D. Stahl, Cluster Analysis, 5th ed., Wiley, 2011.