Machine learning (ML) has transformed healthcare, improving diagnostics, treatment, research, and patient care. However, clinical decision support (CDS) still relies heavily on classical statistical models and manual rules, lacking transparency and accuracy. Starting from the mid-20th century, scoring systems ofer a transparent approach to CDS development. Nevertheless, classical methods for scoring systems like logistic regression may lack predictive accuracy and sufer in handling complex high-dimensional electronic health record data, while black-box ML models pose risks due to their lack of interpretability. To address these challenges, our group focuses on developing interpretable symbolic ML approaches, leveraging multi-objective symbolic regression (MOSR) to accelerate index development, mitigate human bias, and allow for the exploration of new aggregation functions and weighting systems. MOSR optimizes multiple objectives simultaneously, distilling complex phenomena into non-linear yet understandable constructs, a crucial aspect for gaining trust from healthcare professionals. Moreover, MOSR is highly lfexible and extendable to classical statistical models. This paper presents our experience in developing data-driven scoring systems, building on real-world applications such as COVID-19 mortality prediction and risk estimation post-liver transplantation. Our methodology involves designing the entire data pipeline, from feature selection to scoring formula generation, highlighting the importance of developing data-centric and interpretable ML techniques for high-risk domains.
Artificial Intelligence (AI) systems have revolutionized healthcare by ofering opportunities to enhance diagnostics, treatment, research, and patient care. Among these tasks, we want to focus on the development of data-driven solutions to clinical decision support (CDS). Currently, CDS predominantly relies on classical statistical models and manually curated rules or heuristics that are designed to identify patient cohorts with specific characteristics of interest.
Medical scoring systems are considered as the most common CDS tool because they provide healthcare professionals with objective, intelligible, and quantifiable measures to aid in clinical decision-making. Indeed, by interpreting scores in conjunction with clinical expertise and patient-specific factors, healthcare professionals can make more informed decisions regarding diagnosis, treatment planning, and patient management. The history of scoring systems in medicine dates back to the mid-20th century. One of the earliest and most well-known scoring systems is the Apgar score, developed by Virginia Apgar in 1952, which assesses the health of newborns and is still widely used today for quick assessment. Creating and validating clinical scoring systems involves several key steps. Clinicians begin by identifying a patient population with a specific disease or condition and gathering relevant data, including demographics, medical history, and laboratory results. Statistical methods are then employed to identify predictive factors for the outcome of interest, such as mortality or disease progression. Using these factors, a model is developed to generate a score predicting the likelihood of the outcome occurring. Finally, the scoring system is validated by testing its performance on a separate group of patients with the same condition to ensure its accuracy and reliability.
Over the years, scoring systems continued to be developed, incorporating data from new
technologies, such as imaging and laboratory tests, to improve their accuracy and reliability.
However, traditional index creation methods, successful in the past, faced challenges in the
context of complex clinical phenotypes and the vast landscape of Electronic Health Records
(EHRs). In fact, without automated pipelines to reduce variables or identify their importance,
EHR-derived datasets are often high-dimensional, scarce, sparse, and unbalanced [
This work aims to describe the state of the art in the development of data-driven scoring systems and illustrate the progress and applications studied by our research group in recent years for the development of interpretable CDS tools. Specifically, in Section 2, we introduce the concept of scoring systems and their traditional development methods. We then delve into the challenges posed by the rise of AI in automating index creation and explain why interpretable ML techniques are the primary tools for this task. Moving forward, we present state-of-the-art methods for automatically generating scoring systems, with a focus on symbolic regression. This technique serves as the foundation for a new approach developed by our team for constructing lfexible and parsimonious indices in real-world scenarios. Section 3 outlines our comprehensive data pipeline for scoring system development, highlighting key aspects and featuring real case studies from our recent research endeavors. In Section 4, we draw some concluding remarks and summarize the outstanding challenges in this field that we aim to address in the near future.
A score is a mathematical combination of a set of elementary indicators (EIs) representing the diferent components of a multidimensional concept to be measured (e.g. development, quality of life, wealth, risk, etc.). Hence, synthetic scores are used to measure concepts that cannot be captured by a single indicator. In general, a composite index should be based on a theoretical framework, allowing the selection, combination, and weight of the EIs to reflect the size or structure of the phenomenon being measured. Building scores often requires a series of decisions/choices to be made. The process involves multiple steps: i) theoretical framework definition where the concept to be measured and the EIs to be considered are identified; ii) EIs selection based on their relevance, validity, availability, cost, etc., and standardization so as to work with adimensional quantities; iv) aggregation function definition where the final shape of the index combining the selected EI is determined; v) index validation in terms of robustness, generalization and discriminating ability.
There is no general method for the construction of synthetic indices so each score is linked to
the particular application. Classically, finding the right aggregation function and the weighting
of EIs is a highly non-trivial task [
Despite the definition of the theoretical context and the identification of the relevant EIs will
always require some human knowledge to be reliable, the other stages of index creation may be
automated. Indeed, the promise of systematic, targeted, and data-driven scores for CDS becomes
apparent with recent advances in ML and the ever-increasing availability of EHR data. This
would not only speed up index creation but also eliminate any human bias in its construction,
as well as explore further aggregation functions and weighting systems than what was done in
the past. However, the potential benefits are met with equally substantial challenges. Achieving
systematic index creation necessitates the development and deployment of highly accurate
clinical prediction models for a wide range of clinical problems. Moreover, the CDS framework
should be robust enough to cope with and adapt to significant variations in clinical practices and
documentation standards across diferent healthcare providers and systems. For these reasons,
comprehensive assessments should be performed to weigh the potential advantages and risks
associated with each automated CDS solution [
Meeting the previous requirements is made easier by interpretable ML techniques that can
be inspected and questioned by domain experts, ultimately leading to a higher level of trust
and, consequently, facilitating their integration into existing decision-making frameworks. In
fact, it is a view increasingly shared by the scientific community that black box models cannot
be the final answer to increase accuracy or fix cases where the assumptions of classical models
are not met [
State-of-the-art approaches for interpretable data-driven scoring systems formalize finding the
index aggregation formula as an optimization problem. Most of them translate index creation
into a classification problem, primarily focusing on binary outcomes [
The problems previously listed can be solved by using symbolic regression (SR), the
datadriven ML method for regression analysis. In fact, as we will elaborate in the next section, SR is
able to search a much wider space of mathematical formulas to find the model that best fits
a given dataset. Moreover, the use of SR is not limited to non-linear model fitting but can be
adapted to any parametric statistical model. For the above reasons, the use of SR in scoring
system development for high-stakes domains may lead to extremely important results in the
coming years [
SR refers to data-driven ML methods for regression analysis that search the space of
mathematical formulas to find the model that best fits a given dataset. The most classical approach
to SR relies on genetic programming (GP) algorithms [
When the optimization process relies on meta-heuristic processes such as GP, SR can be easily extended to multi-objective tasks. Multi-objective SR (MOSR) has been recently developed to address constrained minimization problems and prevent overfitting the training data. A classical approach to solve the overfitting problem is to consider a bi-objective setup optimizing both complexity and accuracy [19, 20]. In fact, without a goal associated with complexity, models tend to become excessively verbose, losing generalization performance and transparency. Only recently, some work started combining accuracy and parsimony into a single objective through model selection principles such as the Akaike or Bayes Information Criterion (AIC and BIC), or the minimum description length [21]. Other applications of MOSR aim to incorporate knowledge-driven (in)equalities to be satisfied [ 22, 23, 24]. While these types of constraints can be easily incorporated in scoring system development, score construction itself can be a natively multi-dimensional problem that must satisfy or track more than one property.
Our interest in developing data-driven scoring systems has stemmed from our close collaboration with the Department of Infectious Diseases of the Modena University Hospital, particularly with members of the HIV Metabolic Clinic (MHMC), a tertiary-level referral center for the + min
* ** 2 1
3 1
2 (x, ) = 13 + min(2, 1 2 ) diagnosis and treatment of non-infectious co-morbidities in people living with HIV (PWH). In 2022 [25], our research group pioneered the introduction of the first MOSR approach for managing data-driven, continuous, and non-linear scoring systems. The necessity for a multiobjective approach in continuous medical index creation arises from the intricate nature of medical outcomes, often too complex to predict precisely. This challenge is compounded by the characteristics of EHRs, which are typically not only small but also unbalanced. To address these challenges, unlike standard strategies that validate results post-hoc using metrics like stratification power and index balancing, our approach incorporates these desirable properties directly into the optimization phase. This integration provides greater control over the behavior and complexity of the models.
Moreover, despite the scientific community’s main focus being on identifying index aggregation functions, our fruitful collaboration with the MHMC highlighted that the automation of data-driven scoring system development should rely on a complete data pipeline that encompasses the entire data lifecycle, from data preprocessing and feature engineering, to model creation, optimization, and validation. A pictorial view of the data pipeline needed to create scoring systems with MOSR is depicted in Figure 1. While certain phases, such as the problem definition step, necessitate a synergistic approach between data scientists and clinicians, other steps can be partially optimized. In our experience, these are: • EI engineering and selection. The construction of normalized EI can be automatized once the data type, its meaning, and the kind of non-linearities are specified in index formulation. A major challenge in SR is the vast function space to be explored, which grows exponentially with the number of mathematical operators and variables [26]. To improve convergence rate we usually rely on minimum-redundancy-maximum-relevance non-linear feature selections [27, 28, 29] whenever needed. • Optimization objective. The ability to abstract the problem enables the identification of the appropriate class of models wherein risk assessment should be integrated, whether it’s classification, regression, survival analysis, etc. By comprehending the properties essential for the index’s efectiveness, one can discern additional constraints or desiderata, such as constraints/objectives on correlation, calibration, or distribution. Lastly, identifying the final application domain aids in understanding the tolerable nonlinearities and the maximum acceptable complexity level of the final formulas. These considerations collectively facilitate the identification of optimal optimization goals. • Performance metrics and comparison. Performance metrics beyond the optimization criteria should be identified to check the predictive power of the generated scores. These metrics need to be tailored to the specific problem under study. For instance, in the context of a risk score based on binary classification, it is essential to calculate metrics such as sensitivity, specificity, and the area under the ROC curve. Furthermore, it is paramount to compare the results with those obtained from classical statistical methods and other benchmark ML approaches. • Model selection and validation. Model selection should consider performance metrics, safety, and domain expertise. In addition to standard performance evaluation, validating the score typically entails analyzing its distribution within the target population and its correlation with other clinically relevant measures. This ensures the score’s consistency and compatibility with existing medical knowledge.
Below, we will demonstrate that since the publication of the methodological work, our group has persistently advanced in developing, systematizing, and automating various aspects necessary for score development automation, also expanding the methodology to encompass diferent case studies. We start by briefly summarizing two situations where scores are modeled as regression and binary classification problems, afterward, we show in detail how we extended the method to survival analysis problems.
Improving Frailty index assessment for PWH [30]. This study developed a data-driven tool for geriatric assessment of PWH, focusing on enhancing the Frailty Index (FI), a clinical score based on 35 variables. Leveraging data from the MHMC cohort, we designed a pipeline for constructing reduced FIs. Starting from a set of knowledge driven 54 binary EIs, we employed a non-linear feature selection method and identified the 27 most relevant EIs in predicting the FI. We modeled index simplification as a regression problem and used MOSR to replicate the values, distribution, and risk stratification ability of the original FI, minimizing weighted mean square error, Wasserstein distance, and maximizing pairwise Kendall correlation. We evaluated optimal model predictiveness through calibration, correlation with the original index, and associations with established geriatric outcomes such as age, the EQ-5D-5L score, and the SPPB index. The simplest optimal model used only 16 readily available variables, meeting all requirements and was incorporated into the MHMC’s automatic data collection pipeline.
Short-term mortality prediction in patients with Covid-19 [31]. This study aimed to predict short-term in-hospital mortality risk using data collected upon hospital admission. EHRs from 2400 patients with Covid-19 diagnoses were gathered. A non-linear feature selection method reduced covariates from 25 to 10, validated by the medical team and standardized. Index creation involved a tailored non-linear logistic regression using formulas generated by MOSR. Experiments minimized weighted Binary Cross Entropy and maximized the F1 score to optimize the sensitivity-specificity tradeof. MOSR outperformed popular machine-learning algorithms and classical human-generated indices, prioritizing false negative reduction for timely interventions.
The study [32] aimed to quantify the interaction between risk factors in predicting death within the first four months post-LT. The data consisted of 485 EHRs of patients who underwent LT at the University Hospital of Modena between 2010 and 2020. Available exposure variables included patient admission details, preoperative conditions such as colonization by multidrugresistant bacteria, and postoperative risk factors like bloodstream infections. Due to data scarcity and imbalance, EI selection and construction required collaboration between data scientists and clinicians. The chosen set of covariates was: Hospitalization days (HD); Intensive Care Unit days (ICU); Model for End-Stage Liver Disease (MELD); Duration of surgery (DS); LT year (LTY); Post-LT infection (PTI); MDR-Gram negative pre-operative colonization (GnC); on top of death observation and censoring times. Variable selection was followed by feature engineering to create comparable and informative EIs. Specifically, we discretized continuous variables based on clinically meaningful intervals.
Cox’s regression, a classic semi-parametric survival model, is the most classic and easy-tointerpret model that can estimate the efects of exposure variables and adjust for confounding efects. Classically, Cox’s hazard model is written as: (x, ) = 0() × exp {︀ x}︀ (1) where x = {}, = 1 . . . , are the covariates and = { } are the parameters to be estimated by the model. Therefore, the hazard function is given by the product of the baseline hazard 0() and the covariate-dependent relative risk. As the only time dependence lies in 0(), a fundamental assumption underlying the application of the Cox model is the Proportional Hazard Assumption (PHA). The application of the classical Cox model turned out not to be suitable for the indicators created, since, as shown in Table 1, some covariates did not meet PHA. Classical methods for extending the Cox model in this situation are dificult to apply to sparse and unbalanced data, as well as being less interpretable. Therefore, we embedded SR into Cox’s regression by making the covariate-dependent log relative hazard function (CLH) trainable and potentially non-linear: exp {︀ x}︀ → exp { ( , x)}. By optimizing partial AIC (AIC) and model complexity, parametrized as the number of nodes in the syntax tree, we maximized model likelihood while minimizing the number of numerical parameters and the degree of non-linearity for enhanced interpretability. MOSR survival models not only outperformed classical Cox regression in predictive performance but also successfully mined composite data-driven risk factors, overcoming classical model limitations such as the PHA. See the bottom panel in Table 1 for a representative optimal solution. Finally, model selection was guided by diferences in AIC values for theoretical support, calibration and predictive performance metrics for eficiency and usefulness, and out-of-distribution prediction for safety assessment. More details about feature selection and construction and model performance and validation can be found in the original text.
Scoring system development stands out as a prominent application of data-centric AI, where the fusion of data mining, big data analytics, and ML plays a pivotal role. In this work, we have presented the state of the art in automatic scoring system generation, covering data frameworks, algorithms, and select case studies successfully tackled by our research group. By distilling knowledge from retrospective clinical data, our data processing pipeline and analyses ofer seamless adaptability to diverse case studies.
Despite the advancements obtained during the last years in the development of scoring systems, state of the art data pipeline for data-driven scoring system development still lack some crucial milestones to be reached. In addition to being interpretable, CDSs must be able to handle uncertainty. The problem of quantifying and managing epistemic uncertainty becomes paramount when the system under study evolves with time or when dealing with multicenter studies. The ML frameworks devoted to these issues are continual learning (CL) and federated learning (FL) [33].
CL focuses on enabling AI models to learn and adapt continuously over time, incorporating new information while retaining previously acquired knowledge. On the other hand, FL aims to facilitate the development of multicenter studies by overcoming the problems related to centralized learning settings that need to transfer sensitive medical data from multiple centers to a central location. FL and CL approaches to SR have received little attention, with the literature focusing on DL models. Only a few works exist about federated SR algorithms [34, 35], and none of them deal with concept drift.
To foster trust among domain experts, model interpretability may not sufice and their full engagement in model creation and selection should be allowed. In a true human-in-the-loop scenario, physicians should be allowed to contribute their clinical expertise directly to ML model development and selection. This can be facilitated through flexible graphical interfaces enabling iterative feedback processes.
Finally, proposing SR as a method capable of automating the creation of CDS tools will require consolidating and extending it to various parametric models commonly used in clinical score development, such as sub-distribution hazard models and time series analysis. [18] C. Wilstrup, J. Kasak, Symbolic regression outperforms other models for small data sets, arXiv preprint arXiv:2103.15147 (2021). [19] B. Burlacu, G. Kronberger, M. Kommenda, M. Afenzeller, Parsimony measures in multiobjective genetic programming for symbolic regression, in: Proceedings of the genetic and evolutionary computation conference companion, 2019, pp. 338–339. [20] Q. Chen, B. Xue, M. Zhang, Rademacher complexity for enhancing the generalization of genetic programming for symbolic regression, IEEE transactions on cybernetics 52 (2020) 2382–2395. [21] D. J. Bartlett, H. Desmond, P. G. Ferreira, Exhaustive symbolic regression, IEEE Transactions on Evolutionary Computation (2023). [22] J. Kubalík, E. Derner, R. Babuška, Symbolic regression driven by training data and prior knowledge, in: Proceedings of the 2020 Genetic and Evolutionary Computation Conference, 2020, pp. 958–966. [23] C. Haider, F. O. de França, B. Burlacu, G. Kronberger, Using shape constraints for improving symbolic regression models, arXiv preprint arXiv:2107.09458 (2021). [24] J. Kubalík, E. Derner, R. Babuška, Multi-objective symbolic regression for physics-aware dynamic modeling, Expert Systems with Applications 182 (2021) 115210. [25] D. Ferrari, V. Guidetti, F. Mandreoli, Multi-objective symbolic regression for data-driven scoring system management, in: 2022 IEEE International Conference on Data Mining (ICDM), IEEE, 2022, pp. 945–950. [26] M. Virgolin, S. P. Pissis, Symbolic regression is np-hard, Transactions on Machine Learning
Research (2022). [27] H. Peng, F. Long, C. Ding, Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy, IEEE Transactions on pattern analysis and machine intelligence 27 (2005) 1226–1238. [28] M. Yamada, J. Tang, J. Lugo-Martinez, E. Hodzic, R. Shrestha, A. Saha, H. Ouyang, D. Yin, H. Mamitsuka, C. Sahinalp, et al., Ultra high-dimensional nonlinear feature selection for big biological data, IEEE Transactions on Knowledge and Data Engineering 30 (2018) 1352–1365. [29] Z. Zhao, R. Anand, M. Wang, Maximum relevance and minimum redundancy feature selection methods for a marketing machine learning platform, in: 2019 IEEE international conference on data science and advanced analytics (DSAA), IEEE, 2019, pp. 442–452. [30] V. Guidetti, F. Motta, J. Milic, D. Ferrari, F. Mandreoli, G. Guaraldi, Unlocking frailty index: Knowledge distillation with symbolic machine learning to simplify frailty assessment in standard hiv clinics, Under review (2024). [31] D. Ferrari, V. Guidetti, Y. Wang, V. Curcin, Multi-objective symbolic regression to generate data-driven, non-fixed structure and intelligible mortality predictors using ehr: Binary classification methodology and comparison with state-of-the-art, in: AMIA Annual Symposium Proceedings, volume 2022, American Medical Informatics Association, 2022, p. 442. [32] V. Guidetti, G. Dolci, E. Franceschini, E. Bacca, G. J. Burastero, D. Ferrari, V. Serra, F. Di Benedetto, C. Mussini, F. Mandreoli, Death after liver transplantation: Mining interpretable risk factors for survival prediction, in: 2023 IEEE 10th International Conference on Data Science and Advanced Analytics (DSAA), IEEE, 2023, pp. 1–10. [33] L. Cao, H. Chen, X. Fan, J. Gama, Y.-S. Ong, V. Kumar, Bayesian federated learning: A survey, arXiv preprint arXiv:2304.13267 (2023). [34] J. Dong, J. Zhong, W.-N. Chen, J. Zhang, An eficient federated genetic programming framework for symbolic regression, IEEE Transactions on Emerging Topics in Computational Intelligence (2022). [35] D. Nguyen Duy, M. Afenzeller, R. Nikzad-Langerodi, Towards vertical privacy-preserving symbolic regression via secure multiparty computation, in: Proceedings of the Companion Conference on Genetic and Evolutionary Computation, 2023, pp. 2420–2428.