In hospital environments, in-time decision-making is critical to improving patient outcomes and eficiency. However, many decisions are often made too late in the process, resulting in delays and misuse of resources. This research investigates how Process Predictive Modelling (PPM) models can be developed and evaluated to incorporate clinical impact and thus support earlier decisions within hospital workflows. Focusing on not only retrospective models, but also on real-world applications and integration into clinical practice.
The core idea of this research is to develop and evaluate PPM models to support earlier and more efective clinical decision-making within hospital workflows. In our case, we focus on three. For each of the three identified use cases, we develop PPM models: • ED admission: A model that predicts, during a patient’s stay in the Emergency Department (ED), whether the patient will be admitted. This allows logistics, such as ward bed allocation, to begin earlier in the process and hopefully improves the time at the ED for the patient. • Acute Kidney Injury (AKI): A model that uses frequency data from heart-lung machines during cardiac surgeries to predict the likelihood of a patient developing AKI. By detecting early warning signs, clinicians can be alerted in time to lower the risk.
CEUR
ceur-ws.org • Discharge: A model that predicts (1) when a patient is likely to be discharged and (2) the most probable discharge destination. This supports better resource planning from the ward.
When patients arrive at the ED, doctors must decide whether to send them home or admit them to the hospital for further treatment. However, this decision often isn’t made right away and is made after the clinicians are 100% sure the patient will not go home. This leads that the logistics of e.g. preparing a bed on the ward, is done very late in the process. This late notification of admission lead to overcrowding in the ED and less eficient use of hospital beds.
To improve this process, we developed a model that predicts whether a patient is likely to be admitted. If this prediction is made accurately and early, hospitals can start preparing for the admission sooner, for example by reserving a bed in the appropriate department. These supports should smooth patient lfow, reduce waiting times, and improve the overall eficiency.
AKI is a sudden loss of kidney function that can happen after major surgeries, especially heart surgeries involving heart-lung machines. AKI can lead to serious health complications and longer hospital stays if not recognized and treated early. However, warning signs of AKI are often subtle and may be missed until damage has already occurred.
To address this, we want to create a model that predicts the likelihood of a patient developing AKI during or directly after cardiac surgery. It uses process data collected from heart-lung machines that is already available during surgery but never used for predictions and analysis before (to our knowledge).
The model can alert clinicians when a patient is at risk of AKI before current symptoms become clear. This allows for taking preventive measures earlier, ultimately improving patient outcomes and supporting clinical decision-making during critical care processes.
In certain hospitals, patients may need additional support after discharge, such as home care, rehabilitation, or hospice services. This is referred to as aftercare. Organising aftercare requires coordination with external care providers and various administrative tasks, which are managed through a separate process known as the transfer process. Ideally, the patient care process (spanning from admission to medical discharge) should be aligned with the transfer process (which begins when aftercare needs are identified and continues until discharge). This alignment ensures that all aftercare arrangements are completed when the patient is medically discharged.
From my dataset (October 2021 - December 2024) from the Internal Medicine department at St. Antonius Hospital, there are 5000 admissions, of which 1200 patients required aftercare. Among these, 800 experienced prolonged stays. On average, these patients stay 5.6 days beyond their medical discharge date, totaling approximately 4500 extra hospital days, or 2000 days per year. Additionally, aftercare needs are accurately registered on time and 50% of cases, and medical discharge dates in just 60% of instances.
Prolonged hospital stays create several challenges. First, they delay access to specialized aftercare facilities, where patients can receive more targeted care. Secondly, occupied hospital beds hinder the admission of new patients, contributing to bed shortages and longer waiting times. Finally, prolonged stays increase hospital costs, placing additional strain on healthcare resources.
Currently, the process for arranging aftercare begins with nurses placing an order in the electronic health record (EHR), which is then added to the transfer nurses’ working list. This typically occurs in the morning on weekdays. Transfer nurses review these patient files, visit the patients, and assess their aftercare needs, discussing them with families and other healthcare professionals. The diferent types of aftercare include services such as rehabilitation, long-term care, and specialized outpatient services.
Transfer nurses may inform aftercare facilities in advance about potential admissions, depending on when they anticipate that a patient will be ready for discharge, to increase eficiency. However, to make a formal reservation at an aftercare facility, the (1) exact medical discharge date and (2) specific aftercare requirements must be confirmed. Since at least one of these two is often unknown, reservations are delayed until the day of medical discharge, leading to prolonged hospital stays.
The goal of this thesis is to reduce waste (time, wrong decisions, resources, etc.) in hospital workflows by supporting clinical staf in making the right decision at the right time. The research questions are: 1. What are the clinically relevant outcomes that should guide decision support models in hospital workflows? a) Which outcomes matter for patients? b) Which outcomes matter for healthcare professionals? c) Which outcomes matter for the hospital? 2. How can predictive models be trained and optimized to support these outcomes? a) How can models be developed while optimizing a problem-specific developed metric? b) When multiple outcomes are relevant, how can the diferent models be combined to support decision-making efectively? 3. What is the real-world impact of implementing models in clinical practice? a) How do these models influence the outcomes of the patients, healthcare professionals, and hospital? b) How are these impacts perceived by patients, nurses, and other stakeholders?
To address the research goal, a three-phase approach is used for the use cases: (1) Understanding of the use case, (2) Developing AI models, and (3) Implementing the models to validate.
To achieve clinical impact, the first step involves gaining a deep understanding of the hospital process in question. This includes mapping the entire workflow, identifying all relevant stakeholders (e.g., nurses, physicians, transfer staf, external parties), and determining where in the process predictive support could make a meaningful diference. Key questions to explore include: • What kind of prediction would be most helpful? • At what point in time would such a prediction need to be made? • How accurate, early, or interpretable would it need to be to change decision-making? This step is carried out through close collaboration with clinical staf to understand not just the data, but also the practical constraints and decision points within the process. Rather than beginning with advanced analysis, this phase focuses on qualitative insights that lead to a clear, use-case-specific definition of clinical value. This may result in custom metrics that guide both model development and evaluation.
Once the process and its needs are well understood, PPM models are developed using hospital data. Crucially, the models are evaluated not only on standard metrics (such as accuracy or AUROC), but also on the clinical impact criteria defined in Phase 1. Where possible, these criteria are also incorporated during model training.
Each use case may require diferent modelling strategies. For example, in the discharge use case, separate models may be needed to predict the discharge date and the aftercare. In some cases, multiobjective or multi-target models can address these dependencies within a single framework. The insights gathered in Phase 1 help inform which model and inputs are most relevant for input to the model.
True validation of clinical impact requires testing the models in practice. However, full integration of AI models into hospital systems is often complex and subject to regulatory, ethical, and technical constraints. Therefore, this phase is approached in stages: • Shadow testing performed first, where model predictions are generated in real time but not yet used for decision-making. • Selective implementation follows, targeting specific wards, clinicians, or workflows to test impact in a controlled way. • Full integration, where possible, allows for real-time/daily predictions to inform care decisions, but is likely limited to use cases that are ethically and operationally ready (e.g., ED admissions).1
The current focus of my work is the Discharge use case. So, in the next section, we will describe the gaps between existing models and the clinical usefulness of this use case.
Numerous studies have focused on the early prediction of aftercare requirements. There are studies [
The prediction of hospital LOS has also been an important area of research. Some studies simplify the prediction task, transforming it into a classification task. For instance, in [ 6], LOS is classified into short stay (0 to 7 days) and long stay (more than 7 days). Others predict the LOS (usually in terms of the number of days) for target patient groups [7], or surgery patients [8]. Similarly to the aftercare needs predictions, some research is not specific about the time of the prediction. However, there are studies [9] that perform such prediction clearly after 24 hours of admission.
Dynamic prediction models have also been explored, such as those in [10] and [11], which update LOS predictions daily based on the latest data, focusing on whether the patient will be discharged on 1In use cases such as AKI prediction, where clinical risk is high, more validation is required before any implementation can be ethically justified. the same day. Extending this approach, [12] applies the most recent data to predict whether a patient will be discharged today, tomorrow, or the day after, providing a slightly broader target compared to earlier studies.
Most research [
In our recent work, we have addressed some of the identified gaps. Specifically, in [ 13] we proposed a novel metrics to measure the clinical impact of predictive models, moving beyond standard performance metrics. Furthermore, in [14] we developed and evaluated two prediction models that are optimized with respect to this clinical impact measure.
While the first draft of the thesis ofers a clear structure, several questions remain: 1. How to, methodologically and/or in a data-driven manner, identify the prediction time that provides the highest clinical impact? 2. How to balance the earliest prediction time for higher clinical impact with prediction accuracy? 3. How to implement (multi-target) prediction models to optimize for clinical impact? 4. How to correctly consider domain knowledge to measure clinical impact? Coming from example discussions to a more general approach. 5. Each prediction made would lead to a recommendation of action to support the decision process.
How to evaluate if such recommendations are ”good”? We can already think on measuring if it is the right recommendation, if it was given on the right time. There is no general framework in recommendation systems on how to do this.
The authors declare that no generative AI tools were used for the generation of text in this paper. Grammarly and Writefull were used exclusively for grammar and spelling correction. [4] Annika L. Dubbeldam, István Ketykó, Renata Medeiros de Carvalho, Felix Mannhardt, Early Predicting the Need for Aftercare Based on Patients Events from the First Hours of Stay – A Case Study, ICPM Workshops (2023) 366–377. doi:10.1007/978-3-031-27815-0_27, mAG ID: 4360969787 S2ID: 73033a82168f76e5d2974efc89f044c435bbc27b. [5] T. Schrage, G. Thomalla, M. Härter, L. Lebherz, H. Appelbohm, D. L. Rimmele, L. Kriston, Predictors of Discharge Destination After Stroke, Neurorehabilitation and Neural Repair 37 (2023) 307–315. URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10272622/. doi:10.1177/ 15459683231166935. [6] B. Alsinglawi, Osama Alshari, Mohammed Alorjani, O. Mubin, F. Alnajjar, M. Novoa, O. Darwish, An explainable machine learning framework for lung cancer hospital length of stay prediction, Scientific Reports 12 (2022). doi: 10.1038/s41598-021-04608-7, mAG ID: 4205344436 S2ID: 1b3ae70162005599a21ccfb72e3c83501f217e21. [7] K. Goshtasbi, T. M. Yasaka, M. Zandi-Toghani, H. R. Djalilian, W. B. Armstrong, T. Tjoa, Y. M.
Haidar, M. Abouzari, Machine learning models to predict length of stay and discharge destination in complex head and neck surgery, Head & Neck 43 (2021) 788–797. doi:10.1002/hed.26528. [8] Rachda Naila Mekhaldi, Patrice Caulier, P. Caulier, Sondès Chaabane, Sondes Chaabane, A. Chraibi, Sylvain Piechowiak, Using Machine Learning Models to Predict the Length of Stay in a Hospital Setting, WorldCIST (2020) 202–211. doi:10.1007/978-3-030-45688-7_21, mAG ID: 3024251604 S2ID: 415177522f3539ee5e997a1255c1075a924f4851. [9] Nicholas Piniella, Theresa E. Fuller, Laura Smith, Hojjat Salmasian, Cathy Yoon, Stuart R. Lipsitz, Jefrey L. Schnipper, Anuj K. Dalal, Early Expected Discharge Date Accuracy During Hospitalization: A Multivariable Analysis, Journal of medical systems 47 (2023). doi:10.1007/ s10916-023-01952-1, mAG ID: 4376225069 S2ID: fed5e1bde8b1d397cb6cbe3c52d67dab953e57cf. [10] Sean L. Barnes, Eric Hamrock, E. Hamrock, Matthew Toerper, M. F. Toerper, Sauleh Siddiqui, S. Siddiqui, Scott R. Levin, S. Levin, Real-time prediction of inpatient length of stay for discharge prioritization, Journal of the American Medical Informatics Association 23 (2016). doi:10.1093/ jamia/ocv106, mAG ID: 2135195117 S2ID: 87345277dbc325e032f125e47b550cdad56f5626. [11] K. C. Safavi, P. M. Pardalos, T. Khaniyev, M. S. Copenhaver, Development and Validation of a Machine Learning Model to Aid Discharge Processes for Inpatient Surgical Care., JAMA Network Open 2 (2019) 1917221. doi:10.1001/jamanetworkopen.2019.17221, mAG ID: 2995850824 S2ID: 8e191ea882361beddd6be18a0ec9a6c6531c94cf. [12] S. Levin, S. Barnes, M. F. Toerper, A. Debraine, A. DeAngelo, E. Hamrock, J. S. Hinson, E. H. Hoyer, T. Dungarani, E. Howell, Machine-learning-based hospital discharge predictions can support multidisciplinary rounds and decrease hospital length-of-stay 7 (2020) 414–421. doi:10.1136/ bmjinnov-2020-000420, mAG ID: 3113453271. [13] Y. van der Haas, R. Medeiros de Carvalho, T. van Dijk, B. F. van Dongen, R. Plas, A Novel Way to Evaluate Medical Discharge Predictions: A Research Paper, https://research.tue.nl/nl/publications/a-novel-way-to-evaluate-medicaldischarge-predictions-a-research-, 2025. URL: https://research.tue.nl/nl/publications/ a-novel-way-to-evaluate-medical-discharge-predictions-a-research-. [14] Y. van der Haas, R. M. de Carvalho, B. F. van Dongen, R. Plas, T. van Dijk, Optimizing the Discharge Planning Process in a Dutch Hospital, ICPM 2025 (2025).