In this position paper, we discuss the potential use of a reinforcement learning (RL)-based human-in-the-loop recommender system to support clinical management of COVID-19. COVID-19 is a disease of extraordinary complexity that even the most experienced clinicians are struggling to understand. There is an urgent need for an evidence-based model for predicting the severity of the COVID19 disease and its complications that can guide individual clinical management decisions. Such a model will utilize a diverse set of information to determine a patient's disease severity and associated risk of complications. An immediate application would be a clinical protocol tailored for COVID-19 patient care; this is a critical need both today and for future studies of potential treatments.
• Human-centered computing → Human computer interaction (HCI); • Applied computing → Health care information systems. COVID-19; reinforcement learning; human-in-the-loop; staging
The emergence of the Severe Acute Respiratory Syndrome
Coronavirus (SARS-CoV-2) poses significant challenges to the livelihood
of the afected nations and, in the absence of directed treatment
or a vaccine, requires drastic public health measures which have
crippled national and international economies [
There are several challenges in developing a successful human-inthe-loop reinforcement-learning framework that generalizes across the entire disease severity spectrum.
Extracting actionable intelligence from heterogeneous and incomplete data: Owing to the complexity of COVID-19, the identification of distinct clinical stages of COVID-19 progression and patient trajectories requires the integration of multiple data sources. We believe that domain-guided models that integrate machine learning methods and clinical insights will be beneficial. Specifically, probabilistic graphical models, can represent the domain-driven relationships between diferent information sources, and can be transformed into discriminative models that can be trained using the available data. The goals are to improve outcomes, appropriately allocate healthcare resources, and reduce mortality rates while directed treatments and vaccines are being developed. Quantifying the uncertainty in model predictions: Since data are limited in the beginning, uncertainty in the prognostics will be high in the early stages and will gradually decrease as the model is updated using new data. The ability to quantify such uncertainty is critical in order for clinicians to accurately gauge the importance of their own assessments relative to the model’s predictions. We recommend the use of Bayesian methods and attribution-based approaches to quantify the uncertainty and interpret model predictions, respectively, both of which will inform the clinicians. Time-frames for model development and reinforcement: It is unclear how many data points are required to train an accurate initial model. A potential measure that can guide this decision is the convergence of class probabilities to their respective population means. In addition, there have been multiple reports indicating varied lengths of hospitalizations. Such variability depends on multiple factors including disease severity, comorbidities, health provider policies, decisions regarding withdrawal of life support, cost of treatments, etc. Clearly, some of these factors are non-deterministic and data on such factors are typically unavailable for the machine learning model. Learning a suficiently accurate and robust decision scheme with those dificult-to-measure elements and determining when to reinforce the learning algorithm remain challenging tasks. Modeling the human-in-the-loop decision process: A typical RL approach relies on an efective balance between exploration and exploitation such that the algorithm is allowed suficient exploration of the input space prior to basing predictions primarily on the space that it has already explored. However, that paradigm is not usable in this setting, because treatment decisions are a matter of life and death; we cannot take actions that would jeopardize medical ethics. Therefore, our approach requires the presence of a clinical expert who will make decisions after appraising the model’s predictions in light of his/her own assessments.
In this paper, we described a novel domain-guided human-in-theloop RL framework to assist physicians in clinical decision-making to stage COVID-19 patients across the disease severity spectrum. Going forward, the clinical stages as defined by this approach could form the basis for evaluating the eficacy of existing and new drugs related to the patients in diferent stages of disease progression. While the proposed model is specifically designed and trained for COVID-19, the underlying paradigm of our model, i.e., the human in the loop RL, afords the adaptivity to be applicable to other respiratory illnesses and other future pandemics, with re-calibration.
This project has been funded by the Jump ARCHES endowment through the Health Care Engineering Systems Center at the University of Illinois.