Cancer screening is a large, population-based intervention that would bene t from tools enabling individually-tailored decision making to decrease unintended consequences such as overdiagnosis. The heterogeneity of cancer screening participants advocates the need for more personalized approaches. Partially observable Markov decision processes (POMDPs) can be used to suggest optimal, individualized screening policies. However, determining an appropriate reward function can be challenging. Here, we propose the use of inverse reinforcement learning (IRL) to form rewards functions for lung and breast cancer screening POMDP models. Using data from the National Lung Screening Trial and our institution's breast screening registry, we developed two POMDP models with corresponding reward functions. Speci cally, the maximum entropy (MaxEnt) IRL algorithm with an adaptive step size was used to learn rewards more e ciently; and combined with a multiplicative model to learn state-action pair rewards in the POMDP. The lung and breast cancer screening models were evaluated based on their ability to recommend appropriate screening decisions before the diagnosis of cancer. Results are comparable with experts' decisions. The lung POMDP demonstrated an improved performance in terms of recall and false positive rate in the second screening and post-screening stages. Precision (0:02 0:05) was comparable to experts' (0:02 0:06). The breast POMDP has excellent recall (0:97 1:00), matching the physicians and a satisfactory false positive rate (< 0:03). The reward functions learned with the MaxEnt IRL algorithm, when combined with POMDP models in lung and breast cancer screening, demonstrate performance comparable to experts.
Annually, millions of people undergo screening for disease prevention and surveillance. From these tests, physicians aim to make decisions based on the patient's past results and most current observations, determining a subsequent action (e.g., further diagnostic testing, increased monitoring, following regular screening schedules, etc.) that optimizes early detection of health problems while balancing other (pragmatic) concerns (e.g., patient quality of life, resource utilization, cost). Choosing the \best" next step and tailoring screening for each person is challenging: selecting an action of bene t in the immediate future may not be optimal over the long-term, given the particulars of an individual (i.e., a locally greedy approach vs. a global optimization).
Sequential decision making methods provide a potential solution. Such
approaches can integrate and analyze multiple sources of patient data, while
handling issues related to temporal credit assignment. In particular, partially
observable Markov decision processes (POMDPs) have been applied to cancer
screening (e.g., breast, colorectal, prostate [
Here, we propose using the Maximum Entropy Inverse Reinforcement
Learning (MaxEnt IRL) algorithm [
Although Markov decision processes (MDPs) and POMDPs are used in a
number of domains, their application in healthcare is limited and few strategies
exist for estimating the associated reward functions that drive agent behavior
in clinical settings. Classic examples include: Bennet et al. [
Brie y, IRL addresses the problem of obtaining a reward function given an
agent's optimal behavior over time towards a stated goal. A reward function for
the environment is unknown and is hence learned through empirical
investigation of sensory inputs (i.e., observations) that progressively change the agent's
selection of di erent actions. Two families of IRL algorithms exist: 1) linear
programming (LP) methods [
NLST Dataset
The National Lung Screening Trial (NLST) is a multi-site randomized controlled
trial that demonstrated a 20% mortality reduction in lung cancer screening
using low-dose computed tomography (LDCT) relative to plain chest
radiography [
Athena Dataset
The Athena Breast Health Network [
Partially Observable Markov Decision Processes An MDP is represented by a tuple of states, actions, rewards, action-dependent state transition dynamics (i.e., transition probabilities), and a discount factor. A POMDP is an extension to MDPs with two additional components: observations and state-dependent observation dynamics (i.e., observation probabilities). The state of the agent in POMDPs is partially observable. As such, its state is modeled as a probability distribution over the states, called the belief state, that gets updated over time based on the observations experienced by the agent.
We designed and evaluated two separate POMDPs for lung and breast cancer
screening. Each model consists of three states and two actions. The observations
of each POMDP are domain based: in the lung model, they represent ndings
obtained from LDCT imaging studies, including nodule size, consistency, location,
and margins; in the breast model, they represent BI-RADS scores derived from
MG interpretations. Given the nature of each dataset, both the lung and breast
models have a horizon of three and four years, respectively, with 6-month and
1-year epochs. Each epoch represents time points for which we have information
on the cancer status of patient (diagnosed with cancer or not). Transition and
observation probabilities for each POMDP model are learned using the
expectation maximization (EM) algorithm, for learning dynamic Bayesian networks,
from each dataset. Both models were solved using the QMDP approximation
solver [
Lung cancer screening POMDP Figure 1 (left) depicts the lung POMDP, illustrating the state space and allowed transitions between states, as well as the observations of each state. The state space consists of three states: the nocancer (NC) state that represents any case with no suspicious abnormalities (i.e., no pulmonary nodules > 4 mm). The uncertain (U) state that represents any case with a noted nding (i.e., nodules 4mm or larger) but not yet a lung cancer. Lastly, the invasive-cancer (IC) state is any case with a con rmed lung cancer diagnosis through the use of additional diagnostic tests. The IC state is terminal such that any individual who enters it leaves the screening process for treatment. An LDCT action implies continuation of screening, whereas an intervention action refers to any diagnostic procedure (e.g., thoracotomy, biopsies, diagnostic CT, positron emissions tomography (PET) scan). Observations represent LDCT ndings (nodule size, consistency, margins, and anatomic location) and the occurrence of an intervention. To generate initial belief states for each individual in our dataset we used the Tammemagi PLCOM2012 model with demographic and clinical features at baseline to predict the risk of cancer. Demographic features used include age, education, race, and body mass index. Clinical features used were COPD, family history of lung cancer, personal history of cancer, smoking status, smoking intensity, and duration of smoking.
Breast cancer screening POMDP The breast POMDP model also consists of three states: the no-cancer (NC) state in which no abnormalities are seen, the benign (B) state in which benign breast disease diagnosis follows the MG, and the malignant (MA) cancer state in which the disease is con rmed through biopsy. MA is similarly a terminal state in which the patient leaves the screening process for treatment. Figure 1 (right) shows the breast cancer screening POMDP, transitions, observations (BI-RADS scores 1, 2, 3, 4A, 4B, 4C, 5), and actions. Though an intervention (biopsy in the breast cancer context) is possible after each MG, in practice biopsies are only performed after an MG of BI-RADS 4 or higher. For an initial belief, we used the patient's Gail score. The Gail score is an absolute risk estimate derived using age, age at menarche, age at rst birth, the number of rst-degree relatives with breast cancer, the number of previous breast biopsies, and race. 3.4
Maximum Entropy IRL
In IRL, the reward function, r, is assumed to be a linear combination of feature
vectors fs and weights ( T is the transpose of ):
r( ; ) = T f = X T fs (1)
s2
A feature count, (f ), is the sum of feature vectors of the states visited along
a trajectory, where fs represents binary vectors indicating state values. Inputs
to the MaxEnt IRL algorithm are an MDP and a set of trajectories (D) [
As such, trajectories of equal reward are equally likely to be executed by the expert, whereas trajectories of less reward are less likely. The probability distribution over paths with maximum information entropy is parameterized over . Z( ) is the partition function, where Z( ) = P 2D exp r( ; ).
1
Z( ) p( ; ) = exp (r( ; )) The log likelihood of the trajectories (loss function) is shown in Equation 4, M is the number of trajectories:
L = The gradient r L represents the di erence of feature expectations and sum over state visitation frequencies multiplied with feature vectors: r L = f~
X
Dsi fsi
si
A feature expectation, (f~), is de ned as the average of all feature counts across
all trajectories. The frequency of state visitation, Dsi , can be computed using a
dynamic programming algorithm; see [
Adaptive step size
To improve the convergence of the MaxEnt IRL algorithm, we introduce an
adaptive learning rate approach for the update rule of the gradient descent. The
idea behind making the step size adaptive is to calculate the inner product of
r L, the gradient, in the current step, i.e., r Li with r Li 1, its value from
the previous step. If the two are in the same direction then the step size can be
increased, otherwise it is decreased. Following [
!
user-de ned constants obtained from [
Comparison of MaxEnt IRL with & without adaptive step size
(a) Lung cancer states' re- (b) Lung cancer states' rewards. wards. (c) Lung cancer actions' (d) Lung cancer actions' rewards. rewards.
Fig. 3. State and action rewards computed using the MaxEnt IRL and normalized by
range. Left: Using an adaptive step size. Right: Without using an adaptive step size.
The adaptive step size MaxEnt IRL algorithm converges to a solution signi cantly
faster than the MaxEnt IRL without an adaptive step size.
with an adaptive step size. We compare the MaxEnt IRL with and without the
adaptive step size and assess the speed of convergence. Figure 3 depicts the
computed rewards for states and actions for the lung POMDP over the number
of iterations of gradient descent in the MaxEnt IRL algorithm, with and without
an adaptive step size. A similar convergence trend is observed with the breast
POMDP. As shown, the adaptive step size method converges to the correct
solution more quickly than the standard MaxEnt IRL implementation. For the
evaluation of the two models we use a reward function derived from rewards
normalized in the [
Lung and breast POMDP results We used the longitudinal observations from the NLST and Athena datasets as input to POMDPs such that each sequential observation updates the belief state of the agent. The belief state of the POMDP, at each epoch, is then used to select the next (optimal) action, with the objective of early detection of cancer. The POMDP models can suggest to continue screening (i.e., MG, LDCT) or to perform an intervention (i.e., biopsy or diagnostic imaging). If an intervention is performed, the individual is removed from further consideration. Evaluation of the POMDP is posed as a binary problem: if the POMDP suggests continued screening (LDCT/MG) then the patient is classi ed as a negative cancer; if it suggests an intervention, then the patient is classi ed as a positive cancer. Based on this de nition, if the model suggests a LDCT/MG and the patient did not have a con rmed diagnosis of cancer in a given epoch, it is considered a true negative (TN); if the patient had a con rmed diagnosis of cancer then it is a false negative (FN). Conversely, if the model suggests an intervention and the patient did not have cancer in a given epoch, then it is considered a false positive (FP); if the patient had a diagnosis of cancer then it is considered a true positive (TP). Performance metrics were estimated for each epoch of the screening process. Any subject diagnosed with cancer is removed from the subsequent epoch. The POMDP models are compared against the equivalent physician decisions (recommendations) at each epoch, applying a similar framework for TN/FN/FP/TP to the experts, given the known cancer outcomes from each dataset (e.g., if the physicians suggested an LDCT/MG and the patient did not have a con rmed diagnosis of cancer, it is considered a true negative, etc.). Table 2 shows the performance of the lung and breast POMDPs and the corresponding performance of physicians on the same dataset. Notably, both POMDP models show performance comparable to experts. The lung cancer screening model has worse performance in terms of recall in the rst and third screening epochs, but an improved performance in terms of recall and false positive rate in the second screening and post-screening. The breast cancer screening model demonstrates excellent recall (as do the expert physicians) but slightly worse false positive rate. The Cohen's kappa coe cient of agreement was used to assess the concordance between the POMDP models and physicians. The kappa score of the lung POMDP and physicians decreases over time due to the large number of false positives. A large portion of di erent cases are classi ed as false positives between the lung POMDP and physicians. The breast POMDP has a high kappa score demonstrating strong agreement with physicians in terms of false positives and true positives. For both lung and breast models, the variance of kappa per screening is less than 0:03. 5
POMDPs, through the use of beliefs and a hidden state space, can overcome some
of the limitations seen in other sequential decision making models used in cancer
screening. For instance, we modeled a hidden cancer state space in three parts
[
Driven by the need to de ne the reward function in these screening POMDPs,
we explored the use of the MaxEnt IRL algorithm towards generation of
stateaction reward pairs. As noted earlier, cost and utility estimation are frequently
adopted as reward functions in healthcare models. However, cost has certain
limitations as it does not generalize to the whole population equally, and does
not re ect the importance of quality outcomes. Additionally, QALY data are
scarce, and arguably expensive to collect [
The rst limitation of using MaxEnt IRL in this study is the fact that more than one combination of rewards can de ne the same problem to overcome this problem. To overcome this, a policy iteration algorithm can be used rather than value iteration as the policy space is nite in comparison to the rewards space (hence the policy iteration algorithm is guaranteed to optimally converge). A second limitation is the assumption that reward functions are only based on state visitation frequencies. To assess the quality of these reward functions a comparison of suggested recommendations with patient satisfaction could be used. Other limitations are around assumptions about the nature of our datasets. While lung and breast cancer screening tests occurred roughly at one year intervals, we assumed that screening occurs annually (i.e., at xed frequency). Moreover, data imbalance is a function of time, as at each screening point the number of cancer and non-cancer cases changes (i.e., at the outset of a screening period, more cancers are found at the beginning of a dataset). We did not account for this dynamic nature of the dataset during training. Given the small number of cancer cases across each screening point of both datasets, we utilized a strati ed 5-fold cross-validation to obtain an unbiased estimate of model performance. To simplify modeling, our lung POMDP model considered only cases reporting a single pulmonary nodule over the course of the trial; this represents only a subset of the screened individuals, as many subjects have more than one such nding. Lastly, for the Athena dataset, in breast cancer screening, patients with BI-RADS 1, 2, or 3 rarely undergo biopsy, thus the true FN rate is likely underestimated. Future work involves the exploration of MaxEnt IRL in transfer learning between other datasets and domains, by reusing learned weights.
The authors thank the National Cancer Institute (NCI) for access to the National Lung Screening Trial data and Dr. Arash Naeim for access to the Athena Breast Health Network data collected at our institution. This material is based upon work supported by the National Science Foundation under Grant No. 1722516 and the Department of Radiological Sciences under the Data-Driven Diagnostic Decision Support (D4S) initiative.