Clinical trials are important for patients, for researchers and for companies. One of the major bottlenecks is patient recruitment. This task requires to match a great quantity of information about the patient with numerous eligibility criteria, in a logically-complex combination. Moreover, the patient's information required by some of the eligibility criteria may not be available at the time of pre-screening. In such situations, the classic approach based on negation as failure ignores the distinction between a trial for which patient eligibility should be rejected and trials for which patient eligibility cannot be asserted, which resuls in underestimating recruitment. We propose an OWL design pattern for modeling eligibility criteria based on the open world assumption to address the missing information problem.
A major focus in all clinical trials is the recruitment of patients. Adequate
enrollment provides a base for projected participant retention, resulting in evaluative
patient data. Identi cation of eligible patients for clinical trials (from the
principal investigator perspective) or identi cation of clinical trials in which the
patient can be enrolled (from the patient perspective) is an essential phase of
clinical research and an active area of medical informatics research. The
National Cancer Institute identi ed several barriers that health care professionals
claim in regards to clinical trial participation [
Automated tools that help perform a systematic screening either of the
potential clinical trials for a patient, or of the potential patients for a clinical
trial could overcome this barrier [
All these works on data and criteria representation, integration and reasoning
are motivated by the requirement to have the necessary information available at
the time of processing the patient's data, and assume that somehow, that will be
the case. Missing information that is required for deciding whether a criterion
is met leads to underestimating recruitment. Solutions for circumventing this
di culty consist either in making assumptions about the undecided criteria, or
in having a pre-screening phase considering a subset of the criteria for which
patient's data are assumed to be available. Bayesian belief networks have been
used to address the former [
We propose an OWL design pattern for modeling clinical trial eligibility criteria. This design pattern is based on the open world assumption for handling missing information. It infers whether a patient is eligible or not for a clinical trial, or if no de nitive conclusion can be reached. 2 2.1
A clinical trial can be modeled as a pair < (Ii)in=0; (Ej )jm=0 > where (Ii)in=0 is the set of the inclusion criteria, and (Ej )jm=0 is the set of the exclusion criteria. All the eligibility criteria from (Ii)in=0 [ (Ej )jm=0 are supposed to be independent from the others (at least in the weak sense: the value of criterion Ck cannot be infered from the combined values of other criteria). Each criterion can be modeled as an unary predicate C(p), where the variable p represents all the information available for the patient. C(p) is true if and only if the criterion is met.
A patient is deemed eligible for a clinical trial if all the inclusion criteria and none of the exclusion criteria are met.
n m patient eligible , i=^0Ii(p) ^ :(j_=0Ej (p))
Before the nal decision on the list of clinical trials a patient is eligible for, there are intermediate pre-screening phases where only the main eligibility criteria of each clinical trial are considered. Such pre-screening sessions rely on subsets of (Ii)in=0 and (Ej )jm=0, but the decision process remains the same. For the sake of clarity, in addition to the general case, we will consider a simple clinical trial with two inclusion criteria I0 and I1, and two exclusion criteria E0 and E1.
patient eligible , I0(p) ^ I1(p) ^ :(E0(p) _ E1(p)) (2) (1) (3)
For example, these criteria could be: { I0: evidence of a prostate adenocarcinoma; { I1: absence of metastasis; { E0: patient older than 70 years old; { E1: evidence of diabetes.
According to equation 2, a patient would be eligible for the clinical trial if and only if he has a prostate adenocarcinoma and has no metastasis and is neither older than 70 years old nor has diabetes.
Because of De Morgan's laws, equation 1 is equivalent to:
n m patient eligible , (i=^0Ii(p)) ^ (j^=0:Ej (p))
Even though equation 1 and equation 3 are logically equivalent, the latter is often preferred because it is an uniform conjunction of criteria. Note that the negations in front of the exclusion criteria are purely formal, as both inclusion and exclusion criteria can represent an asserted presence (e.g. prostate adenocarcinoma for I0 or of diabetes for E1) or an asserted absence (e.g. metastasis for I1).
For our example: patient eligible , I0(p) ^ I1(p) ^ (:E0(p)) ^ (:E1(p)) (4)
According to equation 3, a patient would be eligible for the clinical trial if and only if he has a prostate adenocarcinoma and has no metastasis and is not older than 70 years old and has not diabetes. 2.2
Distinction between the patients that we know are not eligible and those that we do not know if they are eligible When a part of the information necessary for determining if at least one criterion is met is unknown, the conjunction of equation 3 can never be true. This necessarily makes the patient not eligible for the clinical trial, whereas the correct interpretation of the situation is that the patient cannot be proven to be eligible. This is di erent from proving that the patient is not eligible, and indeed, in reality the patient can sometimes be included by assuming the missing values (cf. next section).
For our ctitious clinical trial, we consider a population of nine patients covering all the combinations of \True", \False" or \Unknown" for the inclusion criterion I1 and the exclusion criterion E1. Table 1 presents the value of equation 4 and correct inclusion decision for the nine combinations. Among the ve patients (p2, p5, p6, p7 and p8) for which at least a part of the information is unknown, three (p2, p7 and p8) illustrate a con ict between the value of equation 4 and expected inclusion decision. A strict interpretation of equation 4 leads to the exclusion of the eight patients: { for three of them (p0, p3 and p4), all the information is available; { for two of them (p5 and p6), some information is unknown, but the available information is su cient to conclude that the patients are not eligible; { for the three others (p2, p7 and p8), however, the cause of rejection is either because one of the inclusion criteria cannot be proven (I1 for p7 and p8) or because one of the exclusion criteria cannot be proven to be false (E1 for p2 and p8).
Therefore, if we generalize, equation 3 alone is not enough in the case of partially-known information to make the distinction between the patients we know are not eligible (the rst two categories, so this also includes patients for whom a part of the information is unknown) and those we do not know if they are eligible (the third category). This is a problem because patients from the rst two categories should be excluded from the clinical trial, whereas those from the third category should be considered for inclusion.
Patient I0 I1 E0 E1 I0 ^ I1 ^ :E0 ^ :E1 p0 p1 p2 p3 p4 p5 p6
T T F T T T F F T T F ? T F F T T F F F T F F ? T ? F T
F cannot assert :E1
F T F F F F
Decision Exclude
(E1) Include
Propose (assume :E1)
Exclude (both :I1 and E1)
Exclude
(:I1) Exclude
(:I1) Exclude (E1) Assuming values for criteria Currently, the case of each patient diagnosed with cancer is examined in a multidisciplinary meeting (MDM), gathering (oncologists, pathologists, surgeons,...). The goal is to determine collectively the best therapeutic strategy for the patient, including consideration of potential inclusion into clinical trials. This preliminary stage is called pre-screening because it takes place before obtaining informed consent (i.e., before enrollment). It mainly relies on retrospective data coming from the patient health record. At this point, all the information necessary for determining the status of each inclusion and exclusion criteria may not be available, but the rationale is to focus on the clinical trials the patient may be eligible for. It should be noted that the missing items may di er between patients. One solution could be to assume the values of the unknown criteria in order to go back to a situation where inclusion or exclusion could be computed using equation 3.
In this case: { inclusion criteria for which the available information is not su cient to compute the status are considered to be met; { exclusion criteria for which the available information is not su cient to compute the status are considered not to be met.
Therefore, in the case where the available information is not su cient to compute the status of a criterion, a di erent status is assumed depending on whether the criterion determines inclusion or exclusion.
Referring to our ctitious clinical trial, the lack of information about the absence of metastasis would lead to the assumption that I1 is true, whereas the lack of information about diabetes would lead to the assumption that E1 is false.
This situation raises several issues: { a di erent status is assumed depending on whether the criterion determines inclusion or exclusion; { the assumed status depends on the nature of the criterion (i.e. inclusion or exclusion) and not on its probability; { one has to remember that the value for at least a criterion has been assumed in order to qualify the inferred eligibility (adamant for p0 or p1 vs \under the assumption that..." for p2, p7 and p8); { this quali cation can be di cult to compute (the status of E1 is unknown for both p2 and p5, but p5 can be con dently excluded whereas p2 can be included assuming E1). 2.3
To determine the extent of the missing information problem, we analyzed the 286 prostate cancer cases examined during the weekly urology multidisciplinary meetings at Rennes' university hospital between October 2008 and March 2009. This involved 252 patients: 25 of them were examined during two di erent MDM, and 5 were examined during three di erent MDM. Before the MDM, the patient's data are collected in a form with 59 elds. The form supports the distinction between known and unknown values (e.g. for \antecedent of neoplasm", the possible answer are \yes", \no", \not speci ed").
Overall, 58.64% of the values were unknown. On average, for each case studied in a MDM, 34.6 elds (among 59) had an unknown value.
All of the 286 cases studied had at least some of the 59 elds with an unknown value. Indeed, the case with the most elds lled still missed 19 of them.
54 elds (91.53% of 59) had a missing value in at least one of the 286 cases. The ve elds that were systematically lled were: the patient identi er, the MDM date, the patient's gender, the tumor anatomic site and the primary histological type.
During this period, 4 clinical trials related to prostate cancer running at Rennes Comprehensive Cancer Center were considered during the MDM. Table 2 presents the composition of the clinical trials elds and their proportion of missing information. It shows that for each clinical trial, all the patients had at least one missing eld that prevented formula 3 to be true (regardless of the values of the known elds).
Nb inclusion elds Nb exclusion elds Nb common elds Missing values Nb patients with all inclusion elds known Nb patients with all exclusion elds known Nb patients with all elds known Nb eligible patients CT1 CT2 CT3 CT4 15 19 17 10 10 9 13 11 3 0 3 3 50.06% 61.72% 52.99% 42.99% 0 0 1 1 4 3 0 1 0 0 0 0 30 23 7 2 We propose an OWL design pattern for modeling clinical trial eligibility criteria. We then explain how the reasoning unfolds using the ctitious clinical trial from table 1. We validate our approach by verifying if the inferred outcome corresponds to the expected value from table 1. We evaluate our approach on the four clinical trials related to prostate cancer and the 286 cases mentionned at section 2.3. This allows us to quantify the impact of missing information on inclusion rates, as we have seen that in some cases, even partially-known information can lead to certain rejection.
{ for each criterion, create a class C i (at this point, we do not care if it is an inclusion or an exclusion criteria, or both) and possibly add a necessary and su cient de nition representing the criterion itself (or use SWRL); { for each criterion, create a class Not C i de ned as
Not C i Criterion u: C i.This process can be automated; { for each clinical trial, create a class Ct k (placeholder); { for each clinical trial, create a class Ct k include as a subclass of Ct k with a necessary and su cient de nition representing the conjunction of the inclusion criteria and of the exclusion criteria (cf. equation 3) (Ct k include n m u I i uju=0 Not E j); i=0 { for each clinical trial, create a class Ct k exclude (placeholder) as a subclass of Ct k; { for each clinical trial, create a class
Ct k exclude at least one exclusion criterion as a subclass of Ct k exclude with a necessary and su cient de nition representing the disjunction of the exclusion criteria (Ct k exclude at least one exclusion criterion m t E j ); j=0 { for each clinical trial, create a class
Ct k exclude at least one failed inclusion criterion as a subclass of Ct k exclude with a necessary and su cient de nition representing the disjunction of the negated inclusion criteria (Ct k exclude at least one failed incl criterion n t Not I i ); i=0 { represent the patient's data with instances (Fig. 1 and 2). For the sake of simplicity, we will make the patient an instance of as many C i as we know he matches criteria, and as many Not C j classes as we know he does not match criteria, even if this is ontologically questionable (a patient is not an instance of a criterion). How the patient's data are reconciled with the criteria by making the patient an instance of the criteria is not speci ed here: it can be manually, or automatically with necessary and su cient de nitions or SWRL rules for the C i and Not C j classes. 4.2
If all the required information is available, after classi cation the patient will be an instance of each C i or Not C i, and therefore will also be instantiated as either Ct k include (like p1 in Fig. 3), Ct k exclude at least one exclusion criterion or Ct k exclude at least one failed inclusion criterion (so at least we are doing as well as the other systems).
Fig. 1. A patient for who all the information is available
If not all the information is available, because of the open world assumption, there will be some criteria for which the patient will neither be classi ed as an instance of C i nor of Not C i (Fig. 2), so he will not be classi ed as an instance of Ct k include either. However, the patient may be classi ed as an instance of Ct k exclude at least one exclusion criterion or of Ct k exclude at least one failed inclusion criterion. As both are subclasses of Ct k exclude, we will conclude that the patient is not eligible for the clinical trial. We will even know if it is because he matched an exclusion criterion (like p0, p3 and p6 in Fig. 4), because he failed to match an inclusion criterion (like p3, p4 and p5 in Fig. 5), or both (like p3).
If the patient is neither classi ed as an instance of Ct k include nor of Ct k exclude (or its subclasses), then we will conclude that the patient can be considered for the clinical trial, assuming the missing information will not prevent it (like p2, p7 and p8, who do not appear in Figs. 3, 4 and 5, consistently with Table 1). By retrieving the criteria for which the patient is neither an instance of C i nor of Not C i, we will know which information is missing. We modeled our ctitious clinical trial from section 2.1 as well as the nine combinations of values from section 2.24. All the results were identical to the decision of table 1. We evaluated our model on the rst clinical trial (work is ongoing on the three others)5. Among the 286 cases, 0 were formally eligible, 122 were potentially eligible, and 164 were not eligible. The 30 cases that were identi ed as eligible by the experts during the multidisciplinary meetings were all among the 122 proposed by our system (precision was 0.24; recall was 1.0).
It should be noted that the a posteriori analysis of the 92 cases proposed by our model but not by the MDM revealed that several were not proposed even if they formally met the eligibility criteria because their Gleason score was deemed too low. We added an inclusion criterion requiring patients to have a Gleason score superior or equal to 7. This resulted in 54 cases potentially eligible, among which were 25 of the 30 actually eligible (precision was 0.46; recall was 0.83). The ve false negative cases had a Gleason score of 6. Among the 29 false positive, at least 15 were rejected during the MDM because of additional information not available at the time of pre-screening: 8 because new results indicated that they did not have cancer, 3 because too much information was missing and 4 because other elements such as a relatively young age resulted in proposing a surgical treatment instead of the clinical trial. 4 http://www.u936.univ-rennes1.fr/dameron/clinicalTrial/ct-validation.tgz 5 http://www.u936.univ-rennes1.fr/dameron/clinicalTrial/ct-getug14.tgz
The analysis of the rst clinical trial demonstrates that missing information would have lead to the rejection of all the 30 patients proposed as eligible by the experts during the multidisciplinary meetings. Our approach correctly identi ed these 30 cases among the 122 it proposed as potentially eligible. This shows that our system con dently rejects non-eligible cases, which leaves more time to examine the others during the multidisciplinary meetings. Moreover, precision can be signi catively improved by adding pragmatic criteria that further discriminate the patients who would not be considered as eligible even if they meet the pre-screening criteria. Note that this second step can be kept separated from the formal determination of eligibility but is useful both for the acceptance of the system by the experts and for maintaining the e ciency of the multidisciplinary meetings.
Missing information can partially be handled even with reasoning based on negation as failure using ad hoc conversion between inclusion and exclusion criteria. For example, the inclusion criterion \absence of ischemic heart disease" can be converted into the exclusion criterion \presence of ischemic heart disease". The former will probably never be met because a patient's record only mentions ischemic heart disease when they are present, whereas the latter will (correctly) only exclude those patients having evidence of ischemic heart disease. The problem is that if \absence of ischemic heart disease" had been an exclusion criterion, it would likewise have been converted into the inclusion criterion \presence of ischemic heart disease" and the system would have (incorrectly, at least during pre-screening) rejected patients whose record does not mention the presence nor the absence ischemic heart disease. Moreover, a criterion can be an inclusion criterion for a clinical trial and an exclusion criterion for another trial, so this strategy is not a general solution to the problem of missing information.
Reasoning about the conjunction of the eligibility criteria should be handled by OWL, which supports the open world assumption, rather than by related technologies such as SWRL which do not. It would be possible to write a SWRL rule that represents the conjunction of criteria (cf. formula 3). However, it is impossible to distinguish situations where we know that one criterion is not met from those where we cannot determine if it is met, because in both cases the rule ll not re.
Potential applications of our approach are not limited to clinical trials [
We showed that ignoring the missing information problem for automatic determination of clinical trial eligibility lead to over-estimate rejection. Systems based on negation as failure infer that the patient is not eligible if it cannot be proved that the is eligible, whereas the situations where it cannot be determined that the patient is eligible nor that he is not eligible should be identi ed and treated separately. A retrospective analysis of 252 patients with prostate cancer showed that for the four clinical trials of interest, all the patients had at least one missing value that had them rejected whereas 62 of them were actually eligible for at least one of the clinical trials.
We proposed a modeling strategy of eligibility criteria in OWL that leveraged the open world assumption to address the missing information problem. Our approach was able to distinguish a clinical trial for which the patient is eligible, a clinical trial for which we know that the patient is not eligible and a clinical trial for which the patient may be eligible provided that further pieces of information (which we can identify) can be obtained.
The ASTEC project is funded by the French Agence Nationale pour la Recherche (ANR 08-TECS-002).