-Despite the unprecedented volumes of Electronic Medical Records (EMRs) generated daily across healthcare facilities, the ability to leverage these data for patient participation in clinical trial remains overwhelmingly unfulfilled. The reason behind this is that matching patient information to the eligibility criteria for clinical trials is a manual, effort-consuming process. Therefore, automating this process is an essential step in improving the number of patients participating in clinical research. To address this issue, we propose a novel framework for automated patients to clinical trials matching. The matching process is based on measuring the similarity score between phrases extracted from patient medical records and the eligibility criterion for a trial. Our solution is based on a combination of NLP techniques and modern deep learning-based NLP models. In this context, we follow pre-training and transfer learning approaches to help the model learn task-specific reasoning skills. Additionally, we perform supervised fine-tuning on large Medical Natural Language Inference (MedNLI) and Semantic Textual Similarity (STSB) datasets. The matching process was performed at semantic phrases level by converting patient information and trial criteria into vector representations. We then used a scoring function that combined cosine similarity and scaling normalization to identify potential patient-trial matches. The experimental results have shown that our framework is highly effective in sorting out patients by their similarity scores. Index Terms-NLP, NLI, EMR, Automated clinical trial eligibility screening, BioBERT, Sentence similarity
The widespread adoption and use of electronic medical
records (EMRs), together with the development of advanced
artificial intelligence models, offer remarkable opportunities
for improving the clinical research sector [
Even though EMRs were designed to record information in
a structured format, such as procedure information, diagnosis
codes, drug prescriptions, and lab results, free text remains
the most flexible way for physicians to express case nuances
and clinical reasoning [
On the other hand, eligibility criteria for a clinical trial describes the characteristics of patients who are qualified to participate in the trial. Each criterion is usually expressed as a descriptive text and specified in the form of inclusion and exclusion criteria. Therefore, free text criteria can not always be transformed into structured data representations.
Authors in [
However, matching clinical notes with eligibility criteria is still a manually performed task, which makes it an expensive process in terms of time and effort. This slows down clinical trials and may delay new drugs from benefiting patients. As a consequence, it might entail the loss of human lives that otherwise would have been able to benefit from new medication. For these reasons, automated matching of clinical notes with eligibility criteria in the eligibility screening workflow would help overcome the bottlenecks of pre-screening practices in a trial setting.
To tackle the above challenge efficiently, we need to execute a matching process at a semantic sentence level, rather than by just checking for the presence or absence of a lexical criterion. The investigation of the potential use of modern deep learningbased NLP(Natural Language Processing) models, led us to propose a framework that would automate the evaluation of the eligibility of patients to be candidates for a relevant clinical trial. As a first step, the framework splits patient clinical report and clinical trial sentences into comparatively basic phrase units. Secondly, it classifies the phrases into various clinical categories (diagnosis, drug, procedure, observation). Thirdly, the framework converts candidate phrases into vector representations using an appropriate deep learning-based NLP model. Finally, it calculates a semantic matching score between patients and a clinical trial by using a combination of cosine similarity alongside a scaling normalization method.
This paper is organized as follows: In section II, we expose the problem definition and review the related works. In section III, we describe our framework and illustrate the different challenges. The evaluation of the results and outcomes is discussed in section IV. Finally, we conclude this paper in section V.
II. BACKGROUND
According to our approach, the problem definition of patient-trial matching can be described as follows:
Finding clinical trial participants is the task of matching Patient Pi(Pi 2 EM R) represented by a Discharge Summary DSi to a Clinical Trial CT represented by an Eligibility Criteria EC. Formally, the solution to this task is to find the top-K highest-values of function M which computes the matching score denoted by:
M(Pi, CT ) = v which represents the score of matching patient Pi to a CT .
This list of the top-K highest-scores reduces the overall number of patients that will need to be screened by clinicians in order to identify eligible patients.
B. Data representation
1) Clinical trial: A clinical trial is a type of research that
provides a longstanding foundation in the practice of medicine
and the evaluation of new medical treatments. Each trial has
eligibility criteria describing the characteristics according to
which a patient or participant must meet all inclusion criteria
and none of the exclusion criteria. In this respect, the criteria
differ from study to study. Authors in [
2) Patient medical records: An EMR typically collects
various types of patient information, including patient discharge
summaries, prior diagnoses, radiology reports, medication
history, and so on. Hospital discharge summaries are a
physicianauthored synopsis of a patient’s hospital stay, which serve
as the main documents communicating a patients care plan
to the post-hospital care team [
In the recent past, several projects have developed tools
and technologies for automated trial-patient matching. Milian
et al. [
In this section, we describe the framework we propose for automating the matching process between patients and a clinical trial. This framework takes into account the following different challenges; (i) In order to treat complex sentences in patient’s data as well as in clinical trials, we break down paragraphs into sentences and complex sentences are then parsed into phrases. These phrases are the basic units for matching. (ii) To avoid costly comparisons without fault dismissals, phrases are partitioned using classification methods, which limits the number of pairs to match. (iii) To match phrases, we represent them in the form of distributed vectors, which enables calculating similarity for formally different but semantically related phrases. Fig. 2 shows an overview of our Patients to Clinical Trial matching framework. Given a Clinical Trial CT and set of Patients P, our task is to calculate a Matching score M(Pi, CT ).
In order to measure the similarity between two sentences,
we have to deal with a simple sentence representing a
linguistically-meaningful unit. This process requires
segmenting both paragraph-level and sentence-level structures into
phrase-level structures. According to [
In our model, we handle each phrase as a primitive semantic unit and find matching phrases between patient and clinical
TABLE I
EXAMPLE OF SENTENCES SEGMENTATION INTO PHRASES Paragraph Phrases
Eligibility Crieteria NCT03484780 Previous open laparotomy 1- Previous open laparotomy or contraindications to 2- contraindications to laparoscopy laparoscopy, as determined by 3- determined by implanting implanting physician. physician
Discharge Summary ifiaHmnrbiytsrtoetihrcloelyarayrptddiaoioissanfetl.apwisHaneirtfiohassxtrtoaaycrnsttiyumtoiscnoaopflaocagsotturriloaanltaiorny 1fi23d4i----bsreswHHitaliaiilstssathettutooisaorryynnptoooicsfftocpamoagryruooolnxacyaatisrromydniaaaillrntaienttrhrfyiaearlpctaisotn. trials by calculating the similarity of each phrase in the discharge summary to each phrase in Eligibility Criteria (EC).
We used paragraph and sentence segmentation of
MetaMap [
A discharge summary report contains information about
different topics. Therefore, the large number of heterogeneous
phrases extracted from the patient reports may affect the
efficiency and effectiveness of pairwise phrase matching [
To minimize the number of required comparisons, we applied a filtering methodology. The latter aims to filter all the classes of phrases that do not correspond to a given class, which limits the number of pairs to match.
Data classification techniques could support achieving this filtering by separating phrases extracted from patient data and clinical trial into different medical categories. This classification filters-out non-matching pairs prior to verification, which increases the efficiency of phrases similarity matching with high precision and without sacrificing recall.
In our study, a total of 1500 eligibility criteria were extracted from a Clinical Trials database1 and were manually labelled by a certified nurse and a data science master student according to four classes (diagnosis, drug, procedure, observation).
In this work, we have empirically explored and compared
four methods widely used in classification as our baseline:
SVM, CNN, LSTM, C-LSTM [
Our experiment indicates that CNN + w2v model has the best prediction performance in comparison to the other models selected in our exploration, with a Precision of 0.87, a Recall of 0.88, and a F1-score of 0.875. We therefore adopted CNN + PubMed-and-PMC-w2v to perform this classification task and were able to categorize the phrases into the four pre-mentioned categories.
The purpose of this work is to allow the matching of patients data and clinical trials by comparing unstructured data from both datasets. Our claim is that by measuring the similarity of primitive semantic medical units (medical phrases) of a patient’s Discharge Summary and Eligibility Criteria, we can generate a score value supporting the matching task.
There are plenty of measures of semantic similarity between
sentences used in NLP. Unsupervised and supervised methods
have been used to calculate the semantic similarity between
two sentences in the biomedical domain [
1) Universal sentence embeddings: The concept of
universal sentence embeddings has grown in popularity as it
leverages models trained on large text corpora. These
pretrained models can be used in a wide range of downstream
tasks, such as providing versatile sentence-embedding models
that convert sentences into vector representations. Notable
works include ELMo [
2) BioBERT: BERT (Bidirectional Encoder
Representations from Transformers) is a neural network language model
trained on plain text for masked word prediction and next
sentence prediction tasks. BERT applies multi-layer bidirectional
transformer encoder with self-attention. According to [
3) Phrase embedding: In this respect, to generate
contextrich phrase embeddings, we chose BioBERT as the language
model in conjunction with the Bert-as-service library [
The similarity between two vectors can be evaluated using various similarity measures such as Cosine similarity, Eu
Diagnosis Drug Procedure Diagnosis Drug Procedure Fine-tuned BioBERT
clidean distance, and Manhattan distance. Since these simi- reducing thus the need for many heavily-engineered
tasklarity metrics are a linear space in which all dimensions are specific architectures.
weighted equally, we perform here the similarity matching In the context of natural language understanding (NLU)
metrics of different phrases by ranking these phrases according technology, comparing the relationship between two
sento the cosine similarity. Therefore, the rank of similarity can tences is based on several downstream tasks such as Natural
be obtained by the equCalintiiocanlsTrpiarelsented in (1) and (2). Language Inference (NLI) and Semantic Textual Similarity
iec1 iec2 iec3 exec·4y (STS) [
BioBERT-based model. We then further fine-tuned on MedNLI
Whereas a pre-trained BioBERT knowledge often shows a dataset. We used the fine-tuning classifier from BERT
sysgood performanceec1 for ecc2ertaienc3 taskesc4, as Swcoere shall see later on, tems [
History of hypercholesterolemia and the patient was in a MVC. spoempteicyuelacresradgiosewasaes sin/pvoglavsetrdicinbayploasws- the patient has no medical history. speed MVC.
the patient has no significant injuries. • DSi = {phi,1, phi,2, ..., phi,r} as the phrases extracted
from Discharge Summary of patient Pi. • I EC = {iec1, iec2, ..., iecp} as the phrases extracted
from Inclusion Eligibility Criteria. • EEC = {eec1, eec2, ..., eecq} as the phrases extracted
from Exclusion Eligibility Criteria. • EC = {ec1, ec2, ..., ecl} = I EC [ EEC | l = p + q as
all phrases extracted from Eligibility Criteria.
• S 2 [
1) Matching Patient to Eligibility Criteria: Once phrases embedding are computed for the patients and the clinical trial eligibility criteria, we calculate the similarity between phrases of the same class (Diagnosis, Drug, Procedure,... ) as defined in sub-section III-B. An element si,j of S represents the similarity between patient criteria Pi and single eligibility criteria ecj . The similarity function is defined by calculating the cosine between each phrase phi,r extracted from DSi and ecj , then only the higher cosine value of similarity is retained for si,j and all other values are discarded.
si,j =
max (cos(phi,r, ecj )) 8 phi,r2 DSi
i 2 [1, n] &j 2 [1, l]
Once the similarity values obtained, the final representation of S would be as follows:
2 maxph1,r (cos(ph1,r, ec1)) S = 4 .
.
.
. 5 maxphn,r (cos(phn,r, ecl))
3 2) Ranking and Scoring Patients: The semantic cosine similarity calculated in the previous paragraph enables a proportional similarity instead of exact text semantic matching.
Therefore, when we compare similarity values obtained for different features (eligibility criteria) in the generated matrix S, we notice that just because the value of similarity is higher, that does not mean that the similarity with the patient is greater. For example if sx,1 and sy,2 represent the highest value of the features ec1 and ec2, respectively, and if sx,1 > sy,2, this does not mean that Px has a phrase more similar to ec1 than Py for ec2 (as a noticed in equation 2), but only means that Px and Py are ranked respectively at the top similar of the list for ec1 and ec2. The same logic applies for the lowest value, which represents the last order of similarity.
This variation in the similarity values between features requires a range normalization step to enable rank similarity instead of cosine similarity, which supports perfectly the computation of a matching score between patients and the Clinical Trial. To this end, we generated a new matrix R by applying the following feature scaling normalization: ri,j = ( n ⇥ maxs8ii,(jsi,mj)in8 mi(insi8,ij()si,j ) ( n) ⇥ maxs8ii,(jsi,mj)in8 mi(ins8i,ij()si,j ) ; ; ecj 2 I EC ecj 2
EEC (3)
Finally, the matching score M of Patient Pi with a Clinical Trial is determined by:
M(Pi, CT ) =
l X rij.
j=1
To validate our framework, we used two datasets;
MIMICIII (Medical Information Mart for Intensive Care) [
MIMIC III Clinical Dataset is a critical care database that contains 2,083,108 medical reports from 46,520 patients. We experimented with a randomly selected dataset of 100 Discharge Summaries from patients last visit, excluding patients whose ages are under 18. The segmentation stage produces an average of 400 phrases per report.
We selected a clinical trial that identifies the role of Aldosterone antagonist in patients of heart failure with preserved ejection fraction (NCT04078425). Fig. 3 shows the five eligibility criteria of this clinical trial.
Table III presents the results for a sample of ten patients. In order to evaluate the clinical correctness of patients matching to the clinical trial(NCT04078425), a validation task was performed manually by a nurse and a computer science student. The noteworthy fact is that the evaluation of the matching does not reveal false positives in the score results. Indeed, the similarity scores reflect the order of matching between patients and the clinical trial. The score distribution ranged from (-15) to (8), and eligible patients to be retained for further screening by experts were those with a score greater than 5.
We should note that the scores would be more realistic if the segmentation process was more accurate. For instance, the sentence ”you were thought to have a blood clot in your right leg” was segmented by Metamap into ”a blood clot in your right leg” which would result in a false outcome.
EMRs contain a large portion of unstructured data that need to be matched with eligibility criteria for trial-patient enrollment. Indeed, the gradual improvement of artificial intelligence technology could reduce the number of physician-hours spent in screening patient eligibility. To tackle the problem, we proposed a framework designed to automatically recommend the most suitable patients for a clinical trial. The framework adopts a pre-trained language model (BioBERT) and uses STS-B and MedNLI datasets to improve the accuracy of the model via transfer learning. This work verified that the fine-tuning of BioBERT shows better performance in calculating the similarity between two medical sentences using embedding-based metrics. In future works, we will also explore EMRs structured tables in order to significantly improve the performance and accuracy of our trial-patient matching framework.
The authors would like to thank Marvin Moughabghab for his efforts and contributions to this work.