Transformer deep learning models, such as BERT, have demonstrated their effectiveness over previous baselines on a broad range of general-domain natural language processing (NLP) tasks such as classification, named entity recognition, and question answering (Devlin et al. 2018). They also exhibit enhanced performance in domain-specific NLP tasks, including BioNLP tasks (Lee et al. 2019; Alsentzer et al. 2019). In this study, we focus on clinical trial protocols: exploring and extracting key terms (a named entity recognition task) as well as their relations (a relation extraction task) from the protocols using transformer pre-trained deep learning models. We compare several model configurations and report their results. Our NLP model achieves good performance considering the complex and unique nature of the language in real-world protocols, and has been integrated into the organization's protocol analytics practice. This approach and the extracted information will greatly facilitate trial feasibility analysis for developing new drugs.
Clinical trial protocols (often called “study protocols”) contain key information specifying the trial design and implementation, but are usually in unstructured or semi-structured format, which presents a huge challenge for running computational analysis on them. Due to protocols’ critical role, drug development businesses, such as contract research organizations, have been devoting significant amount of resources in analyzing study protocols to precisely understand the operational requirements, comprehensively evaluate the systemic challenges, unbiasedly assess the probability of success, accurately forecast the cost implications for optimal business planning. Currently, this protocol analysis work is still performed in a labor-intensive fashion, involving numerous resource checking and cross referencing works. To develop safer, cheaper and more effective drugs faster for better public health, this presses an urgent need for more efficient and effective ways to process text-based protocols.
Here, we present our efforts to facilitate the protocol analysis workflow by automating the process of extracting key information from the protocols using natural language processing (NLP) techniques. More specifically, we focus on the eligibility criteria section in the protocols, which contains patient selection criteria information; we extract key clinically relevant entities (i.e. named entities) and entity relations (i.e. syntactic relations) from this section. Based on the extracted information, the unstructured protocols can be transformed into a structured network with interconnected key entities (e.g. condition, drug, observation etc.) that can be fed into various data-based analytic tasks, for example to query against various real-world evidence databases for patient population estimation, which is critical for clinical trial design in drug development.
Covance Inc. is the world’s largest provider for clinical trial design, monitoring, managing and central lab testing services, and has accumulated large volume of study protocols. The presented work is our first step of a bigger mission towards solving the protocol analysis challenge. To this end, we employed the transfer learning strategy and experiment with deep learning family of algorithms by using the recently developed Bidirectional Encoder Representations from Transformers (BERT) based models and fine-tuning them on our in-house clinical trial protocol corpus to identify the named entities and their relations.
Study protocols are rigorous scientific documents with highly domain-specific terms and complex relations. These characteristics bring both benefits and challenges to NLP work: we concern less about preprocessing due to its rigorous use of language, but need to attend more to its unique yet complex clinical terms and relations. A study protocol’s eligibility criteria section is usually composed of two parts: inclusion criteria and exclusion criteria, which respectively describe the unambiguous characteristics of patients to be included in and excluded from the clinical trial. The general public can access some simplified protocol texts via websites such as ClinicalTrials.gov, which already contain many clinical terminologies. However, the real protocols are much longer with even more domain-specific terms, thus more difficult for the NLP task. We employ pre-trained BERT transformers to tackle this challenging NLP task and our study provides quantified evidence of how BERT performs in the clinical trial domain.
In our practice, the extracted information are stored in a structured format. Figure 1 shows an example: the inclusion criteria is represented as several key-value clauses such that we can query a patient database to find the patients satisfying these criteria. Through extraction we are essentially connecting dots to build a larger graph for knowledge engineering purpose, i.e. we connect protocol text to patient database records, connect protocol to condition terms in a medical ontology, and so on. Once the dots are properly connected, we are empowered to perform many protocol analysis tasks such as building a search engine for precise search, composing graph networks for graph analysis for capturing missing links, evaluating drug effectiveness by comparing with similar drugs, clustering and recommending similar protocols for study feasibility analysis.
Named entities recognition (NER) and relation extraction
(RE) are two classical natural language processing (NLP)
tasks, which we carry out to extract entities and syntactic
relations respectively in our study. Previously, for NER,
researchers have mainly investigated probabilistic sequence
labeling models such like conditional random fields (CRF),
maximum entropy Markov models, and hidden Markov
model
In recent years, with the advances in deep neural network
methods, significant performance improvement has been
achieved for the NER and RE tasks. For NER tasks,
embeddings are widely used in neural network models to
represent words or characters as high-dimensional vectors.
Recurrent neural networks (RNN), including LSTM, GRU, and
their variants, are applied because their architectures
represent better the sentence context as well as the dynamic
sentence length in natural languages
Despite the improvement from previous models, RNN
and LSTM models tend to “forget” earlier context in long
sequences, which limits the model performance.
Transformers are subsequently proposed to counter this issue.
Transformer models use the attention mechanism that attends to
each word in a sequence by replacing the sequence-based
RNN style network structure with dot products and
multiplications between the key/value/query matrices projected
from the embedding vectors
Additionally, transfer learning aims to transfer pre-trained
model from one task to another, usually by training a
general language model on general-domain data set and
transferring it to a downstream task by fine-tuning on the
taskspecific data set. A number of pre-trained language
models have been created to facilitate downstream tasks such
as NER and RE, examples including ELMO, ULMFit,
OpenAI GPT, and BERT, which have outperformed
previous baselines and some even achieved the state-of-the-art
performance
Based on the original BERT architecture, a number of
BERT variants have emerged with alterations for
different purposes. For example, RoBERTa removes next
sentence prediction from the original loss function along with
some other hyperparameter changes; Transformer-XL
captures context both within and between segments for
tackling long-term dependency across sentences; and T5
advocates for encoding-decoding architecture, denoising
objectives and other changes based on extensive experiments
NER and RE have also been longstanding tasks in the
biomedical NLP domain. Researchers have investigated
applying similar yet more customized approaches to
biomedical texts, such as using CRF models and BiLSTM+CRF
neural networks
In the clinical informatics field, it is important to convert
unstructured criteria text to structured format because this
enables people to automatically parse a criteria and query for
proper patients against certain real-world evidence database.
Therefore, NER and RE algorithms are an appropriate and
natural fit to this practice: NER extracts concepts such as
conditions and observations that is related to a patient; RE
provides operational information such as the range for a
particular lab test result for patient selection. Criteria2Query is
a pioneering work in the space of translating study criteria
to SQL queries
To facilitate our NLP approach, we selected 470 study protocols from Covance’s in-house protocol database. And our protocol corpus comprises eligibility criteria sections from these selected study protocols. An eligibility criteria section typically contain 5 - 20 sentences that define the criteria to select and recruit patients for the clinical study. Our data contain a total of 30,183 criteria sentences.
Data Annotation. We have the eligibility criteria
annotated using the IOB format
Entities: Condition, Observation, Procedure, Device, Drug, Investigational product, Event, Refractory condition, Demographics, Measurement, Temporal constraints, Qualifier/modifier, Anatomic location, Negation cue, Permission cue
Syntactic relations: Has value, Has temporal constraint, Modified by, Located in, Is negated, Is permitted, Specified by
Data Split. For the NER task, we randomly split the 30,183 sentences into training (60%, 18,109 sentences) and test (40%, 12,074 sentences) sets. For the RE task, before splitting the data for training and testing, we first check whether a sentence contains multiple relations and if so, we duplicate the sentence for each pair of related entities and make their relation type as the label for classification. This results in 52,470 relation sample sentences, based on which we perform a random split with stratification on relation classes to derive training (60%, 31,482 relation samples) and test sets (40%, 20,988 relation samples). Tables 1 and 2 show data statistics for the NER and RE tasks.
As previously mentioned, we use NER algorithms to extract
clinically relevant entities in eligibility criteria section and
particularly choose BERT, a pre-trained transformer type of
deep learning model, because of its reported superior
performance in many NLP tasks. Due to the attention
transformer in BERT, it is able to provide dynamic context
embedding for tokens, which helps addressing the polysemy
issue. BERT is a language model pre-trained on a large
general domain corpus and can be applied towards downstream
tasks by adding simply structured task layers and fine
tuning on task-specific data set. We hereby follow the fine
tuning practice based on pre-trained models to derive our NER
model
NER task layers. The original BERT paper indicates that
when use for NER tasks, the pre-trained BERT model can
be simply followed by a softmax layer where each token is
classified to their most likely entity class without adding any
CRF layer
Cased or uncased. The BERT model provided by Google includes versions with and without lowercasing preprocessing on the tokens. We experiment with both the cased (not applying lowercasing) and uncased (applying lowercasing) options. Consequently, the two options use different set of subword vocabularies, with cased model of 28,996 subwords and uncased model of 30,522 subwords.
Pre-trained models. We use BERT-base, a smaller version of BERT that comprises 110 millions of parameters, in our first set of experiments. BERT also has a larger version, BERT-large, with 340 millions parameters. We opt to use BERT-base for exploration purposes. In our second set of experiments, we test the BioBERT model that is retrained using large-scale biomedical texts on the basis of the original BERT model. BioBERT has only a cased version and shares the same vocabulary as BERT-base cased (with size of 28,996).
Hyperparameters. For both BERT-base and BioBERT models, we set num of epochs=20, learning rate=2 10 5, training batch size=32, max sequence length=32. For cases when using BiLSTM+CRF as task layers, we set bilstm layer size=128.
The above model options result in 6 NER models:
BERTbase;uncased; Sof tmax: BERT base uncased
pretrained model, softmax as NER task layer
BERTbase;cased; Sof tmax: BERT base cased
pretrained model, softmax as NER task layer
BioBERT; Sof tmax: BioBERT pre-trained model
(cased), softmax as NER task layer
BERTbase;uncased; BiLST M + CRF : BERT
pretrained uncased model, BiLSTM+CRF as NER task layer
BERTbase;cased; BiLST M + CRF : BERT base
pretrained cased model, BiLSTM+CRF as NER task layer
BioBERT; BiLST M + CRF : BioBERT pre-trained
model (cased), BiLSTM+CRF as NER task layer
The layout of the BERT NER neural architecture is shown
in Figure 2.
The RE task is also treated as a downstream task to the
pretrained models. The original BERT paper did not include RE
task as one of their downstream tasks, whereas the BioBERT
study investigated it due to its importance in the biomedical
NLP domain
We employ the following strategy for the RE task: In training, we first scan through each sentence for entities using human annotations, and record the token positions of each entities; if a sentence contains n (n > 1) pairs of entities with human annotated relation, we duplicate this sentence n times so that each instance target represents one pair of entities and their relation; In prediction, we use NER pipeline results to locate entities, enumerate all legitimate entity pairs, and duplicate sentences accordingly. Since we record the token positions of each entity pair, we can get BERT output vectors for them based on their position information, concatenate the two vectors and then feed it to a softmax layer to classify their relation. The result can be one of the 7 relations listed in Table 2 or ‘no relation’.
More specifically, the input fed to the BERT RE model is sentence text along with positions of entity pairs. We do not make use of entity type information for the following reasons: 1) this end-to-end (i.e. tokens-to-relation) practice
Sof tmax
Sof tmax BioBERT; Sof tmax
BiLST M + CRF
BiLST M + CRF BioBERT; BiLST M + CRF
Type strict exact partial macro strict exact partial macro strict exact partial macro strict exact partial macro strict exact partial macro strict exact partial macro makes the RE model more useful as a standalone tool that does not require entity type; 2) when in prediction mode, the errors in entity prediction could propagate to the RE task, which we mitigate by including only the entity position information. Figure 3 shows the neural architecture of our RE task.
For training purposes, we randomly generate negative samples for the ‘no relation’ class as two entities can have no relations with each other. We use two ways to obtain negative samples: one way is to randomly choose two unrelated entities in a sentence, the other is to break an existing related entity pair and establish a non-related pair between one of the entities in the original pair and another unrelated entity in the sentence.
Similar to the NER task, we experiment with 3 pre-trained models with softmax as the task layer for all of them: BERTbase;uncased: BERT base pre-trained model, uncased BERTbase;cased: BERT base pre-trained model, cased BioBERT : BioBERT pre-trained model (cased) Following hyperparameter configuration is num of epochs=20, learning rate=2 10 5, ing batch size=32, max sequence length=32. used: train
We implement the NER and RE tasks using Tensorflow based on the BERT neural architecture and run experiments on an AWS p2.xlarge GPU instance.
We follow the practice in the SemEval-2013’s Drug-Drug Interactions task and evaluate NER performance by 3 matching standards: strict, exact, and partial (Segura-Bedmar, Mart´ınez, and Herrero-Zazo 2013). The strict matching evaluates both boundary and entity type of entity phrases; the exact matching evaluates the exact boundary regardless of entity type; and the partial matching measures the partial boundary of entities regardless of entity type (thus the most lenient). We calculate precision(P)/recall(R)/f1-score(F) for the three evaluation types, and additionally, we also report macro average P/R/F results. The results are shown in Table 3.
In our experiments, fine-tuning the pre-trained BioBERT model achieves slightly better performance than its BERT counterparts. For example, BioBERT; Sof tmax has f1score of 70.61, better than BERTbase;uncased; Sof tmax’s 69.80 and BERTbase;cased; Sof tmax’s 69.68. Similarly, BioBERT; BiLST M + CRF holds a higher f1score than BERTbase;uncased; BiLST M + CRF and BERTbase;cased; BiLST M + CRF for all the four evaluation types.
When comparing the cased and uncased strategies, we notice that the uncased pre-trained models outperform the cased ones with the same neural architecture: e.g. BERTbase;uncased; BiLST M + CRF achieves f1-score of 70.28 for the strict evaluation type, higher than the f1score of 69.89 from BERTbase;cased; BiLST M + CRF . F 69.80 73.16 77.55 64.63 69.68 73.15 77.49 64.63 70.61 73.83 78.06 65.03 70.28 73.62 77.98 64.88 69.89 73.23 77.56 64.45 70.76 74.06 78.36 65.54 This finding suggests that applying lowercase to preprocessing actually enhances performance slightly, which is counter-intuitive for NER tasks as the entities are often casesensitive. Meanwhile, we also find that the two BioBERT models, which are cased, perform better than their peer models of the same neural architecture. But since BioBERT only offers the cased option, we cannot discern the relative contribution from being cased in the BioBERT pre-trained model.
From Table 3, it is not surprising that for a given model, the partial evaluation usually holds the highest score, followed by exact, strict, and macro. Another observation is that when we loosen evaluation type from strict to exact, i.e. focusing on entity boundary without penalizing entity type errors, the performance is improved but still remains in the 73.15-74.06 range, suggesting that the experimented BERT based models fail identify entity boundary very precisely, which can be of interest for future investigation.
In our experiments with simple Softmax as the task layer, we observe more boundary detection errors. This in fact is the motivation for us to add the BiLSTM+CRF layers as the NER task layer. However, the results show that given the same pre-trained model configuration, it is debatable that BiLSTM+CRF could consistently improve performance. For example, BioBERT; BiLST M + CRF slightly outperforms BioBERT; Sof tmax in strict matching precision and f1-score, but BioBERT; Sof tmax beats BioBERT; BiLST M + CRF in strict matching recall.
We also find that the recall score is consistently higher than the precision score for all models at all evaluation stanType micro macro weighted micro macro weighted micro macro weighted dards, indicating that the models tend to have more false positive predictions than false negative predictions. The macro scores show lower performance than strict/exact/partial because it simply averages the performance of different entity types and some small-sample entity types have lower performance due to lack of training data.
Overall, BioBERT ; BiLST M + CRF produces the best precision and f1-scores for all the four evaluation types whereas BioBERT ; Sof tmax holds the highest recalls. These results suggest that fine-tuning BioBERT lends itself better to the NER tasks in the clinical trial domain, which seems intuitive. But for task layer, the choice between Softmax and BiLSTM+CRF does not significantly affect the performance.
RE evaluation results are shown in Table 4, in which we report micro/macro/weighted precision(P), recall(R), and f1score(F).
From the above performance chart, we find that BERTbase;uncased has the highest f1-scores, whereas BERTbase;cased has the lowest. Comparing BERTbase;cased and BioBERT indicates that BioBERT can help with performance slightly, at least for this cased scenario. On the other hand, BERTbase;uncased noticeably improves over its cased peer, BERTbase;cased, by a 4.33 percentage margin. Therefore, just like the NER task, the RE task is also case insensitive, probably because uncased situations reduce vocabulary variations in processing. We also observe that recall and precision are close to each other with precision slightly higher for the macro evaluation, but on the contrary, precision is slightly higher than recall for micro and weighted. These observations suggest that the model has higher precision score than recall score in classes with less samples, such as ‘is located’ and ‘is negated’ (in Table 1). And when doing macro evaluation, the contribution from the smaller classes becomes more visible.
Overall, the BERTbase;uncased model prevails - it outperforms the other two models on each evaluation type and measures. For example, it has f1-score of 78.79 for micro, compared to BERTbase;cased’s 74.46 and BioBERT ’s 74.60. These results indicate again that the lowercasing preprocessing helps the NLP tasks even in the clinical trial domain where many terms are represented in capital letters. Secondly, BioBERT beating BERTbase;cased with a small margin may suggest that although pre-training in the biomedical domain could bring in some benefit, it is still not specific enough for clinical trials. Since there is no uncased BioBERT pre-trained model available, it is unclear whether training on biomedical corpus with lowercasing preprocessing could synergistically improve the performance. Considering the big improvement from BERTbase;cased to BERTbase;uncased, we believe the uncased scenario of current BioBERT model is worth future investigation.
We present and inspect NER prediction results from one of
the models (BERTbase;uncased; Sof tmax) in a Brat server,
an open source tool that can help visualize annotation results
using color bars
The NER errors can be broadly categorized into boundary errors and entity type errors, as reflected by the four evaluation types. For boundary errors, one pattern is that BERT tends to mis-annotate some words inside a multi-word phrase. For example, as shown in Figure 4, “at least a 3 month” is one temporal constraint entity, but the NER model only captures “at”+“3 month” while misses the words in the middle (“least a”). This reflects a potential problem with BERT NER models: although it can assign entity classes relatively well, lack of structure enforcement on its output layer may possibly cause the inconsistent label within a full phrase.
In some cases, the NER model captures longer entities than the human annotator. For example, the model annotates “[cardiac mechanical assist device]jDevice”; whereas the gold standard annotates the same phrase as “[cardiac]jAnatomicLocation” + “[mechanical assist device]jDevice”. In some other cases, the situation reverses and the NER model chunks one entity in the gold standard into multiple ones. For example, “[non-steroidal anti-inflammatory drugs]jDrug” is chunked into a Qualifier/Modifier and a drug: “[non-steroidal]jQualifier/Modifier [anti-inflammatory drugs]jDrug”. The boundary merging and chunking issues, as illustrated by these two examples, occur frequently with the Qualifier/Modifier class as it is arguable that a complex term can be annotated by one whole entity or as a Qualifier/Modifier plus an entity.
For the entity type error, we observe a few cases, such as “urinalysis—Procedure type” is predicted as an Observation entity, and “gastrointestinal motility—Condition type” is predicted as Drug. The type errors occur less frequently than boundary errors according to our manual inspection.
For the RE task, we manually screen the predictions from the BERTbase;uncased; Sof tmax model against the gold standards. We first observe that the NER boundary errors can propagate to the RE task. Note that we only use named entity positions but not types in the RE task, and therefore only NER boundary errors can affect the RE performance. For example, “Transient neurologic deficits”, annotated as one Condition entity in the gold standard, is split into “Transient—Qualifier/Modifier” and “neurologic deficits—Condition”, thus causing the RE task to predict a ‘modified by’ relation between the two entities which actually does not exist in the gold standard. Another major category of RE classification error is that a number of actual relations misclassified as ’no relation’, while misclassification between other classes is much less frequent.
In this study, we focus on extracting clinically relevant terms and relations from protocol eligibility criteria by applying pre-trained transformer deep learning NLP models for NER and RE tasks. We experiment with several configurations of the pre-trained BERT models and report our results and findings.
Our results demonstrated the effectiveness of NLP models in processing clinical trial protocols. Despite of the fact that the processed texts are unique with specific clinical and medical terms and logical relations, BERT and BioBERT models returned acceptable performances. We also find that in general, BioBERT, which is pre-trained on biomedical corpus, outperforms BERT, which is pre-trained on general domain corpus. This agrees with the general understanding of the importance of domain-specific training for achieving higher model performance in domain-specific tasks.
A surprising finding is that even though the clinical trial domain largely contains capitalized terminologies, lowercasing preprocessing improves the performances of both NER and RE tasks. Our hypothesis is that maintaining less token variation (i.e. lowercasing has less variation) is more important than maintaining casing for these tasks.
It is also worth noting that there are rooms to improve the quality of our gold standard. Due to the complex nature of the protocols that cover many different sub-domains in biomedical and clinical sciences such as therapeutic areas, even human experts can easily make mistakes or be inconsistent. In fact, we found many cases that the model predictions are in fact correct, although different from the gold standard. To address this annotation quality issue, we employed an iterative annotating pipeline that asks human experts to verify the pre-annotated documents by the NLP models. We anticipat that this practice can help partly address this issue.
We believe that the model performance can be further improved. To do that, we can further explore in several directions. The first approach is to train a biomedical BERT model using a domain-specific vocabulary from scratch. BERT model handles tokens by splitting them into subwords using a predefined subword vocabulary. For example, ’myocarditis’ and ’pericarditis’, two heart conditions sharing the same suffix ’carditis’, are however represented as ’my’+’##oca’+’##rdi’+’##tis’ and ’per’+’##ica’+’##rdi’+’##tis’ respectively. This way of tokenization does not represent the suffix in a biomedically meaningful way due to the lack of biomedical subwords in the vocabulary. We assume subwords generated from the biomedical domain reflecting word root patterns can further enhance the word representation for BERT models and thus improve downstream task performance. We can train a BERT model from scratch using a biomedical corpus and a biomedical subword vocabulary.
The second strategy is to deploy multi-task co-training: since NER and RE tasks are dependent on each other, namely, knowing one task’s output can facilitate the other task’s, and therefore joint learning on them is expected to improve performances for both.
Our third strategy for future improvement is to reduce unnecessary relations currently predicted from the RE model. Our current greedy prediction pipeline enumerates all possible entity pairs that results in an unnecessarily large testing base set. One way to address this issue is to consider dependency parsing information, which can be used to indicate whether two terms has dependency relations to prune unnecessary entity pairs.
The extracted information from the NER and RE tasks has the great potential of assisting drug development business especially for study feasibility analysis. The derived information is the basis for a local knowledge graph for the protocols and a global graph when merging with external structured information such as drug ontologies. In conclusion, this is our first step towards a greater mission to apply deep learning to business cases in drug development, and the subsequent analysis based on the derived graph can even further enhance our contribution and insights to this research area.
Bidirectional arXiv preprint