Clinical NLP tools that automatically extract cancer concepts from unstructured Electronic Health Record (EHR) text can benefit cancer treatment matching, clinical trials cohort identification, and reportable cancer abstraction. We used a combination of two BERT-based [1] language models, BETO [2] and MBERT [1]; with regular expressions constructed from training data; and ICD-O dictionary based features to participate in the tumor named-entity recognition subtask of the 2020 CANTEMIST (CANcer TExt Mining Shared Task) [3]. Our goal is to explore the incorporation of dictionary-based features into these models to provide better integration between machine learning models and external knowledge resources. Results on the test data set were highest with a regular expression based system (F-Score 0.73) and development set results showed a 5 point drop in F-Score (0.76 to 0.71) when integrating dictionary features into our BETO based system. We suggest that dictionary-based features will need careful integration to improve the performance of masked language models.
The widespread adoption of Electronic Health Records (EHR) has resulted in an explosion in the
volume of clinical data captured electronically. Computerized methods such as data analytics
and clinical decision support (CDS) can be applied on clinical data to accelerate new scientific
discoveries and improve clinical care delivery. This is particularly important in the field of
oncology: cancer treatments are highly specific to cancer subtypes based upon clinical and
tumor attributes (e.g., Philadelphia chromosome-positive B-cell Acute Lymphoblastic Leukemia);
cancer clinical trials require identification of patients who meet high specific eligibility criteria
(e.g., cancer subtype, clinical features, biomarker status); and cancer reporting for public health
surveillance and quality assurance requires abstracting detailed cancer-related attributes from
the clinical record. Each of the above examples would highly benefit from automated extraction
of cancer concepts from the EHR[
The first task of structured data extraction is the identification of the specific span of text
(mention) containing the name of interest. This is referred to as Named-Entity Recognition
(NER), or clinical entity recognition in the context of clinical text. NER software that has
been developed specifically for clinical text includes cTAKES[
Relatively little attention has been given to integrating dictionary features for large
clinical vocabularies into these types of architectures for clinical NER. One recent exception[
For this paper, we explore the integration of transformer-based language models (such as BETO and MBERT) with external knowledge resources, as well as their applicability to clinical entity normalization for cancer concepts.
We developed a total of 8 diferent systems for this task, including BETO-FLAIR, REGEX,
BETOFLAIR-REGEX, MBERT-REGEX, MBERT-PYTORCH, MBERT-DICT, BETO-PYTORCH and
BETODICT. Only 3 systems, BETO-FLAIR, MBERT-PYTORCH and REGEX, were finished in time to be
oficial entries for the CANTEMIST shared task, but results are shown for all systems. All systems
with masked language models were constructed utilizing Huggingface’s implementations of
transformer-based language models[
We used three distinct methods for predicting annotations: fine-tuned masked language
models (BETO-FLAIR, BETO-PYTORCH and MBERT-PYTORCH ), regular expressions alone
or in conjunction with masked language models (REGEX, BETO-FLAIR-REGEX and
MBERTREGEX ), and dictionary features in conjunction with masked language models (MBERT-DICT
and BETO-DICT ). BETO is a BERT language model pretrained on a large Spanish corpus and
outperforms MBERT on several Spanish tasks, including natural language inference, paraphrasing,
NER, and document classification[
Cantemist input files in BRAT standof format (.ann files) were converted to CoNLL format for processing by all systems, except REGEX. Input text was tokenized using the NLTK Spanish tokenizer and converted to either a 2 or 3 column format. The first column specified the token, and the second column specified the Cantemist tag in IOB (Inside-Outside-Begin) format. An optional third column was used in BETO-FLAIR-REGEX and MBERT-REGEX to specify the logits resulting from the pytorch-based MBERT-REGEX system, which was used to adjust the cutof pT2bN0M1
HCC
CCR
Ca
frequency for the regular expression component (REGEX ). During the conversion, many of
the annotations had overlaps which afected the performance of the converter. To handle
overlapping annotations, we kept only the longest of the overlapping annotation, based on span
length.
For our first system, we used the sequence tagger from Flair version 0.4.2. We loaded the
pretrained BETO[
Both implementations use Huggingface’s Transformers[
The MBERT-PYTORCH and BETO-FLAIR systems were extended by integration with the REGEX system. Both systems were modified to output a confidence score, ranging from 0 to 1, associated with the prediction for each token. We obtain these scores by applying softmax to the logit values returned by the BERT model. For tokens that were initially classified as not "MORFOLOGIA_NEOPLASIA" (the "O" class) that were later contained in a span of text the REGEX system classified as "MORFOLOGIA_NEOPLASIA," we would adjust the confidence score by +0.15. If the adjusted confidence score of any of the tokens crossed the 0.5 boundary, we changed the classification to "MORFOLOGIA_NEOPLASIA" for all tokens matched in the regular expression. The confidence score adjustment of +0.15 was chosen empirically after manually reviewing the output of the two systems. The decision to apply the adjustment solely to the "O" class was made in hopes of improving recall without reducing precision. This resulted in only a few adjustments to the predicted results of the two systems: BETO-FLAIR-REGEX resulted in 1998 annotation changes compared with BETO-FLAIR alone across both the test and background sets, and MBERT-REGEX resulted in 59 changes compared with MBERT-PYTORCH alone.
The MBERT-DICT and BETO-DICT systems extends MBERT-PYTORCH and BETO-PYTORCH respectively by the use of dictionary-based features described in the next section. We wrote a custom head for the BertForTokenClassification model which concatenates these dictionarybased features with the logits corresponding to each subword in a sample. These extended samples run through two stacks of dropout and fully-connected layers, first mapping to the HIDDEN_SIZE, then applying the standard mapping to NUM_LABELS. Results for this model converge in 4 epochs.
Table 4 summarizes dictionary-based features used in the BETO-DICT and MBERT-DICT systems. Features were selected in order to assess both the coverage and the cohesiveness of input mentions relative to a term representation of all term names and synonyms in the Cantemistprovided ICD-O dictionary (valid-codes.txt file) for an entry.
For subword-based dictionary features, the presence of input mention text dictionary-based features were calculated for each input subword present in the mention. We utilized the entire 512 BETO subword limit as our lookup window.
Character-based dictionary features were calculated using a Python string similarity library (https://github.com/luozhouyang/python-string-similarity). Character-based comparisons were made between a character-based term representation of the dictionary term and the lookup window. Parameters for character-based dictionaries, including overlap co-eficient and shingle size, were determined empirically, resulting in values of 3 and 5 respectively. All dictionary features are described in detail below.
Highest subword coverage The highest subword coverage calculates the number of overlapping subwords in the subword window for each ICD-O term’s dictionary name and any of its respective synonyms (term representation) to compute a subword overlap count for each term. The highest subword coverage is the count of subword overlaps for the term with the maximum subword overlaps for the dictionary.
Distinct subword Each subword in the subword window is matched against all dictionary term representations resulting in a set of all the entries with at least one subword. One or more subwords will return a set with the lowest cardinality. That minimum cardinality over all returned sets from all subwords in the subword window is the distinct subword. Average matches The average number of subword window term representation over the entire dictionary.
representation is the maximum number of subwords represented in the subword window. Diferential subword coverage The diferential subword coverage is computed by taking the "Highest subword coverage" described above and subtracting the average subword coverage or overlap between the subword window and all subword term representation in the ICD-O dictionary.
Best entry log subword frequencies The subword frequency for each subword in the ICDO dictionary is computed for all subwords overlapping between the subword window and the subword term representation. The log of these frequencies is summed for each comparison and the highest value is used as the "best entry log subword frequencies". Overlap coeficient An overlap coeficient of 3 is used to calculate a Szymkiewicz-Simpson similarity score between the mention subword and 10 adjacent characters on either side ("character extended subword window") to a character-based term representation.
Shingle N-Grams Character 5-gram profiles were pre-computed for each character-based term representation and compared to a profile from the input mention’s character extended subword window using cosine similarity .
Oficial test results are shown in Table 5, and unoficial test results including all systems developed are shown in Table 6. Only 3 systems were submitted in time for completion, but we show test results for all data using the included eval script. The discrepancies between the oficial and unoficial test results are caused by changes made to the utility that converted .conll and .ann files, changes in the regex system to more loosely match escape characters, and processing all files with MBERT-PYTORCH. Our original submission included only 80% of ifles. Our best performing system for the oficial test results was the baseline REGEX system which reflects underdevelopment of the masked language model systems, although BETO-FLAIR had the highest precision. This is replicated in the unoficial results. Additionally we show results on the development set in Table 7. The BETO-PYTORCH obtained the best results in all 3 evaluation metrics.
We were disappointed by the poor performance of pure transformer-based systems on the test data relative to the simple regular expression-based system REGEX. We also tested using a pure regular expression dictionary matching approach, but performance was worse than simply looking for exact matches in the training data (data not shown). However, on the development data BETO-PYTORCH did produce the best results. Our eforts to integrate dictionary features with masked language models also yielded disappointing results. We suspect the poor performance of the dictionary-based approach is due to limited development time, the reliance on subwords (versus words), overly large lookup window (512 subwords instead of a sentence or smaller window) and the lack of dictionary feature validation and testing rather than the integration of these features into the BERT model.
Combining BERT based model with REGEX did not result in a significant improvement. Recall was slightly higher when evaluating on the test data set, but at a cost of lower precision. High recall with lower precision is naturally expected when using regular expression-based systems for NER. Picking a single confidence score adjustment and applying that across multiple trained models also likely caused lower performance, since each model’s average confidence score for a given class was significantly diferent. Future Directions In the short term, we plan to expand the number of dictionary-based features to better account for term variation and head nouns. Word level features also need to be introduced, and the utility of subwords to handle medical abbreviations and relevant Latin and Greek roots needs to be evaluated. Appropriate medical stemming, or use of a clinical subword vocabulary, also needs to be evaluated. We are also interested in cross-language evaluation (English-Spanish) of cancer extraction terms.
Our work sufered from a number of limitations, the most important being the lack of a Spanish speaker in our group, forcing us to rely on Google Translate and the similarity of Latin-based medical terms. Due to time constraints, we did not perform a principled dictionary feature evaluation to assess the relative importance of features. Parameter settings were not fully evaluated for similar reasons.
This publication was supported by internal funding from the Informatics Institute at University of Alabama at Birmingham, and a NvidiaTM grant of a Titan XP GPU used for machine learning.