The full extent of diversity in clinical documents and the efects on natural language processing (NLP) tasks in the medical domain have not been well studied. In supervised NLP tasks, it is vital to have training data that resembles test data [27]. In the medical domain, this often translates to uniform subject matter distribution [16]. We have access to a corpus of 157 million documents from 42 diferent electronic medical record (EMR) vendors, with over 40,000 distinct categories assigned to the documents. The sheer diversity of the documents is an obstacle to an accurate sub-sampling of the data for annotation. We propose that clustering clinical text documents is an efective way to aid the annotation efort and ensure coverage. We demonstrate the efect of lack of coverage in training data for a supervised generic named entity recognition(GNER) task and the impact of clustering on the task. We will also examine the characteristics of clusters generated from a diverse dataset.
This work was presented at the first Health Search and Data Mining
Workshop (HSDM 2020)[
In 2010, the American Recovery and Reinvestment Act was passed into law, requiring all public and private US healthcare providers to adopt electronic medical records (EMR) by January 1, 2014, empowering clinical information extraction and NLP research.
However, access to annotated data with a comprehensive
coverage of subject matter domains remains a major challenge in clinical
NLP. Widely available clinical document datasets are often small or
are a narrow slice of the extant types of documents found in EMR
systems. For example, the MIMIC dataset consists only of intensive
care unit documents [
Real world medical records are very diverse with respect to
subject matter content and context [
We studied the data repository of a large healthcare provider from 42 diferent electronic medical record (EMR) vendors containing in excess of 157 million clinical text documents. EMR vendors are also referred to as source systems. The naming convention for categories assigned to the documents by the source systems are not necessarily consistent. We refer to this assigned category as a document type. The repository contains over 40,000 unique document types.
If we were to sample a set of documents across all the document types, we will end up with 400,000 documents with just 10 documents from each document type. In addition, annotations and their verification are done by subject matter experts. Evidently, this exercise requires large resources financially as well as with respect to time.
Generic named entity recognition is used to generate structured
representation of a clinical text document by identifying biomedical
concepts in the text. GNER can be used to extract hidden
information in a diagnosis [
Supervised machine learning based GNER systems requires
annotated clinical text documents with wide coverage [
We examine the efects of inadequate coverage of training data for a generic named entity recognition (GNER) task and how clustering can mitigate some of these efects. Note that our objective is not to classify clinical text into a certain category, but to ensure coverage for the GNER task.
In this work, we want to answer the following research questions. (1) How does lack of coverage in training data afect a supervised
GNER system ? (2) Can we cluster documents such that sampling from every cluster improves coverage ? (3) How do we cluster documents and what are the characteristics of each cluster ?
In the following sections, we will describe the dataset in further detail, explain the clustering method that we used, experiments demonstrating the impact of lack of coverage in the GNER task and ifnally the impact of clustering on coverage. 2
Classification and clustering of electronic medical documents are
relevant in other contexts within the medical domain. They are
primarily employed to address problems emanating from the
diversity of documents based on subject matter content. BioASQ1
orgainizes a large scale biomedical semantic indexing task every
year to classify PubMed2 documents into classes from the MeSH3
hierarchy. Weng et al. [
Clustering has been used to extract medication and symptom
names by Ling et al. [
Features used in classification and clustering tasks in the medical
domain frequently includes relevant terms extracted with the help
of a concept mapper. cTakes, [
Tang et al. [
We employ clustering as a means to ensure diversity in
annotated data in downstream tasks. As a result of the large number
of document types from many source systems, it is not feasible
to sample from every document type. The Document type dataset
as we will see in Section 3.1 has 168 document types. We aim to
show that sampling from clusters is a feasible strategy to represent
diverse clinical text documents in annotated data. We specifically
choose GNER as the downstream task because biomedical concepts
are correlated with the relevant subject matter domain [
We describe here the corpus of the large healthcare provider to illustrate the scale of the problem. The data corpus is collected with ethical approval. The data processing pipeline anonymizes patient data. The processed data is stored in a HIPAA compliant environment with restricted access. We mentioned earlier that documents are assigned document types by source systems.
Naming document types depends upon 5 axes:
Type of service: e.g. consultation Kind of document: e.g. note, consent 1http://bioasq.org/ 2https://www.ncbi.nlm.nih.gov/pubmed 3https://meshb.nlm.nih.gov/search
Role: e.g. attending, consultant
However, the naming conventions are not uniformly applied even within the same system and source systems might not use all the axes when deciding to assign document types. Because of these incompatible naming conventions, our repository has a very large variety of document types.
The repository has 41,521 document types containing 718,337 documents. The data is diverse and the distribution across subject matter categories is not uniform. There are 32,468 types in IMAGECAST, for example, a source system primarily for radiology. This is because radiology documents have a diferent document type based on the relevant body part and the type of the image (X-ray, MRI).
4,458 of the document types had at least 100 documents (common document types) and the remaining types had less than 100 documents per type (sparse document types).
Among the 4,458 common document types, radiology notes were grouped based on certain conventions (such as first 2 letters of their name) into 18 types. The resulting consolidated dataset has 1296 common document types. We used this dataset to examine high level characteristics.
The most frequent tokens in the documents within a document type can give a certain idea about the document type. This is illustrated in Table 1.
We were able to identify certain patterns among document types and on closer inspection, we found that certain document types were duplicates of each other. The key properties were small euclidean distance between them in the feature vector space (described in Section 4, an high overlap of top terms within the document types’ vocabulary and similar names for the document types. Some examples are shown in Table 2. Each duplicate pair here belong to the same source system. However, we cannot rule out a scenario where there are duplicate types across source systems. Clustering the documents can be a way to ensure that these duplicate note types are always grouped together which can significantly reduce annotation eforts.
The “GNER dataset" and the “Document type dataset" are subsets of the documents from the 1296 common document types. 3.1
The Document type dataset is used to find the methodology for clustering and understand the best features. The resultant parameters are then used to cluster documents for the GNER dataset.
The dataset contains 13,440 documents spanning across 30 unique
subject matter domains. The subject categories are listed in
Appendix A. The Document type dataset’s diversity is a very notable
aspect. With 13,440 documents, it is smaller than the MIMIC [
Subject matter content for documents in this dataset is informed
by the document’s document type. We can map many of the
document types to standard representation from the subject matter
domain axis of LOINC Ontology [
foot liver artery
ankle hepatitis femoral
incision cirrhosis catheter
Orthopaedic Surgery Gastroentrology Cardiology by a data analyst and subsequently verified by a subject matter expert. The 13,440 documents belong to 168 document types across the 30 subject categories. 3.2
Our experiments for the GNER task uses a dataset of 2059 documents. We refer to this as the “GNER dataset". The GNER dataset consists of documents with each token annotated as one of G-B, G-I or O. All generic mentions of medical concepts (e.g. conditions, procedures, labs-observation, medications) are annotated.
We do not have information about the subject categories for these documents. We do however, know the source systems for the documents. The distribution of source systems in the dataset is shown in table 3. Documents from diferent source systems have notable diferences in content. COPATH notes are clinical observations and procedures while IMAGECAST notes are radiology observations and procedures. CERNER and EPIC are typically systems used at hospital level and contains a mixture of note types such as progress notes, consults and discharge summaries. PROVATION notes are gastroenterology procedures while VASCUPRO notes are vascular lab reports. MUSE notes consists of cardiology procedures, CVIS notes are related to cardiac imaging and APOLLO notes are for documenting physical therapy. 4
The motivation behind clustering is that it can create coherent groups of data. We show how closely the clusters are aligned according to their subject matter domain. For this analysis, we use the “Document types dataset". 4.1
Clusters are evaluated based on how coherent they are to a particular subject matter. This is measured using the purity metric. To compute purity, each cluster is assigned to the class which is most frequent in the cluster, and then the accuracy of this assignment is measured by counting the number of correctly assigned documents and dividing by N , the total number of documents.
1 Õ purity(W , C) = N mjax |wk ∩ cj |
k where W = (w1, w2, w3, ...wk ) is the set of clusters and C = (c1, c2, c3, ..cj ) is the set of classes.
Let us look at the specific steps involved in generating document clusters. 4.2
All documents are preprocessed for stopword removal (stopwords from the nltk4 corpus), filtering out highest frequency words (top 0.1%) in the corpus and words with only numbers in them. The documents are then tokenized with nltk word tokenizer. The number of unique tokens in the dataset after preprocessing and tokenizing is approximately 2.7 million. 4.3
Feature generation
4.3.1 N-gram word tokens. The word tokens generated in the
previous process were used to generate unigram, bigram and trigram
word tokens. Looking at the top terms for documents, they indicate
the subject matter domain (e.g. cardiology, neurology) or the kind
of document (e.g. note, letter) in most of the cases. Bigrams and
trigrams are included as features because many medical concepts
are 2 or 3 words or more. Some examples are “intravenous solution"
and “atrial sinus solitus". We only considered the top 200,000
unigrams based on document frequency. As for bigrams and trigrams,
the total size was 8.8 million and 23.5 million respectively. To limit
the size, we put additional restrictions based on corpus frequency
(greater than 30).
4.3.2 N-gram word tokens filtered on medical vocabulary. We also
restricted the word tokens to only those found in a medical
dictionary5. The medical terms are unigram tokens from two corpuses
(OpenMedSpel6 and MTH-Med-Spel-Chek7). Bigrams and trigrams
are selected only when all the constituent terms are part of the
dictionary. This was shown to group documents of the same subject
categories even closer. The clustering purity based on unigrams
with no filtering on the “Document type dataset" was 0.56 while
the one based on filtered unigrams was 0.63. It also had the
considerable advantage of reducing the dimensionality of the features.
Unigram frequency reduced from 200,000 to 17,462. Bigram and
trigram frequencies were 100,000 and 70,000 respectively (Table 4).
4.3.3 Section headers. Medical records typically have sections that
distinguish one type from another. For e.g. A cytology report
contains section headers such as “CLINICAL HISTORY" and “SOURCE
OF SPECIMEN", whereas a test report of lumbar puncture has
“PROCEDURE" and “TECHNIQUE". These section headers act as a
signature for documents from the same source and can be used to
identify format level features for the documents. We extract section
headers from the documents with NobleCoder [
Unigrams Bigrams Trigrams Section headers Concept tokens
N-gram tokens, section headers and concepts are combined and
they are weighted by their tf-idf (Term frequency - Inverse
document frequency) [
The feature matrices are then horizontally stacked8 in a variety of combinations to examine their efects (5.1) Horizontal stacking is simply concatenating the individual feature matrices row-wise. E.g. a 13,440 x 8,432 feature matrix for section headers is combined with a 13,440 x 27,453 concept feature matrix to get a 13,440 x 35,975 combined feature matrix. 4.5
In K-means clustering, (Hartigan and Wong [
K-means clustering initializes ‘k’ centroids when generating ‘k’ clusters. Each centroid will be a particular document in the dataset. In our setting, the centroid is chosen at random for each execution of the algorithm. Predetermined centroids did not change the purity values for the generated clusters. Subsequent members for each cluster is chosen when that document is closer to a particular centroid than other centroids. The centroids are then recalculated after adding each member. The algorithm is run multiple times and we check whether the results are consistent across each run. We tried a wide range of k-values from 10 to 140, but the clusters were found to be fairly stable and consistent across all values. We also tried other clustering techniques and the resultant purity values were comparable to values obtained from the k-means method.
Table 5 illustrates the content for some of these clusters. Cluster 0 is about sleep medicine and cluster 2 is about x-rays. But clusters 3 and 4 are about consults and ofice visits. 5 5.1
Each document in the Document types dataset corresponds to a sample datapoint for the clustering algorithm. The Document type dataset has 13,440 samples. Each document has a subject matter category assignment and there are 30 unique subject categories. The 8https://docs.scipy.org/doc/scipy/reference/generated /scipy.sparse.hstack.html number of clusters are chosen as 30 based on the mean silhouette coeficient of all samples. Silhouette coeficient of a sample i in a cluster C is defined as s(i) =
b(i) − a(i) , |Ci | > 1 max a(i), b(i) s(i) = 0, |Ci | = 1 where ai is the mean distance of i to other points in C and bi is the smallest mean distance of i to all points in any other cluster, of which i is not a member.
The “closeness" of samples in a cluster can be measured by a
cluster’s mean silhouette coeficient. High silhouette coeficient of
a sample in a cluster [
Thus, we use the following parameters for the k-means algorithm, number of runs:10; number of clusters: 30. The “default" information we have about the document types is their source systems. So our baseline purity based on clustering by the 24 source systems in the dataset is 0.44 (Table 6). We use all the features we mentioned in previous sections, namely, unigrams, bigrams, trigrams, section headers, and concept tokens. Each of these features independently outperform the baseline. On the basis of the experiments, we see that simply using unigrams and bigrams (117,462 features) resulted in the highest values. Using unigrams (17,462) on its own or the combination of section headers and concepts (35,975) have comparable purity. Usage of the latter set of features have an added advantage of lower dimensionality as the corresponding feature vector is much smaller. For the experiments on the GNER dataset, we only used unigram features.
APOLLO CERNERH1 COPATH CVIS EPIC IMAGECAST MUSE PROVATION VASCUPRO The GNER task is modeled as a sequence labeling problem. Given a word from the document, the task is to predict one of the labels. The labels used are G-B = beginning of an entity, G-I = inside an entity, and O = outside of an entity. For e.g., consider the input sentence, "The patient is sufering from a cardio-vascular disease." The correct predictions would be (The,O), (patient,O), (is, O), (sufering,O), (from,O), (a,O), (cardio,G-B), (vascular,G-I), (disease,G-I),(.O).
GNER is an important step in the NLP pipeline to extract
biomedical entities and concepts. The GNER model we use is a
Bi-LSTMCRF based model (Huang et al. [
We use the GNER dataset for this task. Recall that we do not have access to subject matter annotations for documents in the GNER dataset. But we find that, in this dataset, the document’s source system is a good indicator of subject matter content (Section 3.2). The lack of coverage in training data is first illustrated using document’s source system as a proxy for subject matter content. We will then show that if we use a document’s cluster in place of the source system, GNER system performance sufers in a similar manner. We also use the clustering methodology as explained in Section 4. Performance of the GNER model is measured as the F1 score on the test data.
There are 3 parts to the experiments in the GNER task. (1) Leave one out cross validation based on document’s source system. This identifies the drop in performance for documents in test data that are from unseen source systems in training. (2) Cluster the documents. Leave one out cross validation based on document’s cluster. We show that performance of GNER on documents in test data that are from unseen clusters in training similarly drops (3) Train and test the documents on a per cluster basis.
Finally, we examine the GNER performance on documents trained and tested on a per cluster basis. 5.2.1 Efect of source systems on the GNER task. Documents in diferent source systems belong to diferent sub-domains and their format is diferent as well. We investigate the efect of source systems on GNER performance with leave one out cross validation on the dataset, based on the source system of the documents. Documents originating from one source system is left out and kept aside as a test set. Remaining documents are shufled to make a train/dev split in the ratio 9:1. The trained models are tested on the corresponding dev set and the corresponding left out test set.
Table 7 shows the efect of source systems on GNER performance. Performance of all 8 trained models on the dev set is much better than their performance on the left out test set. Documents from source systems unseen in training causes a drop in performance. We will refer to this as “the unseen type problem". This indicates that when creating training data, we want to have as broad a coverage as possible. It is not feasible to annotate documents across the overwhelming amount of document types that are available. We will see how clustering the documents ensures that the document types being selected are suficiently diverse. 5.2.2 Efect of clustering on GNER task. For the next part of our experiment, we created 7 clusters from documents in the GNER dataset. Table 3 shows the distribution of the document’s source systems in these 7 clusters. Each of the seven clusters are dominated by documents from a particular system. With the exception of cluster 3 and the source system MUSE, we can almost map a source system with a particular cluster. The features and methodology for clustering is described in detail in Section 4. We perform leave one out cross validation on the 7 clusters. One cluster is selected, and kept aside as the test set. The remaining documents in rest of the documents is split into train/dev sets in the ratio 9:1. Then we train the GNER model on the resulting training set and test the model on the dev set and test sets. This is repeated for every cluster.
We see that there is a drop of in test set performance with the exception of cluster 2 (Table 9). The test performance is low for clusters that are strongly dominated by one source with no or very little documents of that source in other clusters. These are clusters 1,4, and 6. For cluster 0, the source systems of documents in this cluster, CERNER, COPATH and, IMAGECAST are also represented in other clusters. In the case of clusters 3 and 5, the performance is lower (but still higher than 4 other clusters) because it is dominated by documents from CERNER. Even though there are 278 documents from CERNER in the training set, there is a diverse range of documents from that system.
For cluster 2, which is dominated by COPATH documents, there are still 42 documents in training data. Further more, the documents in COPATH are mostly clinical pathology or observation notes with a limited vocabulary. They also tend to be very dense with many generic named entity examples to learn from. For instance, when training on a per cluster basis, cluster 2 has 3857 named entities in the training set from 200 documents. There are only 761 entities from 460 documents in the training set for cluster 0.
Going back to the 7 clusters in the leave one out cross validation, we generated train/test split for each of the clusters. The GNER model was trained and tested on a per cluster basis. Table 10 shows the results for the GNER model when it is trained and tested on documents from each cluster separately. The F1 scores in the test set are better in 3 of the 7 cases, maintained for 2 cases and drops for clusters 1 and 6.
Cluster 2 has a high performance because of the nature of the clinical pathology documents that constitute the cluster. Cluster 4 is dominated by documents from PROVATION which also shares some of the properties of cluster 2. Both clusters have a limited vocabulary of biomedical concepts with a dense distribution within the documents. Cluster 6 has a training and test F1 of 0 because there are only 14 named entities in the entire training set. This brings to a potential pitfall when training on a per cluster basis. We also need to make sure that each cluster has enough training data. Cluster 1 on the other hand, has enough training data, but it is dominated by documents from the EPIC source system, which similar to CERNER is very diverse in terms of subject matter domain. This is the second possible pitfall. We need to choose the features so that clusters are aligned on the subject matter domain as much as possible. CERNER and EPIC documents are grouped together despite the diverse nature of constituent documents because general hospital notes are closer to each other than say, radiology notes.
We looked at 5 clusters generated from the GNER dataset, divided each cluster into train and test sets. We trained the Bi-LSTM-CRF model on the training sets of each cluster separately and tested it on the corresponding test sets from that cluster. We posit that the silhouette coeficient of a cluster is correlated with the GNER performance. In Table 8, the average silhouette coeficients of samples in a cluster are compared against the GNER F1 score when trained and tested on a train/test split from the same cluster.
The high pearson correlation coeficient of 0.63 indicates that well formed clusters have a high linear correlation with a higher test GNER performance. We saw in Section 4 that the “closeness" of the samples in a cluster is influenced by their respective subject matter domain.
Despite the caveats mentioned above, when we choose documents for annotation, we can improve test performance by choosing them from as many clusters as possible. This avoids having to annotate subject matter domain for each document type. 6
The diversity of unstructured clinical text documents has been an under-studied problem in clinical NLP. This paper presented initial explorations into a large real-life data repository of 157 million documents across 42 source systems and found that the source systems reported more than 40,000 document types.
Initial explorations of the document types showed that they vary widely in content and format, with significant ramifications on supervised NLP tasks. When a supervised generic named entity detection model was tested on document types that had not been present in the training data, the performance is much lower compared to a model trained on a more diverse training set (“the unseen type problem"). This indicates a need for careful selection of data when annotating to create a training set for a new NLP task in a real world setting; poorly chosen training data will hinder the creation of generalizable models. Ideally, an annotated training set would have coverage over all subject matter content. However, due to the large number of document types available, this is prohibitively expensive. Our study showed that many of the types reported by the systems are actually quite similar, leading us to explore clustering as a method to mitigate diversity of note types.
We experimented with various features for clustering and found that to generate clusters along subject matter domains, a combination of unigram and bigram features worked well, providing high purity scores on the “Document types dataset". Since we do not have subject matter annotations for the larger data repository of 40,000 document types, we posit that clusters are a reasonable stand-in to ensure representation.
We showed that clustering captures information that helps translate training performance for the GNER task. By clustering the document types to a smaller set of clusters, it is possible to select training data for NLP tasks with good coverage without wasting annotation efort on similar types.
Future work may include utilizing semantic embedding representations of documents and training feature weights for better clustering.
A
B