Providing computer-aided support for the assignment of diagnosis codes has been approached in numerous ways, often by exploiting free-text fields in patient records. Modeling the 'meaning' of diagnosis codes through statistical data on co-occurrences of words and assigned codes-using a method known as Random Indexing-has only recently been explored as an interesting, alternative solution. It involves words in a clinician's notes 'voting' for semantically associated diagnosis codes, the election results yielding a single list of recommendations. This approach is here applied and evaluated on a corpus of over 250,000 coded patient records. The evaluation is performed by comparing the recommended codes generated by the model with those assigned by the clinicians. Applying the tf-idf weighting scheme somewhat improves results for general models (23% recall for exact matches) but has little effect on domainspecific models (32% and 59% recall for exact matches). These results confirm the potential of Random Indexing for diagnosis code assignment support, and merits further attention.
Diagnostic coding is part of a clinician’s everyday work routine, its purpose
being to classify diseases and other health-related issues. This makes it
possible to quantify the complex operations of healthcare and thus enables effective
oversight of hospitals; it also produces statistics on a regional, national and
international level. For such statistics to be comparable, a standard such as the
10th revision of the International Classification of Diseases and Related Health
Problems (ICD-10) needs to be employed [
The important yet somewhat tedious task of assigning appropriate diagnosis codes is typically accompanied by note-taking. This information source is often exploited in attempts to provide computer-aided coding support, given the successful application of natural language processing to other classification problems. Since a clinician’s assessment and other notes often overlap with the content of the assigned diagnosis code(s), the text may be used to infer possible codes, not automating but greatly facilitating the coding process.
Computer-aided diagnostic coding support has long been an active research
area (see [
Pakhomov et al. [
In the Computational Medicine Center’s 2007 Medical NLP Challenge [
The possible application of the word space model to this problem has only
recently been investigated. We have previously proposed the use of Random
Indexing as an interesting alternative [
The word space model is an application of the vector space model (see [
Random Indexing [
A subset of the Stockholm EPR corpus [
We then generate a number of models by constructing a word vector for each token (words and ICD-10 codes) encountered in the training data. The relative directions of these word vectors in the word space are assumed to indicate semantic similarity. The construction of the word vectors requires the context to be defined and represented in some way. The context is often defined as a paragraph or a window of surrounding words; however, in our case, it includes an entire document, as there is no sequential dependency between the diagnosis code and the words in the document. Each document, i.e. the context, is assigned a unique and randomly generated context vector, which is a sparse, high-dimensional and ternary vector. The dimensionality of the context vectors depends on the size and redundancy of the data—we have set it to 1,000—with a very small number (1-2%) of randomly distributed +1s and -1s, while the remaining elements are set 2 This research has been approved by the Regional Ethical Review Board in Stockholm (Etikpro¨vningsna¨mnden i Stockholm), permission number 2009/1742-31/5. Ele4ction ofHDeinargiknsossoins,CHoadsesse:l Words as Responsible Citizens to 0. The word vectors are then built by processing the documents: every time a word occurs in a particular context, the context vector is added to the word vector. The usage of a word is thus represented as a vector, which is the sum of the context vectors of the contexts in which it appears. We build a model for each of our data sets: three general models (general elections ), which are trained on the entire data set, and two domain-specific models (municipal elections ), which are trained on subsets from a single type of clinic: ENT (Ear-Nose-Throat) and rheumatology clinics respectively. The reason for creating domain-specific models is that, by limiting the sparsity of the data and the classification problem (# of possible ICD-10 codes), we may achieve better results. Furthermore, we build bigram versions of some of the above models. As bigrams are more informative than unigrams and represent a means of dealing with multiword expressions, they may prove to be appropriate units in the construction of the models.
These models are then used to produce a ranked list of a number of recommended ICD-10 codes (e.g. 10) for each query document (i.e. excluding codes). For each word in a document, a ranked list of semantically correlated words is retrieved from the model. As we are interested in recommending diagnosis codes, the results are filtered to include only such tokens. The individual lists of all the words in the document are combined to produce a single ranked list. This ensemble method is carried out in one of two ways: (1) by using the ranking positions of the codes in the individual lists and (2) by using the semantic (cosine) similarity scores of the codes in the individual lists.
In this initial approach, we apply the ’one word, one vote’ system, i.e. all words have an equal say in electing the diagnosis codes (democratic approach). This simple yet rather naive approach is refined by implementing the tf-idf (term frequency-inverse document frequency) weighting scheme (meritocratic approach), effectively giving a stronger voice to prominent (tf) words that have a high discriminatory value (idf). Tf is implicit in our method, as each instance of a word has its own vote, while idf is retrieved from the model.
The evaluation is conducted by comparing the codes that were assigned by the clinicians with the model-generated recommendations. This matching is done on all four possible levels of ICD-10 according to specificity (Figure 1). If a clinically assigned code is not given at the most specific level—as is often the case—and that same code is recommended by the model, it naturally counts as an exact match. To gauge the quality of the results, they are compared with a naive baseline for each model. It is created by matching the assigned codes for each document against a list of the most frequent labels in the training set.
The data sets have a distinct number of tokens and unique codes per visit (Table 1). While the set of codes is identical across the general data sets (12,396 labels), these are much smaller in the domain-specific versions (1,713 and 638 labels). The models trained and evaluated on the bigram versions of the above data sets fail to yield an improvement over the simpler unigram models. Those models are therefore not considered in any of the subsequent experiments. Out of the two ensemble methods applied in the production of the final list of recommended codes, the one whereby the cosine similarity scores are taken into account generally performs somewhat better than the more basic variant in which the ranking positions are used.
The models are initially employed in a democratic approach, i.e. all words have an equal vote (Table 2). In the general data sets, approximately 21% of the assigned codes are recommended by the corresponding models. Partial matches are more frequent, with approximately 29% matched on level 2. The results are higher when applying the domain-specific models: 31% exact matches in ENT and 59% in Rheumatology. Similar increases are observed when matching at the less specific levels, with 61% and 93% partial matches respectively.
The models are then applied in a meritocratic fashion, i.e. some words have a greater say than others (Table 3). This approach leads to a small increase in the performance of the general models (up to 3 percentage points), with little impact on the domain-specific models, even having an adverse effect in Rheumatology. 0days general 1days general 2days general 0days ENT 0days Rheumatology
0days general 1days general 2days general 0days ENT 0days Rheumatology When we increase the number of generated recommendations, the results improve to some degree (Figure 2). For recall 15, the results increase by up to 7 percentage points and, for recall 20, they increase by up to 13 percentage points. That is, when 20 recommended codes are generated, 28% of the clinically assigned codes are recommended by the general models, while 38% (ENT) and 69% (Rheumatology) are matched by the domain-specific models. The baseline is beaten by the general models but not always by the domain-specific versions. The definition of a patient visit, which potentially determines the length of the documents and the number of assigned codes, appears to have a negligible impact on the results. As can be seen in Table 1, the average number of tokens per visit only increases slightly, while the average number of assigned codes is more or less equal. The reason for this is probably that the status of inpatients is usually monitored and documented on a near-daily basis.
The domain-specific models perform significantly better than the general models. This was expected considering the extent to which the classification problem is thereby limited: the number of labels is reduced from 12,396 to 1,713 (ENT) and 638 (Rheumatology). The discrepancy between the two domainspecific models, including the average number of codes per visit in ENT (2.1) and Rheumatology (1.2), is likewise a possible reason for the latter performing better than the former.
The meritocratic approach had surprisingly little effect on the results. While it did have a positive yet small impact on the general models, it had little—and even a negative—impact on the domain-specific models. A qualitative analysis of which types of words benefit from this weighting scheme would be interesting. It is likely that a small set of keywords are highly indicative of the diagnoses and need to be given additional weight. In a technocratic approach, this could be achieved by seeking out words used in the ICD-10 descriptions or by looking for SNOMED CT terms, such as diseases and body parts.
The fact that the bigram models performed worse than the unigram models
could possibly be due to the sparsity of the data. That is, there are many more
distinct tokens and fewer instances of each. The same conclusion was reached
by Suominen et al. [
The results presented here are not directly comparable with those in
previous studies, primarily due to the tasks being of different orders of magnitude.
In comparison to the 2007 shared task [
In future work, we plan to exploit fixed fields in patient records, such as age
and gender, to avoid statistically rare correlations, as was successfully done in
[
We have quantitatively evaluated the use of Random Indexing as a means to provide diagnostic coding support on over 250,000 patient records. An array of models and two ensemble methods were evaluated. A meritocratic approach (tfidf weighting) yields little improvement over a democratic approach (one word, one vote) in the election of appropriate diagnosis codes. Domain-specific models produce significantly better results (at best 61% exact matches and 93% partial matches) than general models (22% exact matches and 61% partial matches).