A variety of hierarchical domain taxonomies exist in the medical domain for describing medical concepts such as laboratory tests, medications, and procedures. The structural information contained within domain taxonomies contains rich semantic information pertaining to the described concepts and their relationships to each other. As AI models are successfully applied in many medical areas, it is only natural to explore integrating AI models with medical domain taxonomies. However, only a few, nascent attempts have been made. In this work, we investigate how the structure of hierarchical medical taxonomies can be used to improve the performance of a diagnosis prediction task. Specifically, we suggest a method titled TreeEmb to pre-initialize the node embeddings of a patient graph derived from electronic health records using information from the taxonomy. We expect this method to improve the performance of graph convolution network models over the enriched patient graph. We evaluate our method over a patient graph created from the MIMIC-IV electronic health record dataset enriched by initializing node embeddings using hierarchical medical taxonomies. We use type-specific domain knowledge from hierarchical medical taxonomies such as the ICD-9 procedures, ATC medication, and LOINC laboratory test taxonomies. Experimental results from a multi-label diagnosis prediction task over this graph demonstrate the eficacy of our approach.
LOINC Patient
Procedure
ICD 9 Procedure Demographics
Medication
ATC
Much work has recently been put into the
modelcentric development of novel GCN architectures, such
as RelationalGCN [
As the structure of hierarchical medical domain taxonomies contains human-curated knowledge pertaining
domain knowledge structured as hierarchical taxonomies.
Integrating semantically rich domain knowledge such as
hierarchical taxonomies into Artificial Intelligence (AI)
technologies could improve their predictive capabilities
in numerous medical applications such as patient
diagnosis prediction and protein function prediction using
end-to-end supervised learning [
Patients’ Electronic Health Records (EHR) can be
readily modeled as multi-relational graphs connect patients
with their associated medical histories, such as
prescriptions, laboratory tests, and procedures, as illustrated
in Figure 1. We, henceforth, name such graphs EHR
graphs. The AI technology of Graph Convolution
Networks (GCNs) has recently become the de facto standard
for solving many medical problems over EHR graphs due
to their seamless ability to learn latent node embeddings
for subsequent down-stream tasks, such as node
classification, link prediction, and whole graph classification in
an end-to-end manner [
This paper is structured as follows; in Section 2, we hierarchical auxiliary domain knowledge with a patient present related work using domain hierarchies in the graph consisting of multiple patients and their related initialization of node embeddings and the task of patient medical concepts, not limited to symptoms. The work diagnosis prediction using graph convolution networks. closest to ours is that of [13], in which a knowledge graph Section 3 presents the proposed method and theoreti- is built using auxiliary domain knowledge from the MEDcal concepts. In Section 4, we present the data used for LINE medical corpus for multi-label prediction of patient experimentation, followed by Section 5, where the ex- diseases. Patients are associated with diagnosis codes perimental setup and results are analyzed and explained. related to laboratory tests, habits, and profiles in their Lastly, in Section 6, we conclude and introduce future work. However, diferent from our work, their method of work. diagnosis prediction is not related to graph convolutions, and patients are not associated with each other.
3.2
3.2
Embedding Initialization. Research into integrating
domain information, such as textual descriptions, im- In this section, we formalize our method TreeEmb of using
ages, type-hierarchies, and uncertainty information into hierarchical medical taxonomies to pre-initialize node
graph convolution models has lately shown promise [
Patient Diagnosis Prediction. Diagnosis prediction EHR Data EHR Graph is the vital medical application of finding patient co- EHR Graph 3.2 3.3 3.1 morbidities using the patient’s medical history [10]. Hier- Construction archical domain knowledge has recently been introduced 3.1 3.1 3.1 into various AI models for diagnosis prediction. In [11], Figure 2: Illustration of the overall approach. Blue boxes hierarchical medical taxonomies are used to embed med- reference sections with further details on the specific step. ical concepts to leverage the general problem of data Arrows represent the directional flow of data. Orange boxes insuficiency and model interpretability by learning hi- represent our primary contribution of pre-initializing graph erarchical medical concept embeddings, pre-initialized node embeddings using concept features extracted from hieron co-occurrence information by a weighted sum of con- archical medical domain taxonomies. Orange arrows describe cept paths. Instead, in this work, we propose using the the parts of the approach that are learned using backpropagaconcept taxonomies for pre-initializing node embeddings tion. of a medical patient graph for subsequent GCN-based diagnosis prediction. The approach by Sun et al. [12]
centric graph can be constructed from an EHR dataset and the challenge of multi-label patient diagnosis prediction.
EHR data relate patients to medical concepts such as medications, laboratory tests, and procedures. Given a set of patients and a set of medical concepts , where ⊂ is the subset of distinct medical concepts types, then an EHR dataset can formally be defined as the set of tuples (, ) relating a patient ∈ with an associated medical concept ∈ .
Given the example EHR dataset and a set of patients as illustrated in Figure 3a), we create an EHR graph as follows.
The set of graph nodes is created as the union between the set of unique patients and the set of unique medical concepts from as illustrated in Figure 3b), and the graph edges are created as the set of relations and reversed relations between concepts and patients from . Furthermore, every edge in is given an edge type as specified by the medical concept type involved in the relation. As an example, the edge (1, 1) could have an edge type of as illustrated in Figure 3c), as the patient 1 has been prescribed the medication 1. The final patient graph created from and is illustrated in Figure 3c). For brevity, reverse relations are not depicted in the graph. Over this graph, we define the mapping function : → for getting the type of a node, the function : → for getting the type of an edge, and the function : → for getting the embedding of a node of type = ().
Given an EHR dataset and a set of diagnosis concepts , the challenge of patient diagnosis prediction is to find the subset ′ ⊂ pertaining to a patient ∈ s.t. ′ matches the actual set of diagnosis concepts related to the patient. We model this challenge as a multi-label classification problem.
Node features can be either pre-initialized using entityspecific information or random-initialized and learned as part of the model training process. Pre-initialization of node embeddings can be done by extracting type-specific entity information from the nodes or by extracting features from the graph structure. Examples of the former are pre-trained convolution neural networks for imaging information and natural language processing models for text data. An example of the use of graph structure is by counting sub-structures such as graphlets [14]. However, an overlooked source of rich semantic information can be found in type-specific domain hierarchies prevalent in many domains. Domain hierarchies are curated hierarchies of related concepts. Inherently, their structure contains knowledge regarding the relationship between concepts, and each hierarchical layer contains information about the properties of its concepts. Hence, we argue that the position of a concept within hierarchies contains rich semantic information.
In the medical domain, structured medical concepts such as medications, diagnoses, laboratory tests, and procedures are coded in hierarchical taxonomies. Medication can be coded using the world health organization’s anatomical therapeutic classification system (ATC) [ 15] and classifies medication based on its active ingredients and organ or system. Hence, the location of medications within the hierarchy contains semantic information relevant to the task of diagnosis prediction. As an example, for the medication with code 1002, e.g., metformin, the first level of the ATC hierarchy specifies that the medication targets the alimentary tract and metabolism system. Level two specifies the therapeutic subgroup, e.g., the drug is used in diabetes. Level three defines the pharmacological subgroup, e.g., the drug lowers blood glucose. The fourth level indicates the chemical subgroup of the drug, in this case, biguanides, and the last level specifies the chemical substance, e.g., metformin. Given that a patient has received metformin, the patient likely sufered from type 2 diabetes. Explicitly integrating such hierarchical information into concept embeddings should enable the AI model to learn from the proximity of similar concepts.
Surgical procedures performed on patients can be coded using the ICD-9 Procedures (PROC) taxonomy [16] of tree leaf concepts can then be used to pre-initialize node embeddings. Furthermore, using this embedding technique ensures that concepts closely related in the tree will have similar embeddings compared to concepts far away. Hence, we conjecture that GCNs will be more easily able to learn that groups of closely related concepts are used in treating the same disease, thus decreasing the epistemic uncertainty by adding domain knowledge.
1 4 9 0 2 10 3 5 6 7 8 3 3 6 10 1 4 9
0 5 6 7
8 2 10 encompasses laboratory results, vital signs, diagnoses as- Table 2 certained, administered medications, and demographics. Summarizing disease codes omitted from further analysis. The data is structured as a relational database. Codes Count Description
To disambiguate medical concepts, we transform the 290 − 319 375 Mental Disorders dataset into the observational medical outcomes part- 630 − 679 530 Comp. of Pregnancy nership (OMOP) common data model (CDM) [20] using 780 − 799 330 Injuries and Poison an extract-transform-load (ETL) conversion flow. 1 The 800 − 999 1, 617 Ill-Defined Conditions CDM format disambiguates and standardizes medical and 1, 467 Ext. Causes of Injury concepts and thus provides a means of interoperability for subsequent AI models to operate on disparate medical datasets converted into the CDM. In the CDM format 5. Experiments and Results laboratory tests are coded using the LOINC taxonomy, procedures are coded using the ICD-9 procedures taxonomy, and laboratory tests are coded using the RxNorm taxonomy [21]. Since RxNorm is a flat taxonomy, we map each medication concept through its active ingredients to the hierarchical ATC medication taxonomy.
the EHR graph based on patient diagnostic EHR concept types used in related work in EHR-based diagnosis prediction [22, 23, 10] and end up with demographic information, prescriptions, procedures, laboratory tests, and the task labels as patient diagnosis codes.
Patient diagnosis codes are coded using the 9th version of the International Classification of Diseases (ICD-9) and consist of approximately 13, 000 diagnosis codes [24]. We omit codes related to the ICD-9 E and V hierarchies as these are related to external causes of injury and are generally not discernible by EHR data. We further omit hierarchies of codes as summarized in Table 2. Omitting these hierarchies, we are left with 8, 681 disease codes. Since it is usually not possible to generalize from a low number of cases, we omit codes for which less than 500 patient cases exist. We are ultimately left with 128, 605 patients diagnosed with a total of 1, 054, 670 diagnoses from 537 distinct diagnosis codes. The full list of 537 diagnosis codes are available online2. Table 1 summarizes the number of distinct concepts for each medical EHR concept type.
To investigate the efect of pre-initializing node
embeddings using domain hierarchies, we conduct several
empirical experiments as summarized in Table 3 using the
model pipeline as illustrated in Figure 2. Each
experiment is trained on the problem of multi-label patient
diagnosis prediction using a multi-relational version of
the GraphSAGE algorithm as described in Section 3 with
the input EHR dataset described in Section 4. In the Rand
experimental setting, initial graph node embeddings are
random-initialized using Xavier initialization [25] and
made trainable as part of the supervised model training
phase [26]. Hence, Rand serves as a transductive
baseline experiment. Transductive methods generally
perform better on subsequent downstream prediction tasks,
however, with the cost of not being able to extrapolate
to unseen examples [
initialized using state-of-the-art graphlet and edge count features [14] as in [8]. Graphlet serves as an inductive baseline experiment, as trained models can extrapolate to unseen examples.
The FeatInit experimental setting investigates the effect of pre-initializing node concept embeddings using the latent information contained within hierarchical medical taxonomies using the TreeEmb method as described in Section 3. In FeatInit, node embeddings should already contain domain information relevant to the task of diagnosis prediction; hence embeddings are kept constant during training. Furthermore, in the FeatInit experimental setting, patient features are pre-initialized using categorical values for sex, race, and ethnicity and a continuous variable for the patient’s age. Moreover, as FeatInit does not train node embeddings, trained models can extrapolate to unseen examples.
For each experiment, we perform 100 iterations of treebased Parzen estimation (TPE) [27] for hyperparameter optimization over the set of parameters as summarized a non-linear transformation into the dimensionality of in Table 4. Each iteration is trained using the Adam [28] the number of diagnosis codes in a specific level of ICD-9 variation of stochastic gradient descent with binary cross- aggregation, such that we end up with one output node entropy as the loss function. Each experimental setting for each predictable diagnosis code. We split patients is investigated on the prediction of five sets of diagnosis into training validation and test sets with sizes 80/10/10 codes as in [29, 30], with each set relating to a level of and used early stopping based on validation loss. aggregation on the hierarchical ICD-9 diagnosis taxon- To evaluate and compare across experimental settings, omy. In the first setting, named L5, the task is to predict we use the standard harmonic mean F1 value between the raw comorbidities of patients from the entirety of the micro-averaged precision and recall as it is comthe 537 diagnosis codes as described in Section 4. The monly used in the evaluation of multi-label classification remaining settings investigate diagnosis code prediction tasks [13]. Furthermore, to investigate the robustness of on aggregated levels of the ICD-9 diagnosis taxonomy pre-initializing features using TreeEmb embeddings, we named L4 through L1 with 427 disparate diagnosis codes evaluate the median over all 100 model iterations for each for L4 to 13 disparate diagnosis codes for L1. Aggregat- experiment. All experimental code and data are available ing diagnosis codes enables us to investigate the efect online3. of pre-initializing graph concept embeddings from hierarchical medical taxonomies extracted through TreeEmb 5.2. Results and Analysis on classification problems of varying complexities.
As graph convolutions require the same dimensional- Figure 6 presents the results for each experimental setting ity for each node type, we do an initial transformation on over all iterations of the TPE. Experimental results in node input features using type-specific non-linear trans- terms of the F1 value for the median and best-performing formations into the feature dimensionality required by models are summarized in Table 5. the graph convolution layers. Thus, the transformation As illustrated in Figure 6, using TreeEmb embeddings is learned end-to-end with the task of diagnosis predic- for pre-initializing node features resulted in improved F1 tion. Additionally, we transform the output node embed- scores compared to learning node embeddings as part of dings as computed by the final convolution layer using the training and pre-initialization using graphlet features.
Furthermore, using unpaired t-test between Rand and L1 L2 L3 L4 L5 FeatInit and between Graphlet and FeatInit results for any level of diagnosis code aggregation results in the two70 70 70 70 70 tailed P value < .001, which by conventional criteria roe 60 60 60 60 60 itnwdoicgartoesupass.tatistically significant diference between the cFS1 50 50 50 50 50 As summarized in Table 5, for each setting, the best 40 40 40 40 40 performing FeatInit model outperforms the best per30 30 30 30 30 forming Rand and Graphlet models by 1.42 − 6.14 and GraphletRandFeatInit GraphletRandFeatInit GraphletRandFeatInit GraphletRandFeatInit GraphletRandFeatInit s6opf.8en0cot− divee1fl2ye..a3tT0uhrpeeesserucreseinnstugalgttseheipnohdiiinecrtasatreicnthhtieacratmlthksenooifnwFitl1eiasdlcgiozearectioornen-tained within domain taxonomies could provide valuable
knowledge for solving domain-specific problems such as the medical problem of patient diagnosis prediction.
The embeddings produced by TreeEmb should reflect the structure of the hierarchical taxonomy. Assuming that semantically similar concepts are close in the tree and disparate concepts far from each other, the distance between constructed embeddings should increase as the path length between nodes in the tree increases. To investigate this aspect of the TreeEmb embeddings, we compared the Euclidean distance between pairs of concept embeddings with the length of the shortest path on the tree between the pairs. As illustrated in Figure 7, the Euclidean distance between node embeddings is a monotonic increasing function given the length of the shortest path between nodes. This means that similar concepts will have similar embeddings while dissimilar concepts will have disparate embeddings.
onomies contain valuable knowledge that can be utilized by the pre-initialization of graph node embeddings. We then presented a method termed TreeEmb to do so. We evaluated the proposed method on the medical problem of multi-label diagnosis prediction by constructing TreeEmb embeddings for the pre-initialization of concept nodes in an EHR graph for the three medical hierarchical taxonomies ATC, LOINC, and ICD-9 Procedures. Experimental results from the prediction task on five diferent sets of diagnosis codes of varying dificulty demonstrate the superiority of TreeEmb embeddings over a transductive baseline of learned concept embeddings and an inductive baseline of pre-computed graphlet features. All experimental code and data are available online3.
For future work, we aim to investigate the proposed method in domains beyond the medical. Furthermore, since not all levels of hierarchical domain taxonomies may be equally important for the given prediction task, we aim to investigate trainable attention mechanisms for constructing concept embeddings from only the most relevant hierarchical knowledge. We also aim to explore other graph convolution models, including attention techniques.
[7] Z. Zhao, H. Zhou, L. Qi, L. Chang, M. Zhou, Induc- dorf, P. C. Ivanov, R. G. Mark, et al., Physiobank, tive representation learning via cnn for partially- physiotoolkit, and physionet: components of a new unseen attributed networks, IEEE Transactions on research resource for complex physiologic signals, Network Science and Engineering 8 (2021) 695–706. Circulation 101 (2000) e215–e220. doi:10.1109/TNSE.2020.3048902. [20] G. Hripcsak, J. D. Duke, N. H. Shah, C. G. Reich, [8] R. A. Rossi, R. Zhou, N. K. Ahmed, Deep inductive V. Huser, M. J. Schuemie, et al., Observational network representation learning, in: Companion Health Data Sciences and Informatics (OHDSI): OpProceedings of the The Web Conference 2018, 2018, portunities for Observational Researchers., Studies pp. 953–960. doi:10.1145/3184558.3191524. in health technology and informatics 216 (2015) [9] S. A. Alhosseini, R. B. Tareaf, P. Najafi, C. Meinel, 574–8.
Detect me if you can: Spam bot detection using [21] S. J. Nelson, K. Zeng, J. Kilbourne, T. Powell, inductive representation learning, in: WWW R. Moore, Normalized names for clinical drugs: (Companion Volume), ACM, 2019, pp. 148–153. Rxnorm at 6 years, Journal of the American Medidoi:10.1145/3308560.3316504. cal Informatics Association 18 (2011) 441–448. [10] Y. Si, J. Du, Z. Li, X. Jiang, T. Miller, F. Wang, W. J. [22] A. Hosseini, T. Chen, W. Wu, Y. Sun, M. Sarrafzadeh, Zheng, K. Roberts, Deep representation learning Heteromed: Heterogeneous information network of patient data from electronic health records (ehr): for medical diagnosis, in: Proceedings of the 27th A systematic review, Journal of Biomedical Infor- ACM International Conference on Information and matics 115 (2021) 103671. doi:{10.1016/j.jbi. Knowledge Management, 2018, pp. 763–772. 2020.103671}. [23] Z. Liu, X. Li, H. Peng, L. He, S. Y. Philip, Hetero[11] E. Choi, M. T. Bahadori, L. Song, W. F. Stewart, geneous similarity graph neural network on elecJ. Sun, Gram: graph-based attention model for tronic health records, in: 2020 IEEE International healthcare representation learning, in: Proceedings Conference on Big Data (Big Data), IEEE, 2020, pp. of the 23rd ACM SIGKDD international conference 1196–1205. doi:10.1109/BigData50022.2020. on knowledge discovery and data mining, 2017, pp. 9377795.
787–795. doi:10.1145/3097983.3098126. [24] D. J. Cartwright, Icd-9-cm to icd-10-cm codes: [12] Z. Sun, H. Yin, H. Chen, T. Chen, L. Cui, F. Yang, what? why? how?, in: Advances in wound care, Disease prediction via graph neural networks, Mary Ann Liebert, Inc. 140 Huguenot Street, 3rd IEEE Journal of Biomedical and Health Informat- Floor New Rochelle, NY 10801 USA, 2013, pp. 588– ics 25 (2020) 818–826. doi:10.1109/JBHI.2020. 592. doi:10.1089/wound.2013.0478. 3004143. [25] X. Glorot, Y. Bengio, Understanding the dificulty [13] T. Pham, X. Tao, J. Zhang, J. Yong, Y. Li, H. Xie, of training deep feedforward neural networks, in: Graph-based multi-label disease prediction model Proceedings of the thirteenth international conferlearning from medical data and domain knowledge, ence on artificial intelligence and statistics, JMLR Knowledge-Based Systems (2021) 107662. doi:10. Workshop and Conference Proceedings, 2010, pp. 1016/j.knosys.2021.107662. 249–256. [14] N. K. Ahmed, J. Neville, R. A. Rossi, N. Dufield, Efi- [26] W. L. Hamilton, R. Ying, J. Leskovec, Representacient graphlet counting for large networks, in: 2015 tion learning on graphs: Methods and applications, IEEE International Conference on Data Mining, arXiv preprint arXiv:1709.05584 (2017). IEEE, 2015, pp. 1–10. doi:10.1109/ICDM.2015. [27] J. Bergstra, R. Bardenet, Y. Bengio, B. Kégl, Algo141. rithms for hyper-parameter optimization, in: NIPS, [15] M. Ronning, A historical overview of the atc/ddd 2011, pp. 2546–2554.
methodology, WHO drug information 16 (2002) [28] D. P. Kingma, J. Ba, Adam: A method for stochastic 233. optimization, in: ICLR (Poster), 2015. [16] WHO, et al., International classification of dis- [29] T. Sagi, E. R. Hansen, K. Hose, G. Y. Lip, T. B. eases:[9th] ninth revision, basic tabulation list with Larsen, F. Skjøth, Towards assigning diagnoalphabetic index, World Health Organization, 1978. sis codes using medication history, in: Inter[17] C. J. McDonald, S. M. Huf, J. G. Suico, G. Hill, national Conference on Artificial Intelligence in D. Leavelle, R. Aller, et al., Loinc, a universal stan- Medicine, Springer, 2020, pp. 203–213. doi:10. dard for identifying laboratory observations: a 5- 1007/978-3-030-59137-3_19. year update, Clinical chemistry 49 (2003) 624–633. [30] E. R. Hansen, T. Sagi, K. Hose, G. Y. Lip, T. B. [18] A. Johnson, L. Bulgarelli, T. Pollard, S. Horng, Larsen, F. Skjøth, Assigning diagnosis codes usL. Celi, R. Mark, Mimic-iv (version 0.4). physionet, ing medication history, Artificial Intelligence in 2020. Medicine (2022) 102307. doi:10.1016/j.artmed. [19] A. L. Goldberger, L. A. Amaral, L. Glass, J. M. Haus- 2022.102307.