KR-MED 2006 "Biomedical Ontology in Action" November 8, 2006, Baltimore, Maryland, USA Registration in practice: Comparing free-text and compositional terminological-system-based registration of ICU reasons for admission a a a b Nicolette de Keizer , Ronald Cornet , Ferishta Bakhshi-Raiez , Evert de Jonge a Dept of Medical Informatics, b Dept of intensive Care, Academic Medical Center, Universiteit van Amsterdam, The Netherlands Abstract systems can either enumerate all concepts (pre-coordination), or allow post-coordination, i.e. Reusability of patient data for clinical research or enabling to compose new concepts by qualifying quality assessment relies on structured, coded data. pre-coordinated concepts with more detail. Generally Terminological systems (TS) are meant to support this. it takes longer to select and post-coordinate concepts It is hardly known how compositional TS-based corresponding to a patient's findings, diagnoses, or registration affects the correctness and specificity of tests from long lists of standard terms drawn from information, as compared to free-text registration. In terminological system than to enter a summary in free this observational study free-text reasons for text. Worse, the standard codes and terms provided by admission (RfA) in intensive care were compared to a terminological system may constrain clinical RfAs that were composed using a compositional TS. language [6]. Although the disadvantages of capturing Both RfAs were registered in the Patient Data structured, coded data might be outweighed by more Management System by clinicians during care informative data and automatic processing of data, practice. Analysis showed that only 11% of the evidence on the effect of structured and TS-based concepts matched exactly, 79% of the concepts registration of patient data on the correctness and matched partially and 10% of the concepts did not specificity of these data compared to free-text is match. TS-based registration results in more details hardly available. Many studies compared the content for almost half of the partial matches and in less coverage (correctness and specificity) of a TS by details for the other half. This study demonstrates that retrospectively coding a set of diagnoses [7]. Studies the quality of TS-based registration is influences by in which the feasibility of automated coding has been the terminological system’s content, its interface, and investigated also usually use an experimental design the registration practice of the users. in which free text from a medical record is coded Keywords: Terminological system, information retrospectively by some natural language processing storage and retrieval, medical records, evaluation algorithm (e.g. [8,9]). Cimino et al [10] use an observational, cognitive-based approach for differentiating between successful, suboptimal, and 1. Introduction failed entry of coded data by clinicians. They used the Medical Entities Dictionary (MED) which only Most potential advantages of electronic patient included pre-coordinated concepts. To our knowledge records, such as availability of patient data for no observational field studies exist in which free-text decision support and the re-use of patient data for recording in a medical record is compared with clinical research or quality assessment [1], rely on prospectively recorded compositional TS-based structured, coded data, not free text [2]. Structured diagnoses. data entry (SDE) [3] and terminological systems (TS) The aim of this observational study is to evaluate how [4] are means to support this process of capturing clinicians in every day care practice register reasons patient data in a structured and standardized way. SDE for admission (RfA) by using compositional TS-based is a method by which clinicians record patient data systems. TS-based registration was compared to directly in a structured format based on predefined free-text registration with regard to correctness and fields for data entry. Terminological systems provide specificity of recorded RfA. terms denoting concepts and their relations from a specific domain [5] and can be used within predefined The outcome of this study depends on three factors: fields for data entry. the terminological system’s content, its interface, and the registration practice of the users. In this study, we Nowadays most terminological systems do have a aim at distinguishing the effect of content from the computer-based implementation. Terminological effect of the user interface and the user. If structured 3 TS-based registration of diagnoses results in (at least) DICE implements frame-based definitions of the same information as free-text diagnoses, TS-based diagnostic information for the unambiguous and registration is preferred, as retrieval will be much unified classification of patients in Intensive Care easier and thereby re-use of the data will be much medicine. DICE defines more than 2400 concepts more feasible. If TS-based registration results in including about 1500 reasons for admission information loss we need to investigate the reasons for and uses 45 relations. DICE is implemented as a this to search for possibilities to improve the SOAP-based Java terminology service together with terminological system and its use. clients for knowledge modeling and browsing [13]. DICE is used to add controlled compositional terms to clinical records. The implementation of DICE offered 2. Materials & Methods the physicians two ways to search for the appropriate 2.1 PDMS and Terminological system DICE diagnosis concept: (a) a short list containing the most This study took place in an adult Intensive Care Unit frequently occurring diagnoses, (b) entry of (a part of) with 24 beds in 3 units, with more than 1500 yearly its preferred or synonymous term. Once a concept is admissions. Since 2002, this ward uses a commercial selected, DICE uses post-coordination to provide Patient Data Management System (PDMS), concepts with more detailed information, as shown in Metavision. This PDMS is a point-of-care Clinical Figure 2. The user interface of the client by which Information System, which runs on a Microsoft concepts are browsed stimulates but does not enforce Windows platform, uses a SQL server database and users to specify additional qualifiers of a concept, e.g. includes computerized order entry; automatic data a Coronary Artery Bypass Graft (CABG) can be collection from bedside devices such as a mechanical further qualified by the number of bypasses; the types ventilator; some simple clinical decision support; and of bypasses and whether it was a re-operation or not. (free-text) clinical documentation of e.g. reasons for At the start of the pilot physicians got a 15-minutes admission and complications during ICU stay. As part training on the use of DICE. During the pilot, of the National Intensive Care Evaluation (NICE) registration of DICE-based reasons for admission as project [11], a national registry on quality assurance of part of the NICE minimal dataset was voluntary. This Dutch ICUs, for each patient a minimal dataset among means that after the first 24 hours of ICU admission a which the reason for admission is extracted from the physician could add a controlled term from DICE into PDMS. Since April 1st 2005 a pilot study is running in the PDMS to describe the reason for ICU admission. which the compositional terminological system DICE As the reason for admission is an essential part of the [12] is integrated with the PDMS (see Figure 1) to clinical documentation the regular registration of evaluate its usability for structured registration of free-text-based reasons for admission into the PDMS reasons for ICU admission. The main reasons for the was continued during the pilot for each patient at the development of DICE were the need for a time of admission. terminological system that supports a) registration of intensive-care-specific reasons for admission, 2.2 Data collection and analysis commonly either a severe acute medical condition or For all patients admitted between April 1st 2005 and observation after a large surgical condition b) December 1st 2005 the free-text reasons for admission semantic definitions of concepts, enabling selection of patients by aggregating diagnoses on different features, and c) assignment of multiple synonymous Dutch and English terms to these concepts. Figure 1: Activation of TS-based registration within Figure 2: User interface presenting options for the Patient Data Management System post-coordination 4 and (if available) the structured DICE-based reasons CABG” and “CABG”) or when the concepts were for admission were extracted from the PDMS. As siblings with equal anatomical and pathological free-text recording of reasons for admission is properties (e.g. “hepatitis A” and “hepatitis B”). A mandatory, for all patients admitted to the IC a concept pair is considered a mismatch in all other free-text description was available. Since DICE-based cases. For each partial match the two researchers registration of reason for admission was voluntary it independently assessed which concepts, attributes or could be possible that “difficult or complex” reasons relations were missing or were additionally for admissions were not registered with DICE. To represented in the DICE-based reason for admission. investigate this possible selection bias the free-text Comments on missing details in the DICE-based reasons for admission were compared between the registration were classified either as a) “not registered groups with and without structured DICE-based but available in DICE”, b) “value of relation is reasons for admission. missing in DICE”, e.g. although a CABG can be qualified by type of graft (LIMA, RIMA, PIMA and venous) the value “LIMA-lad” is missing or c) For each admission having both a free-text reason for “relation is missing in DICE”, e.g. “bleeding of the admission and one or more DICE-based reasons for cerebellum, right side” can not completely be admission, these reasons for admission were registered by DICE since the relation “laterality” is compared by two independent researchers, both missing. experienced in DICE and intensive care medicine. Each pair consisting of one free-text and one or more Different scores of the researchers were solved based DICE-based reason for admission was scored as either on consensus and if necessary by asking an intensivist being an exact match, partial match or mismatch. A as an independent third party. match was considered exact when the DICE-based Figure 3 presents an example of a partial match. The reason for admission was semantically equivalent to free text “AVR-bio + CABG” coming from the the free-text registration. For example the abbreviated clinical documentation part of the PDMS is displayed free-text “AVR” was considered an exact match with at the top of the screen dump. In the middle of Figure 3 the DICE concept “aortic valve replacement”. A the DICE-based reasons for admission are presented concept pair was considered as partially matching and at the bottom the scoring of agreement, in this case when one concept subsumed the other (e.g. “3-fold a partial match, is presented. A “+” indicates that the Figure 3: Scorings example of the agreement between free-text “AVR-bio + CABG” and the accompanying set of DICE-based reasons for admission. The bottom part represents the match type, the difference (“+” means DICE has additional detail,” –“ means DICE misses detail), the type of difference, the reason for missing (type of prosthesis is available in DICE) and if the two researchers directly agreed on the differences or after discussion. 5 DICE based registration includes more detail than the 300 free-text registration on type of CABG, number of Concepts with No. of reasons for admission 250 additional and lacking bypasses, dysfunction of the aortic valve and the detail Angina Pectoris diagnosis. The “-” indicates that the 200 Less specific concepts free-text registration includes details on the type of 150 valve prothesis which is not registered in the More specific concepts 100 DICE-based registration, although this qualifier is available in DICE. In this example all differences 50 between the free-text and DICE-based reasons for 0 Exact match Partial Mismatch admission were scored by both researchers which is match indicated by “direct” agreement. In this paper a TS-based diagnosis is regarded as Figure 4: Distribution of exact match, mismatch correct when it exactly or partially matches the and partial match (including whether the DICE free-text diagnosis. Specificity of (correct) diagnoses based reason for admission included more and/or is expressed by as "equal" (exact match), "more less specific detail). specific", "less specific" or "more and less specific" DICE-based registration was available. One free-text depending on differences in detail of the TS-based reason for admission could be described by more than diagnoses compared to the free-text diagnoses. one DICE-based reason for admission, e.g. “CABG + AVR” is one free-text reason for admission encoded 3. Results by two DICE concepts “CABG” and “Aortic valve replacement”. The 359 free-text reasons for admission During the study period 799 admissions to the ICU were described by 457 DICE-based reasons for took place. For all these admissions a free-text reason admission. Half of them were registered as for admission was available and for 359 (45%) of pre-coordinated concepts such as “Pneumonia”, half these admissions a DICE-based reason for admission of them were registered using post-coordination, e.g. was available. Those admissions for which a “Pneumonia; has aetiology Staphylococcus aureus”. DICE-based registration was missing do not represent other reasons for admissions than those for which a Figure 4 shows that we found 38 (11%) exact matches, 284 (79%) partial matches and 37(10%) mismatches. Table1. Example of 5 exact matches, 5 partial matches and 5 mismatches Free-text diagnoses DICE-based diagnoses Exact matches THOCR Oesophageal cardiac resection, entrance: transhiatal SAB Subarchnoid bleeding re-CABG x2 venous CABG, Re-operation: true, Type:Venous graft, Number:2 Staphylococcal sepsis Sepsis, has etiology: Staphylococcus aureus Stomach bleeding GI bleeding; localized in stomach Partial matches SAB Subarchnoid bleeding; closing: coil Respiratory insufficiency Respiratory insufficiency; due to: pneumonia CABGx3 and Ao-biovalve CABG & valve replacement Respiratoire insufficiency bij benzodiazepine Accidental intoxication with sedatives and intoxicatie hypnotics Large posterior infarction Acute pulmonary oedema ; due to acute myocardial infarction Mismatches Abdominal bleeding Renal insufficiency Hypercapnia with reduced consciousness COPD Hyponatremia with cerebral oedema Self intoxication Resp insufficiency after cardiogenic shock Myocardial infarction Respiratory insufficiency due to pneumonia Perforated gallbladder 6 According to our definition 90% ((38+284)/359) of all Table 2. Match scores for reasons for admission concepts were correct but for 79% of all concepts (all (RfA) on or not on the list of most frequently partial matches), there were some discrepancies in occurring reasons for admission. specificity. One-third of the partial matches add some RfA on short RfA not on All details as well as miss some details compared to the list short list registered free-text reason for admission. Twenty-two percent of RfA the partial matches was more specific and forty-four Mismatch 21 (7%) 17 (23%) 38 (11%) percent of the partially matches was less specific compared to the free-text reason for admission. Table Partial 233 (82%) 51 (69%) 284 (79%) 1 shows some examples of exact matches, partial match matches and mismatches. Exact 31 (11%) 6 (8%) 37 (10%) match In total 582 comments were given on the 284 partially matched reasons for admission. Two hundred sixty Total 285 (100%) 74 (100%) 359 (100%) (45%) comments were given on additional concepts, attributes or relations registered in the DICE-based located, e.g. “CABG, LIMA-LAD” can be coded in registration of reasons for admission that were not DICE as “CABG, Type: LIMA” but without “LAD”. described in the free-text reason for admission. On the As described above the DICE user interface supports other hand 325 (55%) comments were given on two ways to search for the appropriate diagnostic missing concepts, attributes or relations in the concept: using a short list or entering (a part of) a term. DICE-based registration of reasons for admission Table 2 shows the scores for reasons for admission compared to the free-text reasons for admission. split up for those that could be selected from the short Figure 5 shows the distribution of the 325 reasons why list of frequently occurring reasons for admissions and the DICE-based reasons for admission were missing those that were not on this list. Twenty percent (n=74) detail. The majority (65%) of the details presented in of all reasons for admission was not on the short list of free text but missing in the DICE-based registration frequently occurring reasons for admission. was available in the DICE terminological system, but Reasons for admission that could be selected from the was not used by the clinicians. short list were scored differently from those reasons The largest group of reasons for admission consisted for admission that were not represented on this list of patients who were admitted to the ICU after cardiac (Chi-Square p<0.001). Significantly more mismatches surgery such as CABG and heart valve operations were scored among the reasons for admission that (n=112). In this patient group we found 95% correct were not on the short list. concepts: 6(5%) exact matches, 100(90%) partial In 82% of the cases the two researchers directly agreed matches and 6(5%) mismatches. Among the partial on the assigned scores, disagreement on the other 18% matches the DICE-based registration of cardiosurgical was easily resolved after short discussion. reasons for admission contains more detail in 48% of the cases compared to the free-text registered ones. 4. Discussion The main reason for missing detail in the remaining 52% cases is caused by the lack of a relation to Terminological systems offer the possibility to describe the area of the heart to which the new graft is structure and standardize medical data, which improves the re-usability of these data for clinical research and quality assessment. In this study we compared the correctness and specificity between prospectively collected TS-based reasons for 13% admission and free-text-based reasons for admission. We focused on the recorded data as such without Relation is missing in DICE taking into account the clinical consequences of the 22% Value is missing in DICE correctness and specificity of these data. We analyzed 359 reasons for admission to a Dutch Intensive Care Available in DICE registered in the PDMS by clinicians during actual 65% care practice by using free text as well as by using the DICE terminological system. According to our Figure 5: Reasons for missing detail in DICE- definition 90% of the concepts were correctly based registration of reasons for admission. registered based on the terminological system DICE. Only 11% of the cases had a perfect match. However, a partial match could be measured in 79% and there 7 were only 10% mismatches. One should be aware that patient and hence both knew the patient’s condition if we change our definition of correctness to only very well. Finally, there are no clear registration rules “concepts with a perfect match” a completely different regarding what constitutes a reason of admission of a conclusion appears. patient. As mismatches seemed to be mainly caused by above mentioned limitations of the registration Among the partial matches about half of the TS-based process rather than the terminological system, they reasons for admissions had additional detail compared have not been further investigated. to the free-text reason for admission. A possible explanation of this result could be the functionality of According to other studies in which the quality of the terminology service in which users are encouraged structured and standardized registration of medical to further specify a medical concept by additional data was audited our study has a strong surplus value qualifiers. Sixty-five percent of the information that is because this data comes from a real-practice situation lacking in the other half of the partial matches was and is not collected retrospectively in an experimental available in DICE but was not specified by the users. setting. Physicians in our observational study who Further training and an improved user interface can recorded the reasons for admission treat the patients contribute to improving these recorded reasons for and were not informed that DICE-based reasons for admissions. Medical concepts on the short list of admission would be compared to free-text reasons for frequently occurring reasons for admission, counting admission. In studies such as [14-16] patient cases for 80% of all reasons for admission, do have a better were selected, and structured, coded data were score than those not on this list. This is not a surprising obtained by independent physicians or coders without result as the frequently occurring reasons for a direct clinical relation with the patient. admission have got more attention during the modeling process of the terminological system than The aim of our study corresponds most with [10] as those not on the list. The reasons for missing concepts, both studies observe coding behavior of clinicians in attributes or relations gave us good insight into actual practice. Although different methods are used possibilities for (simple) improvements in DICE. For (cognitive approach vs. document analysis) both example the concept CABG could be extended with an studies compare TS-based registration with some kind attribute to describe which area of the heart is of free text. We used written text while Cimino et al supported by the new graft. However, although we used video-taped spoken text. Cimino et al found a used free-text reasons for admission as they were larger amount of exact matches than we did. recorded in daily care practice as a kind of golden Differences in definitions of match types partly standard, we observed many cases in which the explain this. Furthermore, the differences in results TS-based registration included more detail than the might be partly explained by the fact that in [10] free-text reasons for admission. Further research is TS-based registration took place at the same time as necessary to determine the relevance of the details free-text registration and because of other methods present in free-text as well as in the TS-based used. Furthermore, in [10] not only diagnoses but also registration. drug information is included. The main difference between the two studies, however, is that our study used a compositional TS instead of MED which only contains pre-coordinated concepts. The availability of One weakness of our study is that the moment on post-coordination might have a large influence on the which the free-text reason for admission is registered specificity of recorded diagnoses. Our study confirms is not exactly the same as the moment on which the the findings of Cimino et al. that correctness and DICE based reason for admission has been registered. specificity of TS-based registration depends on three Although both reasons for admission were registered factors: the terminological system’s content, its in the first 24 hours of admission, changing insight interface and the registration practice of the users. into the patient’s condition could be an explanation for the discrepancy (partial match or mismatch) between the free-text reasons for admission and the 5. Conclusions DICE-based reason for admission. We will investigate This study shows that comparing free-text registration this in further research. Another weakness is the fact of reasons for admission with TS-based registration of that TS-based registration and free-text registration reasons for admission only 11% of the concepts have not necessarily been done by the same physician. exactly matched and 79% of the concepts partially However, when two different physicians recorded the matched. TS-based registration added details in reason for admission of a particular patient both almost half of these partially matches and missed physicians were directly involved in treating the details in the other half. The methods used in this 8 study provide insight into possibilities for further design of an intensive care diagnostic classification. improvement of the content coverage of DICE. Methods Inf Med 1999; 38: 102-112. However, 65% of the information not captured by the 13. Cornet R, Prins AK. An architecture for standardized TS-based reasons for admission was available in termiology services by wrapping and integration of existing applications. In proceedings AMIA Annual Symposium, DICE, indicating that user interaction with the system Washington DC (2003):180-4. is more of an impediment than the contents of the TS. 14. Los RK, Roukema J, van Ginneken AM et al. Are This study shows that availability of concepts and structured data structured identically. Investigating the qualifiers in a TS does not guarantee that physicians uniformity of pediatric patient data recorded using will use them all. We expect that this result is OpenSDE. Method Inf Med 2005;44:631-638. generalizable to other terminological systems using 15. Brown PJ, Warmington V, Laurence M, Prevost AT. post-coordination such as SNOMED CT. Further Randomised crossover trial comparing the performance of research is needed to investigate how physicians will Clinical Terms Version 3 and Read Codes 5 byte set coding be optimally supported in compositional TS-based schemes in general practice. BMJ. 2003 May registration. 24;326(7399):1127. 16.Campbell JR, Carpenter P, Sneiderman C, Cohn S, Chute CG, Warren J. Phase II Evaluation of Clinical Coding References Schemes: Completeness, Taxonomy, Mapping, Definitions 1.van Ginneken AM. The computerized patient record: and Clarity. Journal of the American Medical Informatics Balancing efforts and benefit. Int J Med Inf Association 1997;4:238-251 2002:65(2):97-119. 2. Wyatt JC.Clinical data systems, part II: components and Acknowledgement techniques. Lancet 1994; 344: 1609-14. We would like to thank Antoon Prins for implementing the 3. Moorman PW, van Ginneken AM, van der Lei J, van DICE application. Bemmel JH. A model for structured data entry based on explicit descriptional knowledge. Methods Inf Med 1994; 33(5):454-63. Adress for correspondence 4. Rossi Mori A, Consorti F, Galeazzi E. Standards to N.F. de Keizer, Academic Medical Center, dept Medical support development of terminological systems for Informatics, POBox 22700, 1100DE Amsterdam, The healthcare telematics. Methods Inf Med 1998; 37(4-5): Netherlands. Email: n.f.keizer@amc.uva.nl 551-63. 5. de Keizer NF, Abu-Hanna A, Zwetsloot-Schonk JH. Understanding terminological systems I: Terminology and typology. Methods Inf Med 2000;39(1):16-21. 6. Prowsner SM, Wyatt JC, Wright P. Opportunities for and challenges of computerization. Lancet 1998; 352(9140): 1617-22. 7. Arts DG, Cornet R, de Jonge E, de Keizer NF. Methods for evaluations of medical terminological systems. A literature review and a case study. Methods Inf Med 2005;44(5):616-25.. 8. Elkin PL, Brown SH, Bauer BA, Husser CS, Carruth W, Bergstrom LR, Wahner-Roedler DL. A controlled trial of automated classification of negation from clinical notes. BMC Med Inform Decis Mak. 2005;5;5(1):13 9. Lussier YA, Shagina L, Freidman C. Automating SNOMED coding using medical language understanding: a feasibility study. In proceedings AMIA Annual Symposium, Washington DC (2001):418-22. 10. Cimino JJ, Patel VL, Kushniruk AW. Studying the human-computer-terminology interface. JAMIA 2001;8(2):163-73. 11. de Keizer N.F, de Jonge E. National IC Evaluation (NICE) : a Dutch quality control system. J ICU management 2005; 3:62-64. 12. de Keizer NF, Abu-Hanna A, Cornet R, Zwetsloot-Schonk JHM, Stoutenbeek CP. Analysis and 9