Public health surveillance systems rely on the automated monitoring of large amounts of text. While building a text mining system for veterinary syndromic surveillance, we exploit automatic and semi-automatic methods for terminology construction at different stages. Our approaches include term extraction from free-text, grouping of term variants based on string similarity, and linking to an existing medical ontology.
In the project Veterinary Pathology Text Mining, we are developing tools to exploit veterinary postmortem data for epidemiological surveillance and early detection of animal diseases. This paper describes the work in progress on the construction of a veterinary terminology resource as a basis for a text mining tool to classify, with minimal human intervention, free-text veterinary reports with respect to multiple clinical syndromes that can be monitored.
In human medicine, text mining has been
successfully applied to clinical records in many
public health surveillance systems
The text mining of veterinary reports faces
additional challenges such as multiple species and
a less controlled vocabulary
We propose to develop a process for using text mining methodologies (natural language processing) to efficiently extract relevant health information from veterinary diagnostic submission data with minimal human intervention. Given a sufficient amount of data (i. e. at least a few hundreds of manually classified reports), a machine learning approach will allow us to directly classify these data into syndromes that can be monitored for surveillance.
As recognized in the Swiss Animal Health Strategy 2010+, methods for early disease detection, based on the increasing abundance of data on animal health stored in national databases, can contribute to valuable and highly efficient surveillance activities. Post-mortem data, available from pathology services, are often under-exploited. The main purpose of post-mortem investigations of food production animals is to provide information about the cause of disease or death with regard to treatment, and prevention options for the affected herd. Besides these major diagnoses, all additional pathological findings are also recorded as text and electronically archived as necropsy reports. In addition to the value of this information for veterinarians and farmers, systematic evaluation of necropsy data may be of value the early detection of spatio-temporal clusters of syndromes which may result from a new disease emerging into a population or from changing patterns of endemic diseases. As such, it has the potential to be of value for both nation-wide and international (veterinary) public health early-warning systems.
The rest of this paper is organized as follows: We present our efforts in constructing and exploiting a veterinary terminology resource in Section 2. Section 3 describes our work towards report classification in the context of building a surveillance tool. The next steps and further application scenarios are given in Section 4. 2
In the process of report classification, we have put a lot of effort in the construction of a terminology resource that suited our needs. The resulting term inventory is tailored to a very specific task. Still, the methods, insights and even the resource itself can be of use for other applications. Similar to the work by Rinaldi et al. (2002), we extracted a set of terms from a collection of raw text and used automatic methods to organize them into a hierarchical structure. Section 2.1 introduces the categories we used for classification. In Sections 2.2 and 2.3, we describe the steps that led to the construction of the term inventory. Sections 2.4 and 2.5 show how this resource can be automatically enhanced for a more general usage. 2.1
The work described here is based on post-mortem reports that were compiled by the Institute of Animal Pathology (ITPA) of the Vetsuisse faculty at the University of Bern. The data were entered into a database by veterinary pathologists between 2000 and 2011. We used a subset of approximately 9 000 report entries regarding pigs and cattle. The reports are written in German, with a small fraction (less than 3 %) in English and French.
For subsequent quantitative analysis, we
classified all reports using two categorization
levels. As a coarse-grained categorization, we
annotated each report with the syndromic groups that
were affected by a medical issue. Each report
was assigned zero, one or more of 9 syndrome
categories (gastro-intestinal, respiratory, urinary,
cardio-vascular, lymphatic, musculo-skeletal,
reproductive, neural, other). This categorization
approximately meets the level of granularity found in
other work
The medical reports have a high number of surface variants per term. The variation is caused by inflection, inconsistent spelling and typographical errors. On a higher level, variation is increased by synonymy, i. e. the use of different terms for the same concept (e. g. Lipidose/Verfettung ‘lipidosis’). From the perspective of the given text mining task, certain derivative forms can be considered synonymous variants as well (e. g. Ulzeration besides Ulkus).
We split the report texts into tokens, which we gastro-intestinal abomasal ulcer 178 abomasitis 120 acidosis 41 cheilitis 6 cholangitis 84 colitis 368 congenital
abnormality 21 dilatation 159 displaced
abomasum 55 duodenitis 12 enteritis 2458 esophagitis 46 gastric ulcer 206 gastritis 121 glossitis 16 hepatitis 200 HIS 361 Hoflund
syndrome 10 icterus 39 ileitis 60 invagination 59 jejunitis 30 lipidosis 93 obstipation 13 omasitis 18 pancreatitis 2 parasites 13 perforation pharyngitis proctitis reticulitis rumenitic ulcer rumenitis sialoadenitis steatorrhea stenosis stomatitis trauma typhlitis volvulus defined as consecutive runs of alphanumeric characters or hyphens. We then performed a series of normalization steps in order to reduce the number of term variants when compiling an index.
The bulk of the spelling variation stems from Latin/Greek-originated terms, such as Zäkum ‘cecum’. Besides the German spelling (using the letters ä, ö, z/k), the Latin spelling is often used (ae, oe, c, respectively), and even combinations of the two are encountered. For the previous example, the following variants are present, among others: Caecum, caecum, Cäcum, Cäkum, Zaecum. We normalized the usage of these letters by replacing ä with ae and ö with oe unconditionally, while treating c differently based on its right context: before a front vowel it was replaced by z, before h and k it was kept as c, and in all other cases (including word-final position) we replaced it with k. The complexity of this rule is owed to the fact that this normalization is applied to all words, i. e. including originally German words like Kinn/Zinn ‘chin’/‘tin’, which would be confused by an unconditional conflation of c, k, z. As a side effect, the normalization of German terms occasionally captured closely spelled English terms (which were not systematically gathered), such as Enzephalitis/encephalitis.
Subsequently, we removed inflectional suffixes
using the NLTK1 implementation of the
“Snowball” stemmer for German
Stemming is based on orthographical regularities and uses only a minimal amount of lexical information. Although the method is not flawless – it may be prone to errors with very short and irregularly inflected words – it generally works well for languages with alphabetic script and has been successfully applied to many European languages. Using a stemmer, we were able to considerably reduce the number of inflectional/derivational variants. However, a number of inflectional forms were still missed by the stemmer – especially plural forms with Latin inflection, such as Ulkus/Ulzera, or Enteritis/Enteritiden, which are not covered by the stemming rules for general German grammar. The stemmer also failed to capture most of the spelling errors. Table 2 illustrates the conflation with examples. 2.3
For the syndromic classification of the veterinary reports, we manually created a list of focus terms which served as indicators for the clinical syndromes and diagnoses. Starting from a frequencyranked list of the words found in all of the reports (already grouped by their normalized form), we manually selected terms that were likely to indicate (positive) diagnoses in the reports. The list was refined by inspecting the reports that produced hits for the focus terms.
The focus terms typically consist of a single token, but we also allowed multi-word expressions. The terms are grouped by diagnosis. Thus, each diagnosis refers to a set of terms which either constitute a common name of the diagnosis or describe some of its aspects. For example, concerning injuries caused by foreign bodies, we consider Draht ‘wire’ and Nagel ‘nail’ as focus terms, even though these words only refer to the cause, but not to the injuries themselves.
As each focus term is represented by its
normalized form, a number of variant forms is already
matched, as described above. We aimed to
additionally cover variants produced by misspellings
as well as inflected forms not recognized by the
stemmer. Using approximate string matching, we
searched the reports for similar terms for each of
the focus terms. We used the simstring tool
Figure 1 illustrates how term variants were gathered around the concept of a diagnosis. A number of synonymous and hyponymous terms were added to a specific diagnosis by a human expert. These terms were used as seeds to automatically find more variants, such as inflectional and spelling variants as well as misspellings. Please note that the labeled edges are only added for illustration purposes – the relations between term 2For a study of agreement between human judgement and different similarity measures, see e. g. Efremova et al. (2014); for a general overview of similarity measures cf. Navarro (2001) and Christen (2006).
inflection
synonyms
diagnosis volvulus canonical term misspellings
synonym Darmtorsion hyponym hyponym
spelling variants
forms (such as synonym, misspelling) were not captured during this phase, as they were not needed for syndrome/diagnosis classification. However, we examined ways to partly recover this underlying structure in an automated way, as is described in the following sections. 2.4
The UMLS Metathesaurus3 is a large collection of various medical terminology resources. One of its key features is the assignment of unique concept identifiers to entries from different vocabularies in many languages, thus establishing equivalence relations across them. By creating links to Metathesaurus concepts, we can enrich our own terminology resource with information contained in the Metathesaurus, as well as making it more valuable when sharing it with others.
We used the 2014AA release of the Metathesaurus for this work. For each concept that was represented in a German vocabulary, we normalized its lemma and tried to match it against an entry among our focus terms. With this approach, we were able to establish a link to one or more UMLS concepts for 80.6 % of the diagnoses.
Since our data were organized by diagnosis, each covering a number of terms with sometimes quite disparate meanings, the connection to the Metathesaurus produced a high number of one-tomany mappings (cf. Figure 2). This difference in 3www.nlm.nih.gov/pubs/factsheets/ umlsmeta.html (a) (b) granularity hinders the exploitation of the linked information, as the meaning of many diagnoses appears highly ambiguous in terms of the Metathesaurus. In order to better match the semantic range of the UMLS concepts, we passed on to perform the mapping at the level of terms rather than diagnoses. This required us to add a hierarchical layer to our data structure: We needed to distinguish term variants (spelling and inflectional alternations, such as Caecumtorsion vs. Zäkumstorsion) from separate terms (e. g. Zäkumstorsion vs. Darmtorsion). Please note that synonyms such as Darmtorsion and Darmdrehung are considered separate terms, even though they have the same meaning.
For each diagnosis, we organized all term forms into groups of term variants. The arrangement was performed automatically, based on string similarity. While string similarity is only an unreliable approximation of human similarity judgement, and while there are a number of concurring ways of computing it, it is also difficult to determine a 0.9 0.8 threshold that clearly separates similar from dissimilar pairs of words. We therefore chose to perform supervised machine learning, i. e. automatic learning by example. We compiled a training set of positive instances of inflectional/spelling alternation as well as negative instances, i. e. pairs of unrelated words. For each pair, we computed two different string similarity measures (cf. Figure 3): cosine similarity of character trigram vectors, and Levenshtein ratio. These two measures cover different aspects of similarity, and thus their combination might capture more information than just one of them. We trained a Support Vector Machine on the two-dimensional space of the similarity measures, using a polynomial kernel function.
The automatic term grouping yielded very satisfactory results. We manually evaluated the resulting groups, requiring that all members be orthographical or inflectional variations of each other. We also allowed derivational variants (e. g. Weissmuskelkrankheit/…erkrankung ‘white muscle disease’) to be in the same group, although the separation of derivatives (e. g. Ulkus/Ulzeration) was not counted as false negative. We found that less than 6.7 % of the groups contained unequal terms (false positives), and only 1.9 % of the groups were erroneously isolated instead of being merged with the correct equivalents (false negatives). Many false positive judgements were caused by terms with only small differences in meaning, such as Muskeldegeneration ‘muscle degeneration’ and Muskelfaserdegeneration ‘muscle fiber degeneration’, which might even be regarded equal in a less strict evaluation. As for the false negatives, the number of misses could be reduced by extending the stemmer with Latin-inflection endings like Ulkus – Ulzera. 2.5
Each group of term variants was then linked to a UMLS concept if there was a match between at least one member of the group (i. e. a term variant) and of the German concept descriptions, respectively. Only exact agreement of the normalized forms was counted as a match, as preliminary experiments had shown that fuzzy matching introduced a great amount of false positives (connections between similarly spelled, but otherwise unrelated words) while adding only very few desired links. However, we were able to improve the linkage with simple heuristics, such as the removal of boilerplate expressions like nicht näher bezeichnet ‘not otherwise specified’.
In 42.1 % of the terms, we could find a match with a UMLS concept. Only 6.7 % of the matching terms point to more than one concept, which means that 93.3 % of the terms with a match can be mapped to the Metathesaurus unambiguously. However, for more than half of the terms no corresponding UMLS concept could be found at all, which is mainly due to the different domains of our veterinary texts and the predominantly humanmedicine-based UMLS. Table 3 shows some examples of the mapping.
The connections to the Metathesaurus allowed us to further enrich our data. For example, every UMLS concept has a semantic type assigned to it, such as “Disease or Syndrome” or “Pathologic Function”. Additionally, we used the concept descriptions in Metathesaurus to find more focus terms. By matching the descriptions of connected concepts against our text collection, we were able to enlarge the set of focus terms by almost 10 %.
As next steps, we plan to create links to other widely-used terminology resources, such as the Central key for health data recording by the International Committee for Animal Recording (ICAR).4 4See www.icar.org
The terminology resource described above is a key component in our efforts to create a veterinary surveillance system. We wrote a pipeline of Python scripts that assists our semi-automatic annotation of the pathology reports. The tool performs automatic annotation of syndromes and diagnoses based on the term resource, while also keeping track of manual verfications and rejections. Through a web interface, it accepts a Microsoft Excel workbook as input and produces a modified version in the same format, which allows a veterinary domain expert to inspect and modify the automatic annotations. All relevant information – such as the term resource and the assigned categories, negations (see below), and the previous manual annotations – are contained within this file. 3.1
In a keyword-based system for detecting evidence, negative expressions can play a crucial role. Occasionally, negative outcomes of an analysis are reported in the texts, and suspected diagnoses are rejected quite frequently, such as keine Hinweise auf eine Pneumonie ‘no evidence of a pneumonia’. Therefore, we aimed at identifying occurrences of focus terms that are mentioned in a negated context.
Besides the identification of negated expressions, negation detection heavily depends on the correct determination of their scope. Tanushi et al. (2013) compare different approaches to nega67
UMLS
tion scope detection in Swedish clinical reports.
According to them, “[e]mploying a simple,
rulebased approach with a small amount of negation
triggers and a fixed context window for
determining scope is very efficient and useful, if results
around 80 % F-score are sufficient for a given
purpose”
In order to validate the quality of our annotations, we organized a multi-annotator evaluation. We performed an experiment with six experts of veterinary pathology, which were asked to classify a number of reports with respect to the syndromic categories described in Section 2.1. For this purpose, we created a web interface which displayed the report text together with some metadata, one report at a time, and allowed to mark each of the syndromes as present or absent. The reports were randomly sampled, keeping the distribution syndrome gastro-int. respiratory urinary cardio-vasc. lymphatic musc.-skel. reproductive neural other avg. of species and year of creation as close to the entire collection as possible (approaching stratified sampling). Each annotator was provided with a sample of 20 reports, which was extended to twice or three times the size when an annotator asked for more. In order to increase sample size, the same report was given to only two or three annotators, rather than all of them. In total, 81 distinct reports were annotated.
We evaluated the inter-annotator agreement
with Krippendorff’s Alpha
We will assess the performance of the text-mining tool based on a small number of diseases which have been relevant in Switzerland in the last 10 years: 1. Bovine Viral Diarrhoea in cattle (an eradication campaign for the disease was introduced in 2008) 2. Porcine Circovirus type 2 infection in pigs 3. Gastro-intestinal syndromes in pigs (for which we observe an increasing amount of pathology submissions) Time-series analyses will be performed to quantify trends, seasonality and other effects (day of week, day of month etc.) on the number of submissions for syndromes potentially related to these diseases. For each disease, “in-control” data (data collected in the absence of an outbreak) will be used to establish a baseline model describing the amount of normal “noise” in the data (expected number of submissions in the absence of disease outbreaks). Retrospective analyses of the time-series will be done to see whether alerts (signals) were produced when the number of submissions for syndromes potentially linked to the disease was higher than expected from our baseline model (event detection). This will allow us to evaluate whether the system would have worked as an early-warning system.
The tools developed in this project will be adapted to reports from different pathology institutes throughout Switzerland, thus contributing to a nation-wide syndromic surveillance system. Similarly, the methodology developed may be applicable to the analysis of text-based disease information which is recorded in other contexts. For example, there is a great potential of using such a system to systematically analyse health data which are recorded by veterinary practitioners in their practice management software, slaughter data or by animal health services in their central database.
This work was funded by the Swiss Federal Food Safety and Veterinary Office (Bundesamt für Lebensmittelsicherheit und Veterinärwesen).