We report on a research effort to create a corpus of clinical free text records enriched with annotation for symptoms of a particular disease (ovarian cancer). We describe the original data, the annotation procedure and the resulting corpus. The data (approximately 192K words) was annotated by three clinicians and a procedure was devised to resolve disagreements. We are using the corpus to investigate the amount of symptom-related information in clinical records that is not coded, and to develop techniques for recognizing these symptoms automatically in unseen text.
UK primary care databases provide a valuable source of information for research
into disease epidemiology, drug safety and adverse drug reactions. Analyses of
existing large-scale electronic patient records held in the form of lar ge primary
care datasets such as the General Practice Research Database have almost
exclusively exploited coded data. Such data are readily accessible to the classical
methods of epidemiological analysis, once the complexities of defining and
selecting a patient cohort have been overcome. However, since clinicians can choose
to what extent they code a consultation, an unknown amount of clinical data is
not coded, and ‘hidden’ in free text. Free text records often con tain important
information on the severity of symptoms or on additional symptoms which have
not been coded [
Automatic extraction of complex information from notes written by general practitioners, which may be ungrammatical and often contains ambiguous terms, misspellings and abbreviations, is a very challenging natural language processing (NLP) task. The text is much less uniform than data typically analysed by the NLP research community, and issues of confidentiality make it difficult to gain access to significant amounts of data. 2
This study builds on previous work [
The free text records contain information from a variety of different sources. Mostly they consist of notes typed by the GP during or after a consultation, communication with secondary care (for example referral letters and discharge summaries), and sometimes test results. However, about 90% of the records contain notes typed by the GP, which turn out to be the most challenging category. A typical example is: 5 day Hx of umbilical Dx. Smelly Dx. Red inflammed lump within umbilicus. Swab sent. Try fluclox and review. Also ?mass felt left lower abdo.
No weight loss, bowels reg. No Jaccol.
Some differences between this data and standard English are: (1) inconsistent use of capitalization and punctuation; (2) spelling errors and unusual abbreviations, acronyms, and named entities; (3) anomalous tokenization (e.g. missing spaces); and (4) ambiguous use of question marks. These characteristics make it difficult to process the data automatically. They also impact on readability, making human annotation more time-consuming, and therefore cos tly. 3
In order to annotate symptoms associated with ovarian cancer in the free text
fields of patient records, we first identified the most commonly experienced
ovarian cancer symptoms. Table 1 lists these symptoms, which were taken from a
recent paper by Hamilton [
To facilitate the work of the annotators, we created an easy to use
interactive annotation system. We used the Visual Tagging Tool (VTT), part of the
SPECIALIST NLP Tools [
We drafted a detailed set of annotation guidelines. Over several iterations we
refined the guidelines to minimize potential disagreement between the annotators
— learning from others’ experiences in defining a methodology for an notating
clinical data [
The main supporting instructions are: – Do not infer a symptom from the text : only annotate symptoms if found as such in the text. – Presence or absence : e.g. in the case of T‘here was no evidence of abdominal distension,’ a‘bdominal distension’ should be marked up. – Annotate the bare minimum : only annotate as much text as you need to identify the symptom. E.g. .‘..who has had right upper quadr ant abdominal pain now for some weeks,’ only a‘bdominal pain’ should be marked u p.
– Annotate every occurrence of a symptom in a record .
Five individuals with a medical background were involved in developing the guidelines, iteratively refining them on the basic of exploratory annotation sessions. Once the final version of the guidelines was produced, three people each annotated the whole dataset independently. One of the annotators was also involved with developing the guidelines. All the annotators have a medical background, either as a General Practitioner, a researcher with GP training, or a final-year medical student. The annotators were given a copy of t he annotation software and worked with it at their convenience. The data was presented to them in batches of about 800 records each (but there was no requirement to finish a batch in a single session). 4
We started with 6141 records, determined by the amount of data available for the 344 patients for the period 12 months prior to the date of definite diagnosis (Section 2). After these records were annotated by the three annotators, we created a gold standard corpus. During the development of the guidelines and exploratory annotation sessions we noticed that there were two main areas of disagreement between annotators. Firstly, annotation is a mentally strenuous task, and as a result annotators occasionally miss symptoms, especially when a record contains a large number of them. Secondly, disagreements arise from the fact that annotators are free to decide the points at which each marked-up symptom starts and ends.
Inter-annotator agreement [
In our setup it is difficult to calculate an exact figure for agreement other
than the most basic one. The most straightforward measure is strict agreement,
defined as the proportion of all cases where all three annotators were in full
agreement (those cases where the annotated string starts and ends at exactly
the same position in the text and the string is assigned the same label). We
found three-way strict agreement in about 62% of cases. This figu re is difficult
to compare with other work reported in the literature. The closest comparison we
are aware of is the inter-annotator agreement for finding ‘Signs an d Symptoms’
in [
Both [
In order to maximize the quality of the gold standard, we had to decide
on a method for combining the data individually created by the annotators. A
typical way of resolving disagreement between annotators is to have the data
double annotated and appoint a third annotator to choose between the two
conflicting opinions. However, this method does not allow for discussing cases
that are inherently difficult. We therefore decided to keep the triple annotation,
and employ a variant of the Delphi method [
In order to use the annotators’ time efficiently, we created five ca tegories from the cases of disagreement, each of which could be approached differently. The categories were: 1. Trivial difference: e.g. A‘bdominal pain’ vs. A‘bdominal pai’ 2. Added modifier: e.g. s‘lightly bloated abdomen’ vs. b‘loated abdomen’ 3. 2 agree/1 missed out: one annotator may have overlooked a symptom 4. 2 agree/1 disagree: with respect to the span of the string or associated label 5. Rest: all other instances The first two categories, accounting for around 20% of the disagreements, were easy to resolve. The third and fourth categories had to be checked one by one by the annotators, but were mostly easily resolved. The Rest category was the most challenging. Many cases in this category had multiple disagreements (over both the span of the string and the associated tag) and often the annotators had to view the full context of the annotated string to reach agreement.
The resulting annotated corpus consists of a total of 6141 records, containing about 192K words. The total number of annotated symptoms is 3955. Table 2 summarizes the symptoms labeled in the corpus. Even though the average number of occurrences of each distinct expression is low, the distribution is very skewed (see Figure 2). For example, ‘abdominal swelling’ is an expres sion that the annotators label as ‘Abdominal Distension’; there are ten occu rrences (‘tokens’) of this expression (‘type’) in the corpus. However, there is g reat variation in the expressions used to describe the same symptom. The more formal ones are used frequently, whereas informal expressions (e.g. ‘tummy is str ikingly larger’ or ‘abdo looks sl swollen’) often occur only once.
In one experiment we looked at a subset of symptoms, covering about 3281 annotated expressions. Approximately 1200 of these occur only once in the corpus. Figure 2 shows that the most frequent 100 types account for almost 65% of the tokens. This observation is important for designing an automatic system that finds this information in the text. If recall is not an absolute priority, then it is possible to get a long way by concentrating on the high frequency types. Moreover, the most frequently occurring expressions are generally more formal and concise, which would also be of assistance. However, a system would also need to integrate relevant contextual factors, in particular whether indications of symptoms are negated or are attributed to someone other than the patient. 5
The corpus we have produced is a rich resource, which we are using for two purposes. Firstly, we are investigating the amount of symptom-relate d information available in the free text fields of primary care patient records. By comparing symptom annotations with coded information in the same records we are exploring the hypothesis that a significant amount of information is missed when the contents of free text fields are not taken account of in epidemiological research. Secondly, we are starting to use the data for creating and evaluating techniques for automatic recognition of symptoms in free text. Although we are ultimately interested in developing machine learning-based models that precise ly capture as wide a variety of symptoms as possible, the annotated corpus also allows us to estimate the utility of unsophisticated techniques such as approximate string matching and thesaurus-based expansion. If significant amounts of information can be uncovered with such methods, then the epidemiological research community would not require specialised natural language processing expertise to be able to exploit free text resources. Automatic processing of information in free text fields also opens up opportunities to work with un-anonym ized data. Restricted access to textual data is a major hurdle in research using electronic patient records. The possibility of retrieving information without the need for manual anonymization would open up many new opportunities.
The process of establishing consensus between annotators has given us new insights into the nature of the data and has highlighted issues that need to be addressed when processing this data and when defining further annotation tasks. One of the main issues relates to symptoms recorded as a result of a complaint by the patient versus the outcome of an examination or a test result. From a clinical point of view, a complaint has a different status to an examination result. Even though many potential disagreements of this type were resolved in the course of developing the annotation guidelines, new issues will crop up when annotating large amounts of text. Decisions about how to deal with these cases might need to be informed by the research question being addressed.
There is evidence that important information is missed when epidemiological research using patient records relies only on coded information. The research reported here is a step towards quantifying how much additional information is potentially available, and is a prerequisite for research into automatic retrieval of this information and making it available to the research community. Acknowledgments This work was supported by the Wellcome Trust [086105] (The ergonomics of electronic patient records: an interdisciplinary development of methodologies for understanding and exploiting free text to enhance the utility of primary care electronic patient records (“PREP”) ). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.