We extended Cocoa/Peaberry, our (RelAgent) existing rule based entity and event tagger, to tag attributes associated with diseases in clinical records. The boolean attributes of Negation, Uncertainty and Conditional were handled by an extension of the NegEx algorithm. The multi-valued Course and Severity attributes were detected either within the extended disease spans as output by the system, or by event-based annotation using a predicate-argument framework. The anatomical attribute Body Location was marked up by either nding embedded body parts in the extended disease spans or by being colocated close to the disease span. UMLS IDs for anatomical locations were derived by using a small number of morphological lemmas, and by a few rules derived by manual inspection in case of multiple hits. We used the most frequent value in the training data for Subject, Generic, and time-related attributes.
Tracking the severity, anatomical location, and temporal factors, including the course, pertaining to a disease/symptom is of signi cant value in diagnosis. The incidence and progression of a disease are also relevant to tracking response to treatment with various medications, which is useful both in clinical practice as well as in phase trials.
Tagging of diseases, signs and symptoms themselves as well as their
normalization to SNOMED terminology was the focus of the 2013 ShARe/CLEF
eHealth task 1, which covered a variety of clinical documents, such as discharge
summaries and echo/radiology/ECG reports. Disease-tagging tasks share a
degree of overlap with previous tasks which marked up radiology reports [
The current ShARe/CLEF eHealth task 2 [
Cocoa/Peaberry is our (RelAgent) existing named entity and event tagger
for published literature in the biomedical domain. The system performs
reasonably well in various tasks ranging from tagging entities and events in the
molecular/cellular domain [
We have used the Cocoa/Peaberry system to detect diseases, signs and symptoms
for the 2013 CLEF ehealth task 1 [
Brie y, the system marks up anatomical parts (`liver') and disease headwords (`cancer') separately, and then merges them to get the extended disease entity. Body parts tagged by the system are composed of maximal spans, such that words corresponding to location (`left upper') or laterality are merged into the body part entity. For disease entities, words describing severity (`acute'), frequency (`recurrent', `regular'), state (`unresectable', `disseminated') are also merged if they are located proximally. When anatomical parts are in coordination, and the last occurrence abuts a disease headword (`liver and breast cancer'), all the coordinated anatomical parts are marked up as disease entities. Finally, unusually named diseases or symptoms such as `premature cardiac complex'. `long QT syndrome' are handled. The nal tagged disease entity is thus of maximal span in that all adjacent words in any way pertaining to a disease are merged into the entity.
Many disease entities have disjoint spans, especially when they correspond to pathological changes in body parts (`enlargement of the left ventricle') or to a source (`drainage from wound'). In a predicate-argument formalism, one of the disjoint spans appears as an argument of the other span, where the predicate appears either as a verb or in a nominal form. The system has an event detection module to detect such cases.
For the Severity attribute, we used the training set to make a list of trigger words that correspond to the three severity classes, namely `slight', `moderate' and `severe'. The disease entity was marked up correspondingly for severity if the extended disease span contained any of these trigger words. Similarity, embedded trigger words associated with the various values for the Course attribute were obtained from the training sets. Examples are 'progressive' for 'worsened' and 'healed' for 'resolved'. However, the majority of the Course attribute data derived from verbal markers for the disease entity. An example for the value 'improved' is the fragment: 'mental status changes that responded well to Haldol' which follows the template 'Disease responds to Chemical'. We use the event detection module to mark up the Course attribute for such cases with about 15 trigger word driven event templates (verbs or nominals). However, we ignored the value 'changed' for the Course attribute as it caused too many false positives, and its occurrence in the training set was small (less than 2%)
As the system marked up anatomical parts before merging them with disease headwords, embedded body locations are automatically output by the system. Additionally the event module also marks up cases where the disease entity is linked to an anatomical part or location through an intervening preposition, as in `bleeding in your esophagus' or `loculated e usion seen on the left side'. However, there exist a large number of examples where the anatomical location does not occur in a sentential context, but is implied by the discourse, as when it heads the utterance with a following colon or a hyphen, as in `Abdomen: no masses'. Thus for diseases which do not have an embedded or prepositionally proximal body part, we look for occurrences of anatomical locations within 100 characters of the disease entity span. We constructed about 50 rules matching the anatomical part with the disease, e.g., `murmur' or `gallop' match to cardiac entities such as `CV', `atheroma' or `extravasation' match to `artery', `clubbing' and `edema' with `extremities' and so on.
We mapped anatomical entities to UMLS IDs through a collection of
morphological transformations which converted the entity as it occurred in the text
to a regular expression. The framework for this module is the same as the one we
have used in other shared tasks to map diseases to their UMLS or MeSH IDs [
The Negation, Speculation and Conditional attributes were handled by
using the NegEx algorithm [
We did not address the other attributes. For the Subject and Generic
attributes, the data was somewhat sparse, and we decided to leave the default
value in place. Detecting time attributes is well known to be di cult task [
We tested and re ned the performance of the system against the training set. We then ran two runs against the dataset, which di ered only in that the 2nd run split a word into two tokens if there was a slash (`/') character in the word. The two runs produced results that did not di er in the overall accuracy (0.843), and we have shown the better of the results of the two runs for the individual attributes in the column titled `Test set' in Table 1 below. The column entitled `Best' is the best result over all systems for each attribute, while the `Baseline' column shows the result if the gold template had been returned unaltered, i.e. the accuracy with the default value for each attribute. These `Baseline` gures were taken from the accuracy results for systems which had an F-score of 0:0 for that attribute.
One note of relevance is that we did not directly use the gold annotations
supplied by the task organizers (except for one case; please see below). Instead, we
used the system to itself detect the disease entities while simultaneously marking
up the disease attributes. Then, for every disease span in the gold annotations,
we found the rst disease span in our own annotations that overlapped with the
gold span using the overlap algorithm in the 2013 ShARe/CLEF eHealth Task
1 evaluation script [
We extended Cocoa/Peaberry, an existing multi-class entity tagger for the biomedical domain, to detect attributes of disease entities in clinical records. With fairly minor improvements, the system came second in overall accuracy and performed reasonably well in most attributes.
Except for the Body Location attribute, we did not directly use the gold annotated disease entities, instead using the system itself to detect and tag the disease entities and subsequently (and nally) transferring their attributes to overlapping gold annotated entities. We believe therefore that our results on the test set are likely to be close to results against unannotated data, where disease entities are not tagged beforehand, and our results are encouraging in this regard. However, improvement against the baseline is not very high for Cocoa for many attributes, as re ected in the F-scores (not shown), indicating that system performance could bene t from further improvement generally, but particularly for some attributes such as Body Location.
Acknowledgments. We thank Shereen Broido for discussions. The ShARe/CLEF eHealth shared task was made possible by a grant to the task organizers.