Syndromic surveillance systems aim to support early detection of salient disease outbreaks, and to shed timely light on the size and spread of pandemic outbreaks. They can also be used more generally to monitor disease trends and provide reassurance that an outbreak has not occurred. One commonly used technique for syndromic surveillance is concerned with classifying Emergency Department data, such as chief complaints or triage notes, into a set of pre-de ned syndromic groups. This paper reports our ndings on the investigation of the utility and e ectiveness of two existing North American methods for free-text chief complaint classi cation on a large data set of Australian Emergency Department triage notes, collected from two hospitals in the state of Victoria. To our knowledge, these methods have never before been analysed and compared against each other for their applicability and e ectiveness on free text chief complaint classi cation at this scale or in the Australian context.
existing systems utilize keyword-based, linguistic, statistical, and/or character-level data from CC texts in the
classi cation process. Ivanov et al. [11] conducted a retrospective study to ascertain the potential of free-text
CCs collected in pediatric emergency departments. They used the Bayesian classi er implemented in Complaint
Coder (CoCo) [
In this paper, we present an evaluation of the applicability and e ectiveness of two existing machine
learningbased North American CC classi ers, namely Symptom Coder (SyCo) [
Our analysis of SyCo and CoCo on free text CC classi cation is based on a syndromic surveillance data set
that includes ED triage notes from two di erent hospitals in the Australian state of Victoria. We focus on
three primary medical conditions of particular interest in public health surveillance: Flu Like Illness, Acute
Respiratory, and Diarrhoea. According to [
The only previous research that compared the two systems is the work in [
Symptom Coder (SyCo) [
The Bayesian probabilities that CoCo calculates are determined using a default probability le (developed
by RODS) that was derived from 28,990 CC strings each manually coded by a physician into a syndrome
category [
SyCo di ers from CoCo in that it implements a two-layer classi cation procedure from chief complaints into syndromes. SyCo rst nds the posterior probabilities of a number of symptoms (i.e., 17 in-built symptoms) given the text of the chief complaint. Consequently, SyCo calculates the posterior probabilities of each syndromic group given the (posterior) probabilities of the symptoms. The syndrome with the highest posterior probability is then selected as the syndromic group for the chief complaint.
The textual features that CoCo and SyCo extract from CCs are at the word level only, excluding phrases,
n-grams, or any biomedical terminology. The study conducted by Connor et al. [
Silva et al. [
The SynSurv data is data collected from the Emergency Departments of the Royal Melbourne Hospital and the Alfred Hospital during the period 1998-2010 (predominantly from 2000 to 2009). The data were collected on behalf of the Victorian Department of Health (Vic Health) for syndromic surveillance during the 2006 Commonwealth Games held in Melbourne. The Vic Health remains the custodian of the data; however, for the purposes of this project, we were granted permission by Vic Health to work with it.
Data set Training Testing
Syndromic group Flu Like Illness Acute Respiratory Diarrhoea Other Total: Flu Like Illness Acute Respiratory Diarrhoea Other Total:
SynSurv consists of naturally occurring data collected at triage, consisting primarily of free text notes written by a triage nurse for each patient visit to the EDs of the selected hospitals. These notes have been augmented with annotations of diagnostic codes using the ICD-10 version of the International Classi cation of Disease.
The data set contains a total of 918,330 ED visit records. From these records, there are 730,054 visits with a valid ICD-10 code with the primary diagnosis as entered by a nurse upon triage assessment of the patient. In SynSurv, there are 456,213 records with any textual comment at all. The total number of records with both a diagnosis and a textual nurse note is 316,362 entries. From this data set of 316,362 CC records, we used a subset of 314,629 records that were labelled with one of the three syndromic groups Flu Like Illness, Acute Respiratory, and Diarrhoea, or as other. Table 1 summarizes the distribution of the di erent syndromic groups in the data set. 3 3.1
For the machine learning-based classi cation experiments that follow, the set of records corresponding to a given syndromic group was used as positive examples for that syndrome and all other records in the relevant subset of SynSurv (see Table 1) were considered to be negative instances for the syndrome.
We trained one binary SyCo classi er for each of the syndromic groups with the training portion of the data set for the given syndromic group and then, tested the e ectiveness of the classi er using the corresponding test set for the syndromic group.
To evaluate CoCo, which we did not train with our training data sets, we only used the test sets to measure the classi cation performance of the tool for each syndromic group. For this, we had to nd a mapping between our syndromic groups and those eight syndromic categories that CoCo had been trained with. Table 2 shows the mapping that we considered between the two categories of syndromic groups.
To understand how the performances of SyCo and CoCo compare with those of trivial baseline systems, we developed two trivial baseline classi ers:
All-Positive (All+) Baseline, which assigns a positive class label to every given instance of CCs in the test set. A positive class label means that the chief complaint is labelled as the given syndrome under investigation.
Random-50-50 Baseline, which assigns either a positive or a negative class label to every given instance of CCs in the test set. The assignment of class labels is based on a binary random generator. To more accurately account for the randomness of this baseline system, we generated average evaluation results for 10 consecutive independent runs over the same test set related to each syndromic group.
Syndromic group in Victorian data Flu Like Illness Acute Respiratory Diarrhoea
Syndromic group in CoCo Constitutional Respiratory Gastrointestinal We evaluated the classi cation methods using the number of True Positives (TPs), False Positives (FPs), Precision (i.e., speci city), Recall (i.e., sensitivity), and F1-measure. Table 3 summarizes the results achieved using SyCo and CoCo as well as the two baseline systems All+ and Random-50-50 on the above-mentioned data set. SyCo All+ Flu Like Illness Acute Respiratory Diarrhoea Flu Like Illness Acute Respiratory Diarrhoea
From the results in Table 3, and considering F1-measure, from the two chief complaint classi cation systems, SyCo has been demonstrated to perform better on all of the three syndromic groups. In terms of precision and recall, SyCo outperforms CoCo again in most cases, except for the Acute Respiratory group where CoCo shows a higher precision (0.3603 vs. 0.2483). This single scenario where CoCo outperforms SyCo corresponds to the situation in which SyCo returns with a large number of FPs (5,845) versus only 1,843 FPs returned by CoCo for the same syndromic group. This could not be compensated for by the relatively small di erence between the numbers of TPs that both methods returned (CoCo: 1,038 vs. SyCo 1,931). One major factor that may explain the superior performance of SyCo over that of CoCo in our experiments is the fact that CoCo was not trained with our training data sets whereas each SyCo binary classi er was trained for each of the syndromic groups with the corresponding training set in our data set.
According to the results in Table 3, SyCo and CoCo result in a large di erence in the recall values for the
Flu Like Illness category (CoCo=0.1561 and SyCo=0.5378). These recall values are both relatively di erent
to what Silva et al. [
The results in Table 3 also demonstrate that: i) neither of the two baseline classi cation systems perform well on any of the three syndromic groups, and ii) more importantly, both SyCo and CoCo perform relatively higher than the trivial baseline systems.
To understand how SyCo and CoCo classify each instance of the CCs in the SynSurv data set, we conducted a Cohen's Kappa statistical agreement analysis on the output classi cations of the two systems. For this, a pair of output classi cations for each CC string was created and the entire set of pairs were analysed for their agreement with each other.
The relatively weak agreement between SyCo and CoCo on our data suggest that the two classi er systems model and interpret the CCs in relatively di erent fashions. As a result, there is a dissimilar, possibly complementary, coverage over true positives and true negatives. A possible next step would be to bring the two classi ers together using an ensemble approach.
Syndromic surveillance is a widely-utilised procedure for early detection of salient disease outbreaks. One of the most commonly used techniques in the area of syndromic surveillance is based on supervised classi cation of Chief Complaints into a set of pre-de ned syndromic categories. In this paper, we analysed the performance of two well-known CC classi ers from the Real-time Outbreak and Disease Surveillance (RODS), namely Symptom Coder (SyCo) and Complaint Coder (CoCo). While CoCo was used as an o -the-shelf component, with no training in our experiments, SyCo was trained with the training subsets of the data we used in this work. The results of our analysis on a large data set of Australian ED notes labelled with three syndromic groups Flu Like Illness, Acute Respiratory, and Diarrhoea suggest that, in most cases, SyCo outperforms CoCo in terms of the evaluation metrics. Both SyCo and CoCo outperform the two trivial baseline systems that we developed for performance comparison reasons only. We also found that the two classi ers do not agree on the classi cation outputs, i.e., the classi ers make di erent mistakes, which may suggest that an e ective approach may be required to combine the two classi ers. 6
This work was supported by the Bioterrorism Preparedness task of the Land Personnel Protection Branch, Land Division of the Australian Defence Science and Technology Organisation (DSTO). We also thank the Victorian Department of Health and Human Services for their contribution of the SynSurv data set. This research was conducted under Ethics Approval 21/14 by the Department of Health Human Research Ethics Committee. [9] International Society for Disease Surveillance. Final recommendation: Core processes and EHR requirements for public health syndromic surveillance. Technical report, ISDS, 2011. URL www.syndromic.org/projects/meaningful-use. [10] C.B. Irvin, P.P. Nouhan, and K. Rice. Syndromic analysis of computerized emergency department patients chief complaints: An opportunity for bioterrorism and in uenza surveillance. Annals of Emergency Medicine, 41(4):447{452, 2003. [11] Oleg Ivanov, Per H. Gesteland, William Hogan, Michael B. Mundor , and Michael M. Wagner. Detection of pediatric respiratory and gastrointestinal outbreaks from free-text chief complaints. AMIA Annual Symposium proceedings, pages 318{322, 2003. ISSN 1942-597X.