Online reports are increasingly becoming a source for early warning systems that detect natural disasters. Harnessing the massive volume of information available from multilingual newswire presents as many challanges as opportunities due to the patterns of reporting complex spatio-temporal events. In this paper we propose a role for an automated system based on cross-language text mining. We track the evolution of 16 disease outbreaks using 5 aberration detection algorithms on textmined events. Using ProMED reports as a silver standard, news data for 13 languages over a 129 day trial period showed improved recall and timeliness using cross-lingual events.
As electronic data expands, online
reports are coming to represent a new
modality in early warning surveillance
for natural disasters such as epidemics
Since 2006, BioCaster
In order to support the task of detecting the unusual, we compare five widely used aberration detection algorithms to look for spikes in the geo-temporal event stream. In particular this paper seeks to explore the hypothesis that cross-lingual events from text mining can provide improved detection rates for novel events. Although we focus here on newswire as a source we believe the results should have applicability for other unverified reports such as email lists and the rapidly emerging blog space. 2
The 2009 H1N1 pandemic illustrated
how dependent each country is on the
surveillance capacity in other states.
Reducing public health risk depends on
an overall strengthening of global health
event monitoring as well as locally
available sources such as clinical data and
over-the-counter sales data. The Web
provides a low cost surveillance
infrastructure that has been shown to offer
a timely means of detecting epidemics
such as SARS
Several characteristics of early
epidemic detection make the problem
particularly challenging. Firstly, we want
to catch epidemics as early as
possible before they develop into
humanitarian crises; Secondly, not every
epidemic is of equal importance - those that
are of most concern to the international
community are described by the
International Health Regulations
How is this work different to various
research in topic detection and tracking
(TDT)
In general it is extremely difficult to
determine ground truth for the actual
numbers and durations of disease outbreaks.
As a silver standard we have chosen the
best publicly available human network of
reporters which is ProMED-mail
In this study we have used quite coarse-grained granularity by choosing countries and days as the units. This is due to the current limits of reliable location detection in the system and also the frequency of news that we observe. The recorded time for each event was normalized to system download time which takes place every hour of each day.
Evaluation uses the standard classification test measures of sensitivity (recall), specificity, positive predictive value (PPV or precision), negative predictive value (NPV) and timeliness. We also measured the average number of system alarms per 100 days and compared this to the silver standard. The F-measure (F1) is calculated in the usual way as the harmonic mean of sensitivity and PPV.
The standard for a true positive was to obtain a system alert on a countrydisease event on or before the silver standard alert. The period for a qualifying system alert was set as up to 7 days prior to and including a qualifying ProMED report on the same topic. True positives were increased by 1 if there was any system alert that fell within the 7 day period. Multiple system alerts did not count twice. False positives were increased by 1 for each system alert that fell outside of the 7 day window. False negatives were counted as the number of qualifying alert periods when there were no system alerts. True negatives were counted as the number of days outside of any qualifying alert period when no system alert was given.
In testing we tried to maximize F1 together with timeliness. 3.2
Figure 1 shows the 16 event streams that we explored. The events chosen for this study were determined based on diversity of geographical and media coverage rather than random selection. The 16 event streams contain 2064 surveillance days with 153 events (7.4% of alerting days)1. Since we wanted to explore the hypothesis that linguistic coverage in multiple languages could strengthen detection rates and timeliness we compared English news coverage against all languages including English for each of 16 disease outbreaks. Because cross-lingual events on the 13 languages were only available from December 2009, the trial period was from January to May 2010.
ProMED reports used in the silver
standard excluded those that fell outside
our case definition, based on the
International Health Regulations
The text mining system we explored involves a semantic pipeline of modules running on a high throughput cluster computer with 48 Xeon cores. Throughput is approximately 9000 articles per day. System news was gathered from multiple news sources through Google News and MeltWater News as well as specialized sources such as the European Media Monitor, IRIN and ReliefWeb. (Note that no ProMED-mail messages were included in the system data for this study using a block on the Internet domain and message title). In total this gives us access to over 80,000 news sources globally. The languages used in the study (in ISO-639-1) are: ar,zh,nl,en,fr,de,it,ko,pt,ru,es,vi and th.
Underlying the system is a publicly
available multilingual application
ontology
After data sourcing, translation takes
place from the twelve non-English
languages used in this study using Google
Translate. Following this, text
classification using Naive Bayes (F-score 0.93)
removes non-disease outbreak news
before text mining is applied. Rules are
based on a regular expression matching
toolkit called the Simple Rule Language
The final structured event frames in XML includes slot values normalized to BCO root terms for disease, pathogen (virus or bacterium), time period, country and province. Additionally we also identify 15 aspects of public health events critical to risk assessment. For the purpose of this study we only made use of disease and country slots. Events in the 13 languages are treated in this study as being part of a univariate model for comparison purposes against English events.
Latitude and longitude of events down to the province level are found automatically using Google‘s API up to a limit of 15000 lookups per day, and then using lookup on 5000 country and province names harvested from Wikipedia. 3.4
We experimented with a range of popular
models for early alerting used in the
public health community: the Early
Aberration and Reporting System (EARS) C3,
C2 and W2 models as well as the
Fstatistic and the Exponential Weighted
Moving Average (EWMA). All were
implemented in Excel for the purpose of
this study. The models are what might be
termed ‘snapshot’ models because they
all use short 7 day baselines that assume
a relatively stationary background, i.e.
ignoring medium to long term periodic
variations such as seasonal cycles. The
baselines are used to predict future trends
against which the current day values are
compared. All models also use a 2 day
‘guard period’ just before the target day
t to prevent the current day’s data from
being included in the baseline. All
models use a minimally supervised method
by setting a threshold parameter which
we determined using the same 5 held out
data sets used by
C2
The Early Aberration and Reporting
System (EARS) algorithms
C3
C3 is a modified version of C2 so that the
previous 2 observations (within the guard
period) are added to the test statistic if
the counts on those days does not exceed
a threshold of 3 standard deviations plus
the mean on those days. The rationale
here is to extend the sensitivity of C2.
W2
W2
The calculation for the
F-statistic 2 2
St = σt + σb where σt2 approximates the variance during the testing window and σb2 approximates the variance during the baseline window.
Calculation is as follows: (2) (3) (4) nt test 1 nt nb test (Ct − μb)2 (Ct − μb)2 (5) (6)
Unlike other models in our test, the Exponentially Weighted Moving Average (EWMA) provides for a non-uniformly weighted baseline by down-weighting counts that are on days further from the target day:
Y1 = C1
Yt = λCt + (1 − λ)Yt−1 where 1 > λ > 0 is a parameter that controls the degree of smoothing. The optimal level found from held out data was found to be 0.2. The test statistic is calculated as:
St = (Yt −μt)/[σt ×(λ/(2−λ))0.5] (7)
As above, μt and σt are the mean and standard deviation on the baseline window. 4
Interestingly we found that 80% of news reports covered only about half the ProMED-mail alert disease-country topics, implying that the remaining 20% of news has to provide coverage for almost half the topics. Surprisingly, the trend was broadly similar for both English and all language news. Although the sample size is relatively small, given that the events we chose were from all regions of the world, this implies that having news in more languages may have a deepening effect rather than a broadening effect on event coverage. The three notable exceptions were in the cases of FMD in China (e4 in Figure 1), Dengue in Brazil (e12) and Dengue in Bolivia (e13).
Results for global events on English (Table 1) show an advantage for the Fstatistic if we are primarily concerned with sensitivity (recall) and alerting rates (shown in column B). However the Fstatistic has a clear disadvantage with PPV (precision) which impacts heavily on the number of false alarms. This can be seen clearly by comparing the alarm rate per 100 days of 16.2 in column A with the ProMED average of 7.4. Both advantages and disadvantages are amplified when we add cross-lingual events.
Whilst the F-statistic has the highest overall F1, its high rate of false alarms reflected in the PPV makes it potentially an undesirable choice. If we seek for the best balance of F1 and timeliness with a minimum of false alarms then C3 looks like a more desirable alternative.
Cross-lingual event capture seemed to extend sensitivity in all models, improving F1 and timeliness. To see if we could harden our intuitions about these effects we looked specifically at South East Asia - a region where we would expect the representation of Chinese to be proportionately greater than English. Table 2 shows results which largely mirror those for the world as a whole. The noticable exception though is that EWMA shows a large drop in performance. 5
Although the sample size is limited, the data suggests trends in model performance. C3 seems to perform best when we consider that the high false alarm rate for the F-statistic could desensitize users. Cross-language events generally seem to improve F1 performance by several points across most models except for EWMA. The benefits come from an extension in sensitivity but could be focussed on topics where we already have large coverage of English news. This is not to say that multilingual news is not useful, as we comment below, it could be that it has a greater role to play in extending detection rates of novel events at lower levels of geographic granularity than the country.
The study raises several questions about factors in the imbalance of reporting: why did Dengue in Brazil (e13) or FMD in China (e13) receive such massive local coverage but disproportionately less in the English media? Why did cholera in Angola (e9) or influenza in Romania (e8) receive comparatively low coverage overall? We also observed that the USA epidemics (e15 and e16) were widely reported in English but not so greatly in other languages.
In order to illustrate the potential complexity of the task we provide a detailed drill-down analysis of one of the outbreaks in our data set, i.e. cholera in Angola. Just to put the reporting of this outbreak into context: Angola itself is a former Portuguese colony which has suffered major outbreaks (e.g. 2006 to 2008) of cholera due to poor sanitation, drinking water infrastructure and environmental conditions. Although UNICEF has commented on recent advances, the country remains at risk, especially during the rainy season from January to mid-May. 21/1/2010 BioCaster detects 1 report in
Spanish of 31 cases of cholera from October to December 2009. The report is republished in English and again in Spanish and Portuguese over the next few days. Since the report is for a historical outbreak (> 3 weeks old) it is a false positive. 19/2/2010 ProMED-mail issues a report in English on cholera in Bocoio, Angola between 12/2/2010 and 18/2/2010. The cited source was the Angola Press Agency on 19/2/2010. BioCaster failed to capture this, so it is a false negative.
4/3/2010 The Angop issues a report of 8 Mean number of days that alerts were given before ProMED
mail reports. Figures in parentheses show 95% CI. guage. alarms per 100 days; The mean number of ProMED-mail alerts per 100 days was 8.1.
B
Mean number of days that alerts were given before ProMED
5.1 5.2
NPV 0.9 (0.93,0.97) 0.92 (0.93,0.97) 0.92 (0.92,0.96) 0.82 (0.94,0.97) 0.91 (0.92,0.96) NPV 0.89 (0.94,0.97) 0.91 (0.94,0.98) 0.91 (0.94,0.97) 0.79 (0.95,0.98) 0.89 (0.92,0.96) deaths from cholera in the province of Namibe. The report is cited by ProMED-mail on 19/3/2010. 6/3/2010 BioCaster detects the 4/3/2010 report in its Spanish version. The status of this report is a false positive in the silver standard but should be considered as a true positive since it is a direct translation of a cited source used by ProMED-mail. 10/3/2010 BioCaster detects reports in Spanish of a prevention campaign against cholera in Luanda. This seems to be a false positive but several such reports raise a system alarm. The high average number of reports means that smaller peaks of true positives on 14/3/2010 and 16/3/2010 do not raise system alarms. The F1 scores for multilingual reports are therefore lower than for English. 14/3/2010 BioCaster detects 1 report in French from the Governor of Luanda requesting civil protection measures to prevent the proliferation of cholera following heavy rain and flooding. The indication of infrastructure stress is highly indicative of a true positive. Due to the high frequency of Spanish reports on 10/3/2010 no alarm is given. 19/3/2010 ProMED-mail issues a report in English of cholera deaths in Tombua (Tombwa), southern Namibe, between 1/3/2010 and 3/3/2010. The cited source was Angop on 4/3/2010.
In this case BioCaster was more successful for English than for the multilingual system because a false spike of reports occluded subsequent true positives. In the case of the silver standard report on 19/3/2010, the cited English source was not detected but its Spanish translation was found a few days later - still much earlier than the ProMED-mail report.
The example is a relatively special case that illustrates an event that was not widely re-reported. The reports were made in English, Portuguese, French and Spanish from Angop. Externally, the 4/3/2010 article from Angop was republished in allafrica.com and africanseer.com on the 4th March. It was also referenced in a blog by the Namibia online community. 6
Automated health surveillance using text mining is not intended as a substitute for skilled human analysis but as these results show it does have the potential to reduce their information burden if informed choices are used to govern the selection of models.
Obvious improvements to the techniques described here could take place by modeling lower geographic granularity and reducing size differences between geo-units. More sophisticated approaches might incorporate proximity information between events or model how events propagate through news space.
A more subtle effect of the granularity restriction is that the models we presented do not allow us to follow what might be called ‘late warning’ signals. i.e. follow on events within the country’s borders. For this reason detecting events below the country level is desirable. Future work will need to concentrate on maximizing system sensitivity to overcome the fragmentation of the event distribution that occurs when we bucket events into smaller geographic units.
I greatly acknowledge the comments by the anonymous reviewers. Funding support was provided in part by the Japan Science and Technology Agency under the PRESTO programme.