Micro-blogging services such as Twitter offer the potential to crowdsource epidemics in realtime. However, Twitter posts ('tweets') are often ambiguous and reactive to media trends. In order to ground user messages in epidemic response we focused on tracking self-protective behaviour such as avoiding public gatherings or increased sanitation as the basis for further risk analysis. In initial experiments on influenza tracking we report results for unigrams, bigrams and regular expressions employed in two supervised classifiers (SVM and Naive Bayes) to classify tweets into 4 self-reported protective behaviour categories plus a selfreported diagnosis. We report moderately strong Spearman's Rho correlation for the classifiers against WHO/NREVSS laboratory data for A(H1N1) in the USA during the 2009-2010 influenza season.
Rising awareness of infectious disease
outbreaks and the high costs of extending
traditional sensor networks means that
we have an opportunity to harness new
forms of social communication for
crisis surveillance. The trend is already
underway with automatic map
generation from Twitter reports for earthquakes
and typhoons
A small but growing number of early
warning systems have already developed
to mine event information from low cost
Web sources mainly focussing on edited
newswire reports (see
Recent studies on alerting from
newswire reports
In micro-blogging services such as
Twitter, users describe their experiences
directly in near-real time in short 140
character tweets. As of April 2010 it was
estimated that Twitter had approximately
106 million registered users with 300,000
new users being added each month.
Despite their potential coverage, timeliness
and low overhead, tweets present their
own unique challenges: pre-diagnostic
unedited reports mean that there is a large
trust issue to resolve within the
modeling technique; also social media can
reflect a high degree of reactivity to risk
perception as seen during the H1N1
pandemic in 2009 - redistributing links or
requests for information rather than
generating user experience. To a degree this
reflects newswire coverage and the amount
of uncertainty readers feel. Re-tweets in
themselves may provide useful signal but
their role has yet to be quantified.
Despite these obvious challenges we believe
there is potential for using very short
messages to detect epidemic trends, as
hinted at by the success of Google Flu
Trends
In order to do this we propose to
employ aberration detection for
detecting sharp rises in the features that
signal epidemics. A precursor to this is in
identifying reliable features themselves.
In this study we started by looking at
precautionary actions as identified by
Jones and Salathe´ in their behaviour
response survey
Recently a number of studies have
appeared looking at the effectiveness of
search queries and social media.
Nevertheless challenges in
interpreting query and social networking data
remain.
Taking Jones and Salathe´’s behaviour responses as a starting point we surveyed potential messages in Twitter in relation to H1N1 influenza topics. From an initial group of thirteen categories we decided, due to low frequency counts, to conflate several into a final grouping of four. e.g. avoiding people who cough/sneeze, avoiding large gatherings of people, avoiding school/work and avoiding travel to infected areas were joined into a general ‘avoidance behaviour’ category. To this we added a final category for direct reporting of influenza. The final list of categories is: (A) Avoidance Behavior - behaviours which avoid agents thought to be at risk of infection; (I) Increased sanitation - sanitation measures to promote individual health and prevent infection; (P) Seeking pharmaceutical intervention seeking clinical advice or using medicine or vaccines; (W) Wearing a mask; and (S) Self reported diagnosis - reporting that one has influenza.
As expected there are a number of caveats to each of these broad classes. We list up only a representative sample here: (1) A message is only tagged positive if the user or a close family member is the subject of the tweet; (2) If the message indicates that the action is hypothetical then the classification is negative; (3) The time of the reported event should be within one week of the current time; (4) Messages can belong to more than one category. Examples of (anonymized) messages are shown in Table 1.
At a practical level the problem of identifying self protection messages can be characterised as classifying very biased data. In order to handle this we adopted two stages of filtering. The first stage used a bag of 7 keywords to select tweets on topics related to influenza (flu, influenza, H1N1, H5N1, swine flu, pandemic, bird flu). For 1st March 2010 to April 30th 2010 this resulted in a pool of about 225,000 tweets. This first stage of filtering was also designed to reduce the ambiguity of keywords such as ‘fever’ and ‘cough’ which occur in a wide variety of contexts.
The second stage used hand built patterns to select a total of 14,508 tweets. From these we randomly chose 7,412 tweets spread across the five classes. All duplicates were removed leaving 5,283 messages and the resulting data was then classified by hand using a single annotator as detailed in Table 2. Results for mean character length and standard deviation showed no category-specific trend except to illustrate the wide variety of message lengths.
In order to test the stability of the annotation scheme and our assumptions about its reproducibility we calculated kappa for 2,116 messages balanced across all the classes. For this another annotator was chosen who did not take part in the creation of the guidelines and was not a co-investigator in this study. The simple agreement ratio was 0.88 (the total number of matched class assignments divided by the total number of messages). Kappa was calculated as κ = (pA − pE)/(1 − pE), where pA was 0.88 and pE was 0.12. κ was then found to be 0.86. Both results reveal a high level of agreement in the annotation scheme and give us confidence to move ahead with automated classification. 3.2
We employed two widely used
classification models implemented in the Weka
Toolkit
SRL rules were built from a held out set of tweets not used in training. Rules consist of string literals, skip expressions, word lists, named entity classes and guard expressions for limiting the scope of matched entities. Rule building took approximately 10 hours of work. The rule book contains specialised synonym sets to recognize common and slang terms for medicines (e.g. shot, vaccine, drug, tamiflu, jab, medicine, vacc), physicians (e.g. doctor, doc, dr, physician) and other key domain entities. Verb lists are maintained for specialized lexical classes such as prescribe (e.g. prescribe, perscribe∗). Lists are also built for pronouns, common temporal adverbs, modal verbs and negations. Special rules were built to recognize past events. The exceptional class was I (increased sanitation) where we were not able to identify enough examples with confidence to build meaningful rules by hand. In this case only unigrams and bigrams were used to train the classifiers.
We found that the language used in tweets to express user’s behaviour is very diverse and idiosyncratic so it is challenging to achieve a high degree of coverage in the rules with surety. With this in mind we combined features from the rules with unigrams and bigrams. If a rule matched a tweet its feature value was set to 1, otherwise to 0. 4
All test runs used 10-fold cross validation on each of the 5 test sets. We calculated recall, precision and F-score performance for each category. As we can see in Tables 3 and 4, SVM overall performs better on all categories except for W (wearing a mask). Both model’s performance generally follows the amount of training data except for S (Self diagnosis) where the F-score is slightly lower than the trend in other classes despite large numbers of examples. The overall trend for Naive Bayes is to have stronger recall than precision whereas for SVM precision is generally higher than recall.
The results suggest that our SRL rule book seemed to offer substantial benefits when combined with unigrams but less certain improvements when combined with unigrams plus bigrams. Looking slightly deeper into the results we found a correlation between message length and classification accuracy in Naive Bayes and SVM. For Naive Bayes, whilst the length of messages didn’t seem to make much difference to the false negative rate which remained constant at about 0.2 to 0.25 on messages in the length range of 34 to 144 characters, it impacted to a greater degree on false positives (0.23 on shorter messages of length 34 to 56 down to 0.08 for messages of length 122 to 144). For SVM there appeared to be a general reduction in both false positives and false negatives as message length increased.
As expected, frequent misspellings, abbreviated word forms, slang and lack of punctuation complicated the classification task. Missing auxiliary verbs and articles need to be compensated for within the SRL rules in order to ensure successful matching.
Potentially issues of duplication through re-tweeting still remain which we have not modelled in this study. Clearly we should have more confidence in the alerting model if a larger number of independent sources report an event at the same time. This will form part of our future work. Future work will also need to ensure that the classification model remains relevant over time as the data content in tweets shifts.
In order to provide a proof of concept
we operationalised the classifiers and ran
them on a corpus of Twitter data called
the Edinburgh Corpus
We applied the same keyword
filtering method used on the Edinburgh
corpus for the first set of experiments and
obtained 52,193 tweets for the period of
study. Following this we applied the
SVM UNI model and then compared the
week by week volumes against
laboratory results for weeks 47 to 5 of the
20092010 influenza season in the USA
To measure correlation we calculated the Spearman’s Rho 1 between counts of positive messages in each class and the CDC laboratory data for A(H1N1). Table 6 shows moderately strong correlations. The strongest correlation appeared when A,I and P were combined. Besides W which failed to provide any data, the weakest evidence came from Increased saniation (I). Differences could be due to (a) the global geographic coverage of 1See http://en.wikipedia.org/wiki/Spearmans´rank correlation coefficient tweets in our collection; and (b) the syndromes covered in our self protection behaviour and self reporting messages are wider than A(H1N1) and could actually be other diseases such as common colds, strep throat, adenovirus infection and so on.
Drill down analysis reveals that we still need to do more to remove false positives by strengthening the linguistic features within the limits of the 140 character length. Examples of false positives include interogative sentences, hypothetical sentences, reports on events that took place in the distant past, comments on influenza advice from others, etc. 6
In this paper we have made the first steps towards classifying Twitter messages according to self reported risk behaviour. The results have shown moderately strong correlation with CDC A(H1N1) data but we still need to make further progress in order to achieve the high degrees of correlation reported bewteen Google Flu trends and sentinal influenza data. The next step will be to extend our training data, strengthen the linguistic features and see if we can use these signals to detect emerging disease outbreaks. It was shown in Jones and Salathe´ that after an initial peak in levels of risk concern, anxiety faded once the immediate threat of the A(H1N1) pandemic had passed. In follow up work we intend to look at how closely these signals track epidemic case data.
We also believe that the signals we have modelled make them applicable to a wide range of diseases within the respiratory syndrome and we intend to explore how these features can be used to detect diseases other than influenza.
Besides disaster alerting, results from
analysis of behavioural responses may
also help in the future to evaluate the
success of official prevention campaigns.
For example, it is known that little
notice was taken of antiviral therapies,
goggle or mask wearing advice in the
Netherlands after the Avian Influenza epidemic
was introduced to Europe
We gratefully acknowledge support from the National Institute of Informatics Global Liason Office for internship funding for this study. The study was conceived and directed by NC, corpus collection and analysis was done by NM and the machine learning experiments were done by NS.