In response to the challenges set forth by the 2nd Social Media Mining for Health Applications Shared Task 2017, we describe a framework to automatically detect Twitter tweets with mentioned adverse drug reactions (ADRs). We used a dataset with 10822 annotated tweets provided by the event organizers to develop a framework comprised of a preprocessing module, 6 feature extraction modules, and one predictive modeling module with 5 models (K2 Bayesian network, naïve Bayes, decision tree, random forest, and support vector machines). To evaluate our framework, we employed a blind test dataset with 9961 tweets provided by the event organizer. The area under the ROC curve (AUC) from the K2 Bayesian network model was 0.74 (95% C.I. 0,721-0.759) and F-Measure ranged from 0.339 to 0.342. The described framework in this paper demonstrated a potential public health surveillance tool for ADR surveillance from Twitter tweets.
The training dataset provided by SMM4H 2017 originally comprised of total 15,667 tweet identifiers with annotated
ADR results from two batches: each with 10,822 and 4845 identifiers respectively; after employing the python script
provided by the event organizer, we only retrieved 10,281 tweets for the training dataset. The blind test dataset
released 7 days prior to the competition deadline comprised of 9,961 tweets. All the datasets were annotated and
provided by the event organizer.
Figure 1 summarizes our proposed framework, which contained a data preprocessing module, and 6 feature extraction
modules, and one predictive modeling module. We first preprocessed each tweet in the training dataset in 7 steps: (
We employed six feature extraction modules: N-grams, term frequency-inverse document frequency (TF-IDF), word embeddings, and sentiment statistics.
We used unigram (single word), bi-gram (two words) and tri-gram (three words) as features for predictive modeling. For example, single words like “sick” and “weight” belong to unigram and “Cipro pills” containing two words is a bigram.
II.2.2 Term Frequency-Inverse Document Frequency TF-IDF score measures how important a term, e.g., word, is to a document (tweet) in a corpus (training dataset). The following equation defines TF-IDF.
, , = , ∗ log
+ 1 , + 1 where TF(t,d) denotes the term frequency (number of times that a term t appears in a tweet d), |D| is the total number of tweets, and DF(t, D) is the document frequency (number of tweets that contains the term t in all tweets D). Instead of using words in the above equation, we used hashed terms for computational efficiency. We used the vector of the hashed terms from each tweet’s as a feature vector.
II.2.3 Word Embeddings We employed the Apache Spark Word2Vec7 to obtain a word embeddings matrix from the training dataset. We used Skip-gram model with softmax activation function in a neuron. The Skip-gram model learns word vector representations from sentences by maximizing the average log-likelihood of a word wt in a sentence based on conditional independence assumption as shown below:
log 234 2 , 259 4576 where k is the size of a pre-defined training window, T is the total number of words within a sentence, and the conditional probability p(<|4) is determined based on the softmax model.
For each tweet, we computed an averaged word vector based on the word embeddings matrix, which becomes a feature vector.
II.2.4 Sentiment statistics We employed Stanford NLP toolkit8,9 to annotate each sentence's sentiment level as one of 5 levels in a tweet: very negative, negative, neutral, positive or very positive. From the annotation, we generated 6 sentiment features for each tweet: 1) the count of negative sentences annotated with negative and very negative sentiment levels, 2) the ratio between the numbers of negative sentences and all sentences, 3) the count of sentences annotated with neutral sentiment level, 4) the ratio between the numbers of neutral sentences and all sentences, 5) the count of positive sentences annotated with positive and very positive sentiment levels, 6) the ratio between the numbers of positive sentences and all sentences.
II.2.5 cTAKES findings
We employed cTAKES10, a medical natural language processing application, to extract symptoms and findings from clinical narratives. We further customized cTAKES with 2017 US Edition of SNOMED CT11 as a clinical-term lookup dictionary. We identified 10 annotation types such as drug, disorder, finding, procedure, lab, etc. We employed ADRMine12, a social-media-specific named entity recognition (NER) application, to identify ADR mentions in a tweet. Any non-negative ADR mentions in the training dataset were included in our feature set. We used five machine learning algorithms to build predictive models and validated the models through nested 10-fold cross validation. All the features were obtained from the feature modules stated in Section II.2. For each fold, we applied information gain first followed by correlation-based feature selection. We then used K2, Naïve Bayes (NB), decision tree (DT), random forest (RF), and SVM algorithms to build predictive models.
We identified three different methods to determine probability thresholds. The first method is cross-validation threshold, which used the average threshold that optimized F1-score during cross-validation testing. The second method is prevalence threshold, which calculated the threshold that would produce the same class prevalence observed in the training set. The third method is training threshold, which calculated the threshold that optimized F1-score in the entire training dataset.
The evaluation metrics for predictive models are F-measure (also known as F1-measure), precision, recall, and the area under the ROC curve (AUC).
III.
Compared to other models, the K2 model had the highest average AUC (0.844) and F1-score (0.469) across 10 test folds within the 10-fold cross validation (CV). Table 1 lists predictive results from the 5 algorithms within the 10fold CV. DT had the best recall (0.641) and SVM (0.62) had the best precision. 0.581 0.269 The final K2 model comprised 26 nodes from unigram (9 nodes), bi-gram (2 nodes), tri-gram (1 node), cTAKES (8 nodes), word embeddings (1 node), and ADR tags (5 nodes).
Based on Table 1, we used K2 model for the final test (blind) dataset provided by the organizer released a week before the predictive result submission. Table 2 summarizes the predictive performance in the test dataset. The AUC of the K2 model was 0.74 (95% C.I. 0,721-0.759). The K2 model with the training threshold method had the best F-measure (0.342) and the prevalence threshold method had the best recall (0.394).
Algorithm K2-CV
AUC 0.74 K2-Prevalence 0.74 K2-Training
0.74 IV.
F-Measure Precision Recall 0.341 0.339 0.342 0.333 0.298 0.336 0.35 0.394 0.348 In this study, we built and evaluated a framework to identify individual tweets with mentioned ADR. The framework comprises of preprocessing module, feature extraction modules, and predictive modeling module. The results demonstrated a potential public health surveillance tool for ADR surveillance from Twitter tweets.