This paper describes the systems we developed for all three tasks of the 2nd Social Media Mining for Health Applications Shared Task at AMIA 2017. The first task focuses on identifying the Twitter posts containing mentions of adverse drug reactions (ADR). The second task focuses on automatic classification of medication intake messages (among those containing drug names) on Twitter. The last task is on identifying the MEDDRA Preferred Term (PT) code for the ADR mentions expressed in casual social text. We propose convolutional neural network (CNN) and traditional linear model (TLM) approaches for the first and second tasks and use hierarchical long short-term memory (LSTM) recurrent neural networks for the third task. Among 11 teams our systems ranked 4th in ADR detection with F-score 40.2% and 2nd in classifying medication intake messages with F-score 68.9%. For the MEDDRA PT code identification, we obtained an accuracy of 87.2%, which is nearly 1% lower than the top score from the only other team that participated.
For tasks 1 and 2, we employ traditional methods (e.g. SVMs) and recent deep learning methods (e.g. CNNs) as well as their ensembles as detailed in Section 2.1. For task 3, we use a hierarchical character-LSTM model which is detailed in Section 2.2. 2.1.
For tasks 1 and 2, we used both TLMs, specifically linear SVMs and logistic regression, and deep nets specifically CNNs. In the first task, we simply averaged the probability estimates from each TLM classifier with and without CNN ensemble. The features used in TLMs are itemized below.
Uni/bi grams: Counts and real values of uni/bi-gram features in the tweet via countvectorizer and tfidfvectorizer,
respectively, from Scikit Learn machine learning package [
Sentiment lexicons (11 features): Four count features from the post based on the positive and negative counts of
unigrams and bigrams using the NRC Canada emoticon lexicon1, four additional sentiment score aggregation
features corresponding to the count features, and overall sentiment score of uni-grams/bigrams/uni+bi grams.
Word embeddings (400 features): The average 400 dimensional vector of all the word vectors of a post where
the word vectors (2 R400) are obtained from a pre-trained Twitter word2vec model [
ADR lexicon [
For task 2 only – handcrafted lexical pairs of drug mentions preceded by pronouns (6 features): The count of first, second, and third personal pronouns with and without negation followed, with potentially other intermediate words, by a drug mention2. (e.g., “I did a line of cocaine”) For the CNN models, each tweet is passed to the model as a sequence of words vectors, [w1; w2; :::; wn], where n is the number of words in the tweet. We begin by concatenating each window spanning k words, x j k+1jj : : : jjx j, into a local context vector c j 2 Rkdemb where demb is the dimensionality of the word vectors. Intuitively, CNNs extract informative n-grams from text and n-grams are extracted with the use of convolutional filters (CFs). We define the CFs as Wk 2 Rq kdemb , where q is the number of feature maps generated using filter width k. Next, using a non-linear function f , we convolve over each context vector,
cˆ j = f (Wc j + b); where b 2 Rq. Given the convolved context vectors [cˆ1; cˆ2; : : : ; cˆn k+1], we map them into a fixed size vector using max-over-time pooling
mk = [cˆ1max; cˆ2max; : : : ; cˆqmax]; where cˆ mjax represents the max value across the j-th feature map such that cˆ mjax = max(cˆ1j ; cˆ2j ; : : : ; cˆnj k+1). To improve our model, we use convolutional filters of di erent widths. With filters spanning a di erent number of words (k1; k2; : : : ; kp), we generate multiple sentence representations mk1 ; mk2 ; : : : ; mkp , where p is the total number of filter widths we use.
When datasets contain many noisy features, it can be beneficial to use a simpler model. Simpler models are less prone to overfitting to the noise in the dataset. We augment the features generated from the CNN with a simple average of all the word vectors in a given tweet
mbow = 1n Xi=n1 wi:
Now we have p + 1 feature representations of the final tweet [mk1 ; mk2 ; : : : ; mkp ; mbow]. Prior work using CNNs [
exp(e j;i) Pkp=+11 exp(e j;k)
; where e j;i = vTj tanh(Wami); such that Wa 2 Rt q, v j 2 Rt, and j;i 2 [0; 1]. Intuitively, j;i represents the importance we give the feature representation mi. Next, we merge all feature representations into a matrix M 2 R(p+1) q. Likewise, given s total attentions, each attention weight is combined to form an attention matrix A 2 Rs (p+1). Here, the j-th row of A represents the importance of each mi with respect to the attention weights v j. Finally, we represent each tweet as the weighted sum of the feature representations according to each of the attention weight vectors as where vec represents the matrix vectorization operation and h 2 Rsq. Lastly, h is passed to a final fully-connected softmax layer. 1http://saifmohammad.com/WebPages/AccessResource.htm 2https://www.drugs.com/drug_information.html h = vec(AM); 2.2.
The deep model we propose for task 3 realizes a hierarchical composition in which an example phrase is segmented into N constituent words and each word is treated as a sequence of characters. In our formulation, the word at position i 2 [1; N] is of character length T i. Herein, we formulate the model composition from the bottom up. At the character level, word representations are composed using a forward LSTM over each character and its corresponding character class. The former is a case-insensitive character embedding lookup and the latter is a separate label embedding lookup indicating the class of character: lowercase letter, uppercase letter, punctuation, or other. We denote ci;t and zi;t as the t-th character and class embedding respectively of the i-th word in the sentence for t 2 [1; T i]. For a word at position i, we feed its character embeddings to a forward-LSTM layer with 200 output units such that !g i;t = CLS T M(ci;tkzi;t) for t = 1; : : : ; T i where k is the vector concatenation operation, CLS T M is an LSTM unit composition at the character level, and !g i;t 2 R200 is the output at timestep t. The output state at the last step, or !g i;Ti , encodes the left-to-right context accumulating at the last character and is used to represent the word at position i. Next, we deploy a bi-directional LSTM layer at the word level which is useful for capturing the contextual information of the phrase with respect to each word. Concretely, !h i = W LS T M!(!g i;Ti ); h i = W LS T M (!g i;Ti ); hi = !h ik h i for i = 1; : : : ; N where !h i; h i 2 R200, hi 2 R400, and W LS T M!; W LS T M are LSTM units composing words in the forward and backward direction respectively. We then perform a max-pooling operation over all hi vectors to produce the feature vector hˆ = [h1max; : : : ; h2mdax] where h mjax = max(h1j ; : : : ; h Nj ). We use a fully-connected layer output layer of unit size m, where m corresponds to the number of MEDDRA codes in the label space. The output is computed as q = Wq hˆ + bq p j =
eq j
P mj=1 eq j such that q 2 Rm and Wq and bq are network parameters. In order to get a categorical distribution, we apply a softmax layer to the vector q such that where p j 2 [0; 1] is the probability estimate of the label at index j.
Additional Training Data In addition to the training set provided for this task, we also experiment with using
additional external datasets for training. Namely, we use the TwADR-L [
Model Configuration We train the model with 80% of the training data and use the remaining 20% for validation. We train for 40 epochs with a batch size of 8 and learning rate of 0.01. Here we use the RMSprop optimize with an exponential decay rate of 0.95. The character embeddings are of size 32 and the character class embeddings are of size 8. We apply dropout regularization at a probability of 50% at the feature vector layer. To make the final prediction, we performed model averaging using an ensemble of 10 such models each with a di erent random parameter initialization and random validation split. 3.1.
was slightly better than TLM-ensemble and CNN when considered separately, but CNN-Att was worse o when TLM-ensemble and CNN were combined using model averaging. Our ensemble model has the top precision score among all teams and shows the potential of the ensemble approaches. However, our recall is significantly less than the best performer where it was over 48%. We will further investigate the discrepancies between train and test set performances of various models. Our initial assessment is that the attention model is able to more e ectively weight what di erent CFs are capturing from each tweet. To verify this, we will examine the “popular” n-grams of filters (based on values of cˆ j in Section 2.1) that are consistently being weighted above others by the attention mechanism. In turn, these n-grams can be used as additional features in the regular CNN or TML models to potentially improve their performances. We will need to experiment with these additional approaches to improve our recall without major compromises in precision. Since each task allowed submission of only three models, the ones we submitted may not have been the best compared with additional models we built as part of our participation in the challenge. Therefore, we did some new experiments using unsubmitted models and found newer better performing ensembles on task 1 and 2 as shown in Table 3. The results from our CNN and CNN-Att models are from averaging ensembles of ten models each using both stratified and random 80%–20% splits. We did this because the test dataset distribution may vary from training data, and we believed stratified split may cause overfitting. Therefore, we felt that with the help of random splits it may help our model’s ability to handle test data with a di erent distribution. In this particular case, the test data distribution is similar to that of the training data; so when we only consider the use of stratified splitting to tune the parameters, our results improved. In task 1, we found the ensemble model with CNN-Att and CNN with logistic regression had almost 1% improvement in F1 score. In task 2, we found CNN-Att trained with stratified splitting has an F1 score that is 0.4% higher than that of our submitted model and this score is equal to winning team’s performance.
We disclose our results for task 3 in Table 4. For our initial experiments, we performed testing on 30 di erent runs each evaluating on a di erent random held-out development set; the results are averaged and recorded in the second column of the table. We submitted only two models and these are marked accordingly in the table. Our first submission, the Hierarchical Char-LSTM model as described in Section 2.2, performed best at 87:4% on the validation set which is consistent with its performance on the actual test at 87:2%. Our second submission, which is the same as the first except it is trained using additional external datasets performed reasonably but did not generalize to the test set as initially hoped.
The flat character-based models performed poorly on both the validation and the test set but nevertheless saw a
slight increase on the test set. The Hierarchical Char-CNN model results were underwhelming on the validation;
however, on the test set it is observed to outperform even the Hierarchical Char-LSTM models which is unexpected.
Furthermore, we experimented with appending pretrained word embeddings (trained on the Twitter corpus [
Our work is primarily supported by the National Library of Medicine through grant R21LM012274 and the National Cancer Institute through grant R21CA218231.