We describe our system for Shared Task 1, which involves recognizing tweets containing adverse drug reaction mentions. We used a relatively standard CNN architecture coupled with task-specific features and pre-processing steps to achieve an F-score on the test set of 0.412, placing us as the third best-scoring team and 0.015 points beneath the best score. We observe empirically that the test dataset differs substantially from the training corpus; building models to explicitly account for this shift will likely be important to further improve results. We remove all characters except for those in the Basic Latin Set (U+0000 - U+007F) and replace @usernames, urls, numbers and dollar symbols with corresponding special tokens. We also add a space before and after all nonalphanumeric characters. We heuristically replace all drug mentions in sentences with a DRUG token using Drugbank database1 [2]. Finally, we split the sentence into tokens on whitespace and lemmatize each word using nltk [3]. As mentioned above, we use a CNN architecture, which may be viewed as inducing its own feature map from words. We augment this learned representation with manually crafted indicator features based on prior work by [4]2. These include features that encode information derived via synset expansions and use of an ADR Lexicon, those capturing SentiWord Score, topic based and structural features, and features based on cluster assignments and change phrases .
Briefly, our system comprises a standard CNN architecture [1] augmented with manually crafted features and an embedding projection layer. In this brief note, we describe these components in greater detail. 3.1
Embedding based NLP systems have been shown to benefit from pre-training on large volumes of unlabeled data [
However, if these embeddings are retrained using a small amount of labeled data, this can lead to severe overfitting
for the words that are seen in training. Moreover, word vectors corresponding to tokens not present in the small
training corpus will not be updated at all. Thus, rather than retraining or fine-tuning embeddings, we introduced a
transformation layer to project pre-trained embeddings into a task-specific space. This has the indirect effect that all
embeddings in pre-trained space are updated rather than only those in the training set. Instead of hoping the projection
directly learns a desired underlying mapping, we explicitly let it fit a residual mapping. This is inspired by the residual
layers used in deep convolutional networks [
W 0 = ReLU(W
P ) + W (1) Our intuition here is that it is easier to optimize the residual mapping than to optimize vector representations in the original or source embedding space. In the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping.
We use two sets of pre-trained word embeddings in this project: (i) Google News Dataset (300 dim, 3M words) [
Convolutional Neural Networks (CNN)s have been extensively used in many computer vision and, more recently,
NLP tasks. In NLP, CNNs were previously used successfully in sentence classification and sentiment analysis [
We also train a multilayer perceptron on the feature vectors that performs binary classification. The MLP comprises a single hidden layer and an output layer. We use a softmax activation to produce the output and we minimize cross entropy loss over the training dataset. 3.4
In this model, we concatenate the features returned by pooling layer of convolution network with the output of the hidden layer of feature network before passing them through the softmax layer.
For CNN models, we use filters of size 1-5 with 250 filters for each. We use a dropout layer [
We also used an ensemble of all methods for evaluation. To counter the class imbalance problem, rather than using majority voting, we labeled the data point as positive if at least 2 classifiers labeled it as positive.4 We also set the threshold for a tweet to be labeled as positive to 0.35. This setting reduced our precision but significantly improved recall.
3A ReLU takes an input and returns the original input if it is larger than 0, otherwise, it returns 0.
4This performed better than alternative ensembling approaches such as ‘stacking’ on the dev dataset. 3.6
The training data consists on 11293 tweets with 9330 negative samples and 965 positive samples. The test data consists of 9961 tweets (9190 negative, 771 positive). We will refer this particular train and test split as the original corpus. Because the training and test datasets were generated by sampling at different times with different drug names, we wanted to see whether the train and test sets reflected different input feature distributions. To assess this, we combined the train and test data and shuffled them together. We then randomly divided the total data into 2 parts with the same distribution of labels as given, i.e., we sampled 9330 negative and 965 positive data points for training and kept remaining for testing. We will call this train/test split as randomized.
We use 90/10 split of training data in both sets above for validation purposes. 4
Feature CNN (goog) + Feature CNN (tw) + Feature CNN (goog + EP) + Feature CNN (tw + EP) + Feature Ensemble We will mention following observations in regards to our results :
As can be clearly seen from the table, the models consistently and significantly improved when trained on randomised datasets. This suggests that our model could benefit from better normalization of texts as well as the use of external ontologies to map words to concepts like drugs, side effects, and so on. In this work, We do not explore the differences between training and test set underlying this observation in further detail. Models with the embedding projection outperform ones with no embedding projection by a small margin. All models have better precision than recall in the original train/test split.
Google Embeddings performed better than twitter embeddings, which we found somewhat surprising. 5
We proposed a hybrid CNN and feature based system to perform binary classification of tweets containing mentions of Adverse Drug Reactions (ADRs). We evaluated our system against the training and test dataset provided as part of the shared task. On using these datasets as they were intended, we achieved an overall F1 score of 0.412. We also observe that combining the given datasets and dividing it into training and test set randomly significantly improved the performance of our classifiers. This suggests a shift in the data distributions. In future work, we will further study the reasons behind this shift and how they can be rectified to generate more precise classifiers. [1] Yoon Kim. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882, 2014.