We present the results from our participation in the 2nd Social Media Mining for Health Applications Shared Task Task 2. The goal of this task is to develop systems capable of recognizing mentions of medication intake in Twitter. Our best performing classification system is an ensemble of neural networks with features generated by word- and character-level convolutional neural network channels and a condensed weighted bag-of-words representation. A relatively strong performance is achieved, with an F-score of 66.3 according to the official evaluation, resulting in the 5th place in the shared task with performance close to the best systems created by other participating teams.
The organizers provided a training dataset with manually assigned labels (intake, possible intake, non-intake) for each tweet. The intake class is defined as clearly expressing a personal intake of medication, whereas the possible intake is more ambiguous, yet still suggesting an intake by the tweet writer. The non-intake class includes the rest of the tweets, all of which include a mention of a drug, but refer to an intake by another person or discuss the drugs in general. Approximately 50% of the data belongs to the non-intake class, whereas the intake and possible intake classes constitute 19% and 31% of the data, respectively.
Due to the data sharing restrictions of Twitter, the organizers only provided the IDs of the tweets instead of their actual content. Since we started investigating this task much later than the data was released, we were only able to obtain the contents for 7444 tweets out of the total 8000 annotated tweets as some of the content had been already removed by the Twitter users, i.e. we had 7% less training data than teams who were involved in the shared task since the very *: these authors have contributed equally. beginning. The organizers also provided a separate development dataset, which consists of 2260 annotated tweets, of which we also lost roughly the same proportion. In the Results and Discussion section we evaluate the impact of the lost data in more detail.
For our baseline approach we form term frequency–inverse document frequency (TF–IDF) weighted sparse bag-ofwords (BOW) representations for all given tweets1. These representations are not only constructed for single tokens, but also token bigrams, trigrams and character n-grams of length 1 to 4. These representations are then fed as features to a linear SVM classifier2. The regularization parameter is selected to optimize the micro-averaged F-score of intake and possible intake classes, the official evaluation metric, on the development set. For the final submission in the shared task we merge the training and development sets and train the system on the combined dataset. As the sparse representations are not able to generalize well to unseen vocabulary, we also test various NN approaches on this task. The final system is based on an ensemble of convolutional neural networks (CNNs)3 and utilizes word and character information.
Each tweet is represented as two separate sequences: words and characters, both of which are processed with separate convolutional channels. Each element in these sequences is represented with a latent feature vector, i.e. an embedding. The word embeddings are initialized using word2vec4 trained with approximately 1 billion drug related tweets as provided by Sarker and Gonzalez5. We also tested GloVe vectors trained on 2B general domain tweets6, but these experiments resulted in a decreased performance. The character embeddings are initialized randomly, but the network is allowed to backpropagate to both word and character embeddings.
The convolutional kernels are applied on the aforementioned two sequences using sliding windows. The outputs are subsequently max-pooled and concatenated. The concatenated vectors are further fed through two densely connected layers, the latter having the output dimensionality corresponding to the number of labels in the data set with softmax activation.
In addition to the convolutional layers we utilize the same TF–IDF weighted sparse vector representations as in the baseline method. As these representations have dimensionality in the order of hundreds of thousands, we first densify the representations to 4000 dimensional vectors using truncated singular-value decomposition (SVD)7. These vectors are concatenated alongside the CNN outputs. This dimensionality reduction is performed mainly due to computational reasons since the approach was prototyped on a consumer grade GPU with limited amount of memory. Projecting the sparse vectors to 4K dimensions preserves 74% of the variance in the data and may have caused a minor performance loss.
The network is trained on the official training data using the Nadam optimization algorithm. The network is regularized with dropout rate of 0.2 after the first dense layer, no explicit regularization is applied on the convolutional part of the network. The training is stopped once the performance on the development set is no longer improving, measured with the official evaluation metric. Table 1 shows the comprehensive list of used hyperparameters.
Character embedding dimensionality Word embedding dimensionality Character CNN, number of filters per window size Character CNN, window sizes Word CNN, number of filters per window size Word CNN, window sizes Dimensionality of first dense layer Dropout rate Activation functions
Optimal value
25
400
50
[
Tested Values [25,50,75,100]
pre-trained [50,100,150,200]
[
[0,0.2,0.5] [ReLU, tanh, sigmoid]
Training the network on this dataset resulted in relatively large variance in the measured performance, caused by the random initialization of the weights. Thus we stabilize the system by training 15 networks, all identical apart from the initial (random) weights. We then select the optimal subset of these networks, as measured on the development set, for the final system where the final predictions are created by summing the confidences of all selected networks and choosing the label with the highest overall confidence. The final system included a subset of 6 neural networks out of the 15. We note that this approach may potentially overfit on the development set.
Other NN architectures experimented and tested during this shared task include various versions of BiLSTM and attention based networks8, 9 but none of these experiments resulted in better performance than the CNN architecture described in detail. However, due to the time limits of the shared task, we cannot reject the possibility of these approaches being competitive as well.
We also experimented with a way of (pre-)tuning the utilized word embeddings to this specific classification task in an attempt to give the word-level CNN a better starting point for the training. This was done using the principles underlying the random indexing (RI)10 method. Unique index vectors are first assigned to each of the three classes (intake, possible intake and non-intake), and empty context vectors are assigned to each word in the data set. When traversing the training set, each word, in each tweet, have the index vector associated with the tweet’s class added to their context vector. After training, the resulting word context vectors are normalized to unit length and summed with the corresponding word embeddings/vectors generated using word2vec5 (also normalized to unit length). To make the signal provided by the RI approach have a modest impact on the conjoint vectors, these vectors are first multiplied with a weight of 0.3. However, the described approach did not seem to result in a positive performance impact, compared to using the original word2vec generated embeddings.
We also tested the potential benefits of including part-of-speech (POS) tags, which were produced using the Twitter NLP toolkit11. The sequences of POS tags were treated in similar fashion to the word and character sequences. Although the benefits of POS tagging are intuitive as for instance verbs in past tense are twice as common in the intake class as in the possible intake, we did not see any increase in the performance when POS tags were utilized.
We measure the performance of our systems using micro-averaged F-score of intake and possible intake classes, following the official evaluation, and conduct all our experiments on the official development set. However, the reported results are not directly comparable with other systems as we only had access to a subset of the original data (see the Data section). The results on the test set are as reported by the organizers and thus comparable to other systems.
The overall performance of our baseline (i.e. SVM) and CNN-based systems are relatively strong, resulting in Fscores of 69.6 and 72.7 on the development set respectively (see Table 2). We suspect the main advantage of the CNN approach to be the generalizability of the word embeddings, which leads to the 3.1pp improvement in F-score. We also briefly tested a nonlinear multilayer perceptron with the same BOW features as used in the SVM, which led to a slight improvement over using the linear SVM model, but was not able to outperform our CNN-based system. Thus the model complexity alone does not explain the performance difference between the SVM and CNN approaches. An unexpected observation is that the intake class seems to be harder to predict than the possible intake, although eyeballing the data suggests otherwise and the annotation guidelines provide more precise definition for the intake class. Also for the CNN-based system it seems that the precision and recall are rather well balanced, thus no performance improvements could have been gained through further fine tuning of these metrics.
The test set results follow the same patterns as the development set evaluation: SVM and CNN systems reach F-scores of 64.2 and 66.3, respectively. Thus it seems that either the test set is somewhat harder than the development set or both of the systems are overfitting equally on the development set, even though the implemented ensemble system with CNNs could have caused greater overfitting. According to the official evaluation, our best system loses to the winning system by 3pp in F-score, the difference being roughly the same in both precision and recall. This places our system in the 5th position in the shared task.
By inspecting the confusion matrices it can be concluded that our classifiers tend to confuse intake class with both SVM CNN
Possible Intake Overall Intake Possible Intake Overall Overall possible intake and non-intake classes equally often, whereas the possible intake is more often confused with the non-intake class.
As we only had access to a partial training data, we try to estimate how much the performance of the systems could have been improved with additional data. To accomplish this, we train the CNN-based system with different subsets of the training data, starting from 5K training examples and incrementally increasing the size in steps of 300 up to the whole training data available to us. After every increment we evaluate the system’s performance on the development set. To reduce the variance caused by different initial random weights, we train 5 networks with each subset of the training data, and calculate the mean performance for each subset. Fitting a linear regression on the resulting measurements shows that in this region, the learning curve is fairly linear and decent performance improvements can be gained by adding more training data. Assuming a performance increase equal to the slope of the fitted regression line, having the full training dataset would have increased our performance by 0.7pp in F-score, placing our system close to the top 3 teams in the shared task.
As most approaches we tested, as well as the systems created by other teams, resulted roughly in the same performance level, we wanted to assess what would be a theoretical performance limit for this task. To this end, we manually annotated a random subset of 100 tweets from the development set and evaluated the annotations against the gold standard. Surprisingly our manual annotations reached only an F-score of 59.3, notably lower than the developed systems or what the official inter-annotator agreement would suggest12. This indicates that the task is complex even for humans and deep understanding of the annotation guidelines is required for high quality annotations. We have shown that strong results in detecting tweets describing personal medication intake can be achieved using convolutional neural networks and word embeddings. However, more traditional methods relying on bag-of-words features and linear classifiers also result in competitive performance. Considering that such a system can be implemented in less than an hour with the existing tools and libraries, and is easily interpretable, the simpler methods may be a more practical choice in many use cases.
Since the amount of training data for this task is fairly limited, we plan to explore various approaches for pretraining NN classifiers as a future task. The goal here is to find a suitable proxy task related to the domain for initializing the network before the actual training. Such a task could be, for instance, sentiment detection as many of the tweets expressing drug intake also express a certain sentiment about the condition of the user or the effects of the drug.
This work was supported by ATT Tieto ka¨ytto¨o¨n grant and Tekes – Ra¨a¨ta¨li project (no. 644/31/2015). Computational resources were provided by CSC - IT Center For Science Ltd., Espoo, Finland.