This paper describes Infosys's participation in the “2nd Social Media Mining for Health Applications Shared Task at AMIA, 2017, Task 2”. Mining social media messages for health and drug related information has received significant interest in pharmacovigilance research. This task targets at developing automated classification models for identifying tweets containing descriptions of personal intake of medicines. Towards this objective we train a stacked ensemble of shallow convolutional neural network (CNN) models on an annotated dataset provided by the organizers. We use random search for tuning the hyper-parameters of the CNN and submit an ensemble of best models for the prediction task. Our system secured first place among 9 teams, with a micro-averaged F-score of 0.693.
Deep learning systems have recently shown to achieve top results in shared tasks related to natural language processing on tweets4. Historically, ensemble learning has proved to be very effective in most of the machine learning tasks including the famous winning solution of the Netflix Prize5. Ensemble models can offer diversity over training data splits, random initialization of the same model or model architectures, and a combination of multiple average or low performing learners to produce a robust and high-performing learning model. A convolutional neural network (CNN)6 is a deep learning architecture, that has shown strong performance on sentence-level text classification. Even fairly simple CNNs evaluate at a level of or even better than more complex deep learning architectures7. Therefore, we designed and implemented a stacked ensemble of shallow convolutional neural networks (Figure 1) for solving the classification task presented in this paper. The main intuition behind developing such an ensemble was to take the best of all worlds. Next, we explain stacked ensemble of CNNs. In order to get the best results from any classification model, hyperparameter tuning is a key step and CNNs are no exception. While the existing literature offers guidance on practical design decisions, identifying the best hyperparameters of a CNN requires experimentation. This requires evaluating trained models on a cross-validation dataset and choosing the best hyperparameters manually that produce best results. Automated hyperparameter searching methods like grid search, random search, and Bayesian optimization methods are also popularly used. In our presented system we use random search8, to explore the hyperparameters of a shallow CNN architecture and form an ensemble of the best models, which we refer to as a stacked ensemble. Next, we share the detailed output and analysis of our experiments.
The organizers provided 8000 annotated tweets as a training dataset and 2260 additional tweets as development dataset. We collected the tweets using the script provided along with the dataset, by querying Twitter’s Class 1 Class 2 Class 3 Total API. However, we could not collect all the tweets as Train 1847 3027 4789 9663 some of them were not available at the moment when we executed our collection process. Later, the Test 1731 2697 3085 7513 organizers also shared the test dataset, that was used for calculating the final scores of the submitted Table 1. Shared task data distribution. Class 1, 2 models. A distribution of tweets provided for each and 3 represent personal medication intake, possible class and the mapping of each class is shown in Table medication intake, and no medication intake, 1. It is to be noted over here that for training our respectively. models, we combine the training and development dataset provided and treat it as our training dataset, therefore learning our models using 9663 tweets with 5-fold cross validation.
We use Spacy4 for all our data preprocessing and cleaning activities. We do not remove stopwords. Each document in our training and test dataset is converted to a fixed size document of 47 words/tokens. We use two pre-trained word embeddings godin9 and shin10, shared by the authors. Each of these embeddings are of 400 dimensions. Each word in the input tweet is represented by its corresponding embedding vector, when present in the vocabulary of the model.
We use Xavier weight initialization scheme11, for initializing the weights of the CNNs. Adam with two annealing restarts has been shown to work faster and perform better than SGD in other NLP tasks12. Therefore, we use the same as our optimization algorithm. We use five filters with varying filter sizes in the convolution layer and use dropout during the training process. The models are implemented using TensorFlow5. The entire ranges of the hyperparameters that we give to our random search procedure is shown in Table 2. The word embedding model to be used during training is also treated as a hyperparameter.
Hyperparameter adam_b2 n_dense_output keep_prob (dropout) batch_size learning_rate word_embedding n_filters filter_sizes
Range An ensemble of five CNNs is trained during 5-fold cross-validation training performed on our combined training dataset along with random search on the hyperparameter ranges. We train 99 such ensembles. The performance of the top 20 ensemble on the training data (blue) and on the test data (red) is shown in Figure 2. The models are arranged in the order of their decreasing training performance. We create stacked ensembles from these ensembles by taking top K ensemble models. We show the performances for such top K stacked ensembles (brown), as well. The detailed performances on the evaluation metrics of Top 3, Top 10 and Top 20 stacked ensembles are shown in
By participating in this shared task we showed the generic effectiveness of CNNs and ensembles on identification of personal medication intake from Twitter posts. Our proposed architecture of stacked ensemble of shallow CNNs, out-performed other models submitted in the task. This provided an empirical evaluation of our initial aim of combining ensembles with CNNs along with training the models using random search on the hyperparameters. In the future, we plan to work more on hyperparameter tuning using random search and various other search procedures and analyze their effectiveness. Instead of using pre-trained word embeddings it would also be interesting to look at the performance of our models by training word and phrase embeddings on a domain specific dataset of tweets. We would also like to formalize the architecture of stacked ensembles of CNNs and compare our models with an exhaustive set of other deep learning as well as traditional machine learning models.