Most practical text classification tasks in natural language processing involve training sets where the number of training instances belonging to each of the classes are not equal. The performance of the classifier in such a case can be affected by the sampling strategies used in training. In this work, we describe a cost sensitive and random undersampling variants of convolutional neural networks (CNNs) for classifying texts in imbalanced datasets and analyze its results. The classifier proposed in this paper achieves a maximum F1-score of 0.414 placing 2nd on the ADR dataset and achieves a maximum F1-score of 0.652 placing 6th on the medication intake dataset.
Input The datasets for the tasks contained tweets-ids and their respective categorical annotations. The first set of annotations provided for the task was used as training dataset and the second set was used development/validation set. The original texts were available for only about 40% of the annotations for Task-1 and 60% for Task-2. In Table 1, we show the details the datasets for both tasks and their respective class distributions.
Classifier For the CNN classifier used in this paper, we implemented our models based on the original CNN architecture as proposed by Kim et. al. for sentence classification.10 We use this CNN architecture to construct cost sensitive and random undersampling variants to tackle the class imbalance problem. The random undersampling variant (Undersampling-CNN) is constructed by randomly sampling equal number of class-instances in each epoch. This means that there are far fewer training instances in each epoch. The cost sensitive variant For our experiments we use a fixed maximum sentence length of 50 words. All tweets with length less than 50 words are padded with zeros. Sentences with more than 50 words are truncated. As pre-processing steps we tokenize each tweet and normalize punctuations. For word embeddings, we use the word vectors generated using millions of tweets containing drug names and made available by Sarker et. al.11 for mining health related data online. Hyperparameters For experimentation we use filter sizes in the range of 1 to 5 words with the number of filters i.e. model hidden dimensions in the range 50-150. The best models were obtained for filter combinations of 2,3,4 and 75 filters. A softmax cross-entropy function is used to compute the cost for optimization. For optimization, we use the Adam Optimizer with a learning rate of 0.001.12 We employ dropout keep probability of 0.5 during training to prevent overfitting.13 We also apply L2 regularization rate of 0.001 for training across 50 epochs. The model with the best performance on the validation/development set is saved and used on the evaluation/test set. The Undersampling-CNN had fewer training samples per iteration compared to training on the entire set. Hence, the Undersampling-CNN had to be trained at half the learning rate i.e. 0.0005 and took around 40-50 epochs to arrive at the optimal model as compared to 10-15 for CNN and CostSensitive-CNN models. Although we could add feature embeddings for each word in the architecture, we do not add any task specific features.
In Table 2, we show the results for both tasks. For Task-1, the CostSensitive-CNN model was found to achieve the best score. As described earlier, the Undersampling-CNN takes longer to train on all the randomized training samples in majority class. However, the results did not show its improvement over the CNN or the CostSensitive-CNN model.
In this work we evaluate a CNN classifer for detecting ADR and medication intakes as part of two shared tasks at AMIA-2017. The classifiers presented in this work placed 2nd and 6th in tasks 1 and 2 respectively. As improvements to the proposed classifiers, we would like to experiment with variants of cost sensitive training architectures in CNN for tackling class imbalance problems as well as strategies to introduce controlled synthetic sentence variants for oversampling the minority class.