This paper describes a medical concept normalisation system developed for the 2nd Social Media Mining for Health Applications Shared Task 3. The proposed system contains three main stages: lexical normalisation, word vectorisation and classification. The lexical normalisation stage was aimed to correct spelling mistakes and maximise the coverage of pre-trained word embeddings utilised to generate word vectors in the following stage. We experimented with three different classification models. The multinomial logistic regression model achieved higher accuracy than the recurrent neural networks with gated recurrent unit. However, the ensemble of both classification models based on the mean rule achieved the highest accuracy of 0:885 on the test dataset.
The system architecture consists of three stages: preprocessing, word vectorisation and classification. The preprocessing stage aims addressing challenges related to noisy text and is focused on lexical normalisation (i.e. spelling correction, abbreviation expansion, slang conversion), stemming and stop words removal. During the word vectorisation stage, all words in preprocessed sentences are converted into corresponding vector-space representations that are later utilised as features. Finally, the classification stage is aimed to predict a target concept and consists of an ensemble of multiple classifiers.
As any other social media posts, health-related discussions also have the characteristics of informal communications such as irregular grammar, misspellings, abbreviations and slang. To this end, the lexical normalisation component aimed to reduce the noise and to maximise the effectiveness (i.e. coverage) of pre-trained word embeddings used in the following stage.
Particularly, our lexical normalisation pipeline utilises three types of external resources:
Vocabulary is a set of known words, used to identify unknown (or out-of-vocabulary) words (i.e. candidates for correction). It could be a list of all English words and medical terms. However, we narrowed it down to vocabulary from a given pre-trained word embedding model.
Mappings are represented as translations from one word (or word form) to another, such as abbreviated to expanded forms (e.g. “hbp” to “high blood pressure”) or interjection to synonymous words or phrases (e.g. “ouchy” to “hurt”). Particularly, we used abbreviations and translations collected from the Internet & Text Slang Dictionary (noslang.com) and extended it with manually curated list of popular medical abbreviations and slang observed in the training data.
Language models are used to calculate a probability score of corrected phrase candidates and pick the best candidate based on the combined ranking from all models. We have utilised three different language models: a trigram model generated from Twitter drug discussions6, a trigram model generated on 1 million sentences from popular support groups on health-related social networking site DailyStrength (www.dailystrength.org) and a bigram model generated on medical expressions parsed from DrugInformer, a search engine for pharmaceutical products and their side effects (www.druginformer.com).
Vectorisation is a step in which all words are converted into a numeric vector representation that can be used as features to train a machine learning classification model. We utilised word2vec7 embeddings that automatically learn hierarchical representations of words by training a recurrent neural network. In the proposed classifiers we have used the following models (two of them were trained on data from generic domains and one was trained on a target domain, namely Twitter drug discussions):
GoogleNews: 300-dimensional vectors obtained from a model trained on 3 million words and phrases from Google News8 Twitter: 400-dimensional vectors obtained from a model trained on 400 million tweets9
DrugTwitter: 150-dimensional vectors learned from 1 million user sentences about drugs on Twitter10 The word vectors obtained from the pre-trained models were utilised differently depending on the classification algorithm.
To predict the most suitable medical concept corresponding to the textual description of the health condition, we have used an ensemble model that combines multiple base classifiers using the mean (averaging) rule. Namely, the final prediction is made based on the highest average value for each class derived from predicted probabilities of the base learners: In particular, we have applied this ensemble method in several places in the system to utilise multiple word embeddings in a multinomial logistic regression model and also to combine predictions of our base classifiers into the final system. We have used three different prediction models for medical concept normalisation (which correspond to the three runs submitted for evaluation):
MultiLogReg: Multinomial logistic regression is a model that generalise logistic regression by allowing more than two discrete outcomes that makes it is suitable for multi-class problems. In order to represent the entire phrase as a vector, the mean of a set of corresponding weighted word vectors (or zero vector for unknown words) is calculated, where word weights were calculated as inverse document frequency that shows whether the word is common or rare across all phrases. The averaging rule was applied to combine target and generic domain embeddings from three different pre-trained models (GoogleNews, Twitter and DrugTwitter). Particularly, the word vectors obtained from each word2vec model were used to train multiple logistic regression classifiers and their predictions were combined. The logistic regression model was trained using limited-memory BFGS optimisation11, limited to 100 iterations.
Bi-GRU: Recurrent neural networks (RNN) have an architecture designed to handle sequences of variable lengths and therefore have been successfully used in many natural language processing tasks. Bidirectional Gated Recurrent Unit (GRU)12 is aimed to increase the amount of input information by performing a forward and backward pass over the sequence, where backward hidden states are calculated by feeding the input sequence in the backward order. For this model we utilised only the word vectors obtained from the GoogleNews model, since it was trained on the largest corpora and yielded the highest performance during the preliminary evaluation on the development set. We set number of units in the GRU layer to 70% of embedding dimension. The model was trained using AdaGrad13 optimisation algorithm with the learning rate of 0:01. For regularisation, dropout with the rate of 0:1 was applied on the Bi-GRU output.
Ensemble: The proposed ensemble model is aimed to utilise the predictive power of both MultiLogReg and Bi-GRU models by combining their predictions using the averaging rule shown in Equation 1.
The training dataset for this task contains 6,650 phrases mapped to 472 concepts (14:09 phrases per concept in average, the most popular concept Insomnia contains 634 phrases, whereas 170 concepts have only single mention). The average length of phrase is 2 tokens (min: 1, max: 22). The testing dataset contains 2,500 phrases.
Table 1 shows the evaluation accuracy of the three models on the test dataset. The multinomial logistic regression model (MultiLogReg) trained on both generic and target embeddings outperformed the Bidirectional GRU (Bi-GRU) model trained only on the GoogleNews embeddings. However, the ensemble model yielded the highest accuracy score. This suggests that MultiLogReg and Bi-GRU learn slightly different information which leads to different predictions. The ensemble model was able to pick the correct candidate in majority of cases. We have presented comparison of predicted concepts by different models and gold-standard labels for the test data in Table 2. For example, “weight nightmare” was classified as Nightmare by multinomial logistic regression model, however, despite the mention of a nightmare and lack of information about the fact that the weight was increased, two other models correctly associated it with the weight gain. In the case when the concept of formication was described as “feeling like there’s bugs under my skin” all systems incorrectly associated it with epidermal and dermal conditions, however, despite the “skin” mention, it is actually a neurological disorder. In other cases, when none of the systems has made the correct prediction, at least one of them associated it with a similar concept. For example, “taste buds aren’t working” was predicted as Dysgeusia (a distortion of the sense of taste) that is associated with the correct concept (Ageusia, a complete lack of taste).
Somnolence Pruritus Nightmare Feel abnormal Drug ineffect.
Inj. site pain Insomnia Fatigue Bi-GRU Feeling abnormal Photosens. reaction Weight increased Abdom. discomfort Dysgeusia Burning sens.
Fatigue Insomnia
Feeling abnormal Photosens. reaction Weight increased Abdom. discomfort Dysgeusia Inj. site infl.
Fatigue Insomnia We presented an ensemble system that combines generalised linear and deep learning models for medical concept normalisation in the context of the 2nd Social Media Mining for Health Applications Shared Task 3. The lexical normalisation was performed prior to the classification in order to reduce the noise and maximise the coverage of pre-trained word embeddings generated on generic and target domains. The multinomial logistic regression model achieved higher accuracy than the recurrent neural networks with gated recurrent unit. However, the ensemble of both classifiers based on mean rule yielded the highest performance.