Acronyms disambiguation is considered a word sense disambiguation (WSD) task which consists on determining the correct expansion of an acronym based on a given context. This paper describes three hybrid systems to disambiguate acronyms in scientific documents, which combine three supervised machine learning (ML) models (Support Vector Machine, Naive Bayes and K-Nearest Neighbor) with cosine similarity on SciAD corpus. Our system achieved it's best performance on the independent test set on Naive Bayes and cosine similarity with 92.15% of precision, 77.97% of recall and 84.47% of F1-macro measure.
Acronyms are defined as ’a short form of multiple words or phrases’ which are used in various type of documents. Normally its meaning is represented the first time is used in each document. But there are many cases that it is used alone without its meaning like in the case of clinical documents.
There is no standard rules to create acronyms. Usually each acronym has more than one meaning which is called ’expansion’ or ’long form’. Writing acronyms without their expansions in the same sentence makes it ambiguous. Determining the correct expansion for an acronym depends on many factors like the domain it is used in. For example; acronym ’ED’ could mean ’Emergency Department’ if it is used in documents related to medical domain, or it could mean ’Euclidean Distance’ if it is used in documents related to mathematics domain. Furthermore, acronyms could have many expansions even in the same domain like ’RNN’ which has two possible expansions ’Recurrent Neural Network’ and ’Random Neural Networks’ which both of them are used in computer science domain.
Word Sense Disambiguation (WSD) is a Natural Language Processing (NLP) task which is applied to determine the right expansion of acronyms based on it’s context. There are two types of WSD; all words WSD which disambiguates all words in the given context. The second type is Lexical sample WSD which disambiguates specific word in the context. Disambiguate acronyms are considered a special case of Lexical sample WSD.
Three main approaches are applied extensively on WSD.
The first one is Knowledge based approach that integrates
lexical knowledge bases and exploits semantic similarity and
graph-based approaches. In similarity-based methods each
expansion of the ambiguous acronym is compared to those
of the content words appearing near it (context words) and
the expansion with the highest similarity (for instance, using
cosine distance) is supposed to be the right one.
Unsupervised ML approaches disambiguate by finding
hidden structure in unlabelled data, for instance, clustering
documents or sentences in groups each one representing an
expansion
Finally, supervised ML approaches which require tagged
corpora. WSD based on this approach, is considered as a
text classification problem where the objective is to
predict the correct expansion of an acronym among its
different expansions
We explore word embeddings in this work as features to
be used in ML algorithms; a preliminary analysis is done
in
In this paper three supervised ML models combined with a knowledge based model are used to disambiguate scientific acronyms for SDU@AAAI-21 shared task (Veyseh et al. 2020a). The rest of the paper is organized as follows: Method section describes the data set which is used in this study, the features and the different models that we applied. In strategies section we describe how the proposed methods are experimentally conducted. Finally, we present our results compared to the baseline system.
Acronyms are ambiguous because they could have multiple expansions. Determining the correct expansion of an acronym is a WSD problem. Since SciAD contains an small set of exampled for some acronyms, we combined supervised machine with knowledge based approaches to tackle this problem.
SciAD
Figure 1 shows frequencies of annotated examples per each acronym; 299 acronyms have less than 20 annotated examples in the training data set.
Additionally, the organizers provide the participants with an acronyms dictionary which contains 732 acronyms and 2308 senses with average of 3.15 senses per acronym. Figure 2 shows the distributions of senses for acronyms contained in the dictionary.
Baseline In order to familiarize the participants with the task, the organizers provided a rule-based baseline in code directory. This baseline computes the frequency of the long forms in the training data set. Afterwards, to make prediction for each acronym in the development data set, it selects the long form with the highest frequency as the final prediction. If there is a tie, the long form that appears the first among all tied long forms in the dictionary is selected as the final prediction.
Supervised ML plemented: • Support Vector Machine (SVM): which separates positive samples from negative ones based on the idea of linear hyper-plane from labeled data set differentiating between samples into true or false categories. SVM is adapted to multi-class classification to be used in WSD. • Naive Bayes (NB): a probabilistic approach to estimate probabilistic parameters which has a long history of success in WSD. This approach is based on Bayes theorem to compute the conditional probability for each sense of an abbreviation from a set of features. • K-Nearest Neighbor (KNN): the classification is done by computing the Euclidean distance for each test vector with the most k similar training vectors.
Knowledge Based Approach For acronyms with few examples in the data set, which are insufficient to train a supervised ML method, a knowledge based approach is implemented. This method is based on expansion’s dictionary provided by organizers; cosine similarity was applied in the test examples. Two vectors were said to be similar when the cosine similarity was close to 1, and they were said to be dissimilar when it is close to 0 (Singhal 2001).
Features play an important rule in WSD system, two types
of features were used. WSD Features: Several lexical
features were used to disambiguate acronyms considering both
left and right contexts of the target Acronym. Our system
adopted a set of lexical features that have been used
successful in WSD. Given a sentence s formed by a set of words
[:::w 2; w 1; w0; wt; w+1; w+2:::] where wk is the targeted
ambiguous acronym, we extracted the following features:
1. Word Features: stemmed words for each tokens on both
side of the target acronym.
2. Word features with direction: The relative direction (left
or right side) of stemmed words.
3. POS (Part-Of-Speech) Tag: POS tag feature for each
token on both sides.
4. Position features: The distance between the feature word
and the target acronym.
5. Word formation features from the acronym itself
including special characters, capital letters and numbers.
Pre-trained word embedding features: A pre-trained word
embedding model with 300 dimension vectors was built
used FastText (Joulin et al. 2016) generated from several
English resources such as the Wikipedia and data from the
common crawl project,
Several pre-processing steps were conducted on the dataset including remove stop words, special characters and stemming the words before extracting the features. For supervised ML approaches, features are formed by combining WSD lexical features and the summation strategy from the pre trained word embeddings which are generated based on the following equation:
S = jW j X v(W (i)); i 6= k i=0 where W is a list of words which surrounding the targeted acronym. jW jis the length of the list and v(.) is a Fasttext pre trained word embedding as mentioned in previous subsection and k is the position of the target acronym. (1)
On the other hand, for the knowledge based approach, just the summation strategy of pre trained word embedding vectors were generated for each example and for the candidate expansions which were extracted from expansions dictionary.
In this phase, training and development data sets were combined to increase the size of data set for each acronyms. Our goal was to build a model to predict acronym’s expansion based on a context for each acronym that has more than 20 annotated examples. To achieve this goal the training data was separated based on each acronym data set. Table 2 shows the distribution of the whole data set for ML and Knowledge-based (KB) approaches, 450 acronyms with 53702 annotated examples, are disambiguated by three ML models, SVM, NB, and KNN. While 282 acronyms with 2521 annotated examples disambiguated by cosine similarity method.
ML KB Total
When the testing data set was released by the organizers, the testing data set was divided based on the training data we had previously (see Figure 4). Table 3 shows the distribution of testing data set over the two models; 444 acronyms with 5876 annotated examples in the testing data set, are disambiguated through the three ML models. 174 acronyms in 342 annotated testing examples were disambiguated with cosine similarity method. Figure 3 summarizes the overall process for the proposed system.
The system performance was evaluated by using three metrics, Precision which is defined as the percentage of the instances which actually have a class label X (True Positives) divided by all those which were classified as class label X as the following equation:
P recision =
T rueP ositives
T ruepositives + F alseP ositives Recall is defined as the percentage of the instances which were classified as class X, divided by all instances which correctly have class X as the following equation: Recall =
T rueP ositives
T rueP ositives + F alseN egatives F1-macro is defined as the harmonic mean of Precision and Recall as the following equation:
F1 = 2
P recision
Recall
P recision + Recall
The training data set includes 634 expansions with less than 10 annotated examples from different acronyms, to balance the data set, these expansions were replicated through oversampling techniques using sklearn library. Then 5 fold (2) (3) (4) cross validation was used for all acronyms in ML models. Furthermore, the training data set contains 10 nonambiguous acronyms which their data set contain one expansion.
Table 4 shows our result on training phase, NB with cosine similarity achieved the highest performance with precision 90.31% , recall 87.16% and F1-macro 84.37%.
Table 5 shows the final scores for our systems were reported by the organizers. The best performance achieved precision 92.15%, recall 77.97% and F1-macro 84.47%, for a hybrid approach with NB and cosine similarity.
A sample of low performance on accuracy were achieved on a training phase shows how strongly imbalanced data set size affects on the model. We focus on Naive Bayes approach since the best result was achieved through this approach. Table 6 shows the accuracy of 4 acronyms, ARD acronym with 246 dataset is distributed between two expansions ”accelerated robust distillation” with 46 training examples and ”adversarially robust distillation” with 201 training examples,
In this paper, we introduced a system to disambiguate scientific acronyms. Our system best score was achieved by a hybrid approach combining supervised ML Naive Bayes and cosine similarity with precision 92.15%, recall 77.97% and F1-macro 84.47%.
Thanks to Palestine Technical University-Kadoorie (PTUK) and DeepEMR project (TIN2017-87548-C2-1-R) for partially funding this work.
Singhal, A. 2001. Modern Information Retrieval: A Brief Overview. IEEE Data Eng. Bull. 24(4): 35–43. URL http: //sites.computer.org/debull/A01DEC-CD.pdf.