Acronym Disambiguation (AD) task aims to map the acronym in sentences to the corresponding expansion among candidate expansions. However, these models based on domain agnostic knowledge might perform insuficient when directly applied to the data in some specific areas such as science and law. To track these issues, we propose a prompt-based acronym disambiguation system with special negative sampling. Specially, we design a prompt to combine the input sentences and candidate expansions, followed by a Pre-train Language Model (PLM) to calculate the score. Moreover, negative expansions are randomly sampled for better training, and an additional hinge loss is added to improve the robustness of our system. Experiments show the efectiveness of our system, and we get competitive results in the SDU@AAAI-22-Shared Task 2: AAAI'22: Scientific Document Understanding Commons License Attribution 4.0 International (CC BY 4.0).
Acronyms are abbreviations formed from the initial
components of words or phrases [
However, for people without domain knowledge, acronyms might be confusing at some time, such as “PPP” can be Paycheck Protection Program or Public-Private Partnership. It is necessary to build an acronym disambiguation system that can identify the correct meaning of acronyms in a diferent context to track this issue. As shown in Figure 1, given several sentences containing acronym POS, we need to find out the corresponding expansion among candidate expansions in the given dictionary. Moreover, understanding the correlation between acronyms and their expansion is beneficial for several tasks in natural language processing, including question answering and machine reading comprehension.
Acronym disambiguation is usually considered as a
sequence classification task [
CEUR
Documents Dialogue fillers and acceptance words affect the accuracy of POS tagging.
These heuristics filter out redundant constituents and raise the ratio of POS in the dataset.
POS of first occurrence : Important concepts are expected to be mentioned before less relevant ones.
Dictionary-POS
Part-Of-Speech positive instances Possessive Position postag
The expansion in the same color is just the corresponding
expansion for each POS.
graph structures [
Furthermore, large-scale data during training brings
an extremely long-tail problem. The size of the original
candidate expansions in the dictionary varies, making it
hard to batch the samples during training. To address
this issue, previous works [
In this paper, we proposed a prompt-based acronym
disambiguation framework with a specially designed neg- combinations of prompts have been explored, such as
ative sampling strategy. Firstly, we design a prompt tem- prompt augmentation [
In semi-supervised WSD models, the classifier is
In this section, we mainly introduce the related stud- trained by the integration of annotated and
unannoies for prompt-based models, especially the BERT-based tated corpora. PageRank-based WSD algorithm
commodels. We first review the existing researches on word bined pIWordNet and semantic links from valency
lexisense disambiguation, which is more generalized than con, Wikipedia articles and SUMO ontology [23].
Clusthe acronym disambiguation. tering and labeling strategy was used to generate labeled
data for subjectivity WSD semi-automatically and further
combined with original annotated data [24].
2.1. Prompt-based Learning However, all these methods ignore the interaction
bePrompt is suggestive information to enhance the knowl- tween ambiguous word explanation and its context. In
edge that PLMs (Pre-trained Language Models) learned this work, we propose a prompt-based model to
inteduring pre-training, containing the description of task an- grate better the semantic relationship between acronym
swers and corresponding answers. Prompt-based learn- context and candidate expansions.
ing is a slot-filling method based on language models,
which aims to probabilistically construct the final prompt 3. Methodology
as the prediction of the task. Previous exploration in
prompt-based learning mainly focuses on prompt con- In this section, we present the overall architecture of
struction, including prompt engineering and answer en- our proposed framework, which uses the prompt-based
gineering. Prompt engineering creates a prompt function model to solve the acronym disambiguation problem and
applicable to corresponding downstream tasks [
Multi-prompt learning, an ensemble of these two
engineering prompts, aims to improve the generalization of
models [
Part-Of-Speech positive instances Position Possessive postag
[Part-Of-Speech]
Dialogue fillers and acceptance words affect the accuracy of POS tagging. ⨁ [SEP]POS[SEP]the meaning of POS is or equals <MASK>
[positive instances] [Position] ……
[Sound Pattern of English] [Frequent Candidates] ……
…… …… 1 2 3 k k+1 k+2 N
BERT
⨁ : Concatenate logits CE loss
Hinge loss
Dynamic Mask logit1 logit2 logit3 …… logitk logitk+1 logitk+2 …… logitN 3.1. Problem Statement token is inserted before and after the acronym, followed by a string: the meaning of acronym is or equals expanFormally, given an input sentence = 1, 2, ..., and sion. Finally, BERT with an additional linear layer is acronym = at position , the goal is to disambiguate employed as our encoder. For training, we will calculate the corresponding expansions among candidate ex- the cross-entropy loss and adopted hinge loss [25]. For pansions { 1, 2, ..., }. The candidate expansions are inference, a dynamic mask strategy is adopted, in which given in advance and their size vary. Specifically, in we will drop the logits of added expansions. Specially, this paper, we treat this task as a classification problem we will drop the logits from the added negative samples, by padding the candidate expansions set to fix length which can not be the answer. with randomly chosen unrelated expansions. We will dynamically mask the logit of added expansion in the testing phase and choose the largest one among original 3.3. Prompt Design candidate expansions as the final prediction.
To build a prompt template efectively, we consider a
two-stage strategy. We hope the model to be aware of
3.2. Overview two tasks: finding out the acronym and finding out
the corresponding expansion. Thus, we employ the
As shown in Figure 2, given the acronym POS in the token [SEP] to highlight the acronym, which can help the
sentence, there are candidate expansions which can be model to understand where the acronym is. For second
divided into a positive sample set of size 1 and a hard neg- task, previous works[26, 27] show that a longer prompt
ative sample set of size − 1 . Firstly, in the expansions of usually performs better. To add more tokens, we use the
other acronyms, we randomly sample − samples as the template: the meaning of acronym is or equals expansion.
easy negative sample set to pad the candidate expansions For French and Spanish, we employ the corresponding
into fix size . Secondly, we design a prompt strategy to translation as the prompt templates.
combine the acronym and candidate expansions. [SEP]
3.4. Negative Sampling
The size of candidate expansions in the dictionary varies,
making it hard to train an eficient model. Moreover, we
consider the negative samples in the original candidate
expansions as related to but not exactly the ground
truth. To improve the robustness and convergence of
the model, we adopt a negative sampling strategy. We
set the size of the padded candidate set as and
randomly sample expansions from the candidate expansions
of other acronyms as needed. For example, is set to 6,
and the number of original candidate expansions is 2. We
need to pick up 4 additional expansions. We note that
[
The diference is that we divided the negative samples into hard negative samples and easy negative samples, thus designing extra loss. 3.5. Loss Function For the model, we consider two goals: 1) the ground truth expansion gets the highest score; 2) the original negative expansions get higher scores than the additional negative expansions. For the first goal, we employ the cross entropy loss function. Note the predict label as and the ground truth label as .
Dataset Legal English Scientific English French Spanish where is also a learnable hyperparameter to control the ratio of hinge loss.
= ( , ) (1) Wbye SeDvaUlu@aAteAaAllI-m22od[e2l8s]b.asAeds osnhotwhendiantaTseatblpero1v,idthede where the means cross entropy loss function. For dataset [29] contains training and development datasets the second goal, we follow the idea of hinge loss and in English (both scientific and legal domain), Spanish, and we want the minimum of the original expansion scores French consisting of 497 English Scientific, 303 English = { 1, 2, ..., −1 } is higher than the legal, 546 Spanish, and 669 French acronyms. For each maximum of the additional expansion scores = language, a diction containing acronyms and their candi{ 1, 2, ..., − } by a margin. date expansions is provided. For Legal English, there are 3717 sentences containing 174997 tokens and 625 candi ℎ = max( − min( ) + max( ), 0) (2) date expansions in the diction. The average expansion length of all acronyms is 3.1. The acronyms in the testing where max(⋅) and min(⋅) mean the maximum and mini- set would not appear in the training set. mum function while is a learnable margin. Hence we For Exploratory Data Analysis(EDA), we analyze the get our final loss. statistical features in the dataset. As shown in Figure 3 and Figure 4, we can see that: 1) for most acronyms, the = + ℎ (3) corresponding sentences are more than 10, indicating that the samples are highly similar. 2) for most acronyms, the corresponding candidate expansions are less than 4.
Given the acronyms in sentences, candidate expansions In this section, we first introduce the experimental and ground truth labels, we can calculate the macrodataset and evaluation metrics and then conduct compre- averaged precision, recall and F1 score. hensive experimental studies to verify the efectiveness of our method. 4.3. Implement 4.2. Evaluation Metrics
All models are implemented based on the open-source transformers library of Huggingface [30]. For all datasets, Dataset Legal English Scientific English French Spanish
Model bert-large-cased spanbert-large-cased scibert-scivocab-cased scibert-scivocab-cased ( = 1.5 ) bert-base-french-europeana-cased
camembert-large bert-base-spanish-wwm-cased bert-base-multilingual-cased
3 Epoch of training stage 4 we set the = 0.1 and = 1 . The batch size is 2 and the size of expected expansion = ([ ]) + 2. For example, for French dataset, the maximum of candidate expansions for all acronyms is 12, thus we set = 12 + 2 = 14. As for other parameters, we set the learning rate as 3 − 5 and random seed as 10086. We pad or cut the input into 128 length. For French dataset, the prompt is: el significado de acronym es o igual a expansion. For Spanish dataset, we use: la signification de acronym est ou est égale à expansion. We train our model in one V100 GPU and evaluate the result using the oficial script. 4.4. Comparison 4.4.1. Overall Performance The overall performance results on the validation set are shown in Table 2. For Legal English, we choose the bert-large-cased [31] and spanbert-large-cased as the PLM. For Scientific English, we choose scibert-sci-vocabcased [32]. For French, we choose bert-base-frencheuropeana-cased and camembert-large [33]. For Spanish, we choose bert-base-spanish-wwm-cased [34] and bertbase-multilingual-cased [31]. As shown in Table 2, we can observe that most models sufer from over-fitting after 3 epochs. Moreover, we find that the BERT trained on the specialized corpus performs better than trained on the common corpus. 4.4.2. The Efect of PLM We change the BERT type to study the influence of different backbones on the French dataset. As shown in Figure 5, we can see that the larger models usually get better results. Another interesting observation is that all models sufer from over-fitting at epoch 4. 4.4.3. The Efect of Margin We change the to 0.0 and 1.0 and conduct our experiments in the English science dataset. According to Figure 6, we can find that a large brings a considerable change during training. Actually, a large means a large gap is required, leading to the oscillation in the loss. 4.4.4. The Efect of Ratio We change the to 0.5 and 1.5 and conduct our experiments in the English Science dataset. As shown in Figure 7, the larger leads to a lower result. Actually, a large means a large hinge loss, which pushes the model to 1 2
3 Epoch of training stage
4
In this paper, we proposed a novel prompt-based model, which shows promising and competitive performance in SDU@AAAI-22 - Shared Task 2. We design an efective, prompt template that helps the model utilize the implicit knowledge in the pre-trained language model. A dynamic negative sampling strategy is employed to improve the robustness and performance of our model.
For future work, we will try to adopt a learned prompt
template rather than a fixed template following the CoOp
[26]. Moreover, the acronym disambiguation under a
zero-shot setting would be another interesting and
valuable topic. Utilizing the graph information in given
sentence [
This research was supported in part by the National Key Research and Development Program of China (No. 2018YFB1601102) and the Shenzhen Key Laboratory of Marine IntelliSense and Computation under Contract ZDSYS20200811142605016. We thank the organizers of acronym identification and disambiguation competitions and the reviewers for their valuable comments and suggestions.