Acronym Disambiguation (AD) task is designed to find the exact expansion of the acronym in a given sentence. Since little work has been done in a Machine Reading Comprehension (MRC) way, this paper presents a novel model which leverages the advantages of both MRC and sequence tagging. First, AD is regarded as a multi-choice task and all the candidate expansions are options. We design useful question-answer pairs where Question can be seen as the combination of sentence and acronym while Context consists of the candidate expansions. Second, we apply adversarial learning (i.e. FGM) and normalization methods such as Gradient Centralization (GC) to further improve the robustness and generalization of the model. Third, the ifnal answer is jointly predicted by two tasks which can enhance model's understanding towards AD. Besides, the model also infers the test set to construct pseudo-labelling set to make the most of data. The model we put forward provides a novel way to handle AD and the performance can be competitive.
Acronyms are built in part from the first letters of word components and pronounced like a word (e.g. NASA).
Due to their convenience, acronyms are of widespread use in many scenarios where long words show frequently.
For instance, an original sentence is, “Here we present a non-linear method based on a deep convolutional neural network and this convolutional neural network is quite By replacing the initial long part with the acronym, the sentence can be brief and explicit. “Here we present a non-linear method based on a deep CNN and this CNN is quite powerful”.
Although using acronyms in documents seems like a favorable choice, people who don’t know much about a specific field can sufer from the ambiguity of acronyms.
Therefore, it is beneficial to figure out the relationship between acronyms and their correct expansions for the However, given that acronyms are widely used in considerable fields, it is hard for people to clarify the real sary to build a model which can automatically find the accurate expansions of acronyms used in documents. biguation (AD) task includes a sentence which contains Proceedings of the Workshop on Scientific Document Understanding (SDU 2022), Remote, 2022. nEvelop-O
NER Named Entity Recognito, ...
Named Entity Recognitio
expansions of a given acronym and its true label [
Consequently, AD can be seen as a classification task.
At the beginning, much attention is paid on the rulean inexact pattern matching algorithm is proposed and play a role in the past [4]. Due to the complexity of difermethods can always fail to catch the subtle relationship between an acronym and the according expansion [5].
Thus, an increased number of research shifts focus to exploiting the contextualized information. Based on lexical knowledge, the method computes the similarity between the acronym and words near it [6]. On the other hand, unsupervised methods are put forward to break the limit further aim of eliminating the ambiguity of the acronym. a native speaker of English let alone the poor machine. meanings of acronyms one by one. Thus, it is neces- based methods [2], [3] and they do work. For example, As shown in Table 1, each instance in Acronym Disam- ent acronyms’ meanings in various scenarios, rule-based [ C L S ] Q u e s t i o n [ S E P ] C o n t e x t [ S E P ] FGM g a t e r o c s Question: Sentence [SEP] Acronym Context: Expansion 1 | Expansion 2 ... n o i t c i d e r P r e w s n A l a n i F
Pseudo Labelling of annotated data. Via clustering word embedding [7], and sentence, we design the question as “Context” : sen[8] into diferent groups, machine can learn how to dis- tence [SEP] “Acronym”: acronym. Note that the lowertinguish one expansion from another. Each group just case word just represents the instance in the dataset. represents an expansion [9]. Context is composed of all the candidate expansions sep
Given that supervised methods always have a better arated by “∣” which just highlights diferent expansions performance [10], more researchers tend to apply semi- to make the model easy to learn the diferences. Eventusupervised approaches [11] to further combine both ad- ally, our model is jointly trained on two tasks: sequence vantages. tagging and machine reading comprehension. The re
Over the past few years, pre-training models such as sult of inferring the test set can be used to construct the BERT [12] have already been proven to be powerful. pseudo-labelling set to make full use of all the data to BERT firstly masks the words in contexts at a certain further improve the model’s performances. possibility like “ Beijing is the caption of [MASK] (China) The main contributions of this paper are summarized ” and then predicts the masked part to gain the repre- as 4 points: sentations of the corpus. Also, BERT is pre-trained at a binarized next sentence prediction task. When it is pre- • We introduce Acronym Disambiguation to Readtrained done, BERT is finetuned at the annotated data. ing Comprehension Task naturally and observe By fully making use of the unlabeled and annotated data, something interesting about the components of BERT outperforms all the models before it at many tasks question-answer pairs. on GLUE [13]. Afterwards, a large number of BERT’s • To the best of our knowledge, we should be the variants come out RoBerta [14], ERNIE [15], SpanBert ifrst to train two tasks: sequence tagging and [16], etc. reading comprehension jointly on AD.
BERTLARGE is our base model (L=24, H=1024, A=16, • Our end-to-end model does not need any extra Total Parameters=340M). Diferent from other BERT mod- operations and it is environmentally friendly comels, this model is pre-trained by applying Whole Word pared to the ensemble of many models. Masking technique and then fine-tuned on the SQuAD • Adversarial training is smoothly combined with dataset. Our whole model architecture is in Fig 1. To let Gradient Centralization to improve the perforthe model exploit the relationships between the acronym mance.
This paper is organized as follows: Related work is included in Section 2. Then comes with model structure and experiment which has 4 subsections: The task, finetuning results and training details. And the last two sections are the conclusion and references.
Since neural networks are fragile and vulnerable to perturbations, adversarial training is a good way to enhance model’s robustness by training the model on extra adversarial examples. Based on the observation that the direction of the perturbation (i.e. the gradient) matters most, the Fast Gradient Sign Method (FGSM) is originally yielded to produce adversarial examples [17]. Afterwards, the Fast Gradient Method (FGM) is added on the word embedding to improve the generalization of the model [18]. Diferent from them, the Projected Gradient Descent (PGD) does a range of attacks on the model and can map the perturbation to a specified range every time [19]. It is obvious that PGD performs better at the cost of high computation complexity. By restricting most of the forward and back propagation within the first layer during the adversarial training, YOPO reduces the cost of computation [20]. FreeAT recycles the gradient information when updating the model parameters to cut down the cost [21].
Given that the model is likely to overfit the dataset, we improve the model’s robustness by adding perturbations to the word embedding (e.g. FGM).
Oc Ec CLS E1 ... En T1 ... Tn
Os
Es SEP E1 ... Em T1 ... Tm
Os Es SEP
methods have exceeded the performance of humans.
Inspired by the habit of humans that we first verify if the answer exists and then we can choose to answer it or not, Retrospective Reader is proposed to better tackle the complex problems [33]. Motivated by their work, we consider the acronyms as the named entity and other parts are not. By designing this strategy, we can also do sequence tagging task just like verifying the acronym.
The objective of Machine Reading Comprehension (MRC) 2.2. Reading Comprehension and irseptoreosuentptutthtehegidviesntriqbuuetsiotino n(,|, s)uppowrthinegrec,o,n t∈e xt and∗
Sequence Tagging the prediction answer respectively and are composed Machine Reading Comprehension (MRC) plays a role in of tokens in the vocabulary (Fig 2). Since the context the development of Artificial Intelligence (AI) and still here is all the candidate expansions split by “|”, we exactly faces complicated problems at present. The early Ques- focus on extractive reading comprehension task because tion Answering (QA) system is rule-based and works the answer can be found in the context. We denote ,…, really bad [22]. After that, the system is made up of rules as the answer span where < . The answer span is [23] which compute the similarity of question-answer predicted by maximizing the sum of the possibility of pairs and Bag-Of-Words Model (BOW) [24], [25]. But the start of the answer span ( = |, ) and the rule-based methods always fail. Soon the dataset MCTest possibility for the end ( = |, ) . is put forward whose instance contains a passage and The goal of Sequence Tagging is to predict the question [26]. And the answer must be chosen from the possibility of every position being the special token four choices. Machine Learning approaches come up (e.g. ( ̂ 1, 2̂, … , ̂ ) such as the named entity based on the to minimize the max-margin loss function) to perform given sequence = {( 1, 1), ( 2, 2), … ( , )}. better on the MCTest [27], [28], [29]. With the development of deep learning, the Long Short-Term Memory 3.2. Multi-Task Fine-Tuning (LSTM) [30] with attention is proposed and achieves good results [31]. Since the ratio of noise in the CNN/Daily Mail dataset is high, SQuAD comes up to further boost the development of MRC [32]. Nowadays, BERT-based Our multi-task model gets improved by leveraging the both advantages of MRC and Sequence Tagging task.
Lspan, Ltag represent the loss of MRC and Sequence Tagging task repectively. The whole loss is computed by 3.2.1. MRC Following [12], the input sequence is encoded by multilayer Transformer [34]. Let H = {h1, h2, … , hn} denote the last-layer hidden states of . The start possibility can be computed by Equation 3. And the analogous rule for the end possibility .
FFN(x) = (0,
1 + 1) 2 + 2 = Sigmoid(FFN(H)) And the aim of MRC is to fit a model to the examples drawn from the training dataset and refers to all the parameters.
arg min ,,∼
[− log (|, )]
We can turn the problem into minimizing the binary tions of example and is the size of . cross-entropy loss for the start and end predictions where , are respectively ground-truth start and end posi
1 =1 Lmrc = −
∑[log(
) + log( )] 3.2.2. Sequence Tagging We first label every token in the sequence 0 and 1 for the right expansion part which means other expansions are labeled 0. For instance: tropy Approach to NERn.[SEP]Named Entity
Recognitio|named entity recognition...” • labeled = [0,…,0,1,…,1,0,…,0]
We can train the model by minimizing the crossthe label just like the example’s. entropy loss where ̂ is predicted by the model and is ŷi = Softmax (FFN(H))
1 =1 Ltag = −
∑[ log ̂ + (1 − )log(1 − ̂ )]
Adversarial Training (AT) is a good way to enhance the model’s robustness by training the model on the generated adversarial examples. By adding small perturbations equation 1 where just indicates the importance of sequence tagging task.
=
+ • input = “[CLS]Acronym[SEP]A Maximum En- default as 1. and is the gradient of common training on (1) (3) (4) (5) (6) (2) tences. More specifically, the length ranges from 1.00 to 251.00
Although the performance of Neural Networks like Trans(7) former can be impressive, it is hard to train them because of the oscillation of training process and risks of being trapped in a local minimum. Performing normalization on activation or weights can to some degree improve this situation. Similar to normalization methods, gradient centralization (GC) centralizes the column vectors of weights so that the mean value of the column vectors is 0 [35]. Specifically, Φ , ℒ are the GC operation and loss respectively. ∇w ℒ is the th column of the gradient matrix and ∇w ℒ is the mean value of the th column of the gradient matrix. By removing the mean value from every column vector of the gradient matrix, GC can make the optimization landscape more smoother which contributes to the more eficient training and the better generalization. Diferent designs of QA pairs matter. In Table 2, the choice stands for the candidate expansions split by “|”.
Note that the dropout ratio here is 0.0 to better observe
Also, this task covers several languages: English, the efect of diferent designs. Although the sentence conFrench and Spanish. Interestingly, only the English ver- tains the specific acronym such as CNN, concatenating it sion contains diferent domains: legal and scientific [ 36]. to the question still improves the model’s performances. And we choose the scientific one for it is more related “Acronym”, “Context” and “Option” are the prompts. Into our field. Besides, there is no overlap among the terestingly, just adding prompts to the question and anacronyms in the training set, the development set and swer works. Also, exchanging the position of sentence the evaluation set. The model’s performance is evaluated and acronym can lead to a better score. However, it by the macro-averaged precision (P), recall (R) and F1. should make no diference in humans’ thinking. Maybe the focus of the machine needs to be further explored.
In this subsection, we present diferent plans for the modelling and the according results on the oficial dev test. To reduce the efect of the seed, every experiment is carried out with 3 diferent seeds:2021, 2022 and 2023. 4.2.3. Better Generalization Since the model has the risk of overfitting the training dataset, we apply adversarial Learning such as FGM and Gradient Centralization (GC) to enhance the model’s
In this section, we present our model’s fine-tuning results on shared task 2: Acronym Disambiguation.
This task is designed to find the exact meaning of the ambiguous acronym in a given sentence. The input is a sentence which includes the acronym and the systems is going to figure out the expanded form of the acronym. For instance: • Input Sentence: Here we present a non-linear method based on a deep CNN. • Input Candidate Long-forms: convolutional neural network, Convolutional Neural Network, convolutional neural networks • Output: convolutional neural network Since the case of words difers, we set do lower case as False. For the training process, we use 5 epochs and the batch size of both training and validating is 32. The optimizer is AdamW [37] which applies weight decay on Adam [38] in a diferent way. And the learning rate, adam epsilon, max grad norm and weight decay are 2e−5, 1e−8, 1.0 and 0.0 respectively. Also, we use the linear schedule for warming up and ratio is 0.1. Due to the observation in the dataset (Fig 3), the max sequence length is 160.
All the experiments are done on the GPU RTXA6000 which has 48GB. And time for every independent experiment in Table 2, 3, 4 take 7m 51s, 13m 41s and 14m 59s respectively. Besides, the space for every experiment is 19GB. 4.2.2. Options of QA pairs 4.2.4. Multi-Task Learning Here we do experiments on the efect of the value of in Equation 1. Since it may take more time for convergence in multi-task learning, the epochs here is 10. Also, the dropout ratio is 0.0. We can draw conclusions from 4.2.5. Data Augmentation and Pseudo-labelling We also use data augmentation such as shufling the options during training, which can improve the performance (Table 5). Finally, we combine everything together and construct pseudo-labelling set for second training which leads to the comprehensive improvement.
By leveraging the advantages of both Machine Reading Comprehension task (MRC) and sequence tagging, our multi-task model gets improved in Acronym Disambiguation task. The combination of adversarial training and gradient centralization can further improve the model’s performance. And extra improvement can be made via designing useful prompts related to the specific task. For future work we plan to focus on the interesting phenomena observed in the experiments. 6.
We thank the organizers of this Acronym Disambiguation task for sharing such an interesting topic with us and valuable advice from friends and reviewers. And this research has been supported by the National Natural Science Foundation of China under grants 61876217, 62176175; the Innovative Team of Jiangsu Province under grant XYDXX-086; the Science and Technology Development Project of Suzhou under grants SGC2021078.
Nicole Meister, Multilingual Acronym Extraction [2] A. S. Schwartz, M. A. Hearst, A simple algorithm for identifying abbreviation definitions in biomedical text, in: Biocomputing 2003, World Scientific, 2002, [9] J. Charbonnier, C. Wartena, Using word embeddings for unsupervised acronym disambiguation (2018). [10] S. Melacci, A. Globo, L. Rigutini, Enhancing modern supervised word sense disambiguation models by semantic lexical resources, in: Proceedings of the Eleventh International Conference on Language
Resources and Evaluation (LREC 2018), 2018. [11] S. Sousa, E. Milios, L. Berton, Word sense disambiguation: an evaluation study of semi-supervised approaches with word embeddings, in: 2020 International Joint Conference on Neural Networks bag of words approaches to medical records, Inter(IJCNN), IEEE, 2020, pp. 1–8. national journal of medical informatics 67 (2003) [12] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, 75–83. doi:10.1016/S1386- 5056(02)00057- 6.
Bert: Pre-training of deep bidirectional transform- [26] M. Richardson, C. J. Burges, E. Renshaw, MCTest: ers for language understanding, arXiv preprint A challenge dataset for the open-domain machine arXiv:1810.04805 (2018). comprehension of text, in: Proceedings of the 2013 [13] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, S. R. Conference on Empirical Methods in Natural LanBowman, Glue: A multi-task benchmark and anal- guage Processing, Association for Computational ysis platform for natural language understanding, Linguistics, Seattle, Washington, USA, 2013, pp. arXiv preprint arXiv:1804.07461 (2018). 193–203. URL: https://aclanthology.org/D13-1020. [14] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, [27] K. Narasimhan, R. Barzilay, Machine comprehenO. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, sion with discourse relations, 2015. doi:10.3115/ Roberta: A robustly optimized bert pretraining ap- v1/p15- 1121, publisher Copyright: © 2015 Associproach, arXiv preprint arXiv:1907.11692 (2019). ation for Computational Linguistics.; 53rd Annual [15] Y. Sun, S. Wang, Y. Li, S. Feng, X. Chen, H. Zhang, Meeting of the Association for Computational LinX. Tian, D. Zhu, H. Tian, H. Wu, Ernie: En- guistics and the 7th International Joint Conference hanced representation through knowledge integra- on Natural Language Processing of the Asian Federtion, arXiv preprint arXiv:1904.09223 (2019). ation of Natural Language Processing, ACL-IJCNLP [16] M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, 2015 ; Conference date: 26-07-2015 Through 31-07O. Levy, Spanbert: Improving pre-training by rep- 2015. resenting and predicting spans, Transactions of the [28] M. Sachan, K. Dubey, E. Xing, M. Richardson, Association for Computational Linguistics 8 (2020) Learning answer-entailing structures for machine 64–77. comprehension, in: Proceedings of the 53rd [17] I. J. Goodfellow, J. Shlens, C. Szegedy, Explaining Annual Meeting of the Association for Comand harnessing adversarial examples. iclr’15, arXiv putational Linguistics and the 7th International preprint arXiv:1412.6572 (2015). Joint Conference on Natural Language Processing [18] T. Miyato, A. Dai, goodfellow, ij adversarial train- (Volume 1: Long Papers), Association for Coming methods for semi-supervised text classification, putational Linguistics, Beijing, China, 2015, pp. in: International Conference on Learning Represen- 239–249. URL: https://aclanthology.org/P15-1024. tations (ICLR), Toulon, France, 2017. doi:10.3115/v1/P15- 1024. [19] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, [29] H. Wang, M. Bansal, K. Gimpel, D. McAllester, A. Vladu, Towards deep learning models resistant to Machine comprehension with syntax, frames, and adversarial attacks, arXiv preprint arXiv:1706.06083 semantics, in: Proceedings of the 53rd Annual (2017). Meeting of the Association for Computational Lin[20] D. Zhang, T. Zhang, Y. Lu, Z. Zhu, B. Dong, guistics and the 7th International Joint ConferYou only propagate once: Accelerating adversar- ence on Natural Language Processing (Volume 2: ial training via maximal principle, arXiv preprint Short Papers), Association for Computational LinarXiv:1905.00877 (2019). guistics, Beijing, China, 2015, pp. 700–706. URL: [21] A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, J. Dick- https://aclanthology.org/P15-2115. doi:10.3115/ erson, C. Studer, L. S. Davis, G. Taylor, T. Gold- v1/P15- 2115. stein, Adversarial training for free!, arXiv preprint [30] S. Hochreiter, J. Schmidhuber, Long short-term arXiv:1904.12843 (2019). memory, Neural computation 9 (1997) 1735–1780. [22] W. G. Lehnert, The Process of Question Answering., [31] K. M. Hermann, T. Kociský, E. Grefenstette, L. Espe
Ph.D. thesis, USA, 1977. AAI7728146. holt, W. Kay, M. Suleyman, P. Blunsom, Teach[23] A. Peñas, E. Hovy, Semantic enrichment of text ing machines to read and comprehend, CoRR with background knowledge, in: Proceedings of abs/1506.03340 (2015). URL: http://arxiv.org/abs/ the NAACL HLT 2010 First International Workshop 1506.03340. arXiv:1506.03340. on Formalisms and Methodology for Learning by [32] P. Rajpurkar, J. Zhang, K. Lopyrev, P. Liang, SQuAD: Reading, Association for Computational Linguistics, 100,000+ questions for machine comprehension Los Angeles, California, 2010, pp. 15–23. URL: https: of text, in: Proceedings of the 2016 Confer//aclanthology.org/W10-0903. ence on Empirical Methods in Natural Language [24] Z. S. Harris, Distributional structure, Word 10 Processing, Association for Computational Lin(1954) 146–162. guistics, Austin, Texas, 2016, pp. 2383–2392. URL: [25] P. Ruch, R. Baud, A. Geissbühler, Evaluating and re- https://aclanthology.org/D16-1264. doi:10.18653/ ducing the efect of data corruption when applying v1/D16- 1264. [33] Z. Zhang, J. Yang, H. Zhao, Retrospective reader for machine reading comprehension, 2020.
arXiv:2001.09694. [34] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit,
L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in neural information processing systems, 2017, pp. 5998–6008. [35] H. Yong, J. Huang, X. Hua, L. Zhang, Gradient centralization: A new optimization technique for deep neural networks, in: European Conference on
Computer Vision, Springer, 2020, pp. 635–652. [36] S. Y. R. J. F. D. T. H. N. Amir Pouran Ben Veyseh, Nicole Meister, MACRONYM: A LargeScale Dataset for Multilingual and Multi-Domain
Acronym Extraction, in: arXiv, 2022. [37] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101 (2017). [38] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).