Acronyms and long-forms are commonly found in research documents, more so in documents from scientific and legal domains. Many acronyms used in such documents are domain-specific, and are very rarely found in normal text corpora. Owing to this, transformer-based NLP models often detect OOV (Out of Vocabulary) for acronym tokens, especially for non-English languages, and their performance sufers while linking acronyms to their long forms during extraction. Moreover, pre-trained transformer models like BERT are not specialized to handle scientific and legal documents. With these points being the overarching motivation behind this work, we propose a novel framework CABACE: Character-Aware BERT for ACronym Extraction, which takes into account character sequences in text, and is adapted to scientific and legal domains by masked language modelling. We use an objective with an augmented loss function, adding max loss and mask loss terms to the standard cross-entropy loss for training CABACE. We further leverage pseudo labelling and adversarial data generation to improve the generalizability of the CABACE framework. Experimental results prove the superiority of the proposed framework in comparison to various baselines. Additionally, we show that the proposed framework is better suited than baseline models for zero-shot generalization to non-English languages, thus reinforcing the efectiveness of our approach. Our team BacKGProp secured the highest scores on the French dataset, second-highest on Danish and Vietnamese, and third-highest in English-Legal dataset on the global leaderboard for the acronym extraction (AE) shared task at SDU AAAI-22.
As a result, it is important that NLP systems built for scientific document understanding take this into to understand acronyms and their provenance. For example, in Fig 1, the long-form, Language Neural System is introduced to the reader, and it is referred to by an acronym, LNS thereafter. Systems that can identify acronyms within the passage, and can further link these to their corresponding
Acronyms are short forms used to represent a longer sequence of words in documents, for brevity. Most commonly, they are generated by joining the starting letter/letters of each word in their long-form.
Such shorthand notations help writers save space and avoid redundant mentions of long-forms in text, especially in scientific papers, which have a page/word limit. A large number of acronyms that occur in scientific and legal documents are domainspecific and are almost never found in common text corpora. As a result, these acronyms often go into the (OOV) – Out of Vocabulary category of NLP models, especially for languages other than English. However, these acronyms quite often form the subject of sentences and play a crucial role in document understanding or text analytics in the scientific domain [ 1]. permitted under
Creative Commons License Attribution 4.0 International (CC BY 4.0). long-forms, if present, would enable models to com- proposed model achieved the highest score on the prehend scientific documents much better. The ease French dataset, and the second highest on Danish of availability of scientific documents on arXiv 1 and and Vietnamese, third highest on Legal English, other open-access archives have led to increased and fourth on Spanish and Scientific English on the research interest on scientific documents [ 1, 2, 3]. AE Shared Task [6]2 at SDU AAAI-22. We make
Although acronym extraction has been studied in our code publicly available3. the past, the datasets and models used have mostly To summarize, the key contributions of this work been limited to the biomedical domain, overlooking are: challenges in other domains such as science or legal. With this very objective in mind [1] made the • We propose a novel framework CABACE largest manually annotated acronym identification that leverages mBERT and aggregates charand disambiguation dataset publicly available at acter embeddings using CNN and maxthe SDU Workshop at AAAI-21, which saw sys- pooling to form character-aware token emtems improving over the baselines. The superiority beddings, which are used along with mBERT of human performance in comparison to the best token representations. We also inject domain system proposed was highlighted, validating the knowledge via masked language modelling scope of future research. At SDU-AAAI-22, the performed on scraped data, which is shown acronym extraction task is extended to 6 diferent to improve AE performance. languages, namely English, French, Spanish, Danish, • We further use an objective with an augVietnamese and Persian, making it the first pub- mented loss function, adding max and mask licly available multilingual benchmark for acronym loss terms in addition to standard crossextraction on scientific documents [ 4] . Given that entropy loss, that results in improved permultilingual NLP/low-resource NLP is starting to formance when paired with CABACE. We pick up pace, such a multilingual benchmark fur- use pseudo labelling and adversarial data ther enables us to test the eficacy of systems on a generation to improve model generalizability. diverse set of languages. • We test the zero-shot generalization eficacy
In this paper, we elucidate our unified framework of CABACE across languages and show that CABACE: Character-Aware BERT for ACronym it performs better than vanilla mBERT. To Extraction, adapted for 5 languages (all provided the best of our knowledge, this is the first in the shared task, except Persian), modelling the work reporting zero-shot eficacy of acronym acronym extraction (AE) task as a sequence la- extraction systems on a multilingual benchbelling problem. As mentioned earlier, many of the mark. acronyms in scientific text rarely occur in the vocab- • We perform extensive experiments on the AE ulary of BERT/Multilingual BERT (mBERT) [5]. task of SDU-22, achieving state-of-the-art reTowards this end, we utilise character sequence infor- sults in both normal and zero-shot settings, mation of tokens, and aggregate individual character demonstrating the efectiveness of our apembeddings using a convolutional neural network proach. (CNN) layer, followed by max-pooling. In this way, we inject character sequence information into our model along with mBERT token embeddings. We leverage domain-specific language modelling to en- 2. Related Work rich mBERT embeddings with domain knowledge, and further improve model generalizability using Models trained for acronym extraction tasks, like pseudo labelling and adversarial data augmenta- most NLP approaches, can be divided into three mation. Additionally, we perform masking of tokens jor categories: 1) rule-based methods [7, 8], 2) mawith positive labels (acronyms or long-forms) to chine learning methods that use text-based features encourage CABACE to pay higher attention to the [9, 10], 3) deep learning methods. [11] used BERT context, and use an objective with an augmented and SciBERT with BIO less tagging and blending in loss function, adding max-loss and mask-loss terms an ensemble framework for acronym identification. to the standard cross entropy loss. Finally, we [12] experimented with multi-task learning, feature also evaluate and provide our system performances engineering and CRF, and found that feature-based on zero-shot transfer from English to the French, methods handled the task well. With the advent Spanish, Danish and Vietnamese datasets. Our
3https://github.com/nitkannen/BacKGProp-AAAI
4. Methodology of large pretrained language models such as BERT [5], which is a bidirectional multi-layer transformer encoder that achieved state-of-the-art results on a Given the input sentence, our objective is to jointly number of benchmarks at the time, most NLP tasks predict the character span of acronyms and long have seen use of transformers-based architectures. forms present in the text. Towards this goal, we [13] used the BERT model with adversarial samples ifrst explain how we formulate the problem. Then that were created by perturbing existing inputs such we go into the details of the proposed CABACE that the loss of the model would increase on those framework, following which we describe techniques samples, to improve the robustness of BERT. Their we adopted for robustness and to reduce overfitting, system was the winning solution to the acronym i.e., pseudo-labelling and adversarial data generaidentification shared task at SDU AAAI-21. tion. 4.1. Problem Formulation
With the given dataset being annotated with character span indices of acronyms and long-forms, we The organizers of SDU-22 provide acronym extrac- preprocess the dataset to convert it into sequence tion (AE) datasets for the shared task in 6 lan- tags to model it as a sequence labelling problem. guages, namely English, Spanish, French, Danish, Specifically, we use the BIO tagging scheme [ 14] for Vietnamese, and Persian There are two distinct labelling all tokens in the target sequence. We have splits to the English data, one for texts from the 5 possible label tags: 0) O-None, 1) B-Acronym, 2) scientific English domain and one for texts from the I-Acronym, 3) B-Longform, 4) I-Longform, where legal English domain. All datasets are provided in ’B’ stands for begin, and ’I’ stands for inside. The the form of json files, with each individual data- ifrst token for an acronym/long-form would be given point having raw text, an ID, and a list of ground a B-label, and the rest would be given I-labels. For truth acronyms and long-forms present in the raw all our experiments, we use mBERT that uses the text. Ground truth acronym and long-forms for WordPiece tokenizer, whose property we leverage for each datapoint are provided as separate lists, with converting character spans to BIO-tags as explained elements in the list being of the form – [starting in Algorithm 1. character index, ending character index] for each The evaluation script provided by the task orgaacronym/long-form in the text. The objective of nizers uses character spans to calculate eval. metthe acronym extraction task is to extract character rics, so we convert the sequence tags back to charspans for each identified acronym and long-form in acter span index after model prediction. For this, given sentences. Table 1 lists key statistics of the we leverage ofset map returned by BertTokenizer. datasets provided in 5 languages. More specifically, for each positive prediction by our MaxPool CNN
MaxPool CNN
Character Embeddings
Linear
Linear
Linear
Linear Pred.
Concat.
MaxPool CNN
Mean + Max + Mask Loss Pred.
Concat.
MaxPool .…….
CNN
Pred.
Pred.
Concat.
Concat.
.…….
.…….
CLS ECLS CLS
T1 E1
T 21 E 21
T 31 E 31 mBERT
T 41 E 41 .……. ..…………..
STE1P
ESEE1P TOK1
TOK2
TOK3
TOK4 .……. SEP Consumer
Price
Index (CPI) is the .……. character (using character embeddings). The resulting outputs from both are concatenated and passed through a prediction layer (linear + softmax) before computing the augmented loss function. Note that the token ’(CPI)’ gets split into sub-words by mBERT tokenizer. Masked Language Modelling
Domain Specific Corpus Algorithm 1 Conversion of character spans to BIO 1: 2: 4: 5: 6: 7: 8: 9: tags Input: Output:
, 3: for each _ = = [0] * len( ( in _
= ( for each i in (()) if then end if end for 10: end for 11: return target , ) )
_ _) do do [i : i + len( _
)] = _ FillBIO(i, i + len( _ ),
) model (either acronym or long form), we use the token’s span to finally compute the character span range of predicted acronyms/long forms. 4.2. The CABACE Framework We now present CABACE, Character-Aware BERT for ACronym Extraction. Our proposed framework takes as input the tokens of the sentence, and classiifes each token into one of the 5 possible labels. For this we augment mBERT with the following components as in Fig. 2: 1) Masked language modelling, to inject domain specific knowledge into mBERT to make it better equipped to handle scientific documents, 2) Character-aware token embeddings, to provide information about character sequences in each token, 3) Augmented loss and masking, motivated by the presence of rarely used notations as acronyms. We describe these components in great detail in the upcoming sections. 4.3. Domain-Specific Language
Modelling To inject scientific domain knowledge into mBERT, which is otherwise not a significant portion of mBERT’s pretraining, we perform domain-specific language modeling. For this, we scrape sentences containing acronyms from the web. For scientific English, we scrape data from the arXiv API4 and for French, Spanish, and Danish, we scrape data from Wikipedia. Then, from the scraped sentences, we iflter out those sentences which contain a probable acronym or long-form. A sentence containing a word with more than 50% capitalized letters was used as the criteria to detect sentences with a probable acronym and long-form. We also use the acronym identification data from last year’s SDU-21.
We merge all these datasets for each language with the 4https://arxiv.org/help/api/ datasets provided for this shared task. The total which likely contains the most important character number of sentences thus obtained is 25009 for Enn-gram signal from the token (where ’n’ refers to glish Scientific, 4455 for English Legal, 19769 for the CNN filter size). Let for 6 epochs. The resulting model weights are saved layer for classification: = [
1, 2, .... −1 , ] where is a sentence with k tokens. The computation for each token can be summarized by the following equations, where ℎ̃ is passed into a linear French, 17411 for Spanish, 15196 for Danish and 1593 for Vietnamese. With this data, we perform masked language modeling (MLM), following the same procedure as described in [5].
We start the training from public mBERT checkpoint and train and are an integral part of the CABACE framework.
We use these LM checkpoints with CABACE in all our experiments unless mentioned otherwise.
Each author must be defined separately for accurate metadata identification. Multiple authors may share one afiliation. Authors’ names should not be abbreviated; use full first names wherever possible. Include authors’ e-mail addresses whenever possible. 4.4. Character-Aware Token Embeddings A variety of acronyms found in scientific documents often have repeated sufices/prefixes common to them that can be important signals used to detect IIT and NIT have common sufices that can be overlooked if WordPiece tokenizer of mBERT5 fails to segment out the common sufix ’IT’ that serves as an important signal to detect the acronym. Apart from this, characters/sub-tokens that are out of mBERT’s vocabulary get mapped to the [UNK] token, where important case-specific or character specific information crucial for the task can be lost. Such cases or more common in low-resource languages like Vietnamese. Motivated by these shortcomings, we propose to additionally inject character-aware token embeddings that are concatenated with mBERT ifnal layer token embeddings before being passed to a prediction layer.
With reference to Fig. 2, we pass the character embeddings of each character of a token to a convolutional neural network (CNN) layer of a fixed iflter size. For pre-trained character embeddings, we use the FastText library6, the dimension of the character embeddings being 300. We pad each input token to a constant character length. Hence, for a token of character length 16 (where max pad length is also 16), we would pass a 16 X 300 dimensional vector ∈ R16×300 into the CNN. We then use a maxpooling layer that picks out the CNN filter output that contains the highest signal. This way, we get one embedding vector for each token in the input,
5https://huggingface.co/bert-base-multilingualcased/blob/main/tokenizer.json
6https://fasttext.cc/ such acronyms. For example, the acronyms MIT, applicable to long forms as well, although with less ̃ = ( ( ̃ = ( ) ℎ̃ = ̃ || ̃
)) 4.5. Augmented Loss and Masking Compared to commonly used words, acronyms occur with very low-frequency in the training corpora, and with even lower frequency in the test corpora.
As a result, the model must be encouraged to pay higher attention to the context around acronyms to reduce the dependence on the acronym token itself during prediction. Such a feature could be significance. Towards this end, we randomly mask 10% of tokens with positive labels (B- or I- tokens) during the training phase.
This encourages the model to rely less on the token itself, and more on the context leading up to an acronym/long form.
Inspired by the successful amendment to the loss function in previous works [15], we additionally append a max term that adds the maximum loss value across all token labels from a particular example to the standard cross entropy loss. This ensures that the model learns more from wrong predictions with high losses, as opposed to an uniformly weighted loss that doesn’t special pay attention to the token with the highest loss. Let ℒ = [( 1), (
2), ..., ( where Loss(.) is given by: ( ) ̂ = − ( )]
) ̂ Our new objective function contains a weighted addition of max and mask loss terms along with cross entropy.
= (ℒ ) = (ℒ ) =
∑ =[ MASK] ( ) The max term is weighted by term is weighted by
and the mask which are hyperparameters. The overall augmented loss is given by the following equation. = + + 4.6. Pseudo-Labelling Pseudo-labelling [16] is where additional data is created by running a trained model on unseen data and using the model’s high-confidence predictions on the unseen data as ground-truths, adding them to the training dataset. We train our models from the public mBERT checkpoint on the training dataset of the original SDU-22 dataset. The trained models are used on the unlabeled scraped data to identify acronyms and long-forms in them. We append the datapoints with high-confidence predictions back to our training set. Pseudo-labeling is a very useful semi-supervised data generation technique [17] that helps particularly when working with low-resource datasets. metrics and implementation details, followed by experiments to test cross-lingual zero-shot eficacy. 5.1. Baselines We compare our proposed model with 3 diferent baselines. These are explained in the coming sections: 5.1.1. Rule-Based: We report the results obtained by the rule-based baseline provided by the organizers of the shared task7. This system uses hand-picked rules for extracting acronyms and long-forms. Words that have >60% capitalization are selected as acronyms, and if the initial characters of the preceding words before an acronym can form the acronym, those words are selected as the long-form.
Inspired by the success of BERT [5] for sequence 4.7. Adversarial Data Generation labelling in acronym identification shared task of SDU-21 [13], as well as its ability to efectively We perform adversarial data generation for the En- aggregate contextual information from texts, we glish datasets. Adversarial training is a useful tech- consider it as a baseline in our experiments. We nique that enhances the robustness [18] of models run our inputs through mBERT, and use mBERT’s by adding adversarial samples to the training data. final layer tokens, which are then passed to a linear An adversarial example is an instance with small, classifier followed by softmax to generate 1 out of intentional feature perturbations that induces the the 5 BIO tags as prediction for that token. Problem model to make a false prediction [19]. In the proce- formulation and pre-processing is the same as in dure of adversarial training, input samples are first CABACE. We use Multilingual-BERT (bert-basemixed with some small perturbations to generate multilingual-cased)8 and refer to this model in our adversarial samples. The model is then trained with experiments as Vanilla mBERT. both the original input sample and generated adversarial samples [13]. We use the embedding function 5.1.3. Seq-to-Seq: of the augmenter class from TextAttack [20] to generate text by replacing words with neighbors in the Previous works have reported sequence to secounter-fitted embedding space consisting of GloVe quence approach using the encoder-decoder archivectors, with a constraint to ensure their cosine tecture as an alternative for the sequence tagging similarity is at least 0.8. One such example is: scheme [21, 22]. Following these works, we use a Original Text: "We conduct a showcase study of transformer-based generative framework to autodialectal language in online conversational text by regressively decode the acronyms and long-forms investigating African-American English (AAE) on present in input sentence. For the example given Twitter." in Fig.1, the ground truth target sequence would Generated Adversarial Example: "We conduct a follow the template string: "<Acronyms> LNS case study of dialectal language in online conversa- <Long-Forms> Language Neural System", where tional text by scrutinize African-American English "<Acronyms>" and "<Long-Forms>" contain the (AAE) on Twitter." predicted acronyms and long-forms from the sentence, separated by a comma. We decode the output by searching for occurrences of the predicted 7https://github.com/amirveyseh/AAAI-22-SDUWe compare our model with 3 baselines explained shared-task-1-AE/blob/main/code/baseline.py in the next section. We then go over the evaluation 8https://huggingface.co/bert-base-multilingualcased
5.2. Zero-Shot Transfer to Non-English
Languages
Table 3 lists comparative results between CABACE Multilingual models like mBERT have a shared em- and the Rule-based model, the Seq-to-seq (mT5) bedding space across languages, which they leverage and Vanilla mBERT baselines. We find that to learn language-agnostic properties of the task CABACE improves significantly upon the three along with language-specific features. Zero-shot baselines in all 6 datasets we evaluate the models transfer is a useful way to test how well multilingual models generalize to unseen languages [24].
To this end, we conduct experiments by training Table 3 models on a combined English-Legal + English- Comparison of CABACE with the three baseline models. ReScientific dataset, and testing their performances ported scores are on the dev set. CABACE is seen to outperon the other languages, i.e., Danish, French, Span- form all baseline methods in all datasets. ish and Vietnamese datasets. We comparatively Datasets Model Precision Recall F1 evaluate Vanilla mBERT and CABACE this way to test their zero-shot eficacy. Danish 5.3. Evaluation Metrics The metrics reported here is identical to the ones provided by the organizers of the shared task9. The metrics used were precision, recall and F1-score for the acronyms and long-forms, both individually and combined together. The script uses exact match to pair predicted spans with the gold spans. 5.4. Implementation Details All experiments were done using Pytorch 1.10.0+cu11.1 on Google Colab GPUs (NVIDIA Tesla P100-PCIe-16GB). For the mBERT and mT5 checkpoints, we used the HuggingFace transformers 4.12.5 library10. We used the base
9https://github.com/amirveyseh/AAAI-22-SDUshared-task-1-AE/blob/main/code/scorer.py 10https://huggingface.co/transformers/
Languages upon. Note that the improvement with CABACE in a language like Vietnamese is much more than in English-Legal. This can be attributed to the fact that mBERT classifies many of the tokens in Vietnamese into [UNK], consequently losing crucial information. Injecting character embeddings for these tokens enables CABACE to perform much Table 6 better than Vanilla mBERT. Vanilla mBERT per- CABACE ablation study on English-Scientific forms better than the Seq-to-seq model, which in Ablations Precision Recall F1 turn ups the rule-based model’s performance, in all datasets. Ours 0.8282 0.8876 0.8509
Table 4 shows fine-grained results of the CABACE Ours w/o max loss 0.8223 0.8811 0.8450 architecture on the 6 datasets – precision, recall and Ours w/o mask loss 0.8330 0.8822 0.8503 F1-scores for acronyms and long-forms are provided Ours w/o Char Embd 0.8254 0.8729 0.8485 separately. We observe that CABACE is able to Ours w/o LM 0.8073 0.8658 0.8355 perform very well on extracting acronyms, but loses some points while detecting long-forms, which could be due to lack of components in the architecture drops compared to in-dataset performance. Possible that are specialized to focus on the acronym-long reasons could be due to lexicons and grammar in form interactions during extraction. Recall scores these languages being relatively distant to those are seen to be always higher than precision scores, in English, and relative similarity of Spanish and except on the Vietnamese dataset. English.
A comparison of Vanilla mBERT vs CABACE for zero-shot performance can be found in Figure 3. We ifnd that CABACE outperforms Vanilla mBERT 6.1. Ablation Study on CABACE in 3 out of 4 languages for cross-lingual zero-shot We perform an ablation study to better interpret transfer. Surprisingly, the zero-shot performance in how individual components contribute to perforSpanish using both models isn’t much less than per- mance improvements. The steps we perform include formances of the models when trained on the Span- comparing our full model (CABACE), CABACE ish dataset. However, languages like Vietnamese without the max loss component, CABACE without and Danish see significant zero-shot performance the mask loss component, CABACE without the character embeddings, and CABACE without language modelling. We perform the ablation study on the French dataset (where we top the leaderboard) and on the English-Scientific dataset. Both these studies show similar trends. As seen in tables 5 and 6, removing augmented loss i.e. max and mask loss causes a significant drop in the scores, and so does removing the character embeddings. Not including domain-specific language model checkpoints leads to drops in scores too, as expected.
In this paper, we introduce the CABACE framework for acronym and long-form extraction that integrates character-level information with mBERT representations and uses domain-specific language modelling and an augmented loss function, with pseudo labelling and adversarial data generation for improved generalizability. Experimental results establish the supremacy of our framework over several baselines on 6 datasets spanning 5 languages. We also evaluate zero-shot cross-lingual eficacy of our proposed model and find that it outperforms baseline mBERT results in 3 out of 4 cases. Our system merits the top spot in French, second place in Danish and Vietnamese and third place in Legal English leaderboards on the AE shared task at SDU AAAI-22. [15] Q. Wu, Z. Lin, G. Wang, H. Chen, B. F.
Karlsson, B. Huang, C.-Y. Lin, Enhanced meta-learning for cross-lingual named entity recognition with minimal resources, 2020. a r X i v : 1 9 1 1 . 0 6 1 6 1 . [16] A. Iscen, G. Tolias, Y. Avrithis, O. Chum, Label propagation for deep semi-supervised learning, 2019. a r X i v : 1 9 0 4 . 0 4 7 1 7 . [17] P. Cascante-Bonilla, F. Tan, Y. Qi, V. Ordonez, Curriculum labeling: Revisiting pseudolabeling for semi-supervised learning, 2020. a r X i v : 2 0 0 1 . 0 6 0 0 1 . [18] I. J. Goodfellow, J. Shlens, C. Szegedy, Explaining and harnessing adversarial examples, 2015. a r X i v : 1 4 1 2 . 6 5 7 2 . [19] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, R. Fergus, Intriguing properties of neural networks, 2014. a r X i v : 1 3 1 2 . 6 1 9 9 . [20] J. Morris, E. Lifland, J. Y. Yoo, J. Grigsby, D. Jin, Y. Qi, Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020, pp. 119–126. [21] S. Almasian, D. Aumiller, M. Gertz, Bert got a date: Introducing transformers to temporal tagging, 2021. a r X i v : 2 1 0 9 . 1 4 9 2 7 . [22] H. Yan, J. Dai, T. ji, X. Qiu, Z. Zhang, A unified generative framework for aspect-based sentiment analysis, 2021. a r X i v : 2 1 0 6 . 0 4 3 0 0 . [23] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, C. Rafel, mt5: A massively multilingual pre-trained textto-text transformer, 2021. a r X i v : 2 0 1 0 . 1 1 9 3 4 . [24] H. Choi, J. Kim, S. Joe, S. Min, Y. Gwon, Analyzing zero-shot cross-lingual transfer in supervised nlp tasks, 2021. a r X i v : 2 1 0 1 . 1 0 6 4 9 .