Acronym extraction aims to find acronyms (i.e., short-forms) and their meanings (i.e., long-forms) from the documents, which is important for scientific document understanding (SDU@AAAI-22) tasks. Previous works are devoted to modeling this task as a paragraph-level sequence labeling problem. However, it lacks the efective use of the external knowledge, especially when the datasets are in a low-resource setting. Recently, the prompt-based method with the vast pre-trained language model can significantly enhance the performance of the low-resourced downstream tasks. In this paper, we propose a Prompt-based Sequence Generation (PSG) method for the acronym extraction task. Specifically, we design a template for prompting the extracted acronym texts with auto-regression. A position extraction algorithm is designed for extracting the position of the generated answers. The results on the acronym extraction of Vietnamese and Persian in a low-resource setting show that the proposed method outperforms all other competitive state-of-the-art (SOTA) methods.
Input: Existing methods for learning with noisy labels (LNL) primarily take a loss correction approach.
Output: Existing methods for learning with noisy labels (LNL) primarily take a loss correction approach.
Label:{ ‘acronym’: [ 49 : 52 ], ‘long-form’ : [ 21 : 47 ]} With the development of technology and global informatization, the number of acronyms is increasing rapidly.
The forms of the acronyms are complicated and change- Figure 1: Examples of English acronym extraction.
able because of various languages [
In addition, the carefully designed document reading
system should be able to recognize the correct meaning of
acronyms and their long forms so that these documents
can be processed correctly. This is fairly critical for a
variety of downstream tasks, such as question and answer
[
The acronym extraction task is mainly used to extract
acronyms (i.e., short forms) and their meanings (i.e., long
forms) in the science document [
Recent works tend to model this task as a sequence labeling task [9, 10, 11], which helps the model to capture the local definition of acronyms in the document.
However, the form of acronyms not only appears in the
English scene but also in other language scenes
(multilingual). Traditional sequence labeling methods are weak
in utilizing external knowledge [
In this paper, we propose the Prompt-base Sequence Generation (PSG) method for acronym extraction. Specifically, we design a prompt for sequence generation with the large-scale pre-trained model. With the prompt being used as a template, the acronym form and the long-form can be generated via auto-regression. After obtaining the answer, we designed a position extraction algorithm to locate the labels. The proposed method ranks 1-st under the low-resource language setting (i.e., Vietnamese and Persian) in the shared task 1 of the SDU@AAAI-22, which outperforms all other competitive methods. The main contributions are summarized as follows: • As far as we know, this is the first attempt to adopt prompt-based sequence generation for the acronym extraction task. • We propose a novel acronym extraction method, including prompt-based sequence generation method and position extraction algorithm for obtaining the final labels. • Extensive experiments are conducted on the lowresource datasets (i.e., Vietnamese and Persian).
The results demonstrate the efectiveness of our proposed method compared with other competitive baselines.
We treat the acronym task as a sequence generation problem. Given a series of tokens in the text x = {1, 2, . . . , }, this task aims at finding the corresponding position from the original text. The label indi- 3. Method cates the short-form (i.e., acronym) and the long-form (i.e., phrase). We formulate the above process as fol- In this section, we will introduce our method in detail, inlows: cluding the model architecture, prompt design, sequence , = ℎ (1, 2, . . . , ) (1) generation and position extraction algorithm.
As shown in Table 2, the Persian dataset is divided into training (1336), development (167), and testing according to the data set (160). The training and validation sets of the above two datasets have been manually labeled, where the label is a list of position boundaries. where ℎ is the model which extracts the answers.
The submitted results will be evaluated based on the macro-averaged precision, recall, and F1 scores on the online test set. The final scores represent the prediction correctness of short-form (i.e., acronym) and long-form (i.e., phrase) boundaries in the sentences. The short-form or long-form boundary prediction is counted as correct if the beginning and the end of the predicted short-form or long-form boundaries are equal to the ground-truth beginning and end of the short-form or long-form boundary, respectively. The oficial score is the macro average of short-form and long-form F1 scores.
This acronym extraction task consists of various
multilingual datasets composed of document sentences in
science fields [
labels Encoder Encoder Encoder
Decoder Decoder
Decoder Output MT5
Input Existing methods for learning with noisy labels (LNL) primarily take a loss correc‐ tion approach. The acronyms and their
meanings are:
The overall architecture of our me-thod is shown in
Figure 2, the MT5 model [
We manually design the prompt to extract relevant knowledge from the pre-trained model for sequence generation, which is presented as the fixed tokens, i.e., “The acronyms and their meanings are:”. In addition, the unused tokens are adopted as a placeholder to control the outputs. Specifically, the unused tokens are used as the placeholder to form a template for prompt tuning, where the <extra_id_1> represents the separator of the short forms, the <extra_id_2> represents the separator of the long forms, and the <extra_id_3> represents the separator between the acronym of the long-form and shortform. The <extra_id_4> indicates that no acronym of short-form appears, while the <extra_id_5> indicates that no acronym of long-form appears.
The sequence generation task is designed to generate extracted possible acronyms. The text x = (1, 2, . . . , ) is encoded through the encoder to obtain the context encoding . At the decoding stage, the loss function of the sequence generation task can be performed as auto-regression, which is shown as equation (2): ( ) = − ∑︁ log ( | 0, . . . , − 1, ) (2)
where the is the parameters of the model, the represents the i-th word generated by the decoder, and 0, . . . , − 1 is a sequence of previously generated tokens.
After the possible acron-yms of short-form and long-form are generated with sequence generation, the next step is to consider how to extract their positions. A good extraction algorithm determines the final quality of the generated result. We use a greedy traversal search method, adopting the regular method from left to right to find the corresponding location boundary. At the same time, we need to ensure there is no overlap in the extracted outputs by detecting the boundary margins so that the extracted positions are independent of each other. To take the acronym of short-form as an example, the algorithm is represented in Algorithm 1.
Algorithm 1 Position extraction algorithm Input: Output text y tokenized into list ; Original input text x Output: Short-form acronym position list 1: Let = [] 2: Let truncated text R = ∅ 3: if ‘<extra_id_4>’ in then 4: = [] 5: else 6: = y.find(‘ <extra_id_3>’) 7: R = y[:].strip() 8: for in R.split(‘ <extra_id_1> ’) do 9: for in [.start() for in re.finditer( , x)] do 10: .append([, + len()]) 11: end for 12: end for 13: end if 14: Making the position intervals in list independent
of each other
15: return solution
4. Experiment setup
4.1. Baseline models
• Rule-based method The baseline method
proposed by Schwartz is a rule-based method [8]. In
this baseline, the words that more than 60% of
their characters are uppercased are selected as
acronym. To select long-forms, if the initial
characters of the preceeding words before an acronym
can form the acronym they are selected as
longform. The related codes can be found on the
website1.
• BERT-CRF model The BERT-CRF [9]
architecture is composed of a BERT model [
The matrix contains two states: begin (B) and end (E). We only consider the cross-entropy loss of the first sub-token of each token. • BERT-Span model The task is also considered as the boundary of phrase spans modeled by BERTSpan model [10], including acronyms of shortform and long-form. Two binary classifiers are 1https://github.com/amirveyseh/AAAI-22-SDU-shared-task-1
Notice that the Persian is a right-to-left language, it is diferent from other languages. However, we deem that the whole training process is an auto-regressive task, where the pre-trained model can learn the features of multi-languages via self-supervised pre-training and finetuning. In short, the model can well produce the results with the proposed position extraction algorithm.
4.3. Implementation
adopted to output the multiple start and end
indexes. The prediction represents whether each
token is the start or end tag. Given each token
representation from the BERT model, the
probabilities of each token are predicted as the start or
end position. The weighted cross-entropy loss is
implemented to train the model with parameters
shared at the BERT encoder layers.
• MT5 model The MT5 [
The above models can be found and downloaded on the website2.
All models are implemented based on the open-source
transformers library of huggingface [
In BERT-CRF method, we initialize the model with
mbert [
Danish paragraphs in the legal domain. The training As for multi-lingual fine-tune and single language finesteps are as follows: tuning, we used the AdamW optimizer with an initial learning rate of 1e-4 and annealed it gradually after a warm-up epoch until it reached 1e-5. 1. The trained MT5 model is utilized to initialize the parameters of the encoder and decoder, and ifne-tune with the multi-lingual data. Finally, we train the model with 8 epochs. 2. We fine-tune the MT5 model with a single language for Vietnamese and Persian respectively with 8 epochs being trained. 2https://huggingface.co/models
The main results of our model and baselines are shown in Table 3 and Table 4, where the F1 performance in Vietnamese and Persian are presented respectively. It can be found that the pre-trained model has more advan- the National Natural Science Fund of China (62171183, tages than the rule-based method, since the rule-based 61801178). method has limited generalization capabilities in validation and test datasets. The BERT-CRF and BERT-Span methods have similar performance. This may be because References the sequence labeling method with pre-trained model has limited ability to capture external information in the unseen test dataset, especially in the low-resource setting. It is worth noting that the F1 scores of the method based on the generation method are higher than the sequence labeling method, which indicates that the generation method is more conducive to capturing the relationship between the acronym of the short-form and the long-form. What’s more, our method is higher than other baselines in both the validation set and the test set.
More precisely, on the Vietnamese test set, the proposed method reaches an F1 score of 0.8416, and on the Persian test set, the proposed method reaches a score of 0.7993.
Further conclusion can be found that: 1) compared with the traditional generation method (i.e., MT5), we have designed the prompt method, which is efective to make full use of the knowledge from the pre-trained model. As a result, the proposed method gets a good performance on the test set as its stronger generalization ability. 2) As the scale of the initial pre-trained model increases, the prompt can enhance the generalization of downstream tasks. This work is supported by the National Key Research and Development Project of China (2018YFB1305200) and