Easy-to-use combination of POS and BERT model for domain-specific and misspelled terms

Abstract

In this paper, we present BERT-POS, a simple method for encoding syntax into BERT embeddings without re-training or finetuning data, based on Part-Of-Speech (POS). Although fine-tuning is the most popular method to apply BERT models on domain datasets, it remains expensive in terms of training time, computing resources, training data selection and re-training frequency. Our alternative works at the preprocessing level and relies on POS tagging sentences. It gives interesting results for words similarity regarding out-of-vocabulary both in terms of domain-specific words and misspellings. More specifically, the experiments were done on French language, but we believe that they would be similar on others.

Keywords

• Natural Language Processing
• Language Models
• Semantic Similarity
• Out-of-Vocabulary Words
• Part-Of-Speech

-

For a variety of Natural Language Processing (NLP) tasks, state-of-the-art results have been reported with generic pre-trained language models, such as BERT [ 2 ] and other BERT-like models [ 14,19 ] or task-specific such as GPT [ 23 ] designed for automatic text generation. In these approaches, the pre-trained language models are applied to downstream machine learning tasks using task-specific ifne-tuning. Currently, Transformer models [ 29 ] are trained on diferent sets of generic data (i.e., books, news, Wikipedia, etc.) and are not adapted to domain datasets, both in terms of vocabulary or syntactic structure. Therefore, these models are not intended to be used as is but should be tailored to specific data sets. At the word level, two types of out-of-vocabulary (OOV) words must be correctly processed: application-specific and misspelled words. In this paper, we propose a novel method to improve semantic understanding of domain-specific data. To do this, we present BERT-POS, an easy-to-use technique to integrate external morpho-syntactic context into BERT-like architectures. The proposed method combines BERT with an automatic preprocessing stage which saves 1 Copyright ©2021 for this paper by its authors. Use permitted under Creative

Commons License Attribution 4.0 International (CC BY 4.0) computing time (i.e., Fast learning) and energy consumed (i.e., Green AI). The use of syntax combined with contextual models enable the addition of contextual characteristics in corpora that are dificult to process. The addition of morphosyntactic information allows to compensate for the dificulties related to the processing of OOVs by integrating a knowledge of sentence structure. BERT-POS is based on a pre-training technique that is not only robust on the processing of domain-specific terms but also on misspelled terms. This study is conducted on a French dataset through the CamemBERT model [ 19 ]. 2

Related Work

Fine-tuning language models The problem of adapting the language models was studied [ 24 ] and suggested that combining BERT with other neural networks obtained better results than fine-tuning BERT-like models, which was favored in other studies [ 16,27,33,3 ]. Specific models are shown to perform best when they are specific to the textual genre studied (i.e., SciBERT [ 1 ] and BioBERT [ 15 ]). However, pre-training BERT-like models can be computationally expensive and require having a dataset representative of the target data.

Words segmentation Some studies have shown that the decisions made by BERT tokenizers are dificult to explain when splitting words [ 25 ]. It was demonstrated that the processing of domain-specific OOV terms is strongly impacted by the splitting of the input terms of the model, leading to a significant decrease in the semantic understanding of the words [ 20 ]. Recent works on misspelling generation [ 26,28 ] proved that BERT is not robust on misspellings and performed significantly worse on downstream tasks.

Overcoming OOVs in BERT Several studies have worked on overcoming domain specific OOVs and misspellings in BERT. For instance, [ 4,17 ] proposed to construct representations at the character-level and obtained promising results for domain-specific terms. Other studies have tried to add external features to deal with misspellings such as a word-recognition module [ 22 ] or other strategies [ 5,8 ]. 3

Proposed Method

In this section, we propose BERT-POS, a preprocessing method for encoding morpho-syntactic information into BERT-like embeddings which does not require a complementary phase of fine-tuning [ 4,15 ]. Figure 1 presents the processing chain of our method. For this experiment, we chose CamemBERT because it used SentencePiece [ 13 ], which was easy to use when working with re-constructing words from sub-units. Nevertheless, we assume that this work could be easily applied to architectures that use WordPiece [ 32 ] such as BERT. First, the dataset was split into sentences or sequences of words when the sentences were too dificult to distinguish. Empirically, we split the documents into sequences of 150 tokens. The POS tagging step consists of concatenating each word with its POS using "_" character. Here is an example of annotating a sentence containing n words and m POS tags: word1_posa, word2_posb, word3_posa, . . . , wordn_posm. This annotation technique is commonly used for non-contextual models to disambiguate polysemous words which difer in their grammatical category. Here, our objective is to force the addition of morpho-syntactic information in the embeddings. For a given sentence, if the SentencePiece tokenizer does not recognize a word, it splits it into known sub-units. This creates problems with new sentence structures containing a lot of small words. In parallel, we encode a vector for each word and a vector for each POS tag. For every word, a vector is generated by computing the sum of the sub-vectors associated with the sub-tokens of the words. The same process is done with tags and subtags. We made sure that all the POS tags were not recognized as words so that a unique embedding is re-constructed for each tag. Finally, we computed the average of each occurrence of the pairs {wordi, posj } to construct an unique vector for each word of the corpus. 3 This work is part of a broader study for Electricité De France (EDF) with the aim of improving a classification system. EDF is the leading electricity supplier in France.

Moreover, it contains Energy-specific vocabulary which can be existing words in French or words belonging to the specific domain. Table 1 contains several examples of misspellings, SMS language and domain terms that exist in the corpus. The distribution of POS tags in this corpus, obtained with spaCy4 [ 7 ], is described in Figure 2b.

Dataset

EASY EDF-Emails

Domain #docs Diverse genres 765 Customer emails 99 993

INTJ NUM SYM CSCCOONNXJJ AUX ADV PROPN PRON ADJ VERB DET ADP PUNCT NOUN 0 2,000,000 4,000,000 (a) Description of the datasets

(b) Distribution of POS tags in EDF-Emails 4 We randomly selected and manually annotated the first 300 tokens of EASY and EDFEmails datasets and compared the results obtained with spaCy (fr_core_news_lg ) to calculate a POS tagging accuracy for the respective datasets: 0.95 and 0.83. 5 We worked with the models implemented in the transformers library [ 31 ]. The models were downloaded on May 2021. – Oscar [ 19 ] is a set of monolingual corpora extracted from Common Crawl.

It was selected using a classification model for each language following the approach of [ 6 ] based on FastText [ 12 ]. The classifier was previously pretrained on Wikipedia, Tatoeba and SETimes, and covering 176 languages. – CCNet [ 30 ] is a dataset extracted from Common Crawl but with a diferent filtering from that of Oscar. It was built with a language model using Wikipedia, thus allowing it to filter out noise (code, tables, etc.). CCNET thus contains documents longer on average than Oscar. – Wikipedia is a homogeneous corpus in terms of genre and style which was preprocessed using WikiExtractor.

Models

#layers Dataset Size (GB) In this section, we aim to assess the impact of the training dataset on language models, to analyze its importance in terms of quality and distance towards the applicative dataset. in Table 2. The more tokens are obtained for a single word, the less the model is semantically accurate. We note that for EASY and EDF-Emails, the Wikipedia corpus is the furthest one regarding lexical proximity. It could be explained because the dataset is the lexically poorest from the ones extracted from the web or because our domains of applications are more present in the web-extracted corpora. This result is very relevant, because it shows that the level of cleanliness of the learning corpus (i.e., construction of sentences, order of words, etc.) is not more important than the proximity to the application corpus. Moreover, there is no diferences when using CamemBERT using OSCAR than CCNet, which implies that the pre-processing step of CCNet does not have any impact on our datasets. Therefore, we will not use CamemBERT’s CCNet model in further analysis. The vocabulary of the EASY dataset is known, at best, at 70% while the one from EDF-Emails is only understood at 20%. Those major diferences are expected to be seen while computing similarity, as discussed in Section 6.3. Examples of tokenization with CamemBERT’s models is presented in Table 3, using four frequent words in EDF-Emails: domain-specific (i.e., meter, linky and refund) and Emails-specific (i.e., cordially). The domain-specific words exist in all models’ vocabularies, except for "linky" (i.e., a French electric meter proposed by EDF), which does not exist in general French language. Interestingly, we observe that the Wikipedia model tokenize this word diferently than the others. The segmentation of OOV words is purely based on statistics rather than linguistic properties [ 20 ]. This can lead to a loss of semantics when reconstructing words after their tokenization. Indeed, we expect to obtain diferent words surrounding "linky" when using CamemBERT’s Wikipedia compared to the others, due to the sub-units obtained following the tokenization.

Word

linky remboursement cordialement

Model wikipedia

others wikipedia

others wikipedia others

Tokens [”_l”, ”in”, ”ky”] [”_l”, ”ink”, ”y”] [”_rem”, ”bour”, ”s”, ”ement”]

[”_remboursement”] [”_cord”, ”iale”, ”ment”] [”_cordialement”] (a) EASY Dataset - CamemBERT (top) and CamemBERT-POS (bottom) (b) EDF-Emails Dataset - CamemBERT (top) and CamemBERT-POS (bottom) that we managed to cluster syntactically similar words together by adding POS features into CamemBERT before encoding data. To validate our observations, we carried out a k-means clustering with Euclidean distance. We use two metrics to evaluate clustering results objectively: purity and Normalized Mutual Information (NMI). Given that, we do not seek to obtain a single representative cluster of each morpho-syntactic category but several clusters, the purity metric is particularly interesting in this study. We perform k-means clustering 10 times on EDF-Emails, and on each implementation randomly generate the initial seeds. We select the number of clusters with the elbow method [ 11 ]. The results are detailed in Table 5 and highlight that the size of training data does not modify the syntactic representation of terms. There are two possible explanations for this: 1) the small dataset contains representative examples of the larger one or 2) a small dataset is suficient to model syntactic properties of sentences, as computed by CamemBERT.

Word

Train set: OSCAR

linky (proper noun)

CBERT règlement, débit, transfert, retrait, rétablissement remboursement CBERT-POS sseetrtvleicmese,nitn,tdeervbietn,ttiroann, srfèegrl,ewmiethndt,ratwélaélp,arieminesntta,tbemeseonint (renfouunnd) Fine-tuning sreèrgvleicmese,ntin,tienrfovermntéioe,n,supratyomute,nnt,o nte,legpraatyumitent, need settlement, (is) informed, mostly, no, free cordialement (adv) cordially linky, (proper noun) remboursement (noun) refund

Train set: Wikipedia

CBERT merci, bonne, ph, obtenez, sincère

thanks, good, ph, (you) get, sincere CBERT-POS cordiallement, chaleureusement, sincèrement, infiniment, remerciant *cordiallly, warmly, sincerely, infinitely, thanking Fine-tuning restant, si, merci, quelle, bonne

remaining, yes, thank you, which, good

CBERT linki, linkin, linke, linkey, linké

CBERT-POS linki, ld, li, link, log

Fine-tuning linki, lindky, linly, lynky, linxy

CBERT remboursements, remboursment, remboursse

ment, remboursés, remboursable refunds, *refnd, *refuund, (they were) reimbursed, refundable

CBERT-POS remboursements, rembousementt, reglement, rè

glement, régularisations refunds, refundd, *règulations, regulations, regularizations

Fine-tuning remboursements, remboursemment, remboursr

ment, remboursemenr, reimboursement refunds, *refundds, *refnd, *refundr, reimbursement

CBERT cordialemement, cordialment, cordialemment,

cordiales, cordialementt *cordialylly, *cordialy, *cordiallly, *cordiales, *cordiallyy

CBERT-POS cordiallement, cordiales, franchement, amicale

ment, chaleureusement *cordiallly, *cordiales, frankly, kindly, warmly

Fine-tuning cordialment, cordiales, cordiallement, cordiale,

cordialelent *cordialy, *cordiales, *cordially, *cordial, *cordiallially cordialement (adv) cordially 15 .496 .163 .862 .589 .462 .838 .513 .882 .157 .592 Model Metric CBERT CPOS FT CBERT CPOS FT CBERT CPOS FT Both models demonstrate semantic and syntactic sensitivity regarding word similarity. It is observed through comparing the nearest associates for a given word on EDF-Emails dataset, as presented in Table 4. We use the EDF-Email dataset because it contains more noise than general domain. Nevertheless, we computed similar results with the EASY dataset, as shown in Table 7. We computed cosine similarity between frequent words and the rest of the vocabulary to evaluate the neighbors surrounding these words obtained with both models. Applying the camembert-base-wikipedia-4gb model on EDF-Emails allows to generate strong similarities between terms which share the same root, or which are spelling variants of existing words. On the contrary, using the camembertbase-oscar-4gb model produces clusters of synonyms or words that appear in a similar context. Most of the time, CamemBERT finds similar words according to word structure: it associates verbs with their conjugated forms while not always respecting the proximity regarding the tense of the verbs. However, CamemBERT-POS enhances the possibility of regrouping words that appear in the same context: synonyms and antonyms. However, two distinct phenomena are observed. First, the term "linky", which does not resemble any word in the general field, is now associated with other very specific domain terms, such as another type of electric meter or even meter installation areas. Second, these domain-specific terms are not chosen randomly and have close links, indicating that CamemBERT-POS does not only cluster random OOVs together but keeps the meaning of the terms. Therefore, the proposed method avoids relying on the tokenization step as much by adding morpho-syntactic context. To quantify the diferences between the neighbors generated by CamemBERT and CamemBERTPOS, we use comparative metrics. We implemented the Jaccard distance [ 9 ], which estimates how dissimilar two sets are by computing the number of intersecting elements in two sets. To calculate the distance, the first 50 neighbors obtained by each method were used and we computed the dissimilarity between the sets of neighbors obtained with CamemBERT and CamemBERT-POS. We averaged the similarities obtained for the hundred most frequent words in the corpus. As shown in Figure 5, both models generate significantly diferent neighbors with a Jaccard similarity averaging 0.08, confirming that CamemBERT-POS drastically changes words representation.

Word

CBERT

CBERT-POS kikou idem, grâce, mauvaise, félicitations, ok salut, cool, bonjour, bonsoir, félicitations cool salut sb, ok, combien, gafe, quant bonjour, moi, ok, hello, oui ok, joueur, okidoki, super, gafe cool, bonsoir, bonjour, félicitations, hello We aim to compare the results obtained with CamemBERT-POS regarding OOV terms with CamemBERT after fine-tuning the language model. Our implementation follows the fine-tuning example released in the BERT project to use a vanilla baseline to compare against. All hyperparameters remain as default values. We trained the model on two Epochs, using 100,000 Emails. The results are presented in Table 4. Surprisingly, the results obtained after fine-tuning are not that diferent from the ones with CamemBERT. It mostly generates spelling variations in OOV’s neighborhood. For this application, fine-tuning does not seem adequate when working with domain-specific data when we aim to deal with emerging terms in a context of poor writing. As we do not intend to re-train the model frequently, the process of adding external and automatic features is more adapted to our application study. Furthermore, Table 5 shows that fine-tuning the language model slightly improved the processing of morpho-syntactic words. 6.5

Ablation Study Layer selection BERT encodes multiple types of characteristics depending on the network layer used to represent sentences [ 10 ]: the first layers encode morphosyntactic information better than higher layers. To evaluate the impact of the choice of the layer in our evaluation, we observe the diferences of neighbors for the word "linky" for the dataset EDF-Emails with the model camembert-basewikipedia-4gb in Figures 6 and 7. We note that the neighbors caracteristics remain consistent from one layer to another. CamemBERT-POS regroup similar POS tags together and reduce the distance between semantically close words. With CamemBERT-POS, the new interesting neighbors are either related to electrical ofers ("smart", "blue", "green", etc.), other electrical meters (SMA, CBE, meter, etc.) or installation companies (Scopelec, ENEDIS, etc.). 50 40 rs o30 b h g i e N20 # 10 50 10 40 RDOOOTHOMETARIN irsoehgbN2300

# 50 40 rs o30 b h g i e N20 # 10 50 40 rs o30 b h g i e N20 # 10 ROOT DOMAIN OTHER PROPN

OTHER 0 1 2 3 4 5 6Layer7 8 9 10 11 12 (a) CBERT (b) CBERT-POS (a) CBERT (b) CBERT-POS Fig. 7: Closest first 50 neighbors of "linky" computed using cosine similarity divided in three categories: neighbors that share the same root as "linky" (ROOT), terms that are relevant and domain-specific (DOMAIN) and others (OTHER) Number of POS tags A final experiment was carried out to determine whether a high level of POS knowledge was required, or if only certain POS were relevant. To answer this, we built CamemBERT-POS-Small and calculated the neighbors as before. We chose the most important morpho-syntactic categories with regards to semantics: nouns, verbs, adjectives and adverbs. Results are shown in Table 7. At first sight, we notice that this method does not answer the problem of tokenization with the Wikipedia model as well as CamemBERT-POS for these

Word linky A. Benamar et al.

words. Interestingly, we observe that the cloud is less altered with this method than with the complete CamemBERT-POS, as shown in Figure 5. Yet, we obtain other very relevant synonyms for domain words like "meter" and "refund". We conclude that CamemBERT-POS requires having a fine-grained knowledge of the syntax to get around the processing of OOV terms. However, the word cloud can be impacted by adding a few relevant tags. The addition of these tags allows to obtain interesting clusters of semantically close neighbors.

Oscar

Wikipedia linki, link, compteur, linkie, lo- linki, link, lot, lotissement, li tissement *linki, *link, meter, *linkie, subdi-*linki, *link, sub, subdivision, *li vision remboursement paiement, réglement, rattrapage, remboursements, remboursseretrait, règlement ment, remboursment, remboursez, remboursemen, payment, *séttlement, catch-up, refunds, *refuund, *refnd, (you) rewithdrawal, settlement pay, *refun cordialement bisous, re, bref, client, heureuse- cordiallement, corialement, ment cordilement, sincère, sincerement kiss, re, anyway, customer, fortu-*cordiallly, corially, cordilly, sincere, nately sincerely We studied the efect of syntactic noise (i.e., spelling mistakes) and domainspecific vocabulary in French textual data on the performance of CamemBERT. We further show that, on a dificult corpus, the proximity between words is drastically impacted by the tokenization of OOV words. To address the problem of noisy vocabulary (i.e., OOV), we propose BERT-POS, a method that reduces the impact of tokenization while processing OOV terms. Our work stands out from the literature in two ways. First, the combination of morpho-syntactic markers and language models remains a very limited field of research, in which our work ifts. Even though BERT is a contextual model, new words can alter the structure of the sentences entering the model. External markers (i.e., morpho-syntactic markers) allow sentences to be re-structured when they become too fragmented. Second, we ofer a model that does not require re-training or fine-tuning and is easy to set up, which is, to our knowledge, the first such model built with a goal of improving tokenization issues. In our future work, we want to evaluate the impact of adding syntax on diferent tasks, by conducting a large number of experiments on diferent domain datasets. This will allow us to assess the robustness of our method in diferent domains and on several tasks.

References

1. Beltagy , I. , Lo , K. , Cohan , A. : Scibert: A pretrained language model for scientific text . arXiv preprint arXiv: 1903 . 10676 ( 2019 )

2. Devlin , J. , Chang , M.W. , Lee , K. , Toutanova , K. : BERT: Pre-training of deep bidirectional transformers for language understanding . In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Volume 1 (Long and Short Papers). pp. 4171 - 4186 . Association for Computational Linguistics, Minneapolis, Minnesota (Jun 2019 ). https://doi.org/10.18653/v1/ N19 -1423, https://aclanthology.org/N19-1423

3. El Boukkouri , H.: Ré-entraîner ou entraîner soi-même? stratégies de pré- entraînement de bert en domaine médical . In: Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition) . pp. 29 - 42 ( 2020 )

4. El Boukkouri , H. , Ferret , O. , Lavergne , T. , Noji , H. , Zweigenbaum , P. , Tsujii , J.: CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters . In: Proceedings of the 28th International Conference on Computational Linguistics . pp. 6903 - 6915 . International Committee on Computational Linguistics, Barcelona, Spain (Online) ( Dec 2020 ). https://doi.org/10.18653/v1/ 2020 .coling-main. 609 , https://aclanthology.org/ 2020 .coling-main. 609

5. Fukuda , N. , Yoshinaga , N. , Kitsuregawa , M. : Robust Backed-of Estimation of Out-of-Vocabulary Embeddings . In: Findings of the Association for Computational Linguistics: EMNLP 2020 . pp. 4827 - 4838 . Association for Computational Linguistics, Online (Nov 2020 ). https://doi.org/10.18653/v1/ 2020 .findings-emnlp. 434 , https://aclanthology.org/ 2020 .findings-emnlp. 434

6. Grave , E. , Bojanowski , P. , Gupta , P. , Joulin , A. , Mikolov , T. : Learning word vectors for 157 languages . arXiv preprint arXiv: 1802 . 06893 ( 2018 )

7. Honnibal , M. , Montani , I., Van Landeghem, S. , Boyd , A. : spaCy: Industrial-strength Natural Language Processing in Python ( 2020 ). https://doi.org/10.5281/zenodo.1212303, https://doi.org/10.5281/zenodo.1212303

8. Hu , Y. , Jing , X. , Ko , Y. , Rayz , J.T. : Misspelling correction with pre-trained contextual language model . In: 2020 IEEE 19th International Conference on Cognitive Informatics & Cognitive Computing (ICCI* CC) . pp. 144 - 149 . IEEE ( 2020 )

9. Jaccard , P. : Étude comparative de la distribution florale dans une portion des Alpes et des Jura . Bulletin del la Société Vaudoise des Sciences Naturelles 37 , 547 - 579 ( 1901 )

10. Jawahar , G. , Sagot , B. , Seddah , D. : What Does BERT Learn about the Structure of Language? In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . pp. 3651 - 3657 . Association for Computational Linguistics, Florence, Italy ( 2019 ). https://doi.org/10.18653/v1/ P19 -1356, https://www.aclweb.org/anthology/P19-1356

11. Joshi , K.D. , Nalwade , P. : Modified k-means for better initial cluster centres . International Journal of Computer Science and Mobile Computing 2 ( 7 ), 219 - 223 ( 2013 )

12. Joulin , A. , Grave , E. , Bojanowski , P. , Douze , M. , Jégou , H. , Mikolov , T. : Fasttext.zip: Compressing text classification models . arXiv preprint arXiv:1612.03651 ( 2016 )

13. Kudo , T. , Richardson , J.: SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing . In: Proceedings of the 2018 EMNLP: System Demonstrations . pp. 66 - 71 . Association for Computational Linguistics, Brussels, Belgium ( 2018 ). https://doi.org/10.18653/v1/ D18 -2012, http://aclweb.org/anthology/D18-2012

14. Le , H. , Vial , L. , Frej , J. , Segonne , V. , Coavoux , M. , Lecouteux , B. , Allauzen , A. , Crabbe , B. , Besacier , L. , Schwab , D.: FlauBERT: Unsupervised Language Model Pre-training for French . In: LREC. Marseille, France ( 2020 ), https://hal.archivesouvertes.fr/hal-02890258

15. Lee , J. , Yoon , W. , Kim , S. , Kim , D. , Kim , S. , So , C.H. , Kang , J.: Biobert: a pre-trained biomedical language representation model for biomedical text mining . Bioinformatics 36 ( 4 ), 1234 - 1240 ( 2020 )

16. Li , F. , Jin , Y. , Liu , W. , Rawat , B.P.S. , Cai , P. , Yu , H.: Fine-tuning bidirectional encoder representations from transformers (bert)-based models on large-scale electronic health record notes: an empirical study . JMIR medical informatics 7 ( 3 ), e14830 ( 2019 )

17. Ma , W., Cui , Y. , Si , C. , Liu , T. , Wang , S. , Hu , G. : Charbert: Character-aware pre-trained language model . arXiv preprint arXiv: 2011 . 01513 ( 2020 )

18. van der Maaten , L. , Hinton , G.: Visualizing high-dimensional data using t-sne . Journal of Machine Learning Research 9 : 2579 - 2605 ( 2008 )

19. Martin , L. , Muller , B. , Ortiz Suárez , P.J. , Dupont , Y. , Romary , L., de la Clergerie, É., Seddah , D. , Sagot , B. : CamemBERT: a tasty French language model . In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . pp. 7203 - 7219 . Association for Computational Linguistics, Online (Jul 2020 ). https://doi.org/10.18653/v1/ 2020 .acl-main. 645 , https://aclanthology.org/ 2020 .aclmain.645

20. Nayak , A. , Timmapathini , H. , Ponnalagu , K. , Venkoparao , V.G. : Domain adaptation challenges of bert in tokenization and sub-word representations of out-of-vocabulary words . In: Proceedings of the First Workshop on Insights from Negative Results in NLP . pp. 1 - 5 ( 2020 )

21. Paroubek , P. , Robba , I. , Vilnat , A. , Ayache , C. : Data, annotations and measures in easy the evaluation campaign for parsers of french . In: LREC . pp. 315 - 320 . Citeseer ( 2006 )

22. Pruthi , D. , Dhingra , B. , Lipton , Z.C. : Combating adversarial misspellings with robust word recognition . arXiv preprint arXiv: 1905 . 11268 ( 2019 )

23. Radford , A. , Narasimhan , K. , Salimans , T. , Sutskever , I. : Improving language understanding by generative pre-training ( 2018 )

24. Sahrawat , D. , Mahata , D. , Zhang, H., Kulkarni , M. , Sharma , A. , Gosangi , R. , Stent , A. , Kumar , Y. , Shah , R.R. , Zimmermann , R.: Keyphrase extraction as sequence labeling using contextualized embeddings . Advances in Information Retrieval 12036 , 328 ( 2020 )

25. Singh , J. , McCann , B. , Socher , R. , Xiong , C. : BERT is not an interlingua and the bias of tokenization . In: Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019 ). pp. 47 - 55 . Association for Computational Linguistics, Hong Kong, China (Nov 2019 ). https://doi.org/10.18653/v1/ D19 -6106, https://aclanthology.org/D19-6106

26. Srivastava , A. , Makhija , P. , Gupta , A. : Noisy text data: Achilles' heel of bert . In: Proceedings of the Sixth Workshop on Noisy User-generated Text ( W-NUT 2020 ). pp. 16 - 21 ( 2020 )

27. Sun , C. , Qiu , X. , Xu , Y. , Huang , X. : How to fine-tune bert for text classification? In: China National Conference on Chinese Computational Linguistics . pp. 194 - 206 . Springer ( 2019 )

28. Sun , L. , Hashimoto , K. , Yin , W. , Asai , A. , Li , J. , Yu , P. , Xiong , C. : Adv-bert: Bert is not robust on misspellings! generating nature adversarial samples on bert . arXiv preprint arXiv: 2003 . 04985 ( 2020 )

29. Vaswani , A. , Shazeer , N. , Parmar , N. , Uszkoreit , J. , Jones , L. , Gomez , A.N. , Kaiser , Ł., Polosukhin , I. : Attention is all you need . In: Advances in neural information processing systems . pp. 5998 - 6008 ( 2017 )

30. Wenzek , G. , Lachaux , M.A. , Conneau , A. , Chaudhary , V. , Guzmán , F. , Joulin , A. , Grave , É.: Ccnet: Extracting high quality monolingual datasets from web crawl data . In: Proceedings of The 12th Language Resources and Evaluation Conference . pp. 4003 - 4012 ( 2020 )

31. Wolf , T. , Debut , L. , Sanh , V. , Chaumond , J. , Delangue , C. , Moi , A. , Cistac , P. , Rault , T. , Louf , R. , Funtowicz , M. , Davison , J. , Shleifer , S. , von Platen, P. , Ma, C. , Jernite , Y. , Plu , J. , Xu , C. , Le Scao , T. , Gugger , S. , Drame , M. , Lhoest , Q. , Rush , A. : Transformers: State-of-the-art natural language processing . In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations . pp. 38 - 45 . Association for Computational Linguistics, Online (Oct 2020 ). https://doi.org/10.18653/v1/ 2020 .emnlp-demos.6, https://aclanthology.org/ 2020 .emnlp-demos. 6

32. Wu , Y. , Schuster , M. , Chen , Z. , Le , Q.V. , Norouzi , M. , Macherey , W. , Krikun , M. , Cao , Y. , Gao , Q. , Macherey , K. , et al.: Google's neural machine translation system: Bridging the gap between human and machine translation . arXiv preprint arXiv:1609.08144 ( 2016 )

33. Yang , W. , Xie , Y. , Tan , L. , Xiong , K. , Li , M. , Lin , J. : Data augmentation for bert ifne-tuning in open-domain question answering . arXiv preprint arXiv: 1904 . 06652 ( 2019 )