Creating meaningful text embeddings using BERT-based language models involves pre-training on large amounts of data. For domain-specific use cases where data is scarce (e.g., the law enforcement domain) it might not be feasible to pre-train a whole new language model. In this paper, we examine how extending BERT-based tokenizers and further pre-training BERT-based models can benefit downstream classification tasks. As a proxy for domain-specific data, we use the European Convention of Human Rights (ECtHR) dataset. We find that for down-stream tasks, further pre-training a language model on a small domain dataset can rival models that are completely retrained on large domain datasets. This indicates that completely retraining a language model may not be necessary to improve down-stream task performance. Instead, small adaptions to existing state-of-the-art language models like BERT may sufice.
Since the introduction of BERT, many domain-specific
Proceedings of the Sixth Workshop on Automated Semantic Analysis of language models have been put on the market, for
examInformation in Legal Text (ASAIL 2023), June 23, 2023, Braga, Portugal. ple in the clinical [
Meaningful-Paragraph-Embeddings-for-Data-Scarce-Domains Creating meaningful text embeddings requires multiple steps: first, a tokenizer model tokenizes the text. This tokenization is used by an encoder model to create an embedding. Finally, this embedding can be used by a predictor model to perform a downstream task. We now describe how the tokenizer, language model, and predic- 3.2. Language models tor can be modified to achieve meaningful embeddings in scarce-data domains.
As baselines for our analysis, we select four BERT -based language models that have shown their applicability to
NLP in the legal domain.
For the legal domain, Limsopatham [
However, these models inadequately encode long legal texts, as parts of the inputs are truncated to fit into the language model.
In the clinical domain, Lamproudis et al. [
Art. Name 6 Right to a fair trial P1-1 Protection of property 5 Right to liberty and security 3 Prohibition of torture 13 Right to an efective remedy 8 Right to respect for private and family life 2 Right to life 10 Freedom of expression 14 Prohibition of discrimination 11 Freedom of assembly and association (Other articles)
The European Court of Human Rights (ECtHR) handles BERT-ML The BERT base multilingual cased
(BERTalleged violations of European Convention of Human ML) [
To further pre-train our language model, we use all
facts in training split as used by Chalkidis et al. [
Longformer The Longformer model by Beltagy et al.
[
tokenizer vocabularies, respectively.
We use the extended tokenizers to further pre-train two
encoder models on the ECtHR training set on a machine
with 2 50 GB NVIDIA RTX A6000 cards:3 using the script
provided by Devlin et al. [
Efective text embeddings begin with the tokenization
of the input text. A tokenizer tokenizes a text using
a pre-defined vocabulary. If a word is not in the
vocabulary, it is distributed across vocabulary tokens (e.g., 3.5. Classification model
applicant becomes app, lica, and nt). Due to their
architecture, encoder models limit the max input length We employ a convolutional neural network to classify
(usually 512 tokens). The tokenizer model should respect the documents: for every fact in the document, an
embedthis limit, which usually results in input truncation. How- ding is retrieved using one of the models from 3.2; then,
ever, truncation may negatively afect downstream task the list of embeddings is stacked and fed to the network.
performance [
While a tokenizer that is specifically trained on do- cards.4 Note that the parameters of the encoder model main data may be able to tokenize domain-specific texts as described in the previous subsection remain frozen. most efectively, it may be unfeasible to train a new tok- Furthermore, the focus of this paper lies on finding the enizer; even when training data are available, the encoder meaningful embeddings, and not on the classification model also needs to be retrained, which is a resource- accuracy of the classification model: we investigate how and time-consuming task. Therefore, extending a tok- well the diferent embeddings allow the classification enizer with domain-specific tokens may be more feasible. model to learn the task.
By adding domain-specific words, these words are not split up during tokenization, which leaves more space 4. Results for other tokens. Moreover, the encoder model might be able to capture information concerning the domain- In the following section, we discuss our results for both specific tokens, allowing more meaningful embeddings. tokenization and classification.
For example, the LEGAL-BERT model (which contains domain-specific tokens) only requires a single token for 4.1. Tokenization the word ’applicants’, while the BERT-ML tokenizer requires the tokens ‘app’, ‘lica’, and ‘nts’. We compare the tokenization result of the tokenizer mod
We select the top 1% most common words in the els introduced in Section 3.2, by tokenizing the complete
dataset based on relative frequency using the Scikit-learn ECtHR dataset. Specifically, we note the following:
[
I V TD UT mDT ↓ tDT ↓
For all of the above holds that the lower the values, the more eficient the tokenizer is. The results reported in Table 3 show that the LEGAL-BERT tokenizer is most eficient in tokenizing input texts. The tokenizer requires the fewest tokens to tokenize documents, discards the fewest tokens in comparison to other 512-limited tokenizers, while also having the smallest vocabulary. The Longformer models discard the fewest tokens overall, but require more tokens than the LEGAL-BERT tokenizer.
Extending existing tokenizers slightly decreases the number of discarded tokens (average of 2 for both tokenizers). Thus, retraining the tokenizer model decreases the amount of removed information, but may still be insuficient for long documents. .49 .39 .14 .22 .23 .0 .43 .0 .0 .0 f r e m r o f g n o
L
As the classification task is an unbalanced multi-label
problem, we note the F1-scores in Table 4. We focus on
the classification model’s ability to identify independent
classes, instead of the average F1-score. If the
classification model is unable to identify a class (i.e., 1 = 0), we
take this as an indication that the embedding does not
contain relevant information about that class. Related
work has noted that the multi-label classification is
dificult to solve [
Another limitation is that the performance of the
classification model (Section 4.2) is rather low, which is due
to the minimal efort put into the model. Related work
(e.g., Chalkidis et al. [
A point of caution is that pre-training a language
model like BERT on domain data may introduce
domainspecific bias, especially when the domain dataset
misrepresents identity groups (e.g., males are over-represented)
[
Finally, a limitation is the generalizability of the dataset and tasks; this work only looks at the efect of pretraining on one well-known domain-specific dataset (ECtHR), task (violated article classification). We expect that our findings generalize across other domain-specific datasets and tasks, especially for long texts with domainjargon. Nevertheless, future work is required to further validate this expectation.
In this paper, we investigate the efect of further pretraining large language models on domain-specific data. In order to test this on scarce-domain data, we use the ECtHR dataset as a surrogate (Section 3.1), and further pre-train a BERT-ML and a Longformer language model on this data.
We find that extending tokenizers with domainspecific tokens reduces the number of tokens discarded, albeit slightly (Section 4.1). Retraining a tokenizer results in a much more eficient tokenization result, but also requires more data and retraining an encoder model from scratch, which might be unfeasible. In a data-scarce or resource-scare setting, extending the tokenizer may be a good alternative, as fewer data is required to further pre-train the encoder model.
Embeddings constructed by the original BERT-ML adequately encode legal domain-specific information, but a completely retrained language model may be beneficial for some classification problems (Section 4.2). Moreover, in scarce-data settings, further pre-training BERT -based models using small amounts may be a feasible alternative to training a language model from scratch. In particular, the combination of adding domain-specific tokens to the tokenizer and further pre-training the language model on a small dataset is a promising direction for future research. Whether our findings generalize across other domains and tasks is a question for future work.