Domain-specific contextualized language models have demonstrated substantial efectiveness gains for domain-specific downstream tasks, like similarity matching, entity recognition or information retrieval. However successfully applying such models in highly specific language domains requires domain adaptation of the pretrained models. In this paper we propose the empirically motivated Linguistically Informed Masking (LIM) method to focus domainadaptative pre-training on the linguistic patterns of patents, which use a highly technical sublanguage. We quantify the relevant differences between patent, scientific and general-purpose language and demonstrate for two diferent language models (BERT and SciBERT) that domain adaptation with LIM leads to systematically improved representations by evaluating the performance of the domain-adapted representations of patent language on two independent downstream tasks, the IPC classification and similarity matching. We demonstrate the impact of balancing the learning from diferent information sources during domain adaptation for the patent domain. We make the source code as well as the domainadaptive pre-trained patent language models publicly available at https://github.com/sophiaalthammer/patent-lim.
• Information systems → Document representation; Language models. Language modelling, representation learning, patent domain, BERT
Large scale language models, pre-trained on corpora of general
purpose language [
In this paper we propose a novel domain-adaption linguistically informed masking pre-training method for BERT-style language models. We show its suitability for the patent domain and demonstrate its efectiveness for representation learning on patent language. We publish our BERT-based language model pre-trained on patent language data to make our work readily available for the community.
The suitability of the representations with respect to a given
downstream task relies on the assumption that the language of
the documents of that downstream task comes from the same
or similar distribution as data of the language model. Otherwise
domain-adaptive pre-training becomes necessary, as the further
the language of the downstream task is from the distribution of the
pre-training language, the less relevant information is encoded in
the representations [
We compare the language models BERT [
For BERT and SciBERT we compare domain-adaptive pre-training with and without linguistically informed masking with shifting the masking to 100% and to 75% to the noun chunks. In order to evaluate the efectiveness of the diferent representations for patent language, we evaluate and compare them for the two independent downstream tasks of IPC classification and similarity matching. Here we find that linguistically informed masking does improve the overall downstream performance, thus we reason that shifting the masking probabilities towards the noun chunks improves the representations of patent language. Furthermore we investigate the degree of shifting the masking probability towards the noun chunks.
Our contributions are as follows: • We investigate domain adaptive pre-training of a general purpose language model (BERT) and of a scientific language model (SciBERT) for representing patent language and find that SciBERT is more suitable to represent patent language. We investigate two factors for that: the language pre-training corpus as well as the tokenization • We propose linguistically informed masking for domainadaptive pre-training for patent language and demonstrate efectiveness gains with representations of patent language learned with linguistically informed masking for IPC classiifcation and similarity matching • We make the source code as well as the domain-adapted pre-trained language models available at https://github.com/sophiaalthammer/patent-lim 2
Representation learning. Learning general word representations
continues to be an active research area, from word-level
representations [
Domain adaptation of language models. There is a suite of
BERTlike domain-specific models which have been fine tuned for, e.g., the
social media [
Natural language processing in the patent domain. The use of
machine learning and deep learning methods for patent analysis
is a vibrant research area [
Here we motivate the domain adaptive pre-training method of linguistically informed masking for learning patent representations and we define and introduce linguistically informed masking for BERT-like language models. 3.1
Our goal is to learn better language representations for the patent domain. As the language models which are the subject of our research are pre-trained on a diferent language domain, we aim to quantify the diference between the pre-training language and the target language, here patent language, in terms of their linguistic patterns. Therefore we examine the general-purpose and scientific language on which BERT and SciBERT are pre-trained, respectively, as these models will serve as pre-trained language models which we adapt to patent language.
One of the main distinct characteristics of patent language is the
use of constructed multi-word terms [
We identify the noun chunks using the Spacy natural language toolkit1 after removing tabs and multiple whitespaces as well as mathematical formulas from the raw text. We remove noun chunks which are longer than 10 words as we see that these cases are enumerations when analyzing the noun chunks.
Figure 2 shows the distribution of noun chunk length of the diferent language domains, with the average length in dashed vertical lines. We observe a significant diference of the distribution of the noun chunk length in the patent language (mean: 2.73; sd: 1.27) compared to general-purpose (2.21;1.25) or scientific language (2.37;1.36) as there are more long noun chunks in patent language (K-S test: < .001 for all three language combinations). Therefore we conclude that patent language contains longer noun chunks than general-purpose or scientific language and the longer noun chunks in patent language are constructed using novel combinations of common nouns. Consequently the BERT and SciBERT models are not trained to generate optimal representations of these noun chunks.
Considering that the noun chunks contain domain-specific
information signals in the form of multi-word-terms and technical
terms [
Wikipedia
USPTO13M
The BERT language model [
In this section we describe the novel domain-adaptive pre-training method of linguistically informed masking (LIM) for BERT-like language models, which adapts the MLM task in order to focus the model towards learning specific linguistic information of the domain.
We first give a formal definition of the MLM task. Let be the number of training sequences consisting of encoded sentences in one training batch and let _ be the number of masked positions where the original token needs to be predicted. Then the loss of the MLM task is defined as cross-entropy between the predictions _ and the label _ as := _ Í Í =1 =1 −(log(_ ) _ ) _ Í Í =1 =1 for each position and for each sequence . is a weight for a padding mechanism in case fewer than _ positions are masked in the sequence. The predictions _ ∈ R are a probability distribution over the whole vocabulary with size and the label is a one-hot encoding of the masked token. The predictions _ are the output of the MLM layer and are defined as _ := ( + ) where ∈ R × and ∈ R are the weights and biases of the MLM output layer and ∈ R is the final hidden vector of the masked position with dimensionality . For a sequence we get the final hidden vectors of the masked positions ∈ R ×_ with
= ∀ = 1, .., .
Here ∈ R × are the final outputs of sequence of the BERT model with the input sequence length and the masking matrix ∈ {0, 1}×_ , which shows the masked positions. The masking matrix for a sequence consists of one-hot vectors for each masked position with = 1, .., _:
( 1, if token is masked in th position :=
0, otherwise. 3.3
With the linguistically informed masking method we aim to give
the model the flexibility to focus on learning specific linguistic
characteristics of the language, namely the noun chunks in the
patent domain. Therefore we propose the linguistically informed
masking method where ∈ [
0, otherwise which indicates whether token of sequence belongs to a noun chunk or not. The LIM masking matrix ˆ ∈ {0, 1}×_ is constructed so that it contains with a probability of only masked tokens for which = 1, and with a probability of 1 − only masked tokens for which = 0. With this construction, in of the sequences only tokens that belong to a noun chunk are masked, and in 1 − of the sequences only tokens of non-noun chunks are masked.
In order to examine the diferences of the pre-training methods
of MLM and LIM we analyze the overall noun chunk distribution
for tokens in the Wikitext raw dataset [
We examine the impact of LIM compared to MLM with the choice of = 0.75. The probability of masking the token , which is in a noun chunk in sequence , with the MLM task in the patent domain is ( = 1| = 1) = ( = 1 ∧ = 1)
( = 1) = ( = 1) ∗ ( = 1) ( = 1) 0.15 ∗ 0.75 0.507 = 0.22
This shows how we influence the probability of masking a token , which is in a noun chunk, with the parameter . With = 0.507 LIM reduces to MLM as a special case. In summary, the LIM parameter controls the probability of masking noun chunk tokens. 4
Our experiments investigate which BERT-like pre-trained language model is best suited to represent patent language (RQ1) as well as the influence of domain-adaptive pre-training with linguistically informed masking for the representations of patent language (RQ2). To evaluate these questions we do domain-adaptive pre-training on patent language with BERT and SciBERT with either the MLM or LIM pre-training method. We compare the vanilla model without domain-adaptive pre-training, the MLM and LIM domain-adaptive pre-training for BERT and SciBERT for representing patent language. To assess the quality of the resulting representations we evaluate the performance of the representations for the two independent, patent-related downstream tasks of IPC classification and similarity matching.2 4.1
We leverage the patent corpus from the Google Patents Public
Datasets 3 on BigQuery with the query in Appendix A.1. The
corpus, which we will refer to as USPTO13M, consists of 13 million
granted utility patents in English with title, abstract, claims and
description. The title contains on average 8 words, the abstract
112 words, the claims 1067 and the descriptions 9539 words. We
also retrieve metadata like the filing date and the IPC tags, which
are a consistent, hierarchical topic categorization of the patents
and which are assigned by patent examiners [
Following the definition of Pan and Yang [
take the unsupervised domain-adaptive pre-training dataset as the
title, abstract, claims and descriptions of 320 patent documents
from the USPTO13M corpus containing 3.3 billion words, similar to
the size of the BERT and SciBERT pre-training datasets. We remove
tabs, multiple whitespaces and mathematical formulas. We choose
the cased model with 110 parameters as initialisation
to match the size of the SciBERT model. As the learning rate is a
sensitive parameter that is data- and task-specific, we perform a
grid search over 1 ∗ 10−4, 5 ∗ 10−5, 2 ∗ 10−5, 1 ∗ 10−5 as learning
rate candidates following Beltagy et al. [
For LIM domain adaptive pre-training we consider two values for our domain-dependent hyperparameter ∈ {0.75, 1.0} as our analysis in section 3.3 has shown that = 0.507 reduces to MLM pre-training and as we want to investigate the optimal weighting of LIM hyperparameter . 4.3
Our goal is to compare the performances of the diferent
representations on the downstream tasks rather than to maximise the
absolute performances. As two independent patent-related
downstream tasks for evaluating the quality of the representations of
patent language we choose IPC classification and similarity
matching of two given patents. As baselines we choose the BERT and
SciBERT vanilla models which are not domain adapted to the patent
domain, as well as a convolutional neural network for sentence
classification [
Following the definition of Pan and Yang [
For the BERT model which is fine-tuned on the patent domain
with the MLM task the transfer is defined as only a task adaptation
from the source task of MLM or LIM and NSP to the target task of
IPC classification or similarity matching on the same domain of
patents. The fine-tuning of the BERT LIM0.75 and the LIM1
conifguration are defined analogously for the SciBERT-based models.
Overall we fine-tune and evaluate 8 diferent pre-trained models
on the IPC classification and the similarity matching tasks.
4.3.1 IPC classification. For the IPC classification we use a subset
of up to 480 labelled patent claims of the USPTO13M dataset for
training, similar to Lee and Hsiang [
The patents have 1.73 IPC tags on average. As the document class label we use the single most frequent tag after truncating all tags to the subclass level. We remove tabs, multiple whitespace and mathematical formulas before passing the text to the BERT model for fine-tuning. For the IPC classification fine-tuning we choose the same hyperparameters as in the domain adaptive pretraining except the learning rate. For the learning rate we perform a grid search and choose the same learning rate of 5 ∗ 10−5 for all configurations (see Appendix A.4). We fine-tune in total for 30 steps, which corresponds to one epoch for 480 labelled samples, and we evaluate the models every 10 steps.
In order to analyze the impact of the language representations for downstream tasks with smaller number of labelled training data, we finetune the models on a training dataset size of 160 and 320 and the whole 480 samples and analyze the performance compared to the baseline. Detailed results for the diferent dataset sizes for the BERT model as well as for the SciBERT model can be found in Table 2. Here the baseline performance is compared to the MLM or LIM domain adapted BERT and SciBERT model. 4.3.2 Similarity matching. For the similarity matching task we retrieve pairs of patents which stand in an “X” citation relation, which we interpret as indicating similarity between the two documents. We denote these pairs of patent which cite each other as positive pair. In order to fine-tune the models on similarity matching, we also need negative pairs of patent which do not stand in a citation relation. As the citations of a given patent do not include all possible true citable patents, but rather only those which the patent examiners choose, we must construct negative citation pairs. This construction is done as follows: To a given patent document, which comes from the positive citation pairs, we sample randomly a negative patent document from the positive patent pairs which stand in a citation relation with another patent document. If the pair turns out to be the same document, we drop it and also positive citations pairs are dropped. We choose a training dataset size up of 12 citation pairs with 50.1% positive and 49.9% negative pairs represented by their claims. The test dataset contains 16, 500 pairs, 49.9% positive and 50.1% negative ones. We remove tabs, multiple whitespaces and mathematical formulas from the text before passing it to the BERT model.
The grid search for the BERT vanilla model indicates that 2∗10−5 is the most suitable learning rate for fine-tuning on similarity matching and we choose this rate for all configurations (see Appendix A.5).
In order to analyze the impact of the language representations for downstream tasks with smaller number of labelled training data, we finetune the diferent configurations for one epoch on a training dataset size of 2, 8 and 12 samples and analyze the performance compared to the baseline.
The results of the evaluation for the similarity matching are provided in Table 3 for the BERT-based and for the SciBERT-based models. 5
In the following we examine the downstream task evaluation results regarding our research questions. = 0.50 (MLM) = 0.75 (LIM0.75, ours) = 1.00 (LIM1, ours) Comparing the evaluation results of BERT and SciBERT for IPC classification and for similarity matching leads to the conclusion that the SciBERT based models achieve an overall higher performance. For the IPC classification the results in Table 2 show that the SciBERT model outperforms the equally domain-adapted BERT model (for MLM, LIM0.75 and LIM1) by 1 − 2% downstream task performance. In Table 3 we see the results for similarity matching and comparing the performance of the BERT-based and SciBERTbased models shows the same picture. The domain-adapted SciBERT based model outperforms the corresponding BERT model by 1 − 2% downstream task performance. Overall the SciBERT model domain adapted with LIM with = 0.75 achieves the best performance for IPC classification and similarity matching. 5.2
RQ 2: MLM vs LIM We compare the domain-adaptive pre-training methods of MLM and LIM, by evaluating the downstream task performance of BERT and SciBERT domain adapted either with LIM with = 0.75 or = 1.00 or MLM.
If we compare the evaluation results for the IPC classification in Table 2 we find that for BERT and SciBERT the domain adapted representations with LIM lead to a higher downstream task performance, for BERT LIM0.75 demonstrates the best results, for SciBERT LIM1 shows the highest performance. For the task of similarity matching we find similar results: BERT and SciBERT achieve the best performance for similarity matching based on the domain adapted representations of LIM0.75.
In order to compare the performance gains to the baseline performance of the BERT models without domain adaptation, we can see the relative accuracy improvement compared to the baseline BERT model of the MLM and LIM domain fine-tuned models in Figures 3 and 4 for the BERT-based models and the SciBERT-based models respectively.
For the BERT-based models we can see that the improvement of the LIM domain fine-tuned model with a noun chunk masking of = 0.75 is consistently the highest for each size of downstream task training data for both downstream tasks, besides for the IPC classification trained on 480 samples, where the domain adapted model with = 1.00 improves the performance compared to the domain adapted model with = 0.75. Similarly we observe that the LIM0.75 SciBERT model achieves the highest improvement for the similarity matching for all data set sizes, for the IPC classification the performance improvement for a smaller dataset size of 160 labelled samples is significant.
Especially in the setting of less training data for the downstream tasks, we can observe substantial performance improvements of the LIM0.75 domain fine-tuned models compared to the MLM finetuned model on both tasks and both models. Therefore our experiments show that domain fine-tuning using LIM leads to improved representations, when comparing LIM to MLM on the two independent domain specific downstream tasks.
These evaluation results demonstrate that the representations of patent language, which are domain adapted using the linguistically informed masking training method, achieve higher downstream task performance on two independent patent-related downstream tasks. Especially in the low data regime of the downstream task, the performance gains of the LIM representations are substantial. 6
In this section we analyze our results regarding the better suitability of the SciBERT model to represent patent language than the BERT model as well as regarding the weighting factor of the noun chunk masking in domain adaptive pre-training. 6.1
The language models BERT and SciBERT are trained to encode a
given text into representations. Every language model has a
tokenization, and it has become common practice to train the
tokenization of language models with a subword algorithm [
We measure the suitability of the tokenization by the split ratio,
SciBERT Patent [’f’, ’##em’, ’##to’, ’access’, ’point’] [’fem’, ’##to’, ’access’, ’point’] [’femto’, ’access’, ’point’] which is defined as length of the encoded sentence divided by the number of words in the sentence. A lower split ratio indicates better suitability for the domain because words are not split as often, thus domain-relevant words are retained in full more often rather than being split into less information rich parts.
Following Beltagy et al. [
In order to compare the performance of the diferent vocabularies for encoding patent language we determine the length of the encodings for 1.6 million sentences from patents with the diferent vocabularies.
In Figure 5 is the distribution of the sentence length in total number of words and the distribution of the diferent encoding lengths of the sentences with the average as dashed vertical line.
We can see that the encoding length with the SciBERT tokenization is shorter than the encoding with the BERT vocabulary. We observe an average split ratio for the patent tokenization of 1.16, for the SciBERT tokenization of 1.21 and for the BERT tokenization of 1.29. Because of the shorter encoding length and the smaller split ratio of the SciBERT tokenization relative to BERT, we can conclude that the SciBERT tokenization fits better to encode the patent language. Because the representations are learned for the tokens in the vocabulary, it follows that the better the tokenization ifts to the language domain the more specific information can be captured by the learned representations. As the tokenization of the SciBERT language model fits better to represent patent language than the BERT vocabulary, we suggest this as a reason that the SciBERT language model shows better results for representing patent language than BERT. 6.2
We also want to investigate the efect of balancing the domain adaptive pre-training with the noun chunk masking parameter which gives the ability to control the learning from diferent linguistic information from the target domain. For that we analyze the accuracy improvements of diferent noun chunk masking values for domain adaptive pre-training for BERT and SciBERT on both downstream tasks.
We show the accuracy improvements compared to the baseline models without domain adaptative pre-training in Figure 6. Here one can observe the clear trend that shifting the noun chunk masking towards the noun chunks with a weighting of 75% masked noun chunks and 25% masked non-noun chunks lead to higher accuracy improvements for both downstream tasks. However focusing the domain fine-tuning only on the noun chunks, in other words LIM1, shows suboptimal results, which leads to the conclusion that balancing the masking of noun chunks and non-noun chunks between the values of 0.5 and 1.0 is beneficial for domain adaptive pre-training for patent language. 7
Overall we conclude that domain adaptive pre-training for learning representations of patent language is beneficial for pre-trained masked language models like BERT and SciBERT. We find that the SciBERT-based representations outperform the BERT-based representations of patent language for the two independent downstream tasks of IPC classification and similarity matching. Therefore we conclude that SciBERT is more suitable to represent patent language than the BERT model and see one reason for that in the suitability of the tokenization of SciBERT to patent language. Furthermore we have proposed the empirically motivated domain adaptive pretraining method of linguistically informed masking for BERT-like language models. We demonstrate improvements on both patentrelated downstream tasks for representations of patent language that have been domain-adapted using the LIM method. Furthermore we analyze the impact of the weighting factor for shifting the masking towards the noun chunks. We conclude that domain adaptive pre-training with linguistically informed masking improves the representations of the patent domain for BERT and SciBERT and that balancing the weighting to learn from diferent linguistic information is beneficial for representation learning.
S E L E C T p u b l i c a t i o n _ n u m b e r , p u b l i c a t i o n _ d a t e , f i l i n g _ d a t e , p r i o r i t y _ d a t e , t i t l e . t e x t AS title , t i t l e . t r u n c a t e d as t i t l e _ t r , a b s t r a c t . t e x t AS a b s t r a c t , a b s t r a c t . t r u n c a t e d as a b s t r a c t _ t r , AS ipc , AS c p c F R O M c l a i m . t e x t as claim , c l a i m . t r u n c a t e d as c l a i m _ t r , d e s c r . t e x t as descr , d e s c r . t r u n c a t e d as d e s c r _ t r , A R R A Y _ T O _ S T R I N G ( A R R A Y ( S E L E C T c o d e F R O M
U N N E S T ( p . i p c )) , " ; " ) A R R A Y _ T O _ S T R I N G ( A R R A Y ( S E L E C T c o d e F R O M
U N N E S T ( p . c p c )) , " ; " ) ` p a t e n t s - p u b l i c - d a t a . p a t e n t s . p u b l i c a t i o n s ` p , U N N E S T ( p . t i t l e _ l o c a l i z e d ) as title , U N N E S T ( p . a b s t r a c t _ l o c a l i z e d ) as a b s t r a c t , U N N E S T ( p . c l a i m s _ l o c a l i z e d ) as claim , U N N E S T ( p . d e s c r i p t i o n _ l o c a l i z e d ) A N D A N D A N D A N D A N D as d e s c r W H E R E f i l i n g _ d a t e
Google BigQuery database accessed on the 04.11.2019 A.2
The masking and next sentence accuracies after domain adaptive pre-training each configuration for 2500 steps for the diferent learning rate candidates 1 ∗ 10−4, 5 ∗ 10−5, 2 ∗ 10−5, 1 ∗ 10−5 as well as the learning rate choice for each configuration can be seen in Table 5. Domain adaptive pre-training was performed on 1 GPU and took 2.5 days for each configuration.
A.3
In order to test whether there is a trivial relationship between the IPC tags and the similarity relations of a patent, we train a linear SVM classifier on predicting the similarity matching of a pair of patents from their IPC tag representation. On an equally balanced binary dataset we reach a classification performance which is little better than random (accuracy: 0.59, F1 score: 0.46), and therefore conclude that the two tasks are independent. maximum sequence length masking probability training steps warm-up steps 1 2 dropout probability batch size learning rate maximum sequence length masking probability training steps warm-up steps 1 2 dropout probability batch size
The evaluation of the grid search for the best suitable learning rate for IPC classification fine-tuning can be found in Table 6. The accuracy values for the diferent configurations are shown after ifne-tuning for 2500 steps with the learning rate candidates 5 ∗ 10−5, 2 ∗ 10−5, 1 ∗ 10−5. The IPC classification was performed on 1 GPU and took around 1.5 days for each configuration. The hyperparameter for fine-tuning on similarity matching can be found in Table 7. Fine-tuning the BERT vanilla model configuration for 1000 steps on the diferent learning rates of 5 ∗ 10−5, 3 ∗ 10−5, 2 ∗ 10−5 indicates that 2 ∗ 10−5 is the most suitable learning rate for fine-tuning on similarity matching and we choose this rate for all configurations. The fine-tuning was performed on 1 GPU with 61 RAM and took around 30 hours for each configuration.