Patent documents can be efectively represented using embeddings derived from graphs. These graph-based representations capture the intricate relationships and contextual information within the documents. By leveraging the power of graph embeddings, we can create rich document representations that can be further fine-tuned to enhance their performance for specific tasks. In this paper, we aim to address the fundamental question if search-optimized graph-based document embeddings can be directly used for classification. Traditionally, diferent training pipelines and storage mechanisms were required for each distinct task, resulting in increased complexity and resource consumption. However, by establishing whether the same representations can be efectively used for both search and classification, we can streamline the process and eliminate the need for maintaining multiple sets of embeddings. Our results provide evidence that embeddings optimized for search tasks can be directly employed to perform classification tasks, ofering a promising solution that significantly improves eficiency and resource utilization. By repurposing the same set of optimized embeddings for both search and classification, we not only achieve data eficiency but also reduce computational overhead. This approach allows us to leverage the benefits of existing search-optimized embeddings without sacrificing the accuracy or efectiveness of classification tasks. As a result, we present a novel and eficient classification method that reduces the complexity of maintaining separate training pipelines and storing multiple representations.
mapping from these classes to classes of interest does not
often work [
The categorization of patent documents plays a crucial Due to the discrete nature of certain metrics, direct
role in various aspects of strategic decision-making, such optimization becomes challenging. Consequently, in
varas competitor monitoring, portfolio management, ious machine learning tasks, a common approach is to
and patent landscaping. Document classification itself solve the main objective by optimizing a substitute target
is a foundational task in natural language processing instead. An illustrative case of such a task is ranking,
and a vast amount of research has been done both using where instead of directly solving the discrete ranking
tradi-tional machine learning approaches and deep problem, we turn it into a problem of optimizing
pairlearning-based approaches [
As patent documents are already categorized by the Only training a lightweight classification model on top of patent offices using the International or Cooperative the embeddings is needed. This approach results in a very Patent Classification (IPC/CPC) standards, it may be eficient way to perform classification as such models tempting to try mapping these classes directly to the scale well to larger datasets and in the usual problem classification task of interest. These classes however scale can be trained in a few seconds. This enables online rarely correlate with the actual business tasks, so simple training of new classification models on the fly, allowing PatentSemTech'23: 4th Workshop on Patent Text Mining and for quick verification of results and multiple iterations if Semantic Technologies, July 27th, 2023, Taipei, Taiwan. needed.
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License
CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org)
detect all nouns and noun chunks in the text. The nouns in the text describe the features of the invention and Representations for text documents can be made in vari- become the nodes of the graph. In Figure 1, examples ous ways. In this work, our main focus is to investigate of nouns and noun chunks are snowthrower, motor, and the usability of a search-optimized graph-based embed- handle device. After this, the parser detects, using handding method, where the patent document is first parsed crafted rules, words indicating relationships between into an intermediate graph representation that is then the features of the invention (e.g. comprising, having, turned into an embedding. This is then compared to containing). These words will create edges to the graph. other common document embedding methods. The endpoints of the edges are found using the output of the linguistic analysis done previously. For instance, in 2.1. Graph-based representations Figure 1 the term comprising will result in, among others, optimized for search an edge between snowthrower and motor.
The graph neural network model is trained in a superIn contrast to traditional methods, such as word embed- vised manner using citations reported by patent ofice exding or transformer-based approaches, where the whole aminers, resulting in documents having similar technical document is directly encoded into a vector format with- content being placed close to each other in the embedout task-specific regularization, the graph format adds ding space. The model is trained using triplet loss, where additional prior information about the relations between a patent application acts as the anchor and a patent cited the elements in the document. The idea of the graph is to by the application is the positive sample. The negative describe all the relevant technical features of a patent in sample is chosen to be some other patent document that a concise form that is easily understandable by humans is not cited by the application. This results in a model that and eficient to process by machines. An example of a is useful for searching for prior art for new inventions. patent claim converted to a graph can be seen in Figure The embeddings used for the later classification stage are 1. created from the description graph of the patent document, the description graph including both the claims and the description text of the document.
The details of how the graphs and embeddings are
created are described in [
The parser that converts text to graphs uses the spaCy
[
Ours [
1 for more details). All the embedding models chosen represent conceptually diferent ways of forming the document embeddings. The embeddings are created using the full text of the patent document, including both the claims and the description of the document.
For TF-IDF embeddings we use scikit-learn [12]
library. To form the document embeddings out of the
GloVe embeddings, we use the spaCy [
using stratified 5-fold cross-validation. In both, binary and multi-label cases, only one threshold is selected. For the multi-label case, the threshold that maximizes the micro-averaged F1 score of all classifiers is chosen.
For the experiments where we limit the data amount, we first randomly sample percent of data points (with varying from 0.5 to 75) and then follow the same procedure as with the full data case. The sampling is done so that all the models are trained using the same fixed subset. When training on a subset of data, we repeat the training process times in order to reduce the amount of noise caused by poor train-validation split, where varies from 2 for the largest subsets to 10 for the smallest subsets.
Evaluations are done using a separate holdout test set
independent of training data. For evaluation, we
calculate the standard F1 scores. In the case of the multi-label
dataset, micro averaging is used. We conduct the
evaluation on two diferent datasets, one binary, and one
multi-label. The binary dataset is the gold-standard Qubit
patent dataset [
The same holdout test set is used for all evaluations, both for the experiments with the full data and the experiments with subsets of the data. To convert the predicted probabilities to binary predictions we use the optimal threshold selected using the training phase.
We train each model using a training set separated from We experiment with how diferent choices of embedding the full dataset. The input for the models is the document the documents (see Table 1 for the list of methods) afect embedding and the output is the probability for each label. the performance. Our main interests are classification In the case of the binary dataset, we train one classifier. accuracy (measured using the F1 score), sample eficiency, In the case of the multi-label dataset, we train one binary and training time. When measuring the training time, we classifier for each class following the one-versus-rest do not consider the time required to create the document strategy leading into a collection of separate binary embeddings or include the hyperparameter search but classifiers. assume that the embeddings are readily available and the
For the experiments with full data, we train one clas- optimal hyperparameters are known. sifier for each dataset-model pair. As the outputs are probabilities, we need to find the optimal cut-of threshold that maximizes the F1 score. This threshold is selected
Embedding type Model F1
Embedding type Model F1 Time (s)
TF-IDF knn knn knn knn knn knn lr lr lr lr lr lr knn knn knn knn knn knn lr lr lr lr lr lr 3.1. Experiments on embedding
performance To measure the overall embedding performance, we look into two factors, overall accuracy measured in F1 score and required training time following the process described in Section 2.3.1. The results are summarized in 3.2. Experiments on sample eficiency Table 3.
The F1 scores on the Qubit dataset show much vari- To experiment with how well the models perform when ability between models and embedding methods: For data is scarce, i.e. how much data is actually needed instance, the GloVe (Stanford) embeddings perform the to gain reasonable performance, we limited the amount best when using the k-nearest-neighbors (knn in the fig- of training data to smaller subsets of specific percentures) model but the second worst when using the logistic ages. The same test set was used here as in the previous regression (lr in the figures) model. This suggests that experiment on full data. the results on the Qubit dataset do not give much infor- From Figures 2 and 3, we can see that most models mation about which model or embedding method works start to plateau already when around 30% of full data is the best. For the multi-label dataset, however, the results included. On the Qubit dataset, the same efect of no are more consistent, with our approach reaching the top clear separation seems to be present as well when using performance with both models. smaller subsets of the data, similar to what was seen with From the training times, we can see a direct correla- the full data case. The curves fluctuate over each other, tion between the training time and the embedding size.
Especially the TF-IDF embeddings show an extreme case of this requiring over ten-fold time compared to any other embedding type. The main reason for poor training speed with TF-IDF is, however, that the models used do not support sparse training. and no clear distinction can be seen between models.
In the multi-label case, however, clear diferences show up: When using 0.5% of the data there is almost a 20-percentage-point diference between the best (Ours) and the worst (TF-IDF with lr and BERT (patents) for knn). The performance diference between the models decreases when the number of samples is increased, but the rankings of the diferent models mostly stay the same regardless of the amount of data used, with our method reaching the highest scores on virtually all subset sizes.
In this paper, we showed that patent classification can be done eficiently on rich graph embeddings optimized for a search task. We evaluated the performance on both a binary and a multi-label dataset and demonstrated that search-optimized embeddings work well with a very lim- [11] F. Cariaggi, BERT for Patents, 2023. URL: https: ited amount of labeled samples in the multi-label case. //huggingface.co/anferico/bert-for-patents.
In the binary dataset case, the results were inconclusive.
We showed that training of the classification models can be done in less than a second, enabling users to train classifiers in an online fashion. Due to the limited amount of datasets available, nothing conclusive can be said about the generalization capabilities of the method, but we believe this result generalizes to any rich-enough embeddings optimized for a search task. Further investigations are needed to say anything conclusive, however. [12] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel,
B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830. [13] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, A. M. Rush, Transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online, 2020, pp. 38–45.