Classifying patent documents into categories is an important step in many decision-making processes like competitor monitoring, patent landscaping, and portfolio management. Performing the classification task manually is time-consuming, so the amount of training data available to train machine learning models for classification is often limited. The lack of training data is thus frequently the limiting factor in the performance of the classification model. In this work, we share semi-supervised learning methods used to alleviate the issue of the lack of training data. We utilize pre-trained searchoptimized representations to automatically label additional training data by performing label propagation using nearest neighbor searches in the vast space of patent documents and demonstrate that the classifiers' performance improves compared to using the smaller, original training data sets and to naive oversampling methods.
1. Introduction 2. Literature survey dings is needed. As a logical continuation of the aforementioned work, this paper presents an algorithm already used in our commercial product [4], that utilizes knowledge about the embeddings’ structure in the vector space to enlarge customer datasets using label propagation techniques and -nearest neighbors algorithm. The classification of patent documents plays an important role in strategic decision-making. Supervised machine learning models require labeled datasets for predicting new input data, yet the manual labeling of patents is labor-intensive and often leads to limited or imbalanced training data. This highlights a crucial diference between academic datasets and real-world scenarios in patent classification. In academic settings, researchers typically work with well-curated datasets like CIFAR-10 and IMAGENET, focusing on optimizing models for these datasets [1, 2]. In contrast, when working on real-world patent classification, the data are not fixed. Users can continuously add new patents to the dataset or remove some, change labels, etc. Moreover, in some cases only a few manually labeled data points are provided, while unlabeled data may be abundant. In this work, we cover this specific setting: the user might provide labels for only a small set of documents, and the database could have millions of structured, but unclassified patent documents. Therefore, to improve the classifier performance, we are free not only to modify the customers’ datasets but even to collect additional data.
Our previous work [3] showed that graph-based embeddings optimized for a search task contain rich enough information to be directly applied to a classification task with no additional fine-tuning steps. Only training a lightweight classification model on top of the embedtransductive algorithms are provided with labeled data (, ) and unlabeled data , and output exclusively predictions for the unlabeled data. Since the method we analyze in this work is transductive, the literature survey below focuses mainly on the latter.
The transductive methods typically define a graph over all data points, both labeled and unlabeled, encoding the pairwise similarity of data points with possibly weighted edges [7]. Graph-based semi-supervised learning methods generally involve three separate steps: graph creation, graph weighting (these two steps could also be described as one graph construction phase), and inference [8, 9].
After the first two steps, we have a graph consisting of a set of nodes corresponding to the data points, and a weight matrix, containing the edge weights for all pairs of nodes. Once the graph is constructed, it is used to obtain predictions for the unlabeled data points.
Three common methods for graph creation include -neighbourhood, -neighbourhood, and -matching [6]. The -neighbourhood method connects nodes within a distance of . The -neighbourhood method links each node to its closest neighbors. The -matching method is a post-processing step used with -neighbourhood to ensure the graph’s regularity, checking that the nodes have the same number of neighbors and a specified number of edges. Regarding the graph weighting step, in many cases, graph weights reflect the similarity measure employed for the edge construction, as in the case of the method we are about to introduce.
Concerning the inference stage, the literature survey
[
A similar approach to our work was used in [11], who
experimented with graph construction using a -nearest
neighbors algorithm and the -neighbourhood
(connecting pairs of data points with distance smaller than ).
They kept the edge weights fixed and uniform but
experimented with changing the weight of edges between
unlabeled data points relative to the other edges [6].
Dataset
Qubit [13]
Cannabinoid [
Chemical datasets enlarging and fine-tuning and therefore better classification metrics scores. We compare our suggested semi-supervised method to up-sampling, wherein existing samples within the dataset are duplicated to address class imbalance.
Four datasets were chosen for this study, comprising two
binary datasets and two multi-label datasets. The binary
datasets consist of the well-established Qubit dataset [13]
and the Cannabinoid patent dataset [
Each model is trained using a distinct training set extracted from the complete dataset. The models take document embeddings as input and generate probabilities for each label as output. For binary datasets, a single classifier is trained. Conversely, for multi-label datasets, a binary classifier is trained for each class using the oneversus-all approach, resulting in a set of individual binary classifiers.
For the experiments on the subsets of data, the training set was partitioned based on predetermined percentages (i.e. we randomly sample percent of data points with varying from 0.5 to 100). The criterion for a successful sampling attempt is the presence of at least one positive and one negative sample in each class of the dataset. We would call this subset of training data with at least one positive and one negative sample per class an original subset, ⊆ .
Following this, to enlarge the training data subset 3. Methodology and have higher test scores, we search for additional samples for every sample ∈ , ∈ {3, 5, 7, 10}. The Numerous manipulations can be conducted on datasets algorithm for creating an additional dataset and propagatto enhance metric performance, achieve greater result ing labels to it is detailed in Section 4.3. Here we would stability, and facilitate improved model generalization. only mention that the additional samples are strictly not In this work, our main focus is to investigate the usabil- from the same patent families as the sampled training ity of search-optimized graph-based embeddings for the data and the test data. Therefore, in the final training dataset, where the additional and original training data 3.4. Model evaluation are merged, all samples belong to diferent patent families. Evaluations for all subsets of data were done using a
To provide a comparison, we perform an up-sampling separate holdout test set independent of the training data. procedure for the same original subset. We copy each For demonstration purposes, we calculate the standard F1 scores. In the context of the multi-label dataset, microsample ∈ times, where ∈ {3, 5, 7, 10}. Then we merge copied samples with the original dataset , averaging is applied. To transform predicted probabilities as we did with the semi-supervised additional samples. into binary predictions, we employ an optimal threshold The copied samples have the same labels and embedding determined during the training phase. vectors as the original ones.
Hence, each model discussed in the corresponding 4. Transductive semi-supervised section (see Section 3.3) is trained using two training datasets. One dataset involves up-sampling, where copies method for dataset enlarging of data points are included, while the other is a semi- As highlighted in the literature survey, the proposed semisupervised dataset enriched with additional data. It’s supervised method has a transductive nature, because it crucial to emphasize that for both datasets the original has the following three stages: graph creation, graph samples ∈ are the same and only the enlarged part weighting, and inference. changes. Therefore the models’ training results obtained In the first stage, we create embeddings for patents. from these two datasets can be directly compared to each This involves parsing patent document text to construct a other.
patent document graph and then using a neural network to
When training on a subset, we repeat the random sam- embed this graph into a vector space. We then utilize the pling and training processes times to reduce the amount embeddings for each patent document as the foundation of noise caused by a poor train-validation split, where when creating the search space graph used to perform the varies from 2 for the largest subsets to 10 for the smallest label propagation. It’s crucial to diferentiate between the subsets. concepts of the patent document graph and the search
This methodological approach was designed to main- space graph. The concept of the patent document graph tain a result comparability across iterations. By utilizing is specifically relevant in Section 4.1, as it pertains to the same test set and keeping the same training set for the embedding creation process. Conversely, the search diferent models, the obtained metrics are comparable. space graph is a recurring concept throughout the paper. Metrics from multiple sampling iterations for the same The second phase is the search space graph creation percentage were then averaged. and weighting using the nearest neighbors algorithm.
Since the models’ outputs are probabilities, it’s cru- Finally, the third phase is the inference step, where we cial to determine an optimal cut-of threshold that maxi- ifnd possible new members of the dataset and propagate mizes the F1 score. This threshold is determined through labels for them. a stratified 5-fold cross-validation. In both binary and multi-label scenarios, only a single (global) threshold is selected. In the multi-label case, we choose the threshold 4.1. Graph-based representations that maximizes the micro-averaged F1 score across all optimized for patent search classifiers.
Building upon our previous work [3], we have decided to continue using the same classification models for this study. Our selection criteria emphasize minimizing training costs and ensuring the models can provide probability estimates for sample classification. Specifically, we utilize the basic logistic regression and k-nearest-neighbors classifiers with default parameters, implemented using the scikit-learn library [15] among with XGBoost classifier using library [16].
patent document graph using a specialized parser. The result is a collection of nodes and edges describing the invention found in the original text. 2. A graph neural network is used to embed the graph into a vector space, where the network is trained to perform prior art searches in the patent domain.
The specialized parser works as follows: First, a lin- 4.2. Search space graph creation for label guistic analysis of the text is done using the spaCy [17] propagation library, and uses this information to detect all nouns and noun chunks in the text. The detected nouns and noun To create the search space graph to use for the label chunks are the features of the invention, and they become propagation, we perform for each existing sample in the nodes in the graph. In Figure 1, examples of nouns and training data set a nearest neighbor search in the embednoun chunks turned into nodes are snowthrower, motor, ding space of patent documents, where the embeddings and handle device. are created using the method described in section 4.1.
After the features have been detected, the parser ex- The total size of the search space is over 100 million doctracts relationships between the features utilizing the out- uments, reflecting the large number of available patent put of the linguistic analysis combined with hand-crafted documents. The nearest neighbor searches are done usrules that recognize common phrases used in patent texts. ing the Annoy library [19].
The outcome of this process is a set of edges connecting When creating the graph we exclude from the nearest the pairs of feature nodes. For instance, in Figure 1, the neighbor search results any documents that are already parser will recognize the term comprising, and it will, a part of the dataset, to guarantee that the label propagaamong others, result in an edge between snowthrower tion results in completely new documents being used for and motor. the training.
Due to the removal of duplication and some legal jargon, the graphs are smaller than the original texts and 4.3. Inference stage thus faster to process, while also allowing downstream models to learn faster since the relationships between fea- We are given a set of labeled samples, and a set of tures are explicitly encoded. More details on the patent unlabeled samples. Since in our case, the search space document graphs and how they are created can be found graph has millions of patents, is exceedingly large. in [18]. After the construction of the graph = (, ), weights
The graph neural network is then given the parsed are assigned to the edges between all vertices (i.e. patent graphs as input and trained to perform patent searching documents’ embeddings) based on a cosine distance bein a supervised manner by using patent ofice examiner tween them. To enlarge the set and propagate labels citations. The result is a model that embeds documents for the subset of the set , we perform the next steps: with similar technical content near each other in the 1. For a sample ∈ we find neighbors among embedding space, regardless of the exact words used. ∈ , using the Annoy library [19]. Let’s mark Further information on the training of the graph neural netTwhoerekmisbefodudnindgisnc[r1e8a]te.d by the graph neural network th− essehnoeuilgdhnb’torbsealosng− t.oAthneysfaomunedpnateeignhtbfaomril− y a∈s can then be used for patent classification purposes by saanmy e p∈atenstafmampliel.yIafsththeenseaigmhpbloer,t−hesnhawreesstkhipe training a lightweight classification model on top of the
The motivation for this algorithm is that if edges between samples that are similar to each other are given a high weight, then it’s probable that they share the same labels since this conforms with the basic assumption of many learning algorithms that similar examples should be classified similarly.
it and move to the next, more distant neighbor. and its impact on the F1-score is worthwhile. Our exTherefore, we always find -neighbors for sample periments involved training models with additional data , but they might be not the closest ones. where parameter varies in {3, 5, 7, 10} set. Although 2. We assign the same labels as the sample has the semi-supervised method generally yielded the best to all − ∈ − . Consequently, each neighbor results for = 10, the diferences in metrics across dif− ∈ − becomes + ∈ +. ferent parameters were relatively modest. Notably, 3. Since all + ∈ + samples have labels now, we we observed a noticeable increase in time consumption merge them into the set. and memory usage with higher values, indicating a 4. We repeat this procedure for all samples ∈ . trade-of between the model performance and resource utilization.
There is also a potential to modify the algorithm to fix the class imbalance more efectively rather than simply appending as much data as possible. 6. Conclusions
5. Results and discussions graph embeddings optimized for a search task in the context of semi-supervised learning. Our evaluation focused Results are presented in Figure 2 and Table 2. The semi- on two binary and two multi-label datasets from diferent supervised method showed improvements in metrics for patent domains. The methods used in the work are also XGBoost and logistic regression classifiers when using utilized in our commercial product. The results show small subsets of training data ( ≤ 25%). However, that our semi-supervised approach for dataset enlargefor larger subsets, enlarging the dataset did not lead ment outperforms the naive up-sampling method. We to further improvement in the metrics. This observa- observed improvements in metrics for XGBoost and lotion was expected, especially for the binary datasets, as gistic regression classifiers when using small subsets of lightweight models like logistic regression typically do training data, indicating the potential of our method for not require many samples per class to achieve an F1-score handling limited data scenarios. plateau. Overall, our work contributes to advancing semi
For future work, it would be beneficial to have an un- supervised learning algorithms and their applicability derstanding of why the -nearest neighbors classification in real-world scenarios with limited labeled data. algorithm’s metrics are barely afected by additional data.
Moreover, exploring the influence of varying the parameter in the -nearest neighbors search algorithm