This study presents an ensemble method for patent classification addressing the imbalance patent data problem. To achieve this, the dataset is divided into two data partitions based on the codes' representation magnitude. These partitions are trained separately by two identical classifiers and their results are combined using a stacking meta-classifier. Experiments are conducted using two benchmark patent datasets. The first results showed that the proposed combination of classifiers improves the imbalance patent data problem and outperforms the baseline classifiers, other combinations of classifiers and recent state-of-the-art techniques for patent classification.
Patent classification is an important task of the patent
examination process dealing with the assignment of
one or more classification codes from a classification
scheme. The most widely used classification scheme is
the International Patent Classification (IPC) which
contains approximately 70,000 different IPC codes.
The correct assignment of classification codes is quite
important as it ensures that patents with similar
technical characteristics will be clustered together
under the same classification codes, something which
is crucially important for many subsequent tasks, such
as patent management and search, technology
characterization and landscape [
The manual patent classification, which is
performed by patent officers when a patent
application arrives, includes the finding of relevant
classification codes through the hierarchical
descriptions of classification codes in the classification
scheme. However, it can be very time consuming,
tedious and strongly dependent on patent officer’s
ability and experience [
Research efforts in automated patent classification
[
Learning (DL) models for effective patent modelling and representation, and automatic classification. Most of these patent classification efforts used various simplifications when applied, e.g., working mostly with well-represented codes having many training samples or targeting the higher levels of the classification hierarchy, still they do not attain acceptable performance, i.e., one close to human performance.
The accuracy of the classification model mainly depends on the quality of the dataset and the classification algorithm. The data-related factors which could reduce the accuracy of a patent classification model are many, such as the complex/broad concepts expressed by classification codes, the ambiguous vocabulary or new terminology used, the overlapping concepts among classification codes (which increases as we go down in the level hierarchy), and, last but not least, the imbalanced patent dataset problem. This means that some classification codes have a large number of patent samples and thus high representation magnitude in the dataset. These codes are called major codes/classes. On the other side, there are some other classification codes which have very few patent samples and thus low representation magnitude These codes are called minor codes/classes.
Classification models trained by imbalanced
datasets usually have a very poor prediction ability on
minor codes. In order to solve the imbalanced dataset
issue, lots of research efforts have been carried out.
Improvements are mainly based on two directions, the
dataset level and the algorithm level [
In this study, we adjust the ensemble architecture
for patent classification presented in [
The classification scheme contains numerous codes, of
which a varying number is assigned to each patent [
Similar to the real-life distribution of patents across codes, the distribution of patents across codes in test collections is quite unbalanced. For example, in the CLEFIP-0.54M dataset2 which originates from the CLEF-IP 2011 (see Section 4 for more information), each code has a mean frequency of 740 patents with a standard deviation of 1,930 patents and a median frequency of 169, which is a more informative statistic compared to mean for imbalance datasets where there exist many frequency outliers. Similarly, in the USPTO dataset, each code has a mean frequency of 3,177 patents with a standard deviation of 12,710 patents and a median frequency of 578. Moreover, 392 codes (53.63% of all 731 codes) in the CLEFIP-0.54M dataset and 212 codes (33.76% of all 628 codes) in the USPTO dataset have a low patent frequency between 1 and 200 patents (Figure 1a and 1b).
Trying to explore whether the code’s patent
frequency affects the performance of the patent
classification models, Figure 2a displays the accuracy
of a state-of-the-art DL model, the Bi-LSTM [
As it is observed, high accuracies can be attained as the patent frequency of codes increases, meaning that the number of patent samples representing a specific classification code plays a significant role in the code’s distinguishability and finally in the code’s performance. Considering that the accuracy of the classification model across all codes is 63.76%, we assume that an adequate accuracy (see the “threshold” line in red – Figure 2a) is achieved for codes represented by more than 500 patent samples.
Especially, for classification codes with low representation magnitude the accuracy achieved is quite low affecting significantly the total accuracy of the classifier. e.g., the accuracy for codes with patent frequency between 0 and 50 patents is only 19.09%.
Therefore, the idea behind this study is that if we had a classifier focusing only on these low-represented codes, better performance would be achieved. This is also validated in Figure 2b where the accuracy achieved by a similar classifier trained only with lowrepresented codes is presented. 2a 2b Figure 2a: The accuracy of a state-the-art patent classification model, the Bi-LSTM, as a function of codes’ patent frequency organized into groups of subsequent codes. Figure 2b: The accuracy of the same model trained only on patents with low-represented codes.
An ensemble architecture for automated patent
classification has been introduced by Kamateri and
Salampasis in [
In this study, we apply this ensemble architecture for automated patent classification to address the imbalance patent data issue equipped it with two baseline classifiers and a meta-classifier (Figure 3). The first classifier is trained with high represented classification codes, while the second classifier is trained with low represented codes. Thus, each classifier specializes in a portion of codes having high and low patent frequency, respectively. This means that if a patent application characterized with a classification code of low frequency is submitted to the first classifier specializing to high-represented codes, the classifier will not be able to classify this patent application correctly since the specific classifier is not (probably) trained with similar patents. Conversely, if this patent application belonging to a classification code of low frequency is submitted to the second classifier, which is more delicate to detect codes with low patent frequency, there are better chances to be properly classified under the correct classification code corresponding to the described invention. In such cases, an appropriate combination of two baseline classifiers can better approximate such a boundary by dividing the data space into smaller and easier-tolearn partitions. Then, a meta-classifier is trained on the features that are outputs of the baseline classifiers to learn how to best combine their predictions (stacking). More specifically, the meta-classifier will distinguish if the described invention of a patent application belongs to a high or a low represented classification code and, respectively, coordinate the operation (sigmoid stacking classifier) or selecting the more appropriate (softmax stacking classifier) of the two baseline classifiers to classify a receiving patent application.
To evaluate the real-world performance of the
proposed ensemble method for imbalance patent data,
two patent benchmark datasets have been used: the
USPTO-2M and the CLEFIP-0.54M.
4.1. USPTO-2M 1.1
The USPTO-2M is a large-scale dataset prepared for
patent classification [
The ensemble architecture presented in [
An ensemble of bidirectional LSTM classifiers was
employed, since this ML method has been proved in
[
With respect to the patent representation, the first
60 words from the patent part of interest (e.g., title,
abstract, etc.) were used after undertaking a sequence
of preprocessing steps (cleaning punctuation, symbols
and numbers, and stop word removal). The feature
words were then mapped to embeddings using a
domain-specific pre-trained language model which
has been created on a patent dataset, proposed by
Risch and Krestel [
The dataset was split into training, validation and testing sets (80:10:10). Batch size was set to 128, epochs for baseline classifiers to 15 and epochs for meta-classifier to 20.
In each experiment, two baseline classifiers have been trained on two different data partitions. The first classifier was trained on patents belonging to highrepresented codes, having patent frequency over 500 patents, while the second classifier was trained on patents of low-represented codes, with patent frequency between 1 and 500 patents. Table 1 presents the Accuracy attained by each classifier i) when it is tested on the same data partition where it was previously trained, named as “Testing on the same data partition”, and ii) when it is tested in the entire dataset, containing both data partitions with known and unknown data, named as “Testing on the entire dataset”. It also presents the Accuracy of the metaclassifier combining the outcomes of the two baseline classifiers using a stacking technique. Last, it presents the Accuracy of the ensemble of classifiers combining sigmoid predictions from different patent sections.
In both datasets, the accuracy is much improved when a stacking technique is applied combining the predicted probabilities acquired by individual classifiers specialized in high- and low-represented codes, respectively. Moreover, the stacking technique
T SP O U
IFLCE .405 n Claims
Softmax Sigmoid Average Waveeigrhatgeed
Baseline Ensemble of
classifier predictions
trained for all patent
on the sections
entire (Weighted
dataset average)
[
In this study, a novel ensemble method for patent classification is presented addressing the imbalance patent data problem which is one of the most significant factors that reduces the accuracy in automated patent classification. The results showed that the proper combination of classifiers can attain significantly improved accuracy compared to baseline classifiers and existing classification techniques. Moreover, the combination of the knowledge gained from multiple classifiers could address the problem of low patent sample representation for codes, a phenomenon that is relatively common in the patent domain as the IPC/CPC taxonomy evolves with new codes introduced, codes partitioned into subcategories, etc.
supported by the Hellenic
Foundation for Research and Innovation (HFRI) under the HFRI PhD Fellowship grant (Fellowship Number: 10695). [18] M. Sofean, Deep learning based pipeline with multichannel inputs for patent classification. World Patent Information 66 (2021) 102060. doi: https://doi.org/10.1016/J.WPI.2021.102060. [19] D. Tikk, G. Biró, A. Törcsvári, A hierarchical online classifier for patent categorization. In Emerging technologies of text mining: Techniques and applications, 2008, pp. 244-267. doi: https://doi.org/10.4018/978-1-59904-3739.CH012.