In this study, we proposed a knowledge taxonomy for patents, called KnowMap, which aligns with the CPC schema and reduces the number of classes to 83 at the lowest hierarchical level. We classified patents into these ifne-grained classes within a multi-label setting, fine-tuning a distilled version of the RoBERTa model for this purpose. We employed two sampling techniques: (i) random sampling and (ii) conditional random sampling, and found that conditional random sampling led to less pronounced class imbalance, resulting in more generalisable outcomes. Additionally, our results showed higher F1-Macro scores for minority classes, which will be further explored in future work.
Exploring and leveraging patent-related data is a key task in both scientific and industrial domains. Patent analytics ofers a comprehensive view of emerging innovative technologies across various fields. Consequently, business and research initiatives, including European projects, depend on analysing and enhancing patent datasets with specialised innovation-related taxonomies.
One such initiative, the enRichMyData project [
In this paper, we propose a novel hierarchical knowledge taxonomy that aligns with the widely
used Cooperative Patent Classification (CPC) schema. The CPC classification system organises patents
into hierarchical taxonomies, which helps streamline internal processes and enhances the eficiency of
search queries. In the first level of the CPC hierarchy, there are nine sections, which are divided into
classes, subclasses, groups, and subgroups. Each level of this hierarchy can have several codes ending
in approximately 250,000 classification labels [
CEUR
ceur-ws.org
Patent documents contain various types of information, including text [
Various classification systems exist for organising patents [
Patent classification is a multi-label classification problem since every patent can belong to several
knowledge fields [
In this work, we developed a knowledge field taxonomy using CPC schema labels. We also classified patents into KnowMap’s fine-grained classes by fine-tuning some pre-trained models.
We used the Google Patents Public Datasets on BigQuery 1 and applied preprocessing and sampling
techniques. The dataset contains various information, with the abstract ofering a brief overview of the
patent’s novelty and the description providing more detail. For classification, we concatenated the title,
abstract, and description, filtering out documents with fewer than 100 words, as prior studies suggest
this improves classifier performance [
In developing the taxonomy, we considered both the shared knowledge across fields and the
distribution of documents within each defined class. To have suficiently abstract classes, we set a threshold for
the minimum number of patents in each detailed group at the lowest level of the hierarchy. Prior to
counting the documents in each class, we applied a deduplication step as part of the preprocessing to
remove duplicate and near-duplicate texts, which may refer to the same patent [
Deduplication was performed using Locality Sensitive Hashing (LSH) [
We developed the KnowMap taxonomy by refining the CPC hierarchy and its class entities to create a more abstract representation of patents. Starting from the highest level, we manually merged groups at each level based on shared knowledge and document counts. While all major CPC sections were retained at the first level, groups with fewer than 40,000, 20,000, and 9,000 documents were merged at levels 2, 3, and 4, respectively.
We formulated the classification problem as a multi-label problem, in which each document is assigned
to one or multiple knowledge fields. In this study, we aimed to classify the patents into the fine-grained
classes in the lowest level of the proposed taxonomy (i.e., 83 classes). We used the pre-trained language
model distilroberta-base, a distilled version of RoBERTa [
For model training, we used a learning rate of 4e-5 with a linear scheduler, a weight decay of 0.1, and
trained for up to 5 epochs with early stopping. The best checkpoint was selected to prevent overfitting,
based on validation accuracy. The sampled datasets were split into training, validation, and test sets
with ratios of 0.8, 0.1, and 0.1, respectively. To maintain the class distribution across these sets, we used
stratified splitting 4 proposed by Sechidis et al. [
In this section, we first present the KnowMap taxonomy and then evaluate the performance of classifiers in categorising patents into fine-grained classes of the taxonomy.
Following the methodology described in Sec. 3.2, we established a hierarchy with the root node as level 0 and level 4 as the lowest level. There are nine classes at level 1 and 83 classes at the lowest level of the hierarchy.
Our hierarchical labels, including merged classes and document counts, are available online 5. The taxonomy retains the nine CPC sections at the first level, while subsequent levels include merged CPC groups, all detailed in the shared online source.
In this study, we classified patents into the fine-grained classes at the lowest level of the hierarchical taxonomy, which includes 83 labels. To gain further insights into the datasets generated by the two sampling techniques, we analysed the number of samples per class in each dataset. Fig. 1 illustrates this information through box plots for both sampling techniques. The plots highlight the first quartile (Q1), median, third quartile (Q3), minimum, and maximum values for each sampling method. 2https://huggingface.co/ 3https://github.com/elmotamedi/KnowMap-Taxonomy 4https://github.com/trent-b/iterative-stratification?tab=readme-ov-file#multilabelstratifiedkfold 5https://github.com/elmotamedi/KnowMap-Taxonomy
Max: 81,241 Q3: 20,028 Q1: 4,862
Median: 8,507
Min: 59 Random Sampling
Q3: 23,928 Q1: 10,726
Min: 1,086 Conditional Random Sampling
Median: 15,006 Samples
Based on Fig. 1, random sampling resulted in a broader range of document counts per class, with a minimum of 59 samples and a maximum of 128,417 samples per class. The conditional random sampling technique produced a narrower range, with a minimum of 1,086 samples and a maximum of 81,241 samples per class. For our analysis, we categorise classes into three groups: small classes (those in the first quartile), medium classes (those in the second and third quartiles), and large classes (those above the third quartile). With this categorisation, conditional random sampling appears to ofer more balanced class distributions compared to random sampling, potentially enhancing the generalisability of classification models trained on this dataset. We present the classification results on both the validation and test sets, applied to the datasets generated by the two sampling techniques in Tab. 1.
As observed from the results, the Macro-F1 score is higher than the Micro-F1 score, which may show that the model performs better for minority classes compared to majority classes. To gain more insights into these results, we generated a plot (see Fig.2), showing the F1 scores compared to the number of documents in each class.
The plot shows that the Macro-F1 score is higher for minority classes compared to majority classes for both sampling techniques. The gap between the line plots for random sampling and conditional random sampling (Fig. 2b) highlights the presence of larger classes in the dataset created by conditional random sampling. To provide further insights into the F1-macro scores for small, medium, and large classes across each sample, we have created box plots summarising these scores for each class group and sampling method. The minimum, median, and maximum F1-macro scores for each class group are presented in Fig. 3. Although the increasing trend in F1-macro scores for smaller classes is still visible, it is less pronounced compared to the random sampling technique.
In this work, we proposed a knowledge taxonomy that aligns with the CPC schema, reducing the number of classes to 83 at the lowest level while ensuring a minimum number of documents for each class in the studied dataset.
We created two datasets from the preprocessed original data using two sampling techniques: (i) random sampling and (ii) conditional random sampling. The conditional random sampling technique resulted in class entities with a minimum of 1,086 samples, substantially more than the minimum sample size achieved through random sampling. This suggests that the results from conditional random sampling may be more generalisable compared to those from random sampling.
In terms of performance, classifiers showed comparable results with both sampling techniques. Both datasets were unbalanced, with the imbalance being less pronounced in the dataset created through conditional random sampling. The classification results exhibited higher F1-Macro scores compared to F1-Micro scores, likely due to the unbalanced nature of the datasets. We conjecture that the lower F1-Macro scores for larger classes may result from the varied nature of documents within those classes, possibly due to imprecise patent assignments in the CPC system or the broader scope of these knowledge ifelds. Our future research will focus on analysing the classes that the classifier struggles with.
To improve classification performance, we plan to address the dataset imbalance using techniques specifically designed for multi-label classification with long-tailed distributions. Additionally, we aim to explore the use of a larger or alternative pre-trained model to potentially enhance classification results.
This work was supported by the Slovenian Research and Innovation Agency under grant agreements CRP V2-2272, V5-2264, CRP V2-2146 and the European Union through enrichMyData EU HORIZON-IA project under grant agreement No 101070284.
The sources referenced in this paper, including the proposed taxonomy and the classification implementation, are available at: • KnowMap taxonomy and classification implementation • Multi-label stratified K-fold implementation • Google Patents Public Datasets on BigQuery • Hugging Face library