Patents are an important source of technological innovation. Due to the large number of patents published each year, it has become increasingly dificult to find precise information on specific patent inventions, which requires not only professional search systems but also many years of patent expertise. Patent claims are the backbone of inventions and define the scope of legal protection. They can be classified in diferent types in terms of what they claim, e.g. for a physical entity we can speak of "product claims", while for an activity we can refer to as a "process claim". Manual identification of these claim types for a large set of documents is labor-intensive and time-consuming. To address this challenge, we developed a Patent Claim Type Recognition (PCTR) model based on Deep Learning (DL), which is able to automatically identify pre-defined types of patent claims. Further, we also built a rule-based heuristic approach, to generate training data to be used by the PCTR model. The proposed model was evaluated by using a dataset labeled by Subject Matter Experts (SMEs). Our experimental results demonstrate that the PCTR model accurately identifies the type of given patent claims, ofering a promising approach to streamline patent analysis and evaluation processes.
1. Introduction dependent claims, together building a hierarchy1.
Typically, independent claims contain core inventive
inforPatents enable inventors to disclose their inventions and mation whereas dependent claims specify improvements
protect them legally by preventing others from using, sell- or variations. Such variations can be rather miniscule or
ing, and producing the invention without permission [
The claims are a crucial component of the patent doc- techniques to automate patent classification, claim
clasuments, they define the legal scope of protection of an sification, and claim type identification. However, often
invention. Essentially, they specify the subject matter they require manually labeled large amounts of training
that is sought to be protected and refer to the core in- data. Further, approaches focus on claims designed to
ventive information of a patent. These claims serve as measure diferent aspects of patents such as comparison
the foundation of what aspects of an invention should be of patent claims and economic growth and social
welprotected from infringement. Therefore, an accurate anal- fare [
Overall the main contributions of the paper are as follows: 3. Patent Claim Type Recognition • Introducing a rule-based heuristic model that as- In this section, we give a definition for the regarded probsigns types to claims, facilitating the generation lem and describe the predefined claim types for the model of training data. prediction. • A transformer-based deep neural network archi- Problem Definition: Given a claim text and a predetecture designed to automatically identify patent ifned type list, the task of the PCTR model is to assign claim types. the most relevant type from the predefined type list. • A comprehensive evaluation of the PCTR model Predefined Types: The specific claim types listed below, using data labeled by three diferent Subject Mat- which are defined by name and example in the WIPO 3 ter Experts (SMEs). publicly available documentation, focusing on the subset of claim types that are directed at the nature of the invention.
Due to the significant importance of patent documents
for individuals, enterprises, and industries, there has been
a considerable amount of study and research dedicated to
this domain. These studies cover a wide range of topics
such as patent classification [
As mentioned in the Introduction section patent claims
can have diferent types, [
In contrast to these approaches, our proposed ap- paired with the document sections such as the title and proach difers in two main aspects. First, the focus of abstract. Then the output is the type of the given claim, our work is to build a comprehensive model to identify represented as ( = |), where denotes the patent types of pre-defined patent claims. Second, although our • Method: recites a sequence of steps that com
plete a task or accomplish a result • Use: depicts intended/inventive application of
novelty • Composition: invention pertains to the chemi
cal nature of materials/components used. • Process: claims define a process of manufacture, it should be noted that WIPO documentation labels it as “product-by-process”. • Apparatus: protects an apparatus or device • System: an assemblage or combination of things or parts forming a unitary whole
wipo_ip_phl_16_t5.pdf
Process
Method Composition Apparatus Use #Sample
training data, test data annotated by subject matter exclaim type (e.g., apparatus, composition, system, etc.). perts (SMEs), and the experimental outcomes obtained The model aims to classify the claims based on the pro- from the PCTR models. vided input text. Figure 1 illustrates an example input of a patent section and its associated claim. Initially, the text 4.1. Training Data Generation is tokenized, and the token embeddings serve as input to the transformer block, with these embeddings being To train the PCTR models which are based on the DL randomly initialized. architecture, it is crucial to have a suficient amount of
Classifying patent claims according to given types (as representative training data. Manually, generating mildefined above) is a challenging task. It should be noted lions of labeled data is expensive and time-consuming. that in this study, we distinguish claim types based on For the training data generation only simple claims (cf. their level of complexity as simple claim types and com- Section3), which contain the type information within the plex claim types. Simple claim types, as the name sug- text are considered. We designed a rule-based heuristic gests, are straightforward and include claim-type infor- model that is able to label given simple claim texts based mation within the claim text. On the other hand, complex on a defined regular expression (regex) rule. On the other claim types either do not include explicit claim-type in- hand, the test set contains only complex claims (cf. Secformation in the claim text or refer to more than one tion4.2), which do not include the claim type within the type, making it challenging to identify them. The focus text. of the developed Patent Claim Type Recognition (PCTR) The designed regex baseline relies on start and end approach is to automatically identify the type of com- markers. In between such markers are the targets. plex claims by designing and developing the deep neural Following is an example of the beginning of a patent network. claim: 3.1. Feature Selection
features, including the claim, title, and abstract. Other potential features, such as CPC/IPC4 information and additional contextual or structural elements from the patent document, remain for our future analysis.
To train the PCTR model with diferent feature combinations we designed the following versions of it: PCTR_V1 utilizes claim, PCTR_V2 utilizes claim + title, PCTR_V3 utilizes claim + title + abstract as input to perform the claim type prediction. Essentially, the version of Example 1. "68. A method according to claim 67," with "68. A" being a start marker, "according to" an end marker, and "method" the target.
End markers are diferent depending on whether they are dependent or independent claims. Therefore, such information has to be provided, e.g. by using diferent tools designed for this purpose. For that, the internally developed tool has been employed. Three types of targets are identified. High-probability targets where start and end markers only capture a single word and that word matches one of the known claim types (appara4https://www.epo.org/en/searching-for-patents/helpful-resources/ tus, compound, composition, device, method, process, ifrst-time-here/classification system, use). Medium-probability targets where several
System Process
Method Composition Apparatus
Use #Sample words a captured, but one of them is a known claim type. And lastly, low-probability targets, where more than one known claim type is captured.
After applying the heuristic model to the claim texts of internal sources which encompasses patents from two diferent sources, namely, the World Intellectual Property Organization (WIPO), and the US Patent Ofice. 2 million patent samples were selected employing random sampling and subjected to preprocessing. In other words, a subset of data points has been selected as a training set from a larger dataset. The preprocessing involved removing duplicates as well as invalid claim types, abstracts, and titles. Finally, we had 2,092,385 samples with high probability targets, i.e., claim types. All the claims with low-probability or medium-probability targets are ifltered. The statistics of the training data are shown in Table 1.
It should be noted that to avoid biases towards the rulebased method for labeling training samples and allow for the generalization of the trained models, we removed the claim type information from each patent claim before feeding them into the PCTR models. 4.2. Test Data Generation
it is essential to have representative test data that the model is expected to encounter in real-world scenarios. To this end, the test data was generated and labeled by overall 4 SMEs following an iterative process in order to achieve the required level of agreement. Initially, 300 samples were randomly selected from the two internal data sources to be labeled by the SMEs. Additionally, 200 samples were also selected from an external database to balance the test set, ensuring an equal number of expertlabeled samples for each type. The statistics of the test set are presented in Table2. For determining the final claim type, we adopted a majority vote approach. 4.3. Evaluation of PCTR Models
PCTR_V1, PCTR_V2, PCTR_V3. Table 3 presents the accuracy results of the models, calculated as the ratio of correctly classified data to the total test data. Upon analyzing these results and consulting with SMEs for improvement suggestions, we received valuable feedback.
The analysis by SMEs led to a revised interpretation of the PCTR model’s performance. Some samples initially deemed as falsely identified types were actually correctly identified, according to the feedback of the SMEs. The essential feedback that we applied to our evaluation process to improve the performance of the PCTR models is as follows: • "Product-by-process" claims should be regarded as "process" claims. • "Use" claims can be seen as a sub-category of "method" claims.
• Claims can have multiple (2) types.
Table 4 presents the improved accuracy of the PCTR models after considering this feedback and re-computing the accuracy.
Another aspect has been considered to improve the models’ performance. The generated dataset through random sampling is quite unbalanced and generally, it is a good practice to have a balanced data set for any machine learning model. In our eforts to achieve a balanced dataset, we had to reduce the size of the training data by downsampling the dataset as some claim types were underrepresented. However, this reduction in the training data resulted in a drop in accuracy, as there were fewer samples for each claim type hence a smaller dataset. Therefore, we decided to use the originally randomly sampled training set to train the models. It should be noted that ensuring a balanced training dataset while randomly sampling is a time-consuming process that requires expert assistance to ensure an equal number of training samples for each class or claim type.
identification of predefined claim types based on a Deep Learning model. Looking at the performance of the PCTR model’s performance focusing on both dataset characteristics and feature selection aspects, we can report the following results of a deeper analysis.
Data from diferent Patent Authorities : As claim type definitions have been discovered to vary by jurisdiction, it is needed to customize the developed models per patent ofice (or collections of patent ofices) accordingly. As stated earlier, the claim type definitions are based on WIPO documentation.
The importance of feature selection: The experiments indicate that including more features, specifically the title, and abstract, can significantly improve the accuracy of the PCTR models for claim type identification. In general, providing more context that can be used to describe and distinguish each claim type from each other is helpful. Further, our preliminary experiments with a very small set of datasets suggest that including CPC information, improved the accuracy. Nevertheless, including CPC as a feature requires a systematic evaluation and data sampling considering the various domains and extraction of balanced data of suficient size for each claim type. We leave this as our future work.
The role of SMEs: The results also demonstrate that incorporating the feedback and insights from SMEs can greatly enhance the accuracy of the PCTR models. This highlights the importance of involving subject matter experts in the development process. Herewith, starting from investigated examples in our data analysis it was, for example, confirmed that a claim can be assigned several types. Besides that, it turned out that it is viable to consider several probabilities for the correct target label (type), as for example in the case of product-byprocess claims which were predicted widely as process. With this valuable feedback, we plan to further refine the PCTR implementation to allow for assigning multiple type information in future iterations.
Finally, the developed PCTR model can be applied in various real-world scenarios: (1) as a stand-alone model targeting only complex type claims, or (2) as part of a hybrid system combining a heuristic model (cf. Section 4.1) with the PCTR model.