A fundamental step in the patent application process is the determination of whether there exist prior patents that are novelty destroying. This step is routinely performed by both applicants and examiners, in order to assess the novelty of proposed inventions among the millions of applications filed annually. However, conducting this search is time and labor-intensive, as searchers must navigate complex legal and technical jargon while covering a large amount of legal claims. Automated approaches using information retrieval and machine learning approaches to detect novelty destroying patents present a promising avenue to streamline this process, yet research focusing on this space remains limited. In this paper, we introduce a novel data pipeline, ClaimCompare, designed to generate labeled patent claim datasets suitable for training IR and ML models to address this challenge of novelty destruction assessment. To the best of our knowledge, ClaimCompare is the first pipeline that can generate multiple novelty destroying patent datasets. To illustrate the practical relevance of this pipeline, we utilize it to construct a sample dataset comprising of over 27K patents in the electrochemical domain: 1,045 base patents from USPTO, each associated with 25 related patents labeled according to their novelty destruction towards the base patent. Subsequently, we conduct preliminary experiments showcasing the eficacy of this dataset in fine-tuning transformer models to identify novelty destroying patents, demonstrating 29.2% and 32.7% absolute improvement in MRR and P@1, respectively.
increasingly unsustainable, driving a growing interest in
the usage of information retrieval (IR), machine learning
Patent search is a rich and challenging space which com- (ML), and deep learning (DL) approaches to streamline
prises a diverse set of tasks, including Freedom to Op- search methodologies and optimize result relevance, for
erate (FTO) searches, novelty or patentability searches, example, via query expansion and targeted semantic
simand validity searches. Within this spectrum, patentability ilarity techniques [
Traditionally, prior art searches have been performed can generate multiple novelty destroying patent datasets. manually, with searchers iteratively crafting and revising We leverage publicly available United States Patent and complex keyword and Boolean queries to obtain the most Trademark Ofice (USPTO) APIs in order to curate such relevant documents. However, with the number of patent datasets, alongside web-scraping Google Patents. To simapplications and volume of prior art growing annually, plify the problem, we focus only on identifying novelty the labor-intensive nature of this process has become destroying patents, rather than all potential literature contributing to novelty destruction. Our contributions are as follows: 5th Workshop on Patent Text Mining and Semantic Technologies (PatentSemTech) 2024 $ arav.parikh@uconn.edu (A. Parikh); shiridh@uconn.edu • We construct ClaimCompare, a pipeline utilizing the (S. Dori-Hacohen) USPTO API and Google Patents to generate curated 0000-0002-0877-7063 (A. Parikh); 0000-0002-0877-7063 novelty destroying datasets. (S. Dori-Hacohen) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License • We utilize ClaimCompare to curate a sample dataset Attribution 4.0 International (CC BY 4.0).
PatentMatch v2 CC Sample
there is little work on the important, highly-specialized task of detecting novelty destroying patents.
The now discontinued CLEF-IP tracks in 2012 and 2013
present useful datasets related to patentability searching.
The 2012 edition released a claims to passage dataset
containing 2.3 million European patent documents with 2,700
corresponding relevance judgements [
A key paper on the task of novelty evaluation is
PatentMatch [
To the best of our knowledge, our paper is the first to approach novelty destruction from a US-centric approach, and ofers the first public dataset for this task as well as a pipeline to easily generate additional datasets1.
We now introduce the ClaimCompare data pipeline (Figure 1) which generates domain-centric and agnostic datasets in order to train DL models to assess patent novelty. For the remainder of the paper, we focus on describing our pipeline, sharing information about the sample dataset, and demonstrating its efectiveness for this task via preliminary experiments involving large language models (LLMs) fine-tuned on this sample dataset. 3.1. Approach To ground the use of our ClaimCompare pipeline in the context of novelty determination specifically, we define the novelty destruction search problem as a sub-problem of the prior art retrieval process. Accordingly, ClaimCompare operates under the assumption that a preliminary set of prior art patents of size has already been retrieved by the searcher for a given query (i.e., base patent ), using preexisting methods. While the models trained with ClaimCompare’s datasets can certainly be used to improve direct retrieval of only novelty destroying patents (i.e., when is made suficiently large), in this paper we focus on applying these models as a filter on top of a smaller subset of retrieved patents.
To develop ClaimCompare, we primarily rely on two publicly available USPTO APIs to access the patent data US Patent and
Trademark Office (USPTO)
User-Selected Seed Queries
Claim Compare Pipeline Bulk Data API Call
Keyword Extraction (top 5 keywords) +
Bulk Data API Call Keyword query results Office Action API Call (query by application #)
Query results
Patent Application
(Base) Rejection text
RReel alateteddPatents (N(deeNR(deegRN(dleeeasNgR;(deeesgRNt(dlt;neeraesNtgel;oenreaeosgtt;ldoneryasoetgtt;lPonryaoegittt;dnonnPryioeatt;dnonnoygoeitadnPnntgyoiov)tedPnntogoi)vaetennPngova)tePlnongt)tvaeentlysov)atteelnoystveetlnyvtteelnysttelnysttlysttys (N(NReRge;lnaotetdnoPvaetlteynts destroying) ddeesstrtoroyyiningg))
Citation Extraction
Citation #
Cited
Patent(s)
(Pos; novelty
destroying)
redirect to
up-to-date
patent #
necessary to teach subsequently trained models to se- if applicable, and other related patents. For the cited
mantically diferentiate between novelty destruction and patents, we query the USPTO Ofice Action Citation API 3
mere relevance. Relevant patents constitute “negative" using the base patents’ application numbers. If an ofice
samples in our dataset and are queried for with keywords action is found with a 102 rejection, we take the rejection
extracted from the base patents, mimicking a common text and pass it to a text2text generation base T5 model
prior art search practice. In contrast, novelty destroy- along with a standardized prompt in order to extract the
ing “positive" samples are generated by obtaining USPTO publication number of the novelty destroying patent cited
ofice actions with rejections citing novelty destroying within the text. The T5 performs the task eficiently with
prior art patents. For our positive samples, we only con- a 94% success rate, which is suficiently high for our needs.
sider ofice actions that contain a 102 rejection, indicating We then perform a simple cleanup on the publication
that the citation is considered to be novelty destroying number and use it to query for its claims via Google
for the corresponding application by skilled patent ex- Patents4, which can web redirect to the most up-to-date
aminers. Our focus on 102 rejections is deliberate, as version of the patent if the number is outdated, unlike
they only reference elements from a single document as the APIs. To acquire our negative samples, we extract
suficient to destroy the novelty of a patent application, keywords and phrases from the base patent abstract. We
whereas the more nuanced and complex 103 rejections apply the KeyBERT model to get the top 5 keywords
often refer to elements from multiple patents and/or com- from the abstract [
For each retrieved patent application, we collect the rows, 36 (10%) have two positive samples; there are no application and publication numbers, abstract, and claims rows with three or more novelty destroying patents in our in order to form our set of base patents; we then set dataset. To ensure that the number of samples per row out to acquire their cited novelty destroying patent(s),
download
citations-api-beta 4https://patents.google.com is always , we set _25 and/or _24 and their corresponding publication numbers as null values, depending on the number of novelty destroying patents present. If none are found for a given base patent, then the rejection columns are set as null instead.
Clearly, given the structure of our dataset, there is an inherent lack of balance between the two classes. We deliberately maintain this imbalance for two main reasons. Firstly, it reflects the current state of prior art searching where there are far more relevant samples than novelty destroying samples for any given patent. While imbalanced datasets typically pose challenges for training ML or DL models, we are intrigued to explore how this realistic representation of class distribution influences model performance in our experimentation. Secondly, although we do not directly train a ranking model in our experimentation, we indirectly test the ability of our model to rank prior art patents based on their likelihood of invalidating the novelty of a given base patent, a task which necessitates a larger value for and, consequently, an imbalanced dataset. P@1 0.651 0.651 0.978 0.975 0.934
General Baseline Domain Baseline Fine-Tuned ( = 25) Fine-Tuned ( = 10) Fine-Tuned ( = 5)
We also intentionally downsample the negative
samples in our training dataset, to observe the efect this has
on both the validation and testing metrics. We perform
this downsampling by simply reducing = 25 to = 10
and = 5 on our three shufled train-val-test splits,
retaining all positive sample while randomly sampling the
required number of negatives examples from the larger
set for each base patent.
4.2. Model Fine-Tuning
4. Experimental Setup and Results For our experiments, we fine-tune a sequence
classification model with our sample dataset. We primarily use the
In order to assess the efectiveness of our dataset in train- base DistilRoBERTa model 5 due to its compact size and
ing LLMs to assist with novelty determination, we test robust performance on related tasks [
We use an 80-10-10 stratified train-val-test split, with using a simple logical OR as an ensemble, such that if each of the splits possessing roughly the same propor- any patent in the set is deemed to be novelty destroytion of positive samples. To avoid data leakage, the base ing, the base patent is found not novel. To quantify the patents are restricted to one of the splits such that all 25 model’s performance on this task, we compute classificapairs can be found in that split. To mitigate the efects of tion metrics suitable for our imbalanced dataset such as sampling bias, we randomly subsample the raw dataset average precision (AP) and area under the receiver opertwice more to generate a total of three unique train-val- ating characteristic curve (AUROC). Since these metrics test splits for our model. The results we present are all averaged across these three runs.
require prediction scores as opposed to labels to com- of both the validation and testing sets. Manual examinapute, we take the maximum novelty destruction score tion of a significant portion of the dataset shows no sign (i.e., pairwise probability of the positive class) out of all of data leakage, and the training appears to be sound as the patents as the representative score for the set. well. Thus, we hypothesize that the high absolute results
Additionally, we also evaluate the model in the con- are the consequence of the negative samples selected text of ranking, using their novelty destruction scores via keyword search being far too “easy” to diferentiate for rank ordering the test patents in each set of 25. We from the positive, novelty destroying samples. Whether use ranking metrics such as MRR and Precision@1 to as- this issue stems from the keyword extraction process sess the performance. The unique nature of the novelty itself, the fact that more powerful Boolean queries are destruction task means there are often no members of needed to obtain the most relevant results, or perhaps the positive class in a given set; we therefore introduce a even another unseen factor, requires further experimenplaceholder patent, assigned a score of 0.9, in order to ac- tation. We leave an in-depth exploration of how to levercurately compute these metrics. The placeholder patent age unique inter-patent relationships to build upon our is inserted into the set prior to ranking, in order to denote pipeline to future work; likewise with the task of finda patent which the model regards as reasonably novelty ing more semantically similar non-novelty destroying destroying. To facilitate accurate comparisons across the patents (i.e., harder negatives) to match with the applicaboard on the testing set, we introduce the placeholder tion to improve robustness at the classification boundary. patent to all of the patent sets, such that if a novelty de- We note that the USPTO has a new Citation API that can stroying patent is actually present, it is ideally ranked potentially be of use for these goals. ifrst, above the placeholder patent. Alternatively, when We also invite future work to employ ClaimCompare no novelty destroying patent is present, the placeholder in order to generate additional datasets, inclusive of adpatent is ideally ranked #1. ditional technical fields, for example, by creating queries
Examining the impact of fine-tuning with our dataset that utilize Cooperative Patent Classification (CPC) codes on model performances (Table 2), we see significant im- instead of keywords. With these datasets, future work provement over both baselines. As expected, the models can train state-of-the-art models, such as generative perform quite poorly in the baseline, zero-shot setting, es- LLMs, to improve performance even further. pecially in terms of classification, with BERT for Patents exhibiting only a slight improvement over DistilRoBERTa.
The near-perfect classification and ranking performance 5. Conclusion metrics we see after fine-tuning, however, are quite surprising. The high AUROC and AP, in particular, indicate that the model is able to diferentiate between the positive and negative samples with a high degree of confidence.
The = 25 model is the top performer, but with little distinguishing it from the = 10 model. This is intriguing as the model appears to be performing better in spite of the greater imbalance, perhaps simply due to the presence of more data allowing the model to better learn the relationships defining novelty destruction.
pare, to generate datasets geared towards patent novelty evaluation; ofer a sample dataset generated using ClaimCompare; and assess its utility in fine-tuning a novelty destroying classifier. We leverage USPTO APIs to obtain our novelty and non-novelty destroying data. To the best of our knowledge, this is the first usage of USPTO ofice actions and patent claims as a source of data, providing high quality datasets while being fairly straightforward, lfexible, and easy to replicate.
We believe ClaimCompare holds potential to accelerate research in the patent retrieval field, in conjunction with the use of LLMs and other cutting-edge DL models. Improving novelty determination holds the promise of reducing the time and monetary burden required for patent search and patentability determinations, while also increasing accuracy, thereby saving searchers’ time and making patent databases more accessible for all. In this sense, we hope that our pipeline can facilitate the democratization of what has been a historically complex process, and enable inventors, attorneys, and patent examiners alike to assess patent novelty with greater ease. 4.4. Limitations and Future Work These results, though incredibly promising, point to some limitations due to the large gap between the baseline and fine-tuned models, and exceptionally high testing scores. While we can confidently say that the fine-tuned models are more suited for the novelty evaluation task than the base models, further comparisons are needed to baseline models pre-trained on general legal data, which would likely provide a stronger baseline; however, due to computational resource limitations at submission time, we leave this to future work.
We note that our models do not appear to be overfitting, given that the loss trends during training are nominal, and these models are generalizing well to the unseen data