Patents are an important source for technological innovation, utilized by companies to disclose their inventions as well as protect intellectual properties legally. Due to the exponential growth of available patent data, keyword extraction has become an important task for the eficient analysis and organization of patent documents. Keywords extracted from the text of a patent comprise highly relevant terms or phrases that represent the content of the patent document. Further, such terms are exploited by various patent applications such as freedom-to-operate analysis, prior-art-search, etc. Most of the existing methods are not able to extract useful terms that help to understand the core of the patent, i.e., the invention. Moreover, these approaches focus on either supervised settings, which require large amounts of training data, or unsupervised settings that cannot extract semantically meaningful keywords. To ifll these gaps, this paper proposes a weakly-supervised deep neural network model (DeepKEA) which is designed to extract terms that are closely related to the topic of a given patent document and the invention it describes. It consists of two main modules: (1) a training data generation module, (2) a deep neural network module. The experiments show that our model yields better performance than existing baselines.
Intellectual property (IP) rights play an important role in the creation, dissemination, and use
of new knowledge for further technological innovation. Patent documents which are complex,
heterogeneous, and lengthy in nature, contain scientific, technical, legal, and business-relevant
information. The so-called full text of a single patent document consists of a title, an abstract,
claims, and a detailed description. While the description part describes the embodiment of
the invention, its use, and the benefits it ofers for target applications, the claims give a clear
definition of what the patent legally protects, i.e. they define the scope and boundaries of an
invention for the purpose of legal protection. In order to deal with steadily growing patent
data, researchers started to employ AI-based approaches to support experts in patent retrieval
and analysis processes. One crucial task for patent searchers is to find the right information
that can be used to support business-critical decisions. To seek such crucial information and
explore patent data fast and eficiently, various automatic keyword extraction methods have
been developed [
Recently, several supervised approaches have been proposed [
In this study, we present our preliminary work toward a novel patent keyword extraction model (DeepKEA) based on deep neural networks. DeepKEA starts by extracting noun phrases from the abstracts and claims of patent documents. These extracted noun phrases serve as initial candidates for relevant terms. To refine these phrases expert validation is employed. In other words, each extracted noun phrase is reviewed and validated by a patent expert internally. It is important to note that this study focuses on the abstracts and claims, and the exploration of noun phrases within patent descriptions will be addressed in future work. In the second step, DeepKEA uses the extracted keywords and their corresponding original patent text to train a deep neural network. Finally, the trained model allows to extract a list of keywords for a given arbitrary patent document.
Overall, the main contributions of the paper are: • A pipeline to generate training data for the patent keyword extraction task, • A neural network architecture for generating embeddings of patent documents and keywords, • The adoption of an approximate nearest neighbor search for eficient keyword extraction based on dense vectors.
Problem Formulation. Given an input patent document , which contains a set of noun phrases d = {1, 2, ..., }, the goal is to output the most relevant top- noun phrases as keywords ′ = {1′, 2′, ..., ′ }, where ′ ⊆ .
Overview. The general workflow of DeepKEA is shown in Figure 1. There are two main modules of the proposed workflow: (1) the Training Data Generation Module, and (2) the Deep Neural Network Module. Section 2.1 and Section 2.2 provide a detailed description of each module and the feature sets that have been utilized by each module.
The training data generation module is responsible for assigning meaningful noun phrases present in the abstracts and claims as keywords to the respective patent documents. The noun phrases are identified by employing an NLP pipeline using Spacy 1. These extracted noun phrases are validated and refined internally by patent experts. This examination involves the reviewing of keywords based on the content of the corresponding patent document. This curation process ensures that the extracted noun phrases align with the content of the patent document. Further, they serve as a set of keywords specifically assigned to the corresponding patent documents for the training phase.
The second module of the workflow involves the training and serving of a deep neural network
model. We have adapted an approach which uses deep learning for recommendation systems [
At serving time, we need to compute the most likely classes (keywords) for each patent document based on the generated patent and keyword embeddings. However, scoring a large amount of keywords is expensive. Therefore, to address the challenge of computational expense, an approximate nearest neighbor lookup based on inner-product is performed to eficiently generate the top- keywords, for which Faiss3, a library for eficient similarity search of dense vectors, is used.
Finally, when confronted with a new patent document in the serving stage, the workflow involves creating embeddings for the document’s title, abstract, and claim sections using the sentence transformer with BERT for patents, as previously described. Subsequently, the trained deep neural network, DeepKEA takes these embeddings as input and generates the embedding of the given patent document. Utilizing inner-product similarity top-N keywords are identified by referencing precomputed keyword embeddings.
This section provides a description of the dataset and the baselines, followed by the experimental results and a comparison to the baseline approaches.
For the evaluation of the proposed model, we gathered internally a dataset consisting of 9,664 unique patents. The patent documents are utilized as input to the proposed workflow. Each patent document is paired with its keywords that are extracted from the abstracts and claims by applying the first step of the workflow, i.e., training data generation. The extracted noun phrases are then validated by experts. Further, the keywords that appeared less than 100 times in the entire dataset are filtered out. After applying the first step the average number of extracted keywords per patent is 13.71, resulting in a total of 14,957 unique keywords. The dataset4 is organized into pairs consisting of keywords and their corresponding patent documents. Finally, the data is split into 13,586 and 1,371 pairs as train and test data, respectively. The test data consists of 100 unique documents and their corresponding keywords.
To evaluate the performance of DeepKEA, two diferent baselines are selected: • TF-IDF is a standard baseline due to its simplicity and efectiveness. It is applied on noun phrases of the patent documents and based on the tf-idf score top- keywords are assigned. • BERT for Patents exploits the vector similarity between documents and keywords. Each document and its keywords (noun phrases present in the document) are firstly converted into their vector representations with the help of sentence transformers with BERT for Patents. Based on the vector similarity between the document and keywords, top- keywords are assigned.
0.678 0.572 0.313
In this preliminary study, we present DeepKEA, a deep neural network model for extracting highly relevant keywords (noun phrases) from patent documents. First, the training data is generated by leveraging the abstracts and claims of patent documents. This initial set of noun phrases is then subjected to validation and filtering by domain experts. Second, the training data is used to train a deep neural network for obtaining the embeddings of documents and keywords. Finally, the model assigns keywords to individual (unseen) documents by applying an approximate nearest-neighbor search based on dense vectors. The experimental results showed that DeepKEA outperforms the baselines. As for future work, we aim to (1) improve the training data generation module by exploiting additional semantic information as well as description part, (2) improve the deep neural network module by including more structured features of patent documents, such as CPC (Cooperative Patent Classification) codes, citations, inventors and applicants.