Introduction

Patent documents are a rich source of technical and legal information, but their complexity often makes them difficult to understand quickly. Traditional summarization methods struggle with the specificity and technicality of patent language. Recent advancements in large language models (LLMs) and knowledge graphs (KGs) offer new opportunities for improving patent summarization. This paper explores the augmentation of LLM-based patent summarization using knowledge graphs to generate more concise and knowledgeable summaries. The primary focus is on leveraging Ollama3 for entity and relationship extraction to build a patent knowledge graph and subsequently using it to generate summaries. This approach aims to enhance the readability and informativeness of patent summaries compared to traditional methods.

Literature Review

Automatic patent document summarization is a critical task that has received significant research attention. Several methods have been proposed to solve this problem. For example, [1] presented a non-ontology-based document summarization method that uses keyword recognition and significant information density to extract paragraphs that best represent the patent text. [1,2] presented a new approach called PQPS, a state-of-the-art query-based patent summarizer that uses RBM and Bi-LSTM models for patent summarization. [2,3] used surface mining, NER, and machine learning techniques to extract concepts and conceptual relationships from the claims section of patent documents. Moreover, [4] described a summarization method for extracting surrogate patents to effectively support patent mapping. [5,4] examined the identification of conflicting proposals in patents using the BERT summarization method. Additionally, [6] discussed clustering patent generalizations into homogeneous groups for knowledge generation. Moreover, [7] studied feature term extraction from patent documents. [8] developed a multilingual patent text mining approach to compute relatedness scores of patent documents. [9] used graph representations to improve abstract summarization of multiple documents, demonstrating the potential of graph-based methods in summarization tasks.

Despite progress in this area, several shortcomings in patent document summarization methods have been identified, highlighting the need for more sophisticated and efficient methods to address them. The literature has noted the shortcomings of methods for summarizing patent documentation. According to [4] there is a growing demand for automated tools to support patent engineers and decision makers in patent analysis. Traditional methods face problems such as difficulties in natural language translation and the lack of parallel sets of bilingual documents [10]. Existing patent summarization strategies often fail due to the unique characteristics of patents and the emphasis on specific sections, such as claims, while ignoring important details from other sections [11]. In addition, the huge number of patent documents leads to information overload, which requires more effective summarization and analysis methods [12]. [13] provided a survey on Graph-based representation for text analysis and found out that the graph shows superiority over traditional methods. The author also suggested the novel approach in construction of semi-graph and sentence extraction, which helps finding linear and non-linear relationship between features [14].

Despite advances in text abstracting, annotations generated for patent documents still do not match human annotations [2]. In addition, the complexity and length of patents, as well as the requirement for domain knowledge, contribute to their under-utilization [5]. Moreover, limitations of existing patent analysis tools hinder the creation of impact technology patent maps, highlighting the need for improved methods for efficiently extracting information from patent documents [15]. Problems associated with cryptic descriptions in patent texts limit the effectiveness and comprehensiveness of patent analysis [16].

Several studies show that integrating LLMs into patent document summarization provides significant benefits in terms of accuracy and efficiency. Large language models (LLMs) have significantly impacted the field of automatic abstracting, especially in the context of patent document abstracting. The use of LLMs in summarizing patent documents has various advantages and disadvantages, as follows:

• LLMs are excellent at capturing the semantic meaning of text, which allows them to create contextually rich and coherent summaries [17], • LLMs are adept at handling the intricacies of patent documents, which often contain technical terminology and detailed information, resulting in more accurate and informative summaries [18], • LLMs demonstrate versatility in summarizing different types of documents, such as scientific literature, radiology reports, and patient problem lists [18], • LLMs have the ability to process voluminous input data, making them suitable for summarizing extensive patent documents without leaving out important details [17], • LLMs can be combined with knowledge graphs to improve generalization by integrating domainspecific knowledge and relationships extracted from the graph At the same time, the readability of summaries created using LLM may vary, making it difficult to ensure that all users, including non-specialists, can easily understand [Decker, 2023]. According to Decker [2023], graduates may have difficulty ensuring that their resumes are complete, especially when gathering all the relevant details from lengthy and detailed patent documents. Bush [2024] points out that LLMs can reflect biases present in the training data, potentially affecting the quality and objectivity of the summaries produced. The comprehension of LLM-based summaries can be a concern because understanding the rationale behind a particular summary generated by the model can be challenging [19].

Integration of LLM and KG in patent document summarization

As a solution to these problems, this paper proposes an integrated approach to summarizing patent documents using LLM with knowledge graphs. Knowledge graphs have become integral to various applications, gaining popularity particularly after the comprehensive representation by [20]. The concept of "Knowledge Graph" dates back to 1972 with the work of Schneider, E. W. Today, knowledge graphs are widely utilized in implementations such as the Google Knowledge Graph, Airbnb, and other social application analyses. Despite their broad applications, there has been limited research focusing specifically on patent datasets. By combining the power of understanding of the LLM with the structured representation of knowledge in the form of graphs, you can create more accurate, informative, and contextually rich summaries of patent documents. Large language models, such as pre-trained encoders [21], have shown promise in text summarization tasks. These models can effectively capture the semantic meaning of text and generate coherent summaries. Knowledge graphs play a critical role in organizing and presenting structured information. Research has shown that incorporating knowledge graphs into language models can improve the representation of domain information [22]. Research on question-answering systems has demonstrated the effectiveness of combining large language models with knowledge graphs to improve answer selection [23]. By using semantic relationships encoded in knowledge graphs, the system can better understand and extract relevant information to answer questions. This approach can be extended to patent document summarization to extract key ideas and relationships embedded in patent texts. [24] proposes the framework for predictive summarization highlighting the possible areas of patent applications in the future by analyzing the past data in the legal domain.

Moreover, leveraging external knowledge from open source knowledge graphs has been shown to improve the performance of document summarization models [25]. By incorporating external knowledge into the summarization process, the model gains access to a broader context and can generate more informative and complete summaries. Several studies have explored the integration of knowledge graphs into patent-related tasks, demonstrating their potential benefits. [26] provided a comprehensive overview of knowledge graph embedding approaches and applications, highlighting the importance of structured knowledge representation. [27] discusses the role of knowledge graphs in integrating health information and their application in healthcare, highlighting the benefits of incorporating knowledge graphs into data analytics. [28] implement knowledge proximity in a US patent database using a knowledge graph called "PatNet" built from patent metadata. Chen and Deng [2023] propose an accountable patent recommendation method using a knowledge graph to demonstrate the practical application of structured knowledge in patent-related problems. [29] extract knowledge graphs from patent applications, filling the gap in knowledge extraction from patent documents. [30] use text mining and machine learning techniques to visualize processes in utility chemical patents, demonstrating the utility of knowledge graphs in patent analysis. Additionally, [31] extend patent searching to include text embedding and knowledge graphs, emphasizing the importance of capturing various aspects of knowledge in patent documents.

In [3], the researchers highlighted that patent summarizations could be significantly enhanced using conceptual knowledge graphs. Similarly, [32] demonstrated the implementation of knowledge graphs using Open-IE to produce summaries through Cloze Reward. This paper builds on these studies by integrating language models to generate human-readable summaries from patent documents. LAUREN, as detailed by [33], focuses on summarizing knowledge graphs specifically for question answering systems. This approach leverages graph-based techniques to resize knowledge graphs, making them more manageable and tailored for answering user queries effectively. The summarization process ensures that the key information necessary for answering questions is retained while extraneous details are minimized. [34] addresses the summarization of knowledge graphs for search purposes. Their work proposes a method to compute a summarization set that provides maximal descriptive power with minimal redundancy. This approach enhances the efficiency of knowledge graph searches by presenting users with the most relevant information. [35] discuss the concept of adaptive summarization to create personalized knowledge graphs. This method adapts the summarization process to the user's specific information needs, ensuring that the resulting knowledge graph is both concise and highly relevant. The adaptability of this method makes it particularly useful for applications requiring tailored knowledge delivery. The GATES framework, presented by [36], utilizes graph attention networks for entity summarization within knowledge graphs. This technique addresses the challenge of summarizing entities by focusing on the most significant relationships and attributes. The use of attention mechanisms allows for the generation of summaries that are both comprehensive and focused on the most pertinent aspects of the knowledge graph.

These studies collectively advance the field of knowledge graph summarization by addressing various challenges such as customization, scalability, and efficiency. They provide a solid foundation for further research and development in creating concise, informative, and user-tailored knowledge graph summaries.

Methodology

Building the Patent Knowledge Graph

The initial step involves constructing the knowledge graph by creating triplets that represent entities and relationships extracted from patent documents. This requires identifying subjects, actions, and objects within the text. Patent documents, characterized by complex sentence structures and nounheavy content, present a challenge for entity extraction. Preliminary attempts using BERT models proved inadequate for the patent dataset.

To address this, we employ llama-3 pre-trained language model1 with the parameter of 8.0B, which effectively identifies entities and relationships from the text, specifically from the description and claims sections of patents. The description section provides key terms, while the claims section offers detailed information about the invention, both crucial for generating a comprehensive summary. By excluding other sections, we aim to minimize the hallucination effect of the language model. The extracted data is then used to construct a knowledge graph representing the core elements of the patent. The knowledge graph is created in the Neo4j database using Cypher queries with the extracted relationships and entities on the previous step, ensuring a structured and accessible representation of the patent as shown in Figure 2.

Generating Summaries from the Knowledge Graph

With the knowledge graph established, the next step involves generating summaries using the language model (llama-3:8.0B). This involves transforming the structured data from the knowledge graph into a human-readable format that emphasizes conciseness and informativeness. To obtain an effective abstractive summary, it is essential to instruct the language model with a custom prompt. This step ensures the generation of a coherent summary rather than a disjointed question-and-answer format. Additionally, Cypher queries are utilized to retrieve all relevant nodes and relationships from the knowledge graph, facilitating the creation of a comprehensive summary.

This methodology leverages the strengths of both knowledge graphs and advanced language models, aiming to produce high-quality, informative summaries of patent documents, thereby enhancing the accessibility and understanding of patent information.

Baseline Method: Language Model Summarization

The performance of the proposed method is compared against a baseline summarizing technique that solely utilizes a language model for document summarization. The baseline approach is straightforward: the language model is employed to summarize the content of the patent documents to generate a relevant summarization. We defined this baseline to suit the summarization of one document rather than multiple documents. In this step, we come to notice that we can use multiple document summarization using RAG architecture, which we aims to do the further research. Nevertheless, in this current baseline, the user input is the patent document itself, which is then processed to retrieve relevant sections within the document (such as the description and claims). These sections are fed into the language model, which generates a cohesive summary of the patent.

Our adaptation involves ensuring that the retrieval mechanism focuses on internal sections of the patent document, effectively treating it as a self-contained database. This approach leverages the structure of the patent document to enhance the quality of the generated summary. This summary is intended to capture the essence of the invention described in the patent document. The simplicity of this approach allows for a clear comparison with more complex methods.

Algorithm 1 Summarizing Patent Documents Using Knowledge RAG and Llama3

Input: HUPD dataset, Neo4j credentials, patent number.

Output:

Generation of the structured summary of the patent document.

Dataset

We utilized the HUPD (Harvard University Patent Dataset), which provides an extensive range of raw patent claim texts. This dataset was chosen over other prominent datasets such as BIGPATENT2 , CLEF-IP 2011, and USPTO-2M for several key reasons. Firstly, HUPD houses approximately 4.5 million patent documents, significantly more than BIGPATENT's 1.3 million [37]. This expansive volume offers a richer and more varied base for analysis. Secondly, HUPD encompasses 34 distinct sections, including claims, offering a more comprehensive view compared to BIGPATENT's limitation to just 4 sections. Lastly, unlike the pre-tokenized texts in BIGPATENT, HUPD's provision of raw texts affords greater flexibility and depth in our analysis, allowing for a more tailored approach to data processing [38].

To test our proposed method, we randomly selected approximately 10 documents from the 'G' category of the CPC code (Physics) within the HUPD dataset as representative samples for our study. We generated both baseline summaries and knowledge graph summaries for each patent document and used ROUGE scores to track the performance metrics. Additionally, we employed human evaluation to assess the real-world clarity of the generated summaries.

Evaluation

This section presents the results of our proposed method compared to the baseline summarization technique.The proposed method was evaluated using ROUGE scores to measure the quality of the generated summaries. The evaluation metrics included ROUGE-1, ROUGE-2, and ROUGE-L, which assess the overlap of n-grams and longest common subsequences between the generated and reference summaries. Moreover, we evaluated the performance with human evaluations to assess the quality and understandability of the summaries generated.

Quantitative Result

The ROUGE (Recall-Oriented Understudy for Gisting Evaluation) metric, introduced by Chin-Yew Lin in their pivotal work, is a cornerstone in the field of natural language processing, particularly for evaluating automatic summarization and machine translation [39]. ROUGE assesses the quality of a summary by comparing it with reference summaries.

ROUGE scores are often reported in terms of precision, recall, and F1 score, providing a comprehensive evaluation framework:

• Precision measures the fraction of n-grams in the generated summary that are present in the reference summary. • Recall assesses the fraction of n-grams in the reference summary that appear in the generated summary. • F1 Score is the harmonic mean of precision and recall, offering a balanced metric.

The formulas for these metrics are as follows: Precision = Number of overlapping n-grams Total n-grams in the generated summary

Recall = Number of overlapping n-grams Total n-grams in the reference summary ( 2)

𝐹 1 = 2 × Precision × Recall Precision + Recall(3)

Table 1 shows the ROUGE F1 scores for both the baseline and the proposed method. The baseline method outperforms the proposed method across all ROUGE metrics, indicating a higher F1 score for summarization.

Human Evaluation Result

The human evaluation of our summarization method was a critical component of our research. We aimed to determine the effectiveness of our Knowledge Graph (KG) based summaries in terms of understandability and informativeness. To this end, we conducted an evaluation involving 12 respondents who reviewed summaries of 10 different patent documents. We selected 10 patent documents from the Harvard University Patent dataset [38] to ensure a diverse and representative sample. For each patent, three types of summaries were generated: the original summary, a baseline summary, and the KG-based summary. Each respondent was presented with these three summaries for each patent document without being informed of their origins to prevent any bias.

Voting Process

Respondents were asked to vote for the summary they found easiest to understand and most informative. The summaries were presented in a randomized order to eliminate order effects. The voting was conducted using a simple voting system where respondents chose one summary out of the three options. The criteria for voting were clearly defined for understandability: how easily the summary could be comprehended, and, informativeness: how well the summary conveyed the essential information of the patent document.

To ensure the integrity of the evaluation, respondents were chosen from a pool of individuals with no prior experience in reading or writing patent documents. This choice was made to simulate the perspective of an average reader who might not have specialized knowledge. The human-evaluation results are summarized in Table 2. The KG-based summaries received the highest number of votes in most cases, indicating a clear preference for their comprehensibility and informativeness over the other two types of summaries.

Discussion of Results

The results from the ROUGE F1 score evaluations indicate that the baseline method outperforms the proposed Knowledge Graph (KG)-based summarization technique in terms of automated metrics. ROUGE scores measure the overlap between the generated summaries and reference summaries, focusing on precision, recall, and F1 score.

However, the positive feedback from human evaluations suggests that the proposed KG-based method generates clearer, more coherent, and more informative summaries. This contrast highlights the importance of human evaluations in assessing summarization quality, as they capture aspects of readability and informativeness that automated metrics might miss.

Notably, for Patent Number "US14990535, " the baseline method failed to produce a summary, likely due to hallucination effects commonly observed in large language models (LLMs). In contrast, the KG-based method successfully generated a precise and informative summary from the original patent document, demonstrating its robustness.

It is important to note that our primary focus is on understandability and informativeness as perceived by human readers, rather than solely relying on automated metrics like ROUGE scores. This approach ensures that the summaries effectively convey the essence of the inventions described in the patents.

The integration of knowledge graphs and the adaptation of the RAG architecture have proven effective in enhancing the readability and informativeness of patent document summarization. These findings underscore the value of human-centered evaluations in developing summarization tools.

Limitations

Despite the promising results, our research faced several challenges and limitations. The primary limitation stems from the reliance on large language models (LLMs) LLaMA3 for both entity recognition and summary generation introduces potential vulnerabilities. If the LLM fails to accurately produce entities, the subsequent knowledge graph cannot be generated correctly. Any errors in the entity recognition process can lead to inaccuracies in the final summary. This dependency on LLMs introduces a potential source of error that can affect the overall performance of the summarization method.

Additionally, the LLM used for generating summaries, specifically LLaMA3, may produce sub-optimal results if the prompts are not clearly defined or easy to understand. This can result in summaries that do not accurately reflect the content of the original document. Another limitation is the language support; currently, our method is only capable of handling English documents, which restricts its applicability to patents written in other languages.

Moreover, our approach is designed for single-document summarization. This limits its effectiveness when dealing with multiple documents, which is a common scenario in patent analysis and other legal contexts. Addressing this limitation is a key area for future research.

Nevertheless, the human evaluation results suggest that despite the lower ROUGE scores, the summaries generated by the proposed method are more understandable and informative to readers unfamiliar with patent documents. This highlights the importance of human-centered evaluation metrics in assessing the effectiveness of summarization methods, as automated metrics like ROUGE may not fully capture the nuances of human understanding.

Future Work

To address the limitations identified, future work will focus on several key areas of improvement:

Enhanced Entity Recognition: We plan to develop a dedicated entity recognition model trained on annotated patent data. By creating a custom model tailored to the specificities of patent documents, we aim to improve the accuracy of entity recognition and, consequently, the quality of the knowledge graphs and summaries.

Improved LLM Training: We will train the LLM, such as LLaMA3, on a specialized legal database to enhance its performance in summarization tasks. This targeted training will help mitigate issues arising from unclear prompts and improve the model's ability to generate accurate and informative summaries.

Multilingual Support: Expanding our model to support multiple languages is a priority. This involves training the entity recognition and summarization models on multilingual datasets to ensure accurate performance across different languages.

Multi-Document Summarization: Developing our method to handle multi-document summarization will significantly increase its applicability. We plan to extend the architecture to aggregate and summarize information from multiple documents, which is particularly important in the legal and patent analysis fields.

By addressing these areas, we aim to refine our summarization approach, making it more robust, versatile, and applicable to a wider range of documents and languages. This will enhance its utility in practical, real-world scenarios, particularly in legal and technical domains.

Conclusion

In this paper, we introduced a novel summarization approach that integrates knowledge graphs and large language models (LLMs) to enhance the readability and informativeness of patent document summaries. Our main contribution lies in generating summaries that are easier to understand while maintaining the key inventions from the patent documents. Additionally, our research contributes towards advancing the legal technology by analyzing entities in the patent domain and producing structured data through knowledge graphs combined with the contextual understanding of LLMs for automatic summarization tasks. We emphasize the importance of human-centered evaluations and contextual understandability over automated evaluations when producing summaries. Despite the limitations, our research lays a foundation for future advancements in patent summarization technology, aiming to develop a more robust tool for analyzing complex technical documents in legal domain.