A Patent Semantic Representation Using Technical Compound Sentences

Abstract

The claims of a patent define the scope of exclusive rights to an invention, containing all essential technical features reflecting the novelty and non-obviousness. Current patent text mining methods have not fully leveraged patent claims by considering the expression of technical features in patent claims. In this study, we clarify the textual structure of patent claims and model the claims in a patent as a tree by capturing the denpendency relationships among the patent claims. We derive patent technology compound sentences (TCS), then propose a novel patent semantic representation based on TCS. To evaluate the proposed patent representation, we apply relational and direct strategies of empirical evaluation to a dataset of USPTO. The results show that our TCS-based and quantity-quality-weighted representation for patents outperforms other methods on task of P2P similarity and automated IPC symbol classification, which suggest that TCS enables more eficient use of technical information of the patent claim. The potential application of the novel representation in novelty analysis is discussed as well. The foundamental patent representation method using TCS could unleash the value of patent claims as technical information resource, and have many potentials in improving many subsequent tasks of patent mining.

Keywords

• eol>Claim tree
• patent semantic representation
• technical compound sentence

1. Introduction

proved methods to deal with patent claims. In this study, we propose a method of patent technology compound Patent documents are valuable resources for technology sentences (TCS) to structure patent claims, then apply text mining. As a combination of legal and technical it to design a novel patent semantic representation. We terms, patent text difers significantly from other types evaluate the proposed patent semantic representation on of documents as scientific articles [ 1, 2 ]. The character- a patent dataset. The fundamental patent representation istics of patent text should be considered and utilized method based on TCS could unleash the resource value in patent text mining. To this end, many recent tech- of patent claims, and have many potentials in improving niques of patent mining have increasingly employed a many subsequent tasks of patent mining. few methods like information fusion and text reorganization [ 3, 4 ]. As an important element in patent document, patent claim outlines the scope of an invention’s exclu- 2. Related work sive rights and include all essential technical elements that demonstrate its novelty and non-obviousness. Patent For patent semantic representations, terms and phrases claim has been exploited by many applications of patent [ 14, 15 ] or original text [ 16, 17, 18, 19, 20 ] are used as mining, including patent infringement detection [ 5, 6, 7 ], the input. Keywords extraction and subject-action-object patent evaluation [ 8, 9, 10 ], patent classification and clus- (SAO) analysis are leveraged to describe the technologies tering [ 11, 12, 13 ], patent information representation, etc. embedded in the patent texts. These methods, however, Therefore, it is an essential task to design text process- could be unable to capture the relationships within the ing methods of patent claims by fully leveraging their technical concepts and might overlook some of the techfeatures. However, current studies have not yet clarified nical specifics. The original text may be a superior option the textual structure of patent claims, nor designed im- in terms of information integrity with the advancement of NLP techniques. Title and abstract of patent are dePatentSemTech’23: 4th Workshop on Patent Text Mining and sirable sources of technical information, yet the claim of Semantic Technologies, July 27, 2023, Taipei, Taiwan. patent alone is able to achieve state-of-the-art results [ 12 ]. * Corresponding author. Recently, a growing body of research has concentrated $ [e-mail] (S. Xiang); [e-mail] (J. Mao); on applying patent claim in patent semantic [e-mail] (G. Li) tation for its delicate writing [ 3, 14, 18, 19, 20, 21 ]. Yet 000000-0000-0020-0823-3362-5498-9711(6G9.(LS.i)Xiang); 0000-0001-9572-6709 (J. Mao); the virtue of patent claims’ characteristics on NLP tasks © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License are not always valued, and the particularities of patent CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) claim are not dealt with properly. There have been some further studies which optimize the input by attending technicalities. Furthermore, TCS enables the disambiguato characteristics that distinguish patent text from other tion of claims following the serial dependency and claims text types, such as information enhancement with patent following the parallel dependency. The claims following citation [ 3 ], or input transformation according to claim the serial dependency add into the length of TCS, i.e., the structure [ 20 ]. These methods leverage idiosyncrasies technicalities volume of a full description. The claims of claim text to some extent. To our knowledge, little following the parallel dependency add into the count research on patent semantic representation utilizes the of TCS, i.e., generalize and thus expand the scope of a specific structure and internal logic of technical informa- patent. As shown in fig.1, the example patent claim can tion within patent claim. Therefore, we contribute to the be break down into 12 TCSs, and each of them consists research on patent semantic representation by provid- of 5 claims. ing an embedding method that can capture the nuance internal logic of patent claims.

3. A representation using technical compound sentence

3.1. The tree structure of patent claims

The claims of patent can be classified into independent claims and dependent claims. Independent claims describe diferent embodiments or aspects, uses, or methods of producing the invention. Dependent claims refer back to and further limit another claim or the claims in the same application, to further limit the scope and complete the description with more details. The technological embodiments of dependent claims are embedded in the independent claims. With such structure, the patent claims can be model as a tree. Typically, each patent claim is provided as a separate numbered sentence, and the referenced claim is easily identified in the sentence. Theoretically, it is easy to identify the dependencies of patent claims and construct the tree structure of claims [ 22, 23, 24 ]. We refer such tree structure of claims as claim tree. In a claim tree, a claim follows serial dependency refers to the previous claim, and a claim follows parallel dependency refers to claim or claims before the previous one. Serial dependency between claims adds into the depth, and parallel dependency adds into the breadth, resulting in varying structures. We develop a method for semantic representations of patent based on technical compound sentence (TCS). The embedding vector of a patent is the weighted average of the embedding vector of its TCSs, where the weights are based on the quantity Q(s) and quality F(s) of the information the TCS contains. The representation is obtained through ⃗ = ∑︀

1 ∈ () ∈ ∑︁ ⃗ × () × () (1)

A patent claim can be represented as a graph where nodes

3.2. Construction of Technical Compound are terms of the claim. The graph-of-words of patent claim C is defined asG = (V,E) where V is the set of nodes

Sentence that represents the nouns and verbs of C and E is the The logical connections between technicalities embodied set of edges which represents the co-occurrence of the in the claims are reeflcted by the dependencies of claims. words in a 1-size window. Information quantity Q(s) of a Therefore, a path from the root to the leaf nodes in claim TCS is determined by its cover of level H(s) and cover of tree denotes a chain of claims that together provide a full breadth R(s) of the claims it includes. Cover of level H(s) statement of an aspect, use, or method of fabricating the is the maximum depth of a claim that form the TCS in the invention. A technical compound sentence (TCS) is con- claim tree, which is positively related with more technical structed by combining the claims of the path in sequence. details. And cover of breadth R(s) is measured by radius It is capable of grasping the progressive and explanatory of subgraph of the TCS , which can describe the scope relationships of claims, as well as the superior and sub- of technical information the TCS contains. Information ordinate relationships between technical concepts and quantity Q(s) is calculated with () = () ×

(2) As for the information quality F(s) of a TCS, the k-core approach is employed [ 25 ], which focus on cohesiveness and connections of nodes (terms). The -core of G is a subgraph , in which the degree of nodes is greater than or equal to . In the , for the edge D(,) linking the term and of G, its weight equals to the number of co-occurrences of two terms, and its core degree is . Weight of the edge linking two terms and the core where those two terms appear are combined to calculate the information quality F(s) as

The TCSs are then embedded using a custom

Bert+SimCSE-unsup model, and the claim representation is obtained by taking weighted average of the TCSs embeddings. The whole process is illustrated in Figure 2. similarities is investigated. The latter method analyzes the representation’s performance in the prediction of the associated IPC classes [ 7 ]. Firstly, we demonstrate the benefit of TCS and the weighting strategy, by comparing with: (i.) full text of claim; (ii.) the first claim; (iii.) TCS + unweighted average; (iv.) TCS + quantity weighted average; (v.) TCS + quality weighted average. One should notice the above methods share the Bert+SimCSE-unsup model for embeddings. For good measure, other baseline models include: (vi.) PatentSBERTa [ 20 ]; (vii.) Technological Signature [ 18 ]; (viii.) Doc2vec [ 28 ]; (ix.) tfidf-Mittens [ 29 ]; (x.) Mittens+WR [ 30 ]. Each IPC of a patent can be represented by a tree for it comprises a hierarchically organized taxonomy, and the IPC tree of a patent is structured by additionally inserting a root node to unify the trees of all assigned IPC codes. The dissimilarity space embedding (DSE) is adapted for IPC representation [ 26, 31 ], which transform the IPC tree into a vector space. Given a distance function d, the dissimilarity space embedding of IPC is defined as

() : → ℜ () = ((1, ), (2, ), . . . , (, )) (4) Tree edit distance (TED) is employed as distance function. It is given by the minimal cost sequence of all operations including insertion, deletion, and relabeling transforming one tree to another. Then we calculate similarity by dot product of two representation vector. Besides, the absolute value of diference between 1 and the ratio of two similarities (i.e., the similarity derived from the representation and IPC assignment), which takes the form of

4.3. Application

Avg. 0.5797 0.5929 We apply technical compound sentence (TCS) on novelty Std. 1.1598 1.3834 analysis. Innovation consists in carrying out new combiP valuZe (voanluee-sided) -02..09031965 nations. Actually, innovation is fundamentally the combination of facts, concepts, techniques, theories, goals, etc. [ 32 ]. Thus, for novelty analysis, the combinations Table 4 held by the patent are vital and the combinations should Result of Z-Test (iii.) be considered when conducting patent semantic search

TCS + weighted TCS + unweighted in novelty analysis. Patent claims define the boundary Avg. 0.5409 0.5797 for an exclusive right granted by the patent ofice, and we Std. 0.9152 1.1598 may express the same thing by saying that each patent Z value -10.6267 occupies a certain inventive space of the protecting parts P value (one-sided) 0.0000 of technologies that exclude other inventions. A TCS derived from a patent claim tree, naturally, describes a relatively separate segment of the entire space the claim between highlighting the key details and elaborating the defines, which means it contains the implicit combinafull scope. tions of an aspect or method the patent right intends to

Furthermore, we examine whether the generated vec- protect. Therefore, the relevant patents can be located tors can function as inputs for automated IPC symbol and identified by matching similar TCS. By applying TCS classification for the main section (In this case, binary embedding as the query, we are able to retrieve more of classification of section G and section H). An artificial relevant items which might be novelty-prejudicial to the neural network (ANN) is deployed [ 18 ], which takes the target patent for novelty assessment. Thus, TCS could representations as input and predicts the main section of improve the recall of patent retrieval in patent semantic the patent. Table 5 demonstrates that our method outper- search in novelty analysis. forms the baseline methods on this task, which indicates the capability of the presented method in semantic representation and proves the TCS as well as the weighting strategy efective.

5. Conclusion

A technical compound sentence (TCS) is composed of a set of claims that on the path from the root to the leaf nodes in a claim tree. The experiment’s findings demonstrate that the employment of TCS enhances the performance of patent semantic representation. This indicates the capability of the TCS in technical information organization of patents. Additionally, the balance of emphasizing the key information and elaborating the full scope is achieved by the weight of quantity and quality built on TCS, which further improves the semantic representation. For future work, we will further explore the uses of TCS in the field of patent text mining, attempting to achieve eficient processing, interpretation, and utilization of patent texts.

References

[1] S. Casola , A. Lavelli , Summarization, simplification, and generation: The case of patents, Expert Systems with Applications 205 ( 2022 ).

[2] J. Wang , W. Lu , L. HanTong, A two-level parser for patent claim parsing , Advanced Engineering Informatics 29 ( 2015 ) 431 - 439 .

[3] J. Qi , L. Lei , K. Zheng , X. Wang , Patent analytic citation-based vsm: Challenges and applications , IEEE Access 8 ( 2020 ) 17464 - 17476 . doi: 10 .1109/ ACCESS. 2020 . 2967817 .

[4] Y. Chi , H. Wang , Establish a patent risk prediction model for emerging technologies using deep learning and data augmentation , Advanced Engineering Informatics 52 ( 2022 ).

[5] C. Lee , B. Song , Y. Park , How to assess patent infringement risks: a semantic patent claim analysis using dependency relationships , Technology analysis and strategic management 25 ( 2013 ) 23 - 28 .

[6] H. Jang , S. Kim , B. Yoon , An explainable ai (xai) model for text-based patent novelty analysis , Available at SSRN ( 2023 ). doi:http://dx.doi.org/10. 2139/ssrn.4341594.

[7] L. Du , W. Liu, K. Xiao , S. Gao , Y. Han, Technical function-efect based patent multi-to-one negation game model , in: 2022 IEEE 25th International Conference on Computer Supported Cooperative Work in Design (CSCWD) , 2022 , pp. 1443 - 1448 . doi: 10 .1109/CSCWD54268. 2022 . 9776122 .

[8] S. Wittfoth , Measuring technological patent scope by semantic analysis of patent claims-an indicator for valuating patents , World Patent Information 58 ( 2019 ).

[9] A. J. H , Modeling patent clarity, Research Policy 51 ( 2022 ).

[10] E. Novelli , An examination of the antecedents and implications of patent scope , Research Policy 44 ( 2015 ) 493 - 507 .

[11] D. H. Milanez , L. I. L . de Faria, R. M. do Amaral , J. A. R. Gregolin , Claim-based patent indicators: A novel approach to analyze patent content and monitor technological advances , World Patent Information 50 ( 2017 ) 64 - 72 .

[12] J. Lee , J. Hsiang , Patent classification by nfie-tuning bert language model , World Patent Information 61 ( 2020 ).

[13] S. Huang , H. Ke , W. Yang , Structure clustering for chinese patent documents , Expert Systems with Applications 34 ( 2008 ) 2290 - 2297 .

[14] Z. Qiu , Z. Wang , Construction and application of patent technical element dependency network , IEEE Transactions on Engineering Management ( 2022 ) 1 - 15 . doi: 10 .1109/TEM. 2022 . 3227175 .

[15] S. Yun , W. Cho , C. Kim , S. Lee , Technological trend mining: identifying new technology opportunities using patent semantic analysis , Information Processing and Management 59 ( 2022 ).

[16] Z. Qiu , Z. Wang , What is your next invention?-a framework of mining technological development rules and assisting in designing new technologies based on bert as well as patent citations , Computers in Industry 145 ( 2023 ).

[17] deGrazia Charles AW , J. P. Frumkin , N. A. Pairolero , Embracing invention similarity for the measurement of vertically overlapping claims , Economics of Innovation and New Technology 29 ( 2020 ) 113 - 146 .

[18] D. S. Hain , R. Jurowetzki , T. Buchmann , P. Wolf , A text-embedding-based approach to measuring patent-to-patent technological similarity , Technological Forecasting and Social Change 177 ( 2022 ).

[19] L. Lei , J. Qi , K. Zheng , Patent analytics based on feature vector space model: A case of iot , 2019 . arXiv: 1904 .08100.

[20] H. Bekamiri , D. S. Hain , R. Jurowetzki , Patentsberta: A deep nlp based hybrid model for patent distance and classification using augmented sbert , 2021 . arXiv: 2103 . 11933 .

[21] T. Roh , Y. Jeong , B. Yoon , Developing a methodology of structuring and layering technological information in patent documents through natural language processing , Sustainability 11 ( 2017 ).

[22] Y. T. Demey , D. Golzio , Search strategies at the european patent ofice , World Patent Information 63 ( 2020 ).

[23] L. FuRen , C. K. Chen , L. SzuYin, A hybrid patent prior art retrieval approach using claim structure and description, 2014 , pp. 231 - 248 .

[24] J. Rossi , M. Wirth , E. Kanoulas, Query generation for patent retrieval with keyword extraction based on syntactic features , 2019 . arXiv: 1906 .07591.

[25] H. Mirisaee , E. Gaussier , C. Lagnier , A. Guerraz , Terminology-based text embedding for computing document similarities on technical content , 2019 . arXiv: 1906 . 01874 .

[26] K. Riesen , H. Bunke , Graph classification based on vector space embedding , International Journal of Pattern Recognition and Artificial Intelligence 23 ( 2009 ) 1053 - 1081 .

[27] Y. M. G. Costa , D. Bertolini , A. S. B. Jr. , G. D. C. Cavalcanti , L. E. S. Oliveira , The dissimilarity approach: a review , Artificial Intelligence Review 53 ( 2020 ) 2783 - 2808 .

[28] J. H. Lau , T. Baldwin , An empirical evaluation of doc2vec with practical insights into document embedding generation , 2016 . arXiv: 1607 . 05368 .

[29] N. Dingwall , C. Potts , Mittens: An extension of glove for learning domain-specialized representations , 2018 . arXiv: 1803 .09901.

[30] E. Kawin , Unsupervised random walk sentence embeddings: A strong but simple baseline , in: Proceedings of the Third Workshop on Representation Learning for NLP , 2018 , pp. 91 - 100 . URL: https: //aclanthology.org/W18-3012. doi: 10 .18653/v1/ W18 -3012.

[31] K. Frerich , M. Bukowski , S. Geisler , R. Farkas , On the potential of taxonomic graphs to improve applicability and performance for the classification of biomedical patents , Applied Sciences 11 ( 2023 ).

[32] H. Michael , T. Kiessling , M. Moeller , A view of entrepreneurship and innovation from the economist “for all seasons”: Joseph s. schumpeter , Journal of Management History 16 ( 2010 ) 527 - 531 .