Finding relevant prior art is crucial when deciding whether to file a new patent application or invalidate an existing patent. However, searching for prior art is challenging due to the large number of patent documents and the need for nuanced comparisons to determine novelty. An accurate search engine is therefore invaluable for speeding up the process. We present a Graph Transformer-based dense retrieval method for patent searching where each invention is represented by a graph describing its features and their relationships. Our model processes these invention graphs and is trained using prior art citations from patent ofice examiners as relevance signals. Using graphs as input significantly improves the computational eficiency of processing long documents, while leveraging examiner citations allows the model to learn domain-specific similarities beyond simple text-based matching. The result is a search engine that emulates how professional patent examiners identify relevant documents. We compare our approach against publicly available text embedding models and show substantial improvements in both prior art retrieval quality and computational eficiency.
Companies and individuals can protect their intellectual property by filing a patent application for their inventions. When granted, a patent provides its holder exclusive rights to the invention for a limited time. In exchange, the invention is publicly disclosed. The patent application process can be quite costly and time-consuming, so applicants typically seek a high certainty of it eventually being granted before initiating the application process. Since patents are granted only for novel inventions, conducting a prior art search is important to avoid unnecessary costs and delays. Finding relevant prior art is challenging due to the large number of existing patent documents and the detailed analysis required to distinguish novelty-destroying prior art from merely similar inventions. An efective patent search engine will both speed up the process and improve the results.
In this work, we present an approach for patent searching using graph representations of inventions as input to a Graph Transformer network. The graph of a patent document describes the core of the invention disclosed in that document, namely the features and the relationships between them. This representation condenses the original document, significantly reducing the computational resources required to process the document. The graphs are used as input to a Graph Transformer model trained on patent ofice examiner citations created during the application process for a patent. This enables the model to capture relationships between similar inventions, even when described using diferent terms. Furthermore, examiner citations help the model learn domain-specific terminology across diferent technical domains. The result is a patent search engine that eficiently retrieves relevant novelty-destroying prior art. 6th Workshop on Patent Text Mining and Semantic Technologies (PatentSemTech) 2025 $ krzysztof@iprally.com (K. Daniell); igor@iprally.com (I. Buzhinsky); sebastian@iprally.com (S. Björkqvist) 0009-0006-5959-1804 (K. Daniell); 0000-0003-3713-6051 (I. Buzhinsky); 0009-0006-9039-8623 (S. Björkqvist) © 2025 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). a motor; an auger driven by the motor to rotate; a handle device for a user to operate; an auger housing for containing the auger; and a frame for connecting the handle device and the auger housing; wherein the auger housing is made of at least two different materials.
Traditionally, patent searching has relied on Boolean search, where text-based matching is done using
Boolean operators like AND, OR, and NOT [
Machine learning methods have been explored for patent searching to address the limitations of Boolean searching. Traditional approaches including TF-IDF [9, 10] and word embeddings [11] have been applied. More recently, deep learning models such as BERT [12, 13, 14] and GPT-2 [15] have also been explored. In addition, several commercial machine learning-based patent search tools are available [16, 17, 18].
In our previous work [19], we describe a graph-based patent search engine where a Tree-LSTM model is trained for dense retrieval using patent ofice examiner citations as relevance signals. This paper extends that work by replacing the Tree-LSTM model with a Graph Transformer and improving the training procedure, as detailed in Section 3.
Graph Transformers adapt the Transformer architecture to graph-structured data [20]. Early models, such as Graphormer [21] and SAN [22], used full attention over all node pairs, which becomes computationally prohibitive as graph size grows—a critical concern for large patent corpora. Sparse Graph Transformers [23] address this by restricting attention to actual edges or small neighborhoods, drastically reducing overhead while preserving power for long-range dependencies. This is useful for patent searching where long documents revolving around a limited set of distinct features or relationships are typical. Mapping these concepts to nodes, with edges only where genuine conceptual links exist, yields a much sparser graph than a fully connected alternative.
A recent approach in language modeling is using concept-level representations instead of token-based architectures. Large Concept Models [24] can be used to embed entire sentences—or “concepts”—into a high-dimensional, language-agnostic space rather than embedding every token separately. This perspective parallels our approach, where each key textual segment (e.g., a feature of the invention) is treated as a “concept node.” We reduce computational overhead by applying sparse attention to these conceptual links yet still preserve the broad relationships essential for performing patent searches.
Our patent search engine consists of two main parts: We first convert each patent document into a graph that describes the invention, as described in Section 3.1. We then use a Graph Transformer model, described in Section 3.2, to embed the graph into a vector space for dense retrieval. The Graph Transformer model is trained for patent searching using patent examiner citations as relevance signals, as described in Sections 3.3 and 3.4.
The main diferences to our previous work [ 19] are replacing the Tree-LSTM model with a Graph Transformer and enhancements to the training procedure.
We convert each patent document into a graph to capture its core features and relationships while avoiding the overhead of full-text processing. An example of a graph is seen in Figure 1. Each node corresponds to a key feature of the invention (e.g., snowthrower or motor) or a text snippet indicating a relationship (e.g., frame for connecting handle device and auger housing). These relationships include hierarchical ones—such as part-of (meronym) or example-of (hyponym)—and functional ones, linking specific features to describe how they interoperate. By focusing on the essential technical structure, the graph becomes much smaller than the raw text while still conveying the essential features of the invention. This method aligns with how professional examiners perform novelty evaluations: they isolate core features and examine how they interrelate.
The details of how the graphs are created are described in our previous work [19]. In short, we first detect the features of the invention by doing a linguistic analysis with a natural language processing model. To find the relationships between the features, we use a set of hand-crafted rules designed to mirror the way patent professionals identify core inventive concepts and their interconnections from patent text. These rules detect terms describing relationships (e.g., comprising, connecting, containing) and use the output from the linguistic analysis phase to match the relationships with the correct features.
For each patent document, we create three diferent graphs: One containing only the first independent claim (later referred to as first claim graph ), another containing all the claims (all claims graph), and one containing both the claims and the full description of the document (description graph).
Our approach embeds each invention graph into a vector space for dense retrieval. Figure 2 illustrates four key steps, which we describe in detail below.
Each token sequence (e.g. sentence or phrase) is tokenized using a BPE tokenizer [25] trained on patent documents. We pre-train the token embeddings using FastText [26] to improve the convergence speed of the model training. For each token sequence, we apply a Simple Word-Embedding-based Model (SWEM) [27], which computes mean and max pooled embeddings combined with a linear projection. This approach captures contextual information with a lower overhead than LSTMs or CNNs.
Next, we feed the node embeddings into a Graph Transformer consisting of several layers that operate
on the invention graph. We use Query-Key normalization [28] to stabilize training and reduce gradient
variance. We also adopt a Pre-LayerNorm [
Token Embeddings
Node Embeddings
Node Embeddings
Graph Embedding Text Sequences
Graph
Structure BPE Tokenizer & Embedding Layer
SWEM Graph Transformer
Pooling
Mixture of Experts
Final Graph Embedding 3.2.3. Pooling After the Graph Transformer layers refine each node representation, we assign each node a learned importance weight. We then compute a graph embedding using a weighted sum of all node embeddings, emphasizing the nodes most relevant to downstream tasks.
Lastly, we use a densely gated Mixture of Experts (MoE) [
The relevance signals used for training our model are citations extracted from patent examiner reports of patent applications. This approach ensures high-quality, expert-curated training data, as examiner citations highlight legally and technically relevant prior art—what we wish our search engine to place on top of search results.
We use the following citation categories: novelty-destroying prior art (labeled X by the EPO [
Each citation is represented by a pair of graphs, where the citing graph is typically the first claim graph, and the cited graph is the description graph. At training time, we use the following data augmentations and regularization techniques: 1. Trivial citations: We create artificial citations from the first claim graph and the description graph of the same patent. We observe that having such citations stabilizes the training. 2. Graph type augmentation: For each sample, with a predefined probability (0.4 in our experiments), we randomly replace either the citing graph or the cited graph (but not both) with the all claims Our, base stage Our, dimensionality reduction stage Tree-LSTM, base stage PaECTER1 Stella2 KaLM3 GTE-ModernBert4
N/A N/A
N/A 4×512 3×2048 1×4096 1×4096 Dense retrieval models trained to embed patent documents
Dense retrieval models trained to embed text in general
Non-intersection between training, validation, and test sets is achieved on the document level, and no citations cross the diferent sets. When excluding a document from a set, we exclude all the documents of the same patent family (i.e., other applications corresponding to the same invention).
We train our model using the PyTorch [
The training has two stages: A base stage with an output dimension of 2048, and a dimensionality reduction stage reducing the output dimension to 150 by fine-tuning the base stage and adding the MoE layer described in Section 3.2.4. While the base stage achieves higher recall, the dimensionality reduction stage strikes a better balance in terms of performance and vector storage cost.
The model is evaluated thrice per epoch. The training of each stage stops when the top-3 X citation recall (Recall@3, see Section 4.1) does not improve for three subsequent evaluation runs. When trained on eight L4 GPUs, the training process for both stages combined required approximately 185k updates (12 epochs) and took about 4.6 days. 1mpi-inno-comp/paecter 2NovaSearch/stella_en_400M_v5 3HIT-TMG/KaLM-embedding-multilingual-mini-v1 4Alibaba-NLP/gte-modernbert-base
To evaluate the performance of our model in patent retrieval, we measure its efectiveness in retrieving novelty-destroying (X) citations of patent applications, as described in Section 4.1. The query is the ifrst independent claim of the application, while we use the full text (all claims and description) for the search candidate documents. For our Graph Transformer model, we use the graphs corresponding to the query and search candidate documents as input, and the other approaches use the original texts. Any non-English documents are machine translated into English before being processed.
We compare our model with four text embedding models (Section 4.2), our previous Tree-LSTM-based approach (Section 4.3) and Okapi BM25 (Section 4.4).
To perform the evaluation, we use a test set containing about 161,000 search candidate documents
and around 96,000 queries that cite one of the candidate documents as an X citation. We populate the
search space by embedding each search candidate into a single vector, i.e., we do document retrieval,
not passage retrieval. For each query we then perform a nearest neighbor search among all the search
candidate documents and measure how often, on average, the document cited as X appears in the top
three results (Recall@3). As an auxiliary metric, we use nDCG [
Based on MTEB leaderboard [
We tuned the sequence length of these models to achieve the best Recall@3 on the validation set and then evaluated the models on the test set. To preserve more text, the two models that performed best on our evaluation sets (PaECTER and Stella) were applied on several chunks of the input text and averaged the embeddings.
PaECTER was trained on patent titles concatenated with abstracts as both queries and search candidates. However, we found that using the text of first claims as queries does not afect model performance, while using the full text (truncated to fit the 512 token window) for candidates even improves it. Thus, the only diference in inputs to PaECTER compared to other text embedding models is prepending input texts with patent titles.
Table 1 shows the chosen parameters of all models (resultant sequence lengths and the maximum number of input chunks, which is indicated by a digit before the “× ” symbol), model sizes and the obtained metric values. The results show that our Graph Transformer-based approach outperforms the other models on novelty-destroying patent document retrieval. The recall of PaECTER is about 31% lower than our base stage model, while PaECTER performs about as well as Stella and better than the other text embedding models.
We also compare our approach with our previous approach based on Tree-LSTM [19], which was re-trained using the same training data used to train the Graph Transformer. Table 1 shows that its recall is 22% worse than our Graph Transformer base stage model, while outperforming PaECTER and all other evaluated text embedding models.
BM25 [
Our comparison in Section 4.2 is limited since, apart from PaECTER, the other models were not trained specifically for patent retrieval. On the other hand, we found that using a large batch size is crucial for high recall when using online hard negative mining. If we would fine-tune text embedding models we couldn’t use nearly as large batches as with our approach. With a sequence length of 512 and the GPU setup used for training our model, we could fit around 45-65 positives in a batch (depending on the model), which is more than 13 times less than we achieve with our Graph Transformer approach. This would also make the training process significantly slower and could impact the results due to the truncated texts.
We also note that this evaluation focuses on first-stage retrieval eficiency and efectiveness; comparison with computationally intensive re-ranking models like cross-encoders was considered beyond the scope of this specific study but represents a potential area for future investigation.
In this paper, we presented an approach for patent searching that utilizes a Graph Transformer model as a dense retriever. We compared the performance of our approach to existing text embedding models and showed that our approach achieves a significantly higher recall on novelty-destroying citation retrieval. Additionally, we demonstrated that our approach is more computationally eficient than existing text-based Transformer models.
During the preparation of this work, the authors used Grammarly in order to: Grammar and spelling check. And further ChatGPT-4o in order to: Paraphrase and reword, as well as improve writing style. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. [8] A. Ali, A. Tufail, L. C. De Silva, P. E. Abas, Innovating patent retrieval: a comprehensive review of techniques, trends, and challenges in prior art searches, Applied System Innovation 7 (2024) 91. [9] B. Herbert, G. Szarvas, I. Gurevych, Prior art search using international patent classification codes and all-claims-queries, in: C. Peters, G. M. Di Nunzio, M. Kurimo, T. Mandl, D. Mostefa, A. Peñas, G. Roda (Eds.), Multilingual Information Access Evaluation I. Text Retrieval Experiments, Springer Berlin Heidelberg, Berlin, Heidelberg, 2010, pp. 452–459. [10] E. D’hondt, S. Verberne, CLEF-IP 2010: Prior art retrieval using the diferent sections in patent documents (2010). [11] L. Helmers, F. Horn, F. Biegler, T. Oppermann, K.-R. Müller, Automating the search for a patent’s prior art with a full text similarity search, PloS one 14 (2019) e0212103. [12] M. Ghosh, S. Erhardt, M. E. Rose, E. Buunk, D. Harhof, PaECTER: Patent-level representation learning using citation-informed transformers, 2024. URL: https://arxiv.org/abs/2402.19411. arXiv:2402.19411. [13] U. U. Acikalin, M. Kutlu, Patent search using triplet networks based fine-tuned SciBERT, 2022.
URL: https://arxiv.org/abs/2207.11497. arXiv:2207.11497. [14] M. Freunek, A. Bodmer, Transformer-based patent novelty search by training claims to their own description, Applied Economics and Finance 8 (2021) 37. doi:10.11114/aef.v8i5.5182. [15] J.-S. Lee, J. Hsiang, Prior art search and reranking for generated patent text, 2021. URL: https: //arxiv.org/abs/2009.09132. arXiv:2009.09132. [16] Ambercite, Ambercite, 2025. URL: https://www.ambercite.com/, [Online; accessed 22-Apr-2025]. [17] Amplified AI, Amplified, 2025. URL: https://www.amplified.ai/, [Online; accessed 22-Apr-2025]. [18] IPScreener, IPScreener, 2025. URL: https://ipscreener.com/, [Online; accessed 22-Apr-2025]. [19] S. Björkqvist, J. Kallio, Building a graph-based patent search engine, in: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023, pp. 3300–3304. [20] A. Shehzad, F. Xia, S. Abid, C. Peng, S. Yu, D. Zhang, K. Verspoor, Graph transformers: A survey, 2024. URL: https://arxiv.org/abs/2407.09777. arXiv:2407.09777. [21] C. Ying, T. Cai, S. Luo, S. Zheng, G. Ke, D. He, Y. Shen, T.-Y. Liu, Do transformers really perform bad for graph representation?, 2021. URL: https://arxiv.org/abs/2106.05234. arXiv:2106.05234. [22] D. Kreuzer, D. Beaini, W. L. Hamilton, V. Létourneau, P. Tossou, Rethinking graph transformers with spectral attention, 2021. URL: https://arxiv.org/abs/2106.03893. arXiv:2106.03893. [23] V. P. Dwivedi, X. Bresson, A generalization of transformer networks to graphs, 2021. URL: https: //arxiv.org/abs/2012.09699. arXiv:2012.09699. [24] L. team, L. Barrault, P.-A. Duquenne, M. Elbayad, A. Kozhevnikov, B. Alastruey, P. Andrews, M. Coria, G. Couairon, M. R. Costa-jussà, D. Dale, H. Elsahar, K. Hefernan, J. M. Janeiro, T. Tran, C. Ropers, E. Sánchez, R. S. Roman, A. Mourachko, S. Saleem, H. Schwenk, Large concept models: Language modeling in a sentence representation space, 2024. URL: https://arxiv.org/abs/2412.08821. arXiv:2412.08821. [25] R. Sennrich, B. Haddow, A. Birch, Neural machine translation of rare words with subword units, 2016. URL: https://arxiv.org/abs/1508.07909. arXiv:1508.07909. [26] P. Bojanowski, E. Grave, A. Joulin, T. Mikolov, Enriching word vectors with subword information, Transactions of the Association for Computational Linguistics 5 (2017) 135–146. URL: https: //aclanthology.org/Q17-1010/. doi:10.1162/tacl_a_00051. [27] D. Shen, G. Wang, W. Wang, M. R. Min, Q. Su, Y. Zhang, C. Li, R. Henao, L. Carin, Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms, in: I. Gurevych, Y. Miyao (Eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Melbourne, Australia, 2018, pp. 440–450. URL: https://aclanthology.org/P18-1041/. doi:10.18653/ v1/P18-1041. [28] A. Henry, P. R. Dachapally, S. S. Pawar, Y. Chen, Query-key normalization for transformers, in: T. Cohn, Y. He, Y. Liu (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, Association for Computational Linguistics, Online, 2020, pp. 4246–4253. URL: https: