Most patent retrieval systems are text-based, which under utilizes the multimodal nature of patent documents. Although a few multimodal patent retrieval systems exist, they fall short in providing eficient and informative visualizations, and facilitating the interpretation of retrieval results. To address these shortcomings, this paper presents iPatent, a novel web-based multimodal patent image retrieval system. Unlike previous solutions, iPatent integrates state-of-the-art deep learning models for fine-grained unimodal, cross-modal, and multimodal patent image retrieval. Additionally, it employs both traditional machine learning techniques and modern generative methods for interactive visual exploration and insightful interpretation of retrieval results. iPatent leverages modern web technologies to provide an interactive interface that enables users to explore large patent databases eficiently and in a visually informative way. Source code and demo are publicly available at: https://service.tib. eu/ipatent/.
Patents protect intellectual property by combining detailed textual descriptions with visual
representations. However, most patent search and analysis tools primarily depend on text-based methods to
identify similar patents [
Several systems have been developed to address the unique challenges of patent image retrieval. Early
patent image retrieval systems relied on low-level visual features for content-based image retrieval (e.g.,
PatMedia [
In this paper, we present iPatent, an open-source, interactive web-based search system for multimodal patent image retrieval and analysis. The system enables semantic patent image search through imageto-image, text-to-image, and text-conditioned image-to-image retrieval approaches. Users can also search for specific sub-regions or components within a query image, supporting fine-grained exploration. Additionally, the system organizes retrieval results into visually and semantically coherent clusters, each accompanied by synthetic descriptions, enhancing exploration and interpretability to facilitate further analysis of the results.
The remainder of this paper is organized as follows. Section 2 describes the system architecture and
technical implementation details of iPatent. The multimodal image retrieval, image clustering, and
interpretation approaches are described in Section 3. Section 4 presents practical use cases of iPatent,
and the current functionalities and user interface. In Section 5, we provide quantitative evaluation of
CLIP’s performance on patent retrieval task. Section 6 summaries the paper and outlines future work.
2. System Architecture of iPatent
iPatent facilitates patent image retrieval and analysis through a web-based application system. As
illustrated in Figure 1, iPatent comprises of three primary modules: the user interface and backend
module, the index and retrieval module, and the analysis and generative module. The user interface and
backend module is implemented using Streamlit [
User Interface
d Retrieval Select top k 100 Text weight 3. Multimodal Image Retrieval iPatent operates through a sequence of steps that combine deep feature extraction, flexible and finegrained retrieval, and post-retrieval visualizations. Ofline, a large collection of patent images are processed using a deep learning model to extract high-dimensional feature representations (Section 3.1) which are stored in a vector database. The vector-based retrieval process (Section 3.2) then allows unimodal, cross-modal, and multimodal retrieval of patent images for exploration and discovery. After retrieval, the system further organizes the results through clustering, grouping similar images based on their feature representations (Section 3.3). This organization enhances result navigation and exploration, and together with synthetic descriptions (Section 3.4) enable interpretation and analysis.
Efective patent image search requires retrieving not only visually similar images but also those that
are semantically related. To achieve this, we employ CLIP [
The core image retrieval mechanism in the system is based on a k-nearest neighbor search within a
high-dimensional semantic space, enabling both visual and semantic similarity matching. Central to its
eficient and scalable performance is the use of the HNSW (Hierarchical Navigable Small World) [
iPatent implements three image retrieval approaches by leveraging the shared semantic space of CLIP: (1) image-to-image (unimodal), (2) text-to-image (cross-modal), and (3) text-conditioned image-toimage (multimodal).
The image-to-image retrieval approach allows transcending domain and language barriers during patent search. A user, for example, can upload a technical drawing of a folding chair mechanism to search for images that depict visually and structurally similar folding mechanisms, regardless of diferences in drawing style, and without requiring domain knowledge. In this approach, iPatent takes a single patent image as query, which is then encoded using CLIP, and then performs vector search to retrieve similar images.
The text-to-image retrieval approach facilitates searching for relevant images using descriptive language, which bridges the gap between textual and visual information. For example, a user can enter a descriptive textual query such as "portable solar panel with integrated battery storage" encapsulating concepts that sometimes images alone can not. In this cross-modal retrieval approach, iPatent supports text-based natural language query, which is also embedded into the same semantic space as the images using CLIP. The system, similar to image-to-image approach, performs vector search to identify images whose embeddings are most similar to the text query.
The text-conditioned image-to-image retrieval is a multimodal approach that leverages the complementary strengths of both modalities, enabling more nuanced and precise queries. For example, users can upload an image of a bicycle frame and refine their search with textual attributes or modifications such as "with built-in suspension system" to find images with additional functional or structural requirements. Here, iPatent encodes both the query image and the query text into the same semantic space using CLIP, and performs image-to-image and text-to-image retrieval. The retrieved results are then fused together using weighted averaging, i.e., late fusion. The modality weights are also configurable providing the user further fine-grained query formulation features. Grid View Cluster View
Most retrieval systems present results as an ordered list ranked by similarity scores, typically in a list or grid layout. However, such linear layouts often overlook duplicate or near-duplicate results, do not reveal relationships among retrieved items, and often fail to provide meaningful grouping or clustering similar results. Moreover, fixed grid layouts can also become overwhelming when displaying large volume of results, potentially decreasing eficiency in locating specific images of interest.
Displaying retrieved images in visually and semantically coherent cluster groups significantly reduces
information overload and enhances users’ ability to eficiently navigate and explore the results. To
achieve this, we apply the k-means [
Providing rich context for each cluster group—both in relation to its member images and the original query—greatly enhances the interpretability and navigability of retrieval results. To achieve this, each cluster is paired with a set of descriptive keywords as well as synthetic titles and summaries. The keywords consist of component terms associated with the patent images in the cluster, which are extracted directly from the corresponding patent texts. This gives users immediate insight into the technical content of each group.
Beyond keywords, each cluster is further provided with a synthetic title and a description, both
generated by an LVLM. For this purpose, we use the LLaVA-1.6 [
iPatent has a range of impactful use cases across the patent lifecycle. We discuss some uses cases of iPatent in Section 4.1, and highlight some of the user interface and features of iPatent in Section 4.2.
The multimodal image retrieval capabilities of iPatent allows for various use cases across the patent lifecycle:
Prior Art Search: iPatent’s multiple search approaches allow for quick and focused searches helping patent examiners and applicants eficiently identify existing inventions that may afect the novelty or non-obviousness of a new patent application.
Cross-Domain and Multimodal Search: The cross-modal and multimodal search in iPatent allow for searching cross-domain where users lack precise technical vocabulary or when inventions are best described visually. This feature is especially helpful in fields where technical drawings and schematics are central to the invention, such as mechanical engineering, and design domains.
Infringement Detection and Freedom-to-Operate Analysis: The multimodal search in iPatent can potentially identify a broader range of relevant patents, enhancing the recall of the search process. This allows companies and legal teams to reduce potential infringement or ensure freedom to operate reducing the risk of litigation post-publication. 4.2. Demonstrator iPatent is available at https://service.tib.eu/ipatent/. Users can perform three distinct types of image retrieval: image-to-image retrieval (Figure 1; a), text-to-image retrieval (Figure 1; b), and text-conditioned image-to-image retrieval (Figure 1; a and b). In image-to-image retrieval, users can also crop a specific region of the image, to perform fine-grained searches such as searching for similar sub-figures or components. By adjusting the dimensions of the blue square (shown in Figure 1; a1), the users can acquire a cropped query image (shown in Figure 1; a2).
For each query, the top (adjustable by the user) most relevant images are displayed in a grid view (Figure 1; c1). The layout is designed so that the most relevant image appears in the top-left corner, with images arranged from left to right across each row. Relevance decreases progressively rightward along the rows and downward through the grid, making the bottom-right image the least relevant among the displayed results.
Users are also provided with a cluster view (Figure 2; c2). Here, the retrieved images are organized into groups, each representing a cluster of visually and semantically similar images. Each cluster is displayed as a horizontally scrollable row, which reduces visual information overload by helping users focus on conceptually coherent subsets. The users also have the option to select the number of desired clusters and the number of retrieved images to cluster (Figure 1; d). To further enhance interpretability and eficient navigation, each cluster is also paired with a set of keywords, providing quick insights about the cluster. Beyond the keywords, each cluster is also enriched with synthetic title and description, which are generated using an LVLM.
For evaluation of the CLIP model [
In addition to the quantitative metrics, we present qualitative results to demonstrate the model’s capability to retrieve visually and semantically similar patent images. In Figure 3, for each query image, the top-three retrieved images exhibit strong visual similarity and consistent semantic content. For example, the query image from patent US2020340208A1 retrieves images of similar machinery with comparable structural details. Similarly, the queries of patents US2020339341A1 and US2022267022A1, yield closely related designs and visualizations. These qualitative examples highlight the efectiveness of CLIP in capturing both fine-grained visual features and relevant semantics, confirming its practical utility in multimodal patent image retrieval.
US2020340208A1 EP3670764A1 WO2020022454A1EP3670764A1 US2020339341A1 EP3738925A1 US2021205184A1WO2018051449A1
US2022267022A1 EP3640143A1 US2012207401A1US2011095912A1
In this paper, we have introduced iPatent, a novel web-based platform for patent image retrieval and analysis. Unlike existing systems, iPatent leverages state-of-the-art models for unimodal, cross-modal, and multimodal retrieval of patent images with semantic search capability. The platform also supports ifne-grained searching for sub-figures or components through as interactive query image cropping feature. For organizing search results, iPatent uses k-means clustering, while LVLMs automatically generate informative cluster titles and descriptions. Leveraging modern web technologies, iPatent provides an interactive and visually informative interface, supporting eficient exploration of large patent databases.
In future work, we aim to enhance iPatent with additional ranking and interpretation features such as LVLM-based re-ranking and explanation of ranking results. We plan to improve cluster visualizations with interactive 3D projections for more intuitive exploration. Furthermore, we plan to incorporate faceted search options such as filtering by patent class or figure type. Additionally, LVLMs may introduce biases in the content they generate. A systematic study of the synthetic content and its potential biases is planned as an import direction for future research.
This article has been funded by the Academic Research Programme of the European Patent Ofice (project "ViP@Scale: Visual and multimodal patent search at scale"). We would like to thank Wolfgang Gritz and Matthias Springstein (both TIB - Leibniz Information Centre for Science and Technology) for their feedback and help in developing iPatent.
During the preparation of this work, the author(s) used Grammarly in order to: Grammar and spelling check, Paraphrase and reword. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.