Patent images such as technical drawings contain valuable information and are frequently used by experts to compare patents. However, current approaches to patent information retrieval are largely focused on textual information. Consequently, we review previous work on patent retrieval with a focus on illustrations in ifgures. In this paper, we report on work in progress for a novel approach for patent image retrieval that uses deep multimodal features. Scene text spotting and optical character recognition are employed to extract numerals from an image to subsequently identify references to corresponding sentences in the patent document. Furthermore, we use a neural state-of-the-art CLIP model to extract structural features from illustrations and additionally derive textual features from the related patent text using a sentence transformer model. To fuse our multimodal features for similarity search we apply re-ranking according to averaged or maximum scores. In our experiments, we compare the impact of diferent modalities on the task of similarity search for patent images. The experimental results suggest that patent image retrieval can be successfully performed using the proposed feature sets, while the best results are achieved when combining the features of both modalities.
• Information systems → Image search; Content analysis and feature selection; • Computing methodologies → Visual contentbased indexing and retrieval; Image representations.
Patent experts and researchers often encounter language and
terminology barriers when conducting searches to identify research
or patent gaps, (newly) emerging technology developments, or to
check the patentability of research results. Existing patent retrieval
methods are primarily based on textual searches and largely exclude
illustrations and the relationship between text and image. Often,
however, the innovation of a patent can be identified with the help
of an illustration, and patents with similar or related innovations
can be quickly analysed by looking at illustrations in a comparative
way. In this context, a survey with patent experts confirms the
importance of illustrations in their high informative value and the
demand for an image-based search [
In this paper, we present a novel multimodal system for semantic patent image retrieval in a query-by-example scenario. To extract visually relevant features from images, pre-trained embeddings using deep neural networks are used. Furthermore, scene text spotting is applied in order to extract numerals from the images and map them to their mentions in the patent text. Next, we derive textual features from the relevant sentences in the text utilizing sentence transformers. Finally, textual and visual features are used to index the represented illustrations. Experimental results are presented for semantic image search investigating both unimodal and multimodal feature sets.
The rest of the paper is organized as follows. We review related work in Section 2. Section 3 introduces the proposed approach for multimodal patent image search. We provide an experimental evaluation of the proposed solution in Section 4 and conclude the paper with a short discussion of results in Section 5. 2
Previous approaches to patent information retrieval have been
largely limited to textual information [
The more general problem of searching in image databases
(image retrieval) has been intensively researched in the last decades.
Simpler methods for search in image databases are usually based on
so-called low-level features, which technical descriptions of shape,
color, or texture. However, results based on such features very often
do not meet the search needs of users, which are mostly of a content
or semantic nature ("semantic gap") [21]. In recent years,
significant progress has been made to automatically recognize content in
images (denoted as object recognition or visual concept detection)
[22], especially through deep learning approaches [
An important aspect of the presented approach is the similarity search that follows feature extraction. Current similarity search approaches learn compact codes to replace images [18, 27, 28]. The compact codes usually compress high-dimensional features of a Convolutional Neural Network (CNN) trained on specific datasets suitable for the given task. However, these methods are not optimized for the technical and schematic illustrations in patents, so there is a need for research and development in this area.
So far, there are relatively few specific approaches for searching
visual information in patents [29]. An example is the Patmedia
method for similarity search [25], extensions of this [20, 23, 24], or
other approaches for concept-based graphical search [
We now discuss the proposed system that incorporates multimodal patent features to establish a similarity search based on illustrations. Figure 1 illustrates the individual steps. First, we extract visual and textual features (Section 3.1, 3.2) from the patent images. Then, based on each modality an index of corresponding image feature vectors is built (Section 3.3). Finally, the most similar results to a query image can be retrieved by re-ranking results based on both indexes. 3.1
Patent images are a special category of images that have sketchlike characteristics. They usually consist of technical drawings, diagrams, or graphs and are mostly black and white. While smaller details can often be of great relevance for interpretation, they often also contain redundant patterns. To represent these kind of images, features are extracted using deep neural network. We use the Contrastive Language-Image Pre-training (CLIP) [16] model that was trained on a multimodal dataset of 400 million image-text-pairs collected from the internet. The CLIP model is aimed at learning visual concepts from natural language supervision and is primarily designed for flexible zero-shot computer vision classification on arbitrary image datasets by providing simple textual image descriptions. This powerful approach has improved the state of the art on several benchmark datasets including ImageNet Sketch [26], which contains sketch images with characteristics similar to patent images. This motivates us to utilize CLIP embeddings for the task of patent image similarity search. In particular, we use the pre-trained vision transformer (ViT-B/32) to extract visual features and embed the patent images. 3.2
Patent figures usually contain image text, particularly numbers that can be used to link illustrated concepts to a description in the patent document. To use these textual descriptions, we first apply scene text spotting methods (Section 3.2.1). After relevant sentences have been identified, they are embedded using sentence transformers (Section 3.2.2). 3.2.1 Image-Text Relations using OCR. Optical Character Recognition (OCR) aims to recognize characters in images. Recently, scene text recognition methods based on neural networks have emerged. We use a two-step approach in which we first detect text blocks and then recognize the text they contain. For scene text detection, the CRAFT (Character Region Awareness For Text detection) [2] model for character-level text detection is applied. Subsequently, a four-stage deep scene text recognition (STR) framework [1] is employed to extract the text. Although these methods were trained for recognizing text in real-world scenes, they prove to be very accurate on patent images, for which text detection and recognition is generally easier than for scene text. Once the image text is extracted, we keep the numbers and prune irrelevant text. The numbers are then used to identify the relevant sentences in the XML file of the corresponding patent document. For this purpose, we tokenize the text, search for the numbers and keep all matching sentences that provide a description for the illustrated concepts. Exemplary text mappings resulting from the scene text recognition can be seen in Figure 2. 3.2.2 Sentence Transformers. Sentence transformer neural networks were recently introduced and can be used to compute dense vector representations for sentences. We use a RoBERTa [12] model that was pre-trained to produce semantically meaningful sentence embeddings (accordingly to Sentence-BERT[17]) and optimized for semantic textual similarity (STS) in the English language. We embed all the sentences found in the previous image-text mapping step. Finally, an average vector over all related sentences is created to represent an patent image. 3.3
Based on the extracted feature representations, indexes are built using the FAISS library[7]. An index is based on product quantization [6] and allows for the eficient comparisons between query vectors and stored vectors based on cosine similarity and returns nearest neighbors. We built separate indexes for both the image and textual feature modalities based on a dataset comprised of 30, 379 patent images. Subsequently, the nearest neighbors of a query image can be retrieved by similarity search based a) on the stored visual features, b) on the stored textual features, or c) on the basis of a combination of ranking results of both indexes. For the last option we explore two diferent re-ranking approaches. The first one is based on averaging the resulting similarity scores of each modality, whereas in the second strategy the final ranking is based on reordering according to maximum scores. 4
In this section, the patent image retrieval approaches are evaluated according to the experimental setup in Section 4.2) and based on a predefined patent collection (Section 4.1). We discuss outcomes of the experiments in Section 4.3. 4.1
We conduct our retrieval experiments on a patent collection from the European Patent Ofice (EPO) focusing on the exemplary fields of autonomous driving and wind power. To this end, we download patents from the time period 2007 to 2020 and ensure that each patent contains an XML file to parse the structured text and image information. After excluding formulas, our final patent collection comprises 2, 858 patent documents with a total of 30, 379 figures of technical drawings, diagrams and graphs. Analogously, another 3, 770 images from 300 patent documents are reserved as test data. 4.2
The performance of our system is evaluated using the average precision (AP) score which is the most commonly used quality measure for retrieval approaches. The AP score is calculated from a list of ranked documents as follows:
AP = Õ (1)
( − −1) where and are the precision and recall at the th threshold. In general, AP is the average of the precision scores at each relevant document. To evaluate the overall performance, the mean AP (mAP) score is calculated by taking the mean value of the AP scores across diferent queries. To verify the performance of our system, we randomly selected 20 query images along with their descriptions (described in Section 3.2.1) from the test data and evaluated
mAP results up to rank 50 for randomly chosen queries. Re-ranking (avg) denotes the averaging of the different modalities’ scores. Re-ranking (max) denotes the reordering according to maximum scores.
Textual Features
Visual Features 4.3
The results of our experiments are shown in Table 1. Using only visual features for image retrieval yields a slightly higher mAP score of 0.696 compared to using textual features. The combination of both modalities yields the highest mAP score of 0.715 when scores of the textual and visual similarity search are averaged. Reordering the similarity values according to the maximum scores for both feature sets had a smaller efect on the similarity search performance. The results suggest that combining both modalities can help increase the quality of retrieval results. In general, results based on visual features were easier to annotate since the visual embeddings retrieve mostly visually similar results. It should also be noted that results based on textual features were harder to inspect and thus annotated with the additional help of the sentences representing the retrieved image. In general, it was observed that textual features retrieved semantically relevant images. Thus, the combination of both feature representations presents a good mixture of both visually and semantically related patent images. 5
The discussion of related work for patent image retrieval revealed that existing work is either outdated or insuficient when it comes to exploiting the multimodal information that patents provide. In this paper, we have presented a framework that exploits multimodal features to enable semantic patent image search. Image-text relations are identified through scene text spotting and OCR yielding a mapping of in-figure numbers to the corresponding text. This allowed us to embed relevant text passages in feature vector representations. Additionally, we successfully embedded the shape and topological information in images using powerful deep neural networks. We exploit both textual and image features in order to facilitate semantic similarity for patent images. Experimental results demonstrated the feasibility of the approach, while suggesting that the combination of both modalities is beneficial.
In the future, we plan to exploit further information in images such as non-numeric image text. Moreover, we plan to incorporate multimodal information in an end-to-end network and have a joint framework to conduct patent search. Thereby, we intent to fuse features by exploiting multimodal machine learning architectures.
We would like to sincerely thank the reviewers for their valuable and comprehensive comments. This work is financially supported by the Federal Ministry of Education and Research (BMBF, Bundesministerium für Bildung und Forschung, project reference 01IO2004A).