Patent experts and researchers often encounter language and terminology barriers when conducting searches to identify research or patent gaps, (newly) emerging technology developments, or to check the patentability of research results. Existing patent retrieval methods are primarily based on textual searches and largely exclude illustrations and the relationship between text and image. Often, however, the innovation of a patent can be identified with the help of an illustration, and patents with similar or related innovations can be quickly analysed by looking at illustrations in a comparative way. In this context, a survey with patent experts confirms the importance of illustrations in their high informative value and the demand for an image-based search [8]. Moreover, with the continuous refinement of already patented research, the terminology used changes [3], making it more difficult to find corresponding patents. This problem is exacerbated when cross-linguistic searches are conducted. Therefore, illustrations provide an alternative way to enable the identification of relevant results in patents, regardless of language and terminology. The use of illustrations is also advantageous for domain and patent class independent searches. In this way, intellectual property (IP) rights can be evaluated for further application domains, which is only possible to a limited extent with a purely textual search. This is especially relevant for basic and technical patents, whose scope of application is often not clear at the beginning of the creation of an exploitation strategy.
In this paper, we present a novel multimodal system for semantic patent image retrieval in a query-by-example scenario. To extract visually relevant features from images, pre-trained embeddings using deep neural networks are used. Furthermore, scene text spotting is applied in order to extract numerals from the images and map them to their mentions in the patent text. Next, we derive textual features from the relevant sentences in the text utilizing sentence transformers. Finally, textual and visual features are used to index the represented illustrations. Experimental results are presented for semantic image search investigating both unimodal and multimodal feature sets.
The rest of the paper is organized as follows. We review related work in Section 2. Section 3 introduces the proposed approach for multimodal patent image search. We provide an experimental evaluation of the proposed solution in Section 4 and conclude the paper with a short discussion of results in Section 5.
Previous approaches to patent information retrieval have been largely limited to textual information [19]. However, terminology in patents changes continuously due to the constant evolution of the presented content and is inconsistent for this reason [3]. Often, innovative terminology is "invented" along with the actual invention. One result of this evolution is that search results are often incomplete and do not display all relevant patents. The (additional) evaluation of non-textual information in the form of illustrations, such as technical drawings, graphs and diagrams, can facilitate and significantly improve the search for similar or relevant patents. In addition, references to the relevant text passages are often given in numerical form in these illustrations, so that automatic recognition of these image-text references can also significantly improve the quality of the (multimodal) search results. The more general problem of searching in image databases (image retrieval) has been intensively researched in the last decades. Simpler methods for search in image databases are usually based on so-called low-level features, which technical descriptions of shape, color, or texture. However, results based on such features very often do not meet the search needs of users, which are mostly of a content or semantic nature ("semantic gap") [21]. In recent years, significant progress has been made to automatically recognize content in images (denoted as object recognition or visual concept detection) [22], especially through deep learning approaches [5,10,30]. In this way, search queries of a content-related nature can be more accurately answered.
An important aspect of the presented approach is the similarity search that follows feature extraction. Current similarity search approaches learn compact codes to replace images [18,27,28]. The compact codes usually compress high-dimensional features of a Convolutional Neural Network (CNN) trained on specific datasets suitable for the given task. However, these methods are not optimized for the technical and schematic illustrations in patents, so there is a need for research and development in this area.
So far, there are relatively few specific approaches for searching visual information in patents [29]. An example is the Patmedia method for similarity search [25], extensions of this [20,23,24], or other approaches for concept-based graphical search [11,13]. These methods generally extract textual and visual low-level features from patent images and train detectors that identify a limited number of predefined concepts. Experiments of these works show that the combination of visual and textual features works best for the task of concept detection. More recent approaches [9,14] establish the references of figures and related text passages using an automatic detection of the corresponding numerical referencing in the figures. Another approach [4] uses SIFT-like local histograms as features and represents the images in patents using Fisher vectors. In the experiments based on the 2011 CLEF-IP evaluation [15], the best retrieval results were achieved by late fusion of textual and nontextual results. Bhatti and Hanbury [3] provide an overview of further research regarding specific figure types (photo, flow chart, technical drawings, diagrams, graphs) that may also be relevant for patent retrieval. However, to date no integrated patent retrieval system exploiting multimodal search does exist. The representation and quality of the images in patents as well as their schematic and sketchy character require specific approaches or the recognition of special objects that are particularly relevant in patents.
We now discuss the proposed system that incorporates multimodal patent features to establish a similarity search based on illustrations. Figure 1 illustrates the individual steps. First, we extract visual and textual features (Section 3.1, 3.2) from the patent images. Then, based on each modality an index of corresponding image feature vectors is built (Section 3.3). Finally, the most similar results to a query image can be retrieved by re-ranking results based on both indexes.
Patent images are a special category of images that have sketchlike characteristics. They usually consist of technical drawings, diagrams, or graphs and are mostly black and white. While smaller details can often be of great relevance for interpretation, they often also contain redundant patterns. To represent these kind of images, features are extracted using deep neural network. We use the Contrastive Language-Image Pre-training (CLIP) [16] model that was trained on a multimodal dataset of 400 million image-text-pairs collected from the internet. The CLIP model is aimed at learning visual concepts from natural language supervision and is primarily designed for flexible zero-shot computer vision classification on arbitrary image datasets by providing simple textual image descriptions. This powerful approach has improved the state of the art on several benchmark datasets including ImageNet Sketch [26], which contains sketch images with characteristics similar to patent images. This motivates us to utilize CLIP embeddings for the task of patent image similarity search. In particular, we use the pre-trained vision transformer (ViT-B/32) to extract visual features and embed the patent images.
Patent figures usually contain image text, particularly numbers that can be used to link illustrated concepts to a description in the patent document. To use these textual descriptions, we first apply scene text spotting methods (Section 3.2.1). After relevant sentences have been identified, they are embedded using sentence transformers (Section 3.2.2).
Image-Text Relations using OCR. Optical Character Recognition (OCR) aims to recognize characters in images. Recently, scene text recognition methods based on neural networks have emerged. We use a two-step approach in which we first detect text blocks and then recognize the text they contain. For scene text detection, the CRAFT (Character Region Awareness For Text detection) [2] model for character-level text detection is applied. Subsequently, a four-stage deep scene text recognition (STR) framework [1] is employed to extract the text. Although these methods were trained for recognizing text in real-world scenes, they prove to be very accurate on patent images, for which text detection and recognition is generally easier than for scene text. Once the image text is extracted, we keep the numbers and prune irrelevant text. The numbers are then used to identify the relevant sentences in the XML file of the corresponding patent document. For this purpose, we tokenize the text, search for the numbers and keep all matching sentences that provide a description for the illustrated concepts. Exemplary text mappings resulting from the scene text recognition can be seen in Figure 2.
Sentence transformer neural networks were recently introduced and can be used to compute dense vector representations for sentences. We use a RoBERTa [12] model that was pre-trained to produce semantically meaningful sentence embeddings (accordingly to Sentence-BERT [17]) and optimized for semantic textual similarity (STS) in the English language. We embed all the sentences found in the previous image-text mapping step. Finally, an average vector over all related sentences is created to represent an patent image.
Based on the extracted feature representations, indexes are built using the FAISS library [7]. An index is based on product quantization [6] and allows for the efficient comparisons between query vectors and stored vectors based on cosine similarity and returns nearest neighbors. We built separate indexes for both the image and textual feature modalities based on a dataset comprised of 30, 379 patent images. Subsequently, the nearest neighbors of a query image can be retrieved by similarity search based a) on the stored visual features, b) on the stored textual features, or c) on the basis of a combination of ranking results of both indexes. For the last option we explore two different re-ranking approaches. The first one is based on averaging the resulting similarity scores of each modality, whereas in the second strategy the final ranking is based on reordering according to maximum scores.
In this section, the patent image retrieval approaches are evaluated according to the experimental setup in Section 4.2) and based on a predefined patent collection (Section 4.1). We discuss outcomes of the experiments in Section 4.3.
We conduct our retrieval experiments on a patent collection from the European Patent Office (EPO) focusing on the exemplary fields of autonomous driving and wind power. To this end, we download patents from the time period 2007 to 2020 and ensure that each patent contains an XML file to parse the structured text and image information. After excluding formulas, our final patent collection comprises 2, 858 patent documents with a total of 30, 379 figures of technical drawings, diagrams and graphs. Analogously, another 3, 770 images from 300 patent documents are reserved as test data.
The performance of our system is evaluated using the average precision (AP) score which is the most commonly used quality measure for retrieval approaches. The AP score is calculated from a list of ranked documents as follows:
where 𝑅 𝑛 and 𝑃 𝑛 are the precision and recall at the 𝑛 th threshold.
In general, AP is the average of the precision scores at each relevant document. To evaluate the overall performance, the mean AP (mAP) score is calculated by taking the mean value of the AP scores across different queries. To verify the performance of our system, we randomly selected 20 query images along with their descriptions (described in Section 3.2.1) from the test data and evaluated AP scores for the textual retrieval, visual retrieval and combined retrieval based on re-ranking. To evaluate the relevance of an retrieval results we rely on the annotator assessment (done by one of the authors). Using the additional figure descriptions assists in evaluating the relevance of retrieval results to the query image. The ranked retrieval lists are evaluated for the top-50 ranks using the AP score (AP@50).
The results of our experiments are shown in Table 1. Using only visual features for image retrieval yields a slightly higher mAP score of 0.696 compared to using textual features. The combination of both modalities yields the highest mAP score of 0.715 when scores of the textual and visual similarity search are averaged. Reordering the similarity values according to the maximum scores for both feature sets had a smaller effect on the similarity search performance. The results suggest that combining both modalities can help increase the quality of retrieval results. In general, results based on visual features were easier to annotate since the visual embeddings retrieve mostly visually similar results. It should also be noted that results based on textual features were harder to inspect and thus annotated with the additional help of the sentences representing the retrieved image. In general, it was observed that textual features retrieved semantically relevant images. Thus, the combination of both feature representations presents a good mixture of both visually and semantically related patent images.
The discussion of related work for patent image retrieval revealed that existing work is either outdated or insufficient when it comes to exploiting the multimodal information that patents provide. In this paper, we have presented a framework that exploits multimodal features to enable semantic patent image search. Image-text relations are identified through scene text spotting and OCR yielding a mapping of in-figure numbers to the corresponding text. This allowed us to embed relevant text passages in feature vector representations. Additionally, we successfully embedded the shape and topological information in images using powerful deep neural networks. We exploit both textual and image features in order to facilitate semantic similarity for patent images. Experimental results demonstrated the feasibility of the approach, while suggesting that the combination of both modalities is beneficial.
In the future, we plan to exploit further information in images such as non-numeric image text. Moreover, we plan to incorporate multimodal information in an end-to-end network and have a joint framework to conduct patent search. Thereby, we intent to fuse features by exploiting multimodal machine learning architectures.
We would like to sincerely thank the reviewers for their valuable and comprehensive comments. This work is financially supported by the Federal Ministry of Education and Research (BMBF, Bundesministerium für Bildung und Forschung, project reference 01IO2004A).