This paper addresses the automatic generation of explanatory text from flowchart images in patents. The construction of an explanatory text generator consists of four steps: (1) automatic recognition of flowchart images from patent images, (2) extraction of text strings from flowchart images, (3) creation of data for machine learning, and (4) construction of an explanatory text generator using T5. In this study, a benchmark consisting of 7,099 images was constructed to determine whether an image in a patent is a flowchart. Furthermore, an explanatory text generator was constructed from the images using 11,188 flowchart image-explanatory text pairs. The experimental results showed that a recognition accuracy of 0.9645 was achieved for flowchart images. Although high-quality explanatory text could be generated from flowchart images, some issues remain for flowcharts with complex shapes.
A procedural text is a description of a set of procedures to achieve a particular objective. Our goal is to automatically extract knowledge about a series of procedures in a wide range of fields from texts and systematize them. Here, we describe the automatic generation of explanatory text from flowchart images in patents.
In automatically generating explanatory text for the flowchart images, we focus on the abstract and selected figures of the patent. A selection figure enables us to grasp the outline of the invention quickly and accurately. The applicant usually selects a diagram from among the diagrams in the patent that they consider necessary for understanding the abstract contents. If a classifier that automatically determines whether an image in a patent is a flowchart or not is constructed and only those selected diagrams that are flowcharts are extracted, a large number of pairs of flowcharts and their explanatory texts (i.e., patent abstracts) can be generated automatically. Furthermore, using these pairs, we believe it is possible to construct a system that automatically generates explanatory text from flowchart images using machine learning.
The contributions of this paper are as follows: To determine whether an image in a patent is a configuration diagram, flowchart, or table, we constructed a benchmark consisting of 7,099 images. We used this benchmark to achieve a classification accuracy of 0.9645.
We constructed 11,188 pairs of flowchart images and their descriptions automatically.
Using these pairs, we constructed a system that automatically generates explanatory text from flowchart images through machine learning. PatentSemTech'23: 4th Workshop on Patent Text Mining and Semantic Technologies, July 27th, 2023, Taipei, Taiwan. l l l
2.1.
Services that share flowcharts, such as
myExperiment and SHIWA, have started recently,
which has led to a demand for techniques to search for
similarities between one flowchart and another
flowchart [
Chart to text refers to the task of generating natural
language sentences to describe the important
information derived from charts and tables. Zhu et al.
[
Tan and colleagues [
The construction of the generator of explanatory text consists of the following four steps: (Step 1) automatic recognition of flowchart images; (Step 2) extraction of character strings from the flowchart image; (Step 3) creation of data for machine learning; and (Step 4) construction of an explanatory text generator using T5. Each procedure is described as follows. (Step 1) Automatic recognition of flowchart images
Convolutional neural networks (CNNs) are used to recognize flowchart images in patents. Our method uses seven CNN models trained on a large image data set called “ImageNet” to construct a learning model by fine tuning, and its effectiveness is verified through the experiments described in Section 4. (Step 2) Extraction of character strings from the flowchart image
An optical character recognition function in Google Cloud Vision (https://cloud.google.com/vision) is used to extract text strings from flowcharts. An example of a flowchart image and the character recognition result are shown in Figures 1 and 2 respectively. Here, “\n” indicates a line break. 1 Figures 2 and 3 show examples of a character recognition result and a manually written explanatory text (patent abstract). In this [original] 半導体装置\n の製造方法\n 半導体層の上にフォ トレジストを塗布\n 第 1 波長の紫外線を用いて \n フォトレジストを露光した後に\n フォトレジ ストを現像することによって、\n 開口部の内側 へと迫り出した\n ネガ型レジストパターンを形 成\n 第 1 波長より短い第 2 波長の紫外線を\n ネ ガ型レジストパターンに照射することによって、 \n ネガ型レジストパターンを硬化\n(照射工程)\n ネガ型レジストパターンが形成された\n 半導体 層の上に金属膜を形成\n ネガ型レジストパター ン を 半 導 体 層 か ら 除 去 \n 完 成 \nP110\nP120\nP130\nP140\nP150 [translation] Semiconductor devices \n Manufacturing method for \n photoresist is applied over the semiconductor layer \n using ultraviolet light of the first wavelength \n After exposing the photoresist \n by developing the photoresist, \n The photoresist is developed to form a negative resist pattern that extends into the aperture. \n Negative resist pattern is formed. \n By exposing the negative resist pattern to ultraviolet rays of the second wavelength, which is shorter than the first wavelength \n by irradiating the negative resist pattern, \n curing the negative resist pattern by irradiating it with ultraviolet light of the second wavelength, which is shorter than the first wavelength. \n (Irradiation process) \n The negative resist pattern is formed \n Metal film is formed on top of the semiconductor layer \n Negative resist pattern is removed from the semiconductor layer \n Completion \n P110 \n P120 \n P130 \n P140 \n P150 Figure 2: Character Recognition Results for the Image in Figure 1. (Step 3) Creation of data for machine learning
We build an explanatory text generator by machine
learning, using pairs of character recognition results
and explanatory text from a large number of flowchart
images. In this process, we consider that data with
large differences between the character recognition
results and the manually written explanatory texts
(patent abstracts) are inappropriate as training data1;
therefore, we exclude these data. In this process, we
calculate the similarity between the character
recognition result and the explanatory text of the
flowchart image using Gestalt pattern matching [
We build an explanatory text generator by the language model T5. With respect to the flowchart image in Figure 1, the input and output of T5 are Figure 2 for input and Figure 3 for output. case, the similarity between them is so high that we use them as machine learning data. [original] 半導体装置の製造方法は、半導体層の上にフォト レジストを塗布する工程と；第１波長の紫外線を 用いてフォトレジストを露光した後にフォトレジ ストを現像することによって、開口部の内側へと 迫り出したネガ型レジストパターンを、形成する 工程と；第１波長より短い第２波長の紫外線をネ ガ型レジストパターンに照射することによって、 ネガ型レジストパターンを硬化させる照射工程 と；照射工程を行った後、ネガ型レジストパター ンの開口部から露出する半導体層の上に、ニッケ ル（Ｎｉ）から主に成る金属膜を形成する工程 と；ネガ型レジストパターンを半導体層から除去 する工程とを備える。 [translation] The method of manufacturing a semiconductor device comprises the steps of: applying a photoresist onto a semiconductor layer; forming a negative resist pattern, which is pressed inwards into an aperture, by developing the photoresist after exposing the photoresist using a first wavelength of ultraviolet light; forming a negative resist pattern by irradiating the negative resist pattern with ultraviolet light of a second wavelength that is shorter than the first wavelength; and The negative resist pattern is hardened by irradiating the negative resist pattern with ultraviolet light of a second wavelength shorter than the first wavelength; forming a metal film mainly comprising nickel (Ni) on the semiconductor layer exposed from the opening of the negative resist pattern after performing the irradiation process; and removing the negative resist pattern from the semiconductor layer. The process of removing the negative resist pattern from the semiconductor layer.
Figure 3: Explanatory Text (Patent Abstract) Correspongind to the Image in Figure 1
We performed some experiments to confirm the effectiveness of our method. 4.1. Automatic Recognition of Flowchart Images
Using 7,099 randomly selected images from the 2018 edition of the Japanese Patent Public Gazette, we manually identified whether they were flowcharts or not and obtained 1,120 flowcharts from the 7,099 cases.
As a baseline method, we used Keras, a deep learning library, to build CNN training models with three layers for Conv2D and two layers for MaxPooling2D. As a comparison method, we used seven CNN models: VGG16, VGG19, ResNet50, InceptionV3, MobileNet, DenseNet169, and DenseNet121 trained on a large image data set called ImageNet.
The seven methods and the baseline method were evaluated using Precision, Recall, and F-measure.
The experimental results are shown in Table 1. Among the compared methods, DenseNet121 was the most accurate in detecting flowcharts in terms of Precision. The results from DenseNet121 were used in the subsequent experiments.
Among the Japanese patents published from 2010 to 2019, 11,188 patents that included flowcharts and with a similarity of 0.1 by Gestalt pattern matching were used in our experiments. Of these patents, 90% were categorized as training data and the remainder as validation and evaluation data.
The following hyperparameters were used in the generation of explanatory texts by T5. l Max input length: 280 l Max target length: 256 l Train batch size: 8 l Eval batch size: 8 l Num train epochs: 6
Our method was evaluated using the following
measures:
l ROUGE-N: This is the most basic index and is a
method of taking the degree of agreement in
Ngram units. In this case, N = 1, 2 were used for
evaluation (https://github.com/pltrdy/rouge).
l ROUGE-L: Evaluates the maximum sequence that
matches the generated summary and the
manually generated summary.
l BERTScore [
The results of Recall, Precision, and F-measure for ROUGE-1, ROUGE-2, ROUGE-L, and BERTScore.
For simple geometries with no branches in the flowchart (see Figure 4), we obtained good analytical results. Figures 5 and 6 show the explanatory text and the patent summary (correct answer) generated by our method, respectively. [original] 測定治具を取り付ける取付工程と、複数のガス供 給箇所にて取付工程及び湿度測定工程を実行する 湿度測定工程と、複数のガス供給箇所での湿度測 定工程での測定結果に基づいて、水差発生箇所を 推定する水差推定工程と、を含む。 [translation] The process includes a mounting process to install the measurement jig, which is a humidity measurement process to perform the mounting and humidity measurement processes at multiple gas supply locations, and a water-difference estimation process to estimate the location of water-difference occurrence based on the measurement results of the humidity measurement process at multiple gas supply locations.
Figure 5: Exploratory Text Automatically Generated from the Image in Figure 1 [original] ガス供給箇所へ供給されるガスの流量を測定する ガスメータの一次側へ連通接続する連通空間を有 すると共に当該連通空間の湿度を測定する湿度測 定部を有する水差推定治具を取り付ける取付工程 と、湿度測定部にて連通空間の湿度を測定する湿 度測定工程と、湿度測定工程にて測定される湿度 に基づいて、ガス管に水差しが発生しているか否 かを推定する水差推定工程とを実行する。 [translation] The following processes are performed: An installation process in which a water-difference estimation jig is attached to a gas meter with a connecting space that is connected to the primary side of the gas meter that measures the flow rate of gas supplied to the gas supply point and a humidity measuring section that measures the humidity in the connecting space; a humidity measurement process in which the humidity in the connecting space is measured by the humidity measuring section; and a water-difference estimation process in which whether a water drop occurs in a gas pipe is estimated based on the humidity measured in the humidity measurement process.
Figure 6: Patent Summary for the Image in Figure 1 (Correct Answer)
Flowcharts with complex shapes, such as the one shown in Figure 7, tended to generate low-quality explanatory text. The dash line boxes in the figure were added by the author for the purpose of explanation. The description generated by the process in Figure 7 is shown in Figure 8. [translation] The system moves to the latent mode (s2110), obtains the reliability information (s2110), determines the variable magnification factor (s2110), and starts measuring the elapsed time (s2112). The image display then determines whether or not the predetermined time has elapsed (s2112). When the predetermined time elapses (yes at s2112), the latent report mode is terminated (s2112). When the latent report mode is terminated (yes at s2112), the latent report mode is terminated. Figure 8: Exploratory Text Automatically Generated from the Image in Figure 7
Looking at Figure 8, overall, step IDs such as s2110 do not correspond to the explanatory text, but this is because this time the coordinates of each string in the figure are not considered at all. The first conditional branch is “When the predetermined time elapses (yes at s2112), the latent report mode is terminated.” The correct sentence is generated except for the step ID (s2112) (see Figure 8). However, the dashed box in Figure 5 is not included in the explanatory text. Currently, the character strings output by Google Cloud Vision’s character recognition results are used as input to T5 as they are, but in the future, it will be necessary to perform preprocessing such as considering the coordinate information of the character strings and reordering them appropriately.
In this study, 11,188 flowchart image-description pairs were obtained from patents and these data were used to construct a system that automatically generates descriptions of flowchart images using T5. The experimental results showed that for the detection of flowchart images, an accuracy of 0.9645 was achieved with a fine-tuned model using DenseNet121. In the generation of explanatory text from flowchart images, it was found that high-quality explanatory text could be generated, although some issues remain for flowcharts with complex shapes. In the future, we will examine the possibility of generating appropriate explanatory text for flowcharts with complex shapes, such as those containing multiple conditional branches, by considering the positional information of each character string in the image, rather than using the character strings in the flowchart as is.
This work was supported by JSPS KAKENHI Grant Numbers JP22K12154 and JP20H04210.