Image segmentation is the core computer vision problem for identifying objects within a scene. Segmentation is a challenging task because the prediction for each pixel label requires contextual information. Most recent research deals with the segmentation of natural images rather than drawings. However, there is very little research on sketched image segmentation. In this study, we introduce heuristic (point-shooting) and deep learning-based methods (U-Net, HR-Net, MedT, DETR) to segment technical drawings in US patent documents. Our proposed methods on the US Patent dataset achieved over 90% accuracy where transformer performs well with 97% segmentation accuracy, which is promising and computationally eficient. Our source codes and datasets are available at https://github.com/GoFigure-LANL/figure-segmentation.
Combining information contained in text and images is
an important aspect of understanding scientific
documents. However, patents and scientific documents often
contain compound figures containing subfigures, each
ciate individual subfigures with the appropriate caption
and reference text, we must first segment the full figure
into its individual subfigures. Although much research
has been done on figure understanding and extraction
for scientific documents, existing methods rely on either
(1) manually-designed rules and human-crafted features
machine learning approaches most of which were trained
on natural images [
Image segmentation has been extensively studied with
rule-based methods such as watershed, and machine
learning methods applied to natural images [
with other methods. We release the benchmark dataset that can be used for future work on the task of segmenting technical drawings.
experiment, we chose an empirical value = 2 . If a black pixel in the original figure was detected inside the open dot, the dot retrained. Otherwise, the dot was removed. We constrain the circle centers so they do not fall outside the figure boundary. We then fill all retained circles and draw contours1. Using the contour information, we draw rectangular bounding boxes to segment a figure. 3.2. Deep Learning Methods
States Patent and Trademark Ofice (USPTO). The ground truth dataset is developed on a corpus of 500 randomly selected figures from the design category of patent. The dataset consists of 20 figure files with single drawings and 480 figure files, which containing at least two subifgures. We preprocess each figure to remove text labels.
The number of subfigures in each figure file is inferred by the number of text labels detected identifying subfigures so segmentation is only necessary for figures containing multiple subfigures.
We use VGG Image Annotator (VIA) to annotate our dataset. VIA is a manual annotation open source software for annotating images, videos, and audio. We draw rectangles bounding boxes around subfigures. Each figure consists of 2–12 subfigures. We also performed an independent human verification to ensure the bounding boxes were drawn correctly. VIA allows exporting annotation results including filename, file size, region count (e.g., number of the bounding boxes for each figure in an image), region id, and coordinates of bounding boxes.
The point-shooting method is easy to implement and successfully segments most figures containing multiple technical drawings. However, the method does not generalize well for certain figures in our dataset. One example is shown in Figure 3. One can see that in Row 2, the point shooting method creates wrong bounding boxes (Column 1), while U-Net, a deep learning model, produces the correct bounding box (Column 2).
Therefore, we consider applying deep learning-based
methods including U-Net [
Figure 1 illustrates the procedures of this method. Af- the bounding boxes on the input image. ter removing the figure labels, we randomly pick a pixel in the figure , and draw an open dot of a radius . For our python-c/
Most existing SoTA deep learning based models were ture map from a contracting path, and two convolutions,
pre-trained on natural or medical images, which usually each followed by a ReLU.
contain rich color and/or gradient information compared
with technical drawings, which are mostly sketched im- 3.2.2. HR-Net
ages containing black/grey-scale pixels. Therefore, these
pre-trained models usually result in a poor performance In the contracting path of U-Net, feature maps are
downwhen tested on technical drawings in our dataset. To sampled to the lower resolution using polling and later
overcome this limitation, we fine-tuned pre-trained mod- up-sampled in the decoder part. In this process,
highels or trained them from scratch. To reduce unnecessary resolution information is lost. Although skip
conneccomputational cost, we rescale the resolution of the in- tions are used to copy the high-resolution information
put figure to 128 × 128 and use them to train the deep to the expansive path. They can not fully recover
highlearning models. The model produces the segmentation resolution information. To overcome this drawback, we
mask with a dimension of 128 × 128 × 3 which we use to apply the HR-Net model which retains both high and
draw contours and then bounding boxes around contours. low resolution information throughout the training
proAfter obtaining bounding boxes from the low-resolution cess. The preserved information may be useful to
reconimage, we linearly scale up the predicted bounding boxes struct the segmentation mask. We simplified the original
to fit the original figure. HR-Net, which contains three resolution channels, each
capturing high, mid, and low resolution information,
re3.2.1. U-Net spectively. The three channels contain five, three, and
two convolutional blocks, respectively. Each
convoluThe architecture of U-Net consists of a contracting path tional block contains two convolutional layers followed
and an expanding path. The contracting path is a typi- by a batch normalization layer, and a ReLU activation
cal convolutional network containing a series of convo- layer. The resolution gap between two channels is 2.
lutional layers, each followed by a rectified linear unit
(ReLU) and a max pooling layer with stride 2 for down- 3.2.3. Transformer
sampling. At each downsampling step, the number of
feature channels is doubled. In the expanding path, each step
consists of an upsampling of the feature map followed by
an “up-convolution”, a concatenation with cropped
feaAlthough the CNN-based models have shown
impressive performance on the segmentation tasks [
module is used to train a transformer on a small dataset. tain cases, the point-shooting method produced the
correct segmentation map but deep learning-based methods
positional encoding on non-local context. This architec- failed, such as the segmentation results in Row 1 and
models.
3.2.4. MedT
The core component of MedT is a gated position-sensitive
axial attention mechanism designed for small size datasets
[
In general, deep learning-based methods perform better than point-shooting methods, such as the segmentation results in Row 2 of Figure 3. However, in certure contains two branches, including a global branch that captures the dependencies between pixels and the entire image and a local branch that captures finer dependencies among neighbouring pixels.
The training figures are passed through a convolution
block before passing through the global branch. The
same figure is broken down into patches and sent through
a similar convolution block before passing through the
local branch sequentially. A re-sampler aggregates the
outputs from the local branch based on the position of
the patch and generates output feature maps. Outputs
from both branches are add together followed by a 1 × 1
convolutional layer to pool these output feature maps
into a segmentation mask.
3.2.5. DETR
DETR is an end-to-end object detection transformer model
[
The segmentation task can be seen as a classification problem, in which individual subfigures are foreground objects, and the blank area between subfigures is the background. Although we use a training corpus with noisy labels, the deep learning models successfully capture latent representations and correctly segmented individual drawings. The evaluation results are shown in Table 1.
models of challeging cases are shown in Figure 3.
which all methods failed (Figure 3). This occurred when a subfigure contains relatively isolated fragments without prominent connections, which were treated as individual objects.
In conclusion, we compared heuristic and deep learning methods on the task of segmenting technical drawings in US patents. Both heuristic and deep learning-based models achieve over 90% accuracy. Interestingly, though we trained using data containing noisy labels, generated using the point shooting method, the deep learning models still captured the right features and outperformed the point shooting method. The CNN-based model (e.g., HR