The collector's seal is a stamp for the ownership of the book. There is many important information in the collector's seals, which is the essential elements of ancient materials. It also shows the possession, the relation to the book, the identity of the collectors, the expression of the dignity, etc. Majority of people from many Asian countries usually use artistic ancient characters to make their own seals instead of modern characters. A system that automatically recognizes these characters can help enthusiasts and professionals better understand the background information of these seals more efficiently. However, there is a lack of training data of labeled images, and many images are noisy and difficult to be recognized. We propose a retrieval-based recognition system that focuses on single character to assist seal retrieval and matching.
The use of the collector’s seals in Asian ancient books is a worthy topic of detailed discussions. The collector’s seal has the functions of expressing the sense of belonging, demonstrating the inheritance, showing the identity, representing the dignity, displaying aspiration and interest, identifying the version, expressing speech, etc., in different forms. For individual collectors, in addition to their own names, there are also words, which can show their personal or their family situations including ancestral origin, residence, family background, family status, rank and so on. For example, Natsume Soseki, a famous Japanese writer, had many kinds of collector’s seals. The foreign books, which he had collected during his life, were recorded using various collector’s seals that he used in different periods. We can get wider information about his book-collecting hobbies by analyzing these collector’s seals. One of the most important functions of the collector’s seals is to record where a book has been kept and the footsteps of history of a book being handed over. For book collecting institutions, the migration process of the books and their purchase histories can be also reflected in the contents of the collector’s seals. The names of many book collecting institutions have been gradually changed with the historical changes. Through the collector’s seals, we can understand the books themselves, their collectors, or more background information of the collecting institutions. Through collector’s seals in libraries, we can find out the collective experience and the source of inheritance of a book. In Asian countries where kanji characters are used, ancient characters are usually used to make collector’s seals. Also, as time passes, the shape of kanji may be changed, and multiple variations of a character might be created. There are also characters derived from kanji characters, or the characters that just look like kanji characters. For example, Vietnam’s “Chữ Nôm” is a character system that are originated from kanji’s original shape. Therefore, it is hard for non-professionals to understand all the contents of a collector’s seal or a single character in it. Meanwhile, as far as a single ancient character is recognized, utilizing the data from scholars studying kanji characters, we can also have a broader understanding of ancient character’s culture. Especially for individual collectors, exploring the information in every single character of their names is an important method for us to get a comprehensive understanding of their family affairs.
It is expected to construct a retrieval-based ancient character recognition system that can match the user’s query character even if there is only one labeled typeface image, and can update the character type at any time instead of retraining a model. Therefore, we propose a text extraction system for image data of collector’s seals, and this system is also expected for the future research in automatic text feature generation to find the background information of Asian ancient books from external databases using the extracted text contents in the character recognition task. In this research, we perform character segmentation using Mean-shift clustering and retrieve a single ancient character from ancient character typeface images by using the extracted features.
Character segmentation for off-line recognition. Nguyen et al. (2016)[
Seals image retrieval. Fujitsu R&D Center (2016) [
We describe our approach in the following sections. Since we only extract features from typeface images and store them in a database, we perform some pre-processing for user-provided seals images. We describe our data pre-processing in Section 3.1. In the beginning of Section 3.2, we describe the process of extracting essential features from images. These features include deep features and geometric features, therefore in Section 3.2, we also describe the deep features that we use. In Section 3.3, we describe the extraction of geometric features. Introduction of the feature matching and ranking calculation will be shown in Sections 3.4 and 3.5. 3.1
Data pre-processing and character segmentation As shown in Figure 1, in classical materials, seals often overlap with handwritten words.
Therefore, splitting the seal pattern from the image is an important task. We use k-means clustering
(Hartigan, et al.,1979) [
= √( 1 − 2)2 + ( 1 − 2)2 + ( 1 − 2)2 (1)
Where 1, 2, 1, 2, 1, 2 represent RGB values for pixels 1 and 2 respectively, and we use to cluster pixels with similar colors. According to the principle of k-means algorithm, we regard this task as extracting K groups of pixels with similar colors from images. Our system automatically extracts areas with more red components. As shown in Figure 3, the Pixel group 2 is extracted as the analysis target of our system.
Result of clustering
For the determination of a single character, we use the Mean-shift clustering to segment each
character. Because kanji characters are independent and balanced in structure, we regard every character
as a module, and each module has its centroid. By using these centroids, we can use a clustering to
extract character fields. Since we already know the coordinates of each pixel of the seal areas, the
density information of each pixel can be obtained by kernel density estimation. We use Mean-shift
clustering (Comaniciu, et al.,1999) [
bandwidth 1：60
Bandwidth 3：90 bandwidth 2：80 bandwidth 4：100
In each unit, the graph on the left side is the clustering result of each pixel of the seal, and each color represents a cluster, and as for the graph on the right side, the X-axis is the value of bandwidth, and the Y-axis is the number of clusters. The results that we need can be selected in the process when change rate of the total number of clusters becomes steady, for example, when the bandwidth equals to 90, we suppose each cluster in the result is treated as a candidate result of character segmentation. Hence, we calculate a bandwidth interval, and the bandwidth value is obtained equidistantly in the interval. By using these bandwidth values, segmentation candidates are obtained. The algorithm flow is shown in Algorithm 1.
Input: coordinate set { ( 1, 1), ( 2, 2)… ( , )} from non-background area obtained by Kmeans clustering Output: Set of object location hypotheses U 1: Initialize bandwidth interval：[Bandwidthmin, Bandwidthmax]={Bandwidth∈ ℝ： {Bandwidthmin < Bandwidthmax}，take the value at a moderate distance in the interval to get bandwidth set{b1, b2, b3 … . . bn ∈ [Bandwidthmin, Bandwidthmax]} 2: Get the result of number of clusters{Nclusters_b1, Nclusters_b2….Nclusters_bn: Nclusters_bn∈ fmeanshiftclustering (bn)} for each bandwidth using Mean-shift clustering 3: Find the value of when Nclusters_bn become to the minimum, take as the Endbandwidth. 4: Fit a polynomial Q (b) with {(b1, Ncluster_bs1),
(b2, Ncluster_bs2)…(bn, Ncluster_bsn)} using least squares polynomial fit. 5: Get the second derivative Q′′(b) =(d2Nclusters_bn)of Q(bn)
dbn2 6: Foreach bi ∈{b1, b2, b3 … . . bn ∈ [Bandwidthmin, Bandwidthmax]}do: if Q′′(bi) = Min{(Q′′(bi+1) − Q′′(bi))/Q′′(bi)} then
Initial Initbandwidth← else continue 7: Obtain regions U={u1, u2, u3, u4 ….} using Mean-shift clustering with bandwidth during[Initbandwidth, Endbandwidth]
The algorithm implementation is available on GitHub (Li, K et al., 2019)[
The typeface images converted from the font file (Shirakawa Font project, 2016) [
Modern commonly
used fonts “Shirakawa font”
Typeface ‘人’ ‘文’ ‘科’ ‘学’
First, we normalize the typeface image. Then we crop the typeface image according to maximum and minimum coordinates of the black pixels, and then standardize them to the size of 64*64. As there are many variations of ancient characters, even a slight change of the structure will affect the extraction of geometric features. We use the pre-trained model to extract the deep features (convolutional neural networks (CNN) features) of fonts, trying to subtract some minor changes in character’s structure, to assist the characters of same category become closer to each other in the feature space.
As shown in Figure 6, due to the lack of ancient character dataset, we selected CASIA Online and
Offline Chinese Handwriting Databases (Liu et al.,2011)[
Extracting geometric features from images
To capture some details of the character’s structure, we attempted to extract the geometric features of these characters.
As shown in Figure 8, using the method proposed by Zhang’s research (Zhang et al.,1984)[
We use the following method for calculating similarity to match the typeface image and the query image. The matching process is shown in Figure 9. The segmented user query image and the typeface image use the same feature extraction method to extract the corresponding features including CNN features and geometric features.
The cosine similarity is used to compare the similarity of CNN features, and Hausdorff distance
(Huttenlocher et al.,1992) [
= +
+ (3) Where feature is the total score of similarity, is the sum of similarities of geometric features which calculated use _ℎ .
are the weights of similarity score of CNN and geometric feature t. Finally, we use the total score of similarity to predict the input image’s category. 4
explained in Sections 4.2 and 4.3. 4.1
In this section, we will describe our experiments in three parts. In section 4.1, we will introduce the database used in the experiments. The results of image segmentation and image retrieval will be
We select the test data from Collectors’ Seal Database (National Institute of Japanese Literature,
available from 2011)[
Different types of seals are used to test our proposed segmentation algorithm, and the results are shown in Table 2. From the experiment results, it can be seen that our proposed method achieves good results in processing images with irregular characters distribution. Nevertheless, from the Result 3, larger characters are segmented into other character candidate area, thus more efforts are still needed to conduct further research on dealing with images with different character sizes and close character spacing.
1: Regular seal
irregular glyph
background and overlaps with handwritten words
4.3
Here we show the Mean Reciprocal Rank (4) results of ten characters with the highest frequency in the printed text. For each character, we tested with twenty images. where Q is the total number of images retrieved, i is the image number, and rank is the ranking order.
When all the weights are set as initial value 1, the results are shown in Table 2. We found that characters with simple shapes showed better results.
Characters
MRR score
Characters
MRR score
In this study, we used two clustering algorithms to preprocess seal images and have obtained the good results. Not alike to train a neural network, clustering algorithm is a method that can extract part of the required information from an image without consuming a lot of computational resources. Then we use the combination of deep features and geometric features to retrieve ancient kanji characters through the calculation of similarities. It reduces the time of re-training a model when adding new category of characters and also enables flexible use of intermediate output of neural networks. We can know which seal belongs to which famous person using the recognition results, then we can learn more about this person’s hobbies from his or her collection’s information, and this will be the next target of our research.However, the image retrieval performances need to be further improved. How to make better use of only one typeface image is needed to be focused in our future research.