Character recognition of ancient Japanese documents is a signicfiant research topic, we can comprehend Japanese history and culture from them. However, Kuzushiji (classical cursive handwriting characters) is pretty hard to understand for almost all young Japanese. Digitizing these books through character recognition technology can greatly assist us recognize. But the current dataset does not have enough samples for many classes, this makes it dicfiult to accurately recognize these classes. In this research, we propose a method to extend the dataset by using GAN (Generative Adversarial Network) to generate virtual images. Then we combine the generated images and merge these images with the original dataset. For this experiment, we trained with a decoupled YOLOv4 model. Experimental results show that applying this method to the classes with the number of samples greater than 50 and less than 300 can improve the overall accuracy by 21.65%.
There are nearly 2 million ancient Japanese documents, from which we can learn not only about
the ancient Japanese people’s food, clothing, transportation but also about Japan’s geographical and
ecological information in the past. Many people have contributed to tidy these documents [
This paper analyze the data and nfid some prepositions occur more than 30,000 times, like “ ”. Some characters we use a lot, like“ ”, can appear hundreds of times. But some characters, such as “ ”appears less than 5 times in all the data, which we can hardly nfid in a book. Data distribution is shown in Figure 2. Because of the number of occurrences, we believe it is enough for us to understand ancient Japanese documents by training the classes which sample size greater than 50. Due to some classes having too many data, we set the upper limit of sample size for all classes to 500. For the classes with a sample size greater than 300, because of the enough sample size, we don’t do anything to them. However, for the class with a sample size of 50 to 300, the sample size is far from enough to achieve a good recognition eefct. If we can expand the sample size of this part, and make it achieve a good training eefct, it can provide great help for the reading of ancient Japanese documents.
In order to increase the sample size of these classes, we propose using GAN [
Section 2 of this paper reviews related work on GAN and YOLO [
Since 2016, organizations such as the Center for Open Data in the Humanities (CODH) have successively released many datasets on ancient Japanese documents, and many researchers have devoted themselves to the study of ancient Japanese documents.
In 2018, Clanuwat et al. [
Goodfellow et al. [
Bowles et al. [
In 2016, Redmon et al. [
The core of GAN is generator and discriminator, the former is responsible for generating data based on random signals, and the later is responsible for determining the authenticity of the data generated by the generator. During each round of gradient backpropagation, the discriminator is trained rfist and then the generator is trained. Specicfially, supposing the GAN is now trained for the th time. The discriminator is rfist trained and the gennerator is xfied at this time, that is, the parameters of the gennerator are not updated currently. The real image and the virtual image generated in the previous round ( − 1) are stitched together and labeled respectively with labels 1 and 0. The stitched image is input into the discriminator for scoring and score G*(x) is obtained. According to G*(x) and the loss function, the gradient can be backpropagated to update the discriminator parameters. When the generator is xfied, the optimal solution (maximum point) of the discriminator is as follows, the means sampling from origin database, and means sampling from generator’s output.
∗( ) =
() () + () (1)
Then the generator is trained, and the discriminator is xfied currently. The generator generates virtual images based on the input random signal ( − 1) input discriminator to score D*G(x). The diefrence in value between D*G(x) and label 1 is backpropagated as a loss function. The loss function updates the parameters of the generator by minimizing the value when () and () are closest, that is, the th picture is exactly the same as the ( − 1) th picture, and that is what we want. The process is shown in the Figure 3:
We crop the images of the classes with the sample size of 50-300 and randomly select 30 of them to input the GAN. After training the network, the images are output in diefrent epochs; within 500 epochs, we will get 250 diefrent images as new data.
After the new images are generated, we use YOLOv4 to recognize them. Since YOLO requires the coordinate information of the target while taking object detection, we traverse the generated images to combine them and generate the location information of them. Our combination method is to rfist obtain the size of the generated image, and then take a blank image as the bottom, paste the generated image to the blank image in turn, each paste we will save the location information of the pasted image according to the size of the image and the order of the image paste. In order to ensure the uniqueness of the sample, we adopt the forward order traversal and reverse traversal, and limit the number of sample classes in an image, so as to ensure that there are no repeated classes in each image, and the order is not the same. We merge the generated dataset with the original dataset to obtain a large dataset with enough samples. After that, we put the large dataset into the YOLOv4 model for training. The YOLOv4 structural model is shown in Figure 4.
Kuzushiji recognition using YOLOv4 proceeds as follows.
Step1: Dividing the image into S×S grids, if the center of an object falls in the grid, the grid is responsible for detecting the object, and each grid outputs bounding box information and conditional class probabilities.
Step2: Putting image into the DarkNet53, and the feature information of the image is obtained from 3 diefrent dimensions.
Step3: After feature information is extracted, putting it into the PANet layer to carry out multiscale fusion of features, to enhance the localization ability on multiple scales.
Step4: Using YOLO head for prediction. By setting a threshold of 0.5, CIoU loss function is used to predict the location information of bounding box and the probability information of objects belonging to a certain class. We use decoupled YOLO head to alleviate the inherent conflict between classicfiation and regression tasks, and better tfi our Kuzushiji dataset.
Step5: Using NMS (non maximum suppression) to search and discover potential objects. First, suppress targets with CIoU<0.5. If there are many classes of objects with CIoU >=0.5, this method can get the objects from the higher CIoU score to the lower.
Step6: The nfial step generates the result of Kuzushiji recognition.
The training environment of GAN is Nvidia RTX3090TI GPU accelerator and intel core i712700KF processor. In the training parameters, the batch size is set to 8, and the epoch is set to 6000.
Figure 6 shows the training results for each class. The generated image is diefrent from the original image in subtle places, but we can’t determine which image is real and which is virtual. Most of the generated images are high denfiition, only a few images are blurred or even missing. This is because some of the training data has the same problem, we don’t process these data to simulate the real phenomenon that appears in the documents. After all the classes are generated, we stitch the images. We stitched a total of 2,250 images. After combining the original 5,647 images, they are used as nfial training dataset.
We also deal with the problem of balance of the data. From Figure 2, we can see that the training data is seriously unbalanced, and some classes even have 30,000+ samples. To prevent the experimental results from being one-sided, we set the number of samples within 500 for all classes. So, for the class with a sample size of 50 to 300, we generated 250 images per class. Figure 7 shows the distribution of the number of samples in the dataset before and after processing.
As can be seen from the Figure 7, these 1,073 categories are adjusted from extremely imbalanced data to sample number diefrence less than 200. Therefore, we can use the optimized dataset for YOLO detection.
In order to enhance the recognition eefct of YOLO in the training phase, the initial learning rate is set to 0.001, and condfience is set to 0.5. Due to the large number of classes, we use Adam optimization function to speed up the convergence of the model, the number of epochs is set to 600 and use an uncoupled YOLO head which can predict the position of the bounding box, the classes and whether there is an object in the bounding box, respectively. The use of coupled YOLO head will result in uncertain internal conflicts in the prediction phase. Table 1 shows the comparison between the coupled and uncoupled YOLO head in terms of prediction accuracy. Using the uncoupled YOLO head can improve the mAP by 6%.
Figure 8 shows the recognition of using YOLOv4 to train the original dataset and the large dataset. As can be seen from the gfiure, the model trained with the original dataset fails to recognize many infrequently used characters. However, the model trained with the larger dataset is able to recognize more characters. The detection accuracy of the former is 15.59% for the number of samples is greater than 50, and 46.81% for the number of samples is greater than 300. The latter are 37.24% classes coupled YOLO head
decoupled YOLO head sample num>300 sample num>400 sample num>500 and 45.96%, respectively. It can be seen that with the increase of the number of data sets, the introduction of training data unrelated to the unprocessed data sets will lead to a small decline in the recognition accuracy of this part of data, but the impact is not signicfiant. However, for the data with a sample size of 50 to 300, the recognition accuracy of this part of data has been greatly improved due to the addition of this part of training data. This proves that training with an insucfiient number of samples is far less eefctive than training with a large dataset which is augmented by our method.
In this paper, we use GAN to generate virtual images to augment the Kuzushiji dataset. This method generates amounts of new images by randomly sampling the potential space as input and modifying the parameters of the generated network through continuous discriminant correction. We combine the generated data with the original data to augment the dataset. And we propose to use decoupled YOLO head to train the large dataset, alleviating the inherent conflict between classicfiation and regression tasks. According to our experimental results, the mAP of the original dataset is 15.59%, the model’s mAP of the large dataset is 37.24%. The results veriefid the eefctiveness of the proposed method. We provide a method to solve the imbalance of Kuzushiji, so that we can train a YOLO model that can achieve better detection results. It can better help people understand the ancient history of Japan through AI methods.
In the future, we hope to experiment with diefrent GANs, such as DCGAN, LSGAN, WGAN to compare the data generated by diefrent GANs, and nfid a network that better tfis the dataset. Secondly, trying to improve the YOLO to achieve the highest performance. Thirdly, we want to try another way of stitching the generated images, which is to insert the generated image into the original book image. Augmenting the dataset with the same number of images.