Recently, deep convolutional neural networks have brought great breakthrough in image classi cation, which provide e ective solution for the handwritten Chinese character recognition problem. Researchers have experimented with various networks to increase recognition accuracy. Although good accuracy is achieved on di erent networks, these networks tend to be computation-intensive and memory-intensive that make them di cult to be deployed on resource-constrained devices. To solve the problem, the paper proposes an optimization to reduce the number of model parameters by using pruning network and knowledge distillation. Besides, to improve the model's ability to extract the input features, an attention mechanism is adopted in the proposal. The experimental results show that the number of parameters decreased by nearly 26%. At the same time, the recognition accuracy improves by 1.17% with the value of 96.99% compared with the original model. The optimization method presented in this paper not only improves the accuracy of handwritten Chinese characters recognition but also reduces the number of model parameters.
Attention mechanisms are widely used in deep learning to enhance the performance of CNNs. [7] terms the "Squeeze-and-Excitation" (SE) block, that adaptively recalibrates channel-wise feature responses by explicitly modelling interdependencies between channel. SE blocks bring signi cant improvements in performance for existing state-of-the-art CNNs at slight additional computational cost. [8] proposes convolutional block attention module (CBAM), CBAM sequentially infers attention maps along two separate dimensions, channel and spatial, then the attention maps are multiplied to the input feature map for adaptive feature re nement. [9] proposes an e cient channel attention (ECA) module, which only involves a handful of parameters while bringing clear performance gain. Avoiding dimensionality reduction is important for learning channel attention, and appropriate cross-channel interaction can preserve performance while signi cantly decreasing model complexity. [10, 11] uses weighting and Hu man coding to minimize storage space furthermore. [12] eliminates the unimportant channles by applying L1 regularization to the scale factor of the batch normalization (BN) layer. [13] uses least absolute shrinkage and selection operator (LASSO) regression to sparse the weights and cut out unimportant channels. Then, least squares method is used to ensure that the cropping operation has little impaction on the features by using LASSO. [14] takes adopts a new Taylor expansion-based criterion for approximating the loss function change caused by pruning network parameters. This is a modern formula for achieving e ective reasoning in neural networks through pruning the convolution kernel. [15] introduces a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish ne-grained classes that the full models confuse. [16] presents matching guided distillation (MGD) as an e cient and parameter-free manner to solve the problem of adding the adaptation module in classic methods. 3 3.1
Channel attention mechanism has demonstrated to o er great potential in improving the performance of CNNs, which can be used for classi cation. The attention mechanism in deep learning draws on human attentional thinking to focus on the key information in an image rather than the whole image [17]. As shown in Fig. 1, the lightweight attention mechanism ECA module is used for HCCR.
Adaptive Selection of
Kernel Size: k = Ψ (C) H
C W
GAP
s 1×1×C 1×1×C
H
C
W element-wise product Given an aggregated feature y 2 RC without dimensionality reduction, channel attention ! = (Wy)
Where W is a C C parameter matrix. In order to capture local cross-channel
interaction, employing Eq.(
Wk involves k
C parameters. When all channels share the same learning parameters.
can be learned by Eq.(
Where ik indicates the set of k adjacent channels of yi. Eq.(
Where C1D indicates 1D convolution. Here, the method in Eq.(
Where jtjodd indicates the nearest odd number of t. Setting and b to 2 and 1, respectively. Clearly, through the mapping , highdimensional channels have longer range interaction while low-dimensional ones undergo shorter range interaction by using a non-linear mapping. 3.2
Shown in Fig. 2, channel pruning is a coarser-grained pruning, which is accomplished by deleting
the redundant channels of feature map. A scale factor is added for each channel, which is
then multiplied by the channel output. The network weights and these scale factors are jointly
trained, and the latter is sparsely regularized. The redundant channels which are determined
according to the scale factors, is pruned after training. The training objective feature is given
as Eq.(
Where (x, y ) denotes the training input and output, w denotes the trainable weight, the
rst term of Eq.(
In the course of the experiment, choosing g(s) = jsj, which is known as L1-norm and widely used to achieve sparsity. BN has been adopted by most modern CNNs, as a standard method to achieve fast convergence and better generalization performance. Let zin and zout be the input and output of a BN layer, B denotes the current batchsize, BN layer performs the following transformation: z^ = zin B ; zout = z^ +
p B2 + "
Where B and B are the mean and standard deviation values of input activations over B,
and are trainsble a ne transformation parameters which provides the possibility of linearly
transforming normalized activations back to any scales.
(
The goal of knowledge distillation is to use the large model's knowledge to direct the small
model's training so that the small model can match the large model's output. The teacher
model and the student model are described as the large and small models respectively. Fig. 3
depicts the structure. To obtain a better soft target, the temperature parameter T is quoted,
as shown in Eq.(
qi =
exp(Zi=T )
Pj exp(Zj =T )
Where Zi is the probability of the i-th category in the output vector, j 2 (1; 2; :::; k), and k
is the total number of categories. The exp is an exponential operation, and qi is the soft target
output obtained by the function. For the same input, when T is set to 1, the student network
creat a hard target. Using a higher value for T produces a softer probability distribution over
classes, and the teacher network and student network generate a soft target respectively. The
hard target and the two soft targets are used as the input of the cross-entropy loss function
to learn the weights. As a result, the objective function of the knowledge distillation can be
summed up as Eq.(
L = input x
Layer 1 Layer 1
Teacher network La2yer Ă Student network La2yer Ă
Layer n Layer n
Softmax(T=t) Softmax(T=t) Softmax(T=1)
soft labels distillation loss
soft predictions
hard prediction student loss hard label y ground truth
Loss Fn Loss Fn
Shown in Fig. 4, the data on the left is from the CASIA-HWDB1.1 dataset, which is a publicly available HCCR dataset provided by the Institute of Automation of the Chinese Academy of Sciences. 16 classes are selected from the CASIA-HWDB1.1 dataset as part of the dataset in this paper. The right side is the same type of Chinese character written by di erent volunteers. The two parts are combined to form a new dataset, which is named MiniHWDB dataset. Shown in Table 1, MiniHWDB contains of 12,000 images. The dataset is split into two parts: the training set and test with the ratio of 8:2. In the process of training and inference, the input is resized to 224×224. The batch size is set to be 64. The adaptive optimizer SGD is taken to optimize the loss function. All of the experiments are conducted on a computer with the 3.00 GHz Intel(R) Core(TM) i7-9700 processor, 2×8GB of RAM, and a GeForce RTX 2060 graphics card with 6GB of video memory.
At the rst, ResNet18 is adopted as the original network. As shown in Fig. 4, di erent people have their own writing styles, and there is a lot of useless information (white area) in the input. The attention mechanism ECA is used to improve the feature extraction from the input. After that, the teacher network ECA-ResNet56 and the student network ECA-ResNet18 are obtained respectively. Next, the student network is pruned at the channel level according to the pruning ratios, which can be set 0.4 and 0.6. When the pruning rate is de ned as 0.4, it means that 40% of the channels are pruned. The new student networks CS-ECA-ResNet18(0.4) and CS-ECA-ResNet18(0.6) are obtained after the pruning is completed. Finally, the teacher network ECA-ResNet56 is used to guide the pruned student network. The distilled student network is named KD-SC-ECA-ResNet18. In the course of the experiment, the parameters are set as shown in Table 2.
Description Adaptive selection of kernel size k Temperature Batch size Minimun number of epochs Maximun number of epochs The accuracy of the original network ResNet18 reached 94.40%. By introducing the attention mechanism ECA, the accuracy is improved by 1.42%, while the number of parameters only increases by 4.5%. Since ECA is a lightweight module, it can be seen that the parameters increase by introducing the attention mechanism is negligible. Channel pruning reduce the number of parameters by removing unimportant channels, but the result is loss of accuracy. To improve the loss of accuracy due to pruning, the method of teacher network is taken to guide the pruned network. Teacher networks usually to be deep networks. Although the increasing in depth of the network improves the accuracy, it also brings signi cant increase of parameters. For example, the teacher network is much deeper than the student network, but only 3.39% accuracy improvement. However, the number of parameters is 2.19 times than the student network. Although parameters and accuracy are di cult to balance in the task of HCCR, the parameters are given priority. Because these networks are mostly deployed on devices like mobile phones that do not have large storage.
With the increase in the number of parameters, the model is hard to be deployed on embedded devices. So channel pruning is adopted to reduce the number of parameters, this results in a loss of accuracy. Therefore, knowledge distillation is used to improve the accuracy of the pruned network. When the pruning rate is 0.4. The accuracy of the KD-SC-ECA-ResNet18(0.4) is improved 1.71%, and the number of parameters is reduced 16.7%, compared to the ECAResNet18. When the pruning rate is 0.6. The accuracy of the student network is improved 1.17%, and the number of parameters is reduced 25.6%, compared to before the pruning and knowledge distillation. The pruning rate is over 0.6, it is tough to obtain a good result even after knowledge distillation. The results of di erent models are shown in Table 3. Model ResNet18 ECA-ResNet18 ECA-ResNet56 KD-SC-ECA-ResNet18(0.4) KD-SC-ECA-ResNet18(0.6)
This paper focuses on images classi cation for o ine handwritten Chinese character recognition. The method by using attention mechanism, channel pruning and knowledge distillation, not only obtains higher recognition accuracy, but also has a lower number of parameters than original network. In this paper, the attention mechanism is used to improve the network's ability to extract features, channel pruning e ectively reduces the number of parameters, and the knowledge distillation improves the accuracy. It is bene cial for the model to be deployed on the resource-canstrained devices. In future work, the model can be compressed with other methods to further reduce model size. It is very useful for the development of arti cial intelligence, especially for the eld of computer vision.