In the lack of data, an appropriate inductive bias is one of the key factors for the successful training of a model. One approach to transfer inductive bias between the diferent structures of networks is to utilize knowledge distillation. Several studies have achieved promising results in computer vision datasets using response-based knowledge distillation. However, we observe that the previous method fails to transfer inductive bias when the dataset contains fewer data points or classes. To solve the problem, we propose to use feature-based knowledge distillation instead of response-based knowledge distillation for efective inductive bias transfer. Through extensive experimentation and analysis, we demonstrate that the suggested method can transfer inductive bias and outperform previous methods.
Inductive biases are constraints enforcing the model to
have specific properties [
We wonder about the applicability of transferring the Our contributions to this study are the following: First, inductive bias via knowledge distillation in various real- we analyze the limitation of the previous methods of world datasets. To verify this, we evaluated the technique transferring the inductive bias through knowledge distiltransferring the inductive bias in used DeiT on two types lation. Second, we examine the reason for the failure of of medical datasets: the inductive biases (1) in convolu- the previous methods via rigorous experiments. Third, based on the findings from the experimental results, we propose an efective way to transfer inductive biases through knowledge distillation.
AMLTS’22: Workshop on Applied Machine Learning Methods for Time Series Forecasting, co-located with the 31st ACM International Conference on Information and Knowledge Management (CIKM), October 17-21, 2022, Atlanta, USA $ bytaklee@medicalai.com (B. T. Lee); yy.jo@medicalai. (Y. Jo); cto@medicalai.com (J. Kwon)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org)
There are two possible reasons for the failure of the previous method. First, if the teacher’s inductive bias is weak, the signal from the teacher can be insuficient to provide valid information to the student (Figure 2(a)). The second possible explanation is that, even if the teacher has a suficient inductive bias, the force pushing the student network to encode the teacher’s inductive bias may be insuficient (Figure 2(b)). In this section, we explore the teacher’s and student’s representations and filters to identify the reason for the limitation of the previous approach.
2.1.1. Dataset network for EHR datasets [9]. LSTM is stacked with the 3layer, and each layer has 256 hidden units with a residual connection between each layer.
As a student network, we adopt a transformer [10]. There are two student networks, each of which has eight blocks for the ECG dataset and three blocks for the EHR dataset. Training a transformer on ECG datasets, we split a signal into patches following Dosovitskiy et al. [11]. Each patch consists of 100ms (20 timestamps) without overlapping and is used as the input of a transformer. In EHR datasets, a patient has multiple rows, each of which consists of a medical record at a time. A single row is used as a token of the input.
We set a batch size of 512 for the ECG dataset and a batch size of 256 for the EHR dataset. We use an Adam optimizer with the weight decay and the cosine warmup scheduler that peaks at ten epochs. In the experiment with ECGs, the rand augment policy [12] is adopted with six data augmentation methods, including the gaussian smoothing, time resampling with cut, gaussian noise, baseline wander, time mask, and channel mask. In the case of EHRs, data augmentation is not applied. Hyperparameters, such as the learning rate, weight decay, dropout, and parameters for the augment policy, are randomly selected from predefined search space, tuned by the asynchronous successive halving algorithm [13] using the ray framework [14]. The search space and selected hyperparameters are provided in Appendix A.
Train, validation, and test set are divided into a ratio of 0.7:0.15:0.15.
We used the following datasets with diferent
properties: Physionet 2021 for CNN and Physionet 2019 for
RNN. Physionet 2021 is a public ECG datasets [
We additionally used two external datasets to see if the other hand, if the similarities between the teacher’s
the DeiT preserves the inductive biases of CNN/RNN re- and the student’s representations are low, the teacher’s
gardless of the data distribution. The Hangzhou dataset representation is not efectively transferred to the
stu[
We develop two teacher networks: (1) the ResNet-based network for ECG datasets [8]. Each block of ResNet contains two layers of convolution, and there are eight blocks in total. The architectural detail is identical to Hannun et al. [8]. (2) the long short time memory (LSTM) We first examine the output similarity as shown in Table 1. The number in the table is the r-square value. The similarity between DeiT and its teacher (CNN/RNN) is slightly higher than the similarity between the naive transformer and CNN/RNN; however, the discrepancy between the teacher and the student is still large. This
To examine internal representation similarity driven by the architecture, we exploit central kernel analysis [15]. Figure 3 illustrates the representational similarity between CNN/RNN in comparison to DeiT and Transformer. We observe that CNN/RNN’s feature extraction process difers from that of Transformer. DeiT has higher similarity to CNN/RNN compared to Transformer, but there is still a substantial diference to its teacher. Specifically, in the case of DeiT, only the early layers exhibit a significant dissimilarity between the representations, indicating that the early layers of DeiT failed to learn the CNN/RNN representation.
Suppose the inductive bias of the teacher(CNN/RNN) is appropriately transferred to the student(Transformer). In that case, the student’s self-attention should display the pattern of the teacher, i.e., spatial/temporal invariance and locality (Figure 2(a)). However, the student’s selfattention would not exhibit the pattern of the teacher if the inductive bias of the teacher is not appropriately transferred to the student (Figure 2(b)). Figure 4 depicts the averaged self-attention matrices in each block across all samples and heads. It is dificult to distinguish the pattern of DeiT distinct from Transformer. To elaborate, DeiT does not exhibit the characteristics that demonstrate the inductive bias of CNN/RNN, such as translational/temporal invariance or locality.
The examination reveals that using DeiT, the teacher’s inductive bias is not well transferred to the student, which is the case of Figure 2(b). There could be several reasons why DeiT works with ImageNet but not with our dataset. The first possibility is the size of the dataset.
In the case of ImageNet, large data of 1M is suficient to transfer inductive bias via KD. However, the size of 3. Better solution for transferring
inductive bias the data we utilized is only 8 percent of ImageNet, so it temporal axis and projects them along the depth axis. may be challenging to transfer inductive bias via KD. The ℎ→() = () second possibility is the number of classes. ImageNet consists of one thousand classes, whereas the dataset we utilized consists of twenty-six for Physionet 2021 and two classes for Physionet 2019. With this respect, DeiT may not work with our dataset because distributions obtained from our dataset contain less information than distributions obtained from ImageNet. Based on this, we believe the problem can be alleviated if the student is provided with more information to encode inductive bias. where ℎ(· ) := R× → R′× ′ consist of two-layer: (· ) := R× → R′× represents the resize along the temporal axis, and ∈ R× ′ is linear transformation along the depth axis.
With a transformation function, we match and train each block of the teacher and the student (, ) to be similar as illustrated in Figure 5. In addition to matching between blocks of the teacher and the student, we also perform knowledge distillation between the output of the successive composition of blocks of the teacher and the student ( ∘ · · · ∘ 1, ∘ · · · ∘ 1). Each loss function term is formulated as follows.
(1)
The knowledge distillation utilized in the DeiT is a type of response-based knowledge distillation that distills knowledge using the model’s output. In contrast to the previous works, we impose a stronger signal by using featurebased knowledge distillation to enforce the student network to learn the teacher’s inductive bias. Additionally, knowledge distillation is performed on feature maps in order to transfer spatial information from the teacher to the student efectively.
First, we divide the teacher () and student ( ) into the same number of blocks and then perform the knowledge distillation between corresponding blocks (, ) of the teacher and the student. Since features transverse multiple layers, the dimension of it varies. For example, in the case of CNN, the pooling operation and convolution with stride change the dimensions with temporal direction, and the convolution operation change also increases the dimension of the feature. Because of this, the dimension of features used for knowledge distillation can vary. To solve the problem, we introduce a transformation function (ℎ) that transforms each dimension to be identical. This function resizes the feature’s dimensions along the ⃒ ⃒ 2 ℒ1 = ∑︁ ⃒⃒⃒ ⃒⃒⃒ (− 1) − (ℎ→ ∘ ∘ ℎ→−1)(− 1)⃒⃒ ⃒⃒ 2 , (2) ⃒ ⃒ 2 ℒ2 = ∑︁ ⃒⃒⃒ ⃒⃒⃒ (− 1) − (ℎ→ ∘ ∘ ℎ→−1 )(− 1)⃒⃒ ⃒⃒ 2 , ℒ3 = ∑︁ ⃒⃒ ⃒⃒ (ℎ→ ∘ ∘ · · · ∘
⃒ ⃒ , 1)() − ( ∘ · · · ∘ (4) Incorporating all, the loss function used in transferring the inductive bias is ℒℬℬℳ = ∑︀ (︀ ℒ1 + ℒ2 + ℒ3)︀ . We refer to the proposed method as block-by-block matching (BBM) because it performs knowledge distillation by matching each block of the teacher and the student. Using BBM method, the final loss function used for training is as follows: ℒ = ℒℒ + ℒℬℬℳ, where indicates the loss function for classification with cross entropy and is weight term for BBM. (3)
⃒ ⃒ 2 1)()⃒ ⃒ ⃒ ⃒ 2
We divide Transformer and CNN into four blocks in the ECG experiment, respectively. Each network’s blocks are divided equally, so each ResNet block contains four sub-blocks, and each transformer block contains two sub-blocks. Transformer and RNN are divided into three blocks for the EHR experiment, with each block containing one block of Transformer and one layer of LSTM, respectively. 3.2.2. Result
As shown in Figure 1 and 3, BBM demonstrates higher similarity in representation with its teacher. In addition, as demonstrated in Figure 4, we observe that the self-attention matrix of BBM successfully encodes its teacher’s inductive bias, such as spatial/temporal invariance or locality in the self-attention analysis. These prove that the proposed method encodes the inductive bias of its teacher successfully.
We viewed the network as composite functions, performing knowledge distillation on each function. Here, the question of the optimal number of blocks naturally arises. We perform experiments with varying the number of blocks to answer this question. As shown in Table 3, the performance increases as the number of blocks increases.
We show the limitation of DeiT on the transfer of
inductive bias and demonstrate that this issue can be resolved
using feature-based knowledge distillation. Through
experimental studies in medical data, we demonstrate that
our method consistently outperforms existing methods
as well as the strong inductive bias models.
Additionally, an extensive analysis verifies that the proposed
method transfers meaningful inductive bias to
transformers. Many studies focus on transferring the inductive
bias into Transformer on ImageNet. However, there is
insuficient analysis of other real-world data with
diferent properties to ImageNet. We expect our study will
help bridge the gap between research on ImageNet and
real-world data.
database, a freely available multi-center database A.1. Experiments on Physionet2021
for critical care research, Scientific data 5 (2018)
1–13. A.1.1. Convolution network
[8] A. Y. Hannun, P. Rajpurkar, M. Haghpanahi, The total of 100 search space is explored for maximum
G. H. Tison, C. Bourn, M. P. Turakhia, A. Y. Ng, epoch of 100 with early-stopping rate of 0.5 every 20
Cardiologist-level arrhythmia detection and classi- epochs. Learning rate ∈ [0.00001, 0.01] and weight
deifcation in ambulatory electrocardiograms using a cay ∈ [0.00001, 0.1] are sampled from log-uniform
disdeep neural network, Nature medicine 25 (2019) tribution. And dropout ∈ [0, 0.3], rand-augment
num65–69. ber ∈ [
Jordan, et al., Ray: A distributed framework for
emerging {AI} applications, in: 13th {USENIX}
Symposium on Operating Systems Design and Im- A.1.4. BBM: Knowledge distillation
plementation ({OSDI} 18), 2018, pp. 561–577. The total of 30 search space is explored for maximum
[15] S. Kornblith, M. Norouzi, H. Lee, G. Hinton, Simi- epoch of 500 with early-stopping rate of 0.5 every 30
larity of neural network representations revisited, epochs. Learning rate ∈ [0.0001, 0.1] and weight decay
in: International Conference on Machine Learning, ∈ [0.00001, 0.1] are sampled from log-uniform
distriPMLR, 2019, pp. 3519–3529. bution. And dropout ∈ [0, 0.3], rand-augment number
∈ [
All the hyperparamters in experiments are chosen based on extensive hyperparameter search, which is performed using asynchronous successive halving algorithm. The search space and selected hyperparameters are described in the following.
The total of 100 search space is explored for maximum epoch of 100 with early-stopping rate of 0.5 every 20 epochs. Learning rate ∈ [0.0001, 0.01] and weight decay
For the transformer, the number of head ∈ {4, 8}, the dimension of the model ∈ {128, 256, 512}, the dimension of transformed network in the feed forward layer ∈ {128, 256, 512} are randomly sampled. Other settings are identical to the recurrent network. The chosen set of hyperparameters are 0.0008136 for learning rate, 0.00001523 for weight decay, 0.25 for dropout. For hyperparameters of transformer architecture, each hidden unit, model, and head is 128, 512, and 8.
A.2.3. DeiT For the hyperparameters related to transformer’s architecture, We used the hyperparameter set selected in the transformer trained from the scratch. We performed hardlabel distillation as the experiment in Physionet2021. In the search space, we added loss ratio between classification token and knowledge distillation token. The chosen set of hyperparameters are 0.0001554 for learning rate, 0.0001728 for weight decay, 0.05 for dropout, and 0.2 for the the ratio of knowledge distillation.
The architecture selected in transformer trained from the scratch is used. The total of 30 search space is explored for maximum epoch of 500 with early-stopping rate of 0.5, every 30 epochs. Learning rate ∈ [0.0001, 0.1] and weight decay ∈ [0.00001, 0.1] are chosen with loguniform distribution. And drop out ∈ [0, 0.3] is sampled from quantified uniform distribution with the interval of 0.05. The chosen set of hyperparameters are 0.002099 for learning rate, 0.0001718 for weight decay, and 0.05 for dropout.