Domain adaptation is a technique that tackles the dataset shift scenario, where the training (source) data and the test (target) data can come from different distributions. Current research works mainly focus on either the covariate shift or the label shift settings, each making a different assumption on how the source and target data are related. Nevertheless, we observe that neither of the settings can perfectly match the needs of a real-world bio-chemistry application. We carefully study the difficulties encountered by those settings on the application and propose a novel method that takes both settings into account to improve the performance on the application. The key idea of our proposed method is to select examples from the source data that are similar to the target distribution of interest. We further explore two selection schemes, the hard-selection scheme that plugs similarity into a nearest-neighbor style approach, and the soft-selection scheme that enforces similarity by soft constraints. Experiments demonstrate that our proposed method not only achieves better accuracy for the bio-chemistry application but also shows promising performance on other domain adaptation tasks when the similarity can be concretely defined.
Machine learning has been a high-profile topic and succeeded in various kinds
of real-world tasks due to the vast amount of labeled data. However, collecting
well-labeled data from scratch is time and labor-consuming. Therefore, in many
applications [
A family of techniques that aim at tackling the dataset shift problem is domain adaptation (DA). In this work, we try to solve the more challenging unsupervised domain adaptation (UDA) problem, where we can only access the labeled source data and unlabeled target data in the training phase. The goal of UDA is to learn a model from these data and to achieve good performance on the target domain. Intuitively, learning under UDA is not possible if the source and target domains do not share any properties. Previous works on UDA thus make assumptions about the properties shared by the two domains and design algorithms based on the assumptions. Two major assumptions, covariate and label shift, have been considered separately in previous research works.
The assumption of covariate shift considers the mismatch of feature
distribution between the source and target domains. Furthermore, it is assumed that the
labels of both domains are drawn from the same conditional distribution given
the features. There are two main families of methods designed under this
assumption, namely, the re-weighting method [
Most recent works extend from the two settings and demonstrate promising
performance. However, motivated by a real-world bio-chemistry application, we
find that current domain adaptation methods designed for only one of the two
assumptions cannot cope with all the application needs. We carefully examine the
application and find it comes with the shift of label distribution that can be easily
observed from the polarity of label distribution. However, the assumption that
the conditional distribution given label does not seem to be the same, violating
label shift assumption. Accordingly, we must use covariate shift assumption to
model this application. Here comes the problem: If the application is tackled with
the covariate shift assumption using adversarial training, the label distribution
should be the same on the aligned data, violating the polarity property of the
dataset. Therefore, we conclude that this application requires considering both
the covariate shift and label shift properly. In this paper, we study how to follow
the covariate shift assumption while taking the possible label shift into account
for the bio-chemistry application. [
Inspired by some intuitive toy examples, we find that selecting representative examples from the source data allows us to construct a similar-feature and similar-label subset of the source data that resolves both covariate shift and label shift. If the feature space implicitly encodes the distance between two features with physical meaning, we can construct the subset through the nearest-neighbor algorithm by considering the distance as the similarity measure. Based on this finding, we propose two methods, Hard/Soft Distance-Based Selection, to handle different situations. The hard selection directly uses the subset of the source data we construct to train the model, whereas the soft selection enforces similarity on the subset by adding a soft constraint.
Experiments show that our methods successfully capture the structural information and utilize the distance-based similarity and thus mitigate the impact from the label shift in the application. To test the performance of our methods in high-dimension space (e.g., image space), we also do experiments on the benchmark dataset (digits). Further, we extend our methods to tackle this scenario and have promising experimental results. Finally, we discuss what are the good situations to utilize our methods, through a simple noisy source data experiment.
Our contributions of this thesis include 1. We carefully study the difficulties encountered by concurrent UDA methods on a real-world application. 2. We propose two methods based on representative selection to overcome the difficulties. 3. We study how the proposed methods can be extended in different scenarios. 2 2.1
We consider a K-way classification task and let X and Y represent the random variables for the feature and label respectively, where Y = {0, . . . , K − 1}. We denote the joint distributions for the source and target domains as PS (X, Y ) and PT (X, Y ). The marginal distributions of X and Y in the source domain are defined as PS (X) and PS (Y ). Similarly, PT (X) and PT (Y ) represent the marginal distributions of X and Y in the target domain. The conditional label distributions in the two domains are denoted by PS (Y |X), PT (Y |X). PS (X|Y ) and PT (X|Y ) stand for the conditional feature distribution in the two domains.
We consider the UDA setting in this thesis. There exists a set of labeled data DS = {(xi, yi)}in=1 in the source domain, where each instance (xi, yi) is drawn i.i.d. from PS (X, Y ). In the target domain, we have only a set of unlabeled data m DT = {x˜j }j=1, where each instance x˜j is drawn i.i.d. from PT (X).
Our goal is to train a classifier f : X → Y , based on DS and DT and then
predict the corresponding labels of DT . Note that there are labels for the target
domain, but only used for testing.
DA has been studied in various fields, such as natural language processing for
sentiment analysis [
Most UDA researchers put emphasis on covariate shift setting, which assumes
that PS (X) is different from PT (X). Among these methods, we can roughly
divide them into two main approaches. One is the re-weighting method. The
goal of this kind of method is to estimate the importance weight PT (X)/PS (X)
for each source data. After obtaining the importance weights, they can further
do importance-weighted empirical risk minimization to adapt their model to the
target domain. Different methods estimate the importance weight differently. [
Another setting named label shift is assumed that PS (Y ) 6= PT (Y ). In this
setting, previous works mostly utilize the re-weighting method to solve the
problem. But different from covariate shift, they try to estimate the importance
weight PT (Y )/PS (Y ). The concept of kernel mean matching can spread to label
shift setting [
Motivated by a real-world application, we find that current methods cannot
successfully tackle this application which contains the properties from covariate
and label shift. Therefore, how to promote domain adaptation methods to handle
more general cases is essential. Recently, [
Commissioned by the Industrial Technology Research Institute (ITRI), we
initiated a research project on predicting compound-protein interaction (CPI), which
is a vital topic on drug discovery [
We plot the scatter diagram through t-SNE [
E,C D = 1 Xn [yiT log C(E(xi))] + 1 Xn log[D(E(xi))] + 1 Xm log[1 − D(E(x˜j ))] n i=1 n i=1 m j=1 where Lcls represents a cross-entropy loss for the source data and Ladv is the objective function for encoder and discriminator.
We also train a model on the source domain and directly test it on the target domain, which is called source-only. target-only means that we train the model on training target data then evaluate it on testing target data. Note that, we choose weighted accuracy as the evaluation criterion on Herb dataset because it is an imbalanced dataset.
In Figure 2, we notice that of DANN is worse than source-only. Confounding by the result, we dig deeper to analyze the property of this dataset. One possible reason is if we let PS (E(X)) = PT (E(X)), we can derive PS (Y ) = PT (Y ) based on covariate shift assumption. However, we find that the positive to negative ratio of the number of data is 2:1 in the source domain. In the target domain, the corresponding ratio is 1:4. This finding shows that the label distribution of the source domain is different from the one of the target domain, i.e., PS (Y ) 6= PT (Y ). In this circumstance, if we insist on aligning source and target distribution, we may have bad accuracy. Based on this result, we argue that current adversarial methods designed under covariate shift assumption cannot handle the situation where PS (Y ) is also not equal to PT (Y ). It makes the following assumptions. First, the label distribution changes from source to target (i.e. PS (Y ) 6= PT (Y )). Then it further assumes that the conditional feature distributions stay the same (PS (X|Y ) = PT (X|Y )). Recent works deal with this problem through re-weighting and do importance-weighted empirical risk minimization after getting the weights.
PT (X, Y ) Ex,y∼PT (X,Y ) `(y, h(x)) = Ex,y∼PS(X,Y ) PS (X, Y ) `(y, h(x))
PT (Y )
= Ex,y∼PS(X,Y ) PS (Y ) `(y, h(x))
= Ex,y∼PS(X,Y ) w(y)`(y, h(x)).
(1)
whereh stands for a classifier: x → {0, 1}K , ` represents the loss function: y ×
y → [
We take Regularized Learning under label Shifts (RLLS) [
From the previous discussion, we found that current domain adaptation methods could not cover all various dataset shift cases, e.g., our real-world dataset. We first formally construct the C2H dataset for this particular domain adaptation task. It comprises CheEMBL- and Herb represented as source and target domain respectively. The source domain includes 641,545 data, and the target domain contains 3,916 data. Specifically, both input and label distributions vary between source and target. 4
Based on the observations in section 3.1, if we stop at nothing to use adversarial training to align the source and target data distributions, we may finally get an unexpected bad performance on the target domain due to neglecting that there could also exist label shift at the same time. We illustrate the toy example in Figure 4 to demonstrate this finding. In Figure 4(a), if we use adversarial training to align two distributions and do not take PS (Y ) 6= PT (Y ) into consideration, we would probably have bad accuracy on target data. Figure 4(b) shows that when the embedding space has strong physical meaning, selecting the source data which is close to target data directly could get some benefit on classification. That is, we can regard the distance between two data as an similarity measurement and then accomplish domain adaptation through selection technique. We use a toy example to demonstrate our idea in the following section. 4.1
In this section, we demonstrate that a simple selection technique could accomplish UDA in Figure 4. Figure 5(a) depicts the source-only classifier. We can see that directly applying the source-only classifier to the target domain could have a bad performance. In Figure 5(b), choosing the source data which is close to target data and utilizing it could get a good classifier on the target domain, i.e., achieve domain adaptation. Therefore, when the distance can represents similarity, simple selection technique can improve the performance in UDA task.
Furthermore, we actually implicitly make the continuity assumption, i.e., the points which are close to each other are more likely to share the same label. If the assumption holds, we can achieve domain adaptation through selecting the target-like source data which is close to target data. We define the target-like source data as representatives in this paper. Based on the continuity assumption, we further propose two selection techniques to achieve domain adaptation. (a) adversarial training
(b) data selection The first method is based on K-Nearest Neighbor (KNN), a classic lazy algorithm. KNN takes euclidean distance as a similarity measurement and collaborates with the assumption that for any data point and its neighborhood must belong to the same class, i.e., continuity assumption. We feed the source data into KNN as training data first and then input all the target data to get the corresponding representatives. We let K = 1 for simplicity. The procedure can be formulated from a different perspective as rep = {sj }jm=1,
DS rep denote the representatives we choose. After gathering the represenwhere DS tatives, we can use them as a new source dataset to train a model and apply it to the target domain. However, HS could aggravate bad performance when there exist two problems. First, the continuity assumption could be wrong. For example, in high dimensional space like image space, directly take distance as a similarity measurement to select the representatives may be a catastrophe. In this kind of space, the data sparsity problem exists naturally. We may face that the distance does not represent a sort of similarity. Second, if the source data is noisy, choose the representatives by distance may bring a lot of biases into the model and thus hurt the performance. Thus, to overcome these two problems, we propose the second method called SS-β. The soft means we do not drop the rest of the source data after selecting the representatives. Instead, we add the following constraint into the minimization objective. Supposed we train a neural net N as a classifier with L layers, we add the following constraint on the k-th hidden layer j=1 min 1 Xm ||N k(sj ) − N k(DSrejp)||22.
f m Via this term, we enforce that the close data pair in original space must be close in embedding space. The overall objective can be i=1
j=1 n min 1 X `(N (xi), yi) + β N n 1 Xm ||N k(x˜j ) − N k(sj )||22, m where β is a hyperparameter to control the importance of this constraint and ` represents a cross entropy loss. 5
In this section, we evaluate our proposed methods on three parts: (i) C2H, (ii) Noisy C2H and (iii) Digits. For part (i), we want to show simple selection based methods can improve the performance in our C2H dataset. In part (ii), we test our methods in the noisy source domain and discuss what is the best circumstances for our methods to be used. To evaluate the scalability of our methods to high-dimension space, we do the experiments on digits dataset and show the results in part (iii)
We name our methods as follows: (1) HS: use the representatives selected by Hard Distance-Based Selection to train the model and then direct apply it to the target domain. (2) SS-β: train the model on the source domain and add the Soft Distance-Based Selection constraint which is controlled by the hyperparameter β to restrict the influence of this term. For each result, we repeat 5 times trials with different random seeds and show the average on the table. We also indicate the standard deviation to demonstrate the stability for each method. 5.1
We run the following methods as our competitors (i) KMM-γ [
Table 2 shows that HS has an improvement compared with source-only and other methods in this task. We can see that there is a big performance gap between DANN-like methods and ours. The original dataset already has interpretable and discriminative features. Therefore, aligning the distributions aggressively would lead to declining performance, not to mention label shift would deteriorate the performance too. fDANN and sDANN are expected to somewhat ease the impact of label shift by restricting the model not to align the distribution perfectly, but still have bad performance due to destroying the good feature embedding. In Table 2, we can see that re-weighting methods are competitive to HS. HS can basically be regarded as a re-weighting method that only assign the weight to the representatives and others assign 0 weight. However, HS is computational efficiency because we don’t need to calculate the kernel matrix that KMM should do. We just run the KNN algorithm. We can also see that our SS-β perform poorly because it suffers from difficult hyperparameter tuning.
Furthermore, we do the experiment to see whether accuracy under the different number of neighbors k would change. The results plot in Figure 8. We can find that under different k the accuracy has slightly different. Therefore, we do not have to worry about the parameter k when using our methods. We want to verify that SS-β could handle the situation where the representatives could be disruptive due to the noisy source data. Therefore, we create a noisy C2H dataset and try to choose the better method in this scenario. First, we add Gaussian noise with 0 mean and 0.01 variance into each feature dimension independently for every ChEMBL- data point, while Herb dataset remains the same.
The experiment results are listed in Table 3. From the table, we can see that HS perform poorly than SS-β. As expected, in the noisy source scenario, if we over-rely on the close source dataset selected by HS, we would suffer from the impact of noisy data. In this circumstance, choose SS-β can mitigate the noisy data effect by careful hyperparameter tuning.
Briefly, if we know in advance that the data has strong physical meaning in
your task, use the hard version would gain much more benefit without the effort
for tuning the parameter. On the contrary, in the task where source data could
have some noises, choose soft version selection and coupe with careful parameter
search can avoid over-confidence on the fake representative.
To extend to a more severe shift scenario, we follow the procedure of previous
work [
For task (i), the results are listed in Table 4. From the table, we can discover
that fDANN outperforms other methods on the severe shift setting (i.e. [
For task (ii), Table 5 shows that fDANN still does well in severe shift settings.
However, to our surprise, SS-1 has a great improvement on [
Even though our methods perform well only on [
Table 6 and Table 7 show the results. We can see that our method well generalizes to the target domain, under the feature embedding generating by the extractor method. Using the features generated by PCA to run our methods has bad performance on each task. This result shows PCA lets the target data lose a lot of important information. The ImageNet method performs poorly, either. Because it was trained on a non-digits dataset, the model can not extract the features which are important for digits classification. Motivated by the real-world bio-chemistry application, we indicate the problem that covariate shift and label shift could exist at the same time. We propose HS and SS-β which can handle this situation while other recent UDA methods would suffer from. We also extend our methods to image space which is highdimensional. Our methods are mainly based on the similarity, that is, how to get a feature space with strong physical meaning would be a big problem. A possible extension of this work is regarding our methods as a complement for current domain adaptation methods.