Domain adaptation is one of the most crucial techniques to mitigate the domain shift problem, which exists when transferring knowledge from an abundant labeled sourced domain to a target domain with few or no labels. Partial domain adaptation addresses the scenario when target categories are only a subset of source categories. In this paper, to enable the e cient representation of cross-domain plant images, we rst extract deep features from pre-trained models and then develop adversarial consistent learning (ACL) in a uni ed deep architecture for partial domain adaptation. It consists of source domain classi cation loss, adversarial learning loss, and feature consistency loss. Adversarial learning loss can maintain domain-invariant features between the source and target domains. Moreover, feature consistency loss can preserve the ne-grained feature transition between two domains. We also nd the shared categories of two domains via down-weighting the irrelevant categories in the source domain. Experimental results demonstrate that training features from NASNetLarge model with proposed ACL architecture yields promising results on the PlantCLEF 2020 Challenge.
Automated plant identi cation is important in recognizing plant species. The
availability of massive labeled training data is a prerequisite of machine learning
models. Unfortunately, such a requirement cannot be met in the plant identi
cation problem since we have sparse labels for real-world plant images. Therefore,
we propose to transfer knowledge from an existing auxiliary labeled herbarium
domain to the eld photo domain with limited or no labels. However, due to
the phenomenon of data bias or domain shift [
Domain adaptation (DA) has been proposed to leverage knowledge from an
abundant labeled source domain to learn an e ective predictor for the target
domain with few or no labels, while mitigating the domain shift problem [
Recently, deep neural network methods have been widely used in the domain
adaptation problem. Notably, adversarial learning shows its power in embedding
in deep neural networks to learn feature representations to minimize the
discrepancy between the source and target domains [
Although many methods are proposed for partial domain adaptation, they still su er from two challenges: (1) the models are evaluated on small datasets, while it has lower transferability to the large-scale dataset, and (2) the feature consistency of two domains is inappropriately ignored.
To address the aforementioned challenges, we aggregate three di erent loss functions in one framework: source domain classi cation loss, adversarial learning loss, and feature consistency loss to reduce the discrepancy of the two domains. Moreover, our model is evaluated on a large-scale plant identi cation dataset to improve the estimate of the generalization ability of our model.
Our contributions are three-fold: 1. We propose a novel adversarial consistent learning network (ACL) for PDA, to adversarially minimize the domain discrepancy of the source and target domains and maintain domain-invariant features; 2. The proposed adversarial learning loss and feature consistency loss can distinguish the target domain from the source domain, and preserve the negrained feature transition between the two domains; 3. We impose shared category selection to lter out the irrelevant categories in the source domain. By down-weighting the irrelevant categories in the source domain, we can reduce negative transfer from the source domain to the target domain.
Experimental results show that ACL achieves higher classi cation accuracy than several baseline methods and yields promising results on the PlantCLEF 2020 Challenge. 2
PlantCLEF 2020 is a large-scale dataset of the PlantCLEF 2020 task [
We exclude the classId of \108335" in the photo domain since the major classes are from the herbarium domain. In addition, herbarium domain does not contain the \108335" category. Therefore, eight images are excluded in the photo domain. The statistics of the PlantCLEF 2020 dataset are listed in Tab. 1. and @@LCCoonn ) from class label classi er, domain label predictor and feature consistency regressor. The ACL model consists of three di erent loss functions (source classi cation loss LS , adversarial domain loss LA, and feature consistency loss LCon). The feature extractor G in the shared layers is used for both classi er f and domain discriminator D (The blue dash lines are the backward gradients, and GRL stands for gradient reversal layer). Layers visualization of architecture is shown in Fig. 3. 3 3.1
Previous partial domain adaptation methods [
In this paper, we present our approach: adversarial consistent learning (ACL) on partial domain adaptation. It can align the feature distribution of the source and target domains in the shared categories and guarantee feature consistency across the two domains. Importantly, ACL identi es irrelevant source categories via down-weighting class importance automatically. Evaluation on the large-scale PlantCLEF 2020 challenge dataset shows a high generalizability of our model. 3.2
For unsupervised domain adaptation, given a source domain DS = fXS i; YS igiN=S1 of NS labeled samples across the set of categories CS and a target domain DT = fXT j gjN=T1 of NT samples without any labels (YT is unknown) across the set of categories CT . For partial domain adaptation, the number of categories in CT is less than the number of categories in CS , and CT $ CS . The samples XS and XT obey the marginal distributions of PS and PT . The conditional distributions of two domains are denoted as QS and QT . Due to the discrepancy of two domains, the distributions are assumed to be di erent, i.e., PS 6= PT and QS 6= QT . Our ultimate goal is to learn a classi er f under a feature extractor G, which selects shared categories between two domains, and ensures lower generalization error in the target domain. 3.3
To circumvent the large computation resource requirement of training
largescale PlantCLEF 2020 challenge datasets, we instead focus on deep features
from pre-trained models. Based on Zhang and Davison [
The task in the source domain is trained using the typical cross-entropy loss in following equation:
LS (f (G( (XS ))); YS ) =
NS i=1 c=1 1 XNS XCS YS iclog(f (G( (XS i)))); (1) where YS ic 2 [0; 1]CS is the probability of each class of ground truth for the ith element of S, f is the classi er in Fig. 2, and f (G( (XS i))) is the predicted probability. 3.5
In general adversarial learning, the system learns a mapping from the source domain to the target domain. Given the feature representation of feature extractor G, we can learn a discriminator D, which can distinguish the two domains using the following loss function:
LA(GXS XT ; G( (XS )); G( (XT ))) = 1 XNS log(1 NS i=1 However, Eq. 2 only guarantees source domain data will be close to the target data (GXS XT ), and it does not ensure that the target data will be close to the source data. We hence introduce another mapping from the target domain to the source domain GXT XS in Eq. 3 and train it with the same adversarial loss as in GXS XT as shown in Eq. 2.
LA(GXT XS ; G( (XS )); G( (XT ))) (3) For GXS XT , the source domain has the label of 0 and the target domain has the label of 1, which is corresponding to Domain Label 1 in Fig. 2. Meanwhile, for GXT XS , 1 is the new label for the source domain and and 0 is the new label for target domain, which is corresponding to Domain Label 2 in Fig. 2. Therefore, we de ne the adversarial learning loss as:
LA(G( (XS )); G( (XT ))) = LA(GXS XT ; G( (XS )); G( (XT ))) + LA(GXT XS ; G( (XS )); G( (XT ))): 3.6
To encourage the source domain and target domain information to be preserved during adversarial learning, we propose a feature consistency loss in our model. Details of the feature reconstruction layers are shown in Fig. 2; the reconstructed layers are right behind the feature extractor G in the shared layers, and they aim to reconstruct the extracted features and maintain the invariant features during the conversion process. The feature consistency loss is de ned as: LCon(GXS XT ; GXT XS ; G( (XS )); G( (XT ))) = Exs G( (XS))[`(GXT XS (GXS XT (xs)) + Ext G( (XT ))[`(GXS XT (GXT XS (xt)) xs)] xt)]; where ` is the mean squared error loss function, which calculates the di erence between true features and the reconstructed features. 3.7
In PDA, the set of target domain labels is a subset of the source domain labels, i.e., CT $ CS . In the PlantCLEF challenge, the size of irrelevant label set (CS CT ) is far larger than the size of CT (jCS CT j >> jCT j). If we use all elements of the source domain distribution to match the target domain distribution, it will cause negative transfer since the target domain will also be forced to match the irrelevant labels (CS CT ). Therefore, it is important to identify the shared categories between source and target domains.
To address the aforementioned challenge, we re-weight the source domain label set via reducing the irrelevant label set. During the training, we can get ^ the predicted probability of the target domain: YT j = f (G( (XT j ))), which gives a probability of each source label in CS . As we know, the set of irrelevant source labels and target label set are disjoint, and the target data are signi cantly dissimilar to the source data in the irrelevant label set. Therefore, the probability (4) (5) of irrelevant categories should be su ciently small and can be ignored. We then de ned the weight vector as:
W =
NT j=1 1 XNT Y^T j ; (6) where W is a jCS j-dimensional weight vector. The irrelevant categories (CS CT ) will have a much smaller weight than the shared categories. We then assign the weight as 0 if its element Wc is less than a su ciently small number (e.g., 10e 9). By reducing the weight of irrelevant categories, the shared categories can be emphasized and negative transfer will be mitigated. The weight vector W is applied in both the source classi er and domain discriminator over the source domain data as shown in the following objective function. 3.8
We combine the three aforementioned loss functions to formalize our objective function:
L(XS ; XT ; YS ; GXS XT ; GXT XS ) = LS (f (W(G( (XS )))); YS ) + +
LA(W(G( (XS ))); G( (XT ))) (7)
LCon(GXS XT ; GXT XS ; W(G( (XS ))); G( (XT ))); where and are tradeo parameters between di erent loss functions. Our model ultimately minimizes the di erence during the transition from the source domain to target domain and from the target domain to the source domain. Meanwhile, it maximizes the ability to distinguish the two domains. 3.9
The shared layers consist of the feature extractor G and the feature reconstruction layers. In G, there are two dense layers, a \Relu" activation layer, and a dropout layer. The numbers of units of the dense layer are 1000 and 997, respectively. The rate of the Dropout layer is 0.5. The feature reconstruction layers have a \Relu" activation layer, a dropout layer and a dense layer with the number of units of 1000. The shared layers are jointly optimized by both the source classi cation loss, adversarial domain loss and feature consistency loss.
Let FE ( ; E ) be the output of the shared encoder with parameters of E . In addition, let FS ( ; S ) be the output of class label classi er with parameters of S , FA( ; A) be the output of domain label predictor with parameters of A, and FCon( ; Con) be the output of feature consistency regressor with parameters of Con. Therefore, the shared layers are optimized by these three gradients. The parameters in the shared layers are updated in the following equation: S
S
E
E
where is the learning rate and
layer (GRL) in [
is the adaptation factor from gradient reversal (10) (11) We now formalize the error bound of our model. ACL model is trained with both the labeled source domain and the unlabeled target domain. The error bound of the source domain and the target domain ( (h)) in our model is then formally written as: (h) =
XS (h; YS ) +
XT (h; Y^T ); (9) where Y^T is the predicted label of target domain. The term XS (h; YS ) = Ex XS [jh(x) YS j] and XT (h; Y^T ) = Ex XT [jh(x) Y^T j] denote the expected risk over the source domain and the target domain with respect to the ground truth labels and predicted labels, respectively (where j j is the L1 norm).
During the training, we expect the error XT (h; Y^T )) to be close to XT (h; YT ),
which evaluates the classi er f with true target domain labels. The smaller the
di erence between these two errors, the better the model performs and more
discrepancies of the two domains are reduced. Existing domain adaptation
theory shows that the risk in the target domain can be minimized by bounding the
source risk and discrepancy between source and target domains [
Lemma 1 Let h be a hypothesis in a class H. Then (h) =
XS (h; YS ) +
XT (h; Y^T ) 2 XS (h) + dH(DS ; DT ) + C; where dH(DS ; DT ) = 2 suph;h02H j XS (h; h0) XT (h; h0)j is the H-divergence of training and test data in the hypothesis space H: C = XS (h ; YS ) + XT (h ; YT ) is the adaptability to quantify the error in ideal hypothesis h space of training and test data, which should be small and is the optimal hypothesis via minimizing the joint error in Eq. 11.
h = arg min XS (h; YS ) +
XT (h; YT )
In Lemma 1, the generalization boundary of our model consists of three terms: training data error, data discrepancy dH(DS ; DT ), which is estimated by the disagreement of hypothesis in the space H, and the adaptability C of the ideal joint hypothesis. In ACL model, the rst term is measured by Eq. 1. The domain discrepancy is assessed by adversarial learning loss and feature consistency loss. Furthermore, ACL nds the ideal hypothesis and reduces the training error in each iteration. Hence, our model can nd a minimal boundary for two domains. In other words, ACL can implicitly minimize the target domain risk, domain discrepancy, and the adaptability of true hypothesis h in terms of the hypothesis space H:
As aforementioned, the deep features are extracted from the last fully
connected layer [
The parameters of ACL are rst tuned based on the performance of the domain P, while the model is trained with H + A domain. We then apply these parameters to domain T and submit it to the challenge for the evaluation. Our implementation is based on Keras. The parameters settings are = = 0:5, = 0:31, learning rate: = 0:0001, batch size = 128, the number of iterations is 1000 and the optimizer is Adam. The details of the layers are shown in Fig. 3.
We also compare our results with two domain adaptation methods: DANN [
In the nal stage of the PlantCLEF 2020 Challenge, our solutions are evaluated by the organizers using the test domain data. As shown in Tab. 3, our method achieved mean reciprocal rank (MRR) of 0.032 in the whole test domain, and MRR of 0.016 in the subset of the test domain, and our method places 4th in the contest. 5
There are two compelling advantages of the ACL model. First, we consider the adversarial consistent learning paradigm, which maintains the domain-invariant features from the source domain to the target domain and vice versa. Secondly, we reduce the weight of irrelevant categories in the source domain, which eliminates the negative transfer during the training. Although the performance of our model is better than several baseline methods, the highest accuracy of the photo domain is less than 10%, which illustrates that the transfer learning ability in the real world image is lower. One underlying reason is that PlantCLEF 2020 Challenge has di cult datasets|that there are signi cant di erences between herbarium domain and photo domain, as shown in Fig. 1. Another reason is caused by the weakness of our model since we only train deep features instead of raw images to reduce the computational requirements; some features might be ignored during the training. The performance of the ACL model could be improved if we train the architecture with raw images. 6
In this paper, we propose an adversarial consistent learning network on partial domain adaptation termed (ACL) to overcome limitations in nding proper shared categories and guaranteeing the feature consistency of two domains. Our model is optimized via minimizing a three-component loss function. As each component of our ACL model, explicit domain-invariant features are maintained through such a cross-domain training scheme. Experimental results demonstrate our proposed ACL model yields promising results on the PlantCLEF 2020 Challenge.