=Paper=
{{Paper
|id=Vol-1180/CLEF2014wn-Image-ChidlovskiiEt2014
|storemode=property
|title=Assembling Heterogeneous Domain Adaptation Methods for Image Classification
|pdfUrl=https://ceur-ws.org/Vol-1180/CLEF2014wn-Image-ChidlovskiiEt2014.pdf
|volume=Vol-1180
|dblpUrl=https://dblp.org/rec/conf/clef/ChidlovskiiCG14
}}
==Assembling Heterogeneous Domain Adaptation Methods for Image Classification==
https://ceur-ws.org/Vol-1180/CLEF2014wn-Image-ChidlovskiiEt2014.pdf
Assembling Heterogeneous Domain Adaptation
Methods for Image Classification
Boris Chidlovskii, Gabriela Csurka and Shalini Gangwar
Xerox Research Centre Europe,
6 chemin Maupertuis, Meylan, France
Abstract. In this paper we report the contribution of XRCE team to the Domain
Adaptation Challenge [10] organized in the framework of ImageCLEF 2014 com-
petition [9]. We describe our approach to build an image classification system
when a weak image annotation in the target domain is compensated by mas-
sively annotated images in source domains. One method is based using several
heterogeneous methods for the domain adaptation aimed at the late fusion of the
individual predictions. One big class of domain adaptation methods addresses a
selective reuse of instances from source domains for target domain. We adopt
from this class the adaptive boosting for weighting source instances which learns
a combination of weak classifiers in the target domain. Another class of methods
aims to transform both target and source domains in a common space. In this class
we focused on metric learning approaches aimed at reducing distances between
images from the same class and to increase distances of different classes indepen-
dently if they are from source or target domain. Combined the above approaches
with a ”brute-force” SVM-based approach we obtain a set of heterogeneous clas-
sifiers for class prediction of target instances. In order to improve the overall
accuracy, we combine individual classifiers through different versions of major-
ity voting. We describe different series of experiments including those submitted
for the official competition and analyze their results.
1 Introduction
The shortage of labeled data is a fundamental problem in machine learning applica-
tions. While huge amounts of unlabeled data is generated and made available in many
domains, the cost of acquiring data labels remains high. Domain adaptation addresses
this problem by leveraging labeled data in one or more related domains, often referred
as ”source” domains, when learning a classifier for unseen data in a ”target” domain.
The domains are assumed to be related but not identical.
This situation occurs in domains where machine learning components are inten-
sively deployed, such as event detection in videos, entity recognition across different
text corpora, object recognition in images acquired in different conditions (see [21] for
a survey of domain adaptation methods and [24] for a survey on the related field of
transfer learning).
Domain adaptation has also received a significant attention in computer vision ap-
plications [17,18,19,20,22,26,27], Different aspects have been addressed and various
approaches to the domain adaptation have been proposed [7,25]; this clearly indicates
how complex and multi-faceted the problem is.
448
1.1 Domain adaptation challenge
Current research in domain adaptation focuses on a scenario where (a) the prior do-
main (source) consists of one or maximum two databases (b) the labels between the
source and the target domain are the same, and (c) the number of annotated training
data for the target domain are limited. The goal of this challenge is to push the state of
the art towards more realistic settings, relaxing these assumptions1 .
Indeed researchers and teams participating in the ImageCLEF DA 2014 challenge
were asked to build recognition systems for the target classes by leveraging the knowl-
edge from four source domains. Both source and target data are provided by exploiting
existing available resources.
Specifically, the participants were provided with image features2 extracted by the
organizers from 600 randomly selected images collected from five different image col-
lections: Caltech-256 [2], ImageNet12 [3], PASCAL-VOC12 [4], Bing [1] and SUN [5].
The organizers selected 12 common classes from each datasets, namely, aeroplane,
bike, bird, boat, bottle, bus, car, dog,horse, monitor, motorbike, people. (see Figure 1
for example images for each class and in all collection, showing the variability of classes
between the collections). The first four collections from the list are proposed as source
domains for which the image features and all the labels were provided. The SUN dataset
served as the target domain, with 60 annotated and 600 non-annotated instances. The
task was to provide predictions for the non-annotated target data.
2 XRCE approach to domain adaptation
We started by analyzing the domain adaptation task and the available features for the
different collections. Three key elements of our analysis are the following:
1. Original images are not available neither for target nor for sources domains; the
participants dispose feature sets only. It makes impossible to leverage the interme-
diate knowledge generated during the feature extraction process, such as the choice
of low or high level features or the vocabulary for bags of visual words.
2. Source and target domains are semantically related, however they are different
feature-wise. Figure 2.left compares the PCA projection of four source and tar-
get domains made available at phase 1 of the challenge3 . A similar relationship
(see Figure 2.middle) can be found between PCA projections for the source and
target domain data at phase 2. Worse, as shown in Figure 2.right, the target feature
distribution has been changed between phases 1 and 2. This change is likely due to
1
From http://www.imageclef.org/2014/adaptation.
2
These image features were the concatenation of four bag-of-visual words [11] built on a 2x2
split of the image where the low level features were SIFT descriptors extracted densely from
the corresponding image regions.
3
At the beginning of the challenge, the participants were provided with a similar problem con-
figuration but different feature sets in order to familiarize with the problem. In addition all
labels for the target set were also available in order to allow participants to evaluate their
methods. We will refer to this setting as phase 1. The features for the actual training for the
submission were released in phase 2.
449
Fig. 1. Examples of 12 classes from four source domains (Caletch, InageNet, Pascal, Bing) and
the target domain (SUN).
parameter changes in the feature extraction process hidden from the participants.
Consequently, it made impossible any deployment of models learned at phase 1.
3. According to our experiments made with the features released in phase 1, none
of existing domain adaptation methods for the visual classification appeared as a
perfect match for the challenge task [7,25].
Putting all these elements together, we decided to proceed in two steps. First, we under-
took a number of conceptually uncorrelated approaches to domain adaptation. Second,
we used ensemble techniques to aggregate individual predictions using different major-
ity voting methods.
More precisely, in the first step we tested methods of the following classes:
Brute force : Cross validation on available training data to identify best combinations
of source domains for transferring to the target domain, with the optimal set of
parameter values and kernel functions.
Instance weighting : Boosting-based instance-transfer domain adaptation to identify
different instances in one or more source domains allowing to boost the learning in
the target domain.
Space transformation : Metric learning-based domain adaptation where the idea is to
transform the feature space and bring instances from the same class close to each
other, independently if they are from source or target domains.
The remainder of this working note is organized as follows. In Section 3 we de-
scribe three individual domain adaptation methods. Section 4 describes the ensemble
450
Fig. 2. PCA projections for source and target domains. a) Target and source data at phase 1; b)
Target and source data at phase 2; 3) Target data at phases 1 and 2.
methods over heterogeneous classifiers. Section 5 reports evaluation results including
the ten submissions to the challenge. We analyze the results and the impact of different
components on the overall performance. Finally we outline future work and conclude
in Section 6.
3 Individual domain adaptation methods
There are two main cases in domain adaptation, they depend on available data in the
target domain. In the unsupervised case, no labeled target instances are available. In the
semi-supervised case, a few target instances are labeled. In ImageCLEF DA challenge,
60 labeled target instances are available, this positions the challenge into the second
case.
We suppose that the labeled target dataset Tl is drawn from the same distribution
as the unlabeled target set Tu and hence it will play an important role in building the
classification model. However the size of Tl is too small to adequately train a good
classifier for the test data Tu .
The training data from source domains are more abundant, but the classifiers learned
from these data cannot classify the test data well due to different data distributions. Let
S1 , . . . , SNS denote the source domains, where NS ≥ 1 is the number of sources. Let
Xsk be an instance space of Sk (e.g. a subset of Sk used for training). We denote by
Xt the instance space of the target domain (that can take elements from Tl , Tu or both).
Further, we denote by Y = {c1 , c2 , . . . , cNc } the set of category labels common to all
domains.
3.1 Brute force
We start off with the brute force approach which uses available annotated data to assess
the relatedness between source domains and the target domain and to test a straight-
forward domain adaptation scenario. For the NS = 4 different source domains we
consider NSC = 2N S − 1 = 15 source combinations SCi , i = 1, . . . , NSC which are
generated by an exhaustive enumeration of all possible subsets of source domains, e.g.
451
SC1 = {S1 }, SC6 = {S1 , S3 } and SC15 = {S1 , S2 , S3 , S4 }. For each source com-
bination SCj , we concatenate the target train set Tl with the selected sources SCj and
train the SVM models in a cross validation mode to find out the optimal parameter val-
ues and kernel functions. Two versions have been implemented, the multi-task SVM
and one-against-all binary SVMs, as follows:
Multi-class SVM: We first used the multi-class LIBSVM package4 for the multi-class
classification in the target domain. In the cross validation with k = 10 folds, we
tested different values of SVM parameters µ, ν, C, the standard kernels available
with the package and different source combinations SCi we found as the best
source combination CS13 = {S1 , S2 , S4 } with the linear ν-SVM and the parame-
ters ν = 0.12 and C = 0.01. This classifier denoted by fmcsvm allows a gain of 3%
compared to the baseline multi-class SVM fbf 0 with no source domains and cross
validated parameters.
Binarised SVM: We additionally consider the multi-class classification by a compo-
sition of one-against-all classifiers running again the LIBSVM package and cross
validate to determine the optimal parameters for each of Nc = 12 classes in the
datasets. Unlike the multi-class SVM above where the optimal parameter values,
kernel and the source combination are common for all classes, in the binarized
version we identify the best classifier for each class cj ∈ Y which comes with a
specific set of parameter values, kernels and source combinations for each class
ci . Hence, the multi-class classifier fbsvm is composed of Nc binary classifiers
cj
fbsvm , j = 1, . . . , Nc . For an unseen sample xi , it proceeds by applying all clas-
sifiers and predicting the label ŷbsvm = cj for which the corresponding classifier
reports the highest confidence score:
jc
ŷbsvm = argmax fbsvm (xi ).
cj ∈Y
In a probabilistic setting, all class confidence scores are converted into probabilities
P (yi = c|fbsvm (xi )).
3.2 Instance Transfer with Adaboost
We extended the Transfer AdaBoost learning algorithm (TrAdaboost) [12], an extension
of the AdaBoost [15] for transfer learning by assuming that there is abundant source
training data to learn a classifier, but the target domain is different from the source.
Hence this approach is can be easily adopted for the domain adaptation.
AdaBoost aims to boost the accuracy of a weak learner by carefully adjusting the
weights of training instances and to learn a classifier accordingly. In TrAdaboost, source
and target instances have opposite roles. Target training instances are weighted similarly
as in AdaBoost, but when source training instances are wrongly predicted by the learned
model due to distribution changes, it is assumed that they could be those that are the
most dissimilar to the target instances and therefore TrAdaboost tries to decrease the
weights of these instances in order to weaken their impact.
4
http://ww.csie.ntu.edu.tw/ cjlin/libsvm/.
452
Algorithm 1 Transfer Adaptive Boosting with one source domain.
Require: Target training set Tt = (Xt , Y ) of size Nt ; source training set Ts = (Xs , Y ), of size
Ns ; a base learning algorithm Learner; the maximum number of iterations M .
Ensure: Target learner f : Xt → Y .
1: Initialize the initial source and target weight vectors, wT1 = (wt11 , . . . , wt1Nt ), wS1 =
(ws11 , . . . , ws1Ns ),
p
2: Set w = (wT , wS ), β = 1/(1 + 2 ln Nt /M ) and T = (Tt , Ts ).
3: for r = 1, . . . , M do
4: Normalize wr = wr /|wr |.
5: Call Learner on the training set T with the distribution wr to find a hypothesis fr : X →
Y which minimizes error for (T, wr ).
Pn r t
6: Calculate the error of hr on Tt : r = min 21 , PNt1 r i=1 w ti
· [[f r (x i ) 6
= yi ]] .
i=1 wti
7: Set β r = 1/2 log((1 − r )/r ) .
8: Set Γ r = 2(1 − r ).
9: Update the weight vectors:
wsr+1
j
= Γ r wsrj exp(−β [[fr (xsj ) 6= yj ]]), xsj ∈ Xs ,
r+1
wti = wtri exp(2β r [[fr (xti ) 6= yi ]]), xti ∈ Xt .
10: end for P
M r
11: Output the aggregated estimate ftra (x) = r=1 β fr (x) . In a probabilistic setting, these
scores are converted into probabilities P (yi = c|ftra (xi )).
A formal description of the framework is given in Algorithm 1 for the case of using
one source domain, NS = 1, but the extension to multiple sources is straightforward.
The main idea is that at each iteration round, if a source training instance is mistak-
enly predicted, the instance may likely conflict with the target training data. Therefore,
we decrease its training weight to reduce its effect through multiplying its weight by a
strictly positive factor exp(−β[[fr (xi ) 6= yi ]]) ≤ 1, where [[·]] is the Iverson brackets de-
noting the indicator function that equals one if its argument is true and zero otherwise.
We further add Γ r term inspired by [6] to address some drawbacks of the initial TrAd-
aboost [12]. Due to this, in the next round, the mis-classified source training instances,
which are dissimilar to the target ones, will affect the learning process less than in the
current round. After several iterations, the source training instances that fit the target
ones better will have larger training weights, while the source training instances that
are dissimilar to the target ones will have much lower weights. These instances with
large training weights tend to help the learning algorithm to train better classifiers.
As in the previous section, we consider NSC =15 different compositions of available
sources to be used as one. We use Algorithm 1 on different combinations by simply
concatenating the sources to form the source training set Xs . Note the existence of an
extension of TrAdaboost to cope with multiple source domains [29]. This extension
probes at each iteration instances from different sources, in order to identify a source
which reduces the error the most. In the context of the ImageClef DA challenge, our ex-
perience of using TrAdaboost with multiple sources has not been convincing. Concate-
nating sources was more beneficial that keeping them separately. This phenomenon can
453
be explained by the closeness between the source instances proposed for the challenge
(see Figure 2). In the cases where source domains have very different distributions,
keeping them separately might be a better strategy.
3.3 Metric learning methods
Another class of domain adaptation methods tries to jointly transform the source and
target domains, either by trying to align domains using PCA projections [18,16,14]
or by learning a metric which can bridge the two domains. For the challenge we ex-
perimented with two type of metric learning approaches. On one hand we extended a
metric learning technique that optimizes KNN objectives [13,28] that we will denote
by KNN ML. On the other hand, we adopted for domain adaptation the Nearest Class
Mean (NCM) and Nearest Class Multiple Centroids (NCMC) classifiers from [23].
Metric learning with KNN objectives. The aim of these approaches is to find a lin-
ear transformation W such that, in the new space, distances between examples from
the same class are decreased and distances between images from different classes are
increased. This is done generally by optimizing the sum of the losses on the training
set5 :
Lqpn = 1 + dW (xq , xr ) − dW (xq , xn ) + = 1 + 2x> >
q W W (xn − xp ) + (1)
where (xq , xp , xn ) are triplets such as the image xp is from the same class as xq while
xn is from a different class, dW (xq , xi ) =k W xq − W xi k22 is the Euclidean distance
between image xq and image xi in the transformed space and [a]+ denotes max(0, a).
We consider only projections into a lower dimensional space, i.e. the dimension of
the projected vector W x is lower than the dimension of x (in our experiments we used
target dimensions of 64 and 128). One advantage of this is to have less parameters to
estimate, which is especially important when only relatively small amount of training
examples are available; also it generally leads to better performances.
As optimizing the sum of losses over all possible triplets is typically unfeasible, we
approximate the solution through stochastic gradient descent (SGD) [8] method, where
at each step a set of random triplets are selected and W is updated with a fix learning
rate and the sub-gradients:
>
∇W Lqpn = 2 [[Lqpn > 0]] W Xqnp + Xqnp (2)
where Xqnp = xq (xn − xp )> . If instead of updating only by a single triplet, for each
xq we select a set of random positives-negative pair {(xpi , xni ), i = 1..m}, we have
>
the same updating formula but Xqnp = Xq Xnp , where Xnp is the matrix concatenating
the (xni − xpi ) vectors, and Xq is the matrix concatenating m times the xq vector.
5
The second equality is true if we normalize our features to have L2 norm in the original space,
what we have done in all our experiments
454
Metric Learning for Nearest Class Mean. The nearest class mean NCM classifier
assigns a document to the class c∗ ∈ Y with the closest mean:
c∗ = argmin dW (x, µc ) where dW (x, µc ) =k W x − W µc k22 , (3)
c∈Y
where dW (x, µc ) is the Euclidean distance between an image x and the class mean µc
in the transformed space.
The main idea behind the metric learning for Nearest Class Mean (MLNCM) is
to learn a linear projection matrix W such that, in this new space, the instances of
the same class are closer to each other, and hence to the correct class mean than to
the class centers of the other classes. To optimize such NCM performance objective
in the projected space, Mensink et al. [23] proposed to formulate the NCM classifier
as a multi-class soft-max regression problem using a probabilistic model where the
probability for a class c given a feature vector xi is defined as:
exp − .5dW (xi , µc )
P (yi = c|xi ) = P . (4)
c0 ∈Y exp − .5dW (xi , µc0 )
Then, to learn the projection matrix W , the log-likelihood of the correct predictions
are optimized over the training set using the stochastic gradient descend (SGD) algo-
rithms [8]. At each step, W is updated with a fixed learning rate in the direction given
by the gradient:
1 XX
P (yi = c|xi ) − [[yi = c]] · W · (xi − µc )(xi − µc )> ,
∇W L = (5)
N i
c∈Y
where N is the number of training examples. At each iteration we are using only a
random subset of the training data to update W .
In [23], Mensink et al. proposed an extension to this method, the Nearest Class
Multiple Centroids (NCMC) classifier that allows more flexible class representations
by considering multiple cluster means (centroids) for each class. In their case, NCMC
represents each class by a set of centroids instead of a single class mean. In our multiple
domain case, instead of cluster means, we consider domain specific class means µdc
(averaging over instances from the same class c and same domain d) and we assign a
test instance to a given class based on a weighted soft-max distance to these domain
specific class means:
Nd
1 X
P (yi = c|xi , µdc ) = wd exp − .5dW (xi , µdc )
(6)
Z
d=1
P P d
where Nd is the number of domains and Z = c d wd exp − .5dW (xi , µc ) is
the normalizer. Adding domain specific weights wd in Eq. 6 allows giving different
importance to each domain (in our experiments we used wt = 2 for the target domain
Tl and wsk = 1 for the source domains).
455
To learn the projection matrix W , we maximize the log-likelihood of correct classi-
fication in Eq. 6, for which the gradient w.r.t. W is given by:
1 X p(µcd |xi )
∇W L = p(µcd |xi ) − [[c = yi ]] P d0
W (xi − µdc )(xi − µdc )> ,
N
i,c,d d0 p(µc |xi )
(7)
where p(µcd |xi ) = wZd exp − .5dW (xi , µcd ) . Again, we use Stochastic Gradient De-
scent [8] with fixed learning rate where at each iteration we sample a random subset
from the training set (2 ∗ NC was used in the experiments). We denote this learning
method by NCMC ML.
Learning strategies. Let us now suppose that we have the labeled target dataset Tl ,
the unlabeled target data set Tu , and the source training set (that can contain one or
several source domains S1 , . . . , SNs ). We experimented with the above mentioned met-
ric learning approaches using 3 different strategies and different source configurations
SCi .
1. In the first case we simply merged the source combination SCi with the target train-
ing data Tl and directly optimized the loss in Eq. 2, Eq. 5 or Eq. 7 using all training
examples. The rationale behind this is the following. By decreasing interclass dis-
tances independently from the domains, allows to exploit more efficiently labeled
images from source domains in the projected space to classify target examples and
hence it yield to increased performance of distance based classifiers. The meth-
ods from this first strategy are denoted by KNN ML, MLNCM ML respectively
MLNCMC ML.
2. As we have much less labeled target examples than source ones, their presence is
much rearer in the above mentioned random sampling processes. Hence in the it-
erative approaches they will have less influence in the learning process than the
much larger amount of source images. Therefore as a second strategy, we propose
to refine W obtained with the first strategy such that we impose at each iteration
to have in the sample examples from Tl . In the case of KNN, this is done by con-
sidering at each step xtq ∈ Tl and sampling randomly a set of randomly sampled
positive/negative image pairs from the source and then updating W using the triplet
sets (xtq , xrj , xnj ), j = 1..Nc . In the case of NCM and NCMC we simply ensure
that at least one target image from Tl is present in the selected batch. The meth-
ods corresponding to this second strategy are denoted KNN MLt, NCM MLt and
NCMC MLt respectively.
3. Finally, inspired by the adaptive learning proposed in [27], we consider an itera-
tive learning strategy where we further adjust the learned metric with the second
strategy. To do this, at each step and for each class we add target instances from
the unlabeled set Tu and remove source instances as follows. For each class c we
add the target instance xti for which P (yi = c∗i |xti ) − P (yi = c‡i |xti ) according
to Eq. 4 is the largest difference, where c∗i = c is the predicted label of xti and c‡i
is the second predicted label of xti . Also, for each class c we remove the source
image xsj from SCi for which c∗j = c and P (yj = c∗j |xsj ) − P (yj = c‡j |xsj ) is the
456
lowest difference. Then we refine W using the updated training set. We do several
rounds until the stopping criteria is reached6 . The methods corresponding to the
third strategies are denoted by KNN MLDA, NCM MLDA and NCMC MLDA
respectively.
Classification strategies. In the projected space we have different options to predict the
class labels of the target images. As we learned a metric, we can use any distance based
classifiers such as KNN, NCM or NCMC based classifications as they do not require
further parameter tuning. The only parameter k of the KNN can be fixed (e.g. we used
3 in all experiments) or cross validated. NCM has no parameter and the parameters of
the NCMD are the weights wd that can be fixed or tuned. In our experiments we used
fixed weights. Note that we experimented also with training Multi-Class SVM in the
projected space, but the performance was similar to or below the results obtained in the
original space. As our experiments on the dataset provided in phase 17 have shown that
NCMC in general performs better than KNN and NCM (up to 3-5%) we only retained
the NCMC classifier to compute the class probabilities (using Eq. 6) independently
which metric learning method or strategy was used to compute W .
Furthermore, from our experiments in phase 1 it was not clear which source com-
binations are better when we evaluated the NCMC performance in the projected space.
Furthermore, the combinations that yielded best accuracies in the original space (us-
ing SVM classifiers) were not always the ones that proved to be the best after pro-
jection. Therefore, for each of the above described methods, we learned a metric us-
ing each source combination SCi individually and computed class predictions P (yi =
SCi
c∗ |fml (xti )) for the test set Tu in the corresponding projected space. Then we com-
puted the late fusion of class predictions over all source combination to get final predic-
tions given a metric learning method. These classifiers denoted by fml , were used in two
modes. Two runs have been used in submission to predict directly the labels, namely,
MLNCM MLDA 128 it200 e0.1 p025 and MLNCMC ML 128 it200 e0.1 p025. Also
they were used by ensemble methods in combination with other models to generate the
combin * runs.
4 Ensemble Method
The combination of the decisions of several classifiers has been proposed as a means
of improving the accuracy achieved by any of them. The reasons for combining the
outputs of multiple classifiers are compelling, because different classifiers may implic-
itly represent different useful aspects of a problem, or of the input data, while none of
them represents all useful aspects. In the context of instance classification, the idea of
combining the decisions of several classifiers has been well explored.
6
The classification performance on the initial training set incurs a stronger degradation than a
predefined tolerance threshold or no more target image can be added or source removed.
7
We did 11 fold cross validation on the training set in phase 1. We split the provided the 600
test instances into 10 folds such that each folds contained 5 document per class and we added
the training set as 11th fold.
457
In contrast to approaches which combine models derived from multiple versions of
the same learning method, the specific focus of this paper is on ensemble methods that
are able to combine the decisions of multiple classifiers of different types, so-called
heterogeneous sets of classifiers.
Previous sections described three domain adaptation methods which generate one
or more classifiers for target instances, namely, obtained by the brute force (fmcsvm and
fbsvm ), transfer adaptive boosting (ftra ) and metric learning (fml ). They form a pool F
of several classifiers from each group F = {f1 , . . . , fNf } where any classifier’s output
is class scores or class probabilities. We will denote by g(fk , xti ) the predicted label of
fk for xti and by p(yi = c|fk (xti )) the class probability scores. When the classifier does
not provide such score we can define p(yi = c|fk (xti )) = 1 when c = g(fk , xti ) and
zero otherwise.
The first method we tested was the (unweighted) majority voting that combines
decisions of individual classifiers, so the global ensemble prediction for target instance
xt is: X
c∗ = argmaxc∈Y [[gk (fk , xti ) = c]]
fk ∈F
In the probabilistic setting, we sum up class probabilities and report the class with
the highest probability:
X
c∗ = argmaxc∈Y P (yi = c|fk (xti ))
fk ∈F
The majority voting method is based solely on the output label or probabilities com-
puted by each classifier. In the case of weighting majority voting, classifiers’ opinions
may be taken differently, depending on how accurate they have been in the past. Be-
low we establish weights proportional to each classifier’s accuracy, so each classifier’s
output is considered according to its past performance.
In the probabilistic setting, the fact that instance xti belongs to class c given that
classifier fk output label c0 for x, i.e. g(fk , xti ) = c0 , has an uncertainty which can
be expressed as the conditional probability p(yi = c|g(fk , xti ) = c0 ). We may use the
confusion matrices to approximate the conditional probabilities for all classes c ∈ Y
given a classifier fk ∈ F .
Making assumption of the classifier independence, we use the Bayesian rule to es-
timate the posterior probability of the class c for the target instance xi from the output
of classifiers fk ∈ F :
00
Q t
t c00 ∈Y P yi = c|g(fk , xi ) = c
P (yi = c|xi ) = P Q 0 t 00
c0 ∈Y c00 ∈Y P yi = c |g(fk , xi ) = c
Finally, the class c∗ = argmaxc∈Y P (yi = c|xti ) (with the highest probability) is as-
signed as the weighted majority vote to instance xi .
5 Evaluation results
Both individual and ensemble methods have been tuned by XRCE team during the
phase 1 to select best strategies of the challenge and applied to the submission data
458
(phase 2). Among multiple evaluation runs, ten most promising results have been se-
lected for the submission, they all are reported in Table 1. For each submission, the
table reports the accuracy, score, divergence and a comment on the method used.
The divergence has been proposed to measure the deviation of the submitted pre-
diction vector from a vector with equi-weighted classes:
X N
div = Card({i|g(f, xti ) = c}) −
Nc
c∈Y
where N is the number of test images, Nc the number of classes, {i|g(f, xti ) = c} is
the set of target instances for which the classifier f predicts class c, Card(A) is the
cardinality of the set A and |a| is the absolute value of a. Under the assumption on the
equal class distribution in the submission dataset, it may make sense to choose a run
whose prediction tends to minimize the divergence.
Table 1. Ten runs submitted by XRCE team.
Place Score Accu- Run Name Diver- Comment
racy gence
1 228 38.0 combin6 Np20 108 UMV
2 228 38.0 combin3 Np18 108 UMV
3 226 37.67 combinAll6 Np19 164 UMV
4 217 36.17 combin6A Np19 78 UMV + min divergence
5 214 35.67 MLNCM MLDA 128 it200 e0.1 p025 174 ML
6 212 35.33 combinAll7A Np19 134 WMV
7 208 34.67 combin8A Random Np25 78 WMV + min divergence
8 185 30.83 MLNCMC ML 128 it200 e0.1 p025 168 ML
9 182 30.33 combin2 Np10 134 TrA+UMV
10 158 26.33 svmBoost Mul Power f60 186 TrA
Analyzing the ten submissions made (as well as the non-submitted runs), allowed us
to make the following conclusions on the selected strategy and performance of different
components of the image classification system we have built for the challenge:
1. Among individual domain adaptation methods, the brute force performed poorly
as expected. It did not merit any individual submission, but participated in various
ensembles of classifiers.
2. TrAdaboost (TrA) with SVM as a week learner and the Metric Learning (ML)
performed reasonably well, each merited two individual submissions.
3. Ensembles of heterogeneous classifiers has turned to be a right strategy to boost the
overall performance.
4. Unweighted majority vote (UMV) on a small selection of classifiers performed the
best; these selections include 3 to 6 top performing classifiers from each group of
classifiers.
459
5. Weighted majority vote (WMV) works well on large sets of classifiers but under-
performed against the manual classifier selection8 with unweighted majority vote.
6. Divergence minimization did not play any important role; making hypotheses on
the class distribution and incorporating this prior knowledge in the classification
system should be done in a more systematic way.
6 Conclusion
The strategy of using heterogeneous methods for domain adaptation turned to be a right
one. It has allowed our team to build a image classification system in the target do-
main with an important knowledge transfer from available source domains. Ensembles
of heterogeneous classifiers aggregated with different majority voting scenarios has al-
lowed to get high accuracy in the submission runs and to eventually win the ImageCLEF
Domain Adaptation competition. The thoughtful analysis of obtained results has also
allowed to identify new directions in domain adaptation for image classification.
References
1. Bing. http://vlg.cs.dartmouth.edu/projects/domainadapt/.
2. Caltech-256. http://www.vision.caltech.edu/Image_Datasets/
Caltech256/.
3. Imagenet12. http://image-net.org/challenges/LSVRC/2012/index.
4. Pascal-voc12. http://pascallin.ecs.soton.ac.uk/challenges/VOC/
voc2012/index.html.
5. Sun. http://groups.csail.mit.edu/vision/SUN/.
6. Samir Al-Stouhi and ChandanK. Reddy. Adaptive boosting for transfer learning using dy-
namic updates. In D. Gunopulos, T. Hofmann, D. Malerba, and M. Vazirgiannis, editors,
Machine Learning and Knowledge Discovery in Databases, volume LNCS 6911, pages 60–
75. 2011.
7. Oscar Beijbom. Domain adaptations for computer vision applications. Technical report,
2012.
8. L. Bottou. Large-scale machine learning with stochastic gradient descent. In COMPSTAT,
2010.
9. B. Caputo, H. Müller, J. Martinez-Gomez, M. Villegas, B. Acar, N. Patricia, N. Marvasti,
S. Üsküdarlı, R. Paredes, M. Cazorla, I. Garcia-Varea, and V. Morell. ImageCLEF 2014:
Overview and analysis of the results. In CLEF proceedings. 2014.
10. Barbara Caputo and Novi Patricia. Overview of the ImageCLEF 2014 Domain Adaptation
Task. In CLEF 2014 Evaluation Labs and Workshop, Online Working Notes, 2014.
11. G. Csurka, C. Dance, L. Fan, J. Willamowski, and C. Bray. Visual categorization with bags
of keypoints. In ECCV Workshop on Statistical Learning in Computer Vision, 2004.
12. Wenyuan Dai, Qiang Yang, Gui-Rong Xue, and Yong Yu. Boosting for transfer learning. In
ICML, pages 193–200. ACM, 2007.
13. Jason V. Davis, Brian Kulis, Prateek Jain, Suvrit Sra, and Inderjit S. Dhillon. Information-
theoretic metric learning. In ICML, 2007.
8
Under manual selection we meant that the selection was made based on our intuitions accord-
ing to some of the lessons we learned in phase 1.
460
14. Basura Fernando, Amaury Habrard, Marc Sebban, and Tinne Tuytelaars. Unsupervised vi-
sual domain adaptation using subspace alignment. In ICCV, 2013.
15. Yoav Freund, Robert Schapire, and N Abe. A short introduction to boosting. Journal-
Japanese Society For Artificial Intelligence, 14(771-780):1612, 1999.
16. B. Gong, Y. Shi, F. Sha, and K. Grauman. Geodesic flow kernel for unsupervised domain
adaptation. In CVPR, 2012.
17. Boqing Gong, Kristen Grauman, and Fei Sha. Reshaping visual datasets for domain adapta-
tion. In Advances in Neural Information Processing Systems, pages 1286–1294, 2013.
18. Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Domain adaptation for object recog-
nition: An unsupervised approach. In ICCV, pages 999–1006. IEEE, 2011.
19. Judy Hoffman, Brian Kulis, Trevor Darrell, and Kate Saenko. Discovering latent domains for
multisource domain adaptation. In Computer Vision–ECCV 2012, pages 702–715. Springer,
2012.
20. I-Hong Jhuo, Dong Liu, D. T. Lee, and Shih-Fu Chang. Robust visual domain adaptation
with low-rank reconstruction. In CVPR, pages 2168–2175, 2012.
21. Jing Jiang. A literature survey on domain adaptation of statistical classifiers. Technical
report, 2008.
22. B. Kulis, K. Saenko, and T. Darrell. What you saw is not what you get: Domain adaptation
using asymmetric kernel transforms. In CVPR, pages 1785–1792, June 2011.
23. T Mensink, J Verbeek, F Perronnin, and G Csurka. Distance-based image classification:
Generalizing to new classes at near zero cost. PAMI, 11, 2013.
24. Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. Knowledge and Data
Engineering, IEEE Transactions on, 22(10):1345–1359, 2010.
25. Vishal M Patel, Raghuraman Gopalan, Ruonan Li, and Rama Chellappa. Visual domain
adaptation: An overview of recent advances. IEEE Transactions on Geoscience and Remote
Sensing, 52(2):1019–1029, 2007.
26. Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models
to new domains. In Kostas Daniilidis, Petros Maragos, and Nikos Paragios, editors, ECCV,
volume 6314 of Lecture Notes in Computer Science, pages 213–226. Springer Berlin Hei-
delberg, 2010.
27. Tatiana Tommasi and Barbara Caputo. Frustratingly easy nbnn domain adaptation. In ICCV,
2013.
28. K. Weinberger and L. Saul. Distance metric learning for large margin nearest neighbor
classification. JMLR, 10:207–244, 2009.
29. Zhijie Xu and Shiliang Sun. Multi-source transfer learning with multi-view adaboost. In
Tingwen Huang, Zhigang Zeng, Chuandong Li, and ChiSing Leung, editors, NIPS, volume
7665 of Lecture Notes in Computer Science, pages 332–339. Springer Berlin Heidelberg,
2012.
461