Transfer learning has proved to be efective for building predictive models for a target domain, by exploiting the knowledge coming from a related source domain. However, most existing transfer learning methods assume that source and target domains have common feature spaces. Heterogeneous transfer learning methods aim to overcome this limitation, but they often make strong assumptions, e.g., on the number of features, or cannot distribute the workload when working in a big data environment. In this manuscript, we present a novel transfer learning method which: i) can work with heterogeneous feature spaces without imposing strong assumptions; ii) is fully implemented in Apache Spark following the MapReduce paradigm, enabling the distribution of the workload over multiple computational nodes; iii) is able to work also in the very challenging Positive-Unlabeled (PU) learning setting. We conducted our experiments in two relevant application domains for transfer learning: the prediction of the energy consumption in power grids and the reconstruction of gene regulatory networks. The results show that the proposed approach fruitfully exploits the knowledge coming from the source domain and outperforms 3 state-of-the-art heterogeneous transfer learning methods.
Machine learning algorithms aim to train a model function that describes and generalizes a set
of observed examples, called training data. The learned model function can be applied to unseen
data with the same feature space and data distribution of the training data. However, in several
real scenarios, it is dificult or expensive to obtain training data described through the same
feature space and following the same data distribution of the examples where the prediction
functions will be applied. A possible solution to the challenges raised by these scenarios come
from the design and application of transfer learning methods [
A relevant example of these scenarios can be observed in the energy field, where predicting
the customer energy consumption is a typical task [
In several scenarios, data for the target and the source domains come from multiple data
sources, exhibiting diferent data representations that make the direct adoption of classical
transfer learning approaches unfeasible. A possible solution may consist in performing
timeconsuming manual feature engineering steps, aiming to find commonalities among the data
sources and to somehow make the feature spaces homogeneous. However, such manual
operations can be subjective, error-prone, and even unfeasible when no detailed information are
available about the features at hand, or when the features in the target and source domains
are totally diferent, i.e., heterogeneous. To overcome this issue, several approaches have been
proposed in the literature to automatically identify the best match between features of diferent
domains, or to identify a novel shared feature space. Such methods cover diferent real-domain
applications, such as biomedical analysis [
To overcome these limitations, in this paper we proposed a novel heterogeneous transfer learning method, called STEAL, that simultaneously exhibit the following characteristics: • It is able to work also in the very challenging Positive-Unlabeled (PU) learning setting; • It is implemented using the Apache Spark framework following the MapReduce paradigm, enabling the distribution of the workload over multiple computational nodes; • It exploits the Kullback-Leibler (KL) divergence to align the descriptive variables of the source and target domains and, therefore, to work with heterogeneous domains described through feature spaces having diferent dimensionalities; • It is not tailored for specific application domains, but can be considered a general purpose approach applicable to multiple real-world scenarios.
In this section, we describe the proposed method STEAL, emphasizing its peculiarities. The distribution of the workload over multiple computational nodes is achieved by designing the algorithms using the MapReduce paradigm. The pseudo-code description of the algorithms exploits the Resilient Distributed Dataset (RDD) data structure available in Apache Spark.
The workflow followed by STEAL consists of 3 stages. The first two stages are dedicated to align the feature spaces and to identify a proper matching between target and source instances. Finally, the third stage consists of training a distributed variant of Random Forests (RF) available in Apache Spark, from the obtained hybrid training set. In the following, we report some further details about the first two stages.
Stage 1 - Feature Alignment. In the first stage, in the case of PU learning setting, we first
estimate the labels of the unlabeled instances by resorting to a clustering-based approach.
Specifically, we apply a prototype-based clustering algorithm to identify representatives of
positive instances from the positive examples of the target domain, and exploit them to estimate
a score for each unlabeled instance. Specifically, we adopt a distributed variant of the -means
algorithm, since it is well established in the literature and has been efectively exploited in
previous works in the PU learning setting [
Methodologically, the score is computed according to the similarity between the feature
vector associated with the unlabeled instance and the feature vectors of the cluster prototypes.
√∑︀
As similarity function, we consider (, ) = 1 − =1(− )2 , that computes the similarity
between the instance vectors and , in a -dimensional space, based on the Euclidean distance,
after applying a min-max normalization (in the range [
Subsequently, if the dataset is highly dimensional1, STEAL applies a distributed variant of the PCA, in order to identify a reduced set of (non-redundant) new features. We extract √︀(, ) new features, where and are the initial dimensionalities of the source and the target feature spaces, respectively.
Finally, we focus on the main objective of this stage, namely, the identification of an alignment between the descriptive variables of the source and target domains. This step is necessary in order to make the instances of the source and target domains directly comparable (through a distance/similarity measure), to carry our the following Stage. Specifically, the goal is to find a correspondent feature in the source domain, for each feature of the target domain. To this aim, we compute the asymmetric KL divergence for each target-source pair of descriptive variables, that quantifies the loss of information caused by a given feature of the source domain when used to approximate a given feature of the target domain. Formally, given the -th feature of the target domain , and the -th feature of the source domain , we compute the discrete variant of the KL divergence between and as follows: ( , , , ) =
∑︁ ∈(( ,)∩( ,)) (, , ) ln (, , ) , (, , ) (1) where and are the training instances of the target and source domains, respectively; (, ) represents the set of distinct values of the feature in the dataset ; (, , ) represents the relative number of occurrences of the value on the feature in the dataset , after a proper discretization of the feature 2.
In Figure 1 we graphically show this alignment step, while the pseudo-code is reported in Algorithm 1. STEAL starts by considering the values assumed by the attributes as the key of a paired RDD. The result is then used to find common values assumed by target and source attributes, through a join operation. The frequency of each distinct value is then computed for each target and source attribute, which are then exploited to estimate the probabilities used in 1This can be considered as an optional step, that mainly depends on the availability of computational resources. 2We discretize continuous features using the equal-width strategy (bin size of 0.01), after min-max normalization.
Algorithm 1: alignSourceDomain(, )
Data: , : RDD. Target and source domain instances (in the ⟨_, , ⟩ form) Result: : RDD. Source domain instances aligned in the -dimensional feature space. 1 begin 2 ← .map{case(_, , ) → ⟨, ⟩}; 3 ← .map{case(_, , ) → ⟨, ⟩};
// Join the target and source structures according to the feature values 4 ← .join( )
// Count the frequencies of values for the target features 5 ← .map{case(, , )→⟨⟨, ⟩, 1⟩}.reduceByKey((, )→+) // Compute the number (cardinality) of value matches for each target feature 6 ← .map{case(⟨⟨, ⟩, ⟩)→⟨ , ⟩}.reduceByKey((, )→ + ) // Compute the value probabilities for the target domain 7 ← .map{case(⟨, ⟩, ) → ⟨ , ⟨, ⟩⟩} 8 .join().map{case( , ⟨, , ⟩) → ⟨, , /⟩}
// Compute the value probabilities for the source domain in the same way 9 ← ...
// Compute the KL divergences ← .cartesian( ) .map{case(⟨ , , ⟩, ⟨ , , ⟩)→⟨⟨ , ⟩, · ( / )⟩} .reduceByKey((, ) → + ) // Find the best target-source features matching by minimizing the KL divergences ← .map{case(⟨ , ⟩, ) → ⟨ , ⟨ , ⟩⟩} .reduceByKey{case(⟨ 1, 1⟩, ⟨ 2, 2⟩) →
if(1 < 2) then ⟨ 1, 1⟩ else ⟨ 2, 2⟩} // Align source features to target features, according to the minimum divergences return (, ) the computation of the KL terms. Such probabilities are computed by dividing the frequencies by the number of matches (in the join operation) that involved a given attribute. Finally, we compute the KL divergence for each target-source pair of features, which are then exploited to ifnd the best (i.e., with the minimum divergence) source feature for each target feature.
Note that the proposed approach implicitly performs a feature selection on the source domain, since some of the features could not be matched to any target feature. Analogously, the same feature of the source domain could be selected multiple times for diferent features of the target domain, if it appears strongly aligned to all of them from a statistical viewpoint. Stage 2 - Instance Matching. After the first stage, target and source instances are represented in an aligned feature space, and are also directly comparable through similarity/distance measures. A straightforward approach to exploit the instances of the source domain would be that of appending them to the training set of the target domain. However, this approach would lead to the possible introduction of noisy instances, i.e., not properly representing the data distribution of the target domain, increasing the chance of negative transfer phenomena.
On the contrary, we aim to finely exploit only a subset of source instances, by attaching them to specific target instances, through feature concatenation, according to their similarity. In this way, STEAL augments the descriptive features of target instances with those coming from source domain descriptive variables
Fs1 Fs2 Fs3 Ft1 Ft2 Ft3 Ft4
target domain descriptive variables
Kullback-Leibler divergences divergence matrix
Fs1 Fs2 Fs3 aligned, best-matching, source instances.
STEAL computes the Cartesian product between subsets (randomly selected, with replacement) of target and source instances, and reduce the resultset by taking the source instances with the maximum similarity. The adoption of multiple random subsets, instead of working on the whole set of instances alleviates the negative efects due to the possible match of a target instance to an improper source instance, since the same target instance may be sampled multiple times and associated with diferent source instances. At the end of the instance matching step, we obtain a single target-source hybrid training set by merging the target-source concatenated instances constructed from each sample. For space constraints, we omit the pseudo-code description of this stage.
We conducted our experimental evaluation in a relevant application domain where transfer
learning can be strongly beneficial, namely in biology, and specifically in the reconstruction of
the human gene regulatory network (target domain), using also the knowledge coming from
the mouse gene regulatory network (source domain). The reconstruction of a gene regulatory
network consists in the identification of currently unknown gene regulatory activities. A
gene regulatory network consists of a set of genes (represented as nodes) and a set of known
regulations (represented as edges). The reconstruction of such networks can be performed by
resorting to link prediction methods [
We compared the results achieved by STEAL with some state-of-the-art heterogeneous
transfer learning competitors, namely TJM [
In our experiments, we used the dataset adopted in [
The task at hand naturally falls in the PU learning setting, since unobserved links cannot be considered negative examples. To perform the estimation of the label confidence (Stage 1), we identified 3 clusters and 2 clusters for the mouse and the human positive instances, respectively. Such values were identified through the silhouette cluster analysis. For the heterogeneous version of the dataset, we ran STEAL with the distributed PCA, that led to identify 18 new features. As regards the instance matching, 10 samples were considered (i.e., =10), each involving 10% random training examples from the source and from the target domains.
All the experiments were performed in the 10 fold cross-validation setting, where each fold consists of 9/10 positive examples for training and 1/10 positive examples for testing, while all the unlabeled examples are considered for both training and testing.
As evaluation measure, since the dataset does not contain any negative example, we considered the recall@ defined as + , where is the number of returned true positive interactions within the first top- interactions, and ( + ) corresponds to the number of positive examples in the testing fold, and computed the area under the recall@ curve (AUR@K). Note that the computation of other well-known measures, such as the Area Under ROC curve (AUROC) and the Area Under Precision-Recall curve (AUPR), is not possible at all in the positive-unlabeled setting, without introducing wrong biases in the ground truth.
From the results shown in Table 2, we can observe that STEAL is able to outperform all the considered baselines, in both the heterogeneous and homogeneous settings. In particular, the results show that STEAL outperforms the counterpart that does not exploit the source domain, i.e., T (no transfer), by 27% in the homogeneous setting and by 9.7% in the heterogeneous setting, proving the benefits of exploiting the additional knowledge coming from the mouse organism. On the other hand, using only the source domain (i.e., S (optimal feature alignment)), leads to a reduction of the AUR@K with respect to using the target data only. This means that, although source data can be beneficial for the target task, their exploitation should be done in a smart manner. Note that, as expected, appending source instances to the target dataset, even if the optimal feature alignment is known a priori (see T+S (optimal feature alignment)), does not provide as much as benefits as the method adopted by STEAL, that outperforms this approach by 23.4% and 10% in the homogeneous and heterogeneous settings, respectively.
Method
T (no transfer) S (optimal feature alignment) T+S (optimal feature alignment)
T+S (STEAL)
Method Homogeneous Reduced
JGSA 0.500 TJM 0.554 BDA 0.558 STEAL 0.589
The comparison with competitors on the reduced dataset shows that STEAL is able to outperform all of them. Specifically, STEAL achieves an improvement of 17.8% with respect to JGSA, 6.3% with respect to TJM, and 5.5% with respect to BDA. Note that the superiority of STEAL over these competitors is not limited to the observed AUR@K on the reduced dataset, since JGSA, TJM and BDA were not able to complete the experiments on the full dataset.
To assess the advantages of STEAL also from a computational viewpoint, we performed a scalability analysis. We measured the running times and computed the speedup factor to evaluate the ability of STEAL to exploit additional computational nodes. This analysis was performed on a cluster with 1, 2, or 3 computational nodes, by resorting to a synthetic dataset with 10 millions instances. The results of this analysis, depicted in Figure 2, show that, although the communication overhead in principle increases, STEAL is able to take advantage of possible additional computational nodes. Specifically, the measured speedup factors are close to ideal results, i.e., they quickly converge to 2 and 3, respectively, with 2 and 3 computational nodes.
In this discussion paper, we proposed a novel heterogeneous transfer learning method, called STEAL, that can work in the PU learning setting also with heterogeneous feature spaces, thanks to a feature alignment step based on the KL divergence. The obtained results demonstrate the efectiveness of the proposed method with respect to baseline and state-of-the-art competitors. Moreover, it exhibits very interesting scalability results, emphasizing its possible applicability on large datasets. We evaluated the performance of our method in the reconstruction of gene regulatory networks, that obtained significant benefits from the application of STEAL. Currently, we are evaluating the efectiveness of STEAL in other application domains, including the prediction of the energy consumption in power grids, and we are working on extending it with the possibility of exploiting multiple source domains.
Dr. Paolo Mignone acknowledges the support of Apulia Region through the REFIN project “Metodi per l’ottimizzazione delle reti di distribuzione di energia e per la pianificazione di interventi manutentivi ed evolutivi” (CUP H94I20000410008, Grant n. 7EDD092A).