A comparison between co-training and self-training method for single-target regression based on multiples learners is performed. Data streaming systems can create a signi cant amount of unlabeled data which is caused by label assignment impossibility, high cost of labeling or labeling long duration tasks. In supervised learning, this data is wasted. In order to take advantaged from unlabeled data, semi-supervised approaches such as Co-training and Self-training have been created to bene t from input information that is contained in unlabeled data. However, these approaches have been applied to classi cation and batch training scenarios. Due to these facts, this paper presents a comparison between Co-training and Self-learning methods for single-target regression in data streams. Rules learning is used in this context since this methodology enables to explore the input information. The experimental evaluation consisted of a comparison between the real standard scenario where all unlabeled data is rejected and scenarios where unlabeled data is used to improve the regression model. Results show evidences of better performance in terms of error reduction and in high level of unlabeled examples in the stream. Despite this fact, the improvements are not expressive.
Prediction represents an essential task in data streams contexts that depend on
accurate predictions for decision making or planning [
The main areas where unlabeled data occurs are Engineering Systems ( video
object detection ) [
Semi-supervised Learning (SSL) methodology have been suggested to use
input information from unlabeled data for more accurate predictions [
More formally, X = fX1; :::; Xj ; :::; XJ g 2 RJ represents a vector of input random variables and Y represents a scalar random variable, with a joint probability distribution P (X; Y). The vectors xi = (xi;1; :::; xi;j ; :::; xi;J ) 2 RJ and yi, where i 2 f0; 1; 2; :::g, represent realizations of X and Y, respectively. A stream is de ned as the sequence of examples ei = (xi; yi) represented as S = f(x0; y0); (x1; y1); :::; (xi; yi); :::g. Label absence is represented by yi = ;. The objective of SSL is to use examples (xi; ;) to enhance the regression model yi f (xi) and reduce the error of prediction for both labeled and unlabeled examples.
SSL methods work on batch mode and are applied to classi cation. The
imediate adaption to regression is not possible [
The training stage produces an arti cial label for the unlabeled example from
the regressors predictions of the same example, according to a criterion (e.g.,
mean of all predictions) [
Self-training can be seen as a particular case of Co-training where just one model is trained. This method uses its own predictions to arti cially label the unlabeled examples and use this examples in the respective training.
It worths to say that the active learning can also be easily introduced in these methods. However, once used in the training, the example contribution cannot be removed from the model.
This work focus on the comparison between Self-training method and
Cotraining method which uses several models that learn with each other for online,
single-target regression. Despite expecting better prediction results from
Cotraining, it is important to nd how much the results are superior to Self-training.
In fact, Co-training is more computationaly expensive then Self-training. This
work also may pave the way for the extension to online multi-target regression
using the Random Adaptive Model Rules (Random AMRules) algorithm in
future works [
This document is structured as follows. In Section 2, the fundamentals of SSL for Self-training and Co-training are brie y revised. Section 3 describes the modi cations of the Co-training to online learning and regression using ensembles of rule models. Section 4 describes the evaluation method. The results are discussed in Section 5 and the main conclusions are remarked in Section 6.
This section explains some concepts used in the development of Co-training and Self-training methods. As general pattern, Co-training involves the training of two or more di erent models in some aspects ( e.g., di erent inputs, di erent regressors, di erent parametrization, di erent examples ...). The labeled examples are processed as a supervised processure. The unlabeled examples are arti cially labeled and processed as a supervised procedure. The arti cial label is essentially a prediction (Self-training) or a processed prediction derived from a combination of predictions of complementary learners (Co-training). The learners are considered to predict reliably (con dence driven method).
Co-training methods follow these assumptions: consensus, complementary, su ciency, compatibility and conditional independence. Self-training only considers su ciency and compatibility. Note that these assumptions can be applied for both batch and online(incremental) methods.
{ Consensus assumption states that the more similar the learner predictions
are, the more reliable the arti cial label is [
Concerning the drawbacks, the inaccuracy of the arti cially labeled examples
introduce error into the models and it is the main cause of model degradation.
Moreover, the arti cially labeled examples may not carry the information to
the regressor leading to unnecessary operations [
This section provides the description of a developed a Co-training method and also a Self-training method through the presentation of the main adaptations to the online and regression context. Here, the description was focused in the Co-training algorithm. A small description of the underlying algorithm regressor Random AMRules (ensemble rules based method) is also presented.
The new method that is being proposed divides the inputs variables of the example into two groups randomly which is de ned in the initial stage. Here, weak dependence is assumed since no independence information between pairs of attributes is available. The complementarity assumption is also used since each produced model contains information that other does not contain.
The two groups are forced to share a randomly selected inputs by a prede ned overlap percentage. Two Random AMRules complementary regressors are used to produce arti cial labels through prediction for the unlabeled example. The initial models are obtained previously in a training stage using a dataset portion. The size of dataset portion should be su cient to produce a consistent model. Here, the inputs overlapping increase the number of attributes in each model and contribute for the su ciency assumption.
A score that re ects the bene t or con dence of arti cially labeled example is calculated for the decision of being accepted for training. The score is the relative di erence (RD) compared to the maximum of absolute values of the output found in the stream ymax. Here, the consensus assumption is used. Equation 1 de nes de relative di erence.
RD = jy^i1 ymax
2 y^i j If the score is lower than a pre-de ned threshold, the predictions are used to train the complementary regressor. Otherwise, the arti cially labeled example is rejected. The consensus assumption is used in this step. If the example is labeled, this example is used to compute the mean error for each regressor. Next, the example is used for all regressors training. Here, the compatibility assumption is used since both models are trained with the same output. Algorithm 1 explains the training procedure of the proposed method.
Prediction is performed by combining the regressor predictions through prediction weighting. The weights are computed by inverting the values of the respective error produced by labeled examples in the training stage since the higher the error is, the less the arti cial example bene ts the model. In other words, this strategy gives more credit to the regressor that produces less errors. Algorithm 2 shows the steps of label prediction.
The Random AMRules regressor was employed to train the models and to
produce the arti cial labels for the unlabeled examples [
In essence, Random Rules is an ensemble based algorithm that uses
bagging to create diversity and uses AMRules algorithm as a regressor. AMRules
(1)
Algorithm 2 Prediction algorithm of the proposed method
1: Input: Example (xi; yi) 2 S
2: Output: Example prediction y^i
3: Method:
4: Divide xi into xi1 and xi2
5: y^i1 = P redictM odel1(xi1)
6: y^i2 = P redictM odel2(xi2)
7: w1 = e2=(e1 + e2))
8: w2 = e1=(e1 + e2)
9: y^i = w1 y^i1 + w2 y^i2
divides the input space in order to train local model in each partition. AMRules
partionates the input space and creates local models for each partition. The
local models are trained using a single layer perceptron. Its main advantages are
models simplicity, low computational cost and low error rates [
Modularity is one of the main advantages. In fact, this method allows the train of models for local input sections limited by the rule that are more precise. This algorithm also resorts to anomaly detection to avoid data outliers damage. Moreover, change detection on the stream is also employed by this method in order to avoid the in uence of old information on the current predictions. The ensembles of AMRules can bene t the prediction by creating multiple and diverse regressor models by pruning the input partitions. The multiple regressor predictions create more possibilities to nd a more accurate value. The ensembles also lead to a more stable nal prediction and data change resilience.
The Self-training method basically consists of one learner that arti cially labels the incoming example and uses directly in the training (without a rejection criteria). This method is very simple compared to Co-training, since only one model is trained and it doesnt present a condition for training acceptance. In terms of complexity, the Co-training presents the double complexity in required memory and computing power, when compared to the Self-Learning. In fact, the Co-training method basically trains two modesl while Self-training just trains one. 4
The evaluation method and the material used in the experiments are described in this section. Real-world and arti cial datasets were used to evaluate the proposed algorithm through a data stream simulation. A portion of 30% of the rst examples of the stream were used for a initial consistent model training and the remaining 70% were used in the testing.
In order to produce an unlabeled examples in the test stage, a binary Bernoulli random process with a probability p was used to assign an example as labeled or unlabeled. In case of unlabeled assignment, the true output value is hidden from the algorithm. The p probabilities of unlabeled examples occurrence were 50%, 80%, 90%, 95% and 99%.
For the Co-training method, the score threshold values for algorithm
calibration were 1 10 4, 5 10 4, 0:001, 0:005, 0:01, 0:05, 0:1, 0:5 and 1. These
values of score threshold are justi ed by the possibility of algorithm behaviour
observation in multiple scales of this parameter. The overlap percentages
assume the following values: 0%, 10%, 30%, 50%, 70% and 90%. The evaluation
was performed in Prequential mode where in example arrival, the label
prediction is performed rst and then the example is used in the training [
In these experiments, ve real world and four arti cial datasets were used
to simulate the data stream. The real world datasets were House8L (Housing
Data Set), House16L (Housing Data Set), CASP ( Physicochemical Properties of
Protein Tertiary Structure Data Set), California, blogDataTrain and the arti cial
datasets were 2dplanes, fried, elevators and ailerons. These datasets contain a
single-target regression problem and are available at UCI repository [
Table 2 an shows the features of the real world and arti cial data sets used in the method evaluation.
The performance measure used in these experiments was the mean relative error (MRE). The MRE is used as an intermediate measure to quantify the prediction precision of each test scenario for both labeled and unlabeled examples. The MRE Reduction (MRER) was measured by using the relative di erence (in percentage) between the reference scenario (no unlabeled examples used) M RE0 and the case with the parametrization that lead to the lowest error M RElowest fried ailerons elevators # Examples
# Inputs 22784 22784 20640 45730 52472 8 16 7 9 281 # Examples
# Inputs 40768 40768 13750 8752 10 10 41 18 (includes the reference case M RE0). Equation 2 de nes the MRER performance measure.
M RER = jM RE0
M RElowestj :100 (%) M RE0 (2)
If the reference case yields the lowest error, then the MRER is zero, which means that the algorithm is not useful for that particular scenario.
Massive Online Analysis (MOA) platform was used to accommodate the
proposed algorithm [
In this section, the evaluation results for Co-training and Self-training are presented and discussed. 5.1
Co-training results For each combination of overlap percentage and score threshold, the experiments were performed in 10 runs due to the fact that the inputs are selected randomly. This procedure is important to obtain more consistent values. The results also include the presentation of the MRER for each dataset and the unlabeled examples percentage simulation.
In the experiments was registered, for the particular case of overlap of 50% and score threshold of 0.001, the use of 9.1% of the unlabeled examples in the training lead to reduction of 3.85% of the MRE in average for the House16H dataset. In general, it was also observed that the overlapping decrease the MRE.
The failure in some scenario is explained by the fact of many unlabeled examples lead to model degradation and the arti cial labels were very inaccurate (the curves of unlabeled examples scenarios are above the reference curve). This fact indicates that features of the datasets such as inputs variables distributions may dictate the performance.
This methods are prone to error propagation through the model. The error propagation through the model lead to worst predictions in the arti cial labeling. This e ect leads to a cycle that reinforce the error on each unlabeled example processing. In fact, the more unlabeled examples arrive the higher is the error.
Table 3 provides the MRER values of the experiments on real world datasets for each chosen unlabeled examples probabilities, for Co-training method.
Table 3 suggests that the proposed algorithm seems to improve the performance for most part of the scenarios. The Co-training method can produce error reduction in higher percentage of unlabeled examples than the Self-training method. Despite this fact, the MRER are in general relatively small but superior than the Self-training method.
Table 6 provides the MRER value for real arti cial datasets in similar way as the real world datasets presented in Table 5. 2dplanes fried ailerons elevators
The results on arti cial datasets reinforce the same conclusions that were obtained from real world datasets. The MRER is similarly small.
The results show that for 99% of unlabeled examples probability, the method does not produce bene cial arti cial labels. This high level of unlabeled examples in the stream represents an extreme scenario where the model is training almost with arti cially labeled examples and the high error propagation can frequently occur.
Table 5 suggests that the proposed algorithm seems to improve the performance for few scenarios. In fact, the algorithm fails in high probabilities of unlabeled examples. Inclusively, there is one dataset that didnt produce any favourable result. In successful cases, the MRER are in general relatively small.
Table 6 provides the MRER value for real arti cial datasets in similar way as the real world datasets presented in Table 5. 2dplanes fried ailerons elevators
The results on arti cial datasets also support the view that the more elevated the unlabeled probability is, the less is the bene t of the unlabeled examples. The MRER is similarly small and there are very few successfull cases.
These results show that Self-training is limited by the percentage of unlabeled in the stream. For unlabeled examples higher than 50 %, the Self-training does not produce any error reduction. This limitation is explained by the fact that the arti cially labeled examples produce high errors which does not garantee compability of the predictions. 6
This paper addresses a comparison of an online Co-training and Self-training algorithm for single-target regression based on ensembles of rule models. This work is the base for the development of multi-target regression methodology capable of using unlabeled examples information for model improving.
The results support that Co-training approach which uses the Random AMRules method reduces the error with the appropriate parameters calibration. The main contribution was the overlapping and the consensus measure strategies that contribute to increase diversity and model consistency in a online cotraining scenario. The comparison between Co-training and Self-training reveal that Co-training can in fact lead to higher error reductions that the Self-training. In addiction, Co-training can produce error reduction in higher level of unlabeled examples in the stream.
In fact, the MRER is positive when an amount of unlabeled examples are used in the training in most evaluation combinations. Despite this fact, the model bene t is still relatively small and the performance is highly dependent of a good parametrization tuning (score threshold and overlap percentage). In addition, the amount of unlabeled examples is relatively small to obtain some model improvement.
Considering future work, this work will be extended to multi-target regression. The fact that very few unlabeled examples can lead to some improvement may suggest the study of the conditions that lead to this improvement. To increase the method validity, future works will include a higher number of real world datasets with higher amount of examples. Datasets with particular features such drifts presence are also in view. 7
This work is nanced under the project "NORTE-01-0145-FEDER-000020" which was funded by the North Portugal Regional Operational Programme (NORTE 2020), under the PORTUGAL 2020 Partnership Agreement, and through the European Regional Development Fund (ERDF).