First Principle Models Based Dataset Generation for Multi-Target Regression and Multi-Label Classi cation Evaluation

Abstract

Machine Learning and Data Mining research strongly depend on the quality and quantity of the real world datasets for the evaluation stages of the developing methods. In the context of the emerging Online Multi-Target Regression and Multi-Label Classi cation methodologies, datasets present new characteristics that require speci c testing and represent new challenges. The rst di culty found in evaluation is the reduced amount of examples caused by data damage, privacy preservation or high cost of acquirement. Secondly, few data events of interest such as data changes are di cult to nd in the datasets of speci c domains, since these events naturally scarce. For those reasons, this work suggests a method of producing synthetic datasets with desired properties(number of examples, data changes events, ... ) for the evaluation of Multi-Target Regression and Multi-Label Classi cation methods. These datasets are produced using First Principle Models which give more realistic and representative properties such as real world meaning ( physical, nancial, . . . ) for the outputs and inputs variables. This type of dataset generation can be used to produce in nite streams and to evaluate incremental methods such as online anomaly and change detection. This paper illustrates the use of synthetic data generation through two showcases of data changes evaluation.

Keywords

• Synthetic data
• data streams
• Multi-Label Classi cation
• MultiTarget Regression
• First Principle Models

-

In the areas of Machine Learning and Data Mining, datasets quality and quantity are crucial for evaluation stage of methods development [ 1 ]. Controlled evaluation environments with speci ed challenge problems are required to understand the behaviour of the methods [ 2 ]. Methods of Multi-Target Regression(MTR) and Multi-Label Classi cation (MLC) on online data streams are fair examples that imply these evaluation requirements. The importance of these methodologies has been growing due to reasonable modelling and predicting capabilities [ 3, 4 ]. Formally, let an unbounded data stream be represented by D = f:::; (x1; y1); (x2; y2); :::; (xi; yi); :::g, where xi = [xi;1 xi;j xi;M ] is a M -dimensional vector of real values containing the data descriptive variables xi;j (input variables) of the ith example (considering one example with the index of reference). For Multi-Target Regression, yi = [yi;1 yi;j yi;N ] denotes a real values vector of responses yi;j (output variables) of the ith example. For Multi-Label Classi cation, yi corresponds to a subset of nominal labels k such that yi f 1; :::; k; :::; Lg, where L is the number of possible labels.

Typically, output set of labels yi are transformed into a vector of outputs variables [yi;1 yi;k yi;L], where yi;k 2 f0; 1g are binary. If label k is assigned to the ith example then yi;k = 1, otherwise yi;k = 0. The outputs variables are rede ned as yi = [yi;1 yi;k yi;L]. Finally, the objective of both MTR and MLC methods is to learn a function f (xi) ! yi that maps the input values of xi into the output values of yi.

In the evaluation of the methods, the number of examples are not su cient in many cases by the reasons of sensitive data, data damage or high cost of acquisition [ 2 ]. Data changes scenarios are another prominent challenge for MTR and MLC methods [ 5 ]. Changes in the probability distributions, variables trends or variables rapid shifts of the inputs variables are events that have strong inuence on the method's performance [ 5 ]. Similarly to the scarcity of examples, real world datasets that gather all desired data changes properties also lack [ 2 ]. Moreover, the data change events are not often annotated, since annotation is time consuming.

As an attempt to solve this problem, researchers produce synthetic datasets to create evaluation challenges with desired properties. This alternative allows to produce a signi cant amount of examples or even create a reasonable approximation of an in nite data stream. Moreover, few resources for storage and transmission are required. Despite high complexity, the produced datasets models do not re ect the real world conterpart. In fact, the latent models are based on abstract mathematical concepts.

However, datasets can be constructed through the employment of a First Principle Models (FPM) which are described by established laws without making assumptions (empirical or tted parameters) [ 6 ]. FPM are used to create synthetic data for a wide range of areas such as Chemical Engineering (Industrial Chemical process) [ 7 ] and Mechanical Engineering(Mechanical Systems Diagnosis) [ 8 ]. In the area of Control Systems, Proportional-Integrate-Derivative (PID) systems modelling uses FPM extensively in several contexts of application [ 9 ]. For instance, FPM based software simulators(parametrized with inputs and outputs variables) that mimic those systems are created to reduce cost in the industrials trials [ 7 ]. The abundance of free software simulators and models of PID systems justi ed the focus on PID Systems. Thus, this work suggests a method to produce synthetic datasets that are reproducible for MTR and MLC evaluation. This method applies the FPM to produce more realistic and representative models.

Section 2 brie y reviews some existent methods of dataset generation. Section 3 describes the FPM method and the selected FPM model that is used to produce the synthetic datasets. Section 4 shows the production of the MTR and MLC synthetic datasets and their application through MTR and MLC methods evaluation showcases, under data changes scenarios. Finally, the results are presented and discussed in Section 5 and the main conclusions are reported in Section 6. 2

Related Work

In the literature, most of dataset generators produce Single-Label Classi cation (the output variable yi is a label) datasets, since MLC is an emerging methodology [ 1 ]. Monedero et al [ 2 ], Frasch et al [ 10 ] and Narasimhamurthy et al [ 5 ] are representative examples of Single-Label Classi cation(SLC) dataset generators.

One possible strategy is to produce several Binary Classi cation (where yi;k 2 f0; 1g) outputs (one for each label) and combine them into MLC datasets. However, these datasets does not represent correlation between outputs.

Read et al propose a dataset generator that uses single-label generators and combine them according to con gured label imbalance and probabilities of simultaneous label occurrence [ 11 ]. This dataset generator attempt to create more realistic datasets in terms of label imbalance and concept drifts through an empirical method.

Tomas et al propose a highly con gurable dataset generators for MLC based on the creation of hyper-spheres [ 1 ]. However, the generator is not designed to produce data changes. In addiction, despite high exibility, this dataset generator produces an abstract mathematical challenge.

Similarly, the majority of the existent datasets generator produce SingleTarget Regression(also known as Multivariate Regression) datasets with simple but highly non-linear models. Friedman produced STR datasets from very simple and non-linear models to test the methods of Multivariate Adaptive Regresssion Splines [ 12 ] The same strategy used in MLC can be applied to produce MTR from a STR. For the best of our knowledge, no MTR dataset generators were found in literature. In fact, MLC methodologies received more attention by the researchers. 3

First Principle Model-Tennesse Eastman Process In order to illustrate the application of FPM's to generate MTR and MLC datasets, the Tennessee Eastman Process (TEP) was chosen. TEP consists of an industrial process of continuous chemical production. The process is unstable, non-linear and controlled by PID system. Basically, PID systems are controlled mechanisms based on loop feedback that are used in process stabilization [ 9 ]. Figure 1 shows the model of a generic PID system. These systems involves essentially a Plant and a PID controller.

The Plant consists of a set equations that represent the behaviour of the controlled process. The process is driven by the manipulable variables ui and observed by the measurement variables yi. Some processes present the disturbances variables di (binary variables)to simulate process impairments. The error ei between the desired set points ri and process measurements is computed with the purpose of being minimized. The PID controller consists of weighted sum of proportional(present values), integral(past values) and derivative (possible future values) terms which are calibrated in order to produce stable responses [ 9 ]. This component receives the error and computes new manipulable variables values that stabilizes the Plant process. Figure 2 represents the diagram of TEP Plant. Most process details were originally omitted for simpli cation and protection of intellectual property.

Two products G and H (Product) are produced from four gaseous reactants (A, C, D, E). An inert product B is present but does not intervenes in the chemical reaction. A by-product F (Purge) results from the whole reaction. The chemical reactions are irreversible and exothermic. The process model comprises ve inter-acting major units: a reactor, a condenser, a vapour-liquid separator, a stripper for the product stream and centrifugal compressor for the recycle stream. The model has 41 measurements and 12 manipulated variables and 12 set points. There is also 20 variables that simulate disturbances. The physical quantities that the variables represent are explained in detail in Bathelt et al [ 7 ]. Tables 6 to 5 gives the names of the variables and respective physical meaning (see in Appendix). 4

Methods

This section gives the description of procedures for the generation of MTR and MLC datasets. Two evaluation showcases of MTR regressor and MLC classi er which predict over the generated datasets are also described. The generation of the datasets was focused on data changes robustness tests. Data changes in streams can be analysed in two aspects: nature and rate. The nature re ects the variables statistics that changed such as mean and variance [ 13 ]. The rate of change is an important aspect that in uences the performance of most MTR and MLC methods. Abrupt changes (concepts drifts) are identi ed when the change occur from one example to the next (inexistence of transition phase). Gradual changes (concepts shifts) present a transition phase where the changing statistic is continuously varying [ 13 ]. This work is focused on the abrupt changes (concept drifts) of the mean statistic which is one of the most studied topics in data streams [ 14 ].

In this work, a simulator that implements the model described in Section 3 was used. The les of this simulator can be found at http://depts. washington.edu/control/LARRY/TE and was originally developed by Ricker [ 15 ]. This simulator was partially developed in C and in Matlab(Simulink). TEP simulator is composed essentially by a plant function developed in C language and posteriorly built into a Matlab mex le. The PID controller (a set of small PID controllers) function was developed in Simulink.

To produce data changes events, the set points variables are manipulated. Some set points such as Production Set Point allows to produce gradual changes (concept shifts) and abrupt changes (concept drifts) events, since the variables are related to measurements variables convergence to stable values. The rate can be calibrated using a rate limiter in the simulink environment. Figure 3 shows the parameters of data change event. The curve re ects the variation of a statistical parameter from a value a to a value b. The parameter t is the example index where the change starts and W is the number of examples in the transition phase. The change in the set point value create similar changes in the Plant parameters. The changes can be whether a descending and ascending of a parameter.

The generated datasets were produce with 100000 examples each, for both MTR and MLC evaluation. In this experiments, the changes were created with the periodicity of 5000, 10000 and 20000 examples. The W values were 0 in order to simulate a concept drift. Two special cases were also produced. One dataset does not present any drift(base line) and the other presents a constant decreasing change. Regarding the MTR dataset production, the dataset joins measurement variables yi and manipulable variables ui with purpose of predicting ui from yi. Each data example is de ned as ei = (yi; ui). As performance measures, the error was used for the error evolution and RMSE was used for global evaluation and comparison between algorithms. The error curve was smooth with a median sliding window due to the spiky form. The window length is 1000 examples without overlapping. For MLC dataset generation, the disturbances variables di and measurement variables yi are joined. The purpose is to predict the disturbance variables di from yi. The di disturbance variables are already in the form used in MLC problem transformation [ 11 ]. Each data example is de ned as ei = (yi; di). A Poisson process was used to choose the instant of a disturbance occurrence with a user de ned duration. The duration of the disturbances was 5000 examples. In this evaluation, the inputs variables were smoothed with a low-pass lter (sliding windows of 1000 examples) in other to have stability. The F-Measure, Precision, Recall, Accuracy and Exact Match were used for the classi cation scenario.

The Multi-Target regressor MT-AMRules and the Multi-Label classi er MLAMRules were used to exemplify the application of synthetic datasets in MTR and MLC methods evaluation, respectively. Both MTR regressor and MLC classi er were tested using a prequential mode. 5

Results

In this section, two simple showcases with MTR and MLC are demonstrated. The types of involved data changes events are also visualized. A comparison between a scenario without data changes and several scenarios where several types of data changes occur is performed. Figure 4 depicts 4 data changes in the mean statistic with di erent rates using the Reactor Coolant variable. The porpose is to show the exibility and the diversity of the adapted software to create several types of data changes.

These gradual and abrupt changes are created using rate delimiters in Simulink implementation. The thinner curve represents an abrupt change (concept drift) which is the event that is intended to be evaluated in MTR and MLC showcases.

In the following paragraphs, the MTR and MLC evaluation showcases are presented. Figure 5 shows the evalution of the error for several MTR datasets.

The plots a) to c) represent the smoothed error curve for 5000 (MTR 5k dataset), 10000 (MTR 10k dataset) and 20000 (MTR 20k dataset) examples of concept drift periodicity, respectively. The plot d) represent in solid line the scenario where the dataset presents no data change (MTR NoChange dataset) and the scenario where the data change is constant (MTR Const dataset) in dotted line. The error curves show the e ect of the concept drifts. Figures 5 shows descending (dashed) and ascending (solid) concept drift events. The descending variation cause error increasing with more impact than ascending variation. The impact of ascending variation can be observed in the plot c). Table 1 show the results of MTR regression using several datasets with di erent challenges. This work presented a framework for data set generation for MTR and MLC evaluation. A realistic and representative datasets for MTR regression and MLC Classi cation were obtained for method evaluation. However, the main limitations are the limited number of inputs and outputs. As future work, the main goal is to implement a MTR and MLC data streamer that produces data trough a con gurable FPM in a standalone and portable application.

Acknowledgements

This work was partly supported by the European Commission through MAESTRA (ICT-2013-612944) and the Project TEC4Growth - Pervasive Intelligence, Enhancers and Proofs of Concept with Industrial Impact/NORTE -01-0145FEDER-000020 is nanced by the North Portugal Regional Operational Programme (NORTE 2020), under the PORTUGAL 2020 Partnership Agreement, and through the European Regional Development Fund (ERDF).

Appendix: Tables of TEP variables

Number Variable Name

A feed (stream 1) D feed (stream 2) E feed (stream 3) A and C feed (stream 4) Recycle ow (stream 8) feed rate (stream 6) Reactor pressure Reactor level Reactor temperature Purge rate (stream 9) Separator temperature Product separator level Separator pressure Separator under ow (stream 10) Stripper level Stripper pressure Stripper under ow (stream 11) Stripper temperature Stripper steam dew Compressor work Reactor cooling water outlet temperature Separator cooling water outlet temperature of A in Reactor feed (stream 6) Concentration of B in Reactor feed (stream 6) Concentration of C in Reactor feed (stream 6) Concentration of D in Reactor feed (stream 6) Concentration of E in Reactor feed (stream 6) Concentration of F in Reactor feed (stream 6) Concentration of A in Purge (stream 9) Concentration of B in Purge (stream 9) Concentration of C in Purge (stream 9) Concentration of D in Purge (stream 9) Concentration of E in Purge (stream 9) Concentration of F in Purge (stream 9) Concentration of G in Purge (stream 9) Concentration of H in Purge (stream 9) Concentration of D in stripper under ow (stream 11) Concentration of E in stripper under ow (stream 11) Concentration of F in stripper under ow (stream 11) Concentration of G in stripper under ow (stream 11) Concentration of H in stripper under ow (stream 11) Unit kscmh kg/h kg/h kscmh kscmh kscmh kPa gauge %

C kscmh

C % kPa gauge m3 /h

% kPa gauge m3 /h

C kg/h kW

C

C mol % mol % mol % mol % mol % mol % mol % mol % mol % mol % mol % mol % mol % mol % mol % mol % mol % mol % mol %

References

1. Jimena Torres Tomas , Newton Spolao^r, Everton Alvares Cherman, and Maria Carolina Monard. A framework to generate synthetic multi-label datasets . Electronic Notes in Theoretical Computer Science , 302 : 155 { 176 , 2014 . Proceedings of the fXXXIXg Latin American Computing Conference (CLEI 2013 ).

2. Javier Sanchez-Monedero , Pedro Antonio Gutierrez, Mar a Perez-Ortiz, and Cesar Hervas-Mart nez. An n-spheres based synthetic data generator for supervised classi cation . In Advances in Computational Intelligence - 12th International WorkConference on Arti cial Neural Networks, IWANN 2013 , Puerto de la Cruz, Tenerife, Spain, June 12-14, 2013 , Proceedings, Part I , pages 613 { 621 , 2013 .

3. Changsheng Li , Weishan Dong , Qingshan Liu, and Xin Zhang. MORES: online incremental multiple-output regression for data streams . CoRR, abs/1412.5732 , 2014 .

4. Jesse Read , Albert Bifet , Geo Holmes , and Bernhard Pfahringer . Streaming multi-label classi cation . In Proceedings of the Second Workshop on Applications of Pattern Analysis, WAPA 2011 , Castro Urdiales , Spain, October 19-21 , 2011 , pages 19 { 25 , 2011 .

5. Anand Narasimhamurthy and Ludmila I. Kuncheva. A framework for generating data to simulate changing environments . In Proceedings of the 25th Conference on Proceedings of the 25th IASTED International Multi-Conference: Arti cial Intelligence and Applications , AIAP' 07 , pages 384 { 389 , Anaheim, CA, USA, 2007 . ACTA Press.

6. Manuel Rodr guez and David Perez. First principles model based control . In Luis Puigjaner and Antonio Espun~a, editors, European Symposium on ComputerAided Process Engineering-15, 38th European Symposium of the Working Party on Computer Aided Process Engineering , volume 20 of Computer Aided Chemical Engineering, pages 1285 { 1290 . Elsevier , 2005 .

7. Andreas Bathelt , N. Lawrence Ricker , and Mohieddine Jelali . Revision of the tennessee eastman process model . IFAC-PapersOnLine , 48 ( 8 ): 309 { 314 , 2015 .

8. Jonathan Cagan and Alice Agogino . Innovative design of mechanical structures from rst principles . IA-EDAM , 1 ( 3 ): 169 { 189 , 1987 .

9. K. J. AAstr om and T. Hagglund . PID Controllers: Theory, Design, and Tuning . Instrument Society of America, Research Triangle Park, NC, 2 edition , 1995 .

10. Janick V. Frasch , Aleksander Lodwich, Faisal Shafait, and Thomas M. Breuel . A bayes-true data generator for evaluation of supervised and unsupervised learning methods . Pattern Recogn. Lett. , 32 ( 11 ): 1523 { 1531 , August 2011 .

11. Jesse Read , Bernhard Pfahringer, and Geo Holmes . Generating synthetic multilabel data streams . In ECML/PKKD 2009 Workshop on Learning from Multi-label Data (MLD'09) , 2009 .

12. Jerome H. Friedman . Multivariate adaptive regression splines . The Annals of Statistics , 19 ( 1 ):1{ 67 , 1991 .

13. Jo ~ao Gama. Knowledge Discovery from Data Streams . Chapman & Hall/CRC, 1st edition , 2010 .

14. Albert Bifet , Geo Holmes, Bernhard Pfahringer, Richard Kirkby, and Ricard Gavalda . New ensemble methods for evolving data streams . In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '09 , pages 139 { 148 , New York, NY, USA, 2009 . ACM.

15. N. Lawrence Ricker . Decentralized control of the tennessee eastman challenge process . Journal of Process Control , 6 ( 4 ): 205 { 221 , 1996 .