The purpose of the ROCKFlows project is to lay the foundations of a Software Product Line (SPL) that helps the construction of machine learning work ows. Based on her data and objectives, the end user, who is not necessarily an expert, should be presented with work ows that address her needs in the "best possible way". To make such a platform durable, data scientists should be able to integrate new algorithms that can be compared to existing ones in the system, thus allowing to grow the space of available solutions. While comparing the algorithms is challenging in itself, Machine Learning, as a constantly evolving, extremely complex and broad domain, requires the de nition of speci c and exible evolution mechanisms. In this paper, we focus on mechanisms based on meta-modelling techniques to automatically enrich a SPL while ensuring its consistency.
The answer to the question "What Machine Learning (ML)
algorithm should I use?" is always "It depends." It depends
on the size, quality, and nature of the data. It also depends
on what we want to do with the answer [
The industry of cloud-based machine learning (e.g., IBM's Watson Analytics, Amazon Machine Learning, Google's Prediction API) provides tools to learn "from your data" without having to worry about the cumbersome pre-processing and ML algorithms. To address such a challenge they propose fully automated solutions to some classical learning problems such as classi cation. Some other actors like Microsoft, with the Azure's Machine Learning platform, allow users to build much more complex ML work ows, in a graphical editor that is targeted towards ML experts.
The common point between these solutions is that they
chose to select only a few algorithms, in comparison to the
hundreds that are available. However data scientists know
that the best algorithm will not be the same for each dataset
[
The contribution of this paper is thus to describe a toolsupported approach, responding to this challenge: the ROCKFlows project1.
The remainder of the paper is organized as follows. We discuss in the next section challenges we face and some related works. Section 3 describes the architecture that supports the project and two usage scenarios focusing each on a di erent user of ROCKFlows. We detail the evolution process and the correlated artefacts in Section 4. Section 5 concludes the paper and brie y discusses future work. 2.
The purpose of the ROCKFlows project is to lay the foundations of a software platform that helps the construction of ML work ows. This task is highly complex because of the increasing number and variability of available algorithms and the di culty in choosing the suitable and parametrized algorithms and their combinations. The problem is not only on choosing the proper algorithms, but the proper transformations to apply on the input data. It is a trade-o between many requirements (e.g., accuracy, execution and training time).
Since Software Product Line (SPL) engineering is
concerned with both variability and systematically reusing
development assets in an application domain [
Based on these requirements, we have identi ed the following challenges, addressing the needs for building and evolving a SPL in a domain as complex and changing as ML, ensuring a global consistency of the knowledge and scalability of the system.
Making a selection among the high number of data mining
algorithms is a real challenge: more than hundreds of
algorithms exist that can tackle a single ML problem such as
classi cation. While work exists to try and rank their
performance [
Data scientists often approach new problems with a set of best practices, acquired through experience. However, there is few scienti c evidence as to why an algorithm or pre-processing technique leads to better results than another and in which case. Thus, one of the biggest challenge for this project is so to nd a proper way to characterise algorithms and to compare them, relatively to the very broad spectrum of user needs and data representations.
Collaboration between SPL developers and data scientists
induces a complex software ecosystem [
Given these domain requirements, meta-model driven engineering provides an e cient and powerful solution to address the complexity of the ecosystem through support of separation of concerns and collaborations. In order to operationalize it, we chose to consider this environment as a set of components relying on di erent meta-models for which the evolution mechanisms are exposed through services.
While the number of ML algorithms and techniques constantly grows, the fundamental understanding of ML internal mechanisms is not stable enough to allow us to set any knowledge in stone. Both the domain and our understanding of it evolve quickly, forcing constant evolution of the SPL.
The line evolves in particular through the addition of new algorithms and pre-processings. We run experiments to identify dataset patterns leading to similar behavior of algorithms on di erent concrete datasets. The high number of possible combinations (variability of compositions and algorithms) as well as the frequent changes in ML require evolution mechanisms that are both incremental and loosely coupled with the elements presented to the end user.
As of today, ROCKFlows' SPL contains roughly 300 features, and 5000 constraints, representing 70 di erent ML algorithms, 5 pre-processing work ows, and is mostly focused on classi cation problems.
Evolution in SPLs has been a challenge for many years [
Moreover, despite the huge variability of the system, we
have decided to propose the end user only choices that can
lead to a proper result. Hence, it should not be possible
to build a con guration for which we would not be able to
generate a work ow. For instance, it will not be possible for
a user to select, relatively to a given dataset, a performance
value for which we have no algorithm that can reach such
requirements. It is necessary for us to ensure a consistent
con guration process [
Contrary to the works allowing several users to modify
a model in contradictory manners and aiming to reconcile
those [
We identify three stakeholders, each leveraging challenges. - SPL users are the end users of the SPL: it can be a neophyte who is looking for a solution to extract information from a dataset as well as an expert who wants to check or learn dependencies among dataset properties, algorithms, targeted platforms and user objectives. They want to use a system helping them to master the variability, i.e., to express their requirements and get their envisioned ML workow. Both of these do not know about FMs and may need complementary information like examples of uses of the algorithms, details about the algorithm author or implementation, etc. Thus, the visualization of a FM as in standard tool is not adapted. Creating a user interface dedicated to the SPL is also problematic knowing the changing nature of the system. At the same time, in a agile process, we need to test the SPL with users in order to align it with their needs, which are hard to identify a priori.
Some owcharts have been designed to give users a bit of
a rough guide on how to approach ML problems [
A user willing to con gure a new ML work ow will do so
through a web-based con guration interface2. The process
requires at least the following steps, as visible on the left of
gure 1:
(a) The Graphical User Interface (GUI) requests display
metadata on the Feature Model. Metadata associate each
unique feature of the model with descriptions, references
or other artefacts aiming to help non-experts in their
choices. Figure 2 shows a screenshot of our GUI. Here,
user is presented with the choice of its main objective,
in the form of questions.
(b) Once the FM is loaded and displayed properly with the
metadata, the user con gures the underlying FM by
responding to questions. The Feature Model
component in the gure exposes a web-service that allows for
con guration on any FM, through the use of SPLAR's
API [
Though it is not described here, depending on user's preference, the generated work ow would either be provided to the user or directly executed by the target platform. 3.1.2
We will now focus on the introduction of new ML algorithms in the SPL. Such an action has impacts on several parts of the system: - SPL: At least a new feature representing the algorithm 2The interface is accessible at http://rock ows.i3s.unice.fr should be added in the FM; - GUI: Display metadata should be updated to re ect this new feature in the GUI; - Generation: If we can execute the algorithm, the generator should be updated to allow it either through code or a reference to the corresponding element in target platform(s); - Experiments should be made with this new algorithm in order to compare it to the other known algorithms. Currently, the properties that are considered are accuracy of the results, execution time of the work ow, and memory usage. If experiments can be achieved, i.e., Experiments module has access to algorithm execution and results, the SPL needs to be updated with performance information for this algorithm but also for all the algorithms whose ranking has changed.
The central component SPLConsistencyManager's role is to ensure that all required changes are made across the whole system. Through the present scenario, we describe how this component handles the impacts mentioned above. (1) As an External Developer wants to add a new algorithm, the GUI presents her with proper information that she needs to provide. Because this information is meant to change as the system encompasses more possibilities of Machine Learning, it should be easy to change. Hence, the SPLConsistencyManager provides the GUI with the information needed to add a new algorithm in the tool.
The GUI presents then a generated form to the user. (2) Once the External Developer has provided all information on the algorithm, it is sent to the Consistency Manager and dispatched among the other components. (3) If the External Developer provided code to execute the algorithm, the Experiments component is requested to run tests on the algorithms, in order to nd its performance. This component is described more precisely in the next section. (4) Once experiments are nished, the manager analyses the results to nd whether any inconsistency was found between the information provided by the user and the results. If not, the algorithm can be added both to the Feature Model and base of currently supported algorithms. (5) Finally, once the feature has been added to the FM, its display metadata can be lled with the information provided by the expert. 3.2
As we want even non-expert users to be able to get the
appropriate algorithm for their need, we chose to express
higher level goals such as best accuracy or quickest execution
time in the FM. This representation allows to lter out a
number of algorithms by o ering users with trade-o s over
these goals. In a second step, or a more advanced mode,
using other models such as requirements engineering Goal
Models is considered. In combination with our FM, it could
allow during con guration to present the user with more
precise expected values for her non functional goals [
To be able to express knowledge such as "best", "average" or "worst" accuracy, we need an appropriate way to compare algorithms on a given problem. This ranking among algorithms is computed by our Experiment module. The component runs each known algorithm on the available compatible datasets and stores its performance on the di erent properties for each dataset. On top of that, it will transform the datasets with a set of available pre-processing operations and test again each algorithm with those new sets.
Algorithms results on similar datasets are then compared to one another to get a result similar to the one visible on gure 3. Though it will not be discussed here, we have de ned an algorithm based on classical ML and statistical methods to pull o this comparison and regroup datasets in so-called dataset patterns. This knowledge can then be pushed in the FM in the form of constraints linking a functional objective, an algorithm, a dataset pattern and the ranking for each of the properties, also depending on the possible preprocessings for this dataset.
As presented in section 3.1.2, the Experiment component is driven by the SPL Consistency Manager, that is charged to start experiments, gather and validate results before incorporating them in the SPL. As new algorithms are added, all experiments do not have the need to be executed again, however the ranking of the algorithms must be updated to take in account the newest algorithm. 4.
This section discusses how we allow users to provide information about new algorithms, how we use it to update the system, and how we can ensure consistency despite the multiple impacts of these changes. 4.1
In order to make the system as accessible as possible, additional information on features must be set, such as descriptions or examples, as well as closed questions that will be asked to the end user during con guration. This metadata on the practical features is handled in a dedicated metamodel AlgorithmDescriptionMM and used to build the GUIs that are presented to the end users and external developers. The model is brie y described in subsection 4.2. 4.1.3
Information such as the accuracy ranking of algorithms according to dataset patterns is managed by the Experiment component. The expected format of this data is designed in a dedicated meta-model ExperimentPropertyMM. It evolves as we gain knowledge and experience. 3http://ec2-52-32-1-180.us-west-2.compute.amazonaws. com:8080/SPLOT/sxfm.html 4An excerpt of the meta-model used for both FM and con guration de nition in the library, can be found in https://github.com/FMTools/sxfm-ecore/blob/master/ plugins/sxfm/model/sxfm.png
Data scientists adding their algorithms in the system need to provide at least: - The high level ML objective the algorithm can be used for: classi cation of data, prediction of numerical values (regression), anomaly detection, etc.; - Properties of the algorithm in regards to input data: which data types the algorithm supports, can the algorithm handle missing values in the data, etc.; - A description of the algorithm, examples of its use, references to publications or web pages describing the algorithm. As described in section 4.1.2, those will be displayed in the con guration interface; - If possible, code that will allow us to run the algorithm in our tool, so that we can both compare it to the others algorithms and provide executable work ows to end users.
Through the de nition of theses elements in the
AlgorithmDescriptionMM meta-model, a form is automatically
generated and presented to the domain expert. Thus, the
model can be extended to add new properties for the
algorithms and will be automatically handled by the GUI.
However the impact of such changes on the FM still has to
be handled by the SPL manager. We do not know if tools
such as the one described in [
Even though we have de ned a single FM for ROCKFlows, we put a focus on separating concerns in it. The tree is currently separated in 4 sub-trees handling: input data description, user objectives, the processing algorithms, and expected properties of the generated work ow. This separation of concerns provides a rst level of modularity for the model.
Linking Domain artefact and FM structure. Metadata external to the FM itself de nes particular points in the FM where feature can be inserted. Only the sub-trees that need to be modi ed are considered. In our example, we add all algorithms responding to the classi cation problem in the same sub-tree. This mechanism enables us to extend the model cleanly, and abstract ourselves from the exact hierarchy of the features. So, adding new class of algorithms or modifying the structure of the FM can easily be achieved. Once the feature for the algorithm has been created, additional constraints need to be de ned between the algorithm and other features, in particular those describing input data.
Generating domain constraints. Only certain types of constraints must be de ned among those di erent sub-trees. For instance algorithms can de ne constraints towards input data, such as "SVM implies Numerical Data" but never the other way around. However such a constraint only applies in a work ow if no pre-processing is used. So, this constraint has to be transformed to express a constraint depending on the pre-processings that can be applied. The complexity of these cases, their multitude and the frequency of evolution lead us to encapsulate the generation of those constraints in dedicated operators, working on given ensembles (e.g., preprocessing set that returns numerical Data). They also introduce features that are hidden to the end user, allowing us to tame this complexity.
It is interesting to note that, depending of the semantics associated to the features, di erent constraints should be generated. For instance, if an algorithm cannot deal with missing values, a constraint "algo excludes missing values" needs to be generated. In the other case, a constraint "algo implies missing values" should never be generated because such an algorithm can still be used even if no missing value is present in the input data. Like previously, this higher level knowledge of the features is de ned in our metadata. It allows us to ensure that all necessary constraints are properly de ned for all algorithms we add into the SPL.
In this paper, we have outlined some of the di culties related to building and evolving a SPL for ML work ows. To handle the line's complexity and evolution, we have proposed an architecture organized around a set of meta-models and transformations encapsulated within services, A large number of complementary perspectives are considered, both on mechanisms to build the SPL and on the business approach of ML.
The complexity we are facing requires an agile and
pragmatic approach in which users are given the opportunity
to provide feedback during the early stages of the project.
The necessity for evolution of the system leads us today to
consider moving towards approaches focused on the reuse of
existing components [
The domain of ML is currently booming with propositions
for algorithms, distributivity of execution and the opening to
a larger audience. We consider to extend our approach to
integrate the necessary parameters for distributing work ows'
execution as well as proposing deep learning work ows. A
longer term question is to target Scienti c Work ow
Management systems, allowing so to explore data driven
execution through transformations targeting the speci cation
interchange language called WISP [
Finally, we wish to allow the end user to express her needs in business terms, through an approach similar to IBM Watson Analytics5. It is a question of integrating the state of the art practices in our modelisation, without losing the power of our evolutionary approach, driven by experiments.