A common work ow in science and engineering is to (i) setup and deploy large experiments with tasks comprising an application and multiple parameter values; (ii) generate intermediate results; (iii) analyze them; and (iv) reprioritize the tasks. These steps are repeated until the desired goal is achieved, which can be the evaluation/simulation of complex systems or model calibration. Due to time and cost constraints, sweeping all possible parameter values of the user application is not always feasible. Experimental Design techniques can help users reorganize submission-executionanalysis work ows to bring a solution in a more timely manner. This paper introduces a novel tool that leverages users' feedback on analyzing intermediate results of parameter sweeping experiments to advise them about their strategies on parameter selections tied to their SLA constraints. We evaluated our tool with three applications of distinct domains and search space shapes. Our main nding is that users with submission-execution-analysis work ows can bene t from their interaction with intermediate results and adapt themselves according to their domain expertise and SLA constraints. interactive optimization work ow; parametric sweeping application; design of experiments; high performance computing; user expertise; SLAs
Evaluation of test scenarios and model calibration are common in several industries including nance, aerospace, health, and energy. Normally users run applications that contain a set of parameters, each able to assume a broad range of values. These applications are known as parameter sweeping applications or parametric applications. In this context, it is necessary to run several application instances varying their parameter values to nd a combination that meets a given speci cation or optimization criteria.
A popular practice in this context is to employ Design of
Experiments (DOE) techniques [
Over the years, optimization methods have been created
to solve complex problems in industry and academia.
However, fully automatic solutions face some obstacles to obtain
satisfactory results due to the following reasons [
Generally, it is prohibitive to run application instances with all possible parameter values due to time and cost constraints|cost, in particular, becomes a key factor considering execution of these applications in outsourced environments (e.g., public clouds). Additionally, optimization techniques can be adopted to suggest parameter values in order to reduce the number of evaluated scenarios and consequently reduce the experiment costs. Therefore, SLA-aware prediction methods and mechanisms to suggest parameter values would help engineers and scientists to evaluate parametric applications.
In this work, interactive optimization methods are proposed to take advantage of human expertise/creativity and processing power to solve optimization problems. We study how the decision maker's experience can be leveraged to nd solutions for parameter sweeping applications. The analyst's decisions can be impacted by the DOE analysis that shows which parameters have more impact on the output. For instance, an experiment explorer tool can show which parameter values cause more impact on experiment results and the user can select suitable parameter value combinations based on this information.
We introduce a tool, called Copper (Cognitive Optimizer and Parameter Explorer), to help users in the evaluation of search spaces considering human feedback and domain expertise. As new samples are evaluated, Copper updates the impact of each parameter value on the results and generates (1)
SLA - response time - cost - result quality analyst
(5) analyze intermediate results
track SLA
modify search strategy
new samples for evaluation by using derivative free
optimization (DFO) methods [
A tool to assist users in the evaluation of scenarios with parameter sweep applications using human feedback and domain expertise (x 3); Clustering methods to evaluate user behavior and propose strategy changes in case of imminent SLA violations (x 3.2); Evaluation of the tool using three applications from di erent domains and search space shapes (x 4). 2.
This section presents research e orts related to our work and an overview of the problem investigated in this paper. 2.1
Assisting users in executing applications has been a research goal of several groups. Related to our work there are e orts from multiple areas: computational steering, humanguided search, work ow management, design of experiment, interactive optimization, among others.
Computational steering [5, 11{13, 22, 32] aims at
providing users with tools that enable parameter recon guration
while experiments are in progress. Parker and Johnson [
An important research topic related to our work is
optimization assisted by humans [
Several e orts in the design of experiments happened over
the last years. Kleijnen et al. [
Our research is based on existing work of these aforementioned e orts in order to build a tool able to provide feedback to the user on how her interaction with the parameter sweeping experiments impacts the de ned time (and cost) constraints.
Problems from several industries are tackled by users, also named analysts, running complex computer simulations and optimizers that constitute a software with a set of parameters, where each parameter can receive di erent values. We consider applications with n parameters and each parameter has a nite discrete domain Di where i 2 f1; ; ng. A function f : D1 Dn ! R is adopted to evaluate the quality of parameter values and is speci c for each parametric application. The considered optimization problem P is described as follows:
P max f (x) x 2 D1
Dn
We assume the user is able to generate f (x) according to the simulation output. For instance, if the output g(x) of a simulation process is a real value that should be minimized, then f (x) can be assigned as f (x) = g(x) and the problem de nition remains the same. If the parametric application a generates multiple outputs (e.g., a : D1 Dn ! O1 Om), a wrapper function h : O1 Om ! R should be employed to represent the behavior of f (i.e., f (x) = h(a(x))).
For instance, suppose the analyst wants to calibrate a simulation model that generates a list of predicted values over a given period. In this case, the output of the parametric application corresponds to a list of values, one for each time instant. A wrapper function (e.g., Root Mean Squared Error) can be employed to generate a single quality value to compare the di erence between the generated values and a reference series.
Running all possible values for each parameter can be unfeasible|even with large high performance computing machines. Therefore, users need strategies to select a subset of values for each parameter that covers parts of the exploration space that will give them enough information to answer their questions. Such questions can be: what is the best set of parameter values that provides a (close-to-)optimal solution or what are the parameter relationships that have more in uence on the phenomenon being analyzed. The strategies for parameter-value selections can be de ned by Design of Experiments (DOE), which corresponds to a systematic method to determine the relationship between parameters (also known as features) a ecting a process and the output of that process.
Analysts may have initial insights about the regions of search space that lead to good results. However, these insights may be wrong, and can change as long as intermediate results are evaluated. For several problems, the codi cation of human insights is not feasible due to the following reasons. These insights come from observations of historical data which may not be available. Even when this data is available, the time to analyze and codify the human insight may not be a ordable. Human insights can be based on domain expertise, and the analyst has no experience/time to explain or codify this knowledge. Therefore, a mixed strategy that combines human expertise and optimization approaches may be useful to help analysts in the evaluation of complex optimization problems.
In this paper, we introduce a tool and a set of techniques
to help scientists and engineers to properly prioritize their
experiments, which can comprise traditional HPC jobs, or
jobs following high throughput computing [
Figure 1 presents the submission-execution-analysis workow investigated in this scenario. In Step 1, the user submits to Copper a request to run an application with her SLA constraints, which can contain deadline, cost, or result quality restrictions. In this work we focus on deadline constraints. Others restrictions (e.g., cost or energy) can be mapped into the time it takes to execute jobs on the computing infrastructure. In a scenario where Copper is executed in the cloud, cost may become an important factor to be considered. Users may decide to change their execution strategies so as to meet cost constraints based on the cloud provider instance prices and charging model having job execution time and resource requirements as input. Another example, an energy utilization rate can be employed to de ne the maximum amount of energy to run the experiments depending on the execution time. The user also describes in the request the parameters and possible values to run her application, the job submission strategy (e.g., random search or range of parameter values), and the speci cation of the computing environment (e.g. number of processors and their con guration). In Step 2, Copper requests the resource management system to con gure the environment and creates compute jobs corresponding to parameter-value pairs.
Once jobs are executing, Copper monitors the state of the running/completed/pending jobs to predict SLA violation (Step 3). If SLA is about to be violated, a new strategy is de ned to run the pending jobs. If that is the case, a noti cation of SLA violation and a new strategy is sent to the user (Step 4) who, based on this information, can act on the strategy to execute her jobs (Step 5). Meanwhile, the analyst can also manually change job priorities based on her domain expertise and on her trust on the ongoing strategy (Step 6). As Copper exploits the analyst expertise and her feedback is not instantaneous, the job priority asynchronously changes to avoid unnecessary delays in triggering new jobs. This is particularly important as Copper is designed to manage long execution time jobs while users follow their submission-execution-analysis work ows.
The speci c problems tackled in paper are therefore: How to suggest jobs for execution to the analyst and exploit her expertise to get faster problem resolution? How to predict SLA violations based on clustering strategies and pending/running/completed jobs? How to suggest to the analyst to change her strategy to avoid SLA violations based on intermediate results and her expertise?
The proposed solution comprises the Copper tool, a computing infrastructure, and a web client. Copper is a service, which can be executed in the cloud or on-premise, responsible for learning the user strategy, suggesting strategy changes in case of probable SLA violations, and proposing job executions to the analyst. A simple adapter should be created on the computational infrastructure to connect existing user applications and Copper. This component is speci c to each application and allows Copper to trigger job execution by using REST commands. It is also responsible for sending intermediate results and the list of parameters with their possible values to Copper.
Figure 2 presents an overview of the Copper architecture.
The manager orchestrates the Copper operation, reads the
results, and stores them in a database as soon as jobs are
executed. The SLA evaluator and sampler receive those
intermediate results to update the selected DFO method state
and cluster models. In order the produce new job candidates
(x 3.1), the sampler utilizes DFO strategies such as
generalized greedy randomized adaptive search (GRASP),
general pattern search (GPS), or particle swarm (PS)
optimization [
For SLA violation prediction (x 3.1), the SLA evaluator employs clustering methods to estimate the number of pending jobs according to the user strategy. This number is compared to the number of jobs that can be executed based on the remaining time/budget de ned in the SLA contract. If the user has no resources to evaluate all pending jobs according to a given strategy, Copper sends a warning and suggests the user to change her approach. Algorithm 1: sampling - proposed sampling method. Input : M , P , , n
Output: samples 1 solution construction(M , , P ); 2 samples localSearch(M , P , n, solution); 3 return samples; 4 5 end 6 return solution; Algorithm 2: construction - algorithm for creating a solution candidate.
Input : M , P
Output: solution 1 solution ?; 2 foreach p of P do 3 RCL fe 2 listValues(p, M ) j q(e) qmin + (qmax qmin)g; solution[indexOf(p)] getRandomElement(RCL); Algorithm 3: localSearch - nd neighbors of a given solution.
Input : M , P , n, solution
Output: solutions 1 solutions ?; 2 for i 1 to n do 3 solutions[i] getNeighbor(M , P , solution); 4 end 5 return solutions; 3.1
Combinatorial optimization corresponds to a class of
problems that consist of nding an optimal solution from a
nite set of objects [
The GRASP sampling method is presented in Algorithm 1, which comprises two phases: construction and local search. The construction phase builds a feasible solution for each algorithm execution (Line 1). Next, the local search generates the neighbors of the previous solution (Line 2). Finally, the set of results is returned (including the initial solution). The algorithm inputs are the following: the DOE model (M ), the list of parameters (P ), the greediness level ( ), and the 1Samples and jobs represent an instance of the application with a set of parameter values. These terms are used in the paper interchangeably. number of generated samples (S). The greediness level ( ) is a real value contained in [0; 1] and represents how random is the selection of parameter values according to their estimated quality. If the greediness level is close to one, only high-quality parameter values can be selected. On the other hand, values close to zero allow the selection of a broad range of values including low-quality values.
The construction phase is presented in Algorithm 2 and nds a candidate based on the greedy heuristic. Initially, an empty solution is created (Line 1) to return the list of values for each parameter of the result solution. For each parameter p (Line 2), a reduced candidate list (RCL) is created and this list is randomly sampled to get a value that composes the nal solution.
We use the full factorial DOE method [
Whenever part of the search space has enough evaluated samples to apply the full factorial, the DOE model is updated. Next, a component of RCL is randomly sampled to compose the nal solution (Line 4), and nally the solution is returned (Line 6).
The last phase of the sampling algorithm corresponds to the local search (Algorithm 3). In this phase, n samples close to the candidate solution found in the previous phase are selected. The getNeighbor() function is employed to get a solution candidate neighbor by using euclidean distance between samples. 3.2
The method to predict SLA violations is based on
clustering techniques. We identify the clusters of the search space
that the analyst is interested in and compare the number of
pending jobs in these clusters to the number of jobs that can
be executed according to the SLA contract. In this work, we
use Density-Based Spatial Clustering of Applications with
Noise (DBSCAN) [
Figure 4 presents the basic idea of the proposed solution. Suppose the search space is composed of two parameters and each particular combination (v1; v2) represents a job that will be submitted to the computational infrastructure. v1 and v2 are possible values for parameters 1 and 2 respectively. As long as the analyst submits new jobs, the clusjob not executed job executed + good result job executed + bad result best results (not executed yet) interest regions e c a p s h c r a e s ) s e u l a v ( 2 tr e e m a r a p parameter 1 (values) tering approach identi es the clusters of the search space that provide good results. The quality of job results can be previously de ned by using a threshold value or the analyst can qualify each job result (e.g., good or bad) whenever it becomes available.
In this example, the analyst is searching the best result in three clusters of the search space. The clusters are identi ed by the clustering algorithm and the number of pending jobs is counted. If this number is higher than the remaining jobs de ned by the SLA the user should change her strategy. In this case, the number of pending jobs is 15 (including the best result). If the analyst can run 15 jobs, at most, jobs due to SLA restrictions, Copper advises the analyst to change her strategy.
The strategy can be de ned considering exploration and exploitation aspects. We de ne exploitation level (ex) as the proportion of pending jobs that will be executed considering samples inside the detected clusters. For instance, if the exploitation level is 1:0 all the pending jobs will be executed taking the parameters contained in the clusters of interest. If the exploitation level is 0, pending jobs in not explored clusters (outside the clusters) will be given higher priority.
Considering all possible pending jobs in all clusters (i.e. ex #pending jobs), the user de nes which ones of these jobs will execute for each cluster. The default approach utilizes weighted average according the mean quality of evaluated jobs inside the clusters. For jobs outside the clusters (i.e. (1 ex) #pending jobs), the default GRASP strategy remains the same (x 3.1).
In this section, we evaluate the Copper e ectiveness in helping users run optimization and computer simulations under SLA constraints. We rst describe three applications served as use cases and metrics analyzed to understand the bene ts of the assistant. In particular, we are interested in analyzing three major aspects considering submissionexecution-analysis work ows: (i) the interaction of Copper with analysts of di erent expertise levels; (ii) how the analyst con dence on Copper a ects the evaluation of user applications; (iii) the impact of Copper SLA violation predictions on user experiments. 4.1
This section presents an overview of the setup to evaluate Copper. 4.1.1
Here is a short description of the applications we used in the evaluation with an overview of their input parameters and output users are interested in.
1. ifm: The Integrated Flood Model (IFM) [
2. schedsim: Scheduler Simulator (SchedSim) is a simulator to assist in the creation of policies to manage High Performance Computing clusters. It accepts a variety of parameters including number of processors in the cluster, partitions, and scheduling algorithms. Tuning the scheduler and cluster properties to meet client business goals is not a trivial task and several scenarios must be executed to achieve that. For the evaluation, we had the following input: (i) a workload containing historical data of jobs submitted to a cluster; (ii) a xed number of cluster processors; and (iii) two variable partitions. We wanted to know what was the partition sizes and which jobs (requested time and allocated processors) should go to such partitions having the following goals: (i) minimize the overall job response time and (ii) small jobs (under 1h requested time) should wait no more than 3 hours in the cluster waiting queue.
3. mazerunner: Maze Runner is an in-house created game in which simulated runner has to nd the way out of a maze. The objective of the game is to nd a runner con
guration (e.g., velocity, movement strategy) that leads to a shorter time to nd the maze exit. There are enemies in the maze, and the runner needs to escape from them. Additionally, the runner can slip on obstacles and stay vulnerable for some time depending on her velocity. This game is simple and intuitive and was created to demonstrate concepts of the proposed assistant for a broader audience.
The search spaces related to the applications used in this work are illustrated in Figure 5. Multiple parameter combinations are represented in a single axis and each square corresponds to a parameter combination. The quality of a parameter con guration is represented as a given color (dark colors represent better results). For instance, the top-right square of Figure 5c presents the (V1 1, V2 1, V3 2, V4 3 ) parameter con guration of maze runner. In this work, we assume the output of parametric applications as black box functions, therefore we will not enter in details about the meaning of parameters and their values.
By observing the search spaces of the three applications, SchedSim presents a well behaved variation of the output when the parameters values change. A similar behavior happens to the IFM search space. However, for higher values of P1 the output presents no variation. This ` at' search space region reduces the Copper capacity to predict good parameter values. Regarding the Maze Runner search space, con gurations whose parameter P1 = V1 3 provide better results. However, the output variation does not have the same gradual characteristics as in the other applications. 60 .10 .60 .10 .60 .10 .60 .10 .60 .10 .60 .10 . 10 .60 .10 .60 .10 .60 .10 .60 .10 .60 .10 .0 _1 _2 _3 _4 _5 _6 _7 _8 _9 _10 _11 _12 _13 _14 _15 _16 _17 _18 _19 _20 .660 .071 .760 .810 .860 .910 .960 .10 .60 .101 .160 .210 .2 3 3 4 4 5 5 6 6 7 7 8 860 .910 .960 .10 .60 .101 .160 .210 .260 .310 .360 .4 4 5 5 6 6 7 7 8 8 9 96 2V 2V 2V 2V 2V 2V 2V 2V 2V 2V 2V 2V 2V 2V 2V 2V 2V 2V 2V 2V { 1 _ 3 V 2 _ 3 V {
V4_2 1 _ 3 V 2 _ 3 V { V4_3 1 _ 3 V
P4 2 _3V P3
We consider software agents to simulate the human
behavior [
The second evaluation is performed to nd the best result considering a limited number of jobs (x 4.3). Therefore, agents interact with Copper to obtain strategy suggestions if the ongoing experiments tend to extrapolate the number of jobs established in the SLA.
In this evaluation, we assess the number of jobs that must be executed to nd an optimal solution. The agents have insights about which parameter values may result in optimal solutions. Therefore, we divided the agents into three categories based on the insight quality level. The rst group of agents are related to users that have a previous intuition about good parameter values, and this intuition may lead to bad solutions (bad insights). The second group contains agents that have previous ideas about the parameter values and these values result in proper solutions (good insights).
The last group of agents represents users with no insight about the clusters of search space that may contain the optimal solution (no insights).
For each agent insight level, we vary the user con dence level on the job suggestions that come from Copper and the capacity of an agent to learn from intermediate results. The con dence level determines the percentage of jobs that are submitted taking into account Copper suggestions. For instance, if the user submits 100 jobs and the con dence level is 40%, then 40 jobs are submitted based on Copper recommendation and the rest is triggered based on the agent intuition. This intuition corresponds to a subset of search space in which the analyst believes the optimal result is contained. Agent does not learn from intermediate results
Agent learns from intermediate results 10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100
Percentage of samples from advisor (agent confidence on Copper - confidence level) We evaluate agents that learn or not from intermediate results. When the agent learns from intermediate results, it increases the probability of selecting a given parameter value if this value leads to good intermediate results and decreases the likelihood of choosing a given value if this parameter value leads to bad results. As the focus of this experiment is to present how the framework takes bene ts from users recommendations, we do not explore in details here how the agents learn from intermediate results.
Figure 6 shows the percentage of jobs that are submitted
according to Copper recommendation (con dence level) for
all applications as a function of the number of jobs that must
be submitted to nd an optimal solution. As the proposed
sampling algorithm is stochastic, multiple experiments were
performed to nd a 95% con dence interval of the mean
number of jobs to nd an optimal result. The bootstrap
method [
The rst, second, and third rows of Figure 6 are related respectively to IFM, SchedSim, and Maze Runner results.
The three columns of the same gure, shows the experiments for agents with bad, good and no previous insights for these applications. For agents with bad insights, as the con dence level on recommendations made by Copper increase, the number of jobs to nd an optimal result reduces.
Then, users with bad insights may have bene ts when using Copper. For Agents that do not learn from intermediate results, SchedSim results present the most signi cant di erence between relying on Copper to provide the jobs parameters or not. Increasing the con dence in suggestions made by Copper also reduces the number of jobs to nd the optimal con guration in IFM and Maze Runner. However, this di erence is less signi cant than the previous one. As the SchedSim search space has interesting characteristics about the objective function growing related to the input parameters, the proposed sampling algorithm takes less steps to nd the optimal parameter con guration.
For bad insight agents that learn from intermediate results, there is a signi cant di erence between learning from intermediate results or not. For example, bad insight agents for SchedSim with 10% con dence level presents a big di erence between learning from intermediate results or not. This behavior can be explained by the interesting characteristics of ShedSim search space. Figure 5 shows that low values of SchedSim parameters P1, P2, and P3 lead to good results.
The agents learn that these values lead to good results and increase the probability of choosing combinations of these values.
For agents with good insights, the number of jobs to nd the optimal solution increases with the agent con dence level. This result is the opposite of the previous one and indicates that users with good insights may have a signi cant reduction of steps to nd an optimal solution. Agents with good insights about the behavior of SchedSim and Maze Runner present a regular growth in the number of jobs to
nd the optimal solution when the con dence level on Copper increases. However, IFM does not present this characteristic as there is an abrupt increase from 90 to 100 percent of con dence level. Whenever 100% con dence level is employed, the agent only takes Copper suggestions to submit new jobs. In this case, Copper needs more jobs to select parameters outside the at area of IFM search space. As 90% con dence level agents submit 10% of jobs based on user insight. The DOE model is updated considering the non- at areas (agent good insight) and speeds-up the optimization process. For agents with good insights, there is no expressive di erence between learning from intermediate results or not. As the number of jobs to nd the optimal solution is reduced, the number of samples is not su cient to update the user preference and signi cantly a ect the result.
For agents with no insights, the agent's e ciency is impacted by the con dence level. Whenever the con dence level on Copper increases the results get better. However, this impact is less notable to no insight agents when compared to the results of bad insight agents. Similarly to the bad insight agents, there is a di erence between learning from intermediate results. 4.3
This set of experiments utilizes the same agents to nd the best e ort solution but with limited number of jobs that can be executed due to SLA constraints. We assume that all jobs take the same time to execute, and the SLA constraints correspond only to time requirements. Other conditions such as costs or energy consumption can be used by mapping these requirements into the number of submitted jobs. The quality of a result is given by the ratio of the result found and the optimal result.
For each experiment set, we changed the maximum number of jobs that can be submitted and the exploitation level. The exploitation level is de ned as the proportion of pending jobs that will be submitted considering samples inside the clusters de ned in x 3.2. As the output of an experiment is not deterministic, each bar in the chart presents the mean value and the 95% con dence related to 500 runs for each maximum jobs, and exploitation level pair. In these experiments, the proposed clustering method is applied when the number of submitted jobs reaches 70% of the maximum number of jobs and the agent accepts the Copper recommendation for all experiments. The restricted experiment results for IFM, SchedSim, and Maze Runner are respectively presented in Figures 7, 8, and 9.
The IFM experiment set examines the best e ort scenario with a limited number of jobs. In this case, instead of preferring exploration or exploration of sampling space, it is preferable to adopt a mixed strategy to nd good parameter values. For instance, for 50 jobs the experiment with exploitation level of 33% presents better quality when compared to the others. Similar to the other experiment sets, the variation of results quality is higher when the number of jobs is reduced (in this case 50 jobs). Based on these results we can conclude that for this experiment it is preferable to adopt a mixed strategy to evaluate the search space. Therefore, experienced analysts can take advantage of combined search strategies of exploitation/exploration to achieve better results. For the SchedSim results (Figure 8), it is possible to observe that for a reduced number of jobs (i.e., 30) the exploitation level has a signi cant in uence on the output quality. The impact of exploitation level decreases as the number of submitted jobs increase because the probability of nding better results are higher if more jobs are executed. For this experiment set, it is preferable to explore the search space instead of concentrating the submitted jobs in the cluster areas. This result suggests that the best results are spread through the solution search space.
Figure 9 presents the results of experiments with a restricted number of jobs related to Maze Runner. Di erent from the previous application, whenever the exploitation level is higher, the results have better quality. On the other hand, when the SLA constraint is relaxed (more jobs are allowed) the impact of exploitation levels becomes less important. We can conclude that the best results are grouped in the solution search space, then strategies that exploit intermediate results can be adopted to get better results.
Several areas in science and engineering rely on many executions of a software system with di erent values for each supported parameter. These executions are responsible for evaluating test scenarios and calibrating models. Normally, users follow a work ow where they setup a set of experiments, generate intermediate results, analyze them, and reprioritize jobs for the next batch. Reprioritization is a key step as executing all possible values for all supported parameters is usually not feasible, especially under cost and deadline constraints determine by strict SLAs.
This work introduced a tool to help users in executing their software systems by exploring their domain expertise and design of experiment techniques. By learning user strategies on sweeping parameters, our tool is able to detect if the ongoing strategy is able to meet SLA requirements and suggest how jobs can be reprioritized in case an SLA violation is predicted to happen while users run their submissionexecution-analysis work ows. Our main lessons while developing the tool and evaluating with three applications are: (i) it is important to facilitate user interaction with an automated design of experiment technique as users may bring domain expertise that can reach desired results faster; (ii) as the desired solution may be found by mixing exploration and exploitation strategies (e.g. breadth- rst search and depthrst search), users can bene t from a tool that can identify if the ongoing strategy is correct and can meet SLA constraints; (iii) evaluations with a restricted number of experiments are highly impacted by Copper suggestions. Then, the proposed tool and methods are especially suitable for evaluation of parametric applications with tight deadlines.
We would like to thank Miguel Paredes Quinones and the anonymous reviewers for their valuable comments. This work has been partially supported by FINEP/MCTI under grant no. 03.14.0062.00.