MapReduce is a popular programming model for distributed processing of large data sets. Apache Hadoop is one of the most common open-source implementations of such paradigm. Performance analysis of concurrent job executions has been recognized as a challenging problem, at the same time, that it may provide reasonably accurate job response time at signi cantly lower cost than experimental evaluation of real setups. In this paper, we tackle the challenge of de ning MapReduce performance models for Hadoop 2.x. While there are several e cient approaches for modeling the performance of MapReduce workloads in Hadoop 1.x, the fundamental architectural changes of Hadoop 2.x require that the cost models are also reconsidered. The proposed solution is based on an existing performance model for Hadoop 1.x, but it takes into consideration the architectural changes of Hadoop 2.x and captures the execution ow of a MapReduce job by using queuing network model. This way the cost model adheres to the intra-job synchronization constraints that occur due the contention at shared resources. The accuracy of our solution is validated via comparison of our model estimates against measurements in a real Hadoop 2.x setup. According to our evaluation results, the proposed model produces estimates of average job response time with error within the range of 11% - 13.5%.
Information systems ! MapReduce-based systems; Theory of computation ! Parallel computing models;
MapReduce-based systems are increasingly being used for 2017, Copyright is with the authors. Published in the Workshop Proceedings of the EDBT/ICDT 2017 Joint Conference (March 21, 2017, Venice, Italy) on CEUR-WS.org (ISSN 1613-0073). Distribution of this paper is permitted under the terms of the Creative Commons license CC-by-nc-nd 4.0 large-scale data analysis applications. Minimizing the execution time is vital for MapReduce as well as for all data processing applications, and accurate estimation of the execution time is essential for optimizing. Therefore, we need to build performance models that follow the programming model of such data processing applications. Furthermore, a clear understanding of system performance under di erent circumstances is key to critical decision making in workload management and resource capacity planning. Analytical performance models are particularly attractive tools as they might provide reasonably accurate job response time at signi cantly lower cost than simulation and experimental evaluation of real setups.
Programming in MapReduce requires adapting an
algorithm to two-stage processing model, i.e., Map and Reduce.
Programs written in this functional style are automatically
parallelized and executed on computing clusters. Apache
Hadoop is one of the most popular open-source
implementations of MapReduce paradigm. In the rst version of
Hadoop, the programming paradigm of MapReduce and the
resource management were tightly coupled. In order to
improve the overall performance as well as the usefulness and
compatibility with other distributed data processing
applications, some requirements were added, such as high cluster
utilization, high level of reliability and availability, support
for programming model diversity, backward compatibility,
and exible resource model [
In this paper, we address the challenges of de ning
accurate performance models for estimating the execution time of
MapReduce workloads in Hadoop 2.x. We analyzed the
approaches for Hadoop 1.x and also the architecture of Hadoop
2.x and we decided to base our model on performance model
proposed for the rst version of Hadoop in [
Contributions. The main contributions of this paper can be summarized as follows:
By analyzing the architecture of Hadoop 2.x, we identify cost factors that can potentially a ect the cost of the MapReduce job execution.
We theoretically de ne a MapReduce cost model for Hadoop 2.x that captures the precedence of di erent tasks of MapReduce jobs as well as the synchronization delays due to shared resources.
We evaluate the accuracy of our cost model by implementing the cost estimation prototype and comparing the obtained estimates with real MapReduce executions. 2.
We observe two groups of approaches for analyzing the performance of MapReduce job for the rst version of Hadoop. All performance models described in Subsection 2.1 are static, they do not take into account the queuing delays due to contention at shared resources and the synchronization delays between di erent tasks. In Subsection 2.2, we introduce two most common approaches for constructing dynamic performance models for parallel applications and describe a performance model proposed for Hadoop 1.x that takes into consideration the queuing delays. 2.1
There are signi cant e orts and important results towards
modeling the task phases in order to estimate the execution
of a MapReduce job in Hadoop 1.x. Herodotou proposed
performance cost models for describing the execution of a
MapReduce job in Hadoop 1.x [
There has also been an e ort of de ning the lower and
upper bounds for job completion time and resource allocation
to a job so that it nishes within the required deadline. In
[
For estimating the job completion time authors applied the Makespan Theorem for greedy task assignment, which allows to identify the upper T Up and lower bounds TJLow J for the task completion time as a function of the input dataset size and allocated resources. According to the research TJAvg = TJUp+2TJLow is the closest estimation of job completion time T. It was observed that the relative error between the predicted average time T Avg and the measured J job completion time is less than 15%, and hence, the predictions based on T Avg are well suited for ensuring the job
J completion within the deadline. Nevertheless, this model has signi cant limitations that do not allow us to apply it to the second version of Hadoop. As in Herodotou's cost models, the proposed model uses a xed amount of slots per map and reduce tasks within one node.
There has also been an attempt of evaluating the impact
of task scheduling on system performance. However, current
schedulers neither pack tasks nor consider all their relevant
resource demands. This results in fragmentation and
overallocation of resources and, as a consequence, it decreases
noticeably the overall performance. Robert Grandl et al.
present in [
The main challenge in developing the cost models for MapReduce jobs is that they must capture, with reasonable accuracy, the various sources of delays that job experiences. In particular, tasks belonging to a job may experience two types of delays: queuing delays due to contention at shared resources, and synchronization delays due to precedence constraints among tasks that cooperate in the same job - map and reduce phases. There are two main techniques to estimate the performance of workloads of parallel applications that do not take into account the synchronization delays. One such technique is Mean Value Analysis (MVA)[14,15]. MVA technique takes into consideration only task queueing delays due to sharing of common resources. Thus, MVA cannot be directly applied to workloads that have precedence constraints, such as the synchronization among map and reduce tasks belonging to the same MapReduce job. Alternative classical solution is to jointly exploit Markov Chains for representing the possible states of the system, and queuing network models, to compute the transition rates between states [16,17]. However, such approaches do not scale well since the state space grows exponentially with the number of tasks, making it impossible to be applied to model jobs with many tasks, which is commonly the case of MapReduce jobs.
Vianna et al. in their work [
According to the model validation results, the proposed model produces estimates of average job response time that deviate from measurements of a real execution by less than 15%.
Although this model does not capture the dynamic resource allocation and it assumes a xed amount of threads to process map and reduce tasks per node as one of the input parameters, it has important advantages in comparison with previous models. First of all, unlike Herodotous's model that does not capture resource contention between tasks, this model is taking into account the queuing delays due to the contention at shared resources. Secondly, it is able to capture the synchronization delays introduced by the communication between map and reduce tasks (ARIA and Tetris are not considering this property of MapReduce job execution).
In this section, we analyze the architecture and components of Hadoop 2.x in order to identify the architectural changes that a ect the cost of executing MapReduce jobs. 3.1
To illustrate our approach and facilitate the explanations throughout the paper, we introduce a running example. Assume that we have n = 3; m = 4; r = 1, where n - total number of nodes, m - number of containers required for map tasks, r - number of containers required for reduce tasks. All nodes have the same capacity. Using this data, we will illustrate the main steps of our approach. 3.2
In the second version of Hadoop, the YARN module appeared and changed the architecture signi cantly. It is responsible for managing cluster resources and job scheduling. In the previous versions of Hadoop, this functionality was integrated with the MapReduce module where it was realized by the JobTracker component. The fundamental idea of YARN is to split the two major functionalities of the JobTracker, resource management and task scheduling/monitoring in order to have a global ResourceManager, and application-speci c ApplicationMaster. By separating resource management functions from the programming model, YARN delegates many scheduling-related tasks to per-job components and completely departs from the static partitioning of resources for maps and reduces, considering the cluster resources as a continuum, which brings signi cant improvements to cluster utilization. The YARN module consists of three main components:
RM runs as a daemon on a dedicated machine and
arbitrates all the available resources among various competing
applications. We will not go in detail of all components of
RM [
Application Manager Service that negotiates the rst container (logical bundle of resources bound to a particular node) for the Application Master. AMs are
Based on the core functionalities of YARN components, the general schema of job execution process is presented in Figure 1. The process starts when an application submits a request to the ResourceManager. The AM registers with the RM through AM Service and is started in the container that AM Service dedicated for it. Then, the AM requests containers from the RM to perform actual work. Once the AM obtains containers, it can proceed to launch of them by communicating to a NM. Computation takes place in the containers, which keep in contact with the AM. When the application is complete, AM should unregister from the RM. 3.3
For performance model construction it is necessary to understand in detail the resource request process. AM needs to gure out its own resource requirements, which can be: Static. If the resource requirements are decided at the time of application submission, and when the AM starts running, there is no change to the resource requirement that speci cation. In case of Hadoop MapReduce, the number of map tasks is based on the input splits (i.e., HDFS chunks), and the number of reducers on user-de ned parameter. Thus, the total number of mappers and reducers is xed before the application submission.
Dynamic. When dynamic resource requirements are applied, the AM may choose how many resources to request at run time based on criteria such as user hints, availability of cluster resources, and business logic.
Once a set of resource requirements is clearly de ned, the AM can begin sending the requests in a heartbeat message to the RM. Based on the task requirements, AM calculates how many containers it needs and requests them from the RM via a list of ResourceRequest objects. ResourceRequest object for running example from Subsection 3.1 is presented in the Table 1. In the ResourceRequest object, containers can have di erent priorities. Higher-priority requests of an application are served rst by the RM. There is no cross-application implication of priorities. According to the source code of MapReduce AM (package org.apache.hadoop. mapreduce.v2.app.rm; RMContainerAllocator class), MapReduce AM assigns a higher priority to containers needed for the Map tasks and a lower priority for the Reduce tasks' containers, with default priorities values equal to 20 and 10 respectively.
One thing to note is that containers may not be immediately allocated to the AM. This does not imply that the AM should keep on asking the pending count of required containers. Once an allocated request has been sent, the AM will eventually allocate the containers based on cluster capacity, priorities and the scheduling policy. The AM should request for containers again if and only if its original estimate changed and it needs additional containers. 3.4
There is another di erentiating characteristic in terms of how the scheduling of those resources happens:
Resource usage follows a static all-or-nothing model, when all containers are required to run together. For example, if AM asks for n containers, the job will start when AM receives exactly n containers.
Alternatively, resource usage may change elastically, depending on the availability of resources. In this case, the job starts even if AM receives less than required number of containers.
For cost model construction, it is necessary to understand the way to distribute containers for tasks within di erent nodes. By analyzing the source code of MapReduce (package org.apache.hadoop.mapreduce.v2.app.rm; RMContainerAllocator.java class), we observed that map and reduce tasks have di erent lifecycles that are presented in Figure 2 and Figure 3.
Vocabulary Used: pending ! requests which are not yet sent to RM scheduled !requests which are sent to RM but not yet assigned assigned ! requests which are assigned to a container completed ! requests for which the container has completed the execution
Furthermore, AM can do a second level of scheduling and assign its containers to whichever task that is part of its execution plan. Thus, resource allocation in YARN is late binding. The AM is obligated only to use resources as provided by the container, it does not have to apply them to the logical task for which it originally requested the resources. Thus, the MapReduce AM takes advantage of the dynamic two-level scheduling. When the AM receives a container, it matches that container against the set of pending tasks, selecting a task with input data closest to the container, rst trying data local tasks, and then falling back to rack locality. 4.
As a basis of our MapReduce performance model for
Hadoop 2.x, we decided to take the performance model for
MapReduce workloads proposed for Hadoop 1.x [
For the sake of simplicity, we consider a distributed cluster with a set of computing nodes equal to numN odes, all of them having the same technical characteristics. The workload is composed by N MapReduce jobs executing concurrently in the system. Each job has mi map tasks and ri reduce tasks. We are not dividing the map task into phases. As a partial sort is performed after each shu e, we group each pair of shu e and sort in a single subtask called shu esort. After all partial sorts are nished, a nal sort is executed, followed by the nal phase of reduce tasks that applies the reduce function. We group the nal sort and the reduce function into one merge subtask. Thus, according to our terminology, the reduce task is divided into two subtasks: shu e-sort and merge. The input parameters for our model are presented in Table 2. We consider 2 types of service centers (resources): CPU&Memory and Network. The overall number of task classes C is 3 (i.e., map, shu e-sort, and merge). We would like to emphasize the di erence between the residence and response time for a task. The average response time is the total time that task spends in the cluster. Meanwhile, the residence time of task class i on service center k is the average amount of time that task spends using the corresponding resource k during its execution. 4.2
To solve the queueing network model, we use the modi ed
Mean Value Analysis. An algorithm to solve the MVA for a
closed network system initially was proposed by Reiser and
Lavenberg [
Suppose a system with C task classes and K service centers. Let N~ be a vector, i-th component of which indicates the number of tasks of class i in the system; Sjk is the average demand of class j 2 C task on service center k 2 K (i.e., the average amount of time).
The main steps of the algorithm are presented in Figure 4, which consists of 6 main activities: A1-A6. We start by initializing the average residence time of each type of task at each service center and the average response time of each task in the system. Then based on the average response time of each individual task, the precedence tree is constructed. The next step is to take into account the e ects of the queuing delays by factors representing the overlap in the execution times of tasks belonging to the same job (intrajob overlap) and tasks belonging to di erent jobs (inter-job overlap). These overlap factors produce the new estimates of task average response time. The nal step is to apply the convergence test on the new estimates of average response time. In case that the convergence test fails, we return to the construction of precedence tree step trying to build a new, and more accurate precedence tree based on estimates of task response time obtained during the previous iteration. In case that current estimates are close enough to the previous ones, the algorithm nishes, and as a result, a nal job average response time is produced.
In the following subsections, we explain the activities of the modi ed MVA algorithm. In particular, we extensively explain our modi cation of precedence tree construction procedure in Subsection 4.2.2. 4.2.1
Initialization process consists of two sub processes that can run in parallel: initializing the average residence time of each type of task at each service center and the average response time of each task in the system. For initializing the residence time, we take the average of residence time from the history of corresponding real Hadoop job executions. To initialize the tasks response time, we can apply the following approaches:
Using sample techniques - taking the average of task response time from job pro le.
Obtaining from the existing static cost models, for
example, from Herodotou's cost models [
The second approach leads to faster algorithm convergence due to more accurate response time initialization and, as consequence, less number of iterations of the algorithm. In out model we use the second approach. 4.2.2
In a precedence tree, each leaf represents a task and each internal node is an operator describing the constraints in the execution of the tasks. We will consider a precedence binary tree built from 2 types of primitive operators: serial (S) and parallel-and (P ). The S operator is used to connect tasks that run sequentially, whereas the P operator connects tasks that run in parallel. An example of the precedence tree is presented on Figure 5.
The main goal of building the precedence tree is to capture the execution ow of the job, identifying the parallel or serial order of execution of individual tasks and their interdependencies. Based on new estimates for task response time, we rebuild the precedence tree at each iteration of the algorithm (the complexity analysis of building precedence tree procedure can be found in Subsection 4.3).
The precedence tree depends on the response time of individual tasks and is built using a task response timeline. Based on the obtained timeline, the precedence tree can be constructed uniquely up to graph isomorphism. To be able to distinguish the parallel and sequential task executions, we have to identify the beginning of a new phase in a timeline. By the phase we mean the maximum period of time, during which all tasks are executed simultaneously. Thus, tasks within the same phase are executed in parallel, while tasks from di erent phases are executed sequentially.
Based on the architectural analysis (see Section 3), the core assumptions and factors that in uence the timeline construction process can be divided into two subgroups: (1) related to the job scheduling, and (2) related to the resource management system.
The rst subgroup, related to the job scheduling, consists of the following factors: 1. We assume that RM uses the Capacity scheduler which is the default scheduler of the Hadoop YARN distribution. The fundamental unit of the Capacity scheduler is a queue. We assume that we do not have any hierarchical queues and we have only one root queue. Thus, resource allocation among applications will be in the FIFO order, i.e., the priority will be given to the rst application requesting the resources. 2. Due to architectural changes, some responsibilities of job scheduling are delegated to the AM. We have to
Input: M,R,N Output: TL fst->startTime; et->endTime; d->duration; sd->shu eDuration; an->assingedNode; g 1: for i := 1 to jN j do 2: T L[i] := ;; 3: end for 4: for m 2 M do 5: i := min(T L); m:an := i; m:st := min(T L[i]); m:et := m:st + m:d;
T L[i] := T L[i] [ fmg; 6: end for 7: if (slow start) then 8: border := T L[min(T L)]:et; 9: else 10: border := T L[max(T L)]:et; 11: end if 12: for r 2 R do 13: i := min(T L); r:an = i; r:st := max(T L[i]:et; border); for m 2 M do if (m:an <> i) then
r:d := r:d + mj R:sjd ; 14: 15: 16: 17: end if 18: end for 19: r:et := r:st + r:d; 20: T L[i] := T L[i] [ frg; 21: end for 22: Return T L; determine the way to distribute containers for tasks within di erent nodes. According to ndings in Subsection 3.4, map and reduce tasks have di erent lifecycles that we need to take into account during the timeline construction procedure. 3. We are assuming that AM will use requested containers for the same type of tasks as originally requested, thus we ignore the late binding functionality of AM.
The second subgroup, related to resource management, is composed of the following factors and assumptions: 1. Considering the nding in Subsection 3.2 related to di erent priorities for map and reduce tasks, we provide a container rst to map task and after to reduce task. 2. Assigning containers for map tasks mainly depends on whether we consider or not locality constraints (conguration parameter).
In our model, we consider a node locality constraints for map task and ignore locality constraints for reduce tasks. In case of ignoring the locality constraints, we distribute containers for tasks uniformly among nodes with the highest remaining capacity. Assuming that all nodes have the same capacity, we will take into consideration the occupancy rate and assign containers to the nodes with the lowest value.
Container allocation process for reduce tasks conform to the following steps:
Check for slow start: by default, schedulers wait until 5% of the map tasks in a job have completed before scheduling reduce tasks for the same job.
Check if all maps are assigned: if not, schedule reduce tasks based on the percentage of completed map tasks (conf. parameter). Otherwise, schedule all reduce tasks (map output locality is not taken into consideration, request asks for a container on any host/rack).
The last rule that we have to consider is how to divide the timeline into phases. All tasks within the same phase are executed in parallel, and tasks that belong to di erent phases are executed sequentially. It means that each start or end of a task indicates the start of a new phase.
As a summary, we present below our algorithm for the timeline construction, considering that map tasks have higher priority than reduce tasks. We start in lines 1-6 distributing containers for map tasks.In case the slow start is set, the beginning of the shu e-phase of reduce task will coincide with the end of rst map task on the node that has the lowest occupancy rate. Thus, shu ing starts as earlier as possible. In the opposite case, when we do not have a slow start, the shu e-phase of reduce task starts as late as possible (lines 7-11). Further, in lines 12-21 we distribute containers for reduce tasks.
Then based on the timeline we build a binary precedence tree. In order to reduce the maximal depth of precedence tree, we apply a balancing procedure for each P-subtree of it.
Example. Applying the above timeline construction algorithm to example from Section 3, we obtain the timeline, that is presented in Figure 6. Based on this timeline we are able to construct the precedence tree (Figure 7) 4.2.3
For a system with multiple classes of tasks the queueing
delay of task class i due to task class j is directly
proportional to their overlaps [
Rik = Hk max(Ti; Tj); where Hk = Pis=1 1i , s - is the number of child nodes The precedence tree is a binary tree. Thus, Hk = 32 ; 8k The intuition behind this formula is the response time for a parent node equal to the biggest child response time plus possible delay (multiplication by 23 ). 4.2.5
To solve the queuing network models we apply Mean Value
Analysis (MVA) [
During the convergence test, we are comparing the Total
Response Time from the previous iteration with the Total
Response Time received in the current iteration. In case
they are close enough (i.e., jRcurr Rprevj ), the
algorithm nishes. Otherwise, we return to the precedence tree
construction process and repeat activities A2-A6. We use
= 10 7, which is the recommended value for MVA [
We can nd the complexity of the proposed performance model by analyzing the complexity of the MVA algorithm and the complexity of the precedence tree construction.
According to [
The time complexity to build the precedence tree is equal to the complexity of timeline construction. The cost to construct this timeline can be identi ed by the time required to repeatedly search for the next task to nish until the termination of all the tasks.
Let C be the total number of tasks in the timeline and T be the total number of containers in execution.
C = allMapT asks + allShuf f leSortT asks + allMergeT asks; T = n max(pMaxMapsP erNode; pMaxReduceP erNode), where n - the number of nodes; pMaxMapsP erNode, and pMaxReduceP erNode - the maximum number of containers for map and reduce tasks correspondingly, pMaxMapsP erNode = j
T otalNodeCapacity SizeOfContainerF orMapT ask k pMaxReduceP erNode = j
T otalNodeCapacity SizeOfContainerF orReduceT ask k
Thus, in the worst case, the time complexity to build a precedence tree at each iteration is given by the search for m + r(m + 1) tasks in T containers, that is O(C T ) = O((m + r(m + 1)) (n max(pMaxMapsP erNode; pMaxReduceP erNode))), where m; r -is the number of map and reduce tasks in the job correspondingly. The computational cost of the whole solution: O(C2N2K)+ O(((m + r(m + 1)) (n max(pMaxMapsP erNode; pMaxReduceP erNode))) numberOf Iterations): As we can notice, the computational cost of the whole solution is dominated by the MVA algorithm that has quadratic complexity equal to O(C2N 2K). 5.
This section presents the results of a set of experiments we
performed with the proposed performance model. We
provide the validation results from a comparison of our model
(two approaches: Tripathi-based and fork/join-based) against
measurements of a Hadoop 2.x setup. For evaluation we
decided to use map-and-reduce-input heavy jobs (i.e.,
wordcount1) that process large amounts of input data and also
generate large intermediate data [
We performed a set of experiments analyzing the job response time in terms of the following parameters: the number of nodes: 4,6,8; the size of input data: 1GB, 5GB; the number of jobs that are executed simultaneously in the cluster: 1,2,3,4.
Each node in the cluster has the same technical characteristics: 2x Intel Xeon E5-2630L v2 a 2.40 GHz 128 GB Memory RAM
For each experiment we analyze the job response time xing two out of three parameters. Each experiment we repeated 5 times and then took the median of response time. 5.2
First, we present the response time for di erent number of jobs (1 and 4) that are executed simultaneously in the cluster on di erent number of nodes (4,6,8) with a xed size of input data. In all graphs we use blue continuous line to show results for the real Hadoop setup, red dashed - for Fork/join based approach, green 2 dots 1 dash line - for Tripathi based approach. Results for the input size equal to 1 GB and 5GB are presented in Figures 10 and 11, and Figures 12 and 13, respectively. Figure 14 shows the response time depending on the number of jobs (from 1 to 4) that are executed simultaneously in the cluster with a xed size (i.e., 4 nodes).
We can notice that the Fork/join based approach provides more accurate estimation of job response time with error between 11% and 13,5%, meanwhile the Tripathi-based approach shows an error between 19% and 23%. For 5GB input size, we obtain the bigger value of an error: 13.5% 1WordCount Example from the Hadoop distribution: https: //wiki.apache.org/hadoop/WordCount 4 5 6 7 number of nodes 8 4 5 6 7 number of nodes 8 and 23%, respectively. We observe that the accuracy of our algorithm depends on the number of map tasks and not necessarily on the size of input data. The bigger value of error is connected with the complexity (the maximal depth) of the precedence tree, which is increasing with the higher number of map tasks. In order to prove this hypothesis, we increase the number of map tasks without increasing the input data size. Thus, we reduced the default block size for the map task from 128M B to 64M B and repeated the experiments. The results for the input data size equal to 5GB and number of jobs equal to 1 are presented in the Figure 15. As showed by these results, experiments con rm our supposition, as we obtained the biggest values of errors: 17% and 25% for fork/join and Tripathi-based approaches, respectively. For reducing the maximal depth of the precedence tree and, as consequence, for decreasing the error, we balance it.
The Fork/join approach in our model produces accuracy
improvements over the original model for Hadoop 1 [
In conclusion, we can notice that the Fork/join based approach provides more accurate results than Tripathi-based, but with both approaches we overestimate the execution time. The cost model can be further ne tuned for improving the accuracy of the estimations by changing the calculation the overlap factors. 6.
In this work, we tackled the challenge of creating a
MapReduce performance model for Hadoop 2.x, which takes into
consideration queuing delays due to contention at shared
resources, and synchronization delays due to precedence
constraints among tasks that cooperate in the same job (map
and reduce phases). The modeling approach extends the
solution proposed for Hadoop 1.x in [
We validated our model against the measurements obtained from a real Hadoop setup for di erent number of jobs that were executed simultaneously. Our experiments showed the e ectiveness of our approach: the average error of job response time estimation for standard block size is in the range of 11% and 13.5%. Our model can be used for theoretically estimating of the jobs response time at a signi cantly lower cost than simulation and experimental evaluation of real setups. It can also be useful for critical decision making in workload management and resource capacity planning.
Our future plans focus on the tuning of provided performance model in order to decrease the error of job response time estimation. Furthermore, we are planning to extend our model to be able to estimate the amount of consumed resources for each task and the whole job. 7.
This research has been funded by the European Commission through the Erasmus Mundus Joint Doctorate "Information Technologies for Business Intelligence - Doctoral College" (IT4BI-DC).
The research has been partially supported by the Spanish Ministerio de Econom a, Industria y competitividad, grant TIN2016-79269-R. HadoopSetup Fork/join Tripathi 4 5 4 5 250 200 150 100 50 50