SQL-on-Hadoop processing engines have become state-of-the art, yet the skills required to tune these systems are rare in the job market. Automated tuning advisers can profile the low-level MapReduce jobs and propose appropriate tuning setups, but up-front tuning is time consuming and costly. In this demo, we present DejaVu. DejaVu integrates with Hive and effectively reduces the tuning costs by caching tuning setups for partial query plans: When the SQLon-Hadoop engine Hive compiles SQL queries into physical query plans, single MapReduce jobs tend to be similar between query plans. By recycling the tuning setups for similar low-level MapReduce jobs, DejaVu can effectively cut down the time spent profiling the TPC-H query workload in half, achieving similar impact on the performance of the jobs. While we employ Starfish in this demo, DejaVu can leverage any third-party MapReduce tuning adviser.
More than a decade after the publication of the seminal MapReduce paper [
The expertise required for allocating the right mix of physical resources and for
twiddling with the tuning knobs is rare. Also, Hive [
Automated tuning advisers for MapReduce frameworks rely on profiling
MapReduce jobs [
In this demo, we present DejaVu, a tool that integrates with the SQL-on-Hadoop engine Hive. Rather than contributing yet another tuning adviser, DejaVu employs thirdparty tuning advisers like Starfish, but caches tuning setups for partial query plans to avoid unnecessary job profiling. Our approach relies on two observations regarding the query plans compiled from SQL queries: (1) The individual MapReduce jobs within a query plan often have different resource requirements. (2) Since the jobs are generated, MapReduce jobs tend to be similar across query plans.
Regarding observation (1), we conclude that each job benefits from having a customtailored tuning setup, i.e., the configuration of its tuning parameters. In contrast, uniformly applying a tuning setup to all jobs requires maximum settings, and can be wasteful. Consequently, we pursue non-uniform tuning and assign one tuning setup per job.
Let us now consider observation (2). We regard two MapReduce jobs as similar from the perspective of tuning, if they have the same code signature. Intuitively, the code signature of a MapReduce job captures the SQL operators implemented by this job, as well as the expected size of the input. This information is available through the Hive query compiler. Our hypothesis (which we can confirm in our experiments) is that jobs that share the same code signature benefit from the same tuning setups. We therefore may recycle tuning setups for similar jobs, to reduce profiling time.
Let us consider the TPC-H queries compiled on Hive-0.13, which yields 106 MapReduce jobs3. For 71% of these jobs, there is at least one other job with the same code signature. Fewer than a third of the jobs have a unique code signature.
This inspires the idea of recycling tuning setups for similar jobs. Over time, we may
even be able to assign tuning setups for ad-hoc queries, which we have not encountered
yet. In fact, ad-hoc queries are prevalent in many query workloads [
As we show shortly, by non-uniform tuning and by recycling tuning setups, we can
effectively reduce profiling costs. Note that earlier, we have reported on a predecessor
tool called Chameleon [
We next introduce code signatures, and the code signature cache as the datastructure at the heart of DejaVu.
The SQL-on-Hadoop engine Hive compiles SQL queries into query plans. The lowlevel MapReduce jobs are annotated by Hive, and annotations are accessible via the Hive Java API. Each job carries a list of the physical query operators that are implemented by this job. Further, for each operator, an estimated input cardinality is given. Example 1. TCP-H query 1 is compiled by Hive 0.13 to a sequence of two MapReduce jobs. The first job is annotated with the operators Filter, Select, and GroupBy, each with cardinality 2. The job is further annotated with the operators TableScan, ReduceSink, and FileSink, each with a cardinality of 1. In our setting (c.f. Section 3), the estimated input size is 7.24GB.
We hypothesize that jobs with the same annotations have similar resource requirements. Therefore, they may be executed with the same tuning setups, even though their Java code differs. In our experiments, as discussed shortly, we are able to confirm this. 3 Here, we ignore approx. 50 auxiliary jobs that are executed only locally, rather than as a
MapReduce job, so they do not require a MapReduce tuning setup.
We capture the annotations by the code signature, a tuple stating the input size
(given in order of magnitude), and the annotated query operators with their cardinalities.
Example 2. We continue with TPC-H query 1. The code signature of the first job is
h9; FileSink : 1; Filter : 1; GroupBy : 2; ReduceSink : 1; Select : 2; TableScan : 1i
We have implemented the code signature cache in Java, and integrated it with Apache
Hive 0.13. We leverage tuning setups generated by the Starfish tuning adviser [
Our experiments were executed in a cluster with three physical machines. We isolate the master node on one machine, so that it does not influence the profiling of worker jobs. In particular, each machine has a Intel(R) Core(TM) i3-3240 CPU @ 3.40GHz, 4GB of RAM, 1TB of disk. All our profiling runs are configured with the out-of-the-box tuning setup that we refer to as “Hadoop Standard”. 4 https://www.cs.duke.edu/starfish/release.html, last accessed September 2019. 5 https://issues.apache.org/jira/browse/HIVE-600 job causes a cache miss in the code signature cache job causes a cache hit in the code signature cache Recurring code signatures. We have compiled the TPC-H queries on Hive 0.13. There is a considerable share of recurring code signatures, with 24 code signatures shared by 70% of the 106 MapReduce jobs.
We profile the 22 TPC-H queries in the order of the TPC-H benchmark specification. Figure 2 shows the profiling time per query. In total, over ten thousand seconds are spent on profiling. In Figure 2 We visually distinguish two groups of jobs: 1. Jobs which cause a cache miss in the code signature cache, 2. and jobs which cause a cache hit in the code signature cache.
We can observe that for the first TPC-H query, all jobs would cause a cache miss. Yet already for the second and third queries, we’d have cache hits, even though the savings are minor. With the code signature cache becoming more populated, we get more cache hits, and in some cases, some substantial savings in the profiling time. For instance, for queries Q6 and Q14, we can recycle all tuning setups from the cache. Thus, they require no profiling at all.
You may find more details about the caching mechanism and other results in our
full paper [
To make the internals of DejaVu transparent in our demo, we have built a web interface, as shown in Figure 3: – In the upper left, we can choose among the TCP-H queries and inspect their code. – Upon the push of a button, we have Hive compile a query into a query plan and submit it to be executed by Hadoop, as shown in the left. – The queries will have their jobs profiled in the case of a cache miss. In the case of a cache hit, the job will receive new tuning setup (generated by Starfish). – To the right, our interface visualizes the contents of the code signature cache. We can see how the cache is getting populated over time. – Also, to the right we can see the rank of run times and observe the benefits of the tuning cache.
The web-based GUI is for demonstration purposes only. At this point, DejaVu is fully operational and integrated with Hive. Visitors will be able to interact with DejaVu.
Acknowledgments We thank Herodotos Herodotou for all the support with Starfish. This study was financed in part by the Coordenac¸a˜o de Aperfeic¸oamento de Pessoal de N´ıvel Superior - Brasil (CAPES) - Finance Code 001.