The recently proposed Data Lakehouse architecture is built on open file formats, performance, and first-class support for data transformation, BI and data science: while the vision stresses the importance of lowering the barrier for data work, existing implementations often struggle to live up to user expectations. At fulfill the Lakehouse vision. Since building from scratch is a challenge unfit for a startup, we started by re-using (sometimes unconventionally) existing projects, and then investing in improving the areas that would give us the highest marginal gains for the developer experience. In this work, we review user experience, high-level architecture and tooling decisions, and conclude by sharing plans for future development.
1. Introduction [2] argues that the popular data warehouse architecture will soon be replaced by a new architectural pattern, the Data Lakehouse (DLH). A DLH is built on open file formats (e.g. Parquet), exceptional performance, and ifrst-class support for engineering (data transformation), analytics (BI) and inferential (data science) use cases. The vision of such architecture is first and foremost about lfexibility, making it possible for organizations to choose diferent ways to operationalize data depending on data straints.
This is particularly valuable for large organizations
where data democratization is crucial to achieve agility
[
Aside from security and compliance, the biggest argument in favor of the DLH is flexibility: diferent teams can use diferent tools to process data for diferent use cases. Practically, this implies that any DLH needs to support two diferent use cases: • Query and Wrangle (QW), referring to the scenario where users need to explore data and ask specific questions (e.g. counting how many marketing emails were opened in the previous month). Querying predominantly involves SQL, while Wrangling is often performed in Python. • Transform and Deploy (TD), referring to the scenario where users need to construct codedriven, reproducible data pipelines (DAGs) that generate new artifacts for downstream utilization. For instance, building a dashboard exposing the performances of marketing emails across different user demographics. Due to the distinct strengths and weaknesses of SQL and Python, the combination of both is often optimal.
Importantly, depending on the phase in which developers find themselves in the development cycle, their way to interact with the data can be either Synchronous or Asynchronous. While QW is de facto always synchronous, TD tend to be more nuanced and need to support both. • Synchronous is when a user issues a command (a SQL query, or a DAG run) and awaits for the results to come back. In this scenario, simplicity and fast feedback loop are the key goals [4]; • Asynchronous is when a command is issued (often by another system, such as an orchestrator) and the user is involved in monitoring the outcome at a later time. In this scenario, reliability, resilience and infrastructure ergonomics are the key goals. possibility of interacting with data (QW vs. TD) in both synchronous and asynchronous ways.
To achieve this, we designed Bauplan with the following general design principles in mind: • Serverless experience: to fully leverage the separation of storage and compute, developers should deal with as little infrastructure as possible. We propose to decouple data logic from execution to enable a “serverless” experience based on a declarative approach; furthermore, since data pipelines are functional in nature (output of parent nodes is input for children), a function-as-aservice deployment is prima facie a natural fit. 1 • Software development patterns : it is often the case that the only developers who can bring data applications to production are those who possess a special data engineering skill set. Empowering developers with more general coding skills to do impactful work on data is a fundamental piece of the DLH vision. Systems should allow users to use only familiar tools like SQL, standard Python, the CLI and Git. • Reproducibility and versioning: because the primary factor for building data products involves reproducible and versioned code pipelines, we embrace the idea that code provides a (mostly declarative) way to build data. Likewise, data is treated as code, adopting a life cycle including branching, committing, and merging. • Full Auditability: cloud clusters with long startup time and complex configurations encourage developers to resort to local development to expedite the feedback loop. However, this pattern exacerbates the challenges of software development (e.g. dependency management) while introducing potential security issues. We advocate for a cloud-first approach, ensuring that all work and access are centralized, auditable, and aligned with security and governance policies.
Bauplan, we wish to explain the rationale behind departing from Spark, which is widely regarded as the industry standard for analytics at scale and holds a significant position in numerous DLH implementations.
Given our discussion about the ideal DLH developer experience, we believe Spark falls short for several structural reasons. For instance, slow startup and execution makes Spark sub-optimal for synchronous operations, 1Note that we purposely use the term with some flexibility (Section 4.5). such QW. At the same time, the system has a notoriously steep learning curve [5, 6], both from an API and a debugging perspective: when thinking about TD, it is often hard to reason about it [7, 8]. If the DLH vision is truly about enabling a broader set of practitioners to perform data transformations, these systems are not necessarily the best design choice. 2To fully anonymize the dataset, we used the powerlaw package [14] for distribution fitting: final data are then generated by sampling from the distribution. 3https://ourworldindata.org/grapher/ historical-cost-of-computer-memory-and-storage?time=2010. .latest&facet=metric. • the data lake: while not obvious from the code itself, there is an object storage layer containing the raw data we are starting from: from the developer perspective, users would only interact with logical constructs, such as taxi_table; from an implementation standpoint, handling persistent
While a data lake is ultimately made of files, we wish to provide table-like abstractions to our users: by decoupling the actual storage of the data (the file s3://mybucket/taxifile.parquet ) from their logical function (the list of taxi trips in NYC), we can reuse the same code across data versions: every command that points to taxi_table can be executed over diferent versions of the table with just a configuration change (Section 4.3).
After considering Delta Lake, Hudi8, and Iceberg9 as possible formats to give table-like semantics to the object storage, we chose Iceberg mainly for three reasons: larger community support, full support for time-travel and versioning semantics, limited but increasing compatibility with Python10.
At the time of writing, major formats have full read / write support only for JVM engines (e.g. Spark, Presto, Dremio). Considering our working hypothesis about the Reasonable Scale and Bauplan focus on a serverless experience, two major tasks had to be completed to overcome object transparently is a huge component of the these constraints. First, when running a query over an DLH (Sections 4.2 and 4.3 below). Iceberg table, our code intelligence module needs to first • declarative data assets: we subscribe to the one- parse SQL into a table scan to obtain a dataframe-like query, one-artifact pattern popularized by dbt- object (Section 4.4.2); second, when materializing a data style transformations5: users define artifacts one asset from the DAG back to the data catalog, a Spark by one as SQL queries, and the platform builds up session is created to handle the Iceberg INSERT: followthe DAG based on parsing and naming conven- ing our no infrastructure principle, we created custom tion (Section 4.4). Importantly, no imperative- containers (Section 4.5) optimized for starting a Spark style DAG construction is needed: insofar as command with 300 milliseconds latency – as a result, the users implicitly link together parent and children materialization step looks no slower than running any nodes through their code, functions “are all you other Python function (as opposed to waiting for a Spark need”6; cluster to launch). • data expectations: it is best practice to test the tables produced by a DAG for statistical anomalies. 4.3. Data catalog and versioning This provides the foundation of the transformaudit-write pattern for data development (Section 4.3): just as in software we can debug, test and even run diferent versions of an application in parallel against production, automated testing and versioning becomes the foundation of the same approach for data pipelines.7
Software development best practices and tooling allow developers to work on code (new feature, bug fixing, debugging etc.) in a consistent and sand-boxed way: production code can be cloned, run, modified by developers, but running development code won’t leak into a production environment. Bauplan provides the same best practices for data pipelines, enforcing a transform-auditwrite pattern for all transformations. In particular, we picked Nessie 11 to provide a git-like semantics: Nessie versions an entire catalog at a time, so it is ideal for transformation use cases when multiple artifacts are afected at each run. Fig.4 depicts the basic versioning mechanism in the platform: 1. the user checkouts through Git a new branch in
his project (feat_1), to develop a new pipeline; 8https://hudi.apache.org/ 9https://github.com/apache/iceberg 10https://py.iceberg.apache.org/ 11https://projectnessie.org/ 5https://github.com/dbt-labs/dbt-core 6See also the Appendix for the full code example. 7Following the software analogy further, expectations are akin to integration tests, where a new component is embedded in an existing system, and checks are made to ensure the desired output is achieved. A related but diferent concept is unit tests, which instead work on manually fabricated input-output pairs, to test edge cases or important scenarios irrespective of the system actually seeing this input. Given our abstractions, Bauplan can easily accommodate both types of tests, especially considering that Python primitives for creating unit tests for tables are better than SQL.
Bauplan detects the Git context and creates a Nessie branch with the same name, feat_1, starting from the current production data in the lake main branch (grey node); now both the code (through Git) and the data artifacts (through
Nessie) are production-like and sandboxed; 3. Bauplan executes the DAG into an ephemeral branch (run_12): by executing each run “atomically” we can avoid persisting dirty DAGs – only when all steps and tests are executed successfully, we are allowed to merge the data into the current branch, making the artifacts 1 and 3 visible to any user with branch access (the obvious analogy here is the concept of transaction in databases); 4. when the merge on feat_1 is committed, the ephemeral branch run_12 is deleted.
There is a natural tension in modularity between code and compute: modular code is easier to test, re-use, rea
Once again, we remark that we chose to base the developer experience only on Git and the CLI. While we son about, on the other side, monolith compute is easier expect users to be familiar with Git, all the data version- to spin up, manage, orchestrate. The no infrastructure ing is handled behind the scene transparently. The user principle provides guidance on how to navigate the tradeis not expected to master Nessie or any of the technolo- ofs: on the code side, we subscribe to full modularity (e.g. gies involved. Instead, they are provided with a sand- dbt-style transformations), so that each node in the DAG box environment for data development with a familiar corresponds to one file that is runnable and testable in software-like semantics. isolation; on the compute-side, we let the system opting for modularity or monolith depending on the circum4.4. Code intelligence stances. In other words, the user is exposed directly only to the top layer in Fig. 3: it is the job of the code intelligence module (Fig. 2) to take as input the queries and functions defining a pipeline, together with parameters from the CLI, and produce as output first a logical plan of operations, and finally a physical plan to run the desired transformations. 4.4.1. From code to the logical plan After the pipeline code is ingested (e.g. Section A), the full project is snapshotted in an object storage and fingerprinted in a Postgres database, not dissimilarly from what happens for runs in Metaflow [15]: by assigning an id and immutable artifacts to each run, we guarantee reproducibility for auditing and debugging purposes – following the code is data principle, the same code on the same data version will produce identical results. After versioning, SQL and Python files are parsed: first, logical dependencies are extracted from implicit references – in our example, pickups is build out of another table (SELECT .. FROM trips), so we need to materialize nodes in the right order; second, environment details for Python functions are extracted – in our purely functional implementation, a decorator such as @requirements can be used to pin down the needed packages: because of our serverless setup (Section 4.5), the OS, container, and environment layers are handled by the system, leaving packages as the only degree of freedom left to control to ensure full reproducibility.
Finally, in our example Python is used only to run an expectation. There is no reasons why Python could not be used to also declare new tables starting from existing ones. In essence, transformations are functional mappers from set of tuples (rows in the “parent table”) to set of tuples (rows in the “child table”): as long as two languages can speak a common dialect over those tuples, they can operate together. 4.4.2. The execution plan The output of the parsing step is a logical plan (Fig. 3), so that the system knows which artifacts depends on existing Iceberg tables, which tests need to pass to consider the pipeline healthy, and what needs to be written back into the catalog as a result of running the DAG. The ifrst Bauplan version for executing such a plan was the simplest possible idea, i.e. just mapping the plan to an isomorphic execution, in which each node is executed by one (serverless and stateless) function. However, this naive implementation doesn’t optimize around an important feature of data workloads: at RS, computing artifacts is pretty fast, and the bottleneck is often moving data around. To make a concrete example, consider again our sample pipeline: there, the Python expectation is a Pandas function taking a DataFrame as input (the data artifact we are testing), and returning a boolean. Instead of running an Iceberg command first, a SQL query and then a Python function as three separate executions, we pushed down WHERE filters to obtain a smaller in-memory table, then run in-place the SQL logic and the Python expectation. This optimization results in 5x faster feedback loop even with small datasets, and avoid unnecessary spillover to object storage: notably, the user is not required to know any of the underlying implementation details.
When the execution plan is finalized, the computation
needs to happen in a fast, reliable, scalable way.
Following the functional definitions of pipelines, a serverless
runtime is the natural choice in terms of abstraction:
the user specifies what needs to happen, the Bauplan
platform runs the code in an optimized environment
where OS, container, and runtime are under its control
[16]. In recent years, serverless has become an overloaded
term, used to vaguely denote a cluster of features not
necessarily related [17, 18] and not necessarily important
for (or even, at odds with) data pipelines: scale-to-zero,
price-per-second, “infinite” and instantaneous
concurrency, stateless execution model [19]. We identified few
essential properties for our serverless platform:
• multi-language support with flexible
dependencies (Fig. 2): considering SQL code can be run in
a Python interpreter connected to object storage
(see duckdb below), the requirement can be
satisfied by a Python runtime allowing an arbitrary
combination of interpreter version and
dependencies12;
• runtime hardware allocation: the same
transformation logic should run with 10GB or 20GB of
memory depending on the underlying artifacts;
• data locality: given that data pipelines are first
and foremost about moving data, we need to
maintain function isolation at the runtime level but
allow for shared resources at the artifacts level
- moving data is slow and expensive, and object
storage should be treated as a last resort [
We evaluated AWS Lambda13, OpenWhisk14 and OpenLambda15 as of-the-shelf frameworks, but none 12Note how the function-first approach provides a level of control – i.e. specifying packages per function – that is impossible in conventional Spark applications. 13https://aws.amazon.com/lambda/ 14https://openwhisk.apache.org/ 15https://github.com/open-lambda/open-lambda of them fully satisfied the desiderata above: as typi- • bauplan run: asynchronous, DAG-long intercal use cases for serverless are micro-services and glue actions are handled through run; starting from code in cloud infrastructure, it is not surprising that the pipeline code in the IDE, issuing run starts existing tools would be sub-optimal for our scenarios. the intelligence and execution processes depicted Steps in data DAGs have almost opposite requirements in Fig. 3. As DAGs are modular and snapshotwhen compared to typical functions-as-a-service: startup ted at each execution, additional arguments altime is somewhat important, but since the bottleneck is low to replay an arbitrary DAG for debugging data reading and processing, we play in the 200-1000 and inspection: for example, -run-id 12 -m ms regime, not 0-200 ms; on the other hand, resources pickups+ will re-execute in a sandboxed way the required to compute aggregations require more fine- same code over the same data as the run with grained tuning. For these reasons, we invested, as a difer- = 12 , starting from the pickups artifacts and entiating feature, in building an orchestration and mem- running all its children. ory management layer to support workloads in which horizontal scalability is less important than vertical elas- With the goal of truly lowering the bar for data work, ticity and eficient data processing. the CLI-first approach is easy to learn and easy to ex
To support SQL, we leverage duckdb [
Finally, we wish to stress that containerization is an active area of research, with exciting possibilities ofered by new frameworks such as WASM [24]: through an ongoing collaboration with the research group behind SOCK [22], we are actively iterating on this component.
We started our journey designing Bauplan by
considering – and dismissing – two ways to build towards the
DLH vision: re-purposing existing Big Data tools, or
building a new platform from scratch. Mirroring the
Firebolt experience [
at https://github.com/BauplanLabs/quack-reduce. 17We plan to release a Lambda-based generic runtime for Python
functions that leverages object storage for caching.
We report the full code for the running example of this paper (Section 4.1), as schematically depicted in Fig. 3. Please note that steps are transformed into a DAG thanks to a simple naming convention: children tables refer to parents (Step 3 below referring to Step 1 table), while Python testing functions comply with the table_expectation syntax.
Step 1 (trips): read raw data (as stored under an Iceberg table taxi_table) for a target time window, and extract important columns into a new trips table. SELECT p i c k u p _ l o c a t i o n _ i d , p a s s e n g e r _ c o u n t as count , d r o p o f f _ l o c a t i o n _ i d FROM